Hello all, I have a job that processes 50 tables - 25 belong to finance, 20 belong to master data, 5 belong to supply chain data domains. Now, imagine the job ran for 14 hours and did cost me 1000 euros on a day. If I like to attribute the per day cost to the data domains, which of the below statist...| community.databricks.com
Hi everyone, Not sure how practical this idea is to implement but I'd love to have a space / area where there's voice channels. It could be something like an "Official Databricks Community" Discord? 😀 Alternatively, it could be something that the website offers i.e. meeting rooms. Basically, it would provide us with a space where we could talk to each other on a call and share our screens ☺️. I imagine this would enable us to have study groups and also to provide us with a place to...| New board topics in Databricks Community
Databricks notebooks are a powerful tool for data scientists and engineers to collaborate, explore data, and build machine learning models. This guide will help you get started with creating and using notebooks in Databricks. 📓Why Use Databricks Notebooks? Interactive Development: Write and execute code in real-time. Collaboration: Share notebooks with your team and collaborate seamlessly. Visualization: Easily visualize data with built-in charting tools. Integration: Integrate with variou...| New board topics in Databricks Community
Hi guys, 15 days ago, I posted that I passed the Data Analyst cert: https://community.databricks.com/t5/community-articles/zero-to-hero-data-analysis-certification/m-p/130951#M651. Since then, I've only managed to study the first week of the content in the blended learning for Data Engineering (life gets busy!): This weekend, mainly because I'm a nerd, I want to see if I can study the remaining Weeks 2, 3, and 4. If I'm feeling like a turbo-nerd, I'll squeeze in some practice papers or a...| New board topics in Databricks Community
I am planning to take the Gen AI certification. Any tips and guides to prepare for it.| New board topics in Databricks Community
Dear Databricks Support Team, Today, I attempted the Databricks Certified Generative AI Engineer Associate certification exam, but unfortunately, the exam was suspended within 10 minutes after starting due to environmental and behavioral issues. I kindly request you to investigate this issue and reschedule my exam at the earliest possible time. My exam was booked using the following email address: yaramasus@gmail.com. I would appreciate your prompt assistance in resolving this matter, as I am...| New board topics in Databricks Community
Here’s to another week of incredible contributions from our Databricks Community members! 💬✨ From sharing knowledge and resources to helping peers solve challenges, our members continue to make this space a go-to destination for learning and collaboration. Highlights from this week: @-werners- : Cannot deploy DAB with the job branch using a feature branch in Databricks – Initiated the conversation and guided the member to mark the accepted solution. @szymon_dybczak : ML course a...| New board topics in Databricks Community
Data Science Agent transforms Databricks Assistant into an autonomous partner for data science and analytics tasks in Notebooks and the SQL Editor. It can explore data, generate and run code, and fix errors, all from a single prompt. This can cut hours of work to minutes. Purpose-built for common data science tasks and grounded in Unity Catalog for seamless, governed access to your data. Since its launch two years ago, the Databricks Assistant has become an indispensable partner for dat...| New board topics in Databricks Community
Have you been using Databricks for Analytics and Business Intelligence? We’d love to hear from you! Take a few minutes to share your thoughts on Gartner Peer Insights in the Analytics & Business Intelligence category and claim a $25 gift card once your review is published.ATTENTION: This incentive is available only to current Databricks customers — partners, non-customers, and students are not eligible.Why Participate? Fast & Anonymous – The review process is simple, takes under 10 m...| New board topics in Databricks Community
Hi, I want to expose data to consumers from our non-UC ADB. Consumers would be consuming data mainly using SQL client like DBeaver. I tried SQL endpoint of Interactive Cluster and connected via DBeaver however when I try to fetch/export all rows of table it fails but succeds only for small number of rows upto 8k. What is the best way to expose consumer layer?| New board topics in Databricks Community
Hi, I am trying to query a table using JDBC endpoint of Interactive Cluster. I am connected to JDBC endpoint using DBeaver. When I export a small subset of data 2000-8000 rows, it works fine and export the data. However, when I try to export all rows of table, it fails by saying request timed out. My table has almost 1 million rows. I tried same using Python application by trying to export data in CSV and got the same result i.e. it succeeds for small number of rows but fails for entire table...| New board topics in Databricks Community
What are the new features we are going to see in the Dashboard this year?| New board topics in Databricks Community
Hello everyone! I need some help, unable to get cluster up and running. I did try creating classic compute but fails, is there any limit to use databricks community edition? Error here: { "reason": { "code": "CONTAINER_LAUNCH_FAILURE", "type": "SERVICE_FAULT", "parameters": { "databricks_error_message": "Failed to launch the Spark container on instance i-08e0677f2fe8bb9b9. [details] X_ContainerLaunchFailure: Failed to launch spark container on instance i-08e0677f2fe8bb9b9. Exception: Coul...| New board topics in Databricks Community
After creating a new workspace, if you come across Failed to get instance bootstrap steps from the Databricks Control Plane. Please check that instances have connectivity to the Databricks Control Plane. Instance bootstrap inferred timeout reason: GetRunbook_Failed In the Base64 encoded failure message: Failed to get runbook (may retry). Status code: 888. Content: HTTPSConnectionPool(host='eastus-c2.azuredatabricks.net', port=443): Max retries exceeded with url: /api/2.0/instances/vmboo...| New board topics in Databricks Community
Hi Community, I’m working on capturing Structured Streaming metrics and persisting them to Azure Data Lake Storage (ADLS) for monitoring and logging. To achieve this, I implemented a custom StreamingQueryListener that writes streaming progress data as JSON files using the code snippet below. To avoid generating multiple small files, I used coalesce(1) to reduce the DataFrame to a single partition so that Spark writes only one output file per batch. While this approach functions as intended,...| New board topics in Databricks Community
Hi Databricks team, I have registered for certification on Aug 31st for the Exam Date – September 6th 2025 IST. Unfortunately, I realized later that I opted out Databricks Certified Data Analyst Associate instead of the Databricks Certified Data Engineer Associate. Due to tight schedule of my work I made this mistake, got hurry to select for the Exam. Could you please help me to resolve this and reschedule my exam with Databricks Certified Data Engineer Associate. I have use my coupon co...| New board topics in Databricks Community
found that when I run the pipeline, it shows the message "'Cannot run pipeline', 'PL_TRNF_CRM_SALESFORCE_TO_BLOB', "HTTPSConnectionPool(host='management.azure.com', port=443) It doesn't happen on every instance, but I encounter this case often.| New board topics in Databricks Community
I have started a free trial and was trying to play around the AI models. When trying to serve the model, its giving me the below error - "Endpoint creation with provisioned throughput is not supported for your workspace." Do we know why this error is occuring and what would be the resolution thanks| New board topics in Databricks Community
Could someone help with current version and any update on when it's gonna change| New board topics in Databricks Community
Hello, Is there a sample code snippet that depicts end-to-end OIDC flow - imagine, there exists a service principal, interactive user who connect to an sql warehouse, get authenticated, and run some sql queries as part of a python script (jdbc/odbc) for example...| New board topics in Databricks Community
The digital credentials for the Databricks Certified Data Engineer Associate exam are missing. The certification was successfully passed on August 31, 2025, but the digital certificate or badge has not been received via email. According to the FAQ, credentials are typically issued within 48 hours. S...| community.databricks.com
Hi guys! I'm having a problem at work where I need to process a customer data dataset with 300 billion rows and 5 columns. The transformations I need to perform are "simple," like joins to assign characteristics to customers. And at the end of the process, I need to save a .csv file to S3. Currentl...| community.databricks.com
Hi all, TheOC here with my first of (hopefully!) many blogs on the Databricks Community. I'm hoping in this series to share quick, practical tips to help you get the most out of Databricks. Today's topic is: Widgets. If you're anything like me, you've also fallen into the trap of building notebooks ...| community.databricks.com
Hi team, My Databricks Certified Data Engineer Associate exam got suspended before 10 minutes left to submission. I have answered all the questions and was reviewing them. My exam got suspended due to eye movement. I was not looking away from laptop screen. And moreover I answered all my questions ...| community.databricks.com
I am trying to connect with Mongodb from databricks which is UC enabled, and both the mongodb and databricks are in same VPC, I am using the below code, df = ( spark.read.format("mongodb") .option( "connection.uri", f'''mongodb://{username}:{password}@{cluster_uri}:27017/{database}?authSource={datab...| community.databricks.com
Hello all, Is it possible to persist Databricks job name into the Brooklyn audit tables data model when when a Databricks job calls DBT model? Currently, my colleagues persist audit information into fact & dimensional tables of the Brooklyn data model. This data model has job run id but not the job ...| community.databricks.com
Databricks Community is an open-source platform for data enthusiasts and professionals to discuss, share insights, and collaborate on everything related to Databricks. Members can ask questions, share knowledge, and support each other in an environment that ensures respectful interactions.| community.databricks.com
Authors: Lara Rachidi & Maria Zervou Introduction Welcome to our technical blog on the challenges encountered when building and deploying Retrieval-Augmented Generation (RAG) applications. RAG is a GenAI technique used to incorporate relevant data as context to a large language model (LLM) without t...| community.databricks.com