Rolling out Databricks Genie to 5,000 users: accuracy, scale and Teams

A data engineer recently asked a public forum a question we expect a lot of teams to be asking soon. Their company wants a Databricks Genie agent for around 5,000 users, and leadership has three requirements: answers that are at least 90% accurate, no hallucination or drift as more tables are added, and a design that extends from one business unit to many. Also, can it run in Microsoft Teams?

The replies were a good mix of practical advice and well-earned cynicism. One person described their org running weekly Genie office hours, getting great demos, and three months later nobody was using it. Another pointed out that asking a probabilistic tool for a guaranteed answer is a strange requirement to begin with.

Both points are fair, and neither is a reason not to do it. Here’s how we would approach that brief, piece by piece.

First, rewrite “90% accurate” into something you can measure

“90% accuracy” is not a requirement until you say 90% of what. Accurate on which questions, judged by whom, measured how often?

The workable version looks like this:

A benchmark set. A few dozen real questions per domain, collected from the people who will use Genie, each paired with the SQL or the number a human analyst agrees is correct. Genie has a built-in benchmarks feature for exactly this: you store questions with ground-truth answers and run the whole set to get a score.
A threshold per domain, not per company. Finance at 95% on its benchmark set and marketing at 85% is a more honest report than one blended number for 5,000 people.
A re-run on every change. New table, new instruction, new example query: run the benchmarks again. This is how you catch the “degrades as more tables are added” problem before users do.

Nobody can promise zero hallucination. What you can promise is a measured accuracy figure, on questions the business wrote, that you re-check before every change ships. That is a promise a team can actually keep.

Accuracy comes from the data model, not the prompt

The single most repeated piece of advice in that thread, and in our experience, is that Genie’s answer quality is decided before anyone types a question.

Describe every table and column. Genie only knows what your metadata tells it. A column called amt_2 with no description is a guess waiting to happen.
Use metric views for the measures that matter. Unity Catalog metric views let you define a measure, its business rules and the dimensions it can be sliced by, once. Several practitioners called this the biggest single lever for accuracy, and it matches what we see. When “net revenue” is defined in one governed place, Genie stops reconstructing it from raw tables differently each time.
Declare primary and foreign keys. Joins are where text-to-SQL goes wrong most often. Telling Genie how tables relate removes a whole class of errors.
Give it instructions and example queries. A handful of representative questions with correct SQL teaches the space your business rules faster than any amount of description.
Keep the tables as gold as possible. Clean, modeled, purpose-built tables, not the raw ingestion layer.

One space will not serve 5,000 people

This is the architectural point that matters most for scale. Databricks’ documentation caps a Genie space (now called a Genie Agent) at 50 tables, views or metric views, and its best-practice guidance is to aim for five or fewer. Even without the cap, a space stuffed with every table in the company gets less accurate, not more, because every extra table is another way to pick the wrong one.

The pattern that holds up:

One focused space per domain. Sales, finance, supply chain, support, each with a small set of well-modeled tables and its own benchmarks. Specialists beat generalists.
A front door that routes. Users should not have to know which space to ask. Databricks’ newer Genie One experience, or an orchestrator you build on the Genie Conversation API, can sit in front of the domain spaces and route each question to the right one.
Grow table by table. If a domain genuinely needs more tables, add them a few at a time and watch the benchmark score. If it drops, split the space.

Scaling from one business unit to many is then a repeatable process, not a redesign: build the next domain space on the same pattern, with its own benchmark set, and register it behind the front door.

Roll out in waves, not to 5,000 at once

Every experienced reply in that thread said the same thing. Start with one business unit and around 50 users who are engaged and know the data. Read the questions they actually ask, because they will not be the questions you expected. Add the good ones to the benchmark set, fix what fails, then widen to a few hundred, then further.

This also addresses the “nobody uses it after three months” failure. Adoption dies when early users get wrong answers and quietly go back to emailing an analyst. A small pilot with a feedback loop finds those wrong answers while there are still only a few people to win back.

Measure it in production, not just before launch

Benchmarks tell you how Genie does on the questions you anticipated. Production tells you how it does on everything else. Two practical additions:

Review real conversations. Sample what users ask and what Genie answered, especially questions with negative feedback.
Automate the checking where you can. Databricks publishes an open-source app called Genie Workbench that scores Genie spaces and runs benchmark-driven optimization. It comes from Databricks’ solutions team but is not covered by Databricks support, so treat it as a useful accelerator rather than a supported product. Some teams also log Genie conversations through the API and evaluate them with MLflow-based judges for regression testing.

And yes, it works in Microsoft Teams

Databricks provides a Genie app for Microsoft Teams. Users can message it directly or mention it in a channel, and a channel owner can pin a specific Genie space to a channel so a finance channel always asks the finance space. At the time of writing it is labeled Beta on AWS and GCP and Public Preview on Azure, and it needs admins to enable the relevant previews, so check the docs page for your cloud.

One commenter warned that Teams lacked the richer agent features of the Genie web UI; a Databricks employee replied that the Teams app now uses the same API. Our advice: pilot in Teams with your first wave, because meeting users where they already work is a big part of adoption, but keep the web UI available for power users who want to inspect the generated SQL.

The short version

Turn “90% accurate” into benchmark sets per domain, re-run on every change.
Put your effort into metadata, metric views, keys and example queries.
Build many small domain spaces behind one routing front door, not one giant space.
Pilot with one business unit and 50 engaged users, then widen in waves.
Keep measuring in production, and use Teams to reach people where they work.

None of this is exotic. It is the same semantic-layer and rollout discipline that makes any BI tool trustworthy. Genie simply makes the cost of skipping it visible much faster.

If you are planning a Genie rollout and want a second pair of hands on the readiness work, pilots or governance, see our Databricks Genie implementation page.