ChatBI: The Complete Guide to Conversational Business Intelligence
A practical enterprise guide to ChatBI: how natural-language questions become verified answers, what a semantic layer contributes, how access control and auditability work outside the model, how to handle ambiguity, and how to evaluate a system before committing.
Most enterprise ChatBI evaluations fail in the same place. The demo works, the pilot looks promising, and then a business user asks a question whose answer depends on how "active customer" is defined, and the system returns a confident number that disagrees with the finance dashboard. The language model is rarely the cause. The system was never built as a governed analytical product.
This guide treats ChatBI as an architecture and governance problem rather than a model-selection problem. It defines ChatBI and separates it from the terms it is confused with, traces how a natural-language question becomes a verified answer, compares ChatBI with dashboards and text-to-SQL tools, and examines what a semantic layer contributes, how access control and auditability work outside the model, how to handle ambiguity and refusal, and how to evaluate a system before committing. It is written for IT decision-makers, data architects and analytics leads running an enterprise evaluation, and it treats vendor performance claims as claims to be tested against your own data.
What Is ChatBI and How Does It Relate to Conversational BI and NL2SQL?
The term ChatBI appears in the research literature as a specific system design, not a generic label. The VLDB paper ChatBI: Towards Natural Language to Complex Business Intelligence SQL describes a system that "employs all the modules introduced in this paper, matching Multi-Round Dialogues and performing Single View Selection through a phased processing flow," in which a language model first outputs JSON that is then used to generate SQL (ChatBI, VLDB). Two things matter in that definition: the system handles multi-round dialogue rather than single questions, and it constrains generation through an intermediate structured representation instead of asking a model to write SQL directly.
That gives a working definition. ChatBI is an analytical application in which a user asks a question in natural language, the system resolves the question against a governed data model, generates and validates a query, executes it under the user's own permissions, and returns a result with enough provenance for the user to judge it. The language model is one component. The governed model, the validation loop and the authorization path are the others.
The three terms describe different layers of that stack:
NL2SQL is the translation technique — converting a natural-language question into a SQL query. It is a research problem with public benchmarks (Spider, BIRD) and a substantial literature on its failure modes. A tool that only does NL2SQL has solved one component of ChatBI.
Conversational BI is the interaction pattern — a chat surface over data, usually with follow-up questions, clarification and session context. It describes how the user experiences the system, not how the system is governed.
ChatBI is the integrated product: conversational interaction, NL2SQL generation, a semantic layer that defines business terms, and an authorization and audit path that operates outside the model.
The distinction is not academic. A vendor can ship conversational BI over a semantic model without any NL2SQL generation at all — Looker's Conversational Analytics, for example, is "grounded in the Looker semantic modeling layer" and answers questions against modeled Explores rather than generating free-form SQL against raw tables (Looker documentation). Another vendor can ship NL2SQL generation with a thin semantic layer. Both may be marketed as ChatBI. The architecture determines what you can govern, and the next section explains why.
How the Natural-Language-to-Answer Pipeline Works, Step by Step
The most detailed public account of a production enterprise pipeline comes from LinkedIn's internal Text-to-SQL chatbot, described in Text-to-SQL for Enterprise Data Analytics (arXiv). Its architecture is a useful reference because the authors report what each stage contributed and what failed without it.
Stage 1 — Question interpretation. The system receives the user's question and, in a multi-turn setting, the conversation history. The first job is to decide what kind of question it is: a new question, a follow-up that modifies a previous query, or a request for clarification. The ChatBI paper identifies this as the hardest part of multi-round dialogue — the model "needs to first determine whether a given sentence is part of a MRD [multi-round dialogue] or a SRD [single-round dialogue]" (ChatBI, VLDB).
Stage 2 — Context retrieval. Enterprise schemas are too large to place in a prompt. LinkedIn reports that its data lake contains millions of tables, and that the system narrows the candidate set using table clusters derived from three months of user-table access history, then retrieves example queries and domain knowledge by semantic similarity to the question (arXiv). The retrieval stage is optimized for recall — casting a wide net — because a table that is never retrieved cannot be used.
Stage 3 — Schema ranking. Retrieved tables are ranked, and only the top candidates are passed to the query writer. LinkedIn reports that a column ranker operating on the top-ranked tables improved precision, and that the ranking depth is a trade-off: "higher values improves context ranker's recall, but also increases token usage, slowing down the query writer and leading to errors" (arXiv). The paper reports table recall of 78% and column recall of 56% for its full configuration.
Stage 4 — Query generation. The query writer receives the selected tables and columns, relevance scores, examples, domain knowledge and jargon, and returns assumptions, a query, an explanation, and the tables and columns it used. Returning the assumptions is a design choice worth noting: it gives the user something to check.
Stage 5 — Validation and repair. The generated query is checked for compilation errors and for hallucinated tables and columns, and the loop runs at most twice. LinkedIn reports that this reduced schema hallucination from 23% to 1% and compilation errors from 34% to 4% (arXiv). The validation is mechanical — it checks names against an index and syntax against the query engine — not a judgment about whether the query answers the question.
Stage 6 — Execution and presentation. The query runs, and the result is rendered. In a governed system this is also where row-level and column-level policies apply, because the query executes under the user's identity rather than a service account.
The pipeline view matters because it locates the failure modes. LinkedIn's human evaluation of 124 questions found that 53% were rated 4 or 5 (minor issues or correct) and 77% were rated 3 or higher, with the most common issue being "Filter is incorrect" at 24%, while only 4% had an incorrect join (arXiv). Filtering errors are semantic, not syntactic — the query compiles and runs and returns a plausible number that answers a slightly different question. No amount of syntax validation catches them. That is the gap a semantic layer and an evaluation set are designed to close.
Dashboards, Self-Service BI, Text-to-SQL Tools and Full ChatBI: What Actually Differs
The four categories are often presented as a maturity ladder. They are better understood as different answers to the question "who is allowed to ask what, and how is the answer constrained?" A dashboard and a ChatBI system can both be correct; they fail in different ways.
Dashboards answer a fixed set of questions that an analyst anticipated. The author decides what is measured, how it is filtered and how it is displayed. The strength is that the definition is frozen at authoring time and reviewed before publication. The limitation is coverage: a question outside the dashboard's design requires a new dashboard. Dashboards do not generate queries at request time, so they have no query-generation failure mode — but they also cannot answer the question the user actually had.
Self-service BI gives business users a modeling and exploration environment — drag-and-drop fields, calculated measures, filters — over a governed semantic model. The user composes the query through a structured interface rather than free text. This constrains the query space to what the model exposes, which is a real governance advantage, but it requires the user to know which fields to combine. The skill floor is lower than SQL and higher than a question.
Text-to-SQL-only tools take a natural-language question and generate SQL against a database schema, with little or no semantic layer. They are the most flexible and the least governed. The public benchmarks illustrate the ceiling: on Spider 2.0, a benchmark of 632 real-world enterprise text-to-SQL problems over databases that often contain more than 1,000 columns, the best model's execution accuracy was 31% in April 2025 (Spider 2.0; LinkedIn paper). On BIRD, human data engineers and database students scored 92.96% execution accuracy on the test set, while the top reported model entry scored 80.04% (BIRD leaderboard). Benchmark conditions differ from enterprise conditions, but the gap between human and model performance on realistic schemas is the relevant signal.
Full ChatBI combines the conversational interface with a semantic layer, a validation loop and an authorization path. The semantic layer constrains what the model can generate; the validation loop catches mechanical errors; the authorization path ensures the query runs as the user. The cost is implementation effort — the semantic layer must be built and maintained — and the benefit is that the failure modes are bounded and auditable.
Dimension | Dashboards | Self-service BI | Text-to-SQL only | Full ChatBI |
|---|---|---|---|---|
Who composes the query | Analyst, at authoring time | Business user, via structured UI | Language model, from free text | Language model, constrained by semantic layer |
Coverage of unanticipated questions | Low | Medium | High | High |
Definition consistency | Frozen per dashboard | Governed by model | Depends on prompt and schema | Governed by semantic layer |
Primary failure mode | Question not covered | User mis-composes the query | Semantic error in generated SQL | Semantic layer gaps and ambiguity |
Governance surface | Report permissions | Model permissions | Database permissions | Semantic layer plus database policies |
The practical implication is that these categories are complementary. A mature deployment typically keeps dashboards for recurring reporting, self-service BI for exploration, and ChatBI for the long tail of questions that do not justify a dashboard. The comparison that matters is not which category wins but which questions each one is allowed to answer.
What a Governed Semantic Layer Contributes to Answer Reliability
A semantic layer is "a software layer that sits between your data warehouse and the tools that access it" that "defines what 'revenue' means, how 'active customer' relates to purchase history, and who can see specific data" (AtScale glossary). For ChatBI, it does three jobs: it gives the language model a bounded vocabulary, it fixes the definition of each metric so the same question returns the same number, and it carries the access rules that apply when the metric is queried.
The evidence that this matters comes from the LinkedIn ablation study. The authors compared configurations of their system and reported that "incorporating example queries, table clusters, and table / column attributes improved the percentage of correct or almost correct queries from 9% to 49% compared to using table schemas alone" (arXiv). The largest single drop in their ablation came from removing example queries: table recall fell from 76% to 60% and the quality score from 49% to 24%. Removing table and column attributes dropped the quality score to 11%. Schema alone — the text-to-SQL-only configuration — scored 9%.
That measurement comes from the ablation's context bundle — example queries, table clusters, and table/column attributes — not from a direct comparison of a governed semantic layer against a schema. A governed semantic layer is one way to supply that kind of context, but the 9% and 49% figures belong to the ablation's configurations. The mechanism is not that the model got smarter. It is that the model was given fewer wrong choices. A semantic layer that defines "revenue" as a specific calculation over specific tables removes the possibility of the model choosing a different column with a similar name.
Vendor documentation describes the same architecture from the product side. Snowflake's Cortex Analyst "uses Semantic Views to understand your data and generate" queries, and its API returns an interpretation of the question alongside the SQL — "We interpreted your question as ..." — so the user can see the assumption (Snowflake documentation). Looker's Conversational Analytics is grounded in LookML, and its data agents support "authored context that includes custom instructions, verified queries, business glossaries, and persona frameworks to enforce guardrails and improve accuracy" (Looker documentation). Power BI's Copilot supports "verified answers" — responses a model author configures and validates — and AI instructions that define organization-specific terms, such as defining "busy season" as June to August when the semantic model contains no such term (Microsoft documentation).
The semantic layer is also where the platform's own modeling work connects to ChatBI. Singdata's lakehouse positions a unified architecture as the substrate for AI-driven analytics, and a semantic view built on that substrate is the natural place to define the metrics and access rules a ChatBI layer consumes; the platform's semantic views guide covers how those definitions are modeled and governed. The architectural point is that the semantic layer should be built once and shared by dashboards, self-service BI and ChatBI, so that all three return the same number for the same question.
Two cautions follow. First, a semantic layer is only as good as its coverage; a question about a metric that is not modeled falls back to schema-level generation and inherits the 9% problem. Second, the layer must be maintained — a metric definition that drifts from the warehouse's actual data is worse than no definition, because it is trusted.
How Access Control and Auditability Are Enforced Outside the Language Model
The question that decides a ChatBI deployment's exposure is not whether the product lists role-based access control. Almost all of them do. It is "when a user types a question instead of opening a dashboard, where is authorization actually enforced, and as whose identity does the generated query run?" (Singdata).
Natural language widens the permission surface in ways a dashboard does not. A generated SQL statement can reach tables a curated dashboard never touched. An agent can chain several queries in one turn. A service account used for indexing or compute can flatten per-user identity into a single privileged principal. A shared semantic layer can grant access to a view without requiring access to the tables beneath it. Each of these is documented behavior in at least one mainstream product, not a hypothetical.
The enforcement models fall into two families:
Warehouse-enforced. The generated query runs under the user's warehouse role, and row access policies and masking policies on base tables apply. Snowflake Cortex Analyst works this way: the query runs under the user's role, and row access and masking policies on base tables propagate to semantic views (Snowflake documentation; Singdata analysis). Databricks Genie enforces through Unity Catalog, with row filters, column masks and attribute-based policies.
Semantic-layer-enforced. The query is constrained by the model's access rules before it reaches the warehouse. Looker uses access filters, access grants and model sets tied to the user's permissions and user attributes. ThoughtSpot applies table-level row-level security, with a documented caveat that bypassing RLS on a model does not cover queries on views built on those tables.
The decisive detail in both families is the identity. A system that executes generated queries under a service account has effectively granted every user the service account's access, regardless of what the UI displays. A system that executes under the user's identity inherits the warehouse's policies, which is the only enforcement that survives a user who asks an unexpected question.
There is a specific trap in semantic-layer enforcement. Because a semantic view can grant access without requiring table grants, "the only thing standing between a user and unrestricted rows is the policy on the base table. If your team builds semantic views over tables that lack row access policies, owner's rights becomes a bypass rather than a control" (Singdata). The semantic layer is a convenience, not a substitute for base-table policy.
Auditability is the second half of the requirement. A governed ChatBI system should produce, for every answer, the generated query, the interpretation of the question, the tables and columns used, and the identity under which it ran. Demand these artifacts during evaluation: sample generated SQL for a representative question set, resolver or query logs showing the composed query, an audit trail export covering a test user's activity, and a documentation link for each permission claim made in the sales cycle. A vendor who cannot produce the documentation link is asking you to accept an undocumented assurance.
Handling Ambiguous Questions and Knowing When to Decline
Ambiguity is not an edge case in natural-language analytics. It is the normal condition. The CIDR 2026 analysis of text-to-SQL benchmarks found annotation error rates of 52.8% in BIRD and 66.1% in Spider 2.0-Snow, with error patterns including mismatches between the SQL's semantics and the question's intended logic, and questions that are themselves ambiguous (CIDR 2026). If expert annotators with the schema in front of them produce ambiguous questions at that rate, business users typing into a chat box will produce more.
Share of erroneous translations that were semantic rather than syntactic, measured on outputs from the CodeS NL2SQL model only (NL2SQL-BUGs, KDD '25). This is not a property of all NL2SQL systems.
A system that always answers is therefore not more capable; it is less honest. The design question is what the system does when the question underdetermines the answer. Three responses are defensible, and they should be applied in order:
1. Resolve from the semantic layer. Many apparent ambiguities are resolved by a definition. If "revenue" is defined in the semantic layer, the question "what was revenue last quarter?" is not ambiguous — it has a governed answer. This is the cheapest resolution and the reason semantic layer coverage matters.
2. Clarify with a bounded question. When the ambiguity is real, the system should ask a specific question with a small number of options, not a generic "could you clarify?" A useful clarification names the competing interpretations: "Do you mean gross revenue or net of returns?" A clarification that the user cannot answer is a failure; a clarification that offers two modeled alternatives is a resolution.
3. Decline with a reason. When the question cannot be resolved — the metric is not modeled, the data is not available, the question requires a judgment the system cannot make — the system should say so and say why. A decline that names the missing definition is actionable; a decline that says "I can't help with that" is not.
The failure mode to design against is the confident wrong answer. The NL2SQL-BUGs benchmark found that current language models achieve only 75.16% average accuracy at detecting semantic errors in generated SQL, and that they "consistently struggle with more complex error types, particularly subquery-related errors and those requiring deep database knowledge, such as join type mismatches and function-related errors" (NL2SQL-BUGs). A model that cannot reliably detect its own semantic errors should not be trusted to decide on its own whether a question is answerable.
Practical rules follow. Set a confidence threshold below which the system clarifies rather than answers. Log every clarification and decline, because the pattern of unanswerable questions is the roadmap for semantic layer coverage. And make the interpretation visible — Snowflake's "We interpreted your question as ..." pattern is a good default, because it lets the user catch a wrong interpretation before acting on the number.
A Criteria-Driven Evaluation Framework and Pilot Test Set Design
Vendor benchmarks measure the vendor's system on the vendor's questions. A pilot measures your system on your questions. The framework below is designed to produce a decision, not a score.
Two separate indicators from different studies, not a single scale. Left: agreement between the EX metric and human expert consensus (Cohen's kappa, 200 BIRD query pairs, three experts; FLEX, NAACL 2025). Right: share of benchmark examples with at least one annotation error pattern (BIRD Mini-Dev 498 examples; Spider 2.0-Snow 121 of 547 examples with open-sourced gold SQL; CIDR 2026).
The evaluation criteria. Five dimensions, each with a concrete test:
Answer correctness. Does the system return the right number for questions with a known answer? Test against a hand-verified ground truth, not against the system's own output.
Definition consistency. Does the same question, asked in different words, return the same number? Does the ChatBI answer match the dashboard answer for the same metric? A mismatch is a semantic layer defect, not a model defect.
Governance. Does a user without access to a table get a refusal rather than a number? Does the audit log show the generated query and the identity it ran under? Test with a deliberately restricted test account.
Operability. How long does it take to add a new metric to the semantic layer? How are wrong answers corrected — by editing a definition, or by adding a prompt example that may not generalize?
Latency and cost. LinkedIn reports its full configuration runs in under 60 seconds per question on average (arXiv). Measure your own; a system that takes two minutes per question will not be used for follow-ups.
Designing the pilot test set. The test set is the most important artifact in the evaluation, and it should be built before the vendor demo.
Source questions from real users, not from the vendor. Ask domain experts in each business area to contribute questions they actually ask, with the answer they expect. LinkedIn's internal benchmark collected 133 questions across 10 product areas from domain experts, with 167 ground truth tables, and noted that 60% of the questions had more than one valid response (arXiv). Multiple valid answers are normal; the test set should accommodate them.
Stratify by difficulty and failure mode. Include simple lookups, filtered aggregations, multi-table joins, time comparisons, and questions that require a business definition. Include at least a few questions that should be declined — a metric that is not modeled, a table the test user cannot access — because refusal behavior is part of correctness.
Verify the ground truth by hand. The benchmark literature shows that annotation errors are common even in curated datasets. A ground truth you have not verified is a test of the annotator, not the system.
Include paraphrase pairs. Ask the same question two ways and check that the answers match. This tests the semantic layer's coverage more directly than any single question.
Record the expected interpretation, not just the expected number. For each question, note which tables and filters the correct answer requires. This lets you diagnose a wrong answer as a retrieval failure, a ranking failure or a semantic failure.
Acceptance thresholds. Set them before the pilot and hold to them. A reasonable starting point: correctness on the verified set at or above the level your analysts achieve on the same questions; zero governance failures on the restricted-account tests; definition consistency across paraphrase pairs at 100%, because a metric that returns two numbers is not a metric; and a documented path for correcting a wrong answer that does not require a model change. Treat vendor-reported benchmark scores as claims to be validated in your environment, not as evidence about your data.
What to do with the results. The failures are more informative than the pass rate. Categorize each wrong answer as a retrieval miss, a ranking miss, a semantic error (wrong filter, wrong join, wrong aggregation) or an ambiguity that should have been clarified. The distribution tells you whether the gap is in the semantic layer, the retrieval configuration or the question set — and therefore what to fix before scaling.
Conclusion: ChatBI as a Governed System, Not a Model
The language model in a ChatBI system is the least differentiated component. The same frontier models are available to every vendor, and the benchmark literature shows that even the best of them produce semantic errors at rates that matter on realistic enterprise schemas. What separates a system that can be trusted with a business question from one that cannot is the architecture around the model: a semantic layer that bounds the vocabulary, an authorization path that runs the query as the user, a validation loop that catches mechanical errors, and an evaluation set that measures the failures the demo does not show.
Carry this checklist into the next vendor conversation:
Definition. Ask the vendor to define ChatBI and to say which components — semantic layer, NL2SQL generation, validation, authorization — are in the product and which are the customer's responsibility.
Identity. Ask, in writing, whose identity the generated query runs under, and get a documentation link.
Semantic layer. Ask how a new metric is added, who owns it, and how a wrong answer is corrected.
Ambiguity. Ask the system a deliberately ambiguous question in the demo and watch whether it clarifies, guesses or declines.
Audit. Ask for a sample audit trail showing the generated query, the interpretation and the identity for a test user's session.
Evaluation. Ask for the vendor's own test set methodology, then build your own before you sign.
A ChatBI deployment that passes those six checks is a governed analytical system. One that passes the demo but not the checks is a language model pointed at a database, and the difference will show up in the first quarter after rollout.
Frequently Asked Questions
The questions below come up in almost every ChatBI evaluation, and most of them are settled by a design decision rather than a model choice. Each answer is short and points to the section that develops the reasoning.
Is ChatBI the same as conversational BI? No. Conversational BI describes the chat interaction pattern. ChatBI is the integrated product that combines that pattern with NL2SQL generation, a semantic layer and an authorization path. A conversational BI product built on a semantic model may generate no free-form SQL at all. See "What Is ChatBI and How Does It Relate to Conversational BI and NL2SQL?"
Can a text-to-SQL model replace a semantic layer? It can generate queries without one, but the evidence suggests it should not be relied on to. LinkedIn's ablation found that schema-only generation produced correct or nearly correct queries 9% of the time, versus 49% with example queries, table clusters and column attributes (arXiv). The semantic layer is what closes that gap.
How accurate is ChatBI in practice? It depends on the schema, the semantic layer coverage and the question set. Public benchmarks show a wide range: the best model scored 31% execution accuracy on Spider 2.0 in April 2025, and BIRD's leaderboard lists human performance at 92.96% on its test set with model entries below that (Spider 2.0; BIRD). Benchmark conditions differ from enterprise conditions, so the only reliable number is the one from your own pilot.
Who should own the semantic layer? The business domain owner defines the metric; the data team implements and maintains it. The failure mode to avoid is a semantic layer owned entirely by IT, because the definitions then drift from how the business actually uses the terms.
What should happen when the system is not confident? It should clarify with a bounded question or decline with a reason, not guess. The NL2SQL-BUGs benchmark found language models detect semantic errors in generated SQL with only 75.16% average accuracy, which is not a reliable basis for self-assessment (NL2SQL-BUGs). See "Handling Ambiguous Questions and Knowing When to Decline."
Does ChatBI replace dashboards? No. Dashboards remain the right tool for recurring, anticipated reporting. ChatBI covers the long tail of questions that do not justify a dashboard. The two should share a semantic layer so they return the same number. See "Dashboards, Self-Service BI, Text-to-SQL Tools and Full ChatBI: What Actually Differs."