11. Designing Chat UIs for Uncertain AI Answers

In the previous ten posts, I analyzed the most common SQL generation errors.

I used BIRD and Spider benchmarks, tested several models, and grouped wrong answers into types such as COLUMN_BINDING, VALUE_BINDING, and DERIVED_METRIC. As I kept looking at those SQL errors, I realized I wanted to check something different.

“So how should generated SQL and supporting context appear in the UI?”

I had been focused on accuracy and had missed what users actually need from the interface. In a workplace data analysis tool, the UI has to make it clear where numbers come from. Without traceable context, users end up re-running the SQL even when the result looks correct.

For DataNexus, I treated the LLM’s text output as only one layer. SQL, source, permission scope, validation state, and Fallback state had to be displayed alongside the final response. That is what makes the answer usable in meetings and decisions.

Benchmarking answer evidence patterns across major products

In May 2026, I reviewed publicly available documents and demo screens from major Korean and international products.

International products:

• Databricks Genie
• Snowflake Cortex Analyst
• Strategy(MSTR) AUTO
• ThoughtSpot Spotter 3
• Vanna.ai

Korean products:

• Qurify(SmartMind)
• MISO(GS Neotek & 52g)

I focused on one question: “What does a user need to check before using this answer at work?” Each product answered it differently, but none of them relied only on a single-line text response.

• Databricks Genie uses Trusted assets to mark answers that use verified SQL queries or functions with a ‘Trusted’ label. Users ask questions in natural language inside Genie Space and inspect tables, columns, guidelines, SQL queries, and other context in the Data and Instructions tabs. Unity Catalog permissions also apply, so answers use only data the user is allowed to see.

• Snowflake Cortex Analyst has a REST API response structure that directly informed our screen design. The response content types are text, suggestion, and sql, and SQL content may include fields such as verified_query_used. The API also returns warnings and response_metadata.question_category, so ambiguous questions that are hard to turn into SQL can receive alternative questions as suggestion content.

• Strategy(MSTR) AUTO documentation and demos showed how features like derived metrics, selectors, auto narratives, and auto answers are evolving. The demo’s Interpretation area also helped with the DataNexus Trace UI: it showed which Metric and Dimension the user’s question mapped to.

• ThoughtSpot Spotter 3 came closest to our goal of showing verification context in the UI. Spotter 3 handles not only structured data warehouse data, but also information scattered across work tools such as Slack, Salesforce, Jira, and SharePoint. It brings scattered context into the answer, lets the user inspect the result, and then transitions into follow-up analysis. That flow helped shape the DataNexus design.

• Vanna.ai focuses on the learning loop and permission handling. According to the Vanna 2.0 docs , the LLM is connected to tools for SQL execution and chart generation, and Tool Memory stores successful question-SQL-result pairs in a reusable form. User permissions are applied across the whole pipeline. Progress state, SQL code, tables, charts, and natural-language summaries all appear in one flow.

For Korean products, I relied on public product pages and release notes.

• Qurify is a Text-to-SQL product for interacting with databases in natural language. As of version 0.4.4, it shows generated SQL in the UI and lets users edit, run, and copy it. That pattern for follow-up questions that cite execution results also shaped the DataNexus design.

• MISO is more of an enterprise AI platform and workflow app builder than a pure NL2SQL product. I used its public report automation and document search cases as benchmarks while designing the operational AI screens for DataNexus.

I used those benchmarks to set a simple direction for DataNexus: keep SQL, source, validation state, permission scope, and Fallback state visible so users can trust the final response.

I also revisited the semantic layer. A typical semantic layer maps business terms and metric definitions to SQL. DataNexus Glossary, Ontology, and Knowledge Graph add business rules and concept relationships on top of table- and column-level metadata. If those relationships remain hidden in the system, users have to go back to documents. The answer card needed a way for users to explore them directly.

Knowledge graphs for source verification

DataNexus defines ‘answer evidence’ as a network that connects the concepts used in the answer to sources, related tables, and columns. At first, I also considered a separate analysis screen where users could follow nodes and edges in a knowledge graph. But in real-world data operations, users care less about graph visuals than about what they can verify quickly.

A knowledge graph in the UI needs to make these checks immediate.

• Which business concept was this answer based on?
• Which physical data, such as tables or columns, is that concept connected to?
• Who defined this relationship, and where is the documentary evidence?
• If the metric definition changes, which answers are affected?

While traditional keyword search can detect terms such as “net revenue”, it struggles to infer how that metric is tied to gross revenue, returns, discounts, and allowances. In business queries, the contextual links between concepts matter more than simple term matching. DataNexus uses ontology and a knowledge graph to show those links in the UI.

In DataNexus, ontology acts as a reference map for standardized enterprise terms and relationships.

• Concept: customer / product / order / net revenue
• Relationship: a customer creates an order / net revenue is calculated from gross revenue
• Property: order timestamp, payment method, order status
• Rule: a refund can occur only after payment is completed
• Metric: average order value, year-over-year growth rate

The knowledge graph maps this reference map to physical data. Concepts become nodes, relationships become edges, and documents, tables, columns, and formulas connect into one network.

We are not trying to replace the RDBMS. It handles transactions, while the knowledge graph explains what that data means for the business. We wanted to show the SQL value and the knowledge graph’s business meaning side by side.

Detailed source access inside the knowledge graph

A knowledge graph interface is not enough if it only visualizes concept nodes. It also needs to expose the sources and metadata behind those concepts.

When a user clicks a node or edge, the supporting context behind that relationship should open immediately. For example, clicking the “net revenue” node should expand step by step into adjacent nodes, the Data Catalog Glossary entry, the source passage in the original document, and the mapped DB columns. This lets the user inspect the answer end to end.

Even a clear written definition of “What is net revenue?” is not enough; analysts still need to see the underlying source context.

Net revenue is the amount after subtracting returns, discounts, and allowances from gross revenue.

Even with high accuracy, verification is still needed.

• Where did this business logic come from?
• Is it the current enterprise standard?
• Which physical columns were used in the calculation?

So the answer card needs this metadata at minimum.

 - source: Data Catalog Glossary
 - term: net revenue
 - source_id: urn:li:glossaryTerm:net_sales
 - related concepts: gross revenue / returns / discounts / allowances
 - linked columns: orders.amount, orders.refund_amount, orders.discount_amount
 - validation: PASS

Users treat this metadata as a starting point and follow the lineage inside the knowledge graph.

Try the lineage below: click any node to open the evidence behind that step.

Limits of metric-centered answers

The benchmark review made one thing clear: result screens need verification context alongside the answer.

A typical request might look simple:

What was the total order count in 2024?

The response may seem straightforward:

The total order count in 2024 was 12,486.

The value may be correct and still leave analysts with questions such as:

• Which mart table was queried?
• What physical timestamp defined the 2024 period condition?
• Were cancelled and returned orders included in the count?
• Was this operational data or staging data?

Without context, a metric like “12,486” gives users no way to validate the result or defend it to executives. DataNexus places the SQL query, execution process (Trace), validation state, and access scope below the natural-language answer so users have something concrete to check.

SQL availability in the interface

When designing an NL2SQL UI, it is tempting to hide SQL completely to keep the interface readable for business users. The concern is that displaying SQL clutters the interface and exposes too much implementation detail. But enterprise users need transparency, so SQL cannot be fully hidden.

DataNexus does not bury SQL in internal logs. It implements SQL as a block that can be collapsed and expanded inside the answer.

• Primary natural-language answer layer:

The total order count in 2024 was 12,486.

• Secondary expandable SQL layer:

SELECT COUNT(*) AS order_count
FROM orders
WHERE order_date >= DATE '2024-01-01'
  AND order_date < DATE '2025-01-01';

The SQL block is collapsed by default. When number verification is needed, an analyst or DBA can open it immediately, rerun the query in the same environment, and isolate the error.

For users who are not comfortable reading SQL, the UI first shows an Interpretation summary that explains how the system parsed and mapped the user’s intent. This area helps users catch VALUE_BINDING errors.

Users do not have to read the full SQL to verify that their intent was captured correctly. If something is wrong, they can edit the conditions in place and rerun the query.

Before execution, an SQL Inspect panel previews a dry run of the query.

SQL Inspect

 - EXPLAIN summary: Index Scan
 - Estimated rows: 124,503
 - Full-scan warning: none
 - Estimated cost: low
 - Policy check: passed

Runtime Trace and process visualization

Trace shows which internal path handled the question. DataNexus does not rely on a single model call. The answer pipeline goes through ontology validation, catalog mapping, and question-type classification before routing the question to the appropriate path.

DataNexus then routes each request by question type.

• “What is the definition of net revenue?” -> query the catalog glossary immediately without running SQL.
• “What was the total order count in 2024?” -> run a SELECT statement against the data warehouse (DW).

The detailed Trace layer stores structured logs for audit and debugging.

• Database lookup Trace example:

 - Runtime Trace
 - trace_id: dnq_20260523_001
 - route: NL2SQL
 - source: DW/DM
 - source_id: mart.orders
 - semantic_context: Data Catalog Glossary + Ontology
 - verified_query: none
 - sql_execution: success
 - validation:
   - schema_compliance: pass
   - policy_check: pass
   - source_binding: pass
 - permission_scope:
   - user: sales_analyst
   - rls_policy: sales_region_allowed
   - masked_columns: []
 - fallback: false
 - warnings: []

• Data catalog lookup Trace example:

 - Runtime Trace
 - trace_id: dnq_20260523_002
 - route: Glossary
 - source: Data Catalog Glossary
 - source_id: urn:li:glossaryTerm:net_sales
 - sql_execution: not_applicable
 - validation:
   - source_binding: pass
   - policy_check: pass
 - permission_scope:
   - glossary_visibility: internal
 - fallback: false
 - warnings: []

sql_execution: not_applicable is not an error in this trace. It means the path was a metadata lookup that did not need SQL execution. In this case, the system verifies data integrity through validation.source_binding. The user-facing UI shows only summary indicators such as Source: DW/DM, Validation: PASS, and Fallback: False instead of complex logs.

Route separation by execution path

For operational stability, DataNexus tracks the runtime execution path (route) at a finer granularity. Even for an answer that passed system validation and received a ‘PASS’ label, the trust level differs depending on whether it reused a pre-verified query or executed Raw SQL generated at runtime.

DataNexus records this path and shows enough of it for users to see which route handled the answer. When the system is uncertain or detours into an exception path, users can see that state and respond accordingly.

Ambiguity reduction before answering

Showing context after the answer helps, but ambiguity also needs to be reduced before SQL generation. Real-world work questions tend to be abstract. Before generating SQL, DataNexus first shows ontology-based interpretation candidates and narrows the intent.

[Question] Show me 2024 net revenue

DataNexus checks the query conditions.

 ○ net_revenue_v1: gross revenue - returns - discounts - allowances
 ○ net_revenue_KR: Korean accounting standard, includes tax treatment
 ○ net_revenue_global: IFRS standard

[Run with this definition] [Choose manually]

DataNexus keeps the detailed logic chosen by the user in the session and conversation context. Internally, it uses this state structure.

{
  "persisted_context": {
    "accepted_metric": "urn:li:glossaryTerm:net_sales",
    "accepted_period": "2024",
    "accepted_filters": ["region = KR"],
    "rejected_interpretations": ["gross_sales_only"],
    "scope": "current_thread_only"
  }
}

The scope: current_thread_only setting keeps the selected metric criteria and filter conditions active for the current conversation. The user can continue the conversation with the same assumptions. The state is named persisted_context so the schema reads as session state rather than a generic carryover flag.

Permission scope as answer evidence

Even if the source is correct, the answer must not include information outside the user’s security permissions. The same natural-language question can return a different data scope depending on department or role. Trace records permission scope along with the source.

permission_scope:
  - user: sales_analyst
  - allowed_domains: [Sales]
  - row_level_filter: region IN ('KR-Seoul', 'KR-Gyeonggi')
  - masked_columns: [customer_phone, customer_email]

The standard UI omits the granular details of row-level security (RLS) filters and masking policies. Instead, the backend records them so data security auditors can inspect them later in the Inspect layer.

Fallback state transparency

Silent infrastructure failures are hard to catch: stale cached data can make an outage look like a successful response.

Rather than burying infrastructure issues in a normal answer, DataNexus surfaces database communication outages and query timeouts directly in the UI.

 - Runtime Trace
 - source: fallback demo response
 - sql_execution: failed
 - validation: FALLBACK
 - fallback: true
 - warnings:
   - DB connection failed. Demo response was used.

In production, DataNexus separates FALLBACK · demo_response from FALLBACK · sql_execution_failed. The UI must clearly distinguish between a simple demo mode and a real infrastructure failure. When access constraints or missing data sources prevent execution, the system catalogs the state as ABSTAIN rather than a generic fallback.

Source visibility and business usability

In business queries, the most important reference point is the data catalog. A KPI such as “net revenue” can include or exclude different items depending on the company’s business domain and accounting rules.

A typical DDL schema defines only physical column structures, such as amount or discount_amount. It does not carry business context. The term definition tied to actual business rules must come from the official data catalog at runtime.

 - source: Data Catalog Glossary
 - term: net revenue
 - source_id: urn:li:glossaryTerm:net_sales
 - owner: Sales Data Domain

When the owner field and glossary ID are shown together, users can check the result against the enterprise standard instead of relying on guesswork.

Multi-tab structure inside the answer

A knowledge graph or lineage view helps verification, but users do not need to start from a complex network map every time. Most users prefer to see the summary number first and then trace the details backward.

DataNexus groups summary, SQL, catalog, and graph views into tabs inside one answer card.

• [Default screen] Summary view

The total order count in 2024 was 12,486.

• [Tab 1] SQL view

SELECT COUNT(*) AS order_count
FROM orders
WHERE order_date >= DATE '2024-01-01'
  AND order_date < DATE '2025-01-01';

• [Tab 2] Graph view

Order
 ├─ has property: order_date
 ├─ has property: order_status
 └─ counted as: Order Count

• [Tab 3] Catalog view

 - Glossary Term: Order Count
 - Definition: Number of order records created within the selected period
 - Owner: Sales Data Domain

By default, the UI keeps the card slim and shows only summary information so all verification details are not shown at once. When number verification or domain-rule checking is needed, the user opens the lower layers one by one.

Approval workflow for conversational edits

When a user requests a knowledge base edit directly through the chat interface, such as “change this node type to strategic customer,” the system does not apply the edit immediately. It saves the requested change as a draft for review.

Enterprise systems cannot safely apply conversation results directly to the knowledge base. A metadata change can trigger a cascading effect across the downstream SQL generation pipeline and dashboard reports.

DataNexus puts staging and approval steps around conversational edit requests.

user edit request
  → create change candidate (Draft)
  → impact analysis
  → data owner review and policy check
  → apply to Glossary / Ontology master after final approval
  → resync related knowledge graph indexes

Recording approval steps and change history helps prevent incorrect edits from entering the knowledge base and helps preserve the integrity of data assets.

Separation of accuracy and runtime verification

In the pre-validation stage, we usually measure EX (Execution Accuracy), which checks whether the SQL generated by the model matches the result of the gold SQL. But in real-time runtime environments, there is no predefined Gold Answer, so pre-validation metrics cannot be used as-is.

A runtime PASS label is not just ‘EX passed with no SQL syntax error.’ It requires runtime checks for policy compliance and source validation.

 PASS
   - SQL execution engine has no errors and data catalog mapping is complete
   - schema policy check passed
   - data security policy and row-level security (RLS) filters passed
   - matches an official glossary or verified query asset
   - no important warnings

 REVIEW
   - SQL ran, but Binding confidence is low
   - generated a new query instead of using an approved verified query asset
   - mapping uncertainty stemming from low metadata density or intermittent source connectivity
   - routing shifted to an alternative path because the question was ambiguous
   - data owner or analyst review is needed

 FALLBACK
   - SQL execution failed because of a syntax error or connection timeout
   - falling back to the internal cache due to blocked database queries
   - routed to suggestion-only state or a demo response
   - result did not come from a real operational data source

A successful SQL execution with the correct value (EX passed) is still hard to act on in real decisions unless the UI shows which data and criteria produced that value.

Related papers such as TrustSQL also see Text-to-SQL reliability not only as the ability to return the right answer, but also as the ability to abstain from ambiguous questions that cannot be answered. Enterprise cases such as LinkedIn Text-to-SQL also combine domain knowledge and schema context into a knowledge graph and verify answers in multiple stages.

DataNexus does not evaluate this with one score. It also checks whether the UI provides enough evidence for users to decide what to do next. I call this review principle the “Practitioner Sanity Check.”

Final layout

I applied these principles to the final answer card layout. The top of the card displays the trust state badge and primary source. The middle area auto-renders the key summary data and chart. SQL and Trace for deeper verification are placed in collapsible tabs at the bottom.

+---------------------------------------------------------------+
| [Status: PASS / REVIEW / FALLBACK / ABSTAIN] | [Source: Data Catalog] |
+---------------------------------------------------------------+
| "This quarter's net revenue is KRW 120B."                    |
| [Chart: Bar / Line auto-rendered]                             |
+---------------------------------------------------------------+
| ▶ Show Evidence: SQL & Trace                                  |
+---------------------------------------------------------------+
| [Tabs] 1. SQL  2. Data Lineage  3. Metric Logic  4. Graph      |
+---------------------------------------------------------------+
| [👍 / 👎]       [Request revision]  [Export]                  |
+---------------------------------------------------------------+

Executives can make a quick call based on the status label and summary number at the top. Data analysts or DBAs can open the lower Inspect layer and move from summary to detail across SQL, data lineage, and the knowledge graph.

The feedback area at the bottom does not inject collected user corrections directly into real-time learning. It sends them to a governance review queue so feedback never becomes unreviewed training data.

Verification data should not provide only a simple document link. The card follows the DataHub URN and physical table.column FQN (Fully Qualified Name) and narrows the evidence down to specific values and source passages.

However, if reliability labels such as Trusted or PASS are used too often without conditions, users may start to trust the label itself without actually reviewing the underlying content. That is why DataNexus turns on labels only when runtime validation criteria are met.

 - PASS: ready for a decision or report citation
 - REVIEW: needs lower-level highlights and SQL review for number validation
 - FALLBACK: alternative data or demo response is shown because of infrastructure outage or a similar issue
 - ABSTAIN: no answer because of permission limits or missing data source

The natural-language response at the top of the card gives a concise summary of the metric and its primary criterion. The SQL and Trace sections stay collapsed by default so they do not clutter the interface. Still, the path must remain in the answer card so users can follow the data flow whenever needed.

The operator dashboard does more than collect simple metrics. It tracks which citations users click and which detailed routes fall into Fallback, then uses those signals to adjust the overall AI automation scope.

The central question I kept returning to while planning the BI Agent system was this: “Can users trust the data shown in the UI and use it without second-guessing it?” If supporting context is not provided with the answer, polished text is not enough in enterprise data work.