10. Spider 72%: The Dataset, Not the Model, Shapes the Error Profile

Three multi-candidate experiments from post 9 produced no meaningful improvement. Before moving on to the Schema Binding Plan, I wanted to test one more hypothesis: model tier as the bottleneck. I expected Gemini Pro to improve the current 56% accuracy baseline over flash-lite. I also ran the error-type classification and Spider experiment as part of that test.

Switching to Pro Made Things Worse

Prompts, context, and pipeline stayed the same. Only the model changed, and the same 50 BIRD questions ran against it. The metric is EX (Execution Accuracy): whether the generated SQL produces the correct result when executed against the database.

Δ -8.0%p.

Three previously wrong questions became correct, but seven previously correct questions regressed. Most regressions came from the Czech financial database. QID 168 required only a 2-table JOIN, but Pro constructed a 4-table join, apparently seeking a more exhaustive path, which missed the correct answer. QID 39 produced no SQL output at all. flash-lite generated SQL for the same question without difficulty.

Pro performs slightly better on the in-house 30-question benchmark. However, that benchmark uses familiar database schemas and question patterns, so it is hard to compare directly with BIRD’s 50 unseen databases. For measuring generalization, BIRD is the more credible signal.

Model tier is not the direct bottleneck for this pipeline.

Classifying 19 Errors Into Six Types

The same 50 BIRD questions from post 9 were re-run. 27 came back wrong, but 4 of those were grader artifacts; the same SQL scored differently across grader versions. The remaining 19 represent genuine pipeline failures.

• COLUMN_BINDING: The wrong column was selected for a JOIN or SELECT. In a question asking for the percentage of blue female superheroes, the LLM joined on eye_colour_id when the correct answer required skin_colour_id. Both column names look similar and both reference the colour table, and the schema alone does not indicate which one to use.

• VALUE_BINDING: The value used in a WHERE clause was a wrong guess. A question about cash withdrawal transactions in a Czech financial database expects the actual stored value VYBER, but the LLM wrote cash withdrawal. No rows matched, so the result was empty. This happens when the model has no knowledge of what values are actually stored in the database.

• DERIVED_METRIC: The derivation strategy itself differed, producing genuinely different results. Two patterns exist here. One is a calculation strategy error: when the gold SQL uses INTERSECT to compute an intersection but the LLM approaches it with a JOIN, duplicate handling can lead to different outcomes. The other is an evaluation boundary case: expressions like SUM(CASE WHEN ...) vs COUNT(CASE WHEN ...) usually agree, but diverge in the presence of NULLs; even execution-based grading can produce false negatives depending on the data. The higher rate in Spider is explained by the prevalence of INTERSECT/EXCEPT questions in that dataset.

• HIDDEN_JOIN: Cases where reaching the target table requires passing through an intermediate table not mentioned in the question: “to connect column X in table A with column Z in table C, you actually need to go through table B first.” There are proposals to solve this via graph traversal over the schema, which is why it was tracked. Zero cases appeared in the 19 BIRD errors. Not a significant source of failures here.

• GENERATION_FAILURE: The model declined to generate SQL at all. First observed during the Pro tier test; flash-lite produced SQL for the same questions. More cautious models appear more likely to decline SQL generation in ambiguous situations.

• LEXICAL_NORMALIZATION: Lowercase text from the question made it into the WHERE clause unchanged, mismatching the casing of the stored values. For example, querying with 'active' when the database stores 'Active'.

COLUMN_BINDING and VALUE_BINDING together account for 63.2% of failures. The bottleneck is the schema grounding layer, where natural-language concepts get mapped to the actual columns and values in the database, not the SQL generator. These are not syntax errors; they arise from an incomplete interpretation of the schema.

Running Spider Produced a Different Distribution

Spider, published by Yale NLP, is another Text-to-SQL benchmark with simpler database structures and more general domains than BIRD. The same pipeline ran against Spider: 50 questions, SQLite, 20 databases.

Spider EX: 72.0% (36/50)

That is 16%p higher than BIRD. The raw accuracy gap mattered less than the shift in error distribution across categories.

• VALUE_BINDING dropped to zero in Spider. Unlike BIRD’s Czech financial database, Spider does not contain domain-specific values that are hard for the model to guess. Most Spider databases use general, English-language data where value inference is relatively straightforward.

• DERIVED_METRIC jumped sharply in the other direction. Spider includes many questions based on INTERSECT/EXCEPT set operations, and failing to express those operations the same way as the gold SQL lands in this category.

All 14 Spider errors fit within the existing classification. No new category was needed, confirming that the taxonomy holds across both datasets.

The same pipeline produced a different failure profile on each dataset. VALUE_BINDING was prominent in BIRD due to the Czech financial domain; DERIVED_METRIC dominated in Spider due to the prevalence of set-operation questions. Which error type appears most often depends more on the database characteristics than on the model.

What Two Weeks of Testing Covered

Accuracy measurements across the series, in chronological order:

Post 9 covers the first three directions in detail, including their experimental results and failure modes. The shared conclusion was that selecting after execution does not work; the correct column and value have to be identified before generation begins.

Model scaling was the additional verification in this post. Swapping the model did not change the underlying problem, and the result was -8%p.

One point is worth clarifying: these experiments were not run without an ontology in place. By the start of post 8, Vanna already had DDL and a Glossary YAML with 25 business terms trained in. 66.67% was the first measurement under those conditions, and 80% was the peak built from that baseline.

The issue was scope. That Glossary was built for an e-commerce domain, and BIRD is a benchmark that spans 11 domains, including Czech financial databases, superhero databases, and others. Against domains the Glossary was never trained on, it does not help. Its presence makes no measurable difference.

BIRD 56% is not a number that is low because the ontology is missing. It is evidence that an ontology, even one that is present, has no effect on domains it was not trained on.

The upcoming Schema Grounding redesign follows directly from this conclusion. Pre-defining knowledge in YAML hits a clear ceiling against unseen databases. A separate layer is needed for interpreting value-level semantics: not just column selection, but understanding what values are actually stored in the database.

Closing

The next step is Schema Grounding redesign: improving the stage where the system maps natural-language concepts to the correct columns and values. The work moves into implementation from here.

33 error-labeled cases remain (19 BIRD, 14 Spider), ready for direct use in prioritizing the next phase. COLUMN_BINDING and DERIVED_METRIC are structural problems that deeper schema understanding can address. VALUE_BINDING is a different problem: values like VYBER cannot be inferred from the schema; the system needs a separate mechanism to enumerate what is actually stored in the database. Schema Grounding targets the first two types first; VALUE_BINDING follows in a later stage.

WikiSQL is a candidate for additional validation, but that falls after Phase 1.