The text-to-SQL cliff: why 86% on Spider becomes 6% on your warehouse

Text-to-SQL benchmark scores do not transfer to production warehouses, and the size of the gap is documented rather than anecdotal. The Spider 2.0 project reported that the most advanced language models available at the time, including GPT-4, solved 6.0% of Spider 2.0 tasks against 86.6% on Spider 1.0 and 57.4% on BIRD. That single sentence is the most useful thing published about this problem, and it is the reason a demonstration over a five-table sample database predicts nothing about a warehouse.

The rest of this article does three things: traces those numbers to their sources with dates attached, explains exactly what a warehouse schema contains that a benchmark schema does not, and gives you a harness for measuring the only number that matters, which is your own.

The numbers, and what each one measured

Four figures get quoted in this debate, and they measure four different things. Attaching the date and the setting to each is most of the work.

Spider 2.0 contains 632 real-world text-to-SQL workflow problems over enterprise databases that routinely exceed 1,000 columns, split into Spider 2.0-Snow (547 examples on Snowflake), Spider 2.0-Lite (547 examples across BigQuery, Snowflake and SQLite) and Spider 2.0-DBT (68 tasks on DuckDB). The task is not “write a SELECT”. It is a workflow: navigate the metadata, write multiple queries in the right dialect, transform, and produce a result.

The 2026 leaderboard has moved a long way, and it changes the argument less than it appears

Anyone quoting the 6.0% figure in 2026 without checking the leaderboard is quoting a stale number, so check it. As of this writing the public Spider 2.0 leaderboard shows top submissions well above the original baselines — on the order of 96% on Spider 2.0-Snow, 76% on Spider 2.0-Lite and 66% on Spider 2.0-DBT — with the site itself noting that scores may change slightly as evaluation is verified.

Three things are true about that at once, and the honest position holds all three.

The first is that the improvement is real. Frontier models plus agent scaffolding genuinely close a large part of the gap that existed in 2024, and anyone still claiming text-to-SQL “does not work” is describing a two-year-old state of the art.

The second is that leaderboard entries are purpose-built agent systems submitted by teams optimising against a fixed, public, extensively-studied benchmark. A named submission scoring 96% on 547 Snowflake tasks has been engineered against those 547 tasks in a way that no system is engineered against your warehouse on its first day.

The third is the one that matters operationally: the spread within the leaderboard is itself the finding. The same benchmark family yields 96% on one split and 66% on another, and the hard split — Spider 2.0-DBT — is the one closest to how analytics is actually built, with a transformation layer and project structure rather than a single query. When a benchmark’s own subsets disagree by thirty points, the number does not generalise to a schema nobody has seen.

What your warehouse has that a benchmark database does not

The cliff is not caused by SQL being hard. It is caused by ten specific properties of production schemas, none of which appear in an academic benchmark database and all of which appear in almost every warehouse.

Look at the middle of that table rather than the top. The column-count problem gets the attention because it is easy to describe, and it is genuinely the reason retrieval over schema metadata is necessary. But the expensive failures are the semantic ones — slowly changing dimensions, soft deletes and business definitions — because those produce a query that runs, returns a number, and is wrong.

Benchmarks and production disagree about which failure is worse

Execution accuracy counts a query that errors and a query that silently returns the wrong number identically: both are simply not correct. In production they are opposites.

A query that fails is self-reporting. The user sees an error, the system logs it, nobody makes a decision on it, and the failure enters your backlog. A query that runs and returns a plausible wrong number is the failure mode that costs money, because it is indistinguishable from a right answer until someone reconciles it against a report weeks later — and by then it has been quoted in a meeting.

This has a direct design consequence. The correct behaviour when the system is uncertain is to fail visibly, and that has to be engineered rather than hoped for. In DataCopilot the data plane has a circuit breaker: eight consecutive failures return an honest error rather than an answer assembled from whatever the model had left. The system is designed so that “I could not answer this” is a supported output. Any text-to-SQL system without such a state has only one available response to a question it cannot answer, and it will produce it.

The second design consequence is about where numbers come from. A figure that has passed through a token stream has been through a component that can alter it. In DataCopilot, results reach a chart through a Python execution bridge rather than through generated prose — a charted number has never passed through a token stream. That property is worth more than several points of benchmark accuracy, because it removes an entire class of silent corruption.

Measure it on your own schema — the harness is forty lines

The only number that should influence a purchase decision is execution accuracy on your schema, with your questions, and it takes about a day to produce. Build a gold set of 50 questions with hand-written correct SQL, sampling the actual distribution of what people ask, then run this:

# evalsql.py — SQLAlchemy 2.0, pandas 2.2, Python 3.11
# Produces the three counts that matter: correct, errored, wrong-but-ran.
import json
from dataclasses import dataclass
import pandas as pd
from sqlalchemy import create_engine, text

ENGINE = create_engine("postgresql+psycopg://readonly@warehouse/analytics")


@dataclass
class Case:
    question: str
    gold_sql: str


def run(sql: str) -> pd.DataFrame:
    # Read-only, time-bounded. Never evaluate against a session that can write.
    with ENGINE.connect() as conn:
        conn.execute(text("SET LOCAL statement_timeout = '30s'"))
        return pd.read_sql_query(text(sql), conn)


def equivalent(a: pd.DataFrame, b: pd.DataFrame, tol: float = 1e-6) -> bool:
    """Order-insensitive, column-name-insensitive result comparison."""
    if a.shape != b.shape:
        return False
    a2 = a.copy(); b2 = b.copy()
    a2.columns = range(a2.shape[1]); b2.columns = range(b2.shape[1])
    key = list(a2.columns)
    a2 = a2.sort_values(key).reset_index(drop=True)
    b2 = b2.sort_values(key).reset_index(drop=True)
    try:
        pd.testing.assert_frame_equal(
            a2, b2, check_dtype=False, rtol=tol, atol=tol
        )
        return True
    except AssertionError:
        return False


def evaluate(cases: list[Case], predict) -> dict:
    correct = errored = wrong = 0
    log = []
    for c in cases:
        gold = run(c.gold_sql)
        pred_sql = predict(c.question)
        try:
            pred = run(pred_sql)
        except Exception as exc:                     # noqa: BLE001
            errored += 1
            log.append({"q": c.question, "outcome": "error", "detail": str(exc)})
            continue
        if equivalent(gold, pred):
            correct += 1
            log.append({"q": c.question, "outcome": "correct"})
        else:
            wrong += 1                               # the expensive category
            log.append({"q": c.question, "outcome": "wrong", "sql": pred_sql})

    n = len(cases)
    return {
        "n": n,
        "execution_accuracy": correct / n,
        "error_rate": errored / n,
        "silent_wrong_rate": wrong / n,              # report this separately
        "log": log,
    }


if __name__ == "__main__":
    cases = [Case(**c) for c in json.load(open("gold.json"))]
    result = evaluate(cases, predict=lambda q: your_system.to_sql(q))
    print(json.dumps({k: v for k, v in result.items() if k != "log"}, indent=2))

Two decisions in that harness are the ones people get wrong. Comparison is order-insensitive and column-name-insensitive, because a query that returns the right rows in a different order is correct and a strict string comparison will mark it wrong. And silent_wrong_rate is reported as its own line, not folded into an accuracy figure, because it is the number you will be asked about after the first incident.

Expect your first result to be uncomfortable. That is the point: it is a real measurement of a real system on a real schema, and it is the baseline against which every subsequent change is judged.

What actually moves the number

Four things move production text-to-SQL accuracy, in roughly this order of effect.

Documented schema semantics. The single largest gain comes from the warehouse’s own documentation being available to the system: what each table is, what each column means, which tables are abandoned, and how the organisation defines its terms. This is a documentation and data-modelling exercise, not a modelling one, and it is the reason DataCopilot’s semantic layer is built around schema documentation and an assisted datasheet-drafting flow rather than around retrieval tricks. If your warehouse has no data dictionary, building one is a higher-yield investment than any change to the model.

Correct dialect generation. A query that is semantically right and syntactically invalid for the target engine scores zero. This is unglamorous engineering — DataCopilot carries 30 registered database engine adapters and its adapter layer is 6,471 lines, the largest single file in the backend — and it is a precondition rather than an optimisation.

A bounded, inspectable execution loop. Retrying with the engine’s error message attached recovers a meaningful share of failures. It also runs forever if unbounded, which is why the agent loop has a default ceiling of 128 steps, configurable between 1 and 512. A step budget makes failure legible: the agent finishes inside it or reports that it could not.

Read-only, time-bounded execution. Every query runs inside a read-only, time-bounded transaction. This does not improve accuracy at all. It changes the cost of being wrong, which is what makes it acceptable to let a language model near a production database in the first place.

What does not move the number

Being explicit about this saves procurement conversations.

A larger model helps less than the benchmark deltas imply, because the binding constraint is usually schema knowledge the model has no way to possess. Prompt engineering plateaus quickly on wide schemas for the same reason. And a benchmark score — anyone’s, including the ones cited at the top of this article — predicts your result so weakly that it should not appear in a decision document without your own measurement beside it.

Where this stops applying

This analysis is about analytical queries over a documented relational warehouse, executed read-only, with a human reading the answer. Four boundaries.

If your questions are answered by an existing certified report, text-to-SQL is the wrong tool. Route to the report. The value of natural-language querying is in the questions nobody built a report for.

If the warehouse is undocumented, expect results at the low end regardless of what you buy. No system infers that status_cd = 'X' means deleted. That is knowledge held by people, and it has to be written down before it can be used.

If answers feed an automated action rather than a person, the accuracy bar is a different one entirely, and the correct architecture is a curated set of parameterised queries with a natural-language front end — not open-ended generation.

And if your data is not in a relational warehouse at all, the benchmark literature has little to say. Document stores, search indexes and wide-column databases each have their own query semantics, their own failure modes, and no public benchmark of comparable maturity. Every number in this article, including its title, is about SQL.