AI data analytic. All your agent needs is read-only database access

Renat Iuldashev

Abstract

On 300 questions of BEAVER text-to-sql benchmark, a full database profile did not improve an AI agent's accuracy: raw read-only access answered 27 of 300 questions correctly versus 25 of 300 with the profile—while using about six times fewer input tokens and exploring the database 83% more.

Problem

I expected pre-generated metadata to save the AI agent from rediscovering tables, joins, data types, and values. Winners of the well-known text-to-SQL benchmark BIRD1 recommend2 generating a so-called ‘profile’ using db-snooper,3 i.e. a summary of every table: column names, types, examples, etc.

Another file that should help discover non-trivial table-joining strategies is the output of schema-linker.4 It efficiently compares sets of column values to find which columns belong to the same set, generating a compact .md file.

For the AI agent, I needed something lightweight and easy to modify, so I chose the popular pi coding agent.5 It is as powerful as claude or codex, yet is easy to modify and get telemetry from: log tool calls, limit number of SQL queries an agent can submit to database, limit tool access. See the implementation notes6 for more information on agent implementation.

So I tried three ways of giving the agent database context:

  1. Raw database access: direct read-only access to the database.
  2. Full profile: raw access + complete table profile generated by db-snooper.3
  3. Compact metadata: the agent received a 20–50-line summary generated from the full db-snooper profile and schema links. It kept clarified semantics and likely join strategies while omitting most profile detail.

The question was simple: does extra database context help a coding agent produce more correct SQL? In this setup, the answer was no. Three results support that conclusion:

  1. A full profile answered 25 of 300 questions correctly, versus 27 of 300 with raw database access.
  2. The profile made the agent cheaper in database calls but far more expensive in prompt tokens.
  3. Compact metadata matched raw access at 27 of 300, but did not beat it.

1. The full profile did not improve accuracy

Dataset

I used data from one of the latest text-to-SQL benchmarks, BEAVER.7 It consists of three complete MySQL database dumps called ‘neutron’, ‘nova’, and ‘dw’. The benchmark also provides many pairs of textual requests (‘Give me the highest paid employee’) and corresponding ‘gold’ queries (‘SELECT user.name, MAX(salary) FROM …’). Some queries are extremely difficult, spanning tens of tables with complicated grouping and subqueries. According to BEAVER authors, the golden queries have been verified as correct by real experts. The sample spans all kinds of queries: easy, medium, requiring domain knowledge and others. Evenly distributed across all 3 databases: nova, neutron, and dw.

Results

Execution accuracy: whether the generated SQL returned the same result as the reference query. The tables below show how many questions the agent answered correctly, out of 100 per database and 300 overall; the percentage is in parentheses.

Metric Raw DB access Full profile Change
Correct answers on neutron (of 100) 13 (13%) 13 (13%) 0 pp
Correct answers on nova (of 100) 9 (9%) 8 (8%) −1 pp
Correct answers on dw (of 100) 5 (5%) 4 (4%) −1 pp
Correct answers overall (of 300) 27 (9.0%) 25 (8.3%) −0.7 pp
Cost metric Raw DB access Full profile Change
Execution accuracy (of 300) 27 (9.0%) 25 (8.3%) −0.7 pp
Input tokens/question 1× baseline about 6× about +500%
Turns/question 4.6 2.2 −52%
DB queries/question 7.8 1.3 −83%

The agent read the profile, asked fewer questions, and reached the wrong answer faster. On the largest database, the profile text alone added about nine times the raw-access arm’s input-token volume per run.

2. Side quest: compact metadata

Maybe the full profile was simply too much context? Indeed, the profile size was about 200kb for ‘dw’ database. So, the obviouse idea is to summarize it with a brief 20–50-line file, generated from the full profile and schema-linker outputs. The same coding agent generated each summary, with no access to benchmark questions or answers, or the internet to prevent leaking sensible gld queries or domain info. The summaries clarified semantics and potential join strategies, including predicates and cardinality caveats.

2.1. Results

Metric Raw DB access Full profile Compact metadata Metadata vs raw
Correct answers on neutron (of 100) 13 (13%) 13 (13%) 13 (13%) 0 pp
Correct answers on nova (of 100) 9 (9%) 8 (8%) 11 (11%) +2 pp
Correct answers on dw (of 100) 5 (5%) 4 (4%) 3 (3%) −2 pp
Correct answers overall (of 300) 27 (9.0%) 25 (8.3%) 27 (9.0%) 0 pp

Compact metadata changed which questions the agent answered correctly, but not aggregate accuracy. The per-database differences are only a few questions and are descriptive, not evidence of improvement. The harness also has a fourth arm combining the full profile and metadata, but the headline run used three.

References

Footnotes

  1. BIRD text-to-SQL benchmark ↩

  2. Automatic Metadata Extraction for Text-to-SQL ↩

  3. db-snooper ↩ ↩2

  4. schema-linker ↩

  5. pi coding agent ↩

  6. AI data analytic: Pitfalls in implementing AI agent ↩

  7. BEAVER text-to-SQL benchmark ↩