I tested four ways to build an AI data agent on my text-to-SQL benchmark:¹ medium versus low reasoning, reasoning off, a fresh-context critic, and selective retrieval from a database profile. The low-effort and reasoning-off comparisons each used the same questions from BIRD Mini-Dev.² The output for each question was an SQL query, which was executed and compared to reference (often called ‘gold’ query). Whether generated SQL matched the reference result was the primary metric; the tables below show how many of the 498 questions each setup answered correctly, with the percentage in parentheses.

For an AI agent I chose pi,³ because it is a) open source b) minimal c) is made to be modified and adapted. OpenRouter⁴ was chosen as LLM provider, because it allows to control how much money is spent and has good feedback info. See how agent was implemented in a separate post.⁵ For the model I chose OpenAI GPT-5.6 Luna, because it is cheap, fast, and smart enough.

1. Low-effort raw access is the best default

At low effort, giving an AI agent read-only database access answered 311 of 498 questions correctly (62.4%). Generated table summaries⁶—table of contents, selected profile slices, and pre-generated schema links⁷—answered 306 of 498 questions correctly (61.4%).

Metric	Raw database access	Bundled table summary	Change
Correct answers (of 498)	311 (62.4%)	306 (61.4%)	−1.0 pp
DB queries/question	4.50	1.55	−65.6%
Latency/question	27.8 s	32.7 s	+17.7%
Cost/question	$0.0074	$0.0088	+19.2%

The −1.0-point accuracy difference was not measurable (95% CI −3.3 to +1.3, exact McNemar p=.49). The profile won 14 of the 33 disagreements and lost 19. It reduced database traffic, but ran slower and cost more. The schema links generated with schema-linker⁷ were bundled with the profile.

Low effort also dominated medium effort on cost and speed without a measurable accuracy loss:

Metric	Low effort	Medium effort	Low-effort change
Correct answers (of 498)	311 (62.4%)	306 (61.4%)	+1.0 pp
Total tokens	31.1M	35.1M	−11.3%
Reasoning tokens	0.59M	1.04M	−42.7%
Total cost	$3.69	$4.52	−18.3%
Latency/question	27.8 s	35.9 s	−22.7%
DB queries/question	4.50	5.90	−23.7%

2. Reasoning-off plus the profile is faster

Turning reasoning off made raw access much faster and cheaper than low effort, but reduced accuracy. Adding the profile bundle recovered most of the loss.

Metric	Off, raw access	Off, bundled table summary	Low, raw access
Correct answers (of 498)	287 (57.6%)	303 (60.8%)	311 (62.4%)
Total tokens	31.5M	43.3M	31.1M
Total cost	$2.96	$3.64	$3.69
Latency/question	14.2 s	14.8 s	27.8 s
DB queries/question	5.36	1.57	4.50

The profile used 38% more tokens and cost 23% more than reasoning-off raw access, but made 71% fewer database queries with only 4% more latency. Compared with low-effort raw access, it was 1.6 points lower—a difference that was not measurable (95% CI −4.7 to +1.4, p=.37)—cost about the same, and ran 47% faster. The borderline p=.040 gain deserves replication.

3. Separate ‘critic’ AI agent did not help improving accueacy

Some papers and text-to-sql pipelines include so-called ‘critic’. The critic read candidate SQL in a fresh context and checked projection, joins, filters, aggregation, and ordering. I tried it too, implemented as a subagent (just asked PI in the prompt). This resulted in no measurable accuracy improvement, but increased cost substantially.

Metric	Inline self-critique	Critic sub-agent	Change
Correct answers (of 498)	306 (61.4%)	307 (61.6%)	+0.2 pp
Total tokens	35.1M	54.8M	+56%
Total cost	$4.52	$6.89	+53%
Latency/question	35.9 s	56.9 s	+58%

Side quest: table of contents for table summary

The profiles generated by db-snooper⁶ can get large if there are many tables and columns. And the whole file is bundled into the prompt, eating precious tokens.

Idea. Generate a short table of contents for the generated tables summary, so that the agent can read summaries of only the table he needs to.

So I updated added the table of contents to db-snooper and re-run the same 500 queries.

Signal	Full profile in prompt	TOC + selected slices	Change
Input-token premium over raw	about 40%	about 24%	16 pp less overhead

Retrieval reduced prompt overhead, but the agent replaced SQL exploration with profile reads and did not become more accurate. The earlier full-profile run was faster, although that is not a clean speed comparison because the prompt and runner protocols changed.

References

Footnotes

1. AI data analytic: All your agent needs is read-only database access ↩

2. BIRD text-to-SQL benchmark ↩

3. pi coding agent ↩

4. OpenRouter ↩

5. AI data analytic: Pitfalls in implementing AI agent ↩

6. db-snooper ↩ ↩²

7. schema-linker ↩ ↩²