Using DSPy to Evaluate and Improve Datasette Agent's SQL System Prompts

Summary

Simon Willison walks through using DSPy to systematically evaluate and refine the SQL system prompts behind Datasette Agent's read-only question-answering feature. He built a testing harness where DSPy agents invoked the actual Datasette Agent tools against a live database, using auto-generated gold-standard datasets to score prompt variants rigorously rather than by intuition. A concrete finding: the schema listing should include column names, not just table names — the prior prompt caused "co