Introducing Supabase Evals

Matt Rossman on the open-source benchmark Supabase built to find out how well coding agents actually build with its platform, covering “our CLI, MCP server, agent skills, and docs”. Scenarios are organised by product area (Database, Auth), by cross-cutting topic (SDK, observability), and by journey stage. There are two suites: benchmark scenarios for breadth, and regression scenarios that go deep on known failure modes.

Each run happens in a container with both a hosted-like Supabase stack and a local CLI project, so agents call the real MCP and CLI tools rather than a mock. Scoring mixes deterministic checks (whether a user can access certain data, or an Edge Function returns an expected result) with an LLM judge, and agents get one retry before the final grade.

The results are the interesting part. On the build stage, Opus 5 and Kimi K3 scored 100% with no skills loaded at all, and Sonnet 5 went from 78% to 100% once skills were available. Rewording the description of a Postgres best-practices skill lifted its activation rate from about 10% to 60%, which says more about skill descriptions than about the models. Reading habits differ too: Codex agents pulled in roughly eight doc pages per scenario against about two for Claude Code.