AI Agent Tradeoffs: What Evals Catch and Reading Traces Reveal

Sergiy Dybskiy built a schedule assistant for AI Engineer World’s Fair 2026, with cheap open-weight models serving anonymous visitors and better ones for signed-up users. Asked who the famous speakers were, the free tier confidently named Sam Altman, Andrew Ng and Fei-Fei Li, then invented “Chris Hagen, Founder of Stability AI”. Pressed on its source, it claimed the names came from its getTracks tool, which only ever returned track names.

The point of the piece is what the monitoring showed: 44 LLM calls, 92K tokens, 0 errors, under a cent in cost. By every automatic signal the run was fine.

A tool ran, so the answer reads as grounded. You only catch it by opening that tool call and seeing tracks where speakers should be.

He lists the options honestly, from upgrading the free tier’s model to simply accepting the limitation for anonymous users, and settles on keeping the cheap model while fixing the routing, tightening the prompt and writing a groundedness eval, an assertion that answers only name entities that actually appeared in tool output or the database. That check is ordinary test code, no LLM judge needed, and it would have caught this before release. The lesson is that evals guard against regressions you already know about, and reading traces by hand is how you find the ones you do not. Sentry has since written up its automated debugging workflow, the same instinct pointed at production errors.