How Duolingo Built a Production-Ready AI Agent Platform

Guadalupe Aliseda-Canton on the infrastructure Duolingo built so its teams stopped rebuilding the same agent plumbing. An agent is a registry entry with a name, owner, system prompt, model, required MCP servers and output type; defining it once makes it callable from Slack, the CLI, internal sites and workflows without reimplementation. Standing one up used to be a multi-week project and now takes about ten minutes. The premise, in their words: “every useful agent needs a surprising amount of surrounding infrastructure.”

Underneath it is Temporal, chosen because agents take minutes, call external tools and fail unpredictably, so it “persists state, retries safely, and coordinates long-running work across systems.” Execution is decoupled from definition, which lets them run the Claude Agents SDK, Codex CLI or OpenAI Agents SDK behind the same registry entry, with LLM calls routed through an internal gateway for cost tracking.

The evaluation setup is the part I would steal. Evals run against authored scenarios, capturing output, file changes and git diffs, then grade with deterministic checks: structured_output validates response fields, diff_assertions requires or forbids specific repo changes and caps how many files an agent may touch, and no_op_consistency catches the failure mode where an agent reports work it did not actually do. LLM-as-judge is available for the non-deterministic cases but explicitly secondary, since “deterministic graders are the foundation.” Agents on the platform currently fix CI failures, address review comments, and back a Slack bot that investigates crashes for release managers.