Orchestrating AI Code Review at scale

Ryan Skidmore on the code review system Cloudflare runs across its own repos. The argument against off-the-shelf tools is that they “just didn’t offer enough flexibility and customisation for an organisation the size of Cloudflare”, and the argument against a single reviewing model is that one generic prompt produces noise. Instead the system launches up to seven specialist reviewers covering security, performance, code quality, documentation, release management and compliance, then has a coordinator do a judge pass that deduplicates findings, moves them into the right section and drops nitpicks and false positives.

Cost is controlled by sizing the review to the diff. Trivial changes get two agents at around $0.20, medium ones four, and anything over 100 lines gets the full seven or more at around $1.68 on average. Models are assigned by tier too, with the strongest reserved for the coordinator and lightweight models handling documentation and release checks.

The production data covers 30 days to 9 April 2026: 131,246 review runs over 48,095 merge requests across 5,169 repositories, a median review time of 3m 39s, and 159,103 findings, of which code quality alone accounted for 74,898. Roughly 120 billion tokens went through it at an 85.7% cache hit rate. Engineers overrode the system to force a merge 288 times, about 0.6% of merge requests.

Two things worth keeping. The resilience design treats model providers as unreliable by default, with circuit breakers and failback chains so that “we don’t want to wait for an on-call engineer to make a code change to switch out the models we’re using”. And Skidmore’s prompt engineering claim, that telling a model what not to do is where the real value sits, which is what the coordinator’s filtering pass is really encoding. He is clear this does not replace human review yet.