Benchmarking Coding Agents on Databricks' Multi-Million Line Codebase

Public benchmarks were not telling Databricks what it needed to know, so it built its own from real pull requests against its multi-million-line codebase, which spans more than ten languages including Python, Go, TypeScript, Scala, Rust, Java, Bazel and Protobuf. Tasks were graded against the real test suites rather than an LLM judge, with git history sealed so agents could not read the original solution. Roughly a quarter of the tasks are low complexity and about 60% medium.

The headline finding is that no single model wins. As the authors put it, the Pareto frontier for coding tasks includes models from OpenAI, Anthropic and open source, and the sensible strategy is routing by task difficulty rather than paying for the most expensive model every time. Opus 4.8 scored 87% at $1.94 a task; GLM 5.2 tied with it statistically at $1.28; Sonnet 5 came in at 81% and $2.09, costing more per task despite the lower per-token price.

That inversion is the point worth keeping:

The token price of a model is a poor indicator of actual costs incurred on end-to-end tasks.

Reasoning efficiency, not the price list, decides what a task costs. The same holds for the harness: running one model at one thinking effort through Claude Code or Codex versus Pi changed cost per task by more than 2x at equal quality, because Pi sent about 3x less context per turn. The authors are careful to say the lesson is not that one harness is always cheaper. Companion piece on what this costs in practice: How Databricks Manages Its Own Coding Agent Spend.