Better Models: Worse Tools

Armin Ronacher on a regression he hit in his own agent, Pi: Opus 4.8 and Sonnet 5 get better at the task and worse at calling the tool. Pi’s edit tool takes an edits array of objects with only oldText and newText. The newer models produce the correct values and then append invented keys, things like requireUnique, oldText2, matchCase and in_file. The target text is right; the object around it has junk bolted on. One user’s session saw Opus 4.8 fail roughly 20% of the time.

His explanation is that the models were post-trained against Claude Code’s harness, which is forgiving: it accepts parameter aliases, filters unknown keys and silently repairs bad calls. Claude Code’s own edit tool is flat (file_path, old_string, new_string) rather than Pi’s nested shape, so when a model meets the unfamiliar schema it falls back on learned priors about what an edit call looks like. The failures cluster at high-entropy points, notably the decision whether to close the JSON object after an escaped multiline string. Stripping thinking blocks halved the failure rate; strict tool invocation removed it in testing.

The conclusion is that tool schemas are not distribution-neutral. Models do best on schemas that look like their training data, so a dominant harness quietly exports its quirks to everyone else building one. Ronacher has shifted towards grammar-constrained sampling as a result: if models keep improving at tasks while getting looser about schema compliance, harnesses need stronger guarantees rather than good intentions.