Building an agentic harness that outlasts the model

Zack Deveau on the part of an AI security system that is actually worth building. Models get swapped out every few months, so the engineering that lasts is the scaffolding around them: the test bootstrapping, the partitioning, the verification pass, the scoring. His line for it is that this is the part you’ll keep as new models come and go; this is where the innovation is.

The system is Dispatch, a Ruby orchestrator that runs security scans in eight stages. It finds and verifies the repo’s test commands, writes reusable architecture notes, catalogues and partitions files into token-sized clusters, then fans specialised hunting agents out across those partitions in parallel. Findings go through a verification stage that re-tests them with a different model, then deduplication and scoring, and finally a remediation stage that proposes fixes and opens PRs. Tests are the oracle throughout: a finding that cannot be demonstrated with a real test does not survive.

The numbers, with the caveat that they are self-reported. Over six weeks Dispatch completed full scans of more than 80 unique applications and produced over 300 findings, two of which would have been rated Critical. Shopify conservatively values the set at over $400,000 in equivalent bug bounty payouts. A full scan with frontier models runs $50 to $300 per application; incremental diff scans are $5 to $50.

The design principle running underneath is that precision matters more than coverage, because noise sent to a developer is worse than no finding at all. That is what justifies the expensive verification stage, and it is why the harness, not the model, is the asset.