Closing the Verification Loop: Observability-Driven Harnesses

Alp Keles, Jai Menon, Sesh Nalla and Vyom Shah on what to do when agents write code faster than anyone can read it. Their answer is to stop treating review as the place correctness comes from:

With a harness, code reviews become bloom filters: a fast gate, not the source of correctness.

The harness is a stack of automated checks rather than a single technique. A shadow-state oracle runs a plain HashMap alongside the real executor and compares after every operation. Deterministic simulation testing abstracts away real time and injects faults, at roughly five seconds a seed. Then TLA+ specifications, model checking with Stateright, bounded proofs with Kani, Maelstrom with a linearizability checker, property-based testing, shadow deployment against real traffic, and production telemetry to close the loop. The argument for this ordering is that formal methods used to be too slow to be worth it, and cheap agent labour inverts that.

Two systems carry the evidence, both agent-built and both in staging rather than production. redis-rust started out using eight times the memory of Redis 8.4; the agent proposed and implemented three optimisations for an 87% cut. Helix, a Kafka-compatible streaming engine on object storage, went from 500 simulation seeds per component to 10 million across all of them, and in staging averaged 22.2 ms produce latency against 116 ms for the baseline Kafka cluster while serving the APM profiling stream.

The self-interested conclusion is stated plainly, and it is Datadog’s own product line: once the harness depends on observability to close the loop, the observability platform becomes the control layer for agent-built software. The more portable idea is that a harness compounds in a way review cannot, because every check you add keeps working on every future change. Compare Shopify’s argument that the harness rather than the model is the durable asset, and see also Datadog on developer experience in the AI era.