How to Build a Trust Platform for Your Agent

Jack Gordley on what went wrong as Grafana Assistant grew past its prototype stage, and the observability work that fixed it. The pain is specific and familiar:

Reviewing contributions to the prompt, tool set, and agent harness became a nightmare as changes piled on from multiple teams. Bespoke dashboards just weren’t cutting it.

His underlying point is that agents are inherently non-deterministic and heavily customized for a wide range of use cases, so the usual software review reflexes do not carry over.

The path out is set as five stages, Phase 0 to Phase 4. Phase 0 is the early prototype and first deployment. Phase 1 is monitoring live traffic: engineering metrics like latency, cost and tokens, plus analysis of the conversations themselves. Phase 2 is measuring baseline quality with LLM-judge evaluators and deterministic checks, including built-in templates for PII detection and toxicity. Phase 3 turns failed conversations into annotated regression suites. Phase 4 adds experiment tracking and CI, comparing models and gating PRs.

They also mention o11y-bench, their own observability benchmark of around 70 tasks, used to weigh Haiku against Sonnet on performance and cost. Worth reading with the caveat that the internal tool became Grafana Cloud’s Agent Observability, now generally available, so this is a vendor describing something it sells.