Martin Pllu's public notebook
Linklog
things I've read, with notes
-
Pave The Road ampcode.com
Alex Kemper’s argument is that most companies are buying capable models and then running them over development processes built for a world where code was expensive and mistakes were slow to undo. His line for it:
After all, what good is it if the latest model completes a task in five minutes, but it takes another five days to get deployed?
The suggested shift is from preventing mistakes to making them cheap: isolation rather than permission, quick reversibility rather than careful review, parallelism rather than queues.
Read more
Amp offers its own numbers as evidence. In the month after shipping Orbs, their parallel cloud sandboxes, commit velocity rose 65% and more than 85% of commits started originating in sandboxes rather than local checkouts. One engineer shipped 23 changes during a three-hour livestream. Worth reading with the obvious caveat in mind: this is a coding agent vendor describing the benefits of its own product. See also What I Want to Tell You About Orbs for the same shift told from a single developer’s desk.
-
Efficient Tokens & Effective Teams in Buzz engineering.block.xyz
Atish Patel on Block’s Buzz, an internal workspace where several agents sit in shared channels alongside humans, each with its own persona, memory and seat. The question the post sets out to answer is which agents you should actually put on a job. Block’s answer is a tiered cast: QuickBee on cheap fast models for documented legwork, WorkerBee in the middle owning subtasks on its own, and SmartBee on frontier models kept for coordination and judgment.
The benchmarking is the useful part. Running real Buzz agents over live relays, they found a 5.5x price spread among the top scorers on Terminal-Bench 2.1 separated by only 8.9 points of score, which is a neat illustration that paying more stops buying much fairly quickly. On their long-horizon benchmark a SmartBee with two WorkerBees hit 71.5% against 59.1% for a solo agent, completing 20 of 44 tasks outright versus 15, or about a third more work.
Read more
The finding worth carrying away is the conditional one: team composition only paid off on long tasks. On short, well-specified work the extra coordination bought nothing, and the team structure was just overhead. Their summary is “not every task needs your smartest agent. Pick the right bee. Build the right team.”
-
How We Built an Automated Debugging Workflow at Sentry blog.sentry.io
Dhrumil Parekh on Sentry using its own Seer agent against its own repos. Seer root-causes incoming issues and opens fix PRs; an hourly Claude routine then scans a dedicated Slack channel for PRs opened in the last four hours, asks Seer to work out from commit history who knows the affected code best, checks the PR is still open, and pings that person to merge, close, or give feedback.
The reported effects are modest but consistent: roughly 21% more action on PRs, 13% better response within 48 hours, and 12.5% more PRs closed without merging. They read that last one as a good sign rather than a bad one, since the rejections tend to be duplicate fixes or cases where an engineer wanted a more thorough solution.
Read more
The framing I liked, and the reason this is more interesting than most agent-in-production writeups, is that the automation is aimed at the handoff rather than the fix. The agent finds a plausible patch, but the design work went into getting the right human to look at it at the right moment, down to details like never double-posting and never pinging at the weekend. As Parekh puts it, automation “is all about creating less manual work for your engineers, not reducing the number you have.”
-
How Duolingo Built a Production-Ready AI Agent Platform blog.duolingo.com
Guadalupe Aliseda-Canton on the infrastructure Duolingo built so its teams stopped rebuilding the same agent plumbing. An agent is a registry entry with a name, owner, system prompt, model, required MCP servers and output type; defining it once makes it callable from Slack, the CLI, internal sites and workflows without reimplementation. Standing one up used to be a multi-week project and now takes about ten minutes. The premise, in their words: “every useful agent needs a surprising amount of surrounding infrastructure.”
Underneath it is Temporal, chosen because agents take minutes, call external tools and fail unpredictably, so it “persists state, retries safely, and coordinates long-running work across systems.” Execution is decoupled from definition, which lets them run the Claude Agents SDK, Codex CLI or OpenAI Agents SDK behind the same registry entry, with LLM calls routed through an internal gateway for cost tracking.
Read more
The evaluation setup is the part I would steal. Evals run against authored scenarios, capturing output, file changes and git diffs, then grade with deterministic checks:
structured_outputvalidates response fields,diff_assertionsrequires or forbids specific repo changes and caps how many files an agent may touch, andno_op_consistencycatches the failure mode where an agent reports work it did not actually do. LLM-as-judge is available for the non-deterministic cases but explicitly secondary, since “deterministic graders are the foundation.” Agents on the platform currently fix CI failures, address review comments, and back a Slack bot that investigates crashes for release managers. -
Harness Engineering for Self-Improvement lilianweng.github.io
Lilian Weng’s argument is that recursive self-improvement in AI will arrive through the harness, not through models editing their own weights. The harness, in her definition, is “the system surrounding a base model that orchestrates execution and decides how the model thinks and plans, calls tools and acts, perceives and manages context, stores artifacts, and evaluates results”. Because a harness is just code, it is a much more tractable target for improvement loops than the model itself.
Read more
The post is a survey of how those loops work today: context engineering schemes where structured context evolves through generator, reflector and curator roles; evolutionary search over harness code (AlphaEvolve, Darwin Gödel Machine); and self-editing harnesses that mine their own failures, propose bounded edits, and validate them against held-out tests before accepting. Weng is clear-eyed about the failure modes, with weak evaluators and reward hacking at the top of the list.
The HN discussion is worth a skim. One commenter reports that an incomplete check suite silently reporting success was worse than a weak evaluator, because it looked correct and decisive. Another makes the nice point that coding agents already self-improve in a mundane way, by installing and building tools that change their environment, and asks why we treat the agent and its environment as separate at all. There is also a long Torment Nexus joke thread, which tells you how comfortable the room is with the phrase “recursive self-improvement”.
The complement to OpenAI’s harness engineering write-up: that post is about humans building harnesses so agents can ship software, this one is about the harness becoming the thing that improves itself. See also The Anatomy of an Agent Harness for the Agent = Model + Harness framing.
-
What I Want to Tell You About Orbs ampcode.com
Thorsten Ball opens by refusing to sell Orbs on their feature list, on the grounds that a list of ingredients has never made anyone want a burger. What he describes instead is how ephemeral sandboxed environments changed his own day-to-day work on Amp.
Three changes stand out. He spawns far more agents, because the friction of juggling local checkouts and environments no longer caps how many he can have going. Those agents run longer, eight to thirty minutes on work he would previously have kept local, with real testing attached. And they come back with evidence rather than a diff to audit: screenshots, videos, test matrices, in one case a demo video of a Rust desktop app that had been run headless in a sandbox.
Read more
His own summary of the effect is “more agents building more complicated things; agents running for longer and giving better proof that what they did works; a lighter, less sigh-inducing review load; more things shipped, faster.” The proof-of-functionality point is the one I keep coming back to, since review load is the usual bottleneck once agents are producing more than you can read. Same caveat as ever: this is a vendor writing about its own tool. Pave The Road makes the organisational version of the argument.
-
AI adoption starts with truth replit.com
Jon Eide and Aadil Hussaini on why Replit’s internal data agent got used, which they put down to trust rather than capability:
AI adoption is limited by trust. A user who gets burned by a confidently wrong answer will double-check the next one.
Their fix is an operational-truth layer: a version-controlled repository of business definitions, canonical metrics, source relationships, validation rules and documented past failures, which grounds every internal agent.
Read more
The mechanism I find persuasive is that corrections arrive as pull requests. When someone spots a wrong answer, the fix is diffable, revertible and attributable, it goes through human review, and it then propagates to every downstream agent rather than living in one person’s head. They are explicit that the review gate deliberately slows things down. Agents also validate against the layer before making a claim. Their data agent now handles over 1,000 warehouse-backed questions a week, grown by word of mouth.
The framing worth keeping: “a semantic layer is not the product; it is the shared contract that lets a company safely add a system of specialized capabilities.” The point is that one reviewed corpus makes improvements compound across agents instead of errors compounding.
-
AI financial advice is surprisingly good, especially if you ask the right questions mitsloan.mit.edu
MIT Sloan and Stanford researchers had 1,000 adults ask chatbots for financial guidance, then simulated lifetimes of following it. The advice was better than the researchers expected: higher savings, diversification, sensible risk reduction with age. The catch is that outcomes tracked prompt quality. Women and less financially literate users ended up roughly $50,000 worse off by 60, and people unfamiliar with AI nearly $100,000 worse, because their prompts drew weaker advice. The models also leaned on rules of thumb, failing to adjust to shocks like unemployment.
Good advice being freely available but unevenly extractable is a striking new kind of inequality.
-
Allen Holub: what black-box AI code demands of your tests x.com
Allen Holub: if you really want to treat AI-generated code as a black box, your tests have to be immaculate and extensive, and he lists what that actually means, starting with all functionality and running through a long tail most suites never reach. “‘It works’ is not sufficient.”
The fine print on Uncle Bob’s gauntlet: not reading the code is only as safe as the test suite is complete, and software factories fail on exactly the qualities tests don’t price in.
-
In Defense of Not Understanding Your Codebase seangoedecke.com
Sean Goedecke argues that fully understanding your codebase is a small-team luxury. In large systems nobody understands it all, and effective engineers work from partly correct theories, making educated guesses and dealing with the consequences. He frames “pure” engineering (small codebases, complete understanding) and “impure” engineering (large systems, partial models) as different professional cultures, and treats understanding as one value to trade off among many, like performance or compliance.
A useful counterweight to cognitive debt: if full cognitive coverage was never on offer at scale, agents are not taking away something we actually had. Uncle Bob’s gauntlet of constraints is one way of living with that.
-
Everyone is building LLM routers, we deprecated ours manifest.build
Manifest ran an LLM router for four months, classifying requests into complexity tiers and routing them across providers to cut costs, then killed it. A prompt alone does not reveal how hard a task is, prompt caching (cache reads 75 to 90 per cent cheaper) undercuts the savings, and switching models mid-workflow hurts consistency. Their conclusion: a good router stays sticky to one model, doing its job “by, ironically, not doing it”.
The HN discussion broadly agrees that frontier models are now interchangeably good for everyday work, and that nobody has time to learn each model’s quirks when a new one lands every week. The consistency point rhymes with cognitive coverage: switching tools constantly means never really knowing them.
-
qm: a multiplayer agent harness for work github.com
Y Combinator’s software team has open-sourced qm, an agent platform for a whole company rather than one person, living in Slack and on the web. A headless TypeScript core with Postgres handles identity, dual-scoped memory and permissions (personal workspaces plus shared team projects), sandboxed tool execution, and crons for background automation. The harness itself is pluggable: Claude Code, Codex, OpenCode or Pi. MIT-licensed and designed to be forked privately rather than consumed as a product.
A production-scale data point for the ideas in anatomy of an agent harness and harness engineering.
-
An agent skill that forces ASD-STE100 Simplified Technical English github.com
“Your AI writes like a LinkedIn post. Make it write like a Boeing manual.” SimpleEnglish is a skill that forces LLM documentation into ASD-STE100, the controlled language aerospace has used since 1983 so a tired mechanic cannot misread an instruction: measured 72.9 per cent fewer violations across six Claude models, with AI slop dying as a side effect. The underlying observation (via @geogristle) is that controlled languages built for human safety turn out to be exactly the constraint that stops models writing filler.
In the HN discussion, several people want it for their own emails and Slack messages, one shows an output that is still wordier than real STE, and another notes drily that this is the third ASD-STE100 submission in a fortnight: this week’s trend, though maybe a good one.
-
Introducing Supabase Evals supabase.com
Matt Rossman on the open-source benchmark Supabase built to find out how well coding agents actually build with its platform, covering “our CLI, MCP server, agent skills, and docs”. Scenarios are organised by product area (Database, Auth), by cross-cutting topic (SDK, observability), and by journey stage. There are two suites: benchmark scenarios for breadth, and regression scenarios that go deep on known failure modes.
Each run happens in a container with both a hosted-like Supabase stack and a local CLI project, so agents call the real MCP and CLI tools rather than a mock. Scoring mixes deterministic checks (whether a user can access certain data, or an Edge Function returns an expected result) with an LLM judge, and agents get one retry before the final grade.
Read more
The results are the interesting part. On the build stage, Opus 5 and Kimi K3 scored 100% with no skills loaded at all, and Sonnet 5 went from 78% to 100% once skills were available. Rewording the description of a Postgres best-practices skill lifted its activation rate from about 10% to 60%, which says more about skill descriptions than about the models. Reading habits differ too: Codex agents pulled in roughly eight doc pages per scenario against about two for Claude Code.
-
Uncle Bob on not reading agent code x.com
Uncle Bob Martin (coding since the late 60s) no longer reads the code his agents write. Instead he surrounds them with extreme constraints, from unit and gherkin tests to mutation testing, and trusts code that has “run the gauntlet of all of my constraints and tests”.
The polar opposite of cognitive coverage: commit at zero understanding and bet the harness makes up for it. Harness engineering and Jason Swett’s TDD skill, taken to the logical end.
-
How we set up our cloud agent environment cursor.com
Mathew Hogan and Arvind Saripalli treat the development environment as a product whose users are agents. The scale of the shift is the reason:
In December, cloud agents authored roughly one in ten PRs merged to the Cursor monorepo. Today, they write more than half.
Getting a Mac-centric monorepo running on Linux cloud VMs meant a Cursor-defined Dockerfile carrying the key dependencies, along with network egress restrictions, secret scanning and redaction of secrets in tool results.
Read more
The other half is making the repo legible. They built
anydev, one CLI that consolidates service startup, routes the utility scripts, carries help menus per subcommand, and runs a supervisor that monitors and restarts long-running builds so agents never have to manage processes by hand. Cursor Cloud MCP lets an agent diagnose its own environment health, and an automation called Cloud Doctor periodically hunts for failures, does root cause analysis and opens PRs to fix them. Cloud Doctor also reads agent traces to find misleading skills, avoidable VM problems and inefficient workflows.The framing is the useful bit: environment friction that a human would silently absorb becomes a measurable defect once agents are the main users. Read with the usual caveat, since this is a coding agent vendor reporting numbers from its own product.
-
How to Build a Trust Platform for Your Agent grafana.com
Jack Gordley on what went wrong as Grafana Assistant grew past its prototype stage, and the observability work that fixed it. The pain is specific and familiar:
Reviewing contributions to the prompt, tool set, and agent harness became a nightmare as changes piled on from multiple teams. Bespoke dashboards just weren’t cutting it.
His underlying point is that agents are inherently non-deterministic and heavily customized for a wide range of use cases, so the usual software review reflexes do not carry over.
Read more
The path out is set as five stages, Phase 0 to Phase 4. Phase 0 is the early prototype and first deployment. Phase 1 is monitoring live traffic: engineering metrics like latency, cost and tokens, plus analysis of the conversations themselves. Phase 2 is measuring baseline quality with LLM-judge evaluators and deterministic checks, including built-in templates for PII detection and toxicity. Phase 3 turns failed conversations into annotated regression suites. Phase 4 adds experiment tracking and CI, comparing models and gating PRs.
They also mention o11y-bench, their own observability benchmark of around 70 tasks, used to weigh Haiku against Sonnet on performance and cost. Worth reading with the caveat that the internal tool became Grafana Cloud’s Agent Observability, now generally available, so this is a vendor describing something it sells.
-
Meet Stripe's Knowledge AI Platform stripe.dev
Anna Mason, Sharadh Krishnamurthy and Anupam Upadhyay on Kai, Stripe’s Knowledge AI Platform: the same bet as coding agents, aimed at everything that is not coding. Coding tasks have fairly uniform workflows, and their point is that the rest of the company’s work does not:
Knowledge work is the opposite side of that spectrum.
It launched in April 2026 and 83% of Stripe employees are now weekly active, with connections to more than 1,000 internal tools and skills. Three layers: surface-agnostic APIs (a web app, Slack, and an embedded Chrome extension that puts Kai inside third-party tools), AgentStudio where domain owners build, test and monitor their own agents and skills, and an execution environment on LangChain deepagents and Kubernetes with per-session sandboxes, virtual filesystems and access control.
Read more
The numbers worth keeping are the operational ones. Stripe says Kai has helped shift 25,000 hours per year from administrative work, to revenue generating work, and that Account Executives using it produce twice the sales activity and close 39% more deals. This is the operations counterpart to the coding agents described in Stripe’s minions and its follow-up.
-
How much autonomy should you give your agents? posthog.com
Jina Yoon’s framework: autonomy should depend on the task, not the model. “Trusting your agents just because the models got smarter is like skipping your seatbelt because you got a nicer car.” Two questions decide it: is the work easy to check, and are mistakes cheap to undo? That yields four levels, from assistant mode (hard to check, costly to undo) up to self-driving (easy to check, cheap to undo, like dependency updates), with PostHog examples of moving work up the ladder.
The checkability axis is doing the same work as the constraint-building in constrain the agent.
-
Building an agentic harness that outlasts the model shopify.engineering
Zack Deveau on the part of an AI security system that is actually worth building. Models get swapped out every few months, so the engineering that lasts is the scaffolding around them: the test bootstrapping, the partitioning, the verification pass, the scoring. His line for it is that this is the part you’ll keep as new models come and go; this is where the innovation is.
The system is Dispatch, a Ruby orchestrator that runs security scans in eight stages. It finds and verifies the repo’s test commands, writes reusable architecture notes, catalogues and partitions files into token-sized clusters, then fans specialised hunting agents out across those partitions in parallel. Findings go through a verification stage that re-tests them with a different model, then deduplication and scoring, and finally a remediation stage that proposes fixes and opens PRs. Tests are the oracle throughout: a finding that cannot be demonstrated with a real test does not survive.
Read more
The numbers, with the caveat that they are self-reported. Over six weeks Dispatch completed full scans of more than 80 unique applications and produced over 300 findings, two of which would have been rated Critical. Shopify conservatively values the set at over $400,000 in equivalent bug bounty payouts. A full scan with frontier models runs $50 to $300 per application; incremental diff scans are $5 to $50.
The design principle running underneath is that precision matters more than coverage, because noise sent to a developer is worse than no finding at all. That is what justifies the expensive verification stage, and it is why the harness, not the model, is the asset.