Review Is the Runtime

If You Only Read One Thing

Agents are escaping the chatbox just as their output becomes harder to inspect. One enterprise's 196,212 pull requests show why Review Becomes the Scarcity; OpenAI's embeddable loop explains why The Harness Leaves Chat. The interface is moving into ordinary software, but the scarce component is moving the other way: a human or policy layer that can still say no.

Review Becomes the Scarcity

AI coding's next bottleneck is judgment, not generation. Agents can double the flow of proposed changes. They cannot double the attention available to decide which changes should ship.

OpenAI's new field guide shows how that pressure starts. Engineers use Codex for migrations, tests, incident triage, performance work, and neglected fixes. One product engineer says four pull requests merged during a day of meetings. OpenAI's own guidance still puts the reliable unit of delegation at roughly one hour of human work or a few hundred lines.

Usage is stretching past that unit. A large study of Codex activity found active users grew more than fivefold in the first half of 2026. More than 10% ran at least three agents concurrently in a week. The share of individual users attempting at least one task estimated above eight human hours rose almost tenfold.

The review queue is the receipt. Carnegie Mellon and Stanford researchers followed 802 developers and 196,212 pull requests from January 2024 through April 2026. Per-developer throughput reached 2.09 times its pre-mandate baseline. Per-reviewer load roughly doubled, and automated review overtook human review, while merge and revert rates held steady.

That last result is the strongest counterargument. If throughput doubled without more reversions, perhaps automated review scaled with generation. The authors are careful: adoption was not randomized, the study covers one AI-forward company, and pull requests measure output rather than customer value. Stable reversions are encouraging. They do not measure whether architectural mistakes, security defects, or lost system knowledge arrive later.

A second Carnegie Mellon analysis of 3,100 practitioner accounts makes the mechanism clearer. Agent-authored pull requests were reviewed less often, merged faster, and discussed less. Those directions changed under some reasonable analytical choices, so the repository traces are not a verdict. The robust insight is that review decides whether faster generation becomes faster delivery or faster accumulation of mistakes.

The practical shift is from counting produced code to counting judged work: changes that were understood, tested, approved, and still held after release. That is a harsher measure than pull-request volume. It is also the one that separates software output from software productivity.

The field study now has a clean falsifier. If its 2.09-times throughput persists for two more quarters while escaped defects, incidents, and rework per merged change stay flat, automated review absorbed the load. If those outcome rates rise, the queue merely moved beyond the study window.

The Harness Leaves Chat

The agent interface is becoming a component inside other software. OpenAI's August 19 release matters because Codex no longer has to look like Codex: a host application can own the screen, tools, approval flow, and record of work while the agent loop runs underneath.

An agent loop is the machinery that turns a request into repeated action. It maintains context, calls tools, reports progress, handles failure, and pauses for approval. Think of the model as the engine and the loop as the transmission. An engine swap matters, but the transmission decides how power reaches the workflow.

OpenAI says the same open-source harness powers its app, command line, and IDE extension. The Codex app-server protocol lets another application create threads, start turns, stream events, interrupt work, and answer approval requests. An operations dashboard can therefore add an agent without sending its users into a separate chat product.

The shift is bigger than one release. Anthropic's Agent SDK exposes the Claude Code loop as Python and TypeScript libraries, including tools, hooks, permissions, sessions, subagents, and telemetry. Google's Agent Development Kit supports models from Gemini and Claude to OpenAI, Ollama, and vLLM, then supplies its own runtime, session, deployment, and evaluation layers.

Open does not mean portable. OpenAI publishes the harness and client protocol, but model access and managed services remain separate. Anthropic requires API authentication for third-party products and governs the SDK under commercial terms. Google accepts more model providers, yet its documentation offers no way to turn ADK session and approval state into a live Codex or Claude session.

This is the new platform contest. The old assistant asked users to bring work into its interface. The new runtime lets existing software pull the agent into a support console, security investigation, or internal queue. Distribution improves because the agent appears where the decision already happens.

The strongest objection is that open protocols should commoditize this layer. They may eventually. Today, thread state, tool events, approval semantics, checkpoints, and usage records remain runtime-specific. A company can swap a model call more easily than it can move a half-finished agent job with its authority intact.

The first meaningful adoption signal is not another SDK download. It is a non-coding product shipping its own approval interface on top of Codex app-server, then showing that interrupted jobs resume without losing tool state. That would prove the harness can leave the developer surface without leaving reliability behind.

The Contrarian Take

Everyone says: More agent-written pull requests prove productivity, and open harnesses weaken model-vendor lock-in.

Here's why that's wrong (or at least incomplete): One enterprise reached 2.09-times throughput while reviewer load roughly doubled. OpenAI opened its loop but kept model access separate; Anthropic exposes a comparable loop under commercial terms; Google supports many models inside its own runtime state. Generation is becoming abundant, but judgment and session custody remain proprietary constraints. The lock-in did not disappear. It moved into the layer that decides what an agent may do and whether its work counts as finished.

Under the Radar

Agent use is spreading before productivity is proven. Codex users grew more than fivefold in six months, fastest outside software development; median OpenAI output tokens rose 13 times in legal and more than 50 times in research. The study measures activity, not time saved or work quality. That denominator gap will matter more as agents enter jobs without a pull request as their audit trail.
Multi-agent messages now admit they can overflow. Claude Code 2.1.236 added a one-shot notify_when_idle signal and now rejects a burst before a target session's inbox silently drops it. The changelog detail is small, but it exposes a general truth: once agents coordinate, queue capacity and delivery acknowledgment become part of correctness.

Quick Takes

Claude adds an answer-first mode. Claude Code 2.1.237 introduced a built-in Concise style that leads with results and skips narration, while also repairing prompt caching behind custom gateways. One change reduces supervision noise; the other prevents a proxy from silently erasing the economics of repeated context. (Source)
Agent review expands past the diff. OpenAI's new app-security whitepaper puts plugin manifests, MCP configuration, lifecycle hooks, launched commands, and credential exposure inside the review boundary. The artifact to inspect is no longer only generated code. It is also the package that defines what the agent may execute and which authority it inherits. (Source)
Faster completion can leave weaker owners. In a 54-student experiment, coding agents improved initial website completion but reduced code comprehension and left users less prepared to extend the result unaided. The sample is narrow, but the effect targets the skill review depends on: knowing enough about the change to reject it. (Source)

The Thread

Agents are moving closer to the systems where work happens and farther from the interfaces where a person can inspect every step. That makes review more than a stage after generation. Review becomes part of the runtime: the approval, test, and ownership machinery that converts an agent action into an accepted organizational action. The winning agent platform may not produce the most output. It may make consequential output easiest to judge.

Predictions

New predictions:

I predict: By November 30, 2026, at least one of GitHub, GitLab, or Atlassian will add an agent-authorship filter to code-review analytics and report human review time separately for agent-authored changes. A generic AI label without a review-time metric does not count. (Confidence: medium; Check by: 2026-11-30)

Issue date: August 20, 2026 · Generated at 3:52 AM ET