METR Finds Agents Misjudge Quality
8 stories · ~7 min read

Listen
If You Only Read One Thing
Four extra test passes turned into a tie when a developer inspected the rejected fixes. Gemini Neon's apparent lead depended on how its tests were written. METR's capability update finds the reverse problem: model reviewers overrating weakly checked work. Both stories show why choosing a model includes deciding which judgments the surrounding system can actually support.
METR Finds Judgment Lags Behind Checkable Work
An agent's ability to finish a long assignment says little about whether it can decide what deserves to ship. METR's November 20 assessment separates those abilities. How much an agent can do on its own depends partly on how well it can check its work.
METR's combined task suites show horizons doubling roughly every 3.6 months, through models including GPT-6 Astra. A 50% horizon is the human task length at which an agent succeeds half the time. A dependable unattended shift would require much better odds.
On weak-feedback tasks, including evaluation design, agents' judgment lagged human experts. In one task, model judges overrated similarly strong or stronger models' work relative to an expert scorer.
A second model can catch a concrete error while still admiring an ambitious but unhelpful research design. Reviewers have more reason to agree when they can point to an observed result: a reproduced failure, a recovered record, a measured speedup. Two favorable explanations can share the same blind spot.
Consider two proposed assignments. One asks an agent to make an existing parser pass a specified compatibility suite. Another asks it to decide which customer complaints justify replacing the parser. Both may take a person a day. The first supplies frequent correction; the second requires deciding which evidence counts. A time budget alone treats them as interchangeable when they demand different supervision.
Agents can take on much more ambitious work when they get fast, trustworthy feedback. METR also warns that training environments increasingly resemble cheap evaluations. So a strong benchmark result carries a further question: does the task resemble the real assignment closely enough?
A comparison on equally long tasks could test this explanation: give agents clear acceptance tests for one set and delayed, expert assessment for the other. If the gap disappears when task length is held steady, feedback quality would become a weaker reason to keep human review involved.
Neon Matches Opus Until Tests Demand One Implementation
Gemini 4 Neon looks like a credible everyday coding alternative to Opus 5.5 in a fresh production-code trial. But its apparent four-run lead vanishes when the author inspects what the tests actually rejected. That inspection changes the conclusion from better coding to similar results at lower cost.
The developer writing as paddo replayed six merged changes on clean snapshots, with 20 medium-effort runs per model. Neon's four extra passes came from one change: Opus implemented equivalent behavior differently and failed tests checking exact list contents. The author counted that as a tie.
Neon's list-price token cost was $26.93 against $38.45 for Opus, about 30% less. The developer’s repository trial adds evidence to our November 19 benchmark comparison and explains why the pass rates were misleading.
A test can accidentally make the original patch the only acceptable answer. For example, when ordering has no effect on an application's behavior, an assertion comparing two lists element by element may reject a valid reordering. Changing that assertion requires checking the real contract first: sometimes order is exactly what the customer depends on. A failed check can expose a model error or a test that enforces the wrong contract.
The same trial also challenges automatic orchestration. Google's Orchestrator mode pairs a Neon planner with a faster executor. Across six changes, the orchestrator matched solo Neon's results while taking 16 minutes per fix against seven. Orchestration used xhigh effort; solo Neon used medium. The settings and small sample prevent attributing the entire overhead to delegation.
Splitting a short fix adds planning, handoffs and reconciliation before it creates useful parallel work. A single agent may already have enough capacity to finish. Longer jobs with separable pieces could reverse that calculation. Opus also ran in Claude Code while Neon used Antigravity, so the comparison measures those working combinations.
Switching pays when the models deliver similar accepted behavior at a lower total cost, after failed assertions have been checked for whether they enforce a requirement or merely the original implementation. A saving that disappears once genuine failures are repaired wouldn't justify moving the work.
The Contrarian Take
Everyone says: Google recommends Orchestrator as the default for day-to-day development, including bug fixes.
Here's why that's wrong: An ordinary fix can be too small to repay the cost of planning and handoffs. Paddo's trial rejects that default for his short fixes, even though its effort settings leave the precise cause of the overhead unresolved. A cheaper worker could repay the planner's work if it spends long enough on implementation. Starting with a single capable model makes that extra coordination justify its cost. Treating every bug fix as a management problem gives the planner work to do before proving the job needs one.
Under the Radar
-
A Gemini client omitted a whole effort setting. Willison's November 20 llm-gemini repair exposes Hydrogen's
xhighoption. In his single illustration prompt, moving from high to xhigh raised cost from about four cents to 24 cents. That explains how a client can quietly narrow the model being compared. The example measures one creative task; it gives no general coding-quality gain. The release and experiment make the setting and its cost reproducible. -
Muru shows what happened inside a Python request. Dumpleton's November 21 walkthrough connects the development toolbar to wrapture instrumentation, exposing database queries and outgoing service calls. An agent debugging a slow page can then inspect the calls consuming time rather than infer everything from the total request duration. This is development instrumentation with explicit configuration, including whether SQL text is recorded. The working Flask example provides the artifact; no debugging-time saving has been measured.
Quick Takes
-
Canary credentials exposed inference-provider leaks. Dragos sent nearly two million prompts containing unique test URLs and credentials, then observed attempts to use those credentials against controlled systems. OpenRouter has removed 125 endpoints from zero-data-retention eligibility and plans ongoing checks using the researchers' approach. A provider's privacy label needs verification after inference too: leaked credentials may be used long after a request finishes. (Source)
-
GPT-6.2 binds private reasoning to conversation history. OpenAI's November 20 disclosure says rewriting an earlier message drops the reasoning that followed it. Its internal adaptive test also cut extraction of earlier reasoning from 54% on the old Astra setup to 0.2% on the updated model and safeguards. When applications edit or branch histories, they can lose reasoning state; an identical-looking continuation may have different context. (Source)
-
DaisyRose offers an overlooked open-weight agent option. RadixArk's November 18 release builds on Qwen3.8-27B, with Apache 2.0 weights and serving guidance. Its own Terminal-Bench 4.1 comparison rises from 5.6% to 12% after reinforcement learning. More than doubling a low score still leaves most tasks unsolved. The release is worth testing for bounded local work where failures are visible, rather than assuming a relative gain makes it a dependable general agent. (Source)
-
A published inference recipe still needs unavailable patches. Artificial Analysis's November 21 DeepSeek V4.1 Flash configuration names its model revision, container and launch commands. It also explicitly lists nonpublic NVIDIA Rubin patches. That makes a reported hardware result easier to audit while limiting who can reproduce it. A deployment estimate needs to include access to the same software stack before treating benchmark throughput as achievable capacity. (Source)
The Thread
A review system needs examples it ought to reject. A coding test can pass every valid patch and still accept a broken one; a privacy monitor can record no leaks because nobody has checked whether its alarms work. Dragos created credentials whose only purpose was to reveal unexpected access. Model reviewers could face a separate test set mixed with known defects, with their misses measured. That's a proposed extension, especially for METR's weak-feedback tasks, where experts must first agree on what makes an answer defective.
As models improve, an unchanged pass rate can conceal a deteriorating check. If more bad patches get through, the team may attribute the rising completion rate to better coding. Keeping a few deliberate failures in the evaluation would make that interpretation harder to sustain.
Predictions
- I predict: Google's public Antigravity model picker will still offer Neon-based Orchestrator alongside standalone Neon on December 22, preserving orchestration as a task-specific choice. (Confidence: high; Check by: 2026-12-22)
November 22, 2026 · 3:31 AM ET
Tomorrow morning in your inbox.
Subscribe for free. 10-minute read, every weekday.