Open Models Chase Yesterday

If You Only Read One Thing

Yesterday's benchmark leader and today's best reviewer share an awkward property: their value depends on the comparison point. Open Weights Chase Yesterday separates catching an old score from catching the live frontier; Review Direction Changes Accuracy separates useful error correction from destructive second-guessing. The 19-snapshot record turns “parity” from a verdict into a moving, workload-specific threshold.

Open Weights Chase Yesterday

Open weights are catching proprietary models faster without catching the live frontier. That apparent contradiction matters more than another declaration of parity because it separates capability diffusion from current model choice.

A new analysis of 19 dated model snapshots measures two clocks. Prior-frontier catch-up asks how long an open model takes to reproduce an earlier proprietary score. Live-frontier distance asks how far that same open model remains behind the best closed model available now. Think of a runner reaching yesterday's finish line while the race continues: the first clock measures diffusion; the second measures leadership.

The clean June 21–August 18 window shows why both clocks belong in a deployment decision. Kimi K3 reached 99.6% of Claude Fable 5's earlier coding score after 28 days and 99.7% of its aggregate-intelligence score after 58 days. Qwen3.8 exceeded a 24-day-old Claude Opus 5 agentic score. Yet the live proprietary leader still finished 2.7% ahead on coding, 3.5% on agentic work and 5.4% on aggregate intelligence.

Longer-horizon evidence points in the same direction at a slower cadence. Epoch AI estimates that the best open models lagged the closed frontier by an average of four months from January through May, with an eight-point capability gap and a 90% confidence interval of seven to 11 points. Requiring the open model to strictly exceed the old closed score stretches the estimated lag to six months.

The practice consequence is a threshold test, not an open-versus-closed allegiance. Nine of the September snapshot's 14 intelligence-price frontier points were open-weight models, so an open stack can become the rational cost choice before it leads on capability. The relevant comparison is the workload's coding, agentic, latency and operating thresholds against the complete deployed systems. A downloadable checkpoint still needs serving, tool parsers, recovery and monitoring before it replaces an API product.

The strongest objection is that public benchmarks flatter open models trained aggressively against public tests. The 19-snapshot record also crosses a methodology change, contains no current math result and omits production reliability. Those limits make “weeks behind” a measurement of selected scores, not a universal law. The signal strengthens only if a future same-method snapshot puts an open model within 2% of the live coding frontier while a production endpoint preserves that result.

Review Direction Changes Accuracy

Independent review is not automatically a safety layer. A controlled experiment finds that the reviewer can repair a weaker model's residual errors or replace a correct solution with a worse one; which outcome dominates depends on role assignment.

The study ran six writer-reviewer combinations across 116 LiveCodeBench problems. Codex GPT-5.5 passed 71.6% alone. Claude Opus 4.7 review lifted those same drafts to 89.7%, an 18.1-point gain; Codex self-review reached 84.5%. Claude started at 91.4% alone, stayed there under self-review, and fell to 82.8% when Codex reviewed its drafts.

The mechanism is review headroom: a second pass pays when the writer leaves errors the reviewer can recognize without discarding correct structure. Claude fixed 26 failed Codex drafts and broke five passing ones. In the reverse direction, Codex fixed only three Claude failures but broke 13 successes. The authors observed helpful reviews preserving interfaces and repairing local invariants, while harmful reviews more often replaced a working approach. They did not measure rewrite frequency, so that mechanism remains a grounded hypothesis rather than a proven model trait.

This refines the existing two-model review system. The extension treats writer-reviewer direction as a versioned configuration and separates “should this draft change?” from “produce a replacement.” It also puts executable gates above reviewer prose. A separate peer-reviewed study found that richer explanation-and-fix prompts made models reject correct programs more often; executing the proposed fix as a counterfactual cut average false rejections from 54.8% to 16.3% on HumanEval and from 69.0% to 28.9% on MBPP.

Review has a visible price. Claude-on-Codex added about $0.25 and 74 seconds per task, or roughly $1.40 per net fix. Codex self-review added $0.12 and 29 seconds, or $0.95 per net fix. Claude self-review added 72% to cost and 58% to latency without improving accuracy.

The boundary is substantial. These were self-contained code-generation problems using older models, manual prompts and a static reviewer that could not run tests. The result does not establish the best direction for repository-scale review, architecture, security or current Astra and Fable models. A real-repository replication clears the bar only if it reports fixes and regressions separately; a net-positive review rate with regressions below 5% would make direction a durable workflow setting rather than a one-benchmark curiosity.

The Contrarian Take

Everyone says: Open weights are now only weeks behind closed models, and adding a second model makes agentic code safer.

Here's why that's wrong (or at least incomplete): The weeks measure reaches an older frontier; the current proprietary leader remained 2.7%-5.4% ahead in the same comparable window. A second model is similarly not independent by definition. Codex review broke 13 passing Claude solutions while fixing three failures. Both claims confuse adding another option with adding useful error correction. The gain appears only when the new component clears a measured threshold the existing system does not.

Under the Radar

Shared hardware can make local inference cheaper; a dedicated box can make it dearer. A 56-day coding-agent case study reports that 99.3% prompt-cache hits reduced realized API cost by 88.6%. Shared local GPUs still cut modeled total cost 40.1%, while dedicated reservation cost 43.8% more. The local period's 74.9% repair-commit share versus 45.9% for cloud makes this a workload-and-labor result, not a universal hardware verdict.
Coding agents can differ sixteenfold before reading the request. hibench's intercepted first-turn census counted 20,556 default tokens for Claude Code, 8,710 for Codex CLI and 1,264 for Pi using one tokenizer. It measures an empty repository and a single “Hi,” not full-session cost. That boundary is the point: fixed harness context is a tax that task benchmarks usually hide.

Quick Takes

A vendor benchmark still produced a useful price frontier. Octomind ran four agents on 25 real 2026 pull-request tasks: its own GLM-5.2 setup solved 24, Claude Code 23, Codex 21 and OpenCode 19. Codex cost $14.86 versus Claude's $81.79 and finished in one hour versus 6.7. The participant-designed, one-run suite cannot crown a winner, but it makes cost per solved task harder to ignore. (Source)
Codex now preserves the evidence its reviewer saw. Release 0.153.3 retains Guardian review evidence across compaction, keeps drafts after app-server disconnects and makes permission transforms aware of executor paths. These are small changes with one shared consequence: a resumed session is less likely to silently lose the state that justified a review or approval decision. (Source)
Execution is not always the source of repair success. Across 3,000 attempts on 200 SWE-bench problems, commercial agents lost only 1.25 percentage points when code execution was prohibited, a difference the study could not distinguish from noise, while token and wall-clock cost fell. That does not generalize beyond program repair, but it challenges the assumption that every coding loop needs unrestricted execution. (Source)

The Thread

Today's recurring mistake is counting an added component before measuring its marginal contribution. Open weights add deployable supply, but only one clock asks whether they close today's capability gap. A reviewer adds another pass, but only fix-minus-regression counts show whether it improves the draft. Local hardware adds owned capacity, while utilization and repair labor decide whether it lowers total cost. Even a coding agent's default context spends tokens before the task begins. The useful unit is not models, reviewers, GPUs or tools. It is the measured change in accepted output after each one enters the system.

Predictions

New predictions:

I predict: By November 7, a methodology-compatible public snapshot will show an open-weight leader within 2% of an earlier proprietary coding or agentic frontier while still more than 2% behind the live proprietary leader in that same dimension. (Confidence: medium; Check by: 2026-11-07)

Issue date: September 8, 2026 · Generated: 4:55 a.m. ET