Eleven Gains, Nine Losses

If You Only Read One Thing

Nine lost catches can matter more than eleven new ones. Reviewers Trade Blind Spots examines CodeRabbit’s paired evaluation; Opus Resets the Shortlist explains why stronger, cheaper reasoning deserves another comparison. Together they make model retirement a separate decision from model adoption. The new contender must justify its place, but removing the incumbent requires knowing which protection leaves with it too.

Opus Resets the Shortlist

Claude Opus 5.5 makes the premium-model decision harder to justify by habit. Yesterday’s release brings evidence of stronger professional work into Anthropic’s cheaper Opus tier. The consequential change is the shortlist for demanding tasks, not simply the price of repeating yesterday’s requests.

The independent evidence is substantial. Artificial Analysis’s September 22 evaluation puts maximum-effort Opus 5.5 at 58 on its Intelligence Index, the highest result it has measured. That index combines ten tests rather than representing a percentage of real jobs completed. Opus also leads its private professional-work evaluation on analytical quality and presentation, while matching Astra on its terminal-task test. It still trails on some other tests, including long-context reasoning.

The price story requires separating the rate from the amount of reasoning purchased. Think of the effort setting as authorizing a longer investigation before an answer. More investigation can resolve a difficult problem, but it also consumes paid output even when the visible reply stays short. The useful comparison is an effort-specific result: the completed work at a stated setting, with its total bill attached.

Anthropic’s launch tests make medium effort the important starting comparison. On FrontierCode, which evaluates whether code changes would be merged, medium-effort Opus 5.5 scores 54.6%, against Astra’s best 53.3%, at roughly one-fifth the reported task cost. Those are vendor-reported results. They support a serious substitution test; they do not establish that every existing Fable or Astra workflow has become wasteful.

Maximum effort gives a different picture. Artificial Analysis measured about 119,000 output tokens per index task, versus 73,000 for Opus 5. Its measured cost per task stayed roughly level with Opus 5 despite cheaper tokens.

Those runs enabled Anthropic’s default fallback to other models. The measured result therefore describes that configuration, not necessarily Opus alone.

The strongest objection is that benchmarks cannot price the corrections a real project needs. That matters most when an apparent saving produces extra review or an incomplete deliverable. But it cuts both ways: retaining an expensive incumbent without checking the new medium-effort option also makes an unsupported assumption about quality.

The practical consequence is a fresh comparison at the effort level that can finish the job. The API default is medium, so comparisons need an explicit effort setting. The decisive follow-up is whether medium-effort Opus preserves accepted-output rates on an existing workload while reducing the full task bill, including repair.

Code review exposes why winning that comparison does not automatically justify replacement. CodeRabbit’s September 22 test makes a better migration practice concrete: compare the identities of missed bugs before choosing a reviewer.

Across 80 shared bug patterns, CodeRabbit’s lower-effort “Standard” configuration caught 51, against 49 for its production model mix. Forty catches overlapped. Opus found 11 additional issues but missed nine baseline catches. Standard describes settings across CodeRabbit’s pipeline, not a single Anthropic effort value. These are results after verification and filtering, not raw model responses.

This is marginal coverage, the extra protection a component supplies to an existing system. Think of two inspectors checking the same building: the second inspector earns their place by finding important defects the first missed. A higher individual score says little about that contribution until the actual findings are compared. The relevant question becomes “what does this reviewer add?”

The practice extends the existing two-model review loop without assuming that model diversity is enough. A replay can hold the candidate change and known bug labels fixed, then record catches made by both reviewers, by either reviewer alone, and by neither. Severity belongs beside those categories. Losing one dangerous concurrency defect may matter more than gaining several minor findings.

This differs from September 8’s directional-review result, which asked whether one model’s revisions improved another’s draft. Here the decision is replacement versus additional inspection. A reviewer can deserve a specialist role even when it is a poor universal replacement.

Additional inspection has a human cost. Standard generated 127 comments versus the baseline’s 116. Higher effort caught 50 bugs and produced 140 comments. All tested Opus configurations consumed more tokens; the pipeline’s judge determines precision, while developer acceptance was not measured.

The overlap makes a combined review worth testing, but does not prove that running both improves production outcomes. Duplicate warnings, conflicting explanations and triage time could consume its benefit. The adoption threshold is concrete: the value of confirmed additional defects caught must exceed the added review time, with no unacceptable loss among previously caught defects.

The Contrarian Take

Everyone says: A stronger, cheaper model makes its predecessor obsolete.

Here’s why that’s wrong (or at least incomplete): Replacement is only one way to use an improvement. A new reviewer might be most valuable on changes the incumbent struggles with, while a cheaper general model might handle repeatable work without taking over final deliverables. CodeRabbit’s exchanged catches make that distinction observable. The inference is not that every workflow needs more models; it is that removing the old model should require evidence about what disappears with it. Otherwise, a better average score can conceal a worse system for the failures that matter most.

Under the Radar

Less code can contain more bugs per line. Sonar’s September 22 Java evaluation found Opus 5.5 produced 27.5% fewer lines than Opus 5 and fewer total analyzer-classified bugs, yet bug density rose about 12%; the executable pass-rate comparison covers only 544 tasks, not the full 4,444-task analysis corpus. Compactness reduces the material to maintain, but a shorter diff still needs substantive correctness checks rather than approval for looking economical.
A valid tool schema no longer guarantees a tool call. Opus 5.5’s migration contract rejects forced-tool settings that Opus 5 accepted; strict argument validation remains available for calls the model actually makes. Applications that used a required call as their completion signal now need to detect its absence, which is a different guarantee from receiving correctly shaped arguments.

Quick Takes

Sol gets cheaper; Luna needs a separate verdict. GPT-6 Sol and Luna launched September 22. Artificial Analysis’s Codex tests put Sol two points above its predecessor at roughly half the task cost, but Luna two points below at roughly 60% lower cost. Both also regress on its professional-deliverable benchmark. That supports cheaper coding candidates, not a blanket upgrade for research reports and presentations; workload-specific acceptance matters more than the shared generation number. (Source)
Maximum reasoning can prevent delivery. Simon Willison’s September 22 Opus 5.5 SVG experiment exhausted the 128,000-token output limit while still reasoning, returning no finished drawing. One deliberately whimsical task is not a general reliability benchmark. It does establish a concrete failure mode: reserving enough budget to reason does not reserve enough to deliver, so successful completion belongs beside visual quality when comparing effort settings. (Source)
vLLM expands the deployment menu. Version 0.30.0 adds DeepSeek-V4.1-Flash and GLM-5.3-Flash support, alongside serving optimizations for other recent open models. The useful change is another maintained route from weights to a running service, rather than a new capability claim. Hardware-specific kernel improvements in the release notes do not establish lower end-to-end cost for every deployment; capacity planning still needs the actual model, accelerator and request mix. (Source)

The Thread

The useful asset in a review system is a failure caught before it escapes. CodeRabbit’s nine baseline-only catches give an incumbent a reason to survive even after a challenger wins the aggregate score. This advances yesterday’s model-and-agent fit argument: adapting a model concerns its individual performance; retaining two models depends on how little their useful findings overlap. That creates diminishing returns from adding reviewers. A third model that repeats existing warnings consumes attention without adding protection, however high its standalone score. Conversely, cheaper reasoning could make complementary inspection affordable, but only if the additional findings survive human triage. My inference is that model retirement needs its own evidence: repeated absence of valuable unique catches on representative changes. A component’s value can fall because another model learned its specialty, even when its own benchmark score never changed.

Predictions

New predictions:

I predict: By October 7, Claude Sonnet 5.5 will have a public model identifier and documented paid API availability. Anthropic’s stated coming-weeks release plan is the basis; a preview announcement without access will not count. (Confidence: medium; Check by: 2026-10-07)

2026-09-23 · 03:34 ET