FrontierCode Exposes Unfinished Fixes

If You Only Read One Thing

An agent can fix the reported bug and still leave the software broken. Cognition’s FrontierCode examples show what stronger coding tests now demand; Mike Taylor’s source-selection experiment shows how an agent can miss useful material before it starts. Both point to the same overlooked step: what the agent chooses to inspect shapes what it can finish.

FrontierCode Catches Fixes That Miss Surrounding Behavior

Opus 5.5 still tops the harder coding comparison, but it doesn’t reliably finish the whole repair. That’s the useful model-choice finding from FrontierCode 2.0: the premium models retain an advantage when a task demands judgment about surrounding behavior, and even the leader leaves plenty for a reviewer to catch.

Cognition’s public example makes the failure concrete. Opus repairs a generator-reset bug in Mock Service Worker, a JavaScript testing library. It skips the user’s cleanup code inside finally, the block meant to run when execution finishes. Cognition reports failure at every effort setting. The model finds the visible defect and misses an obligation attached to it.

What would more reasoning buy here? In that example, it didn’t buy a complete fix. You can spend more on the same investigation without making the agent inspect the behavior it overlooked.

Cognition expanded the suite to larger tasks and tougher standards for whether a maintainer would merge the change. Its graders consider correctness alongside test quality, scope and repository conventions. They also check thoroughness: fixing the reported instance doesn’t excuse leaving the same problem elsewhere.

Artificial Analysis’s independent runs put Opus at about 46% on the revised coding suite, against 28% for the best released open-weight model, DeepSeek V4.1 Flash. A team handling repairs with unclear scope has a reason to keep a stronger model available. A cheaper worker can handle well-specified edits while the stronger model investigates what else the change affects.

Artificial Analysis also widened the overall gap to Kimi K3.1 Pro, whose weights are still promised, from three points to twelve after changing the test mix. Kimi looks less interchangeable with the leaders than it did in October 24’s briefing. The models haven’t suddenly deteriorated; the evaluator is asking harder and different questions.

Artificial Analysis includes scientific tasks that may barely matter to a coding queue, and Cognition’s merge judgments include subjective grading. I’m fairly sure the coding results favor stronger models for difficult maintenance; I wouldn’t treat the overall gap as a universal spending rule.

A reviewer gains something more useful than another general request to “check the code”: a specific omitted obligation to investigate. Does the repair preserve cleanup and handle the other affected paths? Cognition’s published walkthrough puts the unfinished cleanup beside the patch. On this task, Opus still fails at its highest effort setting.

Clear Descriptions Change Which Sources Agents Cite

Mike Taylor changed the packaging of an article and changed whether an agent used it. In Every’s search-result simulation, a prompting guide went from two citations out of 240 sessions to citations in every session. He kept the article itself unchanged.

Across Anthropic and OpenAI agents shown shuffled search results, Taylor’s explicit title and answer-like description displaced a broad title and a management-science subtitle. The useful change was making the article’s contents visible before an agent opened it.

An agent doesn’t need to dislike the writing to pass over it. It has a question to answer and a small preview with which to judge relevance. When that preview hides the useful material, the agent can select a worse source without ever reading the better one.

That changes how I’d interpret a research agent’s disappointing answer. The agent may have reasoned competently over the wrong shortlist. A stronger final model can’t recover a document that an earlier selection step discarded. The same problem can appear in an internal knowledge search when a page title names a project but leaves out what the page explains.

You can separate those failures by keeping the returned search results alongside the sources the agent actually opens. If the useful document appeared but wasn’t opened, the investigation starts with selection. If it was opened and misunderstood, the investigation starts with reading. A final answer alone makes those two failures look alike.

Taylor’s experiment used simulated search results, and he hasn’t shown that the citation gain produces real referral traffic. He also changed title and description together. I’d bet on descriptive previews helping selection, but I wouldn’t expect the same dramatic gain across every search system.

Taylor ships an agent-pass skill and test script that compare truthful title-and-description variants against competing results. Its saved responses let someone inspect what the agent selected. That makes the preview a testable part of a research system, alongside the document and the final answer.

Taylor says he’ll test live traffic next. That will show whether clearer previews bring readers to the underlying work as well as winning simulated citations.

The Contrarian Take

Everyone says: Raising reasoning effort is the reliable way to get a more complete coding fix.

Here’s why that’s wrong: Cognition’s cleanup example fails at every effort setting. The agent can keep working on the part of the problem it noticed while leaving another requirement untouched. Extra reasoning helps when the model has a hard problem in view; it can also make an incomplete investigation more elaborate. A stronger prompt about the desired outcome, a repository-specific check or a reviewer who notices the omitted path can change the investigation itself. The spending decision depends on which failure occurred. Paying for longer thought is a poor substitute for discovering the missing requirement.

Under the Radar

Offline coding needs a prepared project. Rittika Jindal’s October 27 walkthrough runs OpenCode, Ollama and Qwen with the network disconnected. She reports a completed expense-calculator repair; the companion CSV and log-analysis tasks remain unrun. The useful detail is preparing dependencies and the exact launch command before disconnecting. A model on disk can’t help with an installer that still needs the internet. The walkthrough includes the attempts that needed help.
Claude Code repairs a missing-policy path. Version 2.1.312 fixes organization policy and managed settings failing to load after a VS Code sign-in. It also fixes large instruction files being overwritten after only their beginning was read. Both affect what the agent is actually constrained by during a session. A correct policy file is useful only when the runtime loads it. Anthropic’s release notes identify the affected paths.

Quick Takes

Liquid AI ships LFM3 for specialized local work. The October 27 release brings a small-model family with downloadable weights, API access and documented support across llama.cpp, MLX and vLLM. Liquid’s decision variants return typed judgments, including extracted fields and probabilities, alongside the text and multimodal models. That offers a narrower alternative to asking a general assistant to write an answer for every step. The launch doesn’t yet supply independent proof of dependable agent decisions. (Source)
Codex adds a native Vertex AI provider. Stable release 0.165.0 introduces the gcp provider and improves session navigation across linked worktrees. The provider change lets the same coding client connect through another infrastructure path. Authentication, model availability and organizational controls still belong to that path, so a familiar client doesn’t make the routes interchangeable. The release supplies no matched claim that Vertex improves task quality or cost. (Source)
Google gives other agents a shell-level Gemini client. The October 27 beta gemini-api lets Claude Code, Codex, Cursor and ordinary scripts call Gemini services without adding an SDK to the project. It can analyze files and start managed-agent tasks through the Interactions API. It reduces integration work while adding another provider’s execution and billing. Google warns that minor beta releases can break compatibility, making version selection part of a reproducible setup. (Source)

The Thread

Agents make consequential choices before the work that usually gets graded. A coding agent decides how widely to investigate; a research agent decides which documents deserve opening. A final reviewer sees the patch or answer, but often has to reconstruct those earlier omissions from scratch. That suggests a more useful form of supervision: preserve the alternatives the agent encountered and passed over. For code, that might be the related call sites it found; for research, the returned results it skipped. Neither record certifies correctness. Each gives a reviewer a cheaper way to ask whether the agent stopped looking too soon. The scarce information may be what the agent declined to inspect, even when its final output looks polished.

Predictions

New predictions:

I predict: By November 11, Google will publish at least one additional tagged release of gemini-api after its October 27 beta, with a documented change to command behavior or compatibility. (Confidence: medium; Check by: 2026-11-11)

2026-10-28 · 03:50 ET