Success Without a Recipe

If You Only Read One Thing

An agent can finish the assignment and leave its owner unable to reproduce it. Astra Delivers Without Its Working Papers exposes that gap in a running-route experiment; Private Code Defeats Familiar Answers finds another in enterprise software. Real-SWE’s private-code comparison tests whether frontier agents can recover company-specific meaning. Both stories make the evidence surrounding a result part of the deliverable.

Private Code Defeats Familiar Answers

The frontier coding model is still a provisional worker on unfamiliar business software. Specific’s new Real-SWE comparison, circulating this weekend, puts Fable 5.1 with Claude Code first at 38.8% completion, followed by Astra with Codex at 33.8%. The useful conclusion is how much work remains unsolved across the field, rather than the five-point ranking gap.

The test covers ten tasks, eight model-and-runtime configurations, and eight independent attempts per task. Those repetitions yield 640 runs, but they do not create 640 different business problems. The underlying repositories are licensed private company code, and the work includes billing, taxes and customer migration. These are small changes in a request that can carry large obligations in a product.

Company-specific meaning is a difficult input even when the instruction supplies it. Real-SWE’s invoice example explicitly distinguishes manually set rates, destination-based tax and exempt customers. The agent must carry those rules through the existing application. Call this the business-rule gap: a plausible implementation can lose a stated requirement while preserving the appearance of correct arithmetic.

This changes the interpretation of model choice. Specific classifies 18 of Astra’s 53 failures as unverified assumptions and another 15 as missed requirements. A broad benchmark can suggest which system deserves the first attempt. It cannot establish that the system has correctly located the rule governing a particular migration. The practical extension to independent code review is to separate implementation correctness from premise correctness: a second reviewer can inspect elegant code and still accept the same unsupported business assumption.

There is a strong objection to treating this result as an enterprise-wide verdict. Ten tasks are a narrow sample, and native runtimes make this a comparison of complete systems. Private repositories also limit outside reproduction. Specific says grading checks are introduced only at evaluation time; that improves separation from the agent’s work without proving the checks capture every business requirement.

The result nevertheless weakens the assumption that a high public coding score makes unfamiliar maintenance routine. It does not measure a context-improvement intervention, so it cannot establish that adding a requirements document would recover the missing performance. That distinction matters after September 9’s evidence about late requirements: identifying a failure mechanism is different from proving its remedy.

The decisive next test is a paired run on the same tasks: does supplying independently verified business rules improve completion while holding the model and runtime fixed?

Astra Delivers Without Its Working Papers

Astra can produce a useful custom application while leaving the calculation behind it unavailable. The reported case separates durable output from reproducibility. Where Real-SWE asks whether an agent found the right business rule, this case asks whether anyone can reconstruct the rule it actually used.

In a September 12 experiment, Simon Willison asked ChatGPT Work with Astra Max for 5K and 10K loops from his home using OpenStreetMap data. After 27 minutes, it supplied a visualization and downloadable route files. Later, it could not provide the original Python code. Willison suspected conversation compaction; the published account does not establish the internal cause.

The surviving artifact is unusually informative. Its public HTML contains route coordinates, surrounding roads and labels, plus JavaScript to draw them. It preserves the result as structured data rather than leaving the map trapped in a screenshot. The drawing library is loaded externally, so the file is portable without being completely self-contained.

A saved result and a saved recipe solve different problems. Imagine receiving a spreadsheet containing final values but no formulas. The values remain useful, and charts can be redrawn, yet a changed input cannot be processed with confidence. The map has that same distinction: its coordinates support reuse, while the missing calculation prevents checking how the route was selected.

The demonstrated practice is to make structured exports part of completion. September 6’s portability test asked whether a patch travels. This account adds a different boundary: an export can travel while the process that generated its data does not. This case supports that narrow benefit. It does not show that a particular prompting formula prevents lost code, or that preserving every transcript would recover files already discarded elsewhere.

The cost is deciding which evidence deserves to survive. For exploratory visualization, coordinates may be enough. For a recurring analysis, the input snapshot, transformation code and dependency versions become part of the product because someone will eventually need to rerun it. That is an engineering inference from the failure, not a measured improvement in Willison’s experiment.

A persuasive-looking map also cannot establish that every segment is accessible or that route constraints were applied correctly. The falsifiable completion test is reproduction from saved inputs and code in a fresh environment, without asking the original conversation to reconstruct its past.

The Contrarian Take

Everyone says: Longer-running agents will steadily remove the need for human intervention.

Here’s why that’s incomplete: More execution time helps only when execution is the missing ingredient. Real-SWE exposes work that depends on company-specific interpretation; Astra’s map exposes a result whose original calculation could not subsequently be inspected. Neither problem is automatically solved by allowing another hour. The more useful distinction is between work that needs additional computation and work that needs a missing fact or a durable record. Autonomy can increase the volume of finished output before it increases the volume of work another person can confidently take over.

Under the Radar

Short failures complicate the long-horizon story. In Real-SWE’s run breakdown, 70 of 98 attempts lasting under ten minutes failed, against 398 of 542 longer attempts. Those are roughly 71% and 73%, respectively. The groups are not randomized, so this does not show that extra time is useless; difficult tasks may simply consume longer. It does show why elapsed time alone cannot diagnose the constraint. A fast wrong assumption and a slow integration failure need different interventions.
Safety tuning can include executable checks. The September 5 EvoSafeHarness paper, surfaced in this week’s research feed, jointly searches written policies and enforcement code for a fixed model and domain. Its authors report attack success falling from 45.6% to 10.0% on one benchmark, with a 3.3-point utility cost, and publish an implementation. The relevant practice is testing both blocked attacks and legitimate work while refining the protection. These are experimental results, not a verified production defense or permission to let an agent weaken its own controls.

Quick Takes

Gemini’s cost advantage survives one stricter denominator. Real-SWE lists Gemini 3.8 Flash at an estimated $2.50 per run and 31.2% completion, versus Fable’s $6.96 and 38.8%. Dividing average spend by completion gives approximately $8 versus $18 per successful run across this task mix. That calculation includes failed-run expenditure, but excludes human repair and does not estimate repeated attempts until success on a particular task. It supports a cost-sensitive shortlist, not universal substitution. (Source)
SenseNova adds the report behind the visual model. The September 11 repository announcement links the U1.5 technical report; the weights arrived August 20. The report examines how shared visual understanding transfers to complex image instructions, relevant to diagrams, layouts and illustrated explanations. This is new supporting evidence for an available model, not a new weekend launch. Claimed instruction-following gains still need independent checks on exact text and preservation during edits. (Source)
A visualization can carry its own data. Willison’s exported map embeds route, road and label data directly in the HTML. That makes its contents inspectable separately from ChatGPT and avoids needing the original conversation merely to recover coordinates. The useful boundary is equally concrete: inspecting or redrawing geometry is not rerunning the route-finding process. Portable presentation and reproducible computation are different deliverables, even when they arrive in the same chat response. (Source)

The Thread

AI work needs working papers on both sides of execution. Before a change, the papers establish which company rule the code must implement. After a calculation, they preserve the inputs and transformations behind the answer. The failure modes differ: Real-SWE agents can overlook rules already supplied, while the route account describes calculation code becoming unavailable afterward. One needs evidence that requirements reached the implementation; the other needs preserved execution evidence. This extends September 11’s distinction between running work and judging success into a handover problem: what must travel with a result for someone else to own it?

The same question reaches agent memory. A stored conclusion such as “customers in this category are exempt” is cheap to retrieve. Its source, scope and effective date make a changed rule detectable. Preserving those fields still cannot prove the agent applied them: the acceptance test must connect the stored rule to the resulting behavior. Durable memory earns its value when it preserves the basis for revision, not simply the previous answer.

Predictions

I predict: By October 13, Real-SWE’s published native-runtime comparison will still have no configuration above 60% completion on its original ten-task set. Confidence is medium: the current leader is below 40%, and the problems include integration and business interpretation rather than only token generation. A higher result on that unchanged set falsifies the prediction; replacement tasks do not. (Confidence: medium; Check by: 2026-10-13)

2026-09-13 · 03:38 ET