Muse Trades Tokens for Work

If You Only Read One Thing

Model labels have stopped being specifications. Muse Spark 1.2 spends more inference than 1.1 to complete better agentic work; the Endpoint Accuracy Index finds that providers can remove capability while serving identical weights. One lab improved what the label delivers, while several hosts diluted it. Buying AI increasingly means auditing the runtime, not merely choosing the model.

Muse Buys Better Work

Muse Spark 1.2 is only three points smarter than its predecessor on Artificial Analysis's aggregate index. On the work benchmark that looks most like an agent's actual job, it improved by 260 Elo. The gap between those numbers is the release.

July 26's briefing covered Muse Spark 1.1 as Meta's move into paid developer inference. Version 1.2 changes what that same API delivers without changing its token rates.

Artificial Analysis tested Meta's third Muse Spark version in four months and scored it at 54 on its Intelligence Index, up from 51 for 1.1. The striking move came on GDPval-AA v2, where Muse rose from 1,371 to 1,631 and ranked fifth. GDPval is closer to a work sample than an exam: models use a shell and web browser to produce artifacts from professional tasks, then graders compare the deliverables pairwise and convert those preferences into an Elo rating. Muse's Terminal-Bench score moved only from 78% to 80%, while several scientific evaluations were flat or worse.

The gain was not free. Meta kept list prices at $1.25 per million input tokens and $4.25 per million output tokens, yet Artificial Analysis measured average task cost rising from $0.29 to $0.40. Muse consumed 53% more input and 36% more output tokens. In other words, Meta did not merely make each generated token better. It taught the system to spend more inference on the tasks where additional work changes the artifact.

Why it matters: Per-token price is becoming a poor proxy for agent economics. A model that uses fewer cheap tokens and leaves a human to repair the result can be more expensive than one that spends eleven extra cents and finishes the deliverable. Muse's aggregate score barely moved because broad indexes average together tasks where more work helps and tasks where it does not. Its agentic score suggests that inference allocation—not just base capability—is becoming part of post-training differentiation. The relevant unit is completed, accepted work.

Room for disagreement: The improvement is concentrated. Scientific reasoning did not share the agentic jump, and no independent repository-level coding study surfaced in today's sweep. GDPval also grades artifacts under one tool environment; it cannot tell us whether Muse will spend its extra tokens well inside Codex, Cline or Aider. The evidence supports a narrow claim about agentic work, not a universal capability upgrade.

The Endpoint Changes the Product

An open-weight model can have a public checksum and still arrive as several different products. The divergence happens after the weights leave the lab.

Artificial Analysis's new Endpoint Accuracy Index runs serverless APIs against a self-hosted reference using the official weights, recommended precision and serving recipe. Think of the model name as a software version: the weights are the source release, while quantization, context limits, output caps and the inference engine are build flags. Two vendors can advertise the same version and execute materially different binaries.

The index gives the reference deployment a score of 100 and currently tests tool calling, scientific reasoning and long-context recall with equal weight. The first results show several failure modes. A hosted gpt-oss-120b endpoint completed only 22% of tool calls against 37% for the reference. Restrictive output limits pushed some GLM-5.2 endpoints to half the reference result or less on Humanity's Last Exam. Lower-performing endpoints often emitted roughly half as many output tokens. DeepSeek V4 Pro providers, by contrast, clustered near parity, and its first-party API edged above the reference.

Provider variance itself is not new. A June serverless comparison measured a 34-fold spread in latency consistency for the same model across unnamed hosts. The new index advances the argument from speed to behavior: a host can reduce the thinking budget, truncate context or change precision without changing the model label on the invoice.

Why it matters: Open weights were supposed to make inference providers interchangeable. Instead, the serving layer retains room to differentiate—or degrade—the product through decisions users cannot see in the model ID. That gives low token prices an adverse-selection problem: the cheapest endpoint may be cheap because it withholds computation needed to reproduce the reference behavior. It also changes evaluation provenance. “Tested GLM-5.2” is incomplete unless the result records provider, context ceiling, output limit, precision and serving configuration.

Room for disagreement: This is a point-in-time index covering three task families, and providers can fix configuration mistakes quickly. A score below 100 can also reflect a deliberate price-latency trade rather than negligence. The durable proof will be repeated measurements showing whether gaps persist and application-specific evaluations showing that they alter completed work, not just benchmark averages.

The Contrarian Take

Everyone says: Open weights commoditize inference because every host starts with the same model.

Here's why that's wrong (or at least incomplete): Weights standardize the starting artifact, not the delivered computation. Artificial Analysis found tool-call success of 22% on one hosted gpt-oss-120b endpoint versus 37% on its reference, while output limits cut some GLM-5.2 results in half. Precision, context, token budgets and serving software recreate product differentiation downstream. Open weights make provider quality measurable; they do not make providers interchangeable.

Under the Radar

• Anthropic inserts a veto before inference: Enterprise administrators can now route every governed prompt from Claude, Cowork and Claude Code through an external inference hook. The security server sees transcript text and tool results, then returns allow or deny before the model runs; organizations choose whether a five-second timeout fails open or closed. The control is prompt-side only, excludes API organizations and does not inspect raw files or images.
• OpenAI makes shared-key waste attributable: The Usage and Costs dashboard can now filter and group activity by API key. That is a small schema change with operational utility: teams can separate one application, environment or agent fleet inside a shared project without waiting for a new billing hierarchy, making anomalous spend and regressions easier to assign to an owner.

Quick Takes

Fast Mode Reaches Long Context

OpenAI's Fast mode now accepts GPT-5.6 Sol, Terra and Luna prompts beyond 272,000 tokens while retaining the advertised speedup of as much as 2.5 times. The change removes an awkward routing cliff for repository-scale and document-heavy agents, but it also makes latency a selectable premium on the workloads most likely to generate large bills. (Source)

Opus 4.1 Leaves the API

Anthropic's scheduled August 5 retirement date for Claude Opus 4.1 has arrived, with Opus 4.8 as the migration target. Any pinned evaluation, fallback chain or cached model catalog that treated retirement as a future concern now has an explicit failure boundary. The useful test is whether the replacement preserves tool behavior, not merely prompt compatibility. (Source)

Muse Learns to Decline

Muse Spark 1.2 reduced its hallucination rate from 38% to 28% by attempting only 67% of questions, down from 82%; overall accuracy slipped from 41% to 38%. That is not a clean reliability win. It is a policy shift toward abstention, which helps only when the surrounding agent can recognize “no answer” and route the work elsewhere. (Source)

The Thread

Today's systems are learning to buy capability with hidden budgets. Meta lets Muse spend more tokens on agentic work; endpoint providers decide how much precision, context and output survive delivery; Anthropic lets an administrator spend latency on a pre-inference veto; OpenAI sells extra speed for very long prompts. The scarce capability is no longer just generating the next token. It is controlling which computation is permitted, paid for and preserved all the way to a finished task.

Predictions

New predictions:

• I predict: By September 6, Artificial Analysis will add Kimi K3 to the Endpoint Accuracy Index, test at least five providers, and find at least one composite endpoint below 90% of its reference deployment. This is wrong if any of those three conditions fails. (Confidence: medium; Check by: 2026-09-06)

Issue date: 2026-08-06 · Generated: 2026-08-06 03:54 AM EDT