Prompts Spend the Budget

If You Only Read One Thing

A prompt can spend more compute than its wording suggests, while an API key can carry less identity than its permissions imply. A preregistered coding-agent study makes The Prompt Spends the Budget measurable; Gemini Gives Keys an Identity turns Google’s September migration into an authorization boundary. Strings that looked like configuration are becoming executable policy.

The Prompt Spends the Budget

The expensive sentence is not necessarily the long one. It is the one that gives a reasoning model permission to keep searching.

A new preregistered study ran six large reasoning models through 24 deterministic coding tasks in two real agent harnesses, producing 4,643 valid runs. Asking models to “develop and compare several approaches” increased reasoning tokens by 2.4–7.4× across all six models without improving correctness. “Think deeply” cost 1.6–2.2× more. A bounded instruction specifying scope, acceptance criteria and a stopping condition added no general overhead and cut one model’s reasoning roughly in half.

That changes the useful definition of prompt efficiency. The familiar concern is input length: shorter prompts cost fewer tokens. Here, wording acts as a compute allocator after the input arrives. A request for alternatives opens more search branches; a stop condition closes them. Irrelevant prose barely moved spending, while misleading architectural hints raised it 2.61×. Semantic permission, not character count, determined the bill.

Why it matters: The harness can outweigh the user instruction. In the study, pi carried a 1,147–1,642-token fixed prefix, versus 15,983–20,330 tokens for Claude Code; user prompts were less than 5% and often less than 1% of total input. For the same model, task and prompt, Claude Code’s extra prefix and tool turns produced a 5–30× spread in cost per success. A native Claude Sonnet 5 replication preserved the central result: the multiple-approaches prompt used 2.52× the reasoning tokens inside Claude Code. Prompt caching cut the estimated bill by about 61%, but did not remove the work. A discount can hide inefficiency without changing behavior.

The authors also published runners, hidden evaluators, raw ledgers and wire captures, making the result unusually inspectable. The next evidence should be harder: large repositories, lower baseline success rates and independently implemented harnesses.

Room for disagreement: These tasks touched at most four files and were easy enough for the lean harness to succeed 92–97% of the time. More search may pay on genuinely ambiguous work. This is not a verdict that one agent is universally cheaper; it is evidence that prompt semantics and harness policy must be measured separately before model prices mean much.

Gemini Gives Keys an Identity

The first story makes text govern compute. Google’s Gemini migration makes another text string govern who is acting.

Gemini’s API-key documentation now distinguishes Standard keys from authorization keys. A Standard key associates traffic with a Google Cloud project for billing and quota, but does not identify the caller. An authorization key binds directly to a service account, so requests execute as that workload identity. New AI Studio keys already use the authorization form; unrestricted Standard keys are rejected, and Google says every remaining Standard key will stop working in September 2026.

The HTTP surface barely changes. Both key types still travel in the familiar x-goog-api-key header, while Google restricts authorization keys to the Gemini API by default and can enforce leaked-key blocking faster. Underneath that compatibility, the control model changes from possession of a project secret to possession of a credential attached to an identity. Permissions can now follow a service account rather than a shared project boundary.

Why it matters: This is an authorization migration disguised as key rotation. Project-scoped secrets make ownership inferential: logs show which project paid, not necessarily which service called. Identity-bound credentials create a durable principal that IAM policy can constrain, revoke and manage. They also raise the deployment bar. Creating the keys requires service-account and API-key-binding permissions, so key issuance becomes an IAM workflow rather than a developer-console convenience.

The awkward part is observability. Google explicitly says authorization-key requests are not recorded in Google Cloud’s service-account usage metrics. The credential therefore becomes more precise while one familiar attribution surface goes dark. Identity enforcement and identity telemetry are separate products; the migration improves the former without completing the latter.

Room for disagreement: Existing restricted Standard keys already limit API scope, and clients need not adopt a new authentication protocol. For a small service with disciplined secret management, the immediate security gain may be modest. The hard evidence will arrive in September: whether rejection happens on schedule, and whether Google adds a first-party per-service-account usage view before operators lose the old keys.

The Contrarian Take

Everyone says: Better prompts are a free optimization, and identity-bound API keys automatically make a service easier to govern.

Here’s why that’s wrong (or at least incomplete): Prompt wording can reduce wasted reasoning, but the fixed harness prefix and number of tool turns can still dominate total cost. A carefully bounded request does not neutralize an expensive runtime. Gemini’s new key can name the caller more precisely, yet Google does not expose those calls in service-account usage metrics. In both cases, the control becomes sharper before the measurement system catches up. Better policy without matching telemetry can improve safety while leaving cost and attribution surprisingly opaque.

Under the Radar

• Agent adoption is mostly invisible — A census of more than 180 million public repositories found that bot-account detection caught only 28,154 of 850,157 Claude Code-attributed commits, a roughly 30× recall gap. Pull-request analysis also missed 79% of commit-detected Claude Code adopters and essentially all Codex adopters. Public PRs overrepresent cloud agents doing feature work; commit and configuration signals reveal quieter maintenance and editor use.
• Agents still prefer doing something — FixedBench gives coding agents 200 repository issues whose correct fix is already present. Leading models still made unnecessary changes on 35–65% of tasks. Explicitly allowing a no-change answer improved abstention, but demanding reproduction before editing created a different failure: agents sometimes stopped after a failed reproduction even when work remained. “Do nothing” is a capability prompts can elicit, not yet a stable default.

Quick Takes

One Agent Score Now Has Three Denominators

Artificial Analysis’s Coding Agent Index v1.3 combines 113 DeepSWE tasks, 84 Terminal-Bench v2 tasks and 124 SWE-Atlas-QnA tasks. Each component averages three pass@1 attempts, while the page reports active time, tokens, cache rate and API cost. The composite is useful for breadth; the model held constant across Claude Code, Cursor CLI and OpenCode is more useful for separating model ability from harness behavior. (Source)

AWS Makes Cloud State Reproducible

The open-source aws-bench provisions disposable AWS environments, gives an agent scoped credentials, then grades diagnosis tasks against references and mutation tasks against live infrastructure. Its basic set has 78 tasks across four scenarios; the advanced set adds 47. The important omission is honest: there is no public leaderboard or reference-results paper yet. AWS has shipped evaluation substrate, not a winner. (Source)

Context Tools Lose Their Headline Savings

A fresh 15-task Django audit found none of five repository-context tools near advertised 60–90% full-session token savings. The best reduction was 15.9% in Claude Code and 34.9% in Codex, with quality differences smaller than evaluator noise. Tool invocation itself varied by harness, and cache warming once made an arm look 43% cheaper. The study is vendor-authored, but sealed tasks and a raw methodology make its caveats testable. (Source)

The Thread

Prompts, credentials and benchmark fixtures were once treated as passive configuration around the intelligent system. They now carry operational authority. A prompt decides how widely a model searches; a key decides which principal exercises an API permission; a resettable AWS account decides whether an infrastructure action can be graded against reality. That makes model output only the visible end of a longer policy chain. The consequential engineering question is increasingly not what the model knows, but which upstream string or state grants it permission to spend, act or stop.

Predictions

New predictions:

• I predict: By September 30, 2026, Google will replace its month-only Gemini Standard-key notice with either a specific shutdown date or an explicit migration-complete notice. This is wrong if the documentation still says only “Starting September 2026” and still describes restricted Standard keys as temporarily supported. (Confidence: medium; Check by: 2026-09-30)

Issue date: 2026-08-08 · Generated: 2026-08-08 03:55 AM EDT