The Denominator Is Software

If You Only Read One Thing

AI's most consequential number today was not a benchmark score but a denominator. Inspect AI fixed scorer state that could turn an unscored value into zero after a retry, while Inkling-Small claims flagship-level coding from 12 billion active parameters. One changes what a run costs; the other changes what the result means. Both show that efficiency is designed after the model speaks.

Inkling Shrinks the Active Bill

A 276-billion-parameter model can now plausibly be called “Small.” The name makes sense only if the unit is computation rather than storage.

July 20's full-Inkling release established the 975-billion-parameter flagship as customizable open inventory. Thinking Machines has now released Inkling-Small's Apache-2.0 weights and model card. The sibling has 276 billion total parameters but activates 12 billion for each token, versus 41 billion active in full Inkling. It ships as standard 16-bit weights and in NVFP4, a four-bit format that reduces the memory needed to serve it, with recipes for vLLM, SGLang and other common deployment stacks.

A mixture-of-experts model works like a warehouse that keeps many specialist teams on payroll but calls only a few for each order. Inkling-Small routes every token through six of 256 specialist blocks plus two shared ones. The active count approximates work per token; the total count determines how much model inventory must still be stored and made reachable. That split is the story's memory handle: cheap work, large warehouse.

The vendor's results are striking. Inkling-Small scores 40 on the Artificial Analysis Index against full Inkling's 41, 55.9% versus 54.3% on public SWE-bench Pro, and 64.7% versus 63.8% on Terminal-Bench 2.1. In other words, the smaller checkpoint claims to preserve or improve coding performance while activating 71% fewer parameters. This is a sibling checkpoint rather than merely a quantized copy, with a total weight count 72% below the flagship.

Why it matters: Open-weight labs have usually treated the largest checkpoint as the capability product and smaller siblings as compromises. Inkling-Small suggests the economically useful product may instead be the smallest checkpoint that preserves the post-training recipe. That moves value from raw scale toward routing, data and reinforcement-learning choices that decide which fraction of the warehouse is useful for a given token. It also changes model selection: full Inkling's stronger brand becomes hard to justify if the smaller sibling matches it on the agent workloads that drive adoption. The confirmation test is not another vendor table. It is a matched-harness run showing comparable completed-task quality at materially lower latency or accelerator cost.

Room for disagreement: Thinking Machines used a bash-only harness for its SWE-bench Verified figures and an internal coding harness for Terminal-Bench; comparison-model figures are partly self-reported. No independent Inkling-Small evaluation or measured serving curve surfaced in today's sweep. And 276 billion stored parameters still make “Small” a server-class deployment, not a laptop model. The score compression is credible enough to test, not yet strong enough to price.

Inspect Fixes the Answer Key

The same model and dataset could produce one score on a clean run and another after a retry. The model did not change. The result crossed a serialization boundary.

Inspect AI uses NaN, or “not a number,” as a sentinel for an unscored sample. A merged fix on August 4 explains that Pydantic's default JSON serialization converted some of those values to null. When a failed evaluation set resumed, a dictionary value could reload as None and be silently counted as 0.0, making the retried result differ from a first-try success. A NaN inside a list could instead fail validation and leave the log unreadable.

Another same-day scorer correction fixed an even simpler problem: exact() compared unordered sets of words. “world hello” therefore matched “hello world,” and duplicate words collapsed. The unreleased changelog warns that existing exact-match scores may fall after upgrading. Related commits repaired decimal answers wrapped in punctuation and preserved working-time fields in streamed logs.

The familiar assumption is that an evaluation log is a receipt: execution happens, then storage records it without changing its meaning. These bugs show the log is part of the measuring instrument. A retry reloads prior samples, a scorer normalizes their text, and a metric decides which values enter the denominator. If any stage changes semantics, the final percentage measures the framework version as well as the model.

Why it matters: Model selection increasingly turns on differences of a few benchmark points, while production evaluations routinely retry failed tasks and resume from stored logs. Inspect's fixes show that retry durability and scorer semantics are capability variables, not bookkeeping. A zero inserted for an unscored sample depresses a mean; an unordered “exact” match inflates success. Both effects can reverse a close comparison without either model generating a different answer. The confirming test is a before-and-after replay of real evaluation suites containing retries, unscored samples and short exact answers, with score deltas published by category rather than hidden inside an aggregate.

Room for disagreement: These are edge cases, and the commits do not quantify how often public or internal suites hit them. Most long-form model grading uses richer scorers than exact string matching, and many runs never serialize a NaN. The unresolved question is prevalence: a corpus-level audit of existing Inspect logs would show whether this is a ranking problem or a correctness fix that rarely changes the winner.

The Contrarian Take

Everyone says: Benchmark uncertainty comes from model randomness, contaminated tasks and different agent harnesses.

Here's why that's wrong (or at least incomplete): The answer key and its storage path can move the score before statistical uncertainty enters. Inspect's exact scorer accepted reordered or deduplicated words, while a retry could turn “unscored” into zero. Those are directional software defects, not random noise that washes out over more samples. A larger benchmark can make the wrong measurement look more authoritative unless the scorer, serializer and retry path are versioned with the result.

Under the Radar

• Ling's endpoint finally became inventory: inclusionAI has posted MIT-licensed Ling 3.0 Flash weights for a 124-billion-total, 5.1-billion-active model, plus an official FP8 checkpoint. The card claims 60%-80% lower first-token latency on long inputs and 56.6% on SWE-bench Pro, but deployment currently requires the lab's SGLang or vLLM forks. The release closes the artifact gap; independent performance and upstream runtime support remain open.
• Codex turns plugin policy into parser rules: A new runtime-boundary commit limits skill discovery to direct children, separates plugin MCP data, rejects configuration files that escape the plugin root, and caps model-visible instructions and schemas. July's plugin manifest distributed behavior; this enforcement narrows which packaged behavior can actually reach the model and tool registry.

Quick Takes

Hackers Leave Agent Traces

Cisco Talos recovered exposed Claude Code, Codex, Cursor and Gemini sessions from threat actors. One AI-assisted pipeline scanned 9,180 hosts and extracted credentials or source code from 54, while novices still struggled to turn generated tools into complete campaigns. The operational split is useful: models amplify execution skill, but exposed session history also creates a new forensic record of intent and iteration. (Source)

SGLang Exposes Effort

SGLang added DeepSeek V4's official reasoning-effort control, allowing the same open serving path to request different amounts of thinking rather than treating reasoning as a separate model identity. That makes effort part of the deployment contract, but it also means benchmark and production traces need to record the setting or nominally identical model names will hide different compute budgets. (Source)

Windows Output Stays Valid

llama.cpp now decodes Windows child-process output from the operating system's OEM code page before feeding it into the server's JSON tool layer. Accented text previously became invalid bytes or replacement characters, silently damaging command results. Local agent reliability depends on this mundane translation layer because corrupted tool output can change the next model decision without producing an obvious execution failure. (Source)

The Thread

Both deep stories are about denominators. Inkling-Small advertises capability per active parameter, but deployment still carries 276 billion stored weights. Inspect reports success per scored sample, but serialization could quietly admit an unscored result as zero. As AI systems optimize their headline ratios, the decisive engineering question becomes which costs, failures and samples the ratio is allowed to omit.

Predictions

New predictions:

• I predict: By September 5, at least one independent evaluator or serving-runtime team will publish a matched-harness Inkling-Small result covering coding quality or end-to-end latency. This prediction is wrong if every public result by then still derives only from Thinking Machines' launch table. (Confidence: medium; Check by: 2026-09-05)

Issue date: 2026-08-05 · Generated: 2026-08-05 03:57 AM EDT