The Denominator Chooses the Winner

If You Only Read One Thing

A 2.8x speedup and a 95.2% pass rate can both be accurate and still favor the wrong system. Local Speed Needs Matched Quality shows NInfer outrunning llama.cpp only after task accuracy holds; Graff’s split benchmark makes The Artifact Must Travel the second test. The denominator, not the model name, decides whether an apparent win survives a production workflow.

Local Speed Needs Matched Quality

NInfer changed one local-inference decision by measuring quality before celebrating speed. On a single RTX 5090, its specialized Qwen3.8-27B path delivered 1.4-2.8x faster decoding and 2.6-4.7x faster prompt processing than llama.cpp without a detected task-quality loss. That is a deployment result, not another tokens-per-second trophy.

The practitioner evaluation used more than 250 items from a production content-intelligence pipeline. Six tiers covered classification, long-context retrieval, multi-document questions, reasoning, structured extraction, and recorded tool episodes. Prompts, seeds, and gold labels were paired across engines; a four-model review found four mislabeled gold items before the suite was rerun.

Speculation supplied the speed. NInfer uses multi-token prediction, which drafts several future tokens and accepts them when the main model agrees. Its three-token drafter achieved 76% acceptance in the tested setup. Decode rose from 114 to 158 tokens per second at 1,000-token context and from 72 to 202 at 128,000. Time to first token fell from 85.9 seconds to 33.6 seconds on the longest prompt.

The quality result is deliberately less dramatic. Relevance scored 86% on both NInfer and llama.cpp. Transcript questions landed at 88% versus 82%, while reasoning was 98% versus 100%; neither gap was statistically meaningful. Every engine failed the tool replay. NInfer skipped structured extraction because it lacks JSON mode, which means it cannot replace llama.cpp or vLLM for every production route.

The strongest objection is comparability. NInfer used NVFP4 weights while llama.cpp used a Q5_K_M artifact, and the author excluded vLLM speed because its timing method differed. This is one operator, one GPU, selected Qwen checkpoints, and one workload. NInfer’s runnable code also supports only Linux on an RTX 5090, one resident model, and no multi-GPU serving.

The practice payoff is a matched-quality gate for local runtime changes. A speed result matters only after fixed production cases preserve accuracy, structured-output behavior, context capacity, and failure rate. The result weakens if a second 5090 reproduction finds a quality drop above five percentage points or multi-token acceptance below 60% at long context.

The Artifact Must Travel

Graff exposes a portability failure that ordinary agent pass rates hide. The same harness can leave a correct live environment yet fail to produce a patch that travels back to the repository. The model did not suddenly become worse. The promotion artifact changed.

September 3’s briefing separated tasks won by finding one valid path from tasks won only when every old path disappears. FrontierHarness adds a portability axis. Its 21 terminal tasks inspect the final live environment: the requested file exists, a service stays running, or recovered data is complete. Nine DeepSWE tasks instead apply the submitted patch to a prepared repository and run the project verifier. One rewards correct machine state; the other demands a movable change that survives outside the agent’s workspace.

Graff with Grok 4.6 passed 20 of 21 terminal tasks, while Graff with Kimi K3 passed 17. Both configurations passed only one of nine DeepSWE tasks. Some failures were ordinary test failures. Others came from patches that did not apply to the prepared base or from cleaned submissions that contained no real project change after scratch files disappeared.

That split changes evaluation practice. A green agent run can still leave no portable artifact, just as a clean patch can fail to preserve a live service the task required. Serious trials need both contracts when the production workflow crosses both boundaries: final-state checks inside the sandbox and a fresh application of the patch against the exact base commit outside it.

The headline comparison has sharp limits. Graff received extra evaluation instructions and ran in local Docker images; several comparison rows used different models, instructions, or runtimes. Each configuration ran once. The article therefore refuses to claim that Graff beats Codex, Claude Code, or Exo as a harness. Its cost figures are also useful only within their accounting method: the Graff/Grok terminal run spent $6.23 for 20 passes, while historical list prices and first-turn cache repricing differ across rows.

This extends hermetic testing with a promotion boundary. The evaluator freezes the task, base commit, environment preparation, verifier revision, and allowable output artifact, then counts missing evidence as failure. The decisive next result is a matched Kimi K3 comparison across Graff and Exo with identical instructions, images, and at least five repetitions. If the gap disappears, today’s apparent harness advantage was configuration, not architecture.

The Contrarian Take

Everyone says: The fastest runtime and the highest coding-agent score are the obvious deployment winners.

Here’s why that’s wrong (or at least incomplete): NInfer’s speed becomes useful only because paired production cases found no meaningful quality loss, and even then JSON output remains unsupported. Graff’s 95.2% terminal score collapses to 11.1% when success requires a patch that applies cleanly and passes repository tests. Performance is not a property of the engine alone. The workload, artifact, verifier, and failure denominator choose the winner.

Under the Radar

Eight months of harness code became a postmortem. The ELAI archive preserves 883 commits and a 659-entry feature catalog covering routing, memory, sandboxing, replay, and verification. Its author abandoned the system because it accumulated ideas rather than solving work he actually used it for. The reusable pieces are narrower: external evidence, fail-closed checks, journals, and bounded workers.
A production workflow paper admits its missing control. Infobip’s four-phase coding-agent method front-loads human research and planning, then increases delegation as artifacts harden. The authors explicitly report no effectiveness metric. That candor keeps the pattern useful as a reproducible hypothesis, while blocking the stronger claim that four phases beat an ad hoc workflow.

Quick Takes

vLLM can restart an engine without reloading its weights. The merged Fast Start path keeps post-quantized, tensor-parallel shards in a per-GPU daemon and maps them into a new process through CUDA IPC, removing another disk-to-GPU copy. The mechanism is clear; the pull request supplies no matched end-to-end restart result and rejects several parallel modes. (Source)
Paddock opened another specialized inference path. The Rust/C++ engine claims 1,062 tokens per second at 32 clients on Qwen3.8-27B, versus 958 for vLLM and 844 for SGLang. Its full board shows only 1.02-1.19x gains over vLLM, while one-model-per-GPU operation and no tensor parallelism keep the result below a general replacement claim. (Source)
A coding-agent leaderboard labels its own scores as estimates. GLSRM ranks 68 configurations to one decimal place, but says the figures are editorial estimates calibrated to a separate model table rather than published external runs. A precise-looking rank can therefore carry less deployment evidence than Graff’s small public task set. (Source)

The Thread

Both deep stories make the denominator part of the system. NInfer does not earn a deployment claim from speed alone; it earns a bounded claim from speed beside paired task quality. Graff does not expose one universal completion rate; it exposes the artifact contract that turned 20 of 21 into one of nine. Measurement is not a report laid on top of capability. It defines which capability exists.

The supporting items extend that mechanism beyond the leads. ELAI’s 883 commits could not prove daily utility, and Infobip’s four phases cannot prove improvement without a control. vLLM removes a weight copy without yet pricing the whole restart. Paddock publishes the losses beside its wins, while GLSRM’s polished decimal rests on estimates. The reliable metric stays attached to the decision it is supposed to change.

Predictions

New predictions:

I predict: vLLM will publish a documented before-and-after engine restart benchmark for Fast Start, or mark the path experimental in release documentation, by October 31. (Confidence: medium; Check by: 2026-10-31)

Issue date: 2026-09-06 · Generated: 3:35 a.m. ET