Cache Misses Leave Fingerprints

If You Only Read One Thing

Identical model calls can carry different bills, and identical weight files can feel like different products. Cache Misses Become Visible traces the prompt mutation that destroys reuse; TensorSharp Splits The Clock separates reading latency from writing speed. Microsoft's Cache Explorer documentation is the must-read because its first-divergence view makes the operating rule concrete: measure where repeated work begins, not only where the run ends.

Cache Misses Become Visible

Agent costs have been itemized before. The new development is that one of their largest discounts can now be debugged like code.

Visual Studio Code's July 8 Cache Explorer compares consecutive model requests inside Agent Debug Logs. Each turn shows its cache-hit percentage, reused input tokens, duration, model, and timestamp. A prompt signature then divides the request into instructions, tool definitions, and messages, marking the first place where two requests diverge.

The underlying mechanism is prefix caching. Think of an agent request as a long form whose first pages repeat on every turn: system rules, available tools, repository context, and conversation history. The provider can reuse the model state already computed for those identical pages. Change an early page, even by reordering a tool definition, and every page after it must be processed again. The memory handle is simple: the first break sets the bill.

Why it matters: Billing dashboards show cached tokens after the fact; Cache Explorer attributes the miss to a specific part of the harness. That turns prompt order, tool discovery, reasoning settings, and context compaction into observable runtime variables rather than folklore. Microsoft's earlier production experiments show why the distinction matters: deferring tool definitions cut median per-user token usage about 11% for GPT-5.5, while persistent WebSocket connections reduced median time to first token about 16% for GPT-5.4. Those are harness gains with the model held constant.

Independent evidence sets a more important boundary. A 500-session study across OpenAI, Anthropic, and Google found selective prompt caching reduced API cost 41-80% and first-token latency 13-31%. Yet caching the full changing context made GPT-4o latency 8.8% worse in one condition. The target is not the highest possible hit rate; it is a stable reusable prefix without paying to write dynamic tool results that will never be read again.

Room for disagreement: This is still a VS Code view of requests assembled by VS Code, not a provider-neutral trace of cache placement inside the serving stack. Optimizing the visible signature could also discourage useful tool or context changes. The signal becomes durable if task-level cost falls without lower completion quality, and if other agent runtimes expose the same first-divergence evidence rather than only aggregate cached-token counts.

TensorSharp Splits The Clock

Local model speed is usually flattened into one number. TensorSharp's more useful claim is that the clock has two hands, and a runtime can move them in opposite directions.

TensorSharp is a BSD-licensed native .NET engine for GGUF, the portable model-file format popularized by llama.cpp. It exposes OpenAI- and Ollama-compatible APIs and runs across Windows, macOS, and Linux with Nvidia, Apple-silicon, and vendor-neutral GPU paths. That compatibility matters: the same compressed weights can be held constant while the execution engine changes underneath them.

The two hands are prefill and decode. Prefill is the initial reading pass over instructions, code, and conversation history before the first answer token appears. Decode is the subsequent token-by-token writing phase. Interactive agents with large repositories can spend conspicuous time on the reading pass even when the output streams quickly. The memory handle is read speed versus write speed.

Why it matters: TensorSharp's revised head-to-head benchmark uses the same GGUF files, a 16 GB RTX 3080 Laptop GPU, one request at a time, greedy decoding, and the same Nvidia backend. On Gemma 4 26B-A4B, it reports 1.32 times faster prefill and 1.30 times faster first-token latency averaged across scenarios, while decode is slightly slower at 0.96 times llama.cpp. Structured JSON generation on the smaller E4B model is the outlier, running 7.7 times faster.

That pattern matters more than a blanket victory claim. GGUF has separated the model artifact from the runtime well enough that a 201-star .NET project can contest a hand-tuned C++ default on selected workloads without retraining weights. TensorSharp combines fused prompt processing, reusable GPU execution plans, paged memory allocation, and prefix sharing. In value-chain terms, model portability moves competition upward into schedulers, kernels, and workload-specific fast paths.

Room for disagreement: These are maintainer-run results from one laptop GPU, and the project previously revised its comparison after short prompts distorted the first benchmark. Its own vendor-neutral GPU results are near parity, long-prompt prefill still trails on two dense models, and the server ships without built-in authentication or encrypted transport. llama.cpp's broader hardware coverage, contributor base, and operational history remain advantages. TensorSharp has established a reproducible challenge, not a new default.

The Contrarian Take

Everyone says: Local inference gets better when a runtime produces more tokens per second, while hosted agents get cheaper when providers lower token prices.

Here's why that's wrong (or at least incomplete): Both views compress a multi-stage system into one headline number. TensorSharp can improve the time before the first token while slightly losing the output race; VS Code can lower cost and latency without changing either model or list price by preserving the right prefix. The scarce asset is increasingly the harness's ability to avoid repeated work. Model intelligence still sets the capability ceiling, but runtime structure decides how much of that capability arrives inside a usable latency and cost envelope.

Under the Radar

Agent frameworks are failing silently, not just crashing — LogicHunter generated valid but extreme inputs for LangChain, LlamaIndex, and CrewAI, then used documentation, source inspection, and runtime probes to judge failures. It found 40 previously unknown bugs; maintainers confirmed 30 and fixed 26. That is stronger production evidence than another framework feature list.
Adreno prefill is moving upstream — llama.cpp's July 12 b9968 release adds eight-bit integer math for dense and sparse-model prompt processing through OpenCL, a standard interface for GPU computation, on Qualcomm Adreno chips. No public speed delta accompanied the patch, so this is a portability signal rather than a performance claim: phone and Windows-on-Arm inference is acquiring architecture-specific fast paths in the main runtime.

Quick Takes

Codex separated reading from writing at the app boundary. Codex CLI 0.144 added a writes approval mode: declared read-only app actions can proceed, while writes still prompt, and Model Context Protocol tools can request authentication without an experimental flag. The useful shift is from approving an integration to approving its state-changing verbs. (Source)
Gemini's API authorization boundary needs a definitive map. A Google Cloud practitioner reports that a valid credential, enabled API, and billable project can be sufficient even without an expected per-call identity-and-access permission. This is anecdotal, not an established vulnerability, but it identifies the exact evidence Google needs to publish: which principal, project, key restriction, and role authorize each request. (Source)
AWS treats agent caching as several databases, not one switch. Its new Agentic AI Lens separates prompt, response, semantic, memory, tool-result, and data caches, each with different freshness and invalidation rules. The structural point is that a high hit rate in the wrong layer can make an agent cheaper and staler at the same time. (Source)

The Thread

Today's stories are about making inference legible at the stage where time is actually lost. Cache Explorer locates the first prompt change that forces repeated hosted compute; TensorSharp separates reading a long prompt from writing the answer; LogicHunter distinguishes ordinary exceptions from framework defects. The common shift is from aggregate outcomes toward stage-level evidence. Once the stages are visible, defaults become contestable: prompt order, runtime choice, approval scope, and cache freshness can be judged independently of the model name.

Predictions

New predictions:

I predict: By September 30, 2026, at least one coding-agent runtime besides VS Code will expose a per-component prompt-cache view that identifies the first cache-breaking instruction, tool definition, or message. (Confidence: medium; Check by: 2026-09-30)
I predict: By August 31, 2026, an independent user will reproduce at least a 20% TensorSharp advantage over llama.cpp on one prefill, structured-output, or tool-call workload using a different GPU from the project's RTX 3080 Laptop benchmark. (Confidence: medium-low; Check by: 2026-08-31)

Issue date: 2026-07-12 · Generated: 03:47 EDT