Inference Learns To Predict

If You Only Read One Thing

Kimi Breaks The Cache Block and Colibrì Makes Storage Predictive look like opposite feats: one prepares a 2.8-trillion-parameter model for clusters, the other runs 744 billion parameters from a laptop SSD. The connection is prediction: inference is shifting from keeping everything fast to forecasting which state, weights, and work will matter next. That changes what “efficient” means.

Kimi Breaks The Cache Block

Kimi K3 has forced vLLM to separate a cache's physical shape from the unit at which it can be reused. That sounds like an implementation detail. It is actually a new serving abstraction for hybrid-attention models.

Monday's briefing covered K3's capability and price. The new development is what its architecture demands from an inference engine before Moonshot releases the full weights on July 27. K3 mixes conventional attention with Kimi Delta Attention, or KDA, which carries a compact recurrent state forward instead of retaining every earlier key and value. That state is much larger than one ordinary token's cache entry, so vLLM stores it in relatively large physical blocks.

The old coupling created a pathological miss: two requests could share nearly their entire prompt but fail to reuse it because the common prefix ended inside a large state block. vLLM's new design separates physical allocation, scheduler alignment, and the much finer interval used to match a shared prefix. A hit inside a block is copied into private state before generation continues, preserving the shared cache without letting one request mutate another's history.

The prior art is PagedAttention, which made the key-value cache behave more like virtual memory: logical sequences no longer needed contiguous physical storage. K3 exposes the next hidden assumption. A physical page need not also define the smallest reusable semantic boundary. vLLM says this machinery belongs in its core cache manager and should apply to other models mixing full attention with recurrent state.

Why it matters: Architecture is now reaching through the model file and changing scheduler semantics. Fewer active operations do not guarantee cheaper serving if shared prompts miss cache or recurrent state cannot move safely from prefill, the prompt-reading phase, to decode, which generates tokens. K3 makes prefix reuse a semantic contract separate from physical allocation, preserving fine-grained matches inside large state blocks. Published cache-hit rates and end-to-end throughput across disaggregated serving would show whether that abstraction actually lowers repeated-prompt work.

The evidence is not complete. Non-disaggregated serving works, and a 16-GPU configuration has passed correctness checks on the optimized MXFP4 path. Disaggregated serving and vendor validation remain in progress; end-to-end performance numbers are not published.

Room for disagreement: this is a preview of infrastructure for weights that are not public. The stronger claim is therefore not that K3 is efficient, but that hybrid models require cache contracts finer than their storage blocks.

Colibrì Makes Storage Predictive

Colibrì's headline is that GLM-5.2, a 744-billion-parameter mixture-of-experts model, can run on a machine with about 25 GB of RAM. Its real contribution is showing exactly what “run” means when storage becomes part of the inference loop.

The Apache-licensed engine keeps roughly 9.9 GB of dense weights resident and places about 370 GB of routed experts on disk. A mixture-of-experts model selects only a small subset of specialist weights at each layer; GLM-5.2 activates about 40 billion parameters per token, with roughly 11 GB of routed weights changing along that path. Colibrì treats VRAM, RAM, and NVMe as one hierarchy, then moves experts upward as routing proves they are useful.

Disk offload is not new. MoE-Infinity showed in 2024 that activation traces could guide an expert cache on personal machines. Colibrì turns that idea into a compact, observable runtime: one layer's router predicts the next layer's experts, the project reports 71.6% routing predictability, and a learned hot set stays resident. Two SSDs can serve different experts in parallel; one measured 9 GB/s plus 3 GB/s pairing improved decode by about 33% over the faster drive alone.

Why it matters: “Does the model fit?” is becoming the wrong binary question. The useful questions are how large the changing working set is, how predictable expert routing is, and what storage bandwidth buys at each latency tier. That makes model architecture and machine topology jointly responsible for inference cost. A future model card that reports only total and active parameters will omit the data movement that determines whether local serving is interactive.

The project is unusually candid about the boundary. Its 25 GB development machine manages only 0.05-0.1 tokens per second, or roughly 10-20 seconds per token, after a 372 GB download. A 128 GB CPU desktop reaches about 1.8 tokens per second; six RTX 5090s reach 5.8-6.8. These are project-reported measurements, with little independent replication so far.

Room for disagreement: a model producing one token every several seconds is not a practical assistant. Colibrì does not democratize interactive frontier inference at the hardware floor. It does make a frontier-scale checkpoint inspectable and measurable on hardware that previously could not load it, turning local inference into a continuum rather than a residency test.

The Contrarian Take

Everyone says: Sparse models are efficient because only a fraction of their parameters activate for each token.

Here's why that's wrong (or at least incomplete): Sparsity replaces arithmetic with a forecasting problem. Colibrì must guess which experts to fetch before the router needs them; vLLM must identify which portion of recurrent state is safely reusable before a request extends it. A low active-parameter count helps only when routing locality, cache boundaries, and data movement cooperate. The decisive metric is not how much of the model sleeps, but how cheaply the system can wake the right part.

Under the Radar

Macaron routes requests across post-training specialists. Macaron V1 adds four one-billion-parameter LoRA adapters, small post-training weight layers for chat, agents, coding, and generated interfaces, atop GLM-5.2. A smaller Qwen3.6-based sibling also ships. The checkpoints, hosted API, and routing harness are available. Its strong coding results remain vendor-reported, and the technical report is still pending, but specialization is moving from separate models into switchable adapter layers.
Pydantic gives tool failure a terminal state. Pydantic AI 2.16 adds ToolFailed, which can show a failure to the model without spending another retry, plus a caller-supplied run_id across normal runs, durable wrappers, and UI adapters. Those two changes separate “the tool failed and the model should react” from “the invocation was invalid and should repeat,” while preserving one identity through replay and display.

Quick Takes

OpenAI is packaging the operating process around agents. Presence is a managed limited-GA product that combines policies, approvals, escalation, simulations, and a Codex-powered change loop with forward-deployed integration. It is not a general developer surface; its significance is that OpenAI now treats continuous agent improvement as part of the product, not customer-side implementation. (Source)
Vercel made silent streams observable. AI SDK 7.0.35 returns response-piping promises so callers can catch read and write errors, and adds a per-step timeout for receiving first content. Neither improves model quality. Both convert a hanging or broken stream from ambiguous absence into an application state that can be measured and handled. (Source)
OfficeCLI fixed a small but revealing spreadsheet failure. Version 1.0.140 preserves in-cell images through its dump-and-rebuild path. For agent-edited documents, round-trip fidelity matters more than whether a parser can merely read the file: an edit that silently discards embedded artifacts is corruption with a successful exit code. (Source)

The Thread

The unit being optimized is the decision made before expensive work starts. Kimi's cache decides which prior state is reusable. Colibrì decides which experts deserve RAM and which can wait on disk. Macaron decides which adapter owns a request; Pydantic decides whether a failure deserves another attempt; Vercel decides how long silence can pass for a live stream. Token prediction remains the model's job. Systems advantage increasingly comes from predicting the work around the token: state reuse, weight placement, specialization, retries, and timeouts.

Predictions

New predictions:

I predict: By 2026-09-30, at least one of Moonshot AI, Alibaba, MiniMax, or Z.ai will publish a first-party serving profile for a new open MoE that reports either expert-cache hit rate or bytes of expert weights read per generated token. If none does, this is wrong; a third-party runtime measurement will not count. (Confidence: medium; Check by: 2026-09-30)

Issue date: July 23, 2026 · Generated: 3:47 AM ET