State Needs Protocols

If You Only Read One Thing

The important shift in AI systems is that state has stopped being implementation detail. SGLang's DeepSeek-V4 stack and Gemini's Interactions migration both say the same thing: model quality now depends on how runtimes preserve caches, steps, tool calls, and resumable context. Start with the SGLang/LMSYS writeup because it shows the systems burden beneath the benchmark race.

SGLang Makes V4 Practical

DeepSeek-V4 is not just a model release. It is a stress test for whether open inference stacks can make a trillion-parameter, million-token model usable without turning every request into a memory-management incident.

The model card says DeepSeek-V4 ships in two open-weight variants: Flash at 284B total parameters with 13B active, and Pro at 1.6T total with 49B active. Both support a 1M-token context window, use mixed FP4/FP8 precision, and claim a hybrid attention design that reduces single-token inference FLOPs to 27% and KV cache to 10% of DeepSeek-V3.2 at full context. That is the prior baseline being displaced: long context used to mean "buy enough memory." V4 makes it "coordinate several kinds of memory correctly."

SGLang v0.5.12 is the practical part of the story. The release adds a full DeepSeek-V4 inference path across tensor, expert, context, and data-parallel attention; targets B300, B200, H200, H100, GB200, GB300, and AMD MI35X hardware; and adds prefill/decode disaggregation, HiSparse CPU offload for inactive KV cache, reasoning and tool-call parsers, DeepGEMM and FlashMLA kernels, MegaMoE paths, HiCache under a unified radix tree, and cookbook deployment recipes.

Why it matters: The key concept is cache coherence for attention. In plain terms, a long-context model keeps reusable memory from earlier tokens so it does not recompute the entire past each time it generates the next token. DeepSeek-V4 complicates that because the model does not have one simple memory pool. It mixes sliding-window attention, compressed sparse attention, heavily compressed attention, and speculative decoding state. SGLang's ShadowRadix/HiCache work is an attempt to make those pools share a common coordinate system so cached state can be reused, evicted, restored, and verified without corrupting the answer.

That moves the open-model race away from "weights are available" toward "the serving recipe is available." The model card can claim 1M context and strong coding scores, but a practitioner cannot deploy a million-token MoE because a README says so. They need launch commands, parser support, parallelism defaults, offload behavior, and failure notes. The SGLang cookbook is unusually explicit about this: low-latency, balanced, max-throughput, long-context, and prefill/decode-disaggregated recipes expose different tradeoffs, and the docs warn that concurrency settings can overflow DeepEP dispatch buffers if they are tuned independently.

The counterweight is that this is still first-party-adjacent systems evidence. The LMSYS/SGLang post says its head-to-head benchmark is a Day-0 snapshot, not a definitive ranking, and it could not get stable 200K-plus context runs on the comparison engine. Artificial Analysis also shows DeepSeek-V4 Pro Max as strong but not dominant: 52 on its Intelligence Index versus 60 for GPT-5.5 xhigh and 57 for Claude Opus 4.7 max, with lower price but slower output. The defensible conclusion is narrower and more useful: for open frontier models, capability now ships as a bundle of weights, kernels, cache policy, and serving documentation.

What to watch: The confirmation variable is whether vLLM, TensorRT-LLM, llama.cpp-adjacent stacks, and hosted inference providers publish their own DeepSeek-V4 long-context recipes with KV/offload metrics, not just compatibility badges.

Gemini Turns Chat Into Steps

Google's Gemini Interactions API change looks like a migration chore until you read it as an argument about what an agent API is supposed to remember.

The Gemini API changelog says the Interactions API schema is changing from outputs to steps; the new schema becomes the default on May 26, 2026, and the legacy schema is removed on June 8. The migration guide says the restructure supports future capabilities such as mid-flight steering and asynchronous tool calls. Instead of a response being a pile of output items, the API returns a timeline: user input, model output, Google Search calls, search results, citations, tool calls, and other typed steps. That is the prior-art shift. Chat APIs returned messages; agent APIs are becoming execution logs.

The reason this clears the practicality floor is that the ecosystem is already adapting. Vercel's AI SDK Google provider added Interactions API support in canary .54, support for cancelling long-running Interactions agents via AbortSignal and intermittent stream processing in .59, and a canary .65 patch specifically to handle the upstream breaking changes coming May 26. A May 16 release also updated @ai-sdk/google for the same schema change.

Why it matters: The memory handle here is the step log. A step log is not a transcript; it is the ordered execution record an agent runtime needs when it pauses, streams, calls tools, resumes, or lets the user steer it mid-flight. This is why outputs was too small a shape. A final answer cannot faithfully represent a long-running interaction where some work is done by server-side tools, some by the model, some by the user, and some by an SDK translating provider state into an app-level stream.

This changes the job of an AI SDK. A thin adapter used to normalize prompt, model, and output fields. A serious agent adapter now has to preserve provider-specific state without leaking provider-specific complexity into the application. Vercel's Google changelog is instructive because it is full of unglamorous state-preservation work: functionCall.id, matching functionResponse.id, thoughtSignature preservation, stateless image consistency, cancellation, and intermittent stream handling. Those are not feature-list ornaments. They are the glue that keeps a multi-step agent from losing the thread.

The strongest disagreement is that API migrations are often just churn. Developers will update a wrapper, tests will pass, and the world will move on. That is true for the average app. The structural signal is that Google is making the provider-native unit of work a step timeline, while framework authors are racing to preserve those semantics. Once providers expose steering, async tools, and resumable agent state this way, generic "messages in, text out" abstractions become lossy.

What to watch: Watch the May 26 Gemini cutover itself: whether provider adapters preserve tool-call IDs, thought signatures, stream cancellation, and citation steps cleanly, or whether apps need provider-specific branches around the migration.

The Contrarian Take

Everyone says: The open-model race is about closing the benchmark gap with GPT, Claude, and Gemini.

Here's why that's wrong, or at least incomplete: Benchmarks decide attention; runtimes decide deployment. DeepSeek-V4 looks competitive partly because its architecture cuts the long-context memory bill, but the usable product is SGLang's cache, kernel, parser, and recipe work around it. Gemini's Interactions migration points to the same conclusion from the API side: agents are not better chat completions. They are stateful execution systems, and the scarce layer is the runtime that keeps state correct.

Under the Radar

• Pydantic is scoring failed calls now - Pydantic AI v1.97.0 adds OnlineEvaluator.run_on_errors, marks streaming ModelResponse.state as incomplete while a response is still arriving, and introduces a new MCPToolset. The missed angle is that online evals are moving from "score the successful trace" toward "instrument the failure path," which is where production reliability actually lives.
• Claude Code is pricing context inside plugins - Claude Code 2.1.143 adds plugin dependency enforcement, projected context cost in the plugin marketplace, background-session model/effort preservation after idle wake, direct working-copy background isolation, and stop-hook loop caps. The pattern is not another agent feature. It is agent runtime governance: dependencies, cost, permissions, worktree policy, and recovery behavior becoming visible.

Quick Takes

• StreamIndex attacks the DeepSeek-V4 memory wall. The StreamIndex paper targets compressed sparse attention's indexer step, where public implementations materialize a score tensor that would hit 256GB at 65,536 tokens for V4-Flash-shaped inputs. Its Triton chunked top-k path runs the same indexer to 1,048,576 tokens with 6.21GB peak HBM on an H200, while explicitly not claiming end-to-end model speedup. (Source)
• Artificial Analysis makes the open-model tradeoff sharper. Its current leaderboard puts GPT-5.5 xhigh at 60, Claude Opus 4.7 max and Gemini 3.1 Pro Preview at 57, Kimi K2.6 and MiMo-V2.5-Pro at 54, and DeepSeek-V4 Pro Max at 52. The useful signal is price and speed: several open or Chinese frontier models sit near the top at far lower blended token cost, but not always with better latency. (Source)
• Vercel fixed an xAI reasoning-state edge. The May 16 @ai-sdk/xai release fixes reasoning-start deduplication on multi-summary-part responses and surfaces full xAI error details instead of generic HTTP status text. That sounds narrow, but it is the same runtime theme: reasoning models create provider-specific state that SDKs must preserve across continuation requests. (Source)

The Thread

Today's thread is that AI systems are becoming less like stateless calls and more like operating systems for model work. The model still matters, but the interesting engineering is now in the places that remember what happened: KV caches, compressed attention indexes, step timelines, tool-call IDs, reasoning signatures, background-session settings, failed-call evals, and plugin dependency graphs. State is where capability becomes reliable, and it is also where abstractions start to leak.

Prediction Ledger

Weekly Scorecard

• By 2026-08-31, at least two coding-agent platforms will expose task-level success, retry, elapsed-time, or model-route metrics alongside usage or billing reports, not just aggregate request counts. - Made 2026-05-15, medium confidence. Pending: Claude Code's projected context-cost UI strengthens the measurement premise, but the prediction is not due.
• By 2026-08-31, at least two agent or AI-app projects named in Microsoft's May 14 research, or their direct equivalents, will ship auth-on-by-default installs or public-exposure warnings in their default deployment docs. - Made 2026-05-16, medium confidence. Pending: no due-date resolution yet; the relevant evidence is still default-install behavior, not security commentary.

New prediction

• I predict: By 2026-08-31, at least two AI SDKs or agent frameworks will expose provider step/timeline events as first-class stream objects rather than collapsing them into assistant-message text. (Confidence: medium; Check by: 2026-08-31)

Generated: 2026-05-18 03:52 ET