Local Models Need Jails

If You Only Read One Thing

The consequential moment arrives after generation: llama.cpp must decide where a shell command lands, while Pydantic AI must decide which compacted history a returning browser may discard. llama.cpp’s first Docker-backed tool runtime turns one choice into explicit execution custody; Pydantic turns the other into typed session state. Reliability now lives in who owns the handoff.

llama.cpp Builds the Cell

Running the weights locally solved the wrong half of the agent-security problem. The dangerous boundary begins after the model emits a tool call.

llama.cpp’s b10328 release adds an initial Docker-backed runtime for llama-server’s built-in tools. --tools-runtime docker:<image> starts a container, reuses it across tool invocations and removes it when the server exits. docker-container:<id> targets an existing container without stopping it. The merged implementation puts a small tools_io_isolate interface between llama-server and the execution environment; today that interface assumes a POSIX shell plus basic commands such as cat, find, mkdir and timeout.

That abstraction is more consequential than the Docker flag. llama-server is becoming responsible for execution custody, not merely token generation and function-call JSON. Follow-up releases expose the working-directory capability only when an enabled tool can actually use it, and report the isolated working directory through get_info. A frontend can therefore discover which filesystem boundary applies instead of inferring it from a prompt.

Why it matters: Local weights protect prompts from a hosted model provider; they do not stop exec_shell_command from reading host files or mutating a checkout. Moving tool execution into a named runtime creates a place to restrict mounts, users, networking and lifecycle. It also gives alternative backends a seam: the pull request discusses Podman and SSH as follow-ons, though neither has shipped. This is the beginning of a runtime contract for local agents, not merely an extra launch option. If that contract stabilizes, agent frontends can target execution environments without reimplementing every tool.

Room for disagreement: Docker is not a hard multi-tenant security boundary, especially when access to a privileged daemon is involved. The server documentation still calls the tools experimental and says not to enable them in untrusted environments. Isolation also covers the built-in tool process, not every secret mounted into the container or every network destination it can reach. The real gain today is inspectable placement of effects, not proof that arbitrary model output is safe.

Compaction Becomes Session State

A compacted conversation is no longer just shorter text. It contains a boundary that tells the next component which earlier state has already been summarized or retained by a provider.

August 4’s Pydantic treatment covered cost control inside the run loop. Pydantic AI 2.27 now moves to a different boundary by round-tripping CompactionPart through both Vercel AI and AG-UI. Before this change, dumping a Pydantic message history to either frontend discarded compaction parts; streaming did not emit them; loading the history could not reconstruct them. A browser-owned transcript could therefore look complete while silently removing the marker that gave the next model call its intended context shape.

The adapters now serialize Vercel data-compaction parts and AG-UI activity messages, then rebuild the Pydantic object when history returns. The trust rule is the interesting piece. Provider details can leave the server faithfully, but client-returned history is sanitized: provenance is stripped, standing system instructions can be restored, and a client cannot use a forged compaction marker to trim server-supplied trusted history. Pydantic also added support for round-tripping Anthropic’s optional encrypted_content, while noting that a live probe currently returns no value. That is forward-compatible plumbing, not evidence that Anthropic is already sending opaque compaction payloads.

Why it matters: Agent history is becoming a typed protocol shared by the provider, framework and UI. If any adapter flattens that protocol into visible text, the system can resend stale instructions, drop retained state or spend tokens summarizing material twice. Preserving the marker makes compaction portable across a common deployment pattern: server-side agent, browser-side history, server-side continuation. Sanitizing it on return also establishes custody: the UI may store state without becoming authoritative over which server context can disappear. That distinction will matter more as providers attach opaque continuation data to summaries.

Room for disagreement: Transport fidelity does not make a summary faithful. Pydantic documents a lossy edge case when a manually inserted compaction part splits a native tool call from its return; the pair is kept together on reload. More broadly, Governance Decay found that compaction can omit constraints that later govern tool use. The patch prevents an adapter from dropping the boundary; it cannot guarantee that the content behind the boundary preserved the right facts.

The Contrarian Take

Everyone says: Put a local model’s tools in Docker and the agent becomes private and safe.

Here’s why that’s wrong (or at least incomplete): Docker changes where commands run; it does not certify what the container can reach. A mounted SSH directory, host Docker socket or unrestricted network can recreate the same blast radius under a different process name. llama.cpp’s important move is architectural: it now has an explicit interface where stronger policy can attach. Calling that interface a sandbox before its mounts, identity and egress are constrained mistakes a placement mechanism for a security guarantee.

Under the Radar

• Remote content gets a ceiling — Pydantic AI disclosed that remote URLs could download unbounded content before processing. The advisory says 1.107.2 and 2.24.0 add a configurable 50 MiB default cap. This is mundane in exactly the right way: multimodal convenience had quietly turned a framework helper into an unbounded network-and-memory surface.
• Agent timeouts reach the loop — Vercel AI SDK 7.0.58 fixes ToolLoopAgent so timeouts configured in agent settings are actually respected. A timeout that exists only in configuration is worse than no timeout because operators believe the budget is enforced. The fix closes the gap between declarative policy and the loop that spends wall-clock time.

Quick Takes

Cline Makes Abort Durable

Cline SDK 0.0.72 keeps queued prompts and session context across user aborts, self-aborts and hub restarts; queue edits made during the abort window now persist. Failed queued turns become explicit run.failed events, while hung MCP initialization gets a 30-second budget. Abort is becoming a recoverable state transition rather than session destruction. (Source)

Pydantic Exposes Cancellation

Pydantic AI 2.26 adds first-party run cancellation plus public AgentRunEvents carrying run and cancel state. That matters less as a convenience method than as a lifecycle primitive: a UI or orchestrator can request cancellation, observe it and distinguish a deliberate stop from a failure without maintaining a parallel private state machine. (Source)

Filesystem Capability Stops Pretending

llama.cpp b10329 shows working-directory controls only when an enabled, served tool actually resolves paths against that directory. b10331 then reports the isolate’s working directory through get_info. The UI is beginning to reflect executable capability instead of displaying a generic control that may not affect the active tool set. (Source)

The Thread

Agent correctness now fails at custody handoffs. llama.cpp hands a model’s requested effect to a process environment; Pydantic hands compacted state to a frontend and accepts it back; Cline carries queued intent across an abort; Vercel carries a timeout from configuration into an execution loop. Each change replaces an assumption with a typed boundary, lifecycle state or enforced budget.

This is how agent infrastructure matures: not through another universal abstraction, but by naming who owns state at the moment it crosses components. The practical audit question is no longer only “What tools can the model call?” It is “Which component can alter the working directory, delete history, ignore cancellation or outlive the timeout?” Reliability improves when those answers become inspectable objects rather than prompt conventions.

Predictions

New predictions:

• I predict: By September 30, 2026, an official llama.cpp release will add either Podman or SSH as a --tools-runtime backend. This is wrong if Docker image and existing-container modes remain the only documented runtime options on that date. (Confidence: medium; Check by: 2026-09-30)

Issue date: 2026-08-09 · Generated: 2026-08-09 04:00 AM EDT