Gemini Scores Without Generating Text
7 stories · ~7 min read

Listen
If You Only Read One Thing
A thousand questions don't require a thousand model conversations. Gemini Decision 1 scores supplied choices without generating prose; AWS Emulator gives coding agents local cloud behavior to test against. Both make repeated checks easier to afford. Their limits matter just as much: a confident classification can be wrong, and a successful local test can miss a production permission failure.
Gemini Scores Many Questions Against One Shared Input
Gemini Decision 1 makes a familiar agent expense easier to separate: deciding which action fits doesn't always require generating an explanation. Google's November 11 release supplies probabilities for predefined answers at $0.10 per million input tokens, with no output-token charge.
The model accepts text, images, audio, video and PDFs. A request can ask up to 1,000 independent questions about one shared input. Yes/no checks, multiple-choice questions and ordered rubric scores can coexist in that request. A support system could inspect the same damaged-product photograph for visible damage, a likely department and escalation criteria without running three separate conversations.
The billing detail matters more than the headline question count. Google charges for the shared input once per request; each question adds its own instructions and criteria. Sending one expensive document through several checks can therefore avoid repeatedly paying to read it. That extends October 15's decision-model comparison from cheap dispatch toward coordinated checks over the same multimodal evidence.
This isn't sequential reasoning disguised as batching. If a second question needs the first answer, Google's API guide says to make another request, or ask both and discard the irrelevant result. The saving comes from sharing evidence across independent judgments. It doesn't remove a dependency between decisions.
Google reports 150–250 milliseconds end to end for contexts up to 16,000 tokens, with latency nearly flat between one and 100 questions. Those conditions are narrower than the advertised maximum request size. They don't promise the same wait for a full document and 1,000 checks.
Nor does cheap input pricing establish the cheapest classifier. Artificial Analysis estimates $345 per million tasks on its decision suite, against a $149 median. Those are benchmark tasks, not customer tickets. As in November 9's decision-model comparison, the complete task bill can reverse the impression left by a token price.
The application gain is a separate, inexpensive path for bounded checks, with uncertain cases sent elsewhere. Google's own documentation says confidence thresholds need labeled examples from the application's traffic. A reported probability doesn't supply that validation. A classifier saves money only if its bill stays lower after counting mistakes and escalations at the same tolerated error rate.
AWS Emulator Gives Agents Local Cloud Feedback
Google makes bounded judgments inexpensive; AWS Emulator brings the behavior being tested onto the developer's machine. Its November 11 launch lets coding agents exercise supported cloud APIs without creating live resources or incurring the corresponding service charges.
The released program exposes a local endpoint to existing AWS SDKs and the CLI. Linux packages are available; macOS and Windows use Docker. Rust tests can embed separate emulator instances, each with its own state. A coding agent can create a table, write an item and inspect the response inside a disposable test environment.
Local cloud substitutes already exist, and ordinary mocks can make unit tests fast. AWS's contribution is a substitute maintained against its own service behavior. That gives an agent a more useful reference than a mock it invented to satisfy the code it just wrote. A test can challenge an assumption instead of merely repeating it.
AWS publishes the comparison suite. Its launch explanation says tests are checked against real AWS, then require the emulator to match response status, body, errors and resulting state. Unsupported operations return HTTP 501. A missing implementation should therefore interrupt the loop rather than quietly supply a plausible success.
An agent can test stateful cloud calls locally before a smaller set of live integration checks. The emulator supplies stateful service interactions while the agent edits application logic. Live AWS then answers questions the substitute cannot settle. That division preserves fast feedback without treating every green local test as permission to deploy.
The documented limitations make that distinction essential. AWS Emulator doesn't evaluate identity and access policies the way AWS does. It also doesn't preserve normal separation between accounts and Regions. Code can succeed locally even though production correctly refuses the request. Load behavior and long-running operation aren't established either.
AWS hasn't demonstrated a controlled improvement in accepted coding work from this release. Even local execution has a cost: embedding the emulator in Rust tests requires compiling it, and AWS asks for at least 100 GB of free build space. Fidelity varies by service. An application test that passes locally but fails against AWS would expose the gap that matters: the substitute accepted behavior the real service rejected.
The Contrarian Take
Everyone says: Agents need faster tests so they can catch their own mistakes.
Here's why that's incomplete: Faster tests help only with mistakes the environment can reveal. AWS explicitly documents that its emulator can accept requests whose permissions real AWS would reject. A coding agent can consequently repair every visible failure and still produce an application that can't run with its production identity. More autonomous retries won't expose a rule the test environment doesn't implement. The failure is in the reference the agent trusts, so changing the coding model can leave it untouched. Fast local tests remain valuable when their scope is explicit and a separate check covers the missing behavior.
Under the Radar
-
Generated tools become searchable through their code. Simon Willison used Fable 5.2 and tree-sitter, a parser that identifies code structure, to classify his collection of 244 small tools by JavaScript APIs, libraries and contacted web hosts. The resulting tag browser makes existing artifacts easier to rediscover. The mechanism is syntax-derived indexing rather than asking a model to recall the collection; it doesn't establish measured time savings or reveal destinations assembled dynamically at runtime.
-
vLLM separates media processing from answer generation. Its November 11 engineering report describes sending processed image, audio and video representations between separately scaled serving stages. Passing the associated layout metadata avoids fetching and processing the original media again downstream. Core support is in v0.32.0; DeepSeek V4.1 Flash's metadata-only path is on main. The deployment implication is independent capacity for media-heavy demand, with extra transfer and routing complexity rather than a universal speed improvement.
Quick Takes
-
LLM 0.37.2 repairs two credential-routing leaks. A missing named provider key could fall back to the OpenAI environment key and send it to a custom endpoint. A separate Responses-to-Chat conversion could also send the OpenAI key there. The conversion advisory and the named-key advisory identify 0.37.2 as patched. This materially changes the security picture since November 3's LLM release coverage: an API-compatible destination doesn't establish the right credential recipient. (Source)
-
ERNIE 5.5 Flash Preview supplies another hosted coding candidate. Baidu reports 280 billion total parameters, 14 billion active and a million-token context. Its announcement makes the Qianfan API available while promising Apache 2.0 weights later. That distinction separates a model that can be evaluated through a service from one that can be privately deployed. The vendor's completed game and design demos don't establish superiority on ordinary coding tasks. (Source)
-
Pydantic AI can reject a plausible but truncated answer. Version 2.68.0 changes default
ModelFallbackhandling for non-streamed responses stopped by the output-token limit: it tries the next model instead of returning cut-off text as the output. The merged implementation preserves different behavior for streaming and the olderFallbackModel. A deliberate output cap can therefore trigger extra model calls after an upgrade; recognizing an incomplete answer and enforcing a spending limit require separate controls. (Source)
The Thread
AI suppliers are starting to supply more of the feedback that applications use to judge AI work. Google sells bounded judgments; AWS supplies a local version of the environment that tests generated code. That can shorten the path from an attempt to a useful correction. It also makes the feedback machinery another dependency worth examining on its own. The LLM credential fixes sharpen the consequence: even a helper that returns the right answer can send a secret to the wrong destination. A check can be accurate and still have too much access. Its accuracy and its permissions need separate tests. As cheap checks spread, dependency review needs to follow the checking path as carefully as the execution path. Otherwise a system can improve its visible answers while quietly expanding which software gets trusted with its inputs and credentials.
Predictions
- I predict: On December 12, AWS Emulator's documentation will still exclude using it to validate IAM authorization. AWS has explicitly separated behavioral emulation from security-policy testing; closing that gap requires more than adding service operations. (Confidence: medium; Check by: 2026-12-12)
Issue date: November 12, 2026. Generated at 03:48 AM ET.
Tomorrow morning in your inbox.
Subscribe for free. 10-minute read, every weekday.