The LLM stopped talking.

Gemma 3 4B as a language-conditioned decision function in native Rust: legal-label logits, softmax, typed answer. No generated text, JSON, or parser.

Gemma System One is a research prototype by Yossi Eliaz that uses Gemma 3 4B (Q4_K_M, Rust + Candle) as a typed decision function: it reads the logits of the legal answer labels instead of generating text. On an Apple M1 Pro the experimental packed-sheet path measured 47 ms per warm decision but changed one demo answer; the default API keeps each question in its own batch row over a shared state prefix. On JevBench's 231 public tasks it scored 58.9%.

By Yossi Eliaz · research prototype · September 2026 · Source

47 milliseconds / warm decision
experimental packed-sheet path (can change answers)
Apple M1 Pro · Q4_K_M

The measured result

47 msp50, warm batched sheet (experimental)
21.2/sfive decisions per batch (experimental)
4/4labelled spot-check (4 cases, too few to rank)
7/7option rotations held

Measured locally—not extrapolated—using a quantized 4B model on Apple Metal. The sheet prefill is about 0.6 seconds and is paid once per state. A fresh one-question request is not 47 ms.

Same transformer. Different exit.

Normal LLM

state + question
      ↓
generate()
      ↓
token → token → token
      ↓
text → parser → maybe valid

System-One-style

state + question sheet
      ↓
shared KV prefill
      ↓
legal-label logits
      ↓
softmax → typed decision

How 400 ms became 47 ms

Independent full prompt

Every decision re-reads the state and question.

~400 ms

Cache the state

Reuse the shared-state KV prefix; run only each question suffix.

173 ms

Cache the questions

Prefill state and all questions. Decisions become two-token pointers.

108 ms experimental

Batch single-position steps

One row per question. No attention mask construction on the hot path.

47 ms experimental

Speed did not waive the checks

What held

  • 4/4 fast-sheet labelled spot-check (4 cases, too few to rank)
  • 7/7 option-order rotations
  • 5/5 deterministic distributions
  • 4/5 decisions match the slow path

What did not

  • The sheet path reads retry_risk as 1.08 where the full prompt reads 3.00—low risk versus high risk.
  • A one-token pointer reached 27 ms—and collapsed accuracy to 1/4.
  • Long-suffix batching was slower than serial.
  • The full-prompt spot-check is 3/4, not 4/4.
  • Raw softmax remains uncalibrated.
  • The model still wrongly recommends an unchanged retry.
Not Jev. This reproduces Jev's typed interface and no-generation inference shape. It does not reproduce TypeSafe's model, RLCD training, calibration, or hosted infrastructure.

This is now an ecosystem, not one trick.

The easy part has become reproducible: many projects can turn a language model into a bounded decision function. The frontier is separating model quality, probability quality, and shared-state execution.

trained decision models
Jev · Laya · Kev · Mapika Decider
Train the representation/readout for decisions instead of leaving a chat model unchanged. This is where calibration, generalization, and explicit question isolation can become model properties.
inference adapters
OpenJev · SemIf · openjev-sglang
Reuse ordinary LMs, restrict the answer space, and read logits instead of prose. This establishes that “stop generating and choose” is useful engineering, but no longer sufficient novelty by itself.
probability quality
AnyJev · Kev · trained Decider/Jev-style systems
Debias option/label effects, fit calibration, or train calibrated heads. Raw restricted softmax is a model-relative score until evidence shows otherwise.
parallel architectures
djev / DiffusionGemma
Use diffusion-style answer slots rather than ordinary autoregressive continuation, making simultaneous bounded outputs a native architectural possibility.
shared-state systems
vLLM prefix caching · Hydragen · DeFT
The broader inference-systems literature already asks how to reuse expensive prefixes and branch efficiently. The decision-model question adds a semantic constraint: branches must stay independent.
this repo
Gemma System One · Rust · Candle · exact legal-label logits
Research target: maximize many-question throughput while measuring whether optimization changes the decision distribution. The production API shares only the state prefix; the faster packed-sheet path is retained as an explicit semantic-drift experiment.
Current thesis: the restricted-logit trick is commodity. The harder problem is one expensive state → many runtime-defined decisions, while keeping each branch semantically isolated and its probabilities empirically trustworthy.

Long-form map: decision-model landscape. Research plan: semantic isolation.

231 decisions. No victory lap.

58.9%public accuracy · 136/231
100%easy · 48/48
38.7%hard public · 43/111
0.402ECE · overconfident

Official JevBench v1.3 adapter, every redistributable public task, 231/231 valid responses. On those exact task IDs: Laya 58.4%, kev-4B 66.2%, Jev 1.13.0 86.6%. Prompted Gemma is a useful local baseline, not a trained decision model. This public-only run is not an official leaderboard rank.

Fresh-state API latency was p50 593 ms and p95 16.6 s; long-context tasks dominate the tail. That workload is deliberately separate from the 47 ms shared-state batch above.

Put it before the expensive model.

Local sidecar

agent event
      ↓
POST /v1/systemone
      ↓
route · policy · escalate

Remote fallback

uncertain or complex
      ↓
remote LLM
      ↓
reason · generate · act

The server speaks TypeSafe's request/response shape, so JevBench and an agent harness use the same API. Route only application-validated cases locally; raw softmax confidence is not a safe escalation policy.

Run the instrument

git clone https://github.com/zozo123/gemma-to-jev
cd gemma-to-jev
cargo run --release                 # conference demo

cargo run --release -- bench        # latency + proof suite
cargo run --release -- serve        # local TypeSafe-compatible API
./bench/run_jevbench.sh              # official 231-task public run

The first run downloads about 2.3 GB of weights. Metal is selected automatically; pass --cpu to force CPU.