The LLM stopped talking.

Gemma 3 4B now acts as a language-conditioned decision function in native Rust. One shared prefill. One batched logit read. No generated answer, JSON, or parser.

47 milliseconds / warm decision
21.2 decisions / second
Apple M1 Pro · Q4_K_M

The measured result

47 msp50, warm batched sheet
21.2/sfive decisions per batch
4/4fast-path failure-class check
7/7option rotations held

Measured locally—not extrapolated—using a quantized 4B model on Apple Metal. The sheet prefill is about 0.6 seconds and is paid once per state. A fresh one-question request is not 47 ms.

Same transformer. Different exit.

Normal LLM

state + question
      ↓
generate()
      ↓
token → token → token
      ↓
text → parser → maybe valid

System-One-style

state + question sheet
      ↓
shared KV prefill
      ↓
legal-label logits
      ↓
softmax → typed decision

How 400 ms became 47 ms

Independent full prompt

Every decision re-reads the state and question.

~400 ms

Cache the state

Reuse the shared-state KV prefix; run only each question suffix.

173 ms

Cache the questions

Prefill state and all questions. Decisions become two-token pointers.

108 ms

Batch single-position steps

One row per question. No attention mask construction on the hot path.

47 ms

Speed did not waive the checks

What held

  • 4/4 fast-sheet labelled spot-check
  • 7/7 option-order rotations
  • 5/5 deterministic distributions
  • 4/5 decisions match the slow path

What did not

  • The sheet path reads retry_risk as 1.08 where the full prompt reads 3.00—low risk versus high risk.
  • A one-token pointer reached 27 ms—and collapsed accuracy to 1/4.
  • Long-suffix batching was slower than serial.
  • The full-prompt spot-check is 3/4, not 4/4.
  • Raw softmax remains uncalibrated.
  • The model still wrongly recommends an unchanged retry.
Not Jev. This reproduces Jev's typed interface and no-generation inference shape. It does not reproduce TypeSafe's model, RLCD training, calibration, or hosted infrastructure.

231 decisions. No victory lap.

58.9%public accuracy · 136/231
100%easy · 48/48
38.7%hard public · 43/111
0.402ECE · overconfident

Official JevBench v1.3 adapter, every redistributable public task, 231/231 valid responses. On those exact task IDs: Laya 58.4%, kev-4B 66.2%, Jev 1.13.0 86.6%. Prompted Gemma is a useful local baseline, not a trained decision model. This public-only run is not an official leaderboard rank.

Fresh-state API latency was p50 593 ms and p95 16.6 s; long-context tasks dominate the tail. That workload is deliberately separate from the 47 ms shared-state batch above.

Put it before the expensive model.

Local sidecar

agent event
      ↓
POST /v1/systemone
      ↓
route · policy · escalate

Remote fallback

uncertain or complex
      ↓
remote LLM
      ↓
reason · generate · act

The server speaks TypeSafe's request/response shape, so JevBench and an agent harness use the same API. Route only application-validated cases locally; raw softmax confidence is not a safe escalation policy.

Run the instrument

git clone https://github.com/zozo123/gemma-to-jev
cd gemma-to-jev
cargo run --release                 # conference demo

cargo run --release -- bench        # latency + proof suite
cargo run --release -- serve        # local TypeSafe-compatible API
./bench/run_jevbench.sh              # official 231-task public run

The first run downloads about 2.3 GB of weights. Metal is selected automatically; pass --cpu to force CPU.