Deterministic demo / scripted roles / Bazel / Crabbox

Code and tests from one spec, with no shared scratchpad.

By Yossi Eliaz · July 2026 · More writing & experiments

A deterministic demo of the "hermetic agents" idea. A scripted code writer and a scripted test writer get the same spec but disjoint input manifests: the code writer cannot see the tests, and the test writer cannot see the implementation. A QA arbiter with a hidden oracle blames the test writer for the one seeded wrong test. No language model is called. The proof runs locally, in Bazel and in GitHub Actions, and the repo includes a Crabbox job for running it on an islo.dev sandbox.

Status: deterministic demo. The code writer, test writer and QA arbiter are scripted roles in scripts/hermetic_agents_demo.py, and the wrong test expectation is seeded on purpose.

Raw e2e JSON Transcript GitHub repo Crabbox

Run matrix

Path	Command	Evidence
Local	`./scripts/run_hermetic_agents_demo.sh`	JSON and Markdown under `docs/metrics/`
Bazel	`bazelisk test //:hermetic_agents_e2e_test`	Hermetic test target for CI and local validation
GitHub Actions	`.github/workflows/ci.yml`	Public green check on every push
Crabbox + Islo	`crabbox job run hermetic-agents`	Required proof JSON downloaded to `.crabbox/proofs/`

The proof

The generated test suite intentionally contains one wrong expectation: it expects Gizmo to remain capitalized, even though the spec requires case-folding. The implementation returns gizmo. QA compares both sides to the oracle and assigns the disagreement to test_writer.

{
  "verdict": "test_writer_fault",
  "case": "uppercase-name",
  "blame": "test_writer"
}

Scripted run: the disagreement is planted to show that QA can blame the test.

DAG contract

Role	Can read	Cannot read
Code writer	`spec/problem.md`, `guides/code-writer.md`	Generated tests and test cases
Test writer	`spec/problem.md`, `guides/test-writer.md`	Implementation output
QA arbiter	Spec, both outputs, hidden oracle	May send blame backward, not forbidden artifacts

Distill

Write one spec and role guides that replace broad RAG/context access.

Isolate

Materialize manifests so each writer gets only its allowed inputs.

Arbitrate

QA compares implementation, tests, and oracle, then assigns blame.

Crabbox path

The repo includes a .crabbox.yaml job for delegated Islo execution. Crabbox syncs the checkout, Islo owns the remote sandbox lifecycle, and the run is gated on the proof JSON existing after the command succeeds.

export CRABBOX_ISLO_API_KEY=ak_...
crabbox job run hermetic-agents

crabbox run --provider islo \
  --require-artifact docs/metrics/hermetic-agents-e2e.json \
  --download docs/metrics/hermetic-agents-e2e.json=.crabbox/proofs/hermetic-agents-e2e.json \
  --shell './scripts/run_hermetic_agents_demo.sh'

Why Crabbox fits

Crabbox does not become the agent framework here. The repo owns the specs, role guides, oracle, and QA policy. Crabbox owns the remote execution receipt: sync this checkout, run this command, require this proof file, download bounded evidence, and clean up the runner.

Concept	In this demo
Run	One execution of the hermetic coder/tester/QA harness.
Provider	Islo by default; any supported Linux runner can execute the same proof.
Artifacts	`hermetic-agents-e2e.json` is the required proof gate.
Station	Future fit for long-running model-backed agents, not required for this one-shot proof.

Run it

./scripts/run_hermetic_agents_demo.sh
python3 scripts/hermetic_agents_demo.py --self-test
bazelisk test //:hermetic_agents_e2e_test
crabbox job run hermetic-agents