Meta-harness on Islo

A harness is the prompt + tools + scaffolding around an LLM agent. A meta-harness is the loop that improves the harness automatically: a proposer agent reads the diagnostic logs of prior candidates, spots a failure mode, and writes a better harness. This prototype is inspired by Yoonho Lee’s work. It explores the eval–inspect–revise control flow with a simulator and a predefined hint table.

This project implements the local control flow: evaluate a harness, record artifacts, select a hint, and inspect the next iteration. The agent is a deterministic Python simulator. The proposer reads pass/fail results and selects a task-specific hint from a hardcoded table. The Islo integration is an illustrative, unvalidated stub with placeholder source and output handling. A real backend and LLM proposer remain future work.

Deterministic simulation result: five designed toy tasks (FizzBuzz, primes, reverse, sum-evens, palindrome) progress 0/5 → 2/5 → 3/5 → 4/5 → 5/5 in four proposer steps. The fixes are encoded in the simulator and proposer. This demonstrates orchestration and artifact inspection; it does not measure real-agent accuracy or generalization.

Proposed Islo integration

Three things meta-harness needs from its runtime:

1. Reproducible eval environments — every candidate harness runs against the same setup, otherwise the score is noise.
2. Massive parallelism — testing N candidates × K tasks adds up fast.
3. Persistent traces — the proposer needs to read stdout/stderr/agent-thoughts from runs that completed an hour ago.

Proposed CLI mapping (not exercised by the simulation):

islo snapshot save meta-base                  # prepare environment once
islo use mh-cand-7 --snapshot meta-base ...   # fork per candidate, in parallel
islo logs mh-cand-7 --type agent              # diagnostic traces, durable

These commands illustrate the integration target; the simulation does not exercise them. Source hydration, output capture, gateway policy, and CLI compatibility need validation. A Harbor workload adapter is not included.

The POC

tasks/        # 5 exact-match toy tasks, each prompt.md + grade.sh
harness/v0/   # baseline system prompt, deliberately mediocre
bin/
  meta-harness    # bash orchestrator (eval | propose | loop | viz)
  agent-sim.py    # deterministic agent stand-in (offline mode)
  proposer.py     # reads runs/, emits harness/vN+1
viz/index.html    # live dashboard
runs/             # populated each iteration

The agent is a Python simulator that’s intentionally buggy — until the system prompt contains the right hint keyword. The loop is therefore deterministic and offline. Real Claude/Islo execution still needs implementation and validation. The proposer reads runs/iter-N/, finds which tasks failed, looks up the missing hint for that task, and appends it to a new harness/v{N+1}/system.md. A real proposer would be:

islo use --snapshot meta-base --agent claude --task "
  Examine /workspace/runs/iter-${N}. Find a common failure mode in the
  grade.sh stderr. Write /workspace/harness/v${N+1}/system.md as a small
  edit on top of v${N}/system.md to fix it."

This is a proposed contract. The Islo path is an unvalidated stub; the current evaluator runs tasks sequentially.

In simulation iteration 1, the score moves from 0/5 → 2/5. The fizzbuzz hint contains inclusive, which also triggers the simulator’s sum-evens fix. This is designed keyword overlap, not evidence of transfer learning or an LLM reasoning over traces. The proposer uses pass/fail flags and predefined hints.

A single static HTML page (viz/index.html) polls runs/state.json every 2 seconds and shows a pass-rate timeline, a task × iteration heatmap, and a click-to-inspect trace viewer.

This POC is intentionally tiny so the loop is observable in one screen. Three obvious next steps:

• Real backend. Complete and validate the BACKEND=islo stub: source hydration, task/harness synchronization, output capture, and the CLI contract. Parallel task execution is not implemented. No real-backend latency is measured here.
• Real workload. Add and validate a Harbor adapter for the prompt.md + grade.sh task shape.
• Real proposer. Replace the predefined hint table with an LLM that reads runs/. Evaluate on tasks whose fixes are not encoded in the proposer before claiming generalization.

The working artifact is the local simulation and dashboard. Real-agent results require a separate evaluation.

Thanks to Yoonho Lee for the meta-harness framing. Islo is the proposed runtime.