01 / PUBLIC SIGNALS
Four systems, four pieces of the machine
The public SOL-ExecBench work already isolates the capabilities a serious kernel campaign needs. The numbers below come from different dates, evaluation stacks, and metric slices. Each result should be read within its original experiment.
Cursor
A planner distributed and rebalanced 235 problems across autonomous workers on 27 B200 GPUs. The run lasted three weeks and reported a 1.38× geomean speedup over its baseline.
Keep the worker contract small. Rebalance compute using live performance evidence.
doubleAI
WarpSpeed reported beating optimized baselines on 90% of 235 problems at 2.24× geomean speedup after one day of search. Its follow-up exposed kernels that passed fixed tests while failing on fresh seeds, unseen shapes, and realistic token distributions.
The verifier controls the meaning of every speedup. Hidden distributions belong in the evaluator.
Recursive
Many research threads ran jointly across all 235 kernels, retaining useful context and combining promising branches. Recursive reported a mean SOL score of 0.754 across the full benchmark.
Verified techniques should travel between related tasks while candidate code stays isolated.
Databricks
Databricks named KDA, Humanize, and Omnigent, with Claude writing code and Codex reviewing. The 28 July B200 / v1.1 snapshot shows Databricks first at 0.751491.
Long-horizon work benefits from explicit roles, state on disk, and a review gate that can reject completion.
02 / INVENTORY
The control plane is largely available
Islo exposes the CPU agent-runtime and orchestration primitives behind the loop. Its public job schema exposes vCPUs, memory, disk, snapshots, and lifecycle controls, with no GPU resource field as of 28 July 2026. A credible implementation therefore adds a hardened B200 evaluation service and a campaign controller. Building and qualifying those systems is substantial engineering work.
Persistent isolated computers
The campaign can keep one logical workspace per kernel. Active microVMs remain bounded by the Islo team quota; paused sandboxes retain their filesystem while releasing compute.
Agent stack
Islo’s default image includes Claude Code and Codex CLI. KDA documents Humanize installation, and Omnigent upstream explicitly lists Islo as a cloud-sandbox provider. This exact KDA + Humanize + Omnigent combination still needs an end-to-end pilot.
Durable parameterized jobs
Jobs provide versioned manifests, typed parameters, timeouts, fanout, concurrency, lifecycle policies, status, and logs.
Snapshots
A snapshot can pin the lab filesystem: toolchain, checked-out commits, prompts, and playbook. The external evaluator image, dataset revision, and GPU configuration must be versioned separately.
Scoped gateways
Host, path, method, response filter, rate, and credential rules give the sandbox access to model APIs and the evaluator without exposing secrets.
Audit and replay
Islo logs expose agent sessions and executed commands. The proposed campaign ledger adds the missing link from those records to candidate hashes and evaluator receipts.
Hardened B200 evaluation service
An external or BYOC service must provision B200 capacity, isolate untrusted submissions, reproduce the pinned benchmark harness, queue work, store artifacts, and issue content-addressed receipts. The Islo gateway supplies the controlled HTTP path to it.
Campaign controller
A controller maintains per-kernel state, ranks queued experiments by expected collection-score gain per B200 minute, enforces model and GPU budgets, and handles retry and stop policy.
03 / SYSTEM DESIGN
Islo runs the research loop; B200s return verdicts
This is a split-plane design. Islo hosts long-lived CPU workspaces, agents, jobs, policy, and logs. B200 capacity remains in a separately operated GPU service under the current public Islo schema. The evaluator is a trust boundary: agents can design any candidate within the task contract, while only the evaluator can issue promotion evidence.
- Fresh container
- Fixed harness
- Hidden tests
- Locked clocks
[job]
name = "sol-kernel"
version = "0.1.0"
description = "Persistent CPU research workspace for one SOL-ExecBench kernel"
[job.params.kernel_id]
type = "string"
required = true
[job.params.playbook_version]
type = "string"
default = "kda-b200-v1"
[run]
timeout = "12h"
teardown_on_complete = false
[run.sandbox]
mode = "ensure"
name = "sol-l1-{kernel_id}"
image = "ghcr.io/islo-labs/islo-runner:latest"
snapshot_name = "sol-kda-b200-v1"
gateway_profile = "sol-kda"
init = { type = "full" }
vcpus = 4
memory_mb = 8192
disk_gb = 40
[run.sandbox.lifecycle]
pause_after_idle = 1800
[[run.tasks]]
name = "optimize"
[[run.tasks.steps]]
name = "run-campaign"
exec = ["./campaign/run-kernel",
"--kernel", "{kernel_id}",
"--playbook", "{playbook_version}"]Schema-valid manifest skeleton, checked against the current Islo CLI. The snapshot, gateway profile, and campaign command are project dependencies that still have to be created.
04 / INNER LOOP
The life of one candidate
Each round leaves enough state for another engineer—or another model—to reconstruct the decision.
-
01
Classify the bottleneck
Memory traffic, tensor-core utilization, occupancy, reduction structure, launch latency, or fusion headroom.
analysis.md -
02
State one hypothesis
The expected mechanism, affected workload buckets, and the profiler signal that would support it.
hypothesis.json -
03
Implement a focused candidate
Source, compiler flags, parent candidate, task contract, and environment version become its identity.
candidate.sha -
04
Run local correctness first
Published workloads, generated seeds, numerical checks, fallback coverage, and output sanity remain available for fast iteration.
local-verdict.json -
05
Request promotion evidence
The trusted service reruns the candidate in a clean environment against withheld cases, then measures correct candidates with pinned warmups, trials, cache policy, clocks, and ABI.
evaluator-receipt.json -
06
Review the claim
Codex checks scope, evidence, specialization, physical plausibility, and overlap with failed ideas.
review.md -
07
Promote, branch, or stop
The scheduler considers score gain, B200 cost, remaining headroom, and the next defensible experiment.
decision.json
sha256:8f0c…91d2
- kernel
- 013_fused_residual_rms_norm_backward
- correctness
- PASS · 16/16 public · 48/48 hidden
- candidate SOL
- 0.7842
- previous best
- 0.7719
- replications
- 3 · max CV 0.41%
- promotion
- REVIEW REQUIRED
Illustrative schema. Values demonstrate the artifact contract; they are not campaign results.
05 / VERIFICATION
The evaluator defines what “fast” means
doubleAI’s public analysis is the clearest warning in the benchmark. A kernel can satisfy a fixed verifier and still corrupt the workload it was built to accelerate.
One published embedding-gradient kernel accumulated repeated updates directly in BF16. Uniformly sampled token IDs rarely collided, so the verifier accepted the result. Natural text follows a heavy-tailed distribution; frequent tokens collide hundreds of times and expose accumulated rounding bias. Training diverged under one optimizer while another optimizer masked the error.
The B200 evaluator should therefore maintain two contracts:
Fast local feedback
- Published shapes and tolerances
- Deterministic smoke tests
- Compiler and launch diagnostics
- Coarse benchmark signal
Promotion evidence
- Fresh RNG seeds and held-out shapes
- Realistic heavy-tailed distributions
- Accumulator-precision stress cases
- Fallback and adjacent-shape checks
- End-to-end workload spot checks
During search, agents can use the public verifier and generate their own randomized tests. Promotion inputs stay withheld. The service returns a bounded verdict rather than hidden tensors, rate-limits promotion attempts, and reserves a final quarantine suite for one-time confirmation. Each receipt includes the evaluator version and content hash so a later campaign can reconstruct the verdict.
06 / SCHEDULING
Spend B200 minutes where they move the collection
Agent computers can remain highly parallel. GPU evaluation is the scarce serializing resource, so the queue needs an explicit objective.
Coverage comes first. A failed kernel contributes zero. Replacing that zero with a baseline-level per-kernel score of 0.5 raises the 94-kernel L1 collection score by approximately 0.005319. Once every task has a defensible candidate, the scheduler shifts toward measured marginal gain.
A kernel near its analytical bound may justify one carefully targeted experiment. A kernel with large headroom and a clear NCU signal may justify ten. A kernel whose last five distinct hypotheses produced noise should pause and release its budget.
best SOL
remaining headroom
last N deltas
review confidence
expected eval cost
candidate attempts
model spend
B200 minutes
wall-clock deadline
human interventions
target reached
plateau
no valid hypothesis
unstable evaluator
budget exhausted
07 / OUTER LOOP
Let evidence teach the next lab
The inner loop improves a kernel. The outer loop improves the search policy used by future labs.
Completed tasks emit small, structured lessons: which tiling family worked, which NCU signal predicted the gain, where numerical stability failed, which shape bucket needed specialization, and which fallback preserved generality.
- Observation
- A second HBM pass dominated medium contiguous rows.
- Successful move
- Fuse variance reduction and affine application; vectorize 16-byte stores.
- Failure boundary
- Persistent blocks regressed rows below 2K elements.
- Evidence
- 3 replicated benchmarks + DRAM and memory-pipe NCU counters.
- Applies when
- BF16, contiguous rows, 2K–16K elements, exact accumulation contract.
A curator proposes a new playbook from these lessons. Promotion requires an A/B campaign from identical Islo snapshots on held-out kernels, with equal model and GPU budgets. A stronger playbook must improve correctness, collection SOL, or cost efficiency without increasing verifier failures.
The lesson record and version sequence are illustrative design artifacts, not outputs from an Islo campaign.
08 / FIRST CAMPAIGN
Eight kernels are enough to prove the system
The first run should validate autonomy, evidence quality, and transfer before spending a full 94-kernel budget.
8 L1 kernels
- 2 elementwise / vectorized
- 2 reduction / normalization
- 2 projection / GEMM
- 2 backward operations
3 controlled runs
- KDA + Claude
- KDA + Humanize + Codex review
- Same loop + verified cross-kernel memory
7 measured outputs
- Final SOL score
- Correct candidates / 100 attempts
- B200 minutes / promoted gain
- Human interventions / kernel
- Repeated failed directions
- Reviewer-caused corrections
- Transfer from playbook memory
What the campaign returns
Pinned tools, harnesses, prompts, task contract, and evaluator client.
One parameterized manifest that can launch, resume, and budget any kernel lab.
Content-addressed candidates, hidden verification, reproducible timing, and signed receipts.
Every hypothesis, failure, review, benchmark, promotion, and lineage edge.
Evidence for the value of independent review and shared memory under equal budgets.
A defensible basis for expanding from eight labs to the complete L1 collection.
THE BUILD
One persistent lab per kernel.
One trusted B200 verdict per candidate.
One playbook that earns every upgrade.
Islo supplies the persistent computers, jobs, snapshots, native Claude and Codex access, gateways, and logs needed for the agent control plane. Omnigent’s upstream Islo support makes the integration concrete, although this exact stack remains unqualified. The first implementation must build the hardened B200 evaluation service and campaign controller, then test the complete loop on eight kernels.
SOURCES
Primary material
- NVIDIA — Introducing SOL-ExecBench
- NVIDIA — L1 Single Operations leaderboard
- Yuchen Jin — Databricks L1 result and disclosed stack
- MIT HAN Lab — Kernel Design Agents
- PolyArch — Humanize / RLCR
- Omnigent — multi-harness agent framework
- Cursor — Speeding up GPU kernels by 38% with a multi-agent system
- doubleAI — WarpSpeed approaches Speed of Light on Blackwell
- Recursive — First Steps Toward Automated AI Research
- Islo — persistent hardware-isolated sandboxes
- Islo — Claude Code and Codex agent integration
- Islo — durable jobs and automations
- Islo — pause, resume, and lifecycle behavior
- Islo — pause, resume, snapshots, and logs
- Islo — scoped gateways and credential injection
This article proposes an implementation. It does not claim an Islo SOL-ExecBench run or benchmark result. Reported third-party numbers retain their original scope and date.