DATA 2027 · Week 08 · Part II — New Access Methods & Engines

Transactions & Branching for Agent Swarms

When ten thousand agents want to try something before they mean it, the transaction abstraction stretches in two directions at once.

Lecture 1 — Distributed Transactions: Calvin vs Spanner · Lecture 2 — Copy-on-Write Branching as Product

Lecture 1 · Tuesday

Distributed Transactions: Calvin vs Spanner

Two poles, both published in 2012, still defining the trade space.

L1 · Why agents stress transactions

Agents are greedy transactional clients

A human clicks “checkout” once.
An agent fleet retries, forks, races itself.
Fifty workers write from the same stale plan.
“Read committed and hope” → anomaly within the hour.
Everything shipping today sits between Calvin and Spanner.

L1 · Sixty seconds of serializability

Two promises, one crucial gap

Serializable: effects equal some serial order.
Existential quantifier — the system picks any order.

Strict serializable (external consistency): adds real time.
T₁ commits before T₂ begins → T₁ ordered first.

L1 · Sixty seconds of serializability

Why the gap bites agents

Agent commits, tells its sibling over HTTP.
Sibling’s read may legally not see the write.
Equivalent serial order simply put the sibling first.
Out-of-band agent communication is the norm.
External consistency stopped being academic around 2024.

L1 · Calvin

Calvin: order first, execute later

Sequencer batches inputs into 10 ms epochs, replicated via Paxos.
Every partition receives the same global log.
Locks granted in log order — deadlock impossible.
Crash? Replay the log; determinism converges.
No two-phase commit — no commit protocol at all.

L1 · Calvin

Determinism scales

500,000

txns/sec on TPC-C-like workloads, 100 commodity machines — in 2012, while Spanner-style designs paid cross-datacenter round trips per commit.

L1 · Calvin

Calvin’s price is structural

Read/write sets must be known before sequencing.
No interactive BEGIN…COMMIT — submit a whole stored procedure.
Dependent txns: recon query first, then re-submit (OLLP).
Two trips each; hot keys make recon perpetually stale.
Irony: an agent’s final intent is usually declarable.

L1 · Spanner

Spanner: trust the clocks

Keeps interactive txns: 2PL + 2PC over Paxos groups.
TrueTime: TT.now() returns [earliest, latest].
GPS + atomic clocks keep ε at 1–7 ms, avg ~4 ms.
Commit wait: hold locks until TT.after(s) is true.
Then timestamp order is real-time order, globally.

L1 · Spanner

The cost of external consistency

≈ 2ε

expected commit-wait stall — about 5–10 ms per commit, paid so non-overlapping transactions need no communication at all.

L1 · Spanner

What Spanner pays — and buys

Pays: 2PC prepare + commit across Paxos groups.
Cross-region read-write txns: 10–100 ms.

Buys: lock-free snapshot reads at any past timestamp.
For read-heavy agent fleets, that beats write latency.

L1 · The poles, side by side

Calvin vs Spanner

Dimension	Calvin (SIGMOD ’12)	Spanner (OSDI ’12)
Ordering	Pre-agreed global log	TrueTime + 2PL
Distributed commit	None — determinism	2PC over Paxos
Interactive txns	No; recon + re-submit	Yes, full `BEGIN…COMMIT`
Consistency	Serializable	Externally consistent
Latency tax	~10 ms epochs + recon	2ε wait + 2PC rounds
Best agent fit	Declared high-throughput writes	Read-mostly global snapshots

L1 · MVCC, the bridge to Thursday

Your engine already keeps many realities

Postgres, InnoDB, Oracle, RocksDB: all multi-version.
Updates create versions; vacuum reaps the invisible ones.
Snapshot isolation falls out nearly free (minus write skew).
Engines keep coexisting realities — then GC after seconds.
Branching: keep one alive, name it, let it diverge.

Lecture 2 · Thursday

Copy-on-Write Branching as Product

The mechanism from one B-tree page up to a cloud product.

L2 · The agentic workload

Speculation is the defining workload

An agent doesn’t know its migration is right.
Honest move: run it somewhere consequence-free.
Pre-2022: shared nightly staging, trusted by no one.
CoW: writable production-size copy in tens of milliseconds.
Cost: only the pages you touch.

L2 · CoW from one page

LMDB: every commit is a tiny fork

A writer never modifies a page in place.
New leaf → new ancestors → flip one meta page.
Readers traverse an immutable tree: no locks, no torn reads.
Old and new roots coexist, sharing unmodified pages.
Same chord: ZFS, APFS, Aurora, RocksDB checkpoints.

L2 · CoW at cloud scale

Neon: time travel as a query parameter

Compute = unmodified Postgres; storage ingests the WAL.
Page versions indexed by (page id, LSN).
GetPage@LSN materializes any page at any retained point.
Once “page P at LSN L” works, a branch is bookkeeping.

L2 · Branch = a new timeline

Forking the WAL at an LSN

Fig. 8.1 — Branches as timelines forking from the parent’s WAL. Solid child: live delta. Dashed child: abandoned, GC’d; parent history stays pinned to the oldest live fork point.

L2 · Branch = a new timeline

O(metadata), not O(data)

~50 ms

to create a branch — one metadata record (child, parent, fork LSN), no pages copied, whether the parent holds 1 GB or 10 TB. The parent can’t even tell it has children.

L2 · The arithmetic

Ten thousand branches an hour

64 GB parent, 8 KB pages → 8.4 M pages.
Typical speculative run touches ~200 pages ≈ 1.6 MB.
5-minute average life → ~833 live branches.
Steady-state deltas ≈ 1.3 GB against the 64 GB base.
Storage grows only with divergence. Trivial — so far.

L2 · The arithmetic

What is not trivial

GC pressure: crashed agents never call delete_branch.
Without TTLs + a reaper: corpses at 16 GB/hour.
They overtake the 64 GB base in four hours.
Parent pinning: oldest live fork point sets retention cost.
Fan-out reads: 833 cold branches hammer shared hot pages.

L2 · Worked example

50 branches, 1% divergence each

Strategy	Formula	Total storage	Create time
50 full copies (pg_dump/restore)	50 × 64 GB	3,200 GB	~40 min each
50 CoW branches @ 1% divergence	64 + 50 × 0.64 GB	96 GB	~50 ms each
Break-even divergence	shared wins until ≈ 98% dirty	—	—

Dirty pages are private — divergent versions can’t be shared, so deltas add linearly.

L2 · Worked example

The headline ratio

33×

storage reduction vs full copies. Caveat: at 1% divergence per day, a branch parked three months pins a quarter-year of parent history — branches are a speculation primitive, not an archival one.

L2 · The agent transaction shape

Long think-time, optimistic wins

Shape: read 50 ms → think 3–30 s → write 20 ms.
Under 2PL: 30 s of held locks = self-inflicted DoS.
Agrawal–Carey–Livny 1987: OCC wins when conflicts rare, footprints long.
So: read a snapshot, think lock-free, validate-and-write.
Retries are cheap — re-prompting is one more API call.

L2 · Field note, 2025

Prompts are not idempotency keys

Refund agents shared a Postgres UPDATE credential.
One retried a half-committed batch — same refund twice.
Fix: revoke the credential; agents write to branches.
A 40-line validator owns the only path to main.
It rejects 3–4% of merges. Incidents since: zero.

L2 · The safe write path

Branch → validate → merge

Fig. 8.2 — Speculation on cheap CoW timelines; commitment through a narrow deterministic gate.

L2 · The safe write path

Calvin, rediscovered at the workflow layer

Exploration was interactive; the merge is declared.
Known write set → one serialization point, no 2PC, no human.
The branch absorbed the interactivity.
The merge inherits the auditability.
That division of labor is the agentic-era transaction model.

A branch is a transaction that lived long enough to get a name.

— Week 8 lecture notes

Checkpoint · Discussion

Before you leave

ε = 4 ms, 3 shards: who acknowledges first — Calvin or Spanner? Which assumption flips it?
30% of agents crash without cleanup: how does reaper TTL R shape steady-state storage?
Merge validates the read set against commits since L: serializable? Where does write skew sneak in?

Readings · Week 08

Read before Thursday

Calvin — Thomson et al., SIGMOD 2012. §3 sequencer; §3.2.1 OLLP.
Spanner — Corbett et al., OSDI 2012. §3 TrueTime; §4.1.2 commit wait.
Neon architecture posts, 2022–24. GetPage@LSN, layer files, branch creation.