Disaggregation & Elasticity

Aurora: the Log Is the Database

Every durable database already contains two complete descriptions of its state: the data pages, and the redo log that produced them. They are informationally redundant — given the log, you can rebuild every page. For forty years we shipped both across every replication boundary, because networks were slow and disks were local, so materializing pages eagerly everywhere seemed obviously right. Aurora’s insight (Verbitski et al., SIGMOD 2017) is brutal in its simplicity: in a cloud datacenter, where the network is fast and the storage fleet can run code, shipping pages is pure waste. Ship only the log. Let storage materialize pages itself, lazily, close to the disks. Everything else in the paper — the quorum, the segments, the recovery story — falls out of that one decision.

The write amplification problem

Start by counting what a conventional highly-available MySQL writes per transaction. InnoDB writes the redo log record, the data page itself (16 KB, even if you changed 120 bytes of it), and a second copy of the page to the double-write buffer to guard against torn writes. MySQL also writes the binlog for replication, plus assorted metadata. Now mirror that instance for HA: networked block storage typically replicates each of those writes again, and a hot standby repeats the entire set on its own mirrored volumes. The Aurora paper’s Figure 2 counts the same logical write traveling the network and hitting media many times over — and several of those writes are sequential and synchronous, so latency stacks.

Make it concrete. A transaction updates 10 rows spread over 8 pages. Mirrored MySQL moves roughly 8 × 16 KB of pages, doubled by the double-write buffer, plus log and binlog — then the whole bundle is replayed down a second instance’s stack: comfortably over half a megabyte of writes for a few hundred bytes of intent. Aurora sends 10 redo records of ~100 bytes each to six storage nodes: about 6 KB total. That is roughly a 100× reduction, and it shows up in the paper’s headline benchmark: over a 30-minute SysBench write-only run, Aurora completed 35× more transactions than mirrored MySQL while issuing 7.7× fewer I/Os per transaction. The transaction’s intent is small; only our habits made it big.

Only the log crosses the network

In Aurora the primary instance still runs InnoDB’s parser, optimizer, transactions, and buffer pool — but the only thing it ever writes is redo records. Each record is tagged with its log sequence number (LSN), grouped by the 10 GB storage segment it belongs to, and streamed to the six replicas of that segment. Commit is asynchronous and quorum-based: a worker thread marks a transaction durable once the volume durable LSN (the highest point at which the log is known complete on a write quorum) passes the transaction’s commit LSN. No page ever crosses the wire in the write path. The primary doesn’t even checkpoint in the traditional sense — checkpointing has been delegated to storage, which is continuously, incrementally applying the log it already holds.

Quorum math for three availability zones

Why six copies, and why 4/6 and 3/6? Quorum systems need two inequalities: the read and write quorums must intersect (V_r + V_w > V), and two writes must intersect (V_w > V/2). A 2/3 scheme satisfies both, and that’s what most systems used. Aurora rejects it because of correlated failure: in AWS, an entire availability zone can vanish — fire, flood, a bad network push — and failures of individual nodes (disk dies, top-of-rack flaps) continue at background rates regardless. The design target is AZ+1: lose a whole AZ plus one unrelated node, and you must not lose data. With 2/3 across three AZs, AZ+1 destroys two of three copies — quorum gone. So: six copies, two per AZ. Losing an AZ (−2) leaves 4/6, so writes continue. Losing AZ+1 (−3) leaves 3/6: writes stall, but the read quorum still stands, so no data is lost and the system can rebuild fresh copies and resume.

The subtle half of the argument is repair speed, because a quorum is only as good as the time you spend degraded. Aurora carves the volume into 10 GB protection groups precisely so that re-replicating a lost segment over a 10 Gbps link takes about ten seconds. For double failure to hurt you, the second failure must land inside that ten-second window in the same protection group. Big segments would stretch that window to minutes or hours and quietly invalidate the whole availability story. Durability is a race between failure rate and repair rate, and Aurora wins it by making the unit of repair small.

Storage nodes materialize the pages

Each storage node runs a small pipeline. Foreground: receive redo records, append to an in-memory queue, persist, ACK. That’s the entire latency-critical path — two steps. Everything else is background: sort records by LSN, gossip with peer replicas to fill holes in the log, coalesce redo onto data pages, back up log and pages to S3 continuously, garbage-collect superseded versions, and scrub blocks against checksums. When the database later reads a page, the storage node returns a materialized page at the requested LSN — applying any not-yet-coalesced redo on demand.

This inverts crash recovery. Traditional recovery replays the log from the last checkpoint before opening for business — minutes of downtime proportional to log backlog. Aurora’s storage is always applying the log, so on crash the database re-establishes the durable LSN by quorum reads and opens immediately; pages are brought to consistency lazily, on first touch. Recovery time stops being a function of how much work was in flight. The log is not a recovery aid for the database; the log is the database, and pages are merely a cache of its prefix sums.

Snowflake and the Elastic Warehouse

Aurora disaggregated storage but kept compute monogamous: one primary writes, replicas read, and the unit you scale is still “a database instance.” Snowflake (Dageville et al., SIGMOD 2016) went further and made compute promiscuous. Storage is just S3. Compute is a fleet of virtual warehouses — clusters of EC2 nodes you summon, resize, and dismiss like contractors — and any number of them can work over the same data simultaneously without coordinating, because the data is immutable and the metadata lives in a third, separately-scaled service tier. Aurora asked “what if storage understood the log?” Snowflake asked “what if compute owned nothing at all?”

Three layers, three failure and scaling domains

Snowflake splits into: cloud services (a multi-tenant, replicated brain holding the catalog, transaction manager, optimizer, and access control), virtual warehouses (stateless worker clusters with local SSD acting purely as cache), and storage (S3, which is slow at first byte but has effectively unbounded aggregate bandwidth and eleven nines of durability). The layers scale independently: a metadata-heavy workload stresses services; a giant join stresses a warehouse; nothing ever stresses S3. Crucially, a warehouse failure loses no state — kill every node mid-query and you’ve lost cache and in-flight work, never data. That property is what makes elasticity safe, not merely possible: you can only treat compute as disposable if it was never entrusted with anything.

Immutable micro-partitions and pruning

Tables are stored as micro-partitions: immutable files each holding roughly 50–500 MB of uncompressed data, internally columnar (a PAX-style hybrid layout), heavily compressed per column. Writes never modify a file; they write new files and update metadata to swap them in — an update to one row rewrites that row’s micro-partition. That sounds wasteful until you notice what immutability buys. First, pruning: cloud services keep min/max values (zone maps) per column per micro-partition, so WHERE event_date = '2026-06-09' over a 100,000-partition table touches only the partitions whose range can contain that date — 99%+ of I/O eliminated before a single byte leaves S3. There are no B-tree indexes in Snowflake; pruning over sorted-ish data is the index. Second, time travel: old micro-partitions stick around for a retention window, so querying the table “as of” an hour ago is just resolving an older file list. Third, zero-copy cloning: a clone of a 50 TB table is a metadata entry pointing at the same files. Versioning collapses into bookkeeping.

The economics of elasticity

The deep argument for disaggregation is economic, and it’s one line: 100 machines for 1 hour costs the same as 1 machine for 100 hours — but only one of them returns your answer today. In a coupled architecture you can’t exploit that identity, because adding machines means re-sharding data, which takes longer than the query. With state in S3 and compute stateless, a warehouse resize is just “boot more nodes and warm their caches.” Per-second billing plus auto-suspend turns capacity planning into a control loop: a nightly transform that needs 64 nodes for 9 minutes costs about the same as one node grinding for 9.6 hours, and an analyst’s warehouse that’s idle 95% of the day costs 5% of list price. The 2020 NSDI follow-up (Vuppalapati et al.), written from production telemetry, confirms the bet and reports the honest residue: real workloads are violently bursty, local SSD caches achieve high hit rates despite holding a sliver of total data (access skew is your friend), and the unsolved annoyance is intermediate data — shuffle and spill during a big join wants storage that is neither as expensive as RAM nor as far away as S3.

Neon: Postgres, taken apart along the same seam

Neon applies the Aurora cut to open-source Postgres, with a cleaner separation and one killer API. Postgres runs unmodified above the storage layer and is stateless: no data directory that matters. Its WAL streams to a trio of safekeepers, which run a Paxos-style consensus to decide the durable WAL frontier — commit means a quorum of safekeepers has the bytes. Downstream, pageservers ingest the WAL, reorganize it into immutable layer files (delta layers holding WAL slices, image layers holding materialized pages), cache them on local NVMe, and tier everything to S3. When Postgres misses in its buffer pool, it doesn’t read a disk — it issues an RPC:

GetPage@LSN(tenant, timeline, rel, blkno, lsn) → 8 KB page image

Read that signature slowly, because it’s the thesis of the week. The page is requested at an LSN — at a point in logical time. The pageserver reconstructs it by finding the newest image layer at or below that LSN and applying any delta records between. Once pages are addressable by (block, time), a branch is trivially cheap: it’s a new timeline defined as “parent timeline up to LSN L, plus my own WAL after.” Creating one copies nothing — reads of unmodified pages resolve through the parent’s layers, copy-on-write, exactly like a Git branch sharing history. A 2 TB database branches in roughly a second, and a hundred branches that diverge by a few megabytes each cost a few hundred megabytes total.

The agent angle: seismograph demand curves

Why does this week sit in a course about agents? Look at the load profile an agent fleet generates. Human-driven OLTP is a diurnal sine wave; you provision for the afternoon peak and eat the overnight waste. Agents produce a seismograph: a flat line of near-zero, then a workflow fires and 400 concurrent sessions appear within two seconds, run hot for ninety seconds, and vanish. The duty cycle can be under 1%. Provisioned-for-peak hardware at a 1% duty cycle is a 100× cost penalty; only scale-to-zero compute over disaggregated storage prices that curve honestly. Neon suspends an idle compute in minutes and cold-starts it in well under a second of compute boot plus cache warming — acceptable to an agent that just spent four seconds thinking about what query to write.

Second, agents are reckless in exactly the way branches make safe. A migration-writing agent should never rehearse ALTER TABLE against the production timeline; it should branch, apply, check, then promote or delete. At human scale that workflow meant a weekend restoring dumps onto a staging box. At agent scale — hundreds of experiments a day — it is feasible only because a branch costs one second and zero bytes up front. Branch-per-experiment is not a convenience feature; it is the isolation mechanism for a client that ships code faster than you can review it. Disaggregation is the precondition: copy-on-write across experiments only works when all experiments share one storage substrate addressed by (page, LSN).

Third, notice what happened to the humble buffer pool. In 1981 it was a few megabytes of RAM in front of a local disk. In these systems the cache hierarchy is the architecture: Postgres shared buffers (~100 ns) miss to pageserver or warehouse NVMe (~100 µs) miss to S3 (tens of milliseconds to first byte, but ~$0.023/GB·month and request-priced GETs). Every design decision this week — what to materialize where, what to keep hot, what an acceptable miss costs — is a buffer-pool policy question scaled up to datacenter size. The five-minute rule from Week 2 didn’t retire; it got promoted to chief architect.

Read Before Thursday

This Week’s Problems