The Lakehouse & Open Formats

• A directory, with subdirectories encoding partitions: dt=2026-06-09/region=eu/
• A metastore row points at the root
• The file set = whatever a LIST returns right now
• Hive isn’t a bad database — it is not a database

keys per S3 LIST call — so planning a query over a 1M-file table costs ~1,000 sequential API round-trips: tens of seconds before a single byte of data is read.

• Atomicity: an 800-file write appears 1 file at a time
• Readers see torn writes for minutes
• Consistency: S3 listings were eventually consistent for years
• Isolation: none — INSERT OVERWRITE deletes files under running queries

• Ryan Blue’s team at Netflix, 2018
• Stop defining a table by where its files live
• Define it by a persistent, immutable metadata tree
• The root is a single swappable pointer
• The catalog is the only mutable thing in the system

1. Write the new Parquet data files
2. Write a new manifest listing them
3. Write a manifest list: old + new manifests
4. Write v4.metadata.json with a new snapshot
5. Ask the catalog to compare-and-swap v3 → v4

on a few hundred bytes — a metastore lock, a DynamoDB conditional put, or a REST catalog’s conditional POST. Steps 1–4 are just unreferenced objects in a bucket.

• Two writers race → one CAS fails
• Loser re-reads metadata, validates for real conflicts
• Appends to disjoint files: re-attach manifests, retry
• Compaction rewrote the same files: abort
• Readers pin a snapshot — never take a lock

• Hive’s “atomic” write: write temp directory, then rename
• HDFS rename: O(1) metadata operation
• S3: copy-then-delete, one object at a time, non-atomic
• Millisecond commits on-prem took 10+ minutes in the cloud
• Design rule: never need more than “write a new object”

• Hive leaks dt into queries — miss it, scan everything, silently
• Iceberg: partition spec is a transform on a real column
• day(event_ts), bucket(16, user_id), truncate(4, zipcode)
• Predicate on event_ts → derived partition bounds → pruning
• Specs evolve: month(ts) for 2024, day(ts) from 2025

• Every column carries a permanent integer ID
• rename just changes the name attached to ID 7
• drop + add of the same name → a fresh ID
• Old Parquet is read by ID — deleted data can’t resurrect
• Kills the classic Hive bug: drop a column, everything shifts left

• FOR VERSION AS OF 8124 plans an old snapshot — free
• expire_snapshots: reference-counting GC over the bucket
• Delta: _delta_log/ of ordered JSON add/remove actions
• Commit = atomically create the next-numbered log file
• Redo log vs. functional tree — converging (Apache XTable)

• A lake of cheap Parquet for ML
• A proprietary warehouse for BI
• A fleet of ETL jobs shuttling data between them
• Every hop adds staleness, cost, and a drifting second copy
• CIDR 2021 names this two-tier world the problem

• Metadata fixes ACID — Tuesday’s transactional layer
• Caching + statistics fix latency — NVMe hides S3’s ~50–100 ms first byte
• The format gap is closable — Parquet ≈ warehouses’ internal layout, standardized
• Nothing claims novelty; the claim is architectural: open at every lock-in interface

• Spark SQL generated Java bytecode per query (whole-stage codegen)
• JIT bails out past 8 KB of bytecode — on the widest operators
• GC pauses scale with heap
• With NVMe caches, CPU — not I/O — became the limit

• Columnar batches through precompiled kernels — MonetDB/X100, Week 5
• Chose interpretation over codegen: observable, debuggable kernels
• Adaptive per batch: ASCII fast path, no-NULL paths, sparse/dense vectors
• Slots under Spark’s optimizer via JNI, reads the same Parquet
• An engine swap invisible above the operator boundary

average speedup over the prior Spark engine on customer workloads — ≈3× on average, beyond 10× on the most compute-heavy queries (SIGMOD 2022).

• Headline datapoint: 100 TB TPC-DS world record, 2021, over open formats
• Apply Week 2’s benchmark skepticism — the direction still stands
• Trino, DuckDB, ClickHouse, Snowflake, BigQuery: all vectorized over Parquet/Iceberg
• Polars, DataFusion, Velox compete inside the open-format world

• Anyone can read the bytes → control moves to the catalog
• Databricks: Unity Catalog, open-sourced under pressure
• Snowflake: Polaris — open Iceberg REST catalog protocol
• Databricks paid a reported ~$1–2B for Tabular that same week
• The REST protocol is the JDBC of this era

• REST catalog → metadata location + vended credential
• Parquet read straight into Arrow with DuckDB or Polars
• No server enforces semantics on that path
• Snapshot, column IDs, transforms — all live in the format
• The format is the database; the spec is its source code

baseline text-to-SQL accuracy on Spider 2.0-style enterprise benchmarks — largely because real catalogs are documentation deserts. Descriptions, join paths, and freshness in context move it dramatically.

• Which single Iceberg statistic skips files within the matching week — and what data property makes it useless for user_id? (Ex 7.1)
• In your toyberg, which racing commits may retry and which must abort? (Ex 7.2)
• How could an agent detect stale catalog metadata rather than trust it? (Ex 7.3)

1. Lakehouse — Armbrust, Ghodsi, Xin, Zaharia, CIDR 2021. Extract the three bets in §3; which carries the most risk?
2. Apache Iceberg Table Spec, v2 — ASF; pair with Ryan Blue’s Netflix 2018 talk. Find where per-column bounds live.
3. Photon — Behm, Palkar, Agarwal, et al., SIGMOD 2022. Focus on §3’s vectorize-and-interpret decision; skim the eval skeptically.