Every query engine on earth just agreed on something, which never happens. This is a first principles walk through Apache Iceberg: the one trick underneath it, what that trick buys you, where it will hurt you, and why the entire industry surrendered to a table format with no company behind it.
It is 03:10. The nightly load died two thirds of the way through, and nobody in the room can answer the only question that matters: what is in the table right now?
You cannot roll it back, because there is no transaction to roll back. You cannot re-run it, because you do not know which files landed. You cannot even read it safely, because the morning dashboards are already querying whatever half written mixture is sitting in that bucket. So you do what everyone did: you list a few hundred thousand objects, sort by timestamp, guess at a cutoff, delete what looks new, and re-run the job while quietly hoping nobody pulled a report in the last four hours.
That night, in some version of it, happened to everyone who ran a data lake at scale. It happened because for fifteen years the data industry ran on a polite fiction: that a folder full of files is a table.
It is not. A table is a contract. When you query it, you see one consistent version of the data. When two jobs write to it, one wins cleanly and nothing tears in half. When you add a column, yesterday's data does not become unreadable. Warehouses spent decades earning that contract, and then we walked away from it, because object storage was cheap and infinite and the warehouse was neither.
So we put Parquet files in S3 buckets, arranged them into directory hierarchies, called the arrangement Hive, and told ourselves the contract still held. It did not. Everyone who ran a data lake at scale has the same scars. A pipeline dies halfway through a write and leaves the table torn, half new files and half old ones, with nothing to say which is which. A backfill and a scheduled job write into the same partition and quietly shred each other. Someone renames a column and three dashboards silently start reading nulls. And under all of it, every single query begins with the slowest operation in the cloud: listing files, sometimes millions of them, just to find out what the table even contains.
None of these are bugs. They are the physics of pretending a filesystem is a database. Iceberg took the world by storm because it stopped pretending. It went back to first principles and asked the only question that matters: what would it take to give warehouse guarantees to files you can never edit in place?
The answer turned out to be one trick. Nearly everything Iceberg does, ACID transactions, time travel, schema evolution, partition evolution, all of it falls out of a single idea. Let me show you the idea first, and then we will watch the rest of the format fall out of it like corollaries from a theorem.
Object storage gives you three primitives and takes away everything else. You can write a whole object. You can read a whole object, or a byte range of one. You can list objects, slowly. That is it. There is no append. There is no rename that is actually atomic across systems. There is no way to edit byte 4,096 of an existing file. Every object you write is, for all practical purposes, immutable.
Databases were built on the opposite assumption. A B-tree updates pages in place. A write ahead log appends. Postgres would not survive ten minutes on primitives like these. So the naive conclusion, the one the industry lived with for a decade, is that transactional guarantees on object storage are simply not available, and you compensate with orchestration, conventions, and hope.
But look closer at what a transaction actually requires. Atomicity does not require mutating data in place. It requires exactly one thing: a single moment where the new version becomes visible, entirely or not at all. Everything before that moment can be as messy, as parallel, and as slow as it likes, provided nobody can observe it.
Immutability, the constraint we thought was the problem, turns out to be the gift. If files are never edited, then a file that exists is always internally consistent. Readers never see a half written page, because there is no such thing. The entire problem of transactions collapses into a much smaller one: not "how do I safely change data" but "how do I safely change which data counts."
That is a pointer problem. And pointer problems have a famous, boring, forty year old solution: compare and swap. Iceberg's core design move is to notice this, and to have the discipline to build nothing else at the bottom of the stack.
Here is the whole format in four sentences. A table is defined by a single metadata file, which describes everything: schema, partitioning, and the full list of data files, indirectly, through a tree. Writers never touch existing files; they write new data files and a new metadata tree beside the old one. To commit, a writer asks the catalog to swap the table's pointer from the old metadata file to the new one, atomically, only if nobody else swapped it first. Readers grab the pointer once and see a perfectly frozen version of the table for as long as their query runs.
Reading that is one thing. Watching it is another. Commit a few times below and keep your eye on two things: the pointer never touches history, and the reader that arrived at v411 stays at v411 no matter how much lands on top of it.
Three commits of history, all still on disk, all still queryable. The accent marker is the pointer: the only value in the entire system that ever changes.
That is it. That is the trick that ate the industry. One mutable pointer, guarded by compare and swap, in front of an immutable tree. Everything you have heard about Iceberg, every feature on every conference slide, is a consequence of this structure. The rest of this article is just pulling on the thread.
The metadata tree has four levels. Each level exists to answer one question fast, without listing a single directory. Read them top down; each one earns its place.
Abstractions are cheap; sequences are honest. Here is exactly what happens when a writer appends a batch to orders. Step through it, and note where the expensive work sits relative to the moment anything becomes visible.
The writer lands new Parquet files next to the existing ones. Nothing references them yet, so readers cannot see them. If the job dies right here, the table is untouched; the orphans get cleaned up later. This is why a crashed write can never tear an Iceberg table.
Notice what fell out of step 4 without anyone designing it: optimistic concurrency. Two writers race, one swap succeeds, the loser re-reads the new state, checks whether its files actually conflict, and usually just re-commits on top. Its Parquet does not need rewriting; only the cheap metadata does. Snapshot isolation for readers falls out too: a reader resolves the pointer once and holds a frozen tree, so a ten minute query is consistent even while ten commits land underneath it. Nobody bolted these features on. The structure implies them.
Once you accept "table = immutable tree + swappable pointer", features that took warehouse vendors decades become almost embarrassingly free.
Every commit produces a new root, but the old roots still exist, still valid, still pointing at files that are still there. Keeping history costs nothing extra; history is the natural state, and expiring old snapshots is the operation you have to do on purpose. Query any point in the retention window as easily as the present. Drag through a table's life:
Initial historical load from the warehouse export. Three years of orders, bulk written by Spark.
Iceberg tracks columns by ID, not by name or position. The name is a label on the ID; the Parquet files store the ID mapping. Rename a column and old files still resolve correctly, because the ID never changed. Add a column and old files simply return null for it. Drop and re-add a column with the same name and it gets a fresh ID, so it can never accidentally resurrect old data. This sounds like a small bookkeeping decision. It is the difference between "rename means migrating nineteen terabytes" and "rename means editing one JSON file". The Hive era treated schema as a property of files. Iceberg treats it as a property of the table, versioned like everything else.
In Hive style tables, partitioning is a leaky abstraction with teeth. The table has a physical event_date partition column, users must know it exists, and the classic failure is a query filtering on event_ts that silently scans the entire table because it did not mention the magic column. I have watched that one mistake produce five figure monthly bills.
Iceberg stores the partition transform in metadata: "partition by days(event_ts)". Users query the real column, event_ts, and the engine derives partition pruning automatically. The partitioning scheme becomes invisible, which is exactly what an implementation detail should be. And because it is metadata, you can change it. Start daily, grow, switch to hourly; old files keep the old spec, new files use the new one, and queries plan across both. Repartitioning without rewriting data was science fiction in 2019. It falls out of the pointer.
Being technically right is table stakes. Delta Lake and Hudi solved overlapping problems, both are serious engineering, and both are running in production at companies you respect. So why did the industry converge on Iceberg, to the point where its competitors now ship Iceberg compatibility as a feature?
Three reasons, and only one of them is technical.
First, it was born neutral. Iceberg came out of Netflix as an Apache project with a specification, not a product with a company attached. Delta was, for its formative years, structurally tied to Databricks and its runtime; Hudi carried Uber's very specific streaming upsert DNA. When Snowflake, AWS, Google, and Databricks are all deciding what to build native support for, the format none of their competitors owns is the only format they can all agree on. Neutrality is not a feature you can add later. It is a property of birth.
Second, the spec came first. Iceberg is defined as a document, with the Java and Python libraries as implementations of it, not the other way around. That is why a Rust engine, a C++ engine, and a Go tool can all implement Iceberg from the text and interoperate. The REST catalog protocol finished the job: catalog access became an HTTP API rather than a JVM dependency, and suddenly any process on earth could commit to a table safely.
Third, the design respects the separation everyone already wanted. Compute vendors compete on engines. Nobody wins by competing on file layout. Iceberg let every vendor keep their moat and share the substrate, which converted potential enemies into contributors. The tipping point stopped being a question of if around 2024, and the sequence since reads like a surrender ceremony:
Born from concrete pain: petabyte tables on S3 where directory listings and eventual consistency made Hive tables unsafe at scale.
The spec hardens. Spark, Trino, and Flink land first class support, proving the multi engine thesis early.
Catalog interop becomes an open HTTP contract. The last vendor specific bottleneck starts to dissolve.
Databricks pays over a billion dollars for Tabular, the company founded by Iceberg's creators, while owning the competing format. Snowflake open sources the Polaris catalog. AWS ships S3 Tables, Iceberg as a native storage primitive. When the format's biggest rival buys the format's founders, the war is over.
BigQuery, ClickHouse, DuckDB, Redpanda, Confluent, Dremio, StarRocks: reading, and increasingly writing, Iceberg is now simply what a data product does. The format stopped being a choice and became the terrain.
There is a fourth reason, quieter than the others: timing. Iceberg matured at the exact moment AI made every company suddenly care about feeding the same data to five different compute frameworks. When your training pipeline is Spark, your analysts live in a warehouse, and your agents query through DuckDB, "one open table, many engines" went from architectural taste to procurement requirement in about eighteen months.
It is worth being precise about the predecessor, because half of Iceberg's design reads as a point by point rebuttal of Hive tables. The comparison is not a strawman; it is the system most of us actually ran. Read it as six pairs: on the left the failure, on the right the same line rewritten by the pointer.
Formats do not matter in the abstract. Here are the patterns where Iceberg changes the shape of the system, drawn from real deployments, mine and others'.
The flagship pattern. Spark writes heavy ETL. Trino serves interactive analytics on the same tables, seconds later, no copies. Flink streams into the same tables. DuckDB reads them from a laptop for that one investigation that always comes on a Friday. One storage layer, one security perimeter, one bill, and the engines become interchangeable parts you can upgrade or evict independently. This is the stack I called the Modern Lakehouse in the previous field guide, and Iceberg is the load bearing wall in it: remove it and the whole "engines are replaceable" property collapses back into per engine data silos.
Debezium tails your Postgres WAL, changes land in Kafka or Redpanda, and a small Flink or Spark job merges them into an Iceberg table every few minutes. What you get is a queryable replica of your operational database that analysts can hammer with zero risk to production, with time travel as a built in audit log. The MERGE semantics that make this correct, upserts and deletes applied atomically, are exactly the thing raw Parquet directories could never give you. This pattern quietly killed a whole category of fragile "sync to warehouse" scripts.
Train a model against snapshot 8812 and record that ID next to the model artifact. Six months later, when someone asks why the model behaves as it does, you re-query the exact bytes it saw. Same trick for incident forensics: yesterday's revenue dashboard changed, so you diff today's snapshot against yesterday's and find the late arriving batch in minutes. Before Iceberg, "what did the data look like on Tuesday" was a question you answered with backups and prayer. Now it is a query clause.
GDPR and friends require deleting specific rows out of billions. On plain Parquet that means rewriting every file that might contain the user, tracked by hand. On Iceberg it is a DELETE statement: delete files mark the rows immediately, compaction physically rewrites the affected data files on schedule, snapshot expiry ages out the versions that still contained the rows. The lifecycle is inspectable at every step, which is the difference between compliance you can demonstrate and compliance you assert.
The quiet economic pattern of the last two years: keep the warehouse for the last mile of BI, move the ninety percent of raw and intermediate data into Iceberg on object storage, and point external tables at it. Storage drops to S3 prices, transformation compute becomes a market where Spark, Trino, and the warehouse itself bid for the job, and you are no longer paying warehouse rates to store data nobody queries. I have seen this halve platform bills without a single dashboard noticing.
Now the part most Iceberg articles skip, which is exactly why you should not trust most Iceberg articles. The pointer trick has real costs. You should walk in knowing them, because every one of these has paged someone at 3 AM.
Honest boundaries, because a format this hyped deserves them. If your workload is point lookups and single row updates, that is OLTP; use Postgres and be happy. If you need sub second analytics on hot data, Iceberg is the archive and system of record, but the serving layer wants ClickHouse or Redis in front of it. If your whole estate is a few hundred gigabytes and one team on one warehouse, the operational surface of a lakehouse buys you nothing; DuckDB or the warehouse alone will feel like a superpower. And if nobody on the team can own maintenance jobs, do not adopt a technology whose failure mode is slow, silent decay. Iceberg pays off in proportion to scale, engine diversity, and operational maturity. Below certain thresholds of all three, it is ceremony.
Suppose the case closed and you are doing this next quarter. The order matters more than the tooling, because two of these steps are expensive to reverse and the rest are not. This is the sequence I would run again, and the one I have watched teams skip to their cost.
The format is portable; the catalog is the commitment. It holds the only mutable state you have, it sits in the availability path of every read, and swapping it later means migrating every table's pointer. Decide on Glue, Polaris, Nessie, Unity or a plain REST catalog before you write a single table, and decide it on operations: who runs it, how it fails over, how access control maps to your existing model.
Pick the one table whose pain you can name in a sentence: the one that tears on failed writes, or costs the most in scans, or that three engines already fight over. One table proves the whole chain end to end, gives you a real benchmark, and if the answer turns out to be no, you have lost a week instead of a quarter.
Compaction, snapshot expiry and orphan cleanup are not follow up tickets. Write them as scheduled jobs the day the table lands, with alerting on file counts and manifest sizes, because the failure mode is invisible for weeks and then sudden. A table without maintenance is a table with a timer on it.
Every engine reads Iceberg. Writing is where the variance lives. Run your actual MERGE, your actual concurrent writers, your actual delete volumes against your actual engine versions, and read the file counts afterwards. Half an afternoon here saves a roadmap.
History is free to keep and expensive to store, which is a trap dressed as a gift. Decide what time travel window the business actually needs, write it down as a table property, and make expiry a routine job rather than an emergency response to a storage bill.
By this point the catalog is chosen, maintenance runs itself, and the write path is measured rather than assumed. Everything after this is repetition, which is exactly what you want a migration to become. Teams that regret Iceberg almost always started here and worked backwards.
Everything above, compressed into the table I wish someone had handed me three years ago.
| dimension | what you gain | what it costs |
|---|---|---|
| Correctness | ACID on object storage, snapshot isolated reads, safe concurrent writers | Optimistic concurrency degrades under heavy single table write contention |
| History | Time travel, rollback, snapshot diffs, reproducible reads by version | Snapshot retention is storage you pay for and expiry you must operate |
| Evolution | Rename, add, drop, repartition as metadata edits, no rewrites | Column ID discipline varies across older writers; validate the edges |
| Performance | Stats based pruning, no directory listing, planning from kilobytes of metadata | Small files and unmerged deletes erode it; compaction is mandatory hygiene |
| Freedom | One table, every engine, no vendor lock, open spec with multiple implementations | Write quality is engine dependent; the catalog choice is a real commitment |
| Latency | Minutes level freshness from streams is routine and reliable | Sub second freshness is the wrong ask; pair with a serving layer instead |
Strip away the ecosystem, the acquisitions, and the conference keynotes, and Iceberg's story is almost uncomfortably simple. The industry spent a decade fighting object storage's immutability, and one team at Netflix asked whether immutability was actually the foundation. Immutable files, an append only history, and a single guarded pointer: from those three commitments, transactions, time travel, and evolution all follow like water finding its level.
That is the pattern worth carrying out of this article, more than any tool name. The best systems of the last decade, Git, Kafka, Iceberg, are the same idea wearing different clothes: stop mutating state, start accumulating immutable facts, and make "current" nothing more than a pointer you move carefully. When you find yourself fighting your storage layer, the answer is rarely a cleverer mutation strategy. It is usually a confession that you needed a log all along.
Which brings us back to that 03:10 outage. The reason nobody could answer "what is in the table right now" was not a missing tool or a weak runbook. It was that the system had no place to put the answer. Iceberg's real contribution is not ACID or time travel; it is giving the question somewhere to live.
An iceberg the size of a country moves on currents you cannot see from the surface. The formats and engines above the waterline will keep churning; the mass below, the boring commitment to immutability plus one atomic pointer, is what actually steers. Study that part. It will outlive every logo in this article.
Start with the problem. Understand the forces. Add complexity only when the pain demands it. And always, always, read the spec before the blog post.
See you in the trenches.
Delta Lake and Hudi remain serious, production worthy formats, and Delta's UniForm plus the broader interop work means the formats are converging at the metadata layer anyway. This article is about why the gravity shifted, not a claim that the alternatives are broken.
Apache Paimon deserves its own piece: a table format designed around streaming upserts first, and the strongest answer today to Iceberg's freshness ceiling.
Catalog wars (Polaris, Unity, Nessie, Gravitino, Glue) are where the real vendor competition moved once the format question closed. That is a separate article, and a spicier one.
Iceberg spec v3 work, deletion vectors, row lineage, new types, is the format's answer to several costs listed above. Directionally right; adoption across engines is the thing to watch, not the announcements.