Stop Treating Apache Iceberg Like an RDBMS

DATA ENGINEERING & ARCHITECTURE

Why applying relational database update patterns to cloud lakehouses causes write amplification, high compute costs, and degraded query performance.

Apache Iceberg is getting a lot of attention—and for good reason. It brings capabilities such as ACID transactions, snapshot isolation, schema evolution, time travel, and support across multiple data engines to files stored in object storage.

But these capabilities are also creating confusion. Because Iceberg supports commands such as MERGE, UPDATE, and DELETE, many teams assume they can use it like PostgreSQL, Oracle, SQL Server, or MySQL.

A Small Update Can Create Large Work

A traditional relational database is designed for frequent row-level transactions. When one customer record changes, the database locates the relevant page, modifies the row, records the transaction, and commits it.

Iceberg does not normally modify rows inside an existing Parquet, ORC, or Avro file. Its data files are treated as immutable: once written, they are replaced or supplemented rather than directly edited. When rows are updated or deleted, the change is generally handled in one of two ways:

Option 1: Rewrite the Affected File (Copy-on-Write)

Iceberg reads the data file containing the changed rows and writes a replacement file.

Imagine a 500 MB Parquet file containing five million rows, and you update only ten rows. The logical change is tiny, but the engine must:

• Read the affected 500 MB file.
• Apply the ten row changes in memory.
• Write a complete replacement file.
• Update table metadata and commit a new table snapshot.

The write becomes expensive, but future queries remain simple because they read the newly rewritten file.

Option 2: Keep the File and Add Change Files (Merge-on-Read)

Instead of immediately rewriting the original file, Iceberg keeps it and creates additional delta files describing which rows were deleted or replaced.

This reduces immediate write costs, but future queries must now read the original data file, the delete files, and replacement row files, combining them at runtime. The write is cheaper now, but reads become more expensive later until consolidated through compaction.

The Real Problem Is Amplification

The problem is not that Iceberg supports ACID. The problem is amplification: one small business change can trigger much larger file, metadata, and compute operations.

Consider two pipelines that each write 100 GB of data:

Comparison of Batched Writes versus Tiny Transactions.

Both pipelines write the same amount of data, but frequent small writes result in thousands of small files, inflated metadata, excessive object-storage requests, and higher cloud compute costs. This is how a lakehouse becomes a slow house.

Why Small Files & Compaction Are Expensive

Every file must be located, opened, inspected, scheduled, and processed. Reading 100 large files is vastly more efficient than opening 10,000 small files containing the exact same volumetric data.

Compaction reads many small files and rewrites them into fewer, larger files:

Compaction improves query performance, but it creates another compute-heavy workload. You end up paying for the original writes, queries against fragmented files, compaction compute, replacement file writes, and temporary duplicate storage. If a table needs constant compaction, the real problem is how data is being written.

Iceberg Should Not Be Your Application Database

An order-management application performs frequent row-level operations (creating orders, changing shipping addresses, issuing refunds). These belong in an operational database designed for low-latency updates. Iceberg is designed to store the resulting analytical history.

Recommended data pipeline flow connecting operational databases to Apache Iceberg.

Treat Iceberg as Append-Oriented

Append-oriented does not mean data can never be corrected. It means updates and deletes should be grouped, controlled, and applied deliberately. Instead of repeatedly updating one order record, store state changes as events (ORDER_CREATED, PAYMENT_CONFIRMED, ORDER_SHIPPED, ORDER_CANCELLED).

MERGE, UPDATE, and DELETE remain useful for applying CDC batches, removing duplicate records, handling privacy-related deletions (GDPR), and updating slowly changing dimensions—as long as they are executed in controlled batches rather than millions of isolated transactions.