The Metadata Structure of Modern Table Formats

This article is part 2 of a 15-part Apache Iceberg Masterclass, focusing on the metadata structures of modern table formats. It explains how Apache Iceberg uses a four-level metadata tree (catalog pointer, metadata file, manifest list, manifest files) to enable fast query planning, concurrent writes, and schema evolution. It also compares Iceberg with Delta Lake's sequential transaction log, Apache Hudi's timeline, and Apache Paimon's LSM trees. The article highlights how metadata organization impacts performance, pruning, and overhead, making it essential for data engineers and developers working with lakehouse architectures.