What Are Table Formats and Why Were They Needed?

This article is Part 1 of a 15-part Apache Iceberg Masterclass, covering the fundamental problem that table formats solve. It describes how data lakes without table formats are just collections of files with no transaction support, leading to issues like partial writes, expensive query planning, schema change difficulties, lack of time travel, and exposed partitioning. The article introduces five major table formats—Apache Iceberg, Delta Lake, Apache Hudi, Apache Paimon, and DuckLake—and explains where each excels, with a focus on why Iceberg has become the default. It provides technical context for IT professionals working with data lakes, lakehouses, and big data analytics.