When Catalogs Are Embedded in Storage

This article is Part 8 of an Apache Iceberg Masterclass, focusing on the embedded catalog model where the catalog is integrated into the storage layer. It contrasts traditional Iceberg architectures with newer approaches like AWS S3 Tables and MinIO AI Stor, which reduce operational complexity by handling metadata, atomic commits, compaction, and snapshot management within storage. The article explains how S3 Tables, launched in late 2024, provides built-in maintenance and REST API access for query engines. It discusses when embedded catalogs make sense, hybrid approaches, and operational planning, targeting IT professionals working with data lakes, table formats, and cloud storage.