Partition Evolution: Change Your Partitioning Without Rewriting Data

This article is part 4 of an Apache Iceberg Masterclass, focusing on partition evolution in data lakes. It details the traditional Hive problem where partitioning is permanent and requires rewriting all data to change. Iceberg solves this by using a partition spec that can evolve independently of data files, allowing partition strategies to change without rewriting data. The article covers how query planning handles multiple specs, real-world scenarios like growing from monthly to daily partitions, and adding or removing partition columns. It also touches on how other formats handle this and provides resources for deeper learning.