A Deep Dive Into File Compression: How Data Gets Smaller, Why Codecs Differ, and What to Actually Use in the Lakehouse

This article provides a comprehensive exploration of file compression in data infrastructure, focusing on lakehouse environments. It explains the fundamental principles of data compression, including redundancy and the pigeonhole principle, then details the two main families of compression techniques. The article profiles seven codecs—gzip, bzip2, LZMA, Snappy, LZ4, Zstandard, and Brotli—highlighting their design goals, trade-offs, and use cases. It also covers how compression integrates with Parquet files, columnar encodings, splittability, hardware acceleration, and the economics of object storage. The practical playbook offers guidance on choosing codecs and settings, with benchmark advice and a worked example. Written by Alex Merced, Head of Developer Relations at Dremio, this is a technical resource for data engineers and architects optimizing storage and query performance.