File Encryption for the Lakehouse: The Terminology, the Machinery, and the Hard Problem of Interoperable Encrypted Tables

This article provides a comprehensive guide to file encryption in the modern lakehouse architecture, focusing on Apache Parquet's modular encryption and Apache Iceberg's table-level encryption (v1.11). It covers terminology, encryption layers, key hierarchies, and the hard problem of interoperable encrypted tables across multiple query engines. The piece includes design patterns, a healthcare example, a decision framework, and operational considerations like key rotation and crypto-shredding. It targets IT professionals and data engineers seeking practical, secure encryption strategies for open lakehouse environments.