High-Performance Columnar Transfers: Combining Apache Arrow Flight and Iceberg REST Catalogs

This article discusses the separation of control plane and data plane in modern lakehouse architectures by combining Apache Iceberg REST Catalogs and Apache Arrow Flight. Iceberg REST Catalogs standardize table metadata, discovery, governance, and authorization, while Arrow Flight provides high-performance columnar data transfer over RPC. The article explains how this composition improves performance, security, and flexibility for BI, notebooks, and AI agents, avoiding lock-in and enabling independent evolution of layers. It includes practical implementation patterns, common mistakes, and observability considerations for the agentic lakehouse.