The File Format Renaissance: Parquet, Lance, Vortex, Nimble, BtrBlocks, and the New Physics of Columnar Storage

This article explores the recent surge of innovation in columnar file formats, driven by AI workloads and hardware evolution. It explains how Apache Parquet's 2013 design assumptions are being challenged by point lookups, GPU-speed training, and multimodal data. The piece details the architecture, trade-offs, and use cases of new entrants like Lance (AI-native), Nimble (Meta's wide-table workhorse), and Vortex (aspiring general-purpose successor), alongside research formats like BtrBlocks and FastLanes. It includes a worked example comparing four formats, discusses the relationship between file formats and table formats, and offers practical guidance for choosing a format by workload in 2026. Written by a Dremio developer relations lead, the article is technical and focused on data engineering and storage.