A Fast Path for Fixed-Length Lists in Parquet

This article discusses the performance challenges of storing fixed-length lists (e.g., coordinates, RGB colors, vector embeddings) in Apache Parquet, which uses Dremel encoding designed for variable-length lists, causing ~3x overhead. It explains how the Parquet community is working on a FIXED_SIZE_LIST logical type, but until then, a fast path optimization detects data pages with uniform list lengths and bypasses Dremel reconstruction. Implemented in the Hardwood release, this yields speedups of 1.1x to 3.7x for row and column readers, bringing performance close to flat columns, especially for long lists like 768-dimensional embeddings.