Optimizing Parquet for fixed-length lists like embeddings, achieving up to 3.7x speedup with a fast path bypassing Dremel encoding.
Explores optimizing Parquet for fixed-length lists like embeddings, achieving up to 3.7x speed-up with a fast path bypassing Dremel encoding.
A deep dive comparing block vs. object storage, explaining how lakehouses made slower object storage fast for analytics.
An in-depth guide to file compression codecs, covering how data shrinks, codec differences, and practical recommendations for lakehouse architectures.
Analysis of the renaissance in columnar file formats, comparing Parquet, Lance, Vortex, Nimble, and BtrBlocks for modern data workloads.
Hardwood 1.0 is a fast, lightweight, multi-threaded Apache Parquet reader for the JVM, written from scratch with no mandatory dependencies.
Hardwood 1.0.0.CR1 release: improved ColumnReader API for Parquet files, initial geospatial support, and documentation overhaul.
Hardwood 1.0.0.CR1 release with improved ColumnReader API, geospatial support for Parquet, and documentation overhaul.
Analysis of a 1.75B-record airport weather dataset using DuckDB and QGIS on a high-end workstation.
Analysis of 10K+ satellites using GCAT dataset, DuckDB, and QGIS on a high-end workstation.
Hardwood 1.0.0.Beta2 release adds VARIANT support, interactive Parquet TUI, and performance improvements for Apache Parquet parsing.
Hardwood 1.0.0.Beta2 release adds VARIANT support, interactive Parquet TUI, and performance improvements.
Strategies for migrating data to Apache Iceberg, including in-place, full rewrite, and shadow migration with zero downtime.
Converting French mobile network datasets into Parquet format using DuckDB and GDAL on a high-end workstation.
Hardwood 1.0.0.Beta1 release: new S3 backend, predicate push-down, Avro bindings, CLI for Parquet parsing.
Hardwood 1.0.0.Beta1 release: new S3 backend, predicate push-down, Avro bindings, CLI for inspecting Parquet files.
A technical walkthrough of converting Canada's wind turbine database to Parquet format and analyzing it using DuckDB, QGIS, and command-line tools.
A technical walkthrough of converting the US Wind Turbine Database to Parquet format and analyzing it using tools like GDAL, DuckDB, and QGIS.
A technical walkthrough of converting the massive OpenBuildingMap dataset (2.7B buildings) into a columnar Parquet format for efficient cloud analysis.
Exploring the Layercake project's analysis-ready OpenStreetMap data in Parquet format, including setup and performance on a high-end workstation.
