A technical deep dive comparing metadata structures of modern table formats like Apache Iceberg, Delta Lake, and Hudi for data lakes.
Explains how Apache Iceberg enables partition evolution without rewriting data, solving a major data lake challenge.
Explains how Apache Iceberg uses metadata for data skipping, enabling fast query performance by eliminating 90-99% of files before scanning.
Explains how Apache Iceberg's hidden partitioning prevents accidental full table scans by automatically mapping source column filters to partition values.
Explains how Apache Iceberg table writes work, including commit steps and ACID guarantees on object storage.
Explains lakehouse catalogs in Apache Iceberg, their role in metadata management, and how to choose between open source and managed options.
Explains five ways Apache Iceberg table storage degrades over time, including small files, orphan files, and metadata bloat, with detection methods.
Explores embedding Iceberg catalogs directly into storage, covering AWS S3 Tables and MinIO AI Stor for simplified metadata management.
Guide to maintaining Apache Iceberg tables with compaction, snapshot expiry, orphan cleanup, and manifest rewriting for optimal performance.
Explains Apache Iceberg metadata tables for querying table internals using SQL, covering snapshots, files, manifests, partitions, and practical use cases.
Guide to using Apache Iceberg with Python libraries (PyIceberg, DuckDB, Polars) and MPP query engines like Dremio, Spark, and Trino.
Explores three streaming architectures for Apache Iceberg: Spark Structured Streaming, Flink, and Kafka Connect, focusing on trade-offs between latency and table maintenance.
A practical walkthrough of working with Apache Iceberg on Dremio Cloud, covering table creation, data ingestion, optimization, and AI-powered analytics.
Explores nine key design tradeoffs in query engines like PostgreSQL, ClickHouse, and DuckDB, covering storage, indexing, execution, and concurrency.
Strategies for migrating data to Apache Iceberg, including in-place, full rewrite, and shadow migration with zero downtime.
Explains row vs column storage layouts in databases, their I/O tradeoffs, compression benefits, and when to use each for query engines.
Explains how databases organize data on disk using heap files, sorted files, and LSM trees, plus open file formats like Parquet, ORC, and Avro.
Explores database indexing tradeoffs between B-Trees, LSM Trees, and other index types in query engine design.
Explains how database query optimizers work, covering rule-based and cost-based optimization, join order, cardinality estimation, and adaptive execution.
Explores query execution models: Volcano (row-at-a-time), vectorized (batch processing), and compiled code generation for CPU efficiency.
