Analysis of Hacker News job posts shows the Data Scientist role declining while ML Engineer roles rise, indicating a shift in the data job market.
Interview with Chad Sanderson on data platform leadership, experimentation culture, data quality, and the rise of data contracts.
Explains Project Nessie, an open-source data catalog for Apache Iceberg tables, and its importance for data engineers and architects building data lakehouses.
How to handle mismatched Parquet file schemas when querying multiple files in DuckDB using the UNION_BY_NAME option.
An update on how Monzo integrated machine learning across its organization in 2022, covering team structure, growth, and new initiatives.
Explores the shift to ELT in data engineering, focusing on modern tools like dbt, Fivetran, and Airbyte for loading and transforming data.
A software engineer explains their decision to join Decodable, a startup building a serverless real-time data platform, focusing on stream processing.
A technical walkthrough of using dbt and DuckDB to clean and analyze session feedback data from a tech conference.
A hands-on exploration of using dbt (data build tool) with DuckDB for local data engineering, based on a tutorial project.
Explains the evolution from ETL to ELT in data engineering, clarifying the role of modern tools like dbt in the transformation process.
A hands-on tutorial exploring LakeFS for data versioning and branching using PySpark and Jupyter notebooks in a data engineering context.
A curated list of essential resources for data engineering, including articles, newsletters, podcasts, and tools.
Explores modern data engineering trends in 2022, focusing on analytical data storage formats, organization, and access patterns.
A data engineer explores the evolution of the data ecosystem, comparing past practices with modern tools and trends in 2022.
A PhD student reflects on the complexities of ML engineering, distinguishing between Task MLEs and Platform MLEs, and shares practical lessons from production systems.
An introduction to modern data systems, explaining OLTP, OLAP, data warehouses, data lakes, and the roles of data engineers, analysts, and scientists.
A guide explaining key data engineering terms like data warehouses, data lakes, data mesh, and data pipelines, with definitions and comparisons.
Guide on configuring an external Apache Hive metastore with Azure SQL for use in an Azure Synapse Analytics Spark Pool, troubleshooting common connection errors.
A recap of 2021 conference talks on Debezium and Change Data Capture (CDC), exploring patterns and integrations with tools like Kafka and Infinispan.
A recap of 2021 conference talks on Debezium and Change Data Capture (CDC), exploring patterns and integrations with tools like Kafka and Pinot.
