List: Data Engineering | Curated by Ricardo Mendes

Nov 9, 2024
23 stories
1 save
Data Engineering
In
Data Engineer Things
by
Matheus Jericó
Cloud-Native Data Engineering: Orchestrating Spark on Kubernetes with Custom Airflow Operator and…Deploying Apache Airflow on Kubernetes with Custom Airflow Operator integrated with Apache Spark and Google Cloud Storage
Nov 9, 2024
Nov 9, 2024
In
TDS Archive
by
💡Mike Shakhomirov
A Guide To Data Pipeline Testing with PythonA gentle introduction to unit testing, mocking and patching for beginners
Mar 9, 2024
1
Mar 9, 2024
1
In
The Modern Scientist
by
janmeskens
Data Ingestion — Part 2:  Tool Selection StrategyThis article is the second one in my series on data ingestion. For an introduction to the topic and to explore ‘data ingestion patterns’…
Jan 17, 2024
5
Jan 17, 2024
5
In
The Modern Scientist
by
janmeskens
Data Ingestion — Part 1: Architectural PatternsOver the course of two articles, I will thoroughly explore data ingestion, a fundamental process that bridges the operational and…
Nov 27, 2023
24
Nov 27, 2023
24
In
TDS Archive
by
Evan Heitman
A Neanderthal’s Guide to Apache Spark in PythonTutorial on Getting Started with Spark for Complete Beginners
Jun 14, 2019
11
Jun 14, 2019
11
In
Blog Técnico QuintoAndar
by
Felipe Miquelim
From Traditional BI to Lake House, a Data Architecture EvolutionHow QuintoAndar evolved its Data Platform to a Distributed Strategy.
Jul 10, 2023
Jul 10, 2023
In
TDS Archive
by
Tuan Nguyen
How to Deploy dbt to Production using GitHub ActionsHow to (and not to) deploy dbt to production.
Aug 21, 2021
4
Aug 21, 2021
4
In
TDS Archive
by
Romain Granger
How to Use Partitions and Clusters in BigQuery Using SQLOptimize your costs and speed up your queries
Jun 1, 2022
3
Jun 1, 2022
3
In
TDS Archive
by
Borna Almasi
How to Measure Data Quality13 Metrics You Should Be Tracking (But Don’t)
Mar 15, 2021
1
Mar 15, 2021
1
In
SeattleDataGuy By SeattleDataGuy
by
Ben Rogojan
Why Building Data Reliability Systems Is HardScaling Out Your Data Quality System
Apr 6, 2022
3
Apr 6, 2022
3
In
TDS Archive
by
Jon Loyens
How Should We Be Thinking about Data Lineage?Get a top-down view of your data and analytics ecosystem with comprehensive lineage
Mar 22, 2022
1
Mar 22, 2022
1
In
Geek Culture
by
Alvin Lee
The Rise of the Data Reliability EngineerEmerging Opportunities in Data
Mar 22, 2022
Mar 22, 2022
In
TDS Archive
by
Eric Broda
Data Mesh Patterns: Change Data CaptureData Mesh uses the Change Data Capture (CDC) pattern to move data reliably around the enterprise. This is a deep dive on the CDC pattern.
Jan 26, 2022
3
Jan 26, 2022
3
In
Tributary Data
by
Dunith Danushka
8 Practical Use Cases of Change Data CaptureHow to apply the practices of Change Data Capture to reliably move data from operational databases to other systems for other purposes.
Jun 20, 2021
1
Jun 20, 2021
1
In
SeattleDataGuy By SeattleDataGuy
by
Ben Rogojan
Can Palantir Take Over The Entire Data StackOr Is It Too Late To The Modern Data Stack Party
Mar 1, 2022
5
Mar 1, 2022
5
In
TDS Archive
by
Elena Goydina
The Missing Piece of Data Discovery and Observability Platforms: Open Standard for MetadataWhat could work for true data democratization at scale?
Aug 3, 2021
2
Aug 3, 2021
2
In
TDS Archive
by
Stephanie Shen
7 Steps to Ensure and Sustain Data QualitySeveral years ago, I met a senior director from a large company. He mentioned the company he worked for was facing data quality issues…
Jul 29, 2019
7
Jul 29, 2019
7
In
TDS Archive
by
Piethein Strengholt
Data domains and data productspractical guidance from the field
Nov 23, 2021
10
Nov 23, 2021
10
In
TDS Archive
by
Barr Moses
7 Questions to Ask When Building Your Data TeamWhen it comes to ensuring your team is happy and productive, answering these questions will make all the difference
Nov 18, 2021
Nov 18, 2021
In
TDS Archive
by
Prukalpa
Modern Data Stack Conference (MDSCON) 2021: The Top 5 Takeaways You Should KnowPro tips on growing data participation, protecting your data, increasing diversity, and more
Oct 23, 2021
Oct 23, 2021