Data Engineering Foundations
Learn how to build reproducible batch and streaming pipelines, model analytical data, orchestrate jobs, and enforce data quality.
Course overview
Learn how to build reproducible batch and streaming pipelines, model analytical data, orchestrate jobs, and enforce data quality.
Who this course is for
Developers who know basic programming and SQL and want to understand the full lifecycle of reliable analytical data pipelines.
Learning outcomes
- Build batch pipelines with validation and repeatable transformations.
- Model facts and dimensions with explicit table grain.
- Orchestrate dependent jobs, retries, and historical backfills.
- Explain Kafka partitions, consumer groups, offsets, and delivery semantics.
- Define data quality checks, lineage, access controls, and freshness signals.
- Deliver a documented end-to-end data engineering capstone.
Curriculum
Module 1: Pipeline fundamentals & modeling
- Data Pipelines & Batch Fundamentals — Design repeatable extract, transform, and load workflows.
- SQL, Data Modeling & Analytics Tables — Build trustworthy analytical models and query them efficiently.
Module 2: Orchestration & streaming
- Orchestration, Dependencies & Backfills — Schedule pipelines with observable, recoverable task dependencies.
- Event Streaming & Kafka Concepts — Understand event logs, consumer groups, ordering, and delivery semantics.
Module 3: Quality, governance & capstone
- Data Quality, Lineage & Governance — Detect broken data before it silently affects decisions.
- Warehouse Performance & Capstone — Optimize analytical workloads and deliver a complete pipeline.
How to study
- Read each lesson and write down the core idea in your own words.
- Complete the practical assignment before reading the suggested solution.
- Keep the work in a small repository with a README and reproducible commands.
- Revisit any lesson where you cannot explain a trade-off or failure mode.
- Finish the capstone with tests, documentation, and a short retrospective.
Capstone
Build a local analytics pipeline from sample CSV/JSON order data to cleaned and aggregated tables. Include schema validation, idempotent runs, quality checks, a scheduled-DAG design, backfill instructions, and a data dictionary.
Prerequisites
Basic programming, SQL, CSV/JSON handling, and Git. You can complete the exercises locally with files and open-source tooling; a hosted warehouse is not required.