Orchestration, Dependencies & Backfills
Schedule pipelines with observable, recoverable task dependencies.
Learning objectives
By the end of this lesson, you should be able to:
- Explain the key concepts and when they are useful.
- Apply the concepts to a realistic engineering scenario.
- Identify common failure modes and choose a practical mitigation.
- Complete the assignment and explain your design decisions.
Core concepts
An orchestrator coordinates tasks, dependencies, retries, schedules, and run metadata. Tasks should be idempotent where possible, and retries should not duplicate output. Backfills rerun historical intervals; they need bounded windows, deterministic inputs, and protection against overwriting newer data.
Worked example
Prefer small tasks with explicit inputs and outputs. Distinguish transient failures from data-quality failures. A successful process exit is not sufficient if output row counts or freshness checks fail.
Engineering principles
- Make assumptions explicit. Document expected inputs, outputs, ownership, and failure behavior.
- Validate at boundaries. Treat user input, files, messages, and third-party responses as untrusted.
- Design for failure. Decide what happens when dependencies time out, data is malformed, or a process is retried.
- Prefer observable behavior. Log meaningful outcomes and expose enough diagnostics to debug without leaking secrets.
- Test the invariant. A test should prove a property that must remain true, not merely repeat the implementation.
Practical assignment
Sketch a DAG for ingest → validate → transform → publish → notify. Define retry behavior and the recovery plan for a failed date partition.
Suggested solution and reasoning
Use partition-aware tasks, write to a temporary destination, validate, then atomically publish. Track run state and make a rerun for the same partition produce the same result.
Use this as a reference design, not the only possible answer. A strong submission states assumptions, includes failure cases, and demonstrates how correctness will be tested.
Knowledge check
- What problem does this design solve, and what simpler alternative did you consider?
- Which input or dependency is most likely to fail?
- What invariant must remain true after a retry?
- How would you test the unhappy path?
Completion checklist
- I can explain the concepts without reading the lesson.
- I completed the practical assignment.
- I tested at least one failure scenario.
- I documented trade-offs and next steps.