Data Quality, Lineage & Governance
Detect broken data before it silently affects decisions.
Learning objectives
By the end of this lesson, you should be able to:
- Explain the key concepts and when they are useful.
- Apply the concepts to a realistic engineering scenario.
- Identify common failure modes and choose a practical mitigation.
- Complete the assignment and explain your design decisions.
Core concepts
Data quality includes completeness, validity, uniqueness, consistency, timeliness, and accuracy. Tests should encode business expectations such as non-negative quantities, unique event identifiers, accepted status values, and freshness thresholds. Lineage answers where a dataset came from and which transformations produced it.
Worked example
Track schema versions and owners. Classify sensitive fields, minimize collection, restrict access, define retention, and avoid copying raw personal data into every downstream dataset. A dashboard should expose freshness and failed checks rather than hide them.
Engineering principles
- Make assumptions explicit. Document expected inputs, outputs, ownership, and failure behavior.
- Validate at boundaries. Treat user input, files, messages, and third-party responses as untrusted.
- Design for failure. Decide what happens when dependencies time out, data is malformed, or a process is retried.
- Prefer observable behavior. Log meaningful outcomes and expose enough diagnostics to debug without leaking secrets.
- Test the invariant. A test should prove a property that must remain true, not merely repeat the implementation.
Practical assignment
Create five quality rules for a daily revenue table and define what happens when each fails.
Suggested solution and reasoning
Block publishing on critical uniqueness or reconciliation failures, quarantine invalid rows when safe, alert the owner, and document whether consumers should use the previous good partition or see the dataset as unavailable.
Use this as a reference design, not the only possible answer. A strong submission states assumptions, includes failure cases, and demonstrates how correctness will be tested.
Knowledge check
- What problem does this design solve, and what simpler alternative did you consider?
- Which input or dependency is most likely to fail?
- What invariant must remain true after a retry?
- How would you test the unhappy path?
Completion checklist
- I can explain the concepts without reading the lesson.
- I completed the practical assignment.
- I tested at least one failure scenario.
- I documented trade-offs and next steps.