Event Streaming & Kafka Concepts
Understand event logs, consumer groups, ordering, and delivery semantics.
Learning objectives
By the end of this lesson, you should be able to:
- Explain the key concepts and when they are useful.
- Apply the concepts to a realistic engineering scenario.
- Identify common failure modes and choose a practical mitigation.
- Complete the assignment and explain your design decisions.
Core concepts
A stream is an ordered log within a partition, not a global ordering guarantee. Consumer groups distribute partitions across consumers. Offsets track progress, while keys influence partition placement and per-key ordering. At-least-once delivery can repeat messages; consumers therefore need idempotent processing.
Worked example
A robust event includes event_id, event_type, occurred_at, schema_version, and payload. Treat producer retries, consumer restarts, poison messages, schema evolution, and replay as normal operational cases.
Engineering principles
- Make assumptions explicit. Document expected inputs, outputs, ownership, and failure behavior.
- Validate at boundaries. Treat user input, files, messages, and third-party responses as untrusted.
- Design for failure. Decide what happens when dependencies time out, data is malformed, or a process is retried.
- Prefer observable behavior. Log meaningful outcomes and expose enough diagnostics to debug without leaking secrets.
- Test the invariant. A test should prove a property that must remain true, not merely repeat the implementation.
Practical assignment
Design an order-created event flow that updates analytics without making checkout wait for the analytics consumer.
Suggested solution and reasoning
Publish a versioned event, validate its schema, partition by order ID if per-order order matters, store processed event IDs or use idempotent upserts, and route repeated failures to a dead-letter process with monitoring.
Use this as a reference design, not the only possible answer. A strong submission states assumptions, includes failure cases, and demonstrates how correctness will be tested.
Knowledge check
- What problem does this design solve, and what simpler alternative did you consider?
- Which input or dependency is most likely to fail?
- What invariant must remain true after a retry?
- How would you test the unhappy path?
Completion checklist
- I can explain the concepts without reading the lesson.
- I completed the practical assignment.
- I tested at least one failure scenario.
- I documented trade-offs and next steps.