Streaming and Big Data Engineering

Understand why distributed processing exists and the core concepts shared by batch and streaming systems.

• Scale, throughput, and latency as design drivers
• Distributed storage and processing fundamentals
• Batch versus stream processing and the Lambda and Kappa architectures
• Event time versus processing time
• Partitioning, replication, and the CAP trade-offs in practice
• Choosing between a database, a queue, and a log

Hands-on lab: Start a local Kafka and Spark environment in Docker, verify both are healthy, and map a sample business scenario to batch and streaming components on a design sheet.

Learn how Kafka's distributed log works and how to produce and consume reliably.

• Brokers, topics, partitions, and replication
• Keys, partitioning, and per-partition ordering
• Producer settings: acks, retries, batching, and idempotent producers
• Consumer groups, offsets, and rebalancing
• Retention and log compaction
• Command-line tools for inspecting topics and consumer lag

Hands-on lab: Create a multi-partition topic, write a Python producer and a consumer group of two consumers, and observe partition assignment and rebalancing when one consumer stops.

Model events well and evolve them without breaking downstream consumers.

• Events versus commands and designing event payloads
• Event sourcing and CQRS at a conceptual level
• Avro and Protobuf with Confluent Schema Registry
• Schema compatibility modes and safe evolution
• Kafka Connect for moving data in and out of Kafka
• Dead-letter topics and poison-message handling

Hands-on lab: Register an Avro schema, evolve it with a backward-compatible change, and confirm that an older consumer keeps working while an incompatible change is rejected.

Reason precisely about duplicates, loss, and what exactly-once really means.

• At-most-once, at-least-once, and exactly-once guarantees
• Idempotent consumers and deduplication keys
• Kafka transactions and read-committed consumers
• Offset commit strategies and their failure windows
• Exactly-once within a pipeline versus end-to-end with external sinks
• The outbox pattern for reliable event publication

Hands-on lab: Build a consumer that writes to Postgres, simulate a crash between processing and commit, and make the sink idempotent so replayed messages produce no duplicates.

Use Spark's distributed engine to process large datasets efficiently.

• Spark architecture: driver, executors, jobs, stages, and tasks
• DataFrames, Spark SQL, and lazy evaluation
• Reading and writing Parquet and Delta tables
• Shuffles, joins, broadcast joins, and data skew
• Partitioning, caching, and the Catalyst optimizer
• Reading the Spark UI to find bottlenecks

Hands-on lab: Run a PySpark job that joins a large fact dataset to a small dimension, inspect the stages in the Spark UI, and improve runtime with a broadcast join and better partitioning.

Express continuous computations with the same DataFrame model used for batch.

• The streaming DataFrame model and output modes
• Reading from and writing to Kafka
• Tumbling, sliding, and session windows
• Watermarks and handling late data
• Stateful aggregations and deduplication
• Checkpointing, triggers, and recovery after failure

Hands-on lab: Build a streaming query that reads events from Kafka, computes five-minute windowed counts with a watermark, writes to a Delta table, and restarts cleanly from its checkpoint.

Operate streaming and big data systems with predictable performance and spend.

• Sizing Kafka partitions, replication factor, and retention
• Monitoring consumer lag, throughput, and under-replicated partitions
• Spark cluster sizing, autoscaling, and spot capacity trade-offs
• Backpressure and managing slow consumers
• Managed offerings: Amazon MSK, Confluent Cloud, Databricks, and Amazon EMR
• Disaster recovery, replay, and backfill strategies

Hands-on lab: Load test your pipeline with a producer at increasing rates, record consumer lag and processing time, and write a sizing and cost note with the changes you would make.