1. From Batch to Streaming: Foundations of Real-Time Data Processing
• Recap of batch processing and its limitations in modern data applications.
• Stream processing paradigms: micro-batch vs. true streaming.
• Event time vs. processing time, watermarks, and late data handling.
• Stateful vs. stateless transformations.
• Delivery semantics: at-most-once, at-least-once, exactly-once.

2. Apache Kafka for Real-Time Data Streaming
• Advanced Kafka: consumer groups, partition assignment strategies, and rebalancing.
• Kafka Streams API: stateful stream processing within Kafka.
• Kafka Connect for source/sink integration with external systems.
• Schema Registry and data contracts with Avro, Protobuf, and JSON Schema.
• Kafka tiered storage and modern deployment patterns (KRaft, Confluent Cloud).

3. Apache Spark Structured Streaming
• The Structured Streaming model: treating streams as unbounded tables.
• Sources, sinks, output modes (append, update, complete) and triggers.
• Windowing operations: tumbling, sliding, and session windows.
• Stateful aggregations, joins (stream-stream, stream-static), and watermarking.
• Checkpointing, recovery, and exactly-once guarantees.

4. Apache Flink for True Stream Processing
• Flink architecture: JobManager, TaskManager, and the dataflow runtime.
• DataStream API and Flink SQL for unified batch and streaming.
• Event time processing, watermarks, and state backends (RocksDB).
• Savepoints, checkpoints, and stateful application upgrades.
• Comparison: Spark Structured Streaming vs. Flink vs. Kafka Streams.

5. The Lakehouse Architecture and Open Table Formats
• From data warehouses and data lakes to the lakehouse paradigm.
• Apache Iceberg: architecture, metadata layer, hidden partitioning, and time travel.
• Delta Lake: ACID transactions, the transaction log, and Z-ordering.
• Apache Hudi: copy-on-write vs. merge-on-read, indexing, and incremental queries.
• Schema evolution, table maintenance (compaction, vacuum), and interoperability.

6. Change Data Capture (CDC) and Real-Time Ingestion
• CDC concepts: log-based vs. trigger-based capture.
• Debezium for database CDC into Kafka.
• Streaming CDC into lakehouse tables (Iceberg / Delta / Hudi).
• Upserts, deletes, and slowly changing dimensions in streaming pipelines.
• End-to-end CDC patterns from OLTP databases to analytics platforms.

7. Unified Batch and Streaming Pipelines
• Lambda vs. Kappa architectures.
• The medallion architecture: bronze, silver, and gold layers.
• Building unified pipelines with Spark / Flink on lakehouse tables.
• Incremental processing and streaming-first data modeling.
• Real-time analytics serving with Apache Pinot, ClickHouse, and Druid (overview).

8. Cloud-Native Data Platforms and Deployment
• Managed Spark and Flink on Databricks, AWS EMR, and Google Dataproc.
• Object storage as the lakehouse foundation: S3, GCS, ADLS.
• Catalogs and governance: AWS Glue, Unity Catalog, Nessie, Polaris.
• Kubernetes for stateful streaming workloads.
• Cost optimization and autoscaling strategies.

9. Observability, Reliability, and Production Operations
• Monitoring streaming pipelines: lag, throughput, backpressure, and state size.
• Metrics, logs, and traces with Prometheus, Grafana, and OpenLineage.
• Data quality and contract testing in streaming contexts.
• Handling failures: dead-letter queues, replay strategies, and disaster recovery.
• Performance tuning for stateful streaming and large-scale lakehouse queries.