Streaming & Real-Time Analytics

Almost everything in this section assumes batch processing: you have a dataset, you analyse it, you get an answer. But a whole class of problems can't wait for the dataset to be complete — you need the answer as the data arrives. Catching fraud the moment it happens, alerting when a sensor crosses a threshold, updating a live dashboard — these need stream processing: analysing data continuously, as an endless flow of events, rather than in periodic batches.

It's a genuinely different discipline, not just "batch but faster," because the data is infinite — and that one fact forces some clever rethinking, especially about time. This page is the practical landscape: how streaming differs from batch, the windowing idea that makes infinite data tractable, the subtle problem of when an event happened, and the stack that runs it. It builds on the distributed computing page.

Batch vs stream: the fundamental split

The two paradigms answer different questions. Batch processing runs over a bounded dataset — all of yesterday's transactions — and produces a complete, correct answer, but only after the data is collected and the job runs (minutes to hours of latency). Stream processing runs over an unbounded flow — each transaction the instant it occurs — producing continuously updated answers with sub-second latency.

The trade-off is latency versus completeness. Batch is simpler and gives you the whole picture but late; streaming is immediate but must reason about data that's still arriving. You reach for streaming when the timeliness of the answer is worth the extra complexity — when an answer an hour from now is worthless.

Data that never ends

The defining challenge is that a stream is infinite. You can't "load the dataset" — there's no end. You can't compute a simple average, because the data the average is over never stops growing. Every familiar aggregate has to be rethought for a flow that doesn't finish.

A stream is a sequence of events, each a small immutable record stamped with a time — a click, a transaction, a sensor reading. The processing must be continuous and stateful (it remembers a running summary as events flow through), because it can never go back and reread the whole history. The key move that makes infinite data tractable is to chop it into finite pieces.

Windowing: making the infinite finite

You can't average "all" of an infinite stream, but you can average "the last five minutes." A window is a finite slice of the stream over which you compute — and windowing is the central idea of stream processing. The main kinds:

Two windowing styles over a stream of events. Tumbling windows are fixed, back-to-back, non-overlapping slices (each event in exactly one). Sliding windows overlap, advancing by a small step — giving a smooth rolling aggregate where each event falls in several windows.

Tumbling — fixed-size, non-overlapping ("every 5 minutes"). Each event belongs to exactly one window. Good for regular, discrete aggregates.
Sliding — fixed-size but overlapping, advancing by a small step ("the last 5 minutes, updated every 30 seconds"). Each event lands in several windows — ideal for smooth rolling averages.
Session — dynamic windows defined by activity gaps (a user's burst of clicks, closed after they go quiet). The window size adapts to the data.

Event time, processing time & watermarks

Here's the subtle problem unique to streaming: there are two different times for every event, and confusing them corrupts your results.

Event time — when the event actually happened (stamped at the source).
Processing time — when your system received and processed it.

In a perfect world they'd match. In reality, events arrive late and out of order — a phone loses signal and uploads its readings twenty minutes later, so an event that happened at 3:00 doesn't arrive until 3:20. If you bucket by processing time, that reading lands in the wrong window and your "3:00–3:05" total is wrong. You almost always want to aggregate by event time to get correct answers.

Delivery guarantees

When events flow through a distributed system that can fail mid-stream, a hard question arises: if a machine crashes and restarts, is each event processed once, never, or twice? The guarantees:

At-most-once — events may be dropped on failure. Fast, lossy; rarely acceptable.
At-least-once — no event is lost, but some may be processed twice on retry (so a count could over-report). The common default.
Exactly-once — the gold standard: each event affects the result precisely once, even through failures. Achieved with checkpointing and careful coordination — more expensive, but essential when double-counting would be a real problem (money, compliance).

Which you need is a genuine engineering decision tied to the cost of an error — the streaming echo of the trustworthy-pipeline concern.

The stack

A streaming system usually splits into two roles. A message broker — Kafka is the standard — is the durable pipe: it ingests events and holds them in ordered logs so producers and consumers are decoupled and nothing is lost. A stream processor — Flink, or Spark Structured Streaming (tied to the Spark engine) — does the actual computation: the windowing, the stateful aggregation, the event-time logic above.

You'll also hear of Lambda and Kappa architectures — broadly, whether you run separate batch and streaming layers (Lambda) or treat everything as a stream (Kappa). And the ML tie-in: streaming is the natural home for online learning and the real-time drift detection from the MLOps page — a model updated or monitored as data flows rather than retrained nightly.

Where it shows up in my work

When the answer can't wait

The questions that need streaming are the ones where timeliness is the whole point — real-time monitoring and alerting on incoming data, catching an anomaly the moment it happens rather than in tomorrow's report. Knowing the discipline is what lets me judge when the extra complexity of streaming is justified over a simple batch job (often it isn't — batch is simpler and fine when an answer in an hour is acceptable).

And the concepts that matter most in practice are windowing (how you aggregate an endless flow) and the event-time vs processing-time distinction — because late, out-of-order data quietly corrupting a real-time count is exactly the kind of subtle error that undermines trust in a live dashboard. It ties to distributed computing (the engines), anomaly detection (the alerts), and MLOps (real-time monitoring).

Refresh in 60 seconds

The batch-vs-stream split, windowing types, event-time/watermark handling, and delivery guarantees reflect current stream-processing references (Kafka/Flink practice) alongside hands-on work.