Notesstream🌱 seedling

Stream Join

Also: Streaming Join · Interval Join · Windowed Join · Stream-Table Join · Temporal Join · Lookup Join · Stream Enrichment

A streaming join must remember one side until the matching record from the other side arrives, so its cost is state. Time bounds (an interval, a window, a versioned table) tell it when it may forget; without them, state grows forever.

Why it exists

Enrichment (orders + customer data), attribution (clicks + impressions) and correlation (payments + fraud signals) all need records from different streams combined, while neither stream waits for the other.

How it works

  • Stream–stream (interval / windowed): buffer both sides in keyed state, emit matches within the interval, and evict once the Watermark passes the interval's end.
  • Stream–table (temporal): keep the table as state, built from a changelog (often a compacted or CDC topic). Each stream record looks up the version valid at its event time, which requires a versioned table.
  • Lookup: query an external store per record at processing time. Little state, but non-deterministic on replay.
  • Table–table (regular): maintain a joined Materialized View of two changelogs, holding both sides forever unless bounded by a TTL.

Trade-offs

Join Gains Costs
interval bounded state, deterministic misses matches outside the bound
temporal correct as-of-event-time enrichment versioned table; watermarks on both sides
lookup tiny state, always fresh external load; different answers on replay
regular full relational semantics unbounded state unless TTL (which drops matches)

In practice

  • Kafka Streams: KStream–KStream joins are always windowed. KStream–KTable joins trigger only on stream records.
  • Flink SQL: regular joins keep both inputs in state forever, and table.exec.state.ttl bounds them at a correctness cost. One published case replaced regular joins with temporal joins and cut state from about 780 GB to checkpoints under 4 GB.

Pitfalls

  • Regular joins on unbounded streams grow state until checkpoints time out.
  • A stream–table join that is non-deterministic on replay, because the table has changed since.
  • Forgetting that both inputs need to be co-partitioned by the join key.

Recall

These cards are also in your review deck, scheduled with spaced repetition.