Notesbatch🌱 seedling

Data Skew

Also: Skew · Skewed Join · Skewed Partition · Hot Key · Salting · Key Salting · Straggler Task

When one key (or a few) has far more rows than the rest, hash partitioning sends all of them to one task, and the whole stage waits for it. The batch-processing form of a [[Hot Spot]].

Why it exists

Hashing spreads keys evenly, not rows. Real data has NULLs, default IDs, bots, celebrities and giant customers.

How it works

  • A stage finishes when its slowest task does; a skewed task can run hours while the other executors idle, and may spill or run out of memory.
  • Filter or split off junk keys like NULL, which often join to nothing.
  • Broadcast the small side, so there's no shuffle for the hot key to pile into.
  • Salt: append a random suffix 0..N−1 to the hot key on the big side and replicate the other side's matching rows N times; the hot key's rows spread over N tasks.
  • Two-phase aggregation: aggregate by salted key, then combine, or use a combiner before the shuffle.
  • Adaptive splitting: Spark's AQE splits a join partition larger than 5× the median and 256 MB into several tasks.

Trade-offs

Fix Costs
salting code changes; replicates the other side N times
adaptive splitting joins only; depends on accurate runtime statistics
broadcast only if one side fits in memory

In practice

  • Facebook engineers described joins where non-skewed partitions finished in minutes to a few hours and skewed ones took hours to days, delaying hundreds of downstream pipelines.
  • AWS's EMR guide suggests isolating and salting a value that holds 80% or more of the data, such as NULL.

Pitfalls

  • Adding machines: more executors don't help when one task holds the hot key.
  • Salting every key instead of only the hot ones multiplies the replicated side needlessly.

Recall

These cards are also in your review deck, scheduled with spaced repetition.