Notesbatch🌱 seedling
Data Skew
Also: Skew · Skewed Join · Skewed Partition · Hot Key · Salting · Key Salting · Straggler Task
When one key (or a few) has far more rows than the rest, hash partitioning sends all of them to one task, and the whole stage waits for it. The batch-processing form of a [[Hot Spot]].
Why it exists
Hashing spreads keys evenly, not rows. Real data has NULLs, default IDs, bots, celebrities and giant customers.
How it works
- A stage finishes when its slowest task does; a skewed task can run hours while the other executors idle, and may spill or run out of memory.
- Filter or split off junk keys like NULL, which often join to nothing.
- Broadcast the small side, so there's no shuffle for the hot key to pile into.
- Salt: append a random suffix 0..N−1 to the hot key on the big side and replicate the other side's matching rows N times; the hot key's rows spread over N tasks.
- Two-phase aggregation: aggregate by salted key, then combine, or use a combiner before the shuffle.
- Adaptive splitting: Spark's AQE splits a join partition larger than 5× the median and 256 MB into several tasks.
Trade-offs
| Fix | Costs |
|---|---|
| salting | code changes; replicates the other side N times |
| adaptive splitting | joins only; depends on accurate runtime statistics |
| broadcast | only if one side fits in memory |
In practice
- Facebook engineers described joins where non-skewed partitions finished in minutes to a few hours and skewed ones took hours to days, delaying hundreds of downstream pipelines.
- AWS's EMR guide suggests isolating and salting a value that holds 80% or more of the data, such as NULL.
Pitfalls
- Adding machines: more executors don't help when one task holds the hot key.
- Salting every key instead of only the hot ones multiplies the replicated side needlessly.
Connections
- Part of
- Batch Processing
- Learn first
- Shuffle
- Related
- Hot SpotJoin Strategies
- In real systems
- Apache Spark
Recall
These cards are also in your review deck, scheduled with spaced repetition.