NotesSystem

Hadoop

Also: Apache Hadoop · HDFS · YARN · Hadoop MapReduce

The open-source implementation of Google's GFS and MapReduce, split out of the Nutch search engine in 2006 and grown up at Yahoo. Its defining choice is moving computation to where the data lives, in a replicated distributed file system.

Design choices

  • HDFS: files split into large blocks, each replicated (three copies by default) across machines and racks.
  • MapReduce: map tasks scheduled on nodes holding a replica of their input; intermediate data on local disk, final output back to HDFS.
  • YARN (Hadoop 2): separates cluster resource management from the MapReduce engine, so Spark, Tez and others share one cluster.

Notable trade-offs

  • Chained MapReduce jobs write replicated intermediate files at every step, which Dataflow Engines avoid.
  • Running HDFS means owning disks, NameNode high availability and capacity planning; many platforms moved to object storage and Kubernetes.

Used by

  • Yahoo ran Hadoop on 25,000 servers holding 25 PB in 2010, its largest cluster 3,500 servers.
  • Pinterest has been moving its Hadoop batch platform to Spark on Kubernetes.
  • Cloudera and Hortonworks, the main Hadoop vendors, merged in January 2019; the project still ships (3.5.0 in April 2026).