Notesfoundations🌱 seedling

Algorithmic Bias

Also: Machine Learning Bias · Algorithmic Fairness · Proxy Discrimination

Algorithmic bias is systematically worse outcomes for some groups from an automated system, usually learned from biased historical labels, proxies for sensitive attributes, or a target that measures the wrong thing, and then applied consistently at scale.

Why it exists

A model learns whatever patterns predict its labels. If the labels are past human decisions, it learns their biases; if the target is a convenient proxy, it learns the proxy's distortions. Removing the sensitive column doesn't help when other features correlate with it.

How it works

  • Biased labels: a résumé ranker trained on a decade of mostly-male hiring learns to penalise signals like "women's" (Amazon's experimental tool, scrapped by early 2017).
  • Proxy targets: a health algorithm predicting cost as a stand-in for need underrated Black patients' illness; fixing it would raise the share flagged for extra care from 17.7% to 46.5%.
  • Feedback loops: decisions shape the next training data, so the skew compounds.
  • Fairness metrics conflict: with different base rates, a score can't be equally calibrated for two groups and also have equal error rates, except in degenerate cases. Choosing which to hold is a values decision.

Trade-offs

Gains Costs
auditing outcomes by group catches harm needs demographic data you may not otherwise collect
human review of high-stakes decisions slower, costlier decisions
choosing a fairness criterion explicitly it may lower aggregate accuracy

In practice

  • COMPAS (2016): Black defendants who didn't reoffend were labelled higher risk 44.9% of the time vs 23.5% for white defendants, while scores were similarly calibrated.
  • Dutch childcare benefits: a risk model using nationality as an indicator; the tax authority was fined €2.75M (2021) and the government resigned.
  • GDPR Article 22 limits solely automated decisions; the Dutch authority fined Uber nearly €825M (2026, under appeal) for fully automated driver deactivations.

Pitfalls

  • "Fairness through unawareness": dropping the sensitive column while proxies remain.
  • Auditing once at launch; drift and feedback loops change outcomes later.

Recall

These cards are also in your review deck, scheduled with spaced repetition.