Notesfoundations🌱 seedling
Algorithmic Bias
Also: Machine Learning Bias · Algorithmic Fairness · Proxy Discrimination
Algorithmic bias is systematically worse outcomes for some groups from an automated system, usually learned from biased historical labels, proxies for sensitive attributes, or a target that measures the wrong thing, and then applied consistently at scale.
Why it exists
A model learns whatever patterns predict its labels. If the labels are past human decisions, it learns their biases; if the target is a convenient proxy, it learns the proxy's distortions. Removing the sensitive column doesn't help when other features correlate with it.
How it works
- Biased labels: a résumé ranker trained on a decade of mostly-male hiring learns to penalise signals like "women's" (Amazon's experimental tool, scrapped by early 2017).
- Proxy targets: a health algorithm predicting cost as a stand-in for need underrated Black patients' illness; fixing it would raise the share flagged for extra care from 17.7% to 46.5%.
- Feedback loops: decisions shape the next training data, so the skew compounds.
- Fairness metrics conflict: with different base rates, a score can't be equally calibrated for two groups and also have equal error rates, except in degenerate cases. Choosing which to hold is a values decision.
Trade-offs
| Gains | Costs |
|---|---|
| auditing outcomes by group catches harm | needs demographic data you may not otherwise collect |
| human review of high-stakes decisions | slower, costlier decisions |
| choosing a fairness criterion explicitly | it may lower aggregate accuracy |
In practice
- COMPAS (2016): Black defendants who didn't reoffend were labelled higher risk 44.9% of the time vs 23.5% for white defendants, while scores were similarly calibrated.
- Dutch childcare benefits: a risk model using nationality as an indicator; the tax authority was fined €2.75M (2021) and the government resigned.
- GDPR Article 22 limits solely automated decisions; the Dutch authority fined Uber nearly €825M (2026, under appeal) for fully automated driver deactivations.
Pitfalls
- "Fairness through unawareness": dropping the sensitive column while proxies remain.
- Auditing once at launch; drift and feedback loops change outcomes later.
Connections
- Part of
- Data Ethics
- Related
- Data Minimization
Recall
These cards are also in your review deck, scheduled with spaced repetition.