Technology
by Cathy O'Neil
Who checks the algorithm?
Software now helps decide who gets hired, who gets a loan and how long someone spends in prison. Cathy O'Neil, a mathematician who worked at a hedge fund, argues that many of these models are unfair, and that the people they hurt rarely find out why.
Hear a sample · read by Laura
O'Neil's three marks of a Weapon of Math Destruction. Opacity means the people being scored can't see how it works. Scale means it's used on millions, so one flaw repeats everywhere. Damage means it harms lives, often the poorest ones. A model with all three is a WMD.
Models often measure one thing as a stand-in for another they can't see. A postcode stands in for creditworthiness, and a friend's arrest record for your own risk of reoffending. O'Neil shows how proxies like these end up tracking race and poverty, even when race is never an input.
A healthy model learns when it's wrong. A WMD often creates the evidence that proves it right. Predictive policing sends officers to poor neighbourhoods, they record more petty crime there, and the new data sends them back. O'Neil calls these pernicious feedback loops.
O'Neil's healthy example is baseball statistics. The data is public, it measures what actually matters, like runs, and the models are corrected every season when their predictions miss. Most WMDs fail on all three, which is why they can stay wrong for years.
Three signs
A harmful model is secret, used on huge numbers of people, and does real damage.
Blind spots
Every model reflects the choices and blind spots of the people who built it.
The teacher
Washington, DC fired teacher Sarah Wysocki over a low score from a formula no one could explain to her.
Feedback loops
Bad models feed on their own results. More police in an area find more crime there.
Rankings
When U.S. News ranked colleges, schools chased the score instead of better teaching.
A story from the book
Kyle Behm, a student who'd been treated for bipolar disorder, applied for a minimum-wage job at a supermarket chain. An online personality test screened him out. Other chains used similar tests, and he was turned away again and again. His father, a lawyer, took the employers on.
One thing to try
Next time software makes a call about you, such as a rejected application, an insurance quote or a sudden change to your shifts, ask what data it used and how you can appeal. O'Neil wants models that people can see, question and correct.
Knotd has the key ideas from 250+ non-fiction books, each in a few minutes, in plain words. Weapons of Math Destruction comes with a Spot the Trap case to make it stick.
Get the whole book on KnotdOpen in the appWho checks the algorithm? Software now helps decide who gets hired, who gets a loan and how long someone spends in prison. Cathy O'Neil, a mathematician who worked at a hedge fund, argues that many of these models are unfair, and that the people they hurt rarely find out why.
Opacity, scale, damage; Proxies; Feedback loops; What a fair model looks like.
Next time software makes a call about you, such as a rejected application, an insurance quote or a sudden change to your shifts, ask what data it used and how you can appeal. O'Neil wants models that people can see, question and correct.
Yes. Knotd has the key ideas from Weapons of Math Destruction in a few minutes, in plain words, plus a Spot the Trap case to test them. It's one of 250+ non-fiction books in the app.
