The Alignment Problem: Machine Learning and Human Values
by Brian Christian
Rating: 4.0 / 5 · Read · Finished 2024-01-20
A book about all the ways things can go wrong when training machines to make decisions for us.
Before I started the book, it sounded like it was going to be about really dangerous and existential questions about AI, but then the first part of the book focuses almost exclusively on bias. I was thinking we were going to talk about a terminator scenario, and instead it was talking about how “doctor” is closely associated with “man”.
But then the book opened up into a a breadth of really fascinating topics in machine learning, human learning and philosophy, and all the ways things can go wrong.
Things that jumped out to me:
- Recidivism models predict policing, not just crime. (More police in an area means more crimes turn into arrests)
- Prediction models can ultimately affect their own future training data. Sending more police to areas with likely crime causes more arrests, which serve as training data for the next version of the model.
- Differential enforcement can embolden the less-affected group more than it deters the more-affected group, thus reducing safety. E.g. pulling over black people more than white people may embolden white people to speed.
- Prediction software is used to decide whether to detain someone in order to ensure they show up for their court date. But other things also increase likelihood of showing up for court date: text message reminders, child care, etc.
- A model was trained to predict risk of death for pneumonia patients. Lower risk patients were sent home. But the model learned that asthma meant low risk of death. Because asthma patients are usually sent to ICU and receive better care. Uh oh.
- Interpretability and performance are often at odds with each other. But uninterpretable models are more likely to have some loophole or unintended behavior. Interpretable models are easier to improve by tweaking the training set or feature set.
- Simpler models are more robust across contexts. A simple model for predicting successful marriage: do you have sex more often than you have fights?
- Really really enjoyed the chapter on curiosity. You take two models. One tries to predict what will happen as accurately as possible. One tries to get into situations that the first model mispredicts. The result is novelty seeking behavior.
- In order to be surprised, you need to have a model of how things work. If you don’t have a model, all possible states are equally likely to you.
- Boredom is a lack of curiosity or a lack of novelty. Addiction is pathological curiosity. (Will I win on the next pull of the slot machine??)
- Learning by watching experts doesn’t teach you how to recover from beginner mistakes.
- Alpha Go Zero sounds so cool. Two models. One searches through a weighted tree of moves, looking for the best outcome. One predicts the result of the search. The predictions are used to tune the weights of the search. Think of the two systems as slow methodical thinking, and fast intuitive thinking.
- Remaining uncertain may be the most high utility thing you can do, so you don’t shut out possible actions when you know more later.