It's worth starting this discussion with the strongest possible form of the argument I think is flawed, which I think is Charles Isbell's fantastically well produced talk You Can't Escape Hyperparameters and Latent Variables: Machine Learning as a Software Engineering Enterprise. It's very good, and contains nothing I actually disagree with. I proactively agree with the overwhelming majority of that talk, and would have some nitpicks about some of the things the guests said (outside of the core dialogue), but nothing that changes the thrust of the talk, which is of the highest quality. It is not technical enough of a talk for you to worry about, so really, please enjoy it.
So, something that isn't explained in that talk, but only referenced, is critical to one of their points, and while their take is correct, I think it missed two big issues. So first, what did they say that caught my ear the wrong way?
They said that engineers shouldn't be the ones making the tradeoffs between overall predictive accuracy and "false imprisonment" (29 minutes into the talk, slide 50). What's that mean? Why would better accuracy mean more false imprisonment? Well, that phrasing is extremely lazy, but he's speaking to a non-technical audience, so let me explain: this paper (among others) proves an impossibility result about subgroup blindness. Which group you're in shouldn't matter to how you're treated by the system. That's a core element of what we mean by fairness: that you are judged individually, not as a member of a group. And they prove, mathematically, that for your system to be even approximately fair, and approximately accurate, reality has to meet certain conditions that don't always apply. This isn't a result about current ML methods. This is a statement of statistical fact, and while nuanced and worth considering on its own (it's a good paper, as are others in this area – impossibility results are perhaps the most interesting sub-category of academic results).
So when he says he doesn't want to make those trade-offs, that's what he is describing. The tradeoffs between general accuracy and something akin to Type I and Type II subclassing error. Reality doesn't allow us to be perfect, and the trade-offs can't be made by us. But it's worth considering that being in jail because of increased statistical inaccuracy isn't somehow more fun than being in jail because of categorization error. Someone has to make the choice, and since they all suck (because they have to suck some minimal amount), I don't think it's reasonable to blame engineers for getting bad outcomes. Good outcomes, by this standard, were never on the table. They're impossible.
This fundamentally isn't an engineering problem. That's my take, even though I do think the talk is correct in saying engineers should care about these attributes. This is a management decision, and it's strange how even talks about bias don't say Sundar Pichai is biased against people of color – they'd be silly, because he's also a person of color, but that's where responsibility for these trade-offs exists.
There is a moment where one of the people appearing uses a bridge analogy, saying that we're building systems for the whole world, so it can't be like building a bridge in New York and just thinking about the needs of people on Wall Street. That was an extremely bad analogy, for a couple of reasons, but most saliently, they're conflating out-of-distribution accuracy with deliberate attempts to ignore the needs of the majority of users. I think that guest, or many others in the "Ethics in AI" field, aren't treating normal AI/ML researchers and engineers as if they are doing their best on the technical questions, which they are. People in the field really do care deeply about, not just accuracy, but also out-of-distribution accuracy and detection. Yes, it is important! And they're right to say the future will look better because we are focusing on this. But: this is a management failure, and a failure to appreciate that real life comes with trade-offs you can't avoid.
And it isn't about race. There is a broader safety concern here, that experts at the highest levels have been shouting from the rooftops for years: we do not know how to make sure machine learning systems do what we want, and are extremely limited in our ability to even detect a cause for their outputs. You might want to read that a few times to have it sink in. We do not know how to tell our most capable systems what to do, nor can we check what they're doing.
This technical project needs massive amounts of work, and by making this so race-focused we are losing the chance to have a more deeply nuanced discussion about the dangers of ML in general (with our current techniques). Our ML capabilities are outpacing our safety work by such a huge margin that Rohin Shah, an expert in the technical field, says the risk of this causing human extinction is only as low as 10% because we're likely to encounter a major enough disaster along the way that we will be scared straight (and that's considered a low estimate of the danger, but obviously there is an unbelievable amount of uncertainty).
This is an extremely important discussion, and I think the current talk about racial bias barely scratches the surface. The reason the above impossibility result is even relevant is that machine learning is good at reducing in the amount of literal bias it has (they're also good at reducing variance, because those are the two factors that form squared error loss functions). And the accuracy, in general, will get better – this is what the experts deal with. But specific issue focused concerns (like racial bias) can't possibly address the core safety concerns: that we don't know how to ensure ML systems do what we want, and have weak tools for determining why they're doing whatever it is they end up doing. If we don't have a broader lens, and more compassion and understanding when literally impossible standards of perfection not being met, our whole species is put at risk.