Really great piece at Vox on how you can over-control tests to the point where the thing you're trying to detect is controlled away in a misleading way.

Statistical controls are great! Except when they're not.

The problem with controls is that it's often hard to tell the difference between a variable that's obscuring the thing you're studying and a variable that is the thing you're studying.

An example is research around the gender wage gap, which tries to control for so many things that it ends up controlling for the thing it's trying to measure. As my colleague Matt Yglesias wrote:

The commonly cited statistic that American women suffer from a 23 percent wage gap through which they make just 77 cents for every dollar a man earns is much too simplistic. On the other hand, the frequently heard conservative counterargument that we should subject this raw wage gap to a massive list of statistical controls until it nearly vanishes is an enormous oversimplification in the opposite direction. After all, for many purposes gender is itself a standard demographic control to add to studies — and when you control for gender the wage gap disappears entirely!

"The question to ask about the various statistical controls that can be applied to shrink the gender gap is what are they actually telling us," he continued. "The answer, I think, is that it's telling how the wage gap works."

It's a difficult chicken and egg problem, very relevant to studies of racism in police enforcement.

Imagine applying these controls to society itself. We still have race, but people of all races have the same amount of money, and they live in the same kinds of neighborhoods, and they do the same kinds of drugs, and they even drive the same kinds of cars. That society would be a lot less racist. But part of the reason we're so far from that society is racism. Discrimination perpetuates itself.

In some ways, what's amazing about many of these studies is that they show a racial effect even after controlling for so much of racism's work. They show that racism exists even in our control society — the one with equality of income, and education, and neighborhood, and car choices. The one where we've wiped out most every difference but pigment. The one where we've left ourselves no excuses for our prejudice. It is remarkable how much discrimination can survive.

Read through Harold Pollack's emailed thoughts at the bottom of the piece.