Results of a recent study seemed to indicate a huge drop in obesity among kids, but only in the 2 to 5 year old band. This generated much puzzlement: what would cause just children in that age range to be less obese?
It turns out, according to several who've analyzed the study and the data, the news might just be a lot of statistical noise and fury, signifying not a whole lot. Emily Oster at the new FiveThirtyEight:
If you look at each age group separately, you’re likely to find large changes in some of them. That’s because as you select smaller and smaller groups within a data set, your problem with noise worsens. For example, say you survey 81 people who are 34 years old in 2003, and 85 people age 34 in 2011. What if you just happen to pick a particularly obese group in 2003 and a thinner group in 2011? With just 80-some people, it’s easy to see how this could happen, and yet you might inappropriately conclude that 34-year-olds in the broader U.S. population have become less obese.
One way to prevent yourself from drawing this erroneous conclusion is to apply a more rigorous t-test to the data, one that takes into account the fact that you looked at many subsets. In the JAMA paper, the authors did not do this. They tested for statistical differences within their smaller age groups, but interpreted their p-values relative to the standard significance level of .05. In the case of the 2- to 5-year-olds, the authors reported a p-value of .03. If this were an analysis of the overall NHANES, that p-value should be taken to indicate a significant change. But since the authors are slicing the data and doing multiple tests — they consider at least six different age groups — the statistics need to be adjusted to account for that. With that adjustment, they’d need a p-value of less than .008 to conclude an effect is significant.
To recap: the CDC study was small and had large error bars; other, larger studies find only slight drops in obesity; and there's no indication of any behavioral changes that might have produced a dramatic weight loss. I'd add to that the fact that the CDC data showed no correlation between lower weight at ages 2-5 and lower weight a few years later at ages 6-11.
Bottom line: I hate to be such a buzzkill, but the CDC result seems highly likely to be nothing more than statistical noise. Childhood obesity has barely budged in the last decade.