The China Study: Invalid from the Start
I recently read one of the most famous books defending veganism: The China Study: The Most Comprehensive Study of Nutrition Ever Conducted And the Startling Implications for Diet, Weight Loss, And Long-term Health. The title of the book derives from a massive nutritional study carried out in China, the original of which I procured as well (Junshi et al., Diet, Life-style, and Mortality in China: A study of the characteristics of 65 Chinese counties, Oxford Univ. Press, 1990).
The "China Study" has been amply criticized elsewhere, but I wanted to highlight one criticism that, to my mind, is fatal.
Here's what Colin Campbell and his co-authors did when they studied China (for more details, see here and here). They started with county-by-county death rates from some 2,400 Chinese counties in 1973-75. They then went to China 10 years later in 1983 and 1984, whereupon they visited 65 selected counties out of 2,400.
In each county, they picked 100 people randomly and tested their blood ("Diet," p. 9). They gave three-day diet surveys to 30 families in each county (p. 16). At the end of all this, they came up with 367 different variables about mortality, urine and blood characteristics, diet, etc.
They originally published this research in the Junshi et al. book mentioned above. Roughly 800 of this book's 894 pages consist of listing each variable one by one, along with that variable's correlation with every other variable. One can hardly imagine anything more tedious and useless, except perhaps for this spoof:
This is because of the ecological fallacy. This fallacy lies in attributing characteristics to an individual when all you know is information about a group that he belongs to. For example, Alabama is more likely to vote Republican than Massachusetts. Alabama also has more black people than Massachusetts. But it would be completely wrong to conclude that black people are Republicans. Within both Alabama and Massachusetts, black people are more likely to vote Democratic than white people. And that's what matters if you're trying to predict how individual people will vote.
So go back to the China Study. They purported to find, for example, that liver cancer was related to blood cholesterol (see The China Study book, p. 78). It would be one thing if the China Study authors were claiming to have tested liver cancer patients and to have found high cholesterol levels. That wouldn't show anything about what causes liver cancer, of course: perhaps liver cancer causes high cholesterol, perhaps there's a third factor that causes both liver cancer and high cholesterol, or perhaps people who are prone to liver cancer are also prone to have high cholesterol for completely separate reasons.
But all of that is beside the point, because there isn't any genuine correlation between liver cancer and high cholesterol in the first place. The researchers didn't test the blood of anyone who died of liver cancer (or anything else, for that matter). The death rates all come from 1973-75, a full decade before the researchers went around testing different people's blood. All that the researchers really found was that if there's a Chinese county with a high liver cancer death rate in 1973-75, and if you go there 10 years later to test the blood of people who almost certainly don't have liver cancer and who are up to four decades younger than those who died in the 1970s, you might find high cholesterol levels.
This is about as valid as a study finding that because I can run a marathon, and because my grandmother died from breast cancer a little over 10 years ago, marathons are correlated with breast cancer. (The new bestselling book: "The Marathon Study: Why Running Endangers Your Health.").
It's a bit sad. For all of the effort and good intentions of the researchers, the China Study isn't even relevant. It certainly isn't a reason to advise people to do anything different about their diet.
The "China Study" has been amply criticized elsewhere, but I wanted to highlight one criticism that, to my mind, is fatal.
Here's what Colin Campbell and his co-authors did when they studied China (for more details, see here and here). They started with county-by-county death rates from some 2,400 Chinese counties in 1973-75. They then went to China 10 years later in 1983 and 1984, whereupon they visited 65 selected counties out of 2,400.
In each county, they picked 100 people randomly and tested their blood ("Diet," p. 9). They gave three-day diet surveys to 30 families in each county (p. 16). At the end of all this, they came up with 367 different variables about mortality, urine and blood characteristics, diet, etc.
They originally published this research in the Junshi et al. book mentioned above. Roughly 800 of this book's 894 pages consist of listing each variable one by one, along with that variable's correlation with every other variable. One can hardly imagine anything more tedious and useless, except perhaps for this spoof:
FORTHCOMING WORKS BY DR. BOLI.
Dr. Boli’s Handy Table of Computations. Every number in the world added to, subtracted from, multiplied by, and divided by every other number, with complete operations shown, for the benefit of students of mathematics. The largest and most comprehensive work of its kind. Now available: Vol. 1, “1, Part 1.”Tediousness is the least of the China Study's faults, however. We all know that correlation is not causation, but the China Study doesn't even rise to the level of producing meaningful correlations in the first place.
This is because of the ecological fallacy. This fallacy lies in attributing characteristics to an individual when all you know is information about a group that he belongs to. For example, Alabama is more likely to vote Republican than Massachusetts. Alabama also has more black people than Massachusetts. But it would be completely wrong to conclude that black people are Republicans. Within both Alabama and Massachusetts, black people are more likely to vote Democratic than white people. And that's what matters if you're trying to predict how individual people will vote.
So go back to the China Study. They purported to find, for example, that liver cancer was related to blood cholesterol (see The China Study book, p. 78). It would be one thing if the China Study authors were claiming to have tested liver cancer patients and to have found high cholesterol levels. That wouldn't show anything about what causes liver cancer, of course: perhaps liver cancer causes high cholesterol, perhaps there's a third factor that causes both liver cancer and high cholesterol, or perhaps people who are prone to liver cancer are also prone to have high cholesterol for completely separate reasons.
But all of that is beside the point, because there isn't any genuine correlation between liver cancer and high cholesterol in the first place. The researchers didn't test the blood of anyone who died of liver cancer (or anything else, for that matter). The death rates all come from 1973-75, a full decade before the researchers went around testing different people's blood. All that the researchers really found was that if there's a Chinese county with a high liver cancer death rate in 1973-75, and if you go there 10 years later to test the blood of people who almost certainly don't have liver cancer and who are up to four decades younger than those who died in the 1970s, you might find high cholesterol levels.
This is about as valid as a study finding that because I can run a marathon, and because my grandmother died from breast cancer a little over 10 years ago, marathons are correlated with breast cancer. (The new bestselling book: "The Marathon Study: Why Running Endangers Your Health.").
It's a bit sad. For all of the effort and good intentions of the researchers, the China Study isn't even relevant. It certainly isn't a reason to advise people to do anything different about their diet.
posted by Stuart Buck at
6:36 PM
1 Comments:
Great job, Stuart. You've done a great job identifying the "1st-order" issue with this study, even ahead of the wonderful nuances pointed out by folks like Mike Eades and Denise Minger -- both who did great jobs of expanding on these flaws.
Post a Comment
Subscribe to Post Comments [Atom]
<< Home