Median Income

A lot of liberal bloggers have been citing this graphic that originally comes from the Detroit Free Press, purporting to show that most states have lost median household income from 1999 to 2005:



None of the bloggers that I've seen, however, seem to have any interest in where the figures came from. When you look at the original, the source is this: "U.S. Census Bureau data analysis by VICTORIA TURK and MARISOL BELLO/Detroit Free Press."

Is the analysis accurate? I don't think it is. Here's what led me to that conclusion:

Here is the Census Bureau's webpage listing median household income by state, from 1984 to 2005 -- the very thing that the Detroit Free Press was supposedly measuring. What's more, if you scroll halfway down the page, there is a separate set of tables that gives state-by-state figures all in 2005 dollars.

Let's take my home state of Arkansas. According to the Census Bureau's page, Arkansas' 1999 median household income -- in 2005 dollars -- was $34,770. Then in 2005, the median household income was $36,658. That's an increase of 5.4%, as opposed to the 7.2% decrease that the Detroit Free Press claims to have found.

How about another state: Utah. In 1999 (again, in 2005 dollars): $53,943. In 2005: $54,813. That's a rise of 1.6%, not a decline of 10.5% as the Free Press claims.

Even more oddly: In Michigan, the Free Press seems to understate the drop in income. In 1999: $53,989. In 2005: $45,933. That's a drop of 17.5%, although the Free Press has it at 12%. Michigan itself may be an outlier, though: The Census Bureau has Michigan's median household income rising from $46,990 in 1997 to $53,989 in 1999 -- a gain of 14.9% in just two years. That seems anomalous.

As for the nationwide median -- In 1999: $47,671. In 2005: $46,326. That's a 2.8% decrease, not the 6% decrease found by the Free Press. Not that the overall figures are comforting; a nationwide drop of 2.8% is nothing to sneeze at (although you'd have to know if the composition of households changed between 1999 and 2005).

* * *

So, with all of this in mind, it occured to me to ask the Detroit Free Press journalists what their data source was. I got an email response from one of them saying that they had used data from the Census Bureau's "American Community Survey," while the data I cite above came from the "Current Population Survey." Moreover, the journalist said that "both surveys are different and are conducted differently, so you can’t compare the data in one to the data in the other." Another one of the journalists responded that they didn't have "much time" to tell me where the data came from.

The first journalist then followed up and explained further that the 1999 data came from the 2000 Census (it's available here). They used the inflation calculator recommended by the Census Bureau. And then the 2005 data came from the American Community Survey (here).

I don't think this is a legitimate comparison. For present purposes, it doesn't matter which survey is more accurate or "better." What matters is that you simply can't determine whether there was a rise or fall in income if you're mixing and matching different datasets.

Indeed, the Census Bureau's publications regularly offer this sort of warning:

The Census Bureau recommends that people use the [Community Population Survey] as the data source for national estimates of income and poverty. While both the [American Community Survey or ACS] and the [Community Population Survey] offer income and poverty estimates at the state level, it is important not to draw conclusions from comparisons across surveys. For example, it is inappropriate to compare a state estimate of poverty in the ACS to a different state estimate in the [Community Population Survey].

Or this, from a webpage describing several different types of surveys:

The Census Bureau reports income and poverty estimates from several major national household surveys and programs:

Annual Social and Economic Supplement to the Current Population Survey (CPS ASEC)
American Community Survey (ACS)
Survey of Income and Program Participation (SIPP)
Census 2000 long form
Small Area Income and Poverty Estimates program (SAIPE)

Each of these surveys differs from the others in some ways, such as the length and detail of its questionnaire, the number of households included (sample size), and the methodology used to collect and process the data. The Small Area Income and Poverty Estimates program creates statistical models to produce income and poverty estimates by combining survey results with administrative records. As a result of this multiplicity of sources, it is important to understand that different surveys and methods, which are designed to meet different needs, also produce different results. * * *

[E]stimates from any one survey will almost never exactly match the estimates from any other (unless explicitly controlled), because of differences such as in questionnaires, data collection methodology, reference period, and edit procedures.

Most importantly here, the American Community Survey seems, for whatever reason, to produce lower results than the official Census figures. For example, in one detailed analysis comparing ACS to the Census in a couple of counties, the Bureau reported:

There were significant differences in the estimation of median household income. In Tulare County, the Census reported a value of $33,983 compared to the ACS estimate of $31,467. This is consistent with Census Bureau research in other ACS sites that generally found lower income values reported in the ACS . . . .

Indeed, if you look at Table 3 in this Census Bureau article, the Census 2000 figures for each state are usually a few percentage points higher than the ACS figures for 2000. In other words, ACS understates the Census 2000 figures. This suggests that if you start with Census 2000 figures, and then look at ACS figures for 2005, you might think there was a decrease in some states' income -- even though, as shown above, if you look at the same survey for both years, some of the same states' incomes actually rose.

To be sure, I have no intention of impugning the journalists' integrity or motives here. Census Bureau statistics can be maddeningly difficult to decipher, and the Free Press's results seem to be consistent with the common perception that the past few years were hit hard by an economic recession.

The moral: As I've said before, you can't always be 100% sure that a news story is correct about even the simplest of facts. When a newspaper article touches on a complex subject, use your head, and, if you can, look up the original sources for yourself.

UPDATE: Ezra Klein and Kevin Drum have, to their credit, acknowledged the problems in the Free Press's map.

UPDATE: Most of the people visiting this post will have seen this already, but Megan McArdle makes a similar point with her typical intelligence and wit.