Heckman is Wrong

James Heckman (Chicago, Nobel prize winner) has been arguing that society should offer universal preschool for youngsters. His main evidence for this claim -- which I agree with, by the way -- is a couple of extremely small studies from the 1970s. What bothers me, however, is how he depicts these studies. To wit, here's what he writes in the New York Times:

Also holding back progress are those who claim that Perry and ABC are experiments with samples too small to accurately predict widespread impact and return on investment. This is a nonsensical argument. Their relatively small sample sizes actually speak for — not against — the strength of their findings. Dramatic differences between treatment and control-group outcomes are usually not found in small sample experiments, yet the differences in Perry and ABC are big and consistent in rigorous analyses of these data.

Contrary to what Heckman says, dramatic differences between treatment and control groups are MOST likely to show up in small samples. This is one of the most basic facts that any empirical scholar learns: when the sample size is small, sampling error will be the largest (for example, if a particular small sample happens to include a few outliers, that can swing the results in either direction rather dramatically). When the sample is large, that's when you'd expect to see smaller effects (e.g., in large samples, there's not as much opportunity for a few outliers to swing the results).

It's disturbing that Heckman seems to say the opposite of such a basic point.