The Buck Stops Here: Peer Review

Sunday, November 22, 2009

Peer Review

Out of all the recent emails that were apparently hacked or leaked from a prominent laboratory in England ("Climategate," some are calling it), this one bothers me. It shows how "peer review" worked from the inside:

From: Keith ***
To: Edward ***
Subject: Re: Review- confidential REALLY URGENT
Date: Wed Jun 4 16:02:09 2003

[OMITTED]

At 09:50 AM 6/4/03 -0400, you wrote:

Hi Keith,
Okay, today. Promise! Now something to ask from you. Actually somewhat important too. I got a paper to review (submitted to the Journal of Agricultural, Biological, and Environmental Sciences), written by a Korean guy and someone from Berkeley, that claims that the method of reconstruction that we use in dendroclimatology (reverse regression) is wrong, biased, lousy, horrible, etc. They use your Tornetrask recon as the main whipping boy. I have a file that you gave me in 1993 that comes from your 1992 paper. Below is part of that file. Is this the right one? Also, is it possible to resurrect the column headings? I would like to play with it in an effort to refute their claims.

If published as is, this paper could really do some damage. It is also an ugly paper to review because it is rather mathematical, with a lot of Box-Jenkins stuff in it. It won't be easy to dismiss out of hand as the math appears to be correct theoretically, but it suffers from the classic problem of pointing out theoretical deficiencies, without showing that their improved inverse regression method is actually better in a practical sense. So they do lots of monte carlo stuff that shows the superiority of their method and the deficiencies of our way of doing things, but NEVER actually show how their method would change the Tornetrask reconstruction from what you produced. Your assistance here is greatly appreciated.

Assuming that this email is valid -- and no one seems to be claiming that any emails were altered -- this guy is being asked to peer review a manuscript. Does he think the manuscript is actually wrong? No -- he admits that it's full of math that "appears to be correct" (to be sure, he implies that he finds the math hard to understand). The only thing he can find wrong with the paper is that it's too theoretical, but he obviously isn't content recommending rejection on that basis. So he's worried that the paper "could really do some damage" to another paper by Keith Briffa. Unable to determine the paper's merits quite yet, but also without any doubts about trying his best to get the paper rejected anyway, he seeks help from the very person being criticized as to how to ding the article.

Is this any way for the peer review mechanism to work? I could hardly imagine anything more directly opposite to the ideal of giving manuscripts on a blinded basis to independent and unbiased reviewers who have sufficient expertise to judge an article's merit for themselves.

19 Comments:

Michael Drake said...: Stuart, "Keith" spends the entire second paragraph of the email describing how he thinks the paper is - methodologically - actually wrong. The errors don't need to be in the math. (There are plenty of incorrect science papers that contain correct mathematics; cf. the phenomenon - one you've adverted to - of including actual, "correct" brain scans in otherwise specious papers.) Given that's so, there's no reason "Keith" might not have accepted the math arguendo on an initial read-through of the paper to see where the broad argument is going.

I think whether this writer's posture toward the paper is appropriate depends on the content of the particular paper. If the paper to be reviewed uses a lot of (at-first-blush) correct mathematics to make an erroneous argument, it's appropriate. If not, not.

None of this is to say that Keith and Edward couldn't be raving hacks.

Side note: There is no inherent objection to authors like "Keith" and "Edward" setting out to advocate for their own approach (of which the target article is critical). The only problem would be if they held themselves out as objective referees without disclosing their interest.; 9:17 AM
Stuart Buck said...: I don't think we have to know ourselves whether the paper was erroneous to know that peer review shouldn't mean "secretly give someone who is criticized a chance to help veto someone else's publication." I don't think anyone would choose that ex ante as a good rule.; 9:19 AM
Stuart Buck said...: Besides, he doesn't have a good reason (not yet) as to prevention of publication. All he says is that it's too theoretical and doesn't provide a better way to do reconstruction. Well, so what? If a paper shows -- via math that is admittedly "correct" -- that the way things are being done is stupid, then that is a valid point in and of itself.; 9:28 AM
Michael Drake said...: Maybe (probably) there's something I don't know about the position of power or influence these authors hold, but it doesn't seem clear from the face of what you've quoted (and your post is the only post I've read about this affair) that anyone's being "secretly give[n] a chance to help veto someone else's publication." Which, I'd quite agree, would be problematic, if they had.

In any event, there's nothing wrong with a paper's pointing out that the current way of doing things is stupid; then again, there's nothing wrong with defending against the charge, or pointing out that a paper limited to finding fault with current methods without offering a competing method or research program are pretty worthless from a scientific perspective.; 7:17 PM
Stuart Buck said...: Maybe not a literal "veto," but the whole point of peer review is that the peer reviewers do have some sway over whether the article gets published.

I disagree with your last point -- showing that a current method is wrong is far from worthless. If a small group of scientists at a particular point in time are caught up in the equivalent of alchemy or astrology, it's perfectly legitimate and scientific to say, "This is bogus," without having to face the inquiry, "Well, then, show us YOUR plan for turning lead into gold, or for predicting the future via the stars." Maybe it's just impossible to do the thing that the scientists are trying to do -- and THAT's worth knowing.; 8:04 AM
Michael Drake said...: Yes, "worthless" was a bit overstated and presumptuous. It certainly would be worthwhile to know whether there's an authentic problem with the state of the art. But what may be going on here (because I assume neither that the target report is factually correct nor that it is incorrect) is what you see with the the intelligent design movement: the assertion that there are problems with the leading theory or method, completely unreflective of the (far worse) problems with the competing theory or method (such as it is). Motivated skepticism of that sort, I take it, would be relatively worthless. But of course I don't know that that's what the target paper is engaged in. My point (mostly) is that as far as I can tell from what you've quoted here, it might be, in which case it seems to me that "Keith"'s posture would be appropriate.; 9:34 AM
Stuart Buck said...: It might be indeed, but I read the guy's email as admitting that the authors were "correct" in demonstrating a mathematical case that his own methods were "wrong, biased, lousy, horrible, etc." So I read the email as basically amounting to something like this: "I've got a paper to review that has all this really hard math showing that our astrological methods are crap. Unfortunately, the math looks correct and it will do a lot of damage to us astrologers if it is published, so I need to refute it on the ground that the authors don't have a better way of using the stars to make predictions."; 11:21 AM
Michael Drake said...: Sure, that's one reading. Another is: Here's a paper that claims our methods are wrong. It uses a lot of convincing-looking, possibly correct, but in any case irrelevant math that could give global warming deniers some colorable authority they can cite to in their on-going public relations campaign. I'd like your help to see if we can refute its claims so that it won't be published. I see absolutely no reason within the four corners of your excerpt to prefer your reading to mine (or mine to yours). Do you?; 2:29 PM
Stuart Buck said...: Well, I don't see where he claims or insinuates that the criticisms are actually "irrelevant." Rather, he seems to be saying that the criticisms are correct but too theoretical because not accompanied by a better way of doing X. That's not quite the same as "irrelevant."

And in any event, seeking the help of someone who is criticized shouldn't be part of the peer review process. Imagine the same situation in reverse -- Soon and Baliunas managed to publish some article that a lot of climate scientists disagreed with. Suppose Baliunas were a peer reviewer for some other journal, and was found to have sent an email to Soon saying, "Hey, I have this paper to review with a lot of math that looks correct and that seems to show your previous work to be rubbish. This could really hurt us if it's published, so you need to help me come up with a reason to recommend rejection."; 3:49 PM
Michael Drake said...: That he thinks it's irrelevant is (by my lights) a fair inference from his assertion that the criticisms had no bearing on the Tornetrask reconstruction (whatever that is).

But, look. Ed (I've inadvertently been calling the writer "Keith") should not be on any panel that purports to comprise blind, neutral referees; he is clearly an interested player, or is partial to an interested player.

What's not clear to me, however, is that Ed was part of any such panel refereeing this paper. Was he? If so, I think that's a problem.; 5:09 PM
Stuart Buck said...: I think the header I quoted was from Keith to Edward, but it was quoting an original email from Edward to Keith. Keith wasn't on the panel of referees, as far as I can tell, but it seems that Edward was (he says that he has a "paper to review" that had been submitted to a journal).

Fair enough on the "irrelevant" point, but again that goes to whether the authors came up with a *better* way of doing "reconstruction", not whether they had a valid point that current methods are "wrong, biased, lousy, horrible." And to my mind, if a current method of doing something is indeed "wrong, biased, lousy, horrible," then if there's not a better way, then the answer isn't to sweep the discussion under the rug, but maybe to stop doing that thing entirely.; 9:44 PM
Michael Drake said...: "[that doesn't speak to] whether they had a valid point that current methods are "wrong, biased, lousy, horrible."

How does merely claiming (because that's the word Ed uses in the email) that a paper is "wrong, biased, lousy, horrible, etc." constitute a "valid point"? And if it doesn't, why would the onus be on Ed or Keith (or me) to speak to it?; 12:25 PM
Stuart Buck said...: I'm a little lost here. The point I'm making is that if Edward is tasked with peer reviewing an article, and the article shows -- using admittedly "correct" math -- that a current method of doing something is just plain wrong (mathematically speaking), then Edward can't dismiss the article as irrelevant just because it doesn't show that there's a better way. Maybe there isn't a better way.; 1:43 PM
Michael Drake said...: But you presume that the paper actually does show "that a current method of doing something is just plain wrong." I don't see an iota of support for that presumption in Ed's email. Yes, there's plenty in Ed's email that's consistent with that presumption; but it's also consistent with the presumption that the paper is a hack job that uses apparently (not "admittedly") correct but in any case irrelevant data to purport to find problems with an established method or theory in order to sustain artificial controversies, after the standard denialist manner.

It's clear that you prefer something like the first interpretation and I something like the second. But my rather humble point is this: neither of these interpretations can be justified based only on the content of Ed's email.; 7:29 PM
Stuart Buck said...: I don't get the impression that the email author thinks the paper is a hack job; he seems rather to think it has a lot of hard math that is beyond his ability to refute directly.; 10:10 PM
Michael Drake said...: Ed is asking Keith whether they can "resurrect the column headings" explicitly so that Ed can "play with [the recon] in an effort to refute their claims." By my way of thinking, it would be very odd for a person consciously to set out to refute math that he thought was beyond his ability to refute.; 11:51 PM
Stuart Buck said...: Put it this way: beyond his ability to refute without seeking behind-the-scenes help from someone else who wasn't supposed to be part of a peer review process. It seems very motivated reasoning here: the guy wants to refute it because he's afraid that it will damage him for people to know that he's doing crap work.; 9:40 AM
Michael Drake said...: Well, here again
your argument turns on heavily loaded paraphrasing of Ed's language. What you characterize as "behind-the-scenes help" is, on its face, a request for a complete file with repopulated column headings - in other words, a more complete data set. ("I have a file that you gave me in 1993 that comes from your 1992 paper. Below is part of that file. Is this the right one? Also, is it possible to resurrect the column headings?") On no fair reading is that request improper.

Nor, incidentally, would it even have been improper had Ed asked for other kinds of clarification from Keith, or even for "behind-the-scenes help" with the math. Referees can talk to other parties, even interested parties, to allow them to clarify their statements or illuminate technical points.

With that, I'll submit and take my answer of-air. ;-); 8:51 PM
Stuart Buck said...: Well, I can see we're not going to convince each other. (Didn't Daniel Dennett lament somewhere the fact that people make reasonable arguments to each other all the time, and then walk away without having been affected by anyone else's argument?) Anyway, I hope I'm not being too cynical in this, but the guy's email strikes me as a paradigm example of motivated reasoning. It all seems to boil down to, "Geez, here's a paper that argues I've been doing crap research, and what's worse, the math looks correct. I've got to pull out all the stops to undermine this before it gets published somewhere."

Now maybe in this particular case, the email writer is actually right on the substance -- I certainly wouldn't know -- but it strikes me as a very bad idea for "peer review" to ever consist of letting a paper's chief targets themselves have a chance to keep the paper from getting published.

Again, we can't just think about this case. What happens in the many cases where a good scientist attacks a lesser scientist's work -- should the lesser scientist be on the peer review panel, and thus have a chance to stop good science from being published?; 10:37 PM

The Buck Stops Here

Sunday, November 22, 2009

Peer Review

19 Comments:

About Me

Previous Posts