Nature News has a story about one of the most difficult topics in science: paper retraction. For any number of reasons, a published scientific paper are sometimes discovered to be so flawed that they must be withdrawn from publication (which, these days, mainly means that it disappears from online databases of the various scientific publishers).
According to the article, the number of papers being withdrawn has risen dramatically over the past few years, though from a very low base: from about 0.001% of all scientific papers, to about 0.02%. However, it’s pretty doubtful that this represents the true extent of papers that contain flaws of such severity that they should be withdrawn from circulation. Even if you believe that scientific fraud is that uncommon, honest mistakes do occur on a pretty regular basis, and peer review doesn’t always pick them up – I’ve personally seen this from both sides of the ledger.
But given that withdrawals (and presumably corrections, too) are becoming more common, there are questions about how they are best handled. Will full disclosure of the reasons for paper withdrawal, or allowing quiet withdrawals without stating the reasons, result in a scientific corpus with less fraud in it? The article’s author comes down on the side of full disclosure, but I’m not entirely convinced. Would insistence of full disclosure encourage the withdrawal of papers where the issue was authorship disputes, for instance?
If you’re interested in the area, Retraction Watch follows retractions and how they are handled across the scientific press.
Personally, I don’t see why formal retraction is required at all. There are any number of papers in the scientific literature – indeed, I have been responsible for at least one – that subsequent research has shown are in whole or in part wrong. Such papers merely get quietly ignored, and research gets progressed through the alternative paths that the subsequent research has opened up. The flawed papers once the flaws have been founddon’t mislead anyone by their continued existence, since anyone working seriously in the field concerned will of necessity have to acquire a good enough grasp of the literature to know they have been shown to be wrong.
If the issue is really fraud or fabrication of evidence, not just flaws from honest error or incomplete results – and the Nature article doesn’t seem too sure whether that is what it is actually getting at or not – I am not even sure that that makes a difference. Fraud of course is far more serious, and needs to be dealt with, but penalties through employers, professional associations, naming and shaming, etc, which actually hit the fraudster seem likely to be more effective than simply disappearing a paper.
Wozza, I think retraction is important in fields where decisions made on the basis of the paper might reasonably affect human life. In health these issues are very important. This blog post, for example, points out the possible consequences of acting on the recommendations of a study about cesarean section that was published in the Lancet and later received a significant edit (though no retraction).
I guess it’s not just the retraction itself but the message it sends that policy decisions based on the results need to be rescinded or reconsidered.
This is very different to when a paper is written on the basis of current best understanding and then later scientific investigation shows it’s wrong. Retraction usually only applies when the errors are egregious or something deliberate has been found.
I’m guessing a lot of the increase in retraction rates has to do with the way that pharmaceutical and tobacco company funding is deliberately hidden or obscured by authors. I recently read a classic example of this kind of process in action.
Scientific papers are a lot more accessible these days too (a good thing). So as sg mentioned its more important that there are formal retractions. And I think it will also increase confidence in the scientific process if people admit when they’re wrong and explain why. Less scope for conspiracy theories to grow
Precisely, Chris. If it turns out that a paper contains a serious, out-and-out error, it’s important that that error is acknowledged so that the mistake doesn’t get propagated.
Wozza: As you say, serious researchers will be aware when results and conclusions in their field that have been challenged or withdrawn. However, in many cases, the data and conclusions in these papers will be used by people who are outside of the in group.
For these people, knowing that a paper has been withdrawn or modified significantly is important information. Journals should have a list of withdrawn/modified papers. Links to withdrawn papers should give a paper withdrawn/modified message to anyone who uses the old link. Where appropriate, the message should highlight any conclusions that are no longer considered reliable.
You are also right to point out that the conclusions/data in older papers may no longer be reliable. In these cases it is undesirable to withdraw papers since it is important to be able to see at times how thinking, procedures etc. change over time. However, there may be a case for having an “outdated” message when later papers have led to changed conclusions.
Scanning Retraction Watch, a couple of things seem apparent. Formal retractions take quite some time – years. No doubt this is inevitable as checks are made and processes followed. But frankly the horse has bolted in terms of “so the mistake doesn’t get propagated “ by that time. Those working in the field will be aware from developments in the literature that the work is suss, and ignore it, well before retraction happens. I mean, who discovers the suss-ness anyway? Other scientists in the field of course.
The other is, as one post there puts it, “it comes back to this dual function of retractions: cleaning up the literature versus pointing the finger to those who have committed misconduct.” The post suggests that the former isn’t worth it. The real issue is exposing and preventing actual fraud. Many of the examples of retraction on Retraction Watch are in about serial fraudsters. This where the attention should be focused.
Look, if journals want more formal retractions as part of ensuring flaws in papers are noticed, I have no objection. The more flaws exposed the better, obviously. I just think that this could require a lot of effort for little practical effect. At least in the areas I am familiar with, the approach of minimum fuss, papers shown to be wrong fall by the wayside naturally in terms of losing the attention of researchers through ordinary literature survey processes, works pretty well in ensuring that mistakes don’t get propagated in ways that damage ongoing research.
One exception: we should be very cautious of sg’s “a lot of the increase in retraction rates has to do with the way that pharmaceutical and tobacco company funding is deliberately hidden or obscured by authors.” Funding has got nothing to do with it. Flaws in papers must be assessed only to objective criteria of scientific soundness. One of the reasons I am instinctively concerned about increasing attention on retraction is the potential for non-scientific judgments to be brought to bear. Some of the ethics committees who exercise more and more influence at the start of the research process are already arguably impacting deleteriously on ability to research properly and objectively.
sg I am not of course accusing you of advocating retraction or non-publication of papers for reasons of political correctness, merely saying that there are dangers if Pandora’s box is opened. A field that we often spar about is an example of where there is already an impact of this sort, imho.
I’m with Wozza. A Scientific paper can be enduring if they report a new method for finding answers to a problem, even if the particular answer in that paper becomes superseded by later evidence. And aside from this, it can be useful and important to understand the evolution of a scientific literature. Science proceeds by accepted truths being eventually overturned or qualified or refined.
Einstein was famously wrong on a lot of things, as it turned out. But that doesn’t mean we want to suppress his papers.
Retraction Watch is interesting, so thanks for the link.
I reckon the only papers that should be withdrawn are those that should not have been published in the first place:
- Based on fraudulent data, cherry-picked and manipulated experiments
- Systematic plagiarism
- Zombie science (already invalidated methods and findings) waved through by sympathetic reviews
Honest mistakes – many of those on RWatch are such. Authors could request they be withdrawn, of course. One paper recently withdrawn did not have an author’s consent. Given that a form must be signed to give consent, this is misleading behaviour by the principal author.
Re funding. Wozza, science is a socially-mediated process. Scientific objectivity that links theory, observations and experiment is not absolute but is restricted to local conditions (the philosophy always founders on one paradox or other). Probative values (scientific proofs) are context specific, change over time and have a social dimension in that a community of scholars approves or disapproves of specific methods. Making funding visible, along with data, makes the process more public. It’s about maintaining trust, not revealing influence. Non-scientific judgements always accompany controversial projects – this is one way of managing it.
Fair enough, Roger and Sam.
That said, it’s hard enough to get journals to publish corrections sometimes.
As far as I know, Wozza, the vast majority of retractions are because of fraud and the like.
Regarding the role of competing interests and hiding them, I’m sorry but funding really has a lot to do with it, and there is a very good reason why medical journals demand statements of competing interests. It’s not just so that people can use ad hominem arguments. I’ll give an example in my next paragraph (and it’s a doozy, for all that it’s irrelevant); but before I do I should point out that you’re more than happy to accuse Muller (in that other field we stoush about) of talking up his papers for commercial advantage: pot, kettle, etc.
Now, for an example consider this paper in PLoS One that was presented by a student of my university at our journal club. It’s by Roger Bate, from the American Enterprise Institute (AEI), which is pharmaceutical company funded; no big deal, he states this upfront and makes clear that his fellowship is not so funded. The flaws in his analysis are clear to anyone reading, so as you say the flaws in the paper can be assessed by “objective criteria of scientific soundness.” You can look up Bate’s work online and find that he advocates on behalf of big pharma for laws cracking down on counterfeit drugs; no problem, we know he’s got interests in this field because the paper clearly states so. Any flaws in the analysis stand on their own merits, right?
Except that they can’t, because there’s something wrong with the randomization process. The data is not representative. Why? The clue is in the acknowledgements. The data is not properly randomized; the random walk process by which the data was collected started in areas that were chosen by an “expert.” The expert is not an author and is not specified in the paper; but in the acknowledgements we see that it was one Suresh Sati. He chose the areas for randomization. Do a bit of research on Suresh Sati and you will find that he runs an intellectual property business and is paid by pharmaceutical companies to hunt down counterfeit drugs in India. His choice of sample areas is highly likely to be biased, but this fact is not described anywhere in the paper. The authors are presenting their findings as representative (because randomized, objective) but in fact they’re probably a massive over estimate of the true prevalence of counterfeit drugs in India.
We would know this if Sati were registered as an author and his business interests revealed, but it was buried in the acknowledgments. This study is pretty much useless as an indicator of the magnitude of the problem of counterfeit drugs, but it’s a pilot study demanding more research on the same problem, by someone who is paid to advocate against the problem by pharmaceutical companies, with a carefully hidden process of sampling bias. No one would know about any of this if Bate had not declared his interest.
Do you see the point? These issues are not just some kind of left-wing beat up. There are real reasons, learnt the hard way, why this sort of stuff is done in medical journals (think of a certain Andrew Wakefield if you aren’t convinced by the PLoS example).
sg and Roger, fair enough, I agree that science like anything else can’t be divorced from its context. It is a question of balance and where lines are drawn, and as sg suggests (I think) that is difficult to generalise about in the abstract; the devil is always in the detail of specific cases.
What a love-in this is turning out to be. Not what I am used to at all.
If we want to get a little more heated, one thing that does shit me in discussions of this sort is that so often there are assumptions that funding from big tobacco, big oil, big pharma etc needs to be regarded very suspiciously as influencing the objectivity and publications of those that receive it, whereas funding from big Government, no problem, pure as the driven snow, Government never prosecutes agendas through its funding power. Frankly, if anything the reverse is true. At least business is concerned only with profits, not actively promoting ideology.
But I suppose no-one has actually said that explicitly on this thread, so far be it from me to urge us O/T.
I’ll engage that, Wozza… grant funding has to be declared in a publication just the same way as any other funding. Not only that, but regardless of the source, the role of the funding source has to be disclosed – if they have a contract with you that stipulates their approval or oversight before publication, that has to be declared. Most American journals have additional special rules for US government funded papers (mostly to do with copyright, I think?) Check out the Lancet instructions for authors or the BMJ’s disclosure rules if you’re uncertain about that.
And bear in mind that a lot of research into new drugs that is published is funded by pharma. They aren’t the enemies of good research; they just need to be very very carefully monitored.
“According to the article, the number of papers being withdrawn has risen dramatically over the past few years”
Robert, if you subtract Thompson Reuters’ figures for China and India, the number of retractions per papers published appears to have actually decreased on average these last 10 years.
http://en.wikipedia.org/wiki/Scientific_plagiarism_in_India
Note that one professor alone ‘plagiarised and/or falsified’ 70 papers between 2004 and 2007. That’s about 70% of India’s retractions for those years, assuming they’ve all been retracted.
Google or search Nature for scientific plagiarism in China.
In Van Noorden’s words, “With such small numbers, repeat offenders with a string of retractions can quickly alter the balance from year to year.”
I’m going to come down on the side of Woz as well (has hell frozen over?). I think the naming and shaming should be reserved for the cases that Roger outlined, where significant scientific misconduct has taken place.
Honest mistakes tend to get pointed out by later papers anyway – and if you find someone else’s published mistake the usual thing is to get a publication out of the journal yourself by publishing a paper correcting this. I think this is ok, and gives more of an insight into why the mistake was made (since you can go into detail) than a simple retraction does. Other researchers will be able to pull up a series of linked papers online anyway, so this seems to be the best way.
Jess – why does withdrawing of a paper have to be about shaming? It’d be no surprise that people who made honest but significant errors would be very hesitant about withdrawing a paper if that is the perception.
Isn’t there a benefit from having scientists being able to publicly and officially nadmit when they make mistakes without any shame attached? Rather than just plod along like nothing is wrong and when someone queries it say “well everyone important in the field knows that wasn’t right!”.
Chris: Maybe withdrawing a paper doesn’t need to be about shaming, but in the current scientific context it’s certainly a pretty big stigma. As far as I can see, we could try to stamp out the stigma, or we could stay with the status quo – that is reserving withdrawal for egregious breaches of scientific conduct, and additional publications for honest errors.
It took twelve years for Wakefield’s paper to be fully retracted from The Lancet.