V  & V.  That’s shorthand in project design for “validation and verification.”  Does the scientific method provide V & V?  We are all taught to think that peer review, publication and replication help science to be self-checking, so as to avoid error.  Some recent articles show that ain’t necessarily so.  It may sound good in theory, but in practice, the ideal doesn’t always match the real.

Publish and perish:  In Nature (480, 22 December 2011, pp. 449-450, doi:10.1038/480449a) Adam Marcus and Ivan Oransky reminded readers of the world’s premiere science journal that in science publishing, “The paper is not sacred.”  Peer review needs to continue long after a paper appears in print, they argued.  Their concern was prompted by a 15-fold number of retractions over the last decade.  During the same time period, papers increased by 50%.  This is not necessarily bad, Marcus and Oransky continue, because it indicates corrections are being made.  But what about bad papers that don’t get retracted?  They pointed out disturbing cases where peer review was poorly checked by journal editors, sometimes with “massive” numbers of errors in a paper, under the excuse that peer review is supposed to be secretive.  Often readers are given no explanation for a retraction other than, “This paper has been withdrawn by the authors.”   Notice how extensive the problem is in their words:

Editors have many reasons to pay more attention to retraction and correction notices. For one, scientists often cite papers after they’ve been retracted, and a clear, unambiguous note explaining why the findings are no longer valid might help to reduce that. But, more importantly, a vaguely worded note that includes further claims from researchers whose work has been seriously questioned, in turn raises questions about the integrity of the journal itself, and about the overall scientific record.

Marcus and Oransky pointed to new online methods that might reduce the number of mistakes making their way into the corpus of “scientific knowledge”—even the radical idea that the new methods may reduce the publication of scientific papers in journals.  But their article raises other serious questions.  Since World War II we have been led to believe that peer review provided the V & V science needed.  How do we know that new, untested methods will do better?….

Continue Reading on crev.info