Wednesday, January 4, 2017

Who is responsible for bad science: researchers, peer reviewers or editors?

The following is partly a case study of how research can take a wrong turn in modern molecular biology/biochemistry and partly a reflection on the sociology of the science.

In the current scientific culture it is considered a big deal to publish in particular “high profile venues”, such as the Science and the Nature magazines. We can attest based on personal experience as well as reports from our peers that publication in these venues often entails enormous difficulty to the researchers involved on account of the peer review practices at these venues. Thus, pushing a paper through to these venues can earn one a virtual badge of a survivor or even hero of a great battle. This badge confers tangible benefits in the modern scientific system: 1) the tenure decisions of scientists at many institutions is favorably influenced by such magazine-publications. 2) The scientific productivity reviews of researchers at many institutions often receives a great boost from such publications. 3) A scientist stands to get press, awards, and higher visibility (as measured by citations) – more generally “fame” from such publications. 4) Last and perhaps most importantly in academia, such publications could be a big factor in earning grant money for continuing research.

Unfortunately, this system of incentives makes the magazine-publication an end in itself, ahead of the actual science. In the below case we illustrate how this, together with the system at the magazines, can engender bad science.

Recently a paper was published in the Science magazine entitled: “A nuclease that mediates cell death induced by DNA damage andpoly(ADP-ribose) polymerase-1”. Given our interest in novel biochemistries and our long-standing investigations of self-inflicted nucleic acid damage in programmed cell death our interest was piqued by this paper. A closer look soon revealed that the paper had problems. Briefly:

MIF, a protein with previously reported tautomerase activity, is a member of the tautomerase superfamily which does not feature nucleases. The authors suggest that MIF is a DNase by claiming a structural relationship to nucleases of the Restriction endonuclease (REase) fold, which frequently but not always contain a PD-(D/E)XK motif. They claim that MIF contains three copies of this motif implying that it contains three copies of the REase fold. However, none of this is supported by structure or sequence evidence: 1) A DALI search with the structure of MIF does not recover any REase fold structures with Z-scores suggestive of genuine relationships (Z>3); as expected it recovers several tautomerase superfamily structures. 2) The REase fold is topologically unrelated to the tautomerase fold, which emerged from an internal duplication of a simple two-strand-helix unit. 3) REase fold catalysis requires residues (not conserved in MIF) beyond the metal-coordinating acidic residues of the PD-(D/E)XK. Moreover, the aspartates and glutamates identified by the authors are often on opposite ends of different structural elements and not proximal to coordinate a metal ion. 4) These motifs, unlike the catalytic prolines of the tautomerases, are not well-conserved even among animal orthologs.

Further, the authors claim glutamate 22 to be the catalytic residue equivalent to that of the so-called Exonuclease-Endonuclease-Phosphatase (synaptojanin-like) domain (Pfam: PF03372). This fold is unrelated to both REases (so-called PD-(D/E)XK) and tautomerases. Their claim that MIF contains a “CxxCxxHx(n)C Zinc finger” (the Rad18 Zn-finger present in FAN1/KIAA1018) is also untenable as the side chains of these residues are nowhere proximal to coordinate a Zn-ion in MIF. Hence, this proposal of DNase function in MIF is based on a flawed structural hypothesis and should be viewed with utmost caution. We have explained the above in great detail in this preprint available from bioArxiv.

Given the serious problems with this published paper we decided to resort to the standard process of post-facto scientific engagement. Within 10 days of reading the said paper we submitted a technical comment along the lines of the above-linked preprint to the Science Magazine (10/26/2016) detailing why the paper is problematic. We believe that this was important because the extraordinary claims made in the article flew in the face of the foundations of biology, i.e. the evolutionary theory, and we felt this should be made apparent to the research community. After more than a month (12/6/2016) we heard back from the editor at the Science Magazine who had handled the original article. Despite all that time spent we received no peer reviews of our technical note. Rather the editor decided that even though “your discussion of our recent paper is interesting” it was not worth publishing. However, the editor suggested that we submit a short summary of our note as an “eLetter” (which allows only 500 words; similar to the summary we provide above). These are non-peer reviewed comments that are posted below the article on the Science Magazine website at the discretion of the editor. They are neither visible in any obvious way with the article nor are they available on Pubmed, which is the standard resource used by researchers to find literature of relevance. We followed the suggestion of the editor by posting an “eLetter” (12/9/2016); eventually, the editors decided to post our comment on the magazine website (12/20/2016). Thus, almost 2 months after the original submission some form of dissenting commentary appeared on the magazine.

So what are the lessons to learn from this story regarding bad science? We see that there are three parties, each of which has to be blamed for different even if somewhat overlapping reasons:

1) The authors. They are to blame because they have utterly disregarded the foundations of biology in planning their experiments. In any other science, like physics, a researcher is unlikely to have any chance of being accepted as a serious scientific player in the community if s/he were unaware of the basic foundations of the science, like say classical mechanics, leave alone publish in the Science Magazine. However, unfortunately, in biology several researchers can spend their publishing career without having any more than a sketchy grasp of the foundations of biology, i.e. the evolutionary theory, and how it is applied to study the functions of biomolecules. This is exactly what we see with this paper. The authors blithely disregard very basic principles of protein structure and sequence evolution to form their starting conjecture which they then go to support with wet lab experiments. Now, if the wet lab experiments supported their results then we have reason to be suspicious of the way they were done – contamination, poor controls, or even worse, some kind of unscrupulous practice. There are indeed suspicious features regarding the experimental result of nuclease activity as pointed out in our preprint.

If this were not enough, the authors responded to our eLetter on 12/24/2016 as can be seen on the Science website. This response betrays not just a lack of understanding of the evolutionary theory but even more fundamental issues. Though the authors aligned a monomer to claim the presence of the so-called “PD-(D/E)XK” motif in MIF in their original paper, now they claim that the REase fold is seen only in the trimer of the Tautomerase fold! This is not merely an about-face regarding their original inference but indicates an even deeper lack of an idea of what comprises a protein fold. In conclusion, based on this all we can say is that they are either visually challenged or lack discernment regarding elementary geometric issues such as symmetry.

2) The reviewers. The Science Magazine helpfully provides the following details regarding the review of this paper: It was handled by a single editor who had it reviewed by three full reviewers and one advisor by the single blind method. The whole process took from 11/02/2015 to 8/22/2016 with two rounds of review prior to acceptance. This is an extraordinary overkill both in terms of time spent and number of reviewers for an article that should have been rejected outright in the first round of review itself or by the advisor if such were consulted by the editor before formal review. Why did this not happen? Given the point we make above, we fear that sadly the reviewers too, like the authors, lacked the basic qualification i.e. sound knowledge of the implications of the evolutionary theory as applied to biomolecules. Instead, as is typical of such magazine venues it appears that the reviewers sent the authors around for almost 11 months on a wild-goose chase of doing more experiments which are utterly worthless given that the starting premise itself is flawed. What this also shows is that at magazine venues reviewers are wont to giving trouble to authors for irrelevant things while not really focusing on the key scientific issues in the paper. It is a certain mentality which is sadly not uncommon among wet lab scientist where technical issues take center-stage before asking whether the science could meaningful (and useful) or not.

3) The Editors. Given the vastness of the field and literature and the technical expertise needed for things like sequence/structure analysis we would not blame the editor overly for missing the bad science in the original submission. Nevertheless, at venues like the magazines and high-profile journals the editors are typically former scientists themselves. Hence, we would expect that in the least they would have some quick intuition for good versus bad science. From personal experience we can say that editors at such venues often fail to see merit in genuinely good and interesting science submitted to them, while taking bad pieces like this one. Thus, we may say that the intuition for discriminating between submissions might be weak among the editors. However, even more damaging is their failure to get proper peer reviews as doorkeepers of these magazines which, as noted above, play an important role in the career of researchers. Finally, we believe that the editor’s unwillingness to properly publish a technical comment that reveals why the article amounts to bad science amounts even further damage to science. Although the editors did finally post an eLetter from us, as noted above, this is not visible via the regular channels of literature search. Hence, no corrective is available to go hand in hand with the original article. Thus, this move on part of the editor strikes against the much valued “self-correcting process” that exists in science.

In conclusion, while we are not involved in any kind of design of science policy, we still have a few recommendations to make. 
  • First, relates to biology education. Modern biology education necessarily needs to go hand-in-hand with proper teaching of the evolutionary theory as it applies to biomolecules, along with the accompanying biochemistry that is needed to properly understand it. Technical skills with handling laboratory equipment and experimentation, however important, cannot be privileged over such education in the above-stated fundamentals. 
  • Second, the scientific status-measuring apparatus needs to go slow on emphasizing publications in magazine-like venues as a “badge of honor”. Publications at such venues are short, thereby giving little space for detailed scientific description that helps develop a foundational argument properly and spot flawed ideas. Their peer-review system is aimed more at “causing sufficient trouble” rather than providing honest peer comments on the science.
  • Third, the members of the editorial system should come out of the echo-chambers fostered by certain “big-name researchers” or artificially constructed “hot science” and pay independent attention to the literature from diverse journals in their respective fields.
Finally, in case a reader were to think that this is an example of us making much ado about nothing or that we are sensationally drumming up an isolated case, then all we’ll say is that this just the tip of the proverbial iceberg. Right in this venue we hope to briefly bring up more examples as and when time permits. One might also look at an earlier paper we had written on this topic. Apparently, things have not entirely changed in all those years.