Often the exalted scientific and medical journals sitting atop the impact factor pyramid are considered the only publications that offer legitimate breakthroughs in basic and clinical research. But some of the most important findings have been published in considerably less prestigious titles.

Take the paper describing BLAST—the software that revolutionized bioinformatics by making it easier to search for homologous sequences. This manuscript has, not surprisingly, accumulated nearly 30,000 citations since it was published in 1990. What may be surprising, however, was the fact that this paper was published in a journal with a current impact factor of 3.9 (J Mol Biol, 215:403–10, 1990). In contrast, Nature enjoys an impact factor more than 8 times higher (34.5), and Science (29.7) is not far behind.

One of the most commonly voiced criticisms of traditional peer review is that it discourages truly innovative ideas, rejecting field-changing papers while publishing ideas that fall into a status quo and the “hot” fields of the day—think RNAi, etc. Another is that it is nearly impossible to immediately spot the importance of a paper—to truly evaluate a paper, one needs months, if not years, to see the impact it has on its field.

In the following pages, we present some papers that suggest these two criticisms are correct, at least in part. These studies were published in lower-profile journals (all with current impact factors of 6 or below), suggesting they should have had less of an impact. But these papers eventually accumulated at least 1,000 citations. Many were rejected from higher-tier journals. All changed their fields forever.

P.A. Zuk et al., “Multilineage cells from human adipose tissue: Implications for cell-based therapies,” Tissue Eng, 7: 211–28, 2001. Times cited: 1,175

P.A. Zuk et al., “Human adipose tissue is a source of multipotent stem cells,” Mol Biol Cell, 13: 4279–95, 2002. Times cited: 1,010

Findings: Fat contains pluripotent stem cells.

Impact: More than 25 clinical trials have occurred or are in progress that use fat-derived stem cells to treat a wide range of indications, including diabetes and cardiovascular disease.

Not long after Patricia Zuk began as a postdoc in August 1999 in Marc Hedrick’s lab at the University of California, Los Angeles School of Medicine, “Marc tossed this manuscript on my desk and said ‘Fix this,’” she recalls. The manuscript had to do with a population of cells in adipose (fat) tissue, and based on the work he had conducted with his colleagues at the University of Pittsburgh, Hedrick suspected they could be multipotent stem cells. But clearly the evidence that had raised his suspicion wasn’t strong enough to warrant publication—the manuscript had been rejected.

The problem, Zuk explains, was that “people just auto-assumed the cells were just committed preadipocytes”—if you put the cells in culture, you would get fat. But, Zuk adds, many diseases cause fat and other tissues to calcify—in other words, turn into bone—suggesting fat contains at least some cells capable of differentiating into other tissues.

So Zuk, along with another recently hired postdoc, Min Zhu, took on the responsibility of more fully characterizing these cells. Using fat samples obtained from liposuction procedures, the researchers ran a series of molecular and biochemical tests—expression assays, immunofluorescence experiments, flow cytometry tests, and more—and concluded that the cells were indeed multipotent and capable of differentiating into a variety of different lineages, including fat, bone, muscle, cartilage, and even neural tissues.

“Fat is definitely an underappreciated tissue source,” Zuk says. In addition to being far easier to obtain than bone marrow—the most commonly used source of mesenchymal stem cells—there’s usually a lot more of it to spare, she adds. “There is no tissue in the human body that is as expendable as adipose tissue.” Indeed, the tissue has proven to be a useful clinical application since the publication of these papers, with more than 25 clinical trials completed or in progress for a wide range of indications, including diabetes and cardiovascular disease.

Once the lab members felt they had a strong enough case, they sent their manuscript out to Cell, but it was immediately rejected. “They obviously didn’t think it was high impact enough because they didn’t even send it out for review,” Zuk says. They tried again at Tissue Engineering, and this time they were successful. A follow-up paper was published the following year in Molecular Biology of the Cell.

“There was just something about the papers that really caught everybody’s attention,” says Bruce Bunnell of Tulane University School of Medicine, who has cited the two papers 16 times in his own work. “Those two papers have become [the go-to papers] in the field of adipose stem cell research if you need to reference the basic biology.”

“We were lucky enough to be the seminal paper on this topic,” Zuk says. “When I think hematopoietic [stem cells], I think of McCulloch and Till; when I think of the mesenchymal stem cell population, it’s Friedman, 1969.” Now, she muses, “I guess I’ll be permanently associated with fat.”

S.P. Gygi et al. “Correlation between protein and mRNA abundance in yeast,” Mol Cell Biol, 19:1720–30, 1999. Times Cited: 1,607

Finding: It is not possible to infer protein levels in a cell by measuring RNA transcripts, an easier technique than quantifying proteins directly. This affirmed why the field of proteomics should exist.

Impact: The search term “proteomics” retrieves more than 24,000 papers on PubMed.

For 2 years, researchers at the University of Washington toiled away, running gel after gel to isolate, label, and count the proteins in yeast. The idea of a “proteome” was still a new one in the late 1990s, but throughout the decade, researchers had developed better and better techniques to measure the amount and type of proteins in a cell.

Two years earlier, in 1997, a team at the Johns Hopkins University School of Medicine had published the yeast transcriptome—the set of genes expressed from the yeast genome (Cell, 88:243–51, 1997). Ruedi Aebersold, then a biologist at the University of Washington, realized he finally had all the tools to answer a pressing question: Do RNA transcripts directly correlate with protein levels in a cell?

If so, it would be good news to the research community, since technologies to measure RNA transcripts have always been more advanced and easier to use than those to quantify and identify proteins, more biochemically complex molecules. But to use mRNA levels to predict protein levels, researchers had to assume there was a direct correlation between the two. Unfortunately, they had long suspected that wasn’t the case, recognizing that there are a host of post-transcriptional events that control protein translation and degradation rates, skewing the ratio of the two sets.

To set the suspicion to rest, Aebersold and postdoc Steven Gygi spent years using high-resolution, two-dimensional gel electrophoresis to separate proteins in yeast cells, then excised and identified them using mass spectrometry and database searching. Finally, they compared their data with the mRNA levels of the 1997 paper, and found a very poor correlation between protein and mRNA levels. For genes with equal mRNA levels, protein levels varied by more than 20-fold. For proteins with equal abundance, mRNA levels varied by as much as 30-fold. “I personally wasn’t surprised,” says Aebersold, “but as is always the case, we wanted to show it with data.”

The team submitted the paper to Molecular and Cellular Biology only, since half the data had already been published and it was such a new field. It went right in, recalls first author Steve Gygi. The results were published in March of 1999. Aebersold received nice comments from colleagues, he recalls, who were pleased to finally have data confirming their belief that there was no strong correlation between the two data sets. With one paper, they singlehandedly illustrated why the field of proteomics needed to exist.

“This is a very important paper,” says Matthias Selbach, a proteomics researcher at the Max Delbrück Center for Molecular Medicine in Berlin. “Many people from the proteomics field like to cite this work. They all want to show that protein levels have nothing to do with RNA levels, so then they cite this paper to make [the point] and to justify why they are doing proteomics rather than transcriptomics,” says Selbach. A range of subsequent studies analyzed the same correlation in other cell types, always with similar conclusions.

“It helped my career a lot,” says Aebersold, who went on to continue working in the field of proteomics and cofounded the Institute for Systems Biology in Seattle in 2000. Today he is a professor at the Institute of Molecular Systems Biology at ETH Zurich in Switzerland. “Proteomics was a fringe field for a long time,” says Aebersold. “Compared to 10 years ago, there’s been pretty amazing progress.”

JE Meredith et al., “The extracellular matrix as a cell survival factor,” Mol Biol Cell, 4:953–61, 1993. Times cited: 1100

Finding: The extracellular matrix prevents programmed cell death, sparking a new field that another paper termed “anoikis.”

Impact: Using “anoikis” as a search keyword retrieves more than 700 articles on PubMed.

Most field-changing observations are completely serendipitous. The discovery that a cell’s external environment is essential to its survival—as in, without it, the cell undergoes apoptosis—was no different. Martin Schwartz, then at Scripps Research Institute in La Jolla, Calif., left endothelial cells he was studying in suspension overnight. When he realized his mistake, the cells were dead, and they had formed prototypical “bleb” shapes—the hallmark of programmed cell death, a trendy concept at the time.

Schwartz hypothesized that the extracellular matrix (ECM) contained elements that protected cells from programmed cell death, and he planned a couple of straightforward experiments to confirm it. He had his new postdoc, Jere Meredith, repeat his “accident” and confirm the cells were undergoing apoptosis. Meredith, now at Bristol-Myers Squibb, remembers it as one of the most satisfying experiments of his career. “Everything just seemed to work,” recalls Meredith. “It was kind of neat because science never seems to work that way.”

And like many other science stories, another lab was unknowingly pursuing the same project—across the street from Schwartz’s, in fact. “We could see each other’s buildings,” says Steven Frisch, then at the La Jolla Cancer Research Foundation, now called the Sanford-Burnham Medical Research Institute. “And neither of us knew what the other was doing.” He had also discovered the role of the ECM in preventing apoptosis accidentally, while doing experiments on the E1A adenovirus gene, which he says can convert human tumor cells into normal epithelial cells. When plating tumor cells into soft agar, which prevents cells from forming an ECM, he saw that all cells with an inserted E1A gene disappeared. Frisch reasoned that the cells with the E1A gene were converted to a normal epithelial phenotype, making them dependent on the ECM for survival. But the cancerous cells did not depend on the ECM for survival, enabling them to spread around the body, or metastasize. Additional experiments supported his hypothesis.

Recognizing the importance of his finding and what it could mean for cancer research, Frisch sent his manuscript to “some of the very top-tier journals,” he says. “It kept getting rejected.” Meanwhile, Schwartz, now at the University of Virginia, says he did not know of Frisch’s work, but had heard about a similar project by another scientist at a meeting, and knew he needed to publish fast. Plus, the finding was relatively “simple,” he says, with no mechanism, and higher-impact journals “don’t publish ‘simple’ papers.” They sent their manuscript to only one journal—Molecular Biology of the Cell, then a new publication, but with a reputation for reviewing papers quickly. It was accepted with just small changes, and published a few months after submission. Meanwhile, Frisch’s paper was trapped in the review process, and didn’t appear until 5 months later, in the Journal of Cell Biology (124:619–26, 1994), complete with a citation of Schwartz’s paper. “I didn’t know about Steve Frisch’s work until it was published,” Schwartz says. Both papers showed that the lack of the ECM induces apoptosis, although Frisch gave it a name—“anoikis,” or “homelessnes” in ancient Greek. (He originally named it “homelessness,” but the journal nixed it. After calling a Greek restaurant and ancient Greek scholar, he settled on “anoikis.”)

The findings had obvious implications for cancer research, but additional research has also shown that anoikis is critical in various phases of embryogenesis, and may even play a role in neurodegenerative disease.

Frisch says he “obviously” wishes he hadn’t tried to publish his paper in a top-tier journal, so he could have been the first to describe anoikis. But his publication has accumulated more than 1800 citations, and when the field caught on to the importance of anoikis, he was asked to write reviews and speak about his work at more conferences than he could attend. “For a while, our lab was pretty well known,” says Frisch, now at West Virginia University School of Medicine. To this day, when the journals that rejected it asked him to write articles about anoikis, he reminds them that they rejected the first paper to describe it.

A. Carr et al. “A syndrome of peripheral lipodystrophy, hyperlipidaemia and insulin resistance in patients receiving HIV protease inhibitors,” AIDS, 12:F51–58, 1998. Times Cited: 1,234

Finding: HIV-positive people receiving protease inhibitors develop a syndrome of acquired lipodystrophy.

Impact: The terms “HIV” and “lipodystrophy” bring up more than 1900 papers on PubMed.

In the mid 1990s, science and medicine began gaining ground on the recently characterized HIV virus, and patients were taking new medications that had a dramatic impact on the disease. But with these new treatments came unforeseen side effects.

Andrew Carr, of the University of New South Wales, was hearing complaints from his HIV patients who were receiving new drugs called protease inhibitors that prevented viral replication. Carr says that many came to him to ask why their limbs seemed to have shriveled, the veins on their arms and legs bulging. They were living longer than HIV patients in the previous decade, but also showing off-the-charts cholesterol measurements, and complaining of gaunt faces and protruding bellies.

Carr consulted Donald Chisolm, an endocrinologist at Sydney’s Garvan Institute of Medical Research. Chisolm told Carr it sounded like lipodystrophy, a generic medical term for the abnormal distribution of lipids around the body, a disorder that is normally either inherited or, more rarely, acquired.

“That’s when we decided to systematically survey patients,” remembers Carr. With the help of Chisolm and others, Carr conducted a cross-sectional study of nearly 150 HIV-positive patients, some of whom were taking protease inhibitors. The team submitted their findings—that protease inhibitors were causing a syndrome of acquired lipodystrophy that included high blood lipids, reduced body fat (especially in the extremities), a migration of fat to the abdomen and other areas, and insulin resistance—to The Lancet in the latter half of 1997.

Reviewers at the premier medical journal were unimpressed. “[The paper] got those sort of surprised reviews you get when something completely new comes up and you’re not sure whether you should believe it or not,” says Carr.

Early the next year, Carr and his colleagues took their findings to a large HIV meeting in the United States, and presented a poster positioned next to posters from authors of two recent Lancet papers showing an enlargement of the fat pad on the upper back and expanding abdominal fat in HIV patients. He says that conference attendees were lined up at the three posters 10 people deep, all day long. When Carr and his coauthors returned home from the conference, they submitted their paper to the journal AIDS, which had a fast-track publication process. The paper was accepted, and since its publication, it’s been cited more than 1200 times. Dozens of researchers around the world now devote their careers to studying the ancillary health problems that accompany HIV treatments.

“The whole complexion of HIV changed from a horrible, catastrophic, fatal disease to suddenly have to start worrying about metabolic problems,” says University of California, San Francisco, endocrinologist Morris Schambelan, who authored one of those early Lancet papers on lipid problems in HIV patients. “I think they did a service to the field by throwing that paper out there and letting us chew on it.”

“[The paper] played a major role in getting people to focus on the abnormal loss of fat that was occurring in HIV,” agrees Carl Grunfeld, University of California, San Francisco professor of medicine. “It was the paper that everyone took notice of and started the field” of studying lipid abnormalities in HIV patients.

But Grunfeld and Schambelan fault the authors for laying all of the blame on protease inhibitors, when the study subjects were being treated with a panoply of HIV drugs. “We now understand that the [symptoms] had different causes,” Grunfeld says. Carr agrees. “Attributing [the syndrome] to protease inhibitors might have been wrong.”

This misattribution, Carr says, led some HIV patients to alter their treatment regimens, which could have led to increased HIV transmission in some populations. “I suppose part of me feels guilty that I contributed a little bit to patients stopping their pills.”

S. Sakaguchi et al., “Immunological self-tolerance maintained by activated T cells expressing IL-2 receptor a-chains (CD25). Breakdown of a single mechanism of self-tolerance causes various autoimmune diseases,” J Immunol, 155:1151–64, 1995. Times cited: 2,069

Finding: A distinct cell population (Tregs) keeps the immune system in check.

Impact: The term “Tregs” brings up nearly 1500 papers on PubMed.

For over a decade, immunologists had poured their resources into searching for a cell that suppresses the immune system, only to have their hopes dashed. By the late 1980s, most had given up hope.

The thought that such a powerful suppressor cell could exist stemmed from the late 1970s, when researchers found evidence the body had a fail-safe mechanism to clamp down an immune reaction before it became too aggressive, speculating that it might prevent autoimmune diseases that killed normal as well as infected tissue. Researchers followed one lead after another to find the population of cells that was responsible. After a decade of work, with the best hypotheses proven false, most immunologists abandoned the field. Publication rates for suppressor immune cells dropped from some 1,300 per year in the 1980s to around 150 in the 1990s (Semin Immunol, 16; 69–71, 2004). “The suppressor field had its heyday and failed,” says Ethan Shevach from the National Institute of Allergy and Infectious Diseases, who himself abandoned suppressor cell research. “The field was dead.”

So when researchers led by Shimon Sakaguchi, then at Tokyo Metropolitan Institute for Gerontology in Japan, found a distinct population of immune suppressive cells, publishing their results proved challenging.

Their work demonstrated that a small subset of T cells studded by the surface molecule called CD25 kept the immune system in check. When Sakaguchi used an anti-CD25 antibody to deplete this population of immune cells, mice immediately developed a spate of autoimmune disease such as gastritis, arthritis, and adrenalitis. And when the researchers reinserted cells with CD25, the inflammation directed at healthy tissue was quashed.

With its clear clinical implications, Sakaguchi and colleagues submitted the paper to the Journal of Experimental Biology. They were rejected. “It was a bit of disappointment for us,” says Sakaguchi, now at Kyoto University. They decided to resubmit to the Journal of Immunology, where it was published.

“I was one of the biggest nonbelievers” in this suppressive cell, says Shevach. He read the paper critically as soon as the issue hit his desk and noticed that the antibody Sakaguchi used to deplete CD25-expressing cells came from his own lab. Sakaguchi had used the antibody correctly, Shevach says, and everything just “clicked” in his mind. Right away, Shevach set off to replicate Sakaguchi’s findings. “When we confirmed his studies, I certainly underwent a bit of a ‘religious’ transformation,” says Shevach, becoming a sort of “cheerleader” for the Tregs, which was the new name given to these suppressive cells.

What Sakaguchi’s paper had done that none of the earlier research had accomplished was to identify a reliable surface marker that enabled scientists to isolate and study the cell population in a controlled manner. “Immunologists are attracted by cell surface markers,” says Shevach.

With an identifiable subset available, researchers flocked back to the field. In the years that followed, T regulatory cells were also studied for their role in transplantation medicine (by suppressing the immune reaction that normally rejects transplanted tissue) and cancer biology (by suppressing the cells that could potentially attack and kill tumors). But researchers are still scratching their heads about the best methods for manipulating Tregs. “In the future, we must think more seriously about clinical application,” says Sakaguchi.

Have a comment? Email us at mail@the-scientist.com

It is expected that scientists will do extensive literature survey and cite publications mainly for their relevance - no matter how high or low the impact factor of the journals are. The relevance is established only after reading the paper published in the journal. This is the only kind of citation that can truly build and reflect the value of the impact factor of a journal.

An artificially high impact factor can be built up if citations are cited without reading of the articles or possessing reasonable knowledge of the contents of the paper. In such a case, impact factors are misleading values.

Recently I was compelled to do an in-depth analysis on each citing article of one of my papers, coauthored in 2006. I had to read each one to find why mine was cited, what had been cited, where my paper was mentioned and how many times it was mentioned. Fortunately (not really...) not many citing papers were there o read. To my surprise and dismay, more than often, my paper was cited for no good reason I could identify. I would only count it when a specific point related to our data/results were mentioned, being supported or dismissed, as a valid citation. And a few citing papers actually had NOT mentioned my paper at all, except for an entry in their "References" section.
Hey, I don't mind my citation numbers going up and up. But it's depressing that people who cited your paper didn't really care about the data/results you had sweated, bled and lost a girlfriend (or two) to get!
Then my boss told me that one of his early papers, cited for >500 times, credited as a classic and foundation to its field, was mostly cited inappropriately! That paper was only the beginning of his discovery. The most crucial evidence, including precise numbers and accurate mechanisms, was provided in the following paper, which had been cited for merely 10% of the number of the first paper.
Moral of my stories: being cited doesn't mean being read.

All of this post hoc justifying of how important certain papers are, and how this means they shouldn't have been published in the journals they were, is missing two very important points.

The first is that quite often no one knows what will be a hit - so submitting it to a perfectly appropriate journal (like the Altschuh et al. BLAST paper) and thereby raising its impact factor, is simple justice.

The second is that paradigms are most often most entrenched in the "more prestigious" (or possibly pretentious) journals - which means that anything challenging the status quo will get short shrift from those upholding same.

Face it: many papers in top tier journals get very few citations, while some in lower IF journals get a lot. No problem; world moves on....

When I cite a paper I generally search those papers which support my findings irrespective of their impact factor. In my opinion Impact factor is not very important thing to decide importance of a study.
Avnish Kumar
Lecturer, Dr. Bhim Rao Ambedkar University Agra, India

The comments of Barry Barclay, Walter Wolf, John Markuszka, and Ellen Hunt have been enlightening. The impact factor needs to multiplied by a fudge factor to arrive at a realistic figure. The message is that the quality of work will stand out in due course. Publication of papers depends upon the acceptance which in turn depends upon what the reviewers have said about them. We may look upon with hope that some day published work will get recognition even if published in less esteemed journals. This may come posthumously and there ought to be no sulking on the part of scientists whose work got mangled in this exercise! After all, past is past!!

Nirmal Kumar Mishra
Retd. University Professor of Zoology,
Patna University, Patna

The fact that high quality papers are not published in particular journals does not reflect the defects in the peer review system. In deed, the papers identified are published in journals respected in their fields. The problems lie with the defects in 'rating journals' by simplistic 'impact factors,' and the use of defective compilations by "authorities" such as granting agencies and promotion & tenure committees. This is not a new problem. It has been seen for decades in 'rankings of colleges and universities' or almost any other simplistic classification system. There is no fundamental problem here: good papers were published in good journals.

Years ago I sat down with Madelbrot to talk about some thoughts on fractals and gene transcription. He was still bitter about how his papers had to be published in out of the way journals. And yet, there he was, a living icon, sitting with me. I thought, "How sad. What a waste of energy, time and emotion. Why does he bother with it?" And I vowed that I would not do that to myself.

I think this is analogous to the feelings people have over divorce. Some hang on to bitterness 20 years later, like it was yesterday. Such a waste of life, and then that precious life is over and what has consumed it? Bitterness over injustice. Why compound injustice by allowing it to turn one inside out, to end in a virtual gutter of one's own mind, bitterly bemoaning long ago events?

I rarely read articles just because they are published in leading journals anymore. I read articles from services I subscribe to. And I find articles using web searches now. I subscribe to a number of online journals and services. I stopped subscribing to Nature. It's not worth it. Science I still get, and skim. But that's it.

I refrained from posting a comment regarding this article for sometime, but my heritage got the best of me, so here goes.

First, what is the difference if this work was published in Science or Nature versus the quality journals in which it was published? Would the citations be any greater? I doubt it, good work is cited no matter where it is published.

Second, this article makes the supposition that all work published in Nature or Science is outstanding, ground breaking work. I am not sure that this is the case and certainly these and other high profile journals have had a fair number of retractions in the past years. If the process was so refined at these high impact journals, why so many retractions?

Third, this is a dangerous article for young scientist and tells them to place an erroneous high value on impact factors. It is important first to understand what is an impact factor and how it is calculated. An impact factor is for a journal, not for any given paper. Hence, you can have a poorly cited paper published in a high impact journal or the alternative, a highly cited paper in a lower impact journal. In the end, it is he quality of the work that is important and frankly papers that offer novel ideas are like the proverbial salmon swimming up stream, yet in the end the work will be recognized as its importance reaches greater understanding by the community.

Fourth, it appears that the community places a higher value on science published in the highest impact journals. Why is that value put into place? Certainly the bulk of the work in any given field is published in much lower impact journals than Nature or Science, and this work combined with that of others is what moves fields forward.

While I certainly enjoyed the article, I cannot stop from thinking that there is a misplaced use of impact factors and certainly a misplaced value on striving for the highest impact factor journal. It is important for us to publish our work in relevant journals and in a timely manner, rather than always going for the glitz and glamor of the high impact journals. That said, unfortunately, as many individuals suggested, the entire system, from rewards from an investigator's institution to perception by a study section of an individual's value and the value of their work, is all based upon this warped value placed on where a paper is published.

However, this is a societal problem, not a problem of peer-review. Clearly some of the individuals commenting on this article have not tried to find peer-reviewers for manuscripts, but those of us in the trenches place a high value on the peer-review process. It is the important for the handling or associate editors to examine the reviews for proper rigor and to ensure fairness, but these people are all over worked as well and miss things. Thus, the buck ultimately stops with the editor-in-chief, who has the final say and must deal with all of the missteps. The quality of the EIC's decision making process in the end is all part of the issue, but again this is not a problem with peer-review, but rather the management of peer-review.

Lastly, why is peer-review blinded? Some individuals herein have suggested that peer-review be unblinded. These individuals have not struggled to find peers to review manuscripts, as it is hard enough to find 2-4 reviewers for a blinded process, let alone try to find them for an open process. This protects the reviewers from unsolicited e-mails from authors of rejected manuscripts or from authors coming up to them at a meeting and demanding an explanation as to why their latest manuscript was rejected. Think the process has problems now, put the idea of unblinded peer-review into the process and see what happens.

beauty and importance are in the eyes of the beholder!

If you are in science for the acceptance or even casual note of other scientists, then you are doing it for the wrong reason. I judge how successful I have been by how many people have plagerized, stolen, or paid enough attention to intentionally scoop my work. It is far more flattering than an actual reference which is more likely to be done for purely political motives.

The whole article is based on the premise that "peer review is broken", implying that the peer review process excludes important papers. On the contrary, all the articles cited in this piece were subject to peer review and were published. Furthermore, these papers were read and have become highly cited.

Perhaps what IS broken is the concept that only papers in Nature, Science, or Cell are worth reading. This article only serves to perpetuate this myth. However, most astute scientists know to keep track of all papers in their fields of expertise. They probably are also aware that even the so-called top journals are contain results that are incorrect or irreproducible.

"Peer Review" has been for 30 years+ driven by Money.. grant money. Those who "peer review" innovative ideas that disagree or contradict their own "ideas" rarely will get money or support. It is the Corporatizing of Scientific Research in this country. To get along you has to go along. Any wonder why so many innovative ideas come from small to medium size companies rather than corporate giants? The same rules apply to scientific research since teh grants began to shrink with Nixon in 1969.. Peer review is the money gatekeeper
Jack Markuszka, Ph.D.

I find the peer review system to be a discouraging reminder of how entrenched the old boy network is in science. The professional editors routinely discriminate against young scientist by not sending their articles out for review and the bar to publishing in high profile journals is clearly lower for a well established lab. No question about that.

But, the scientific community effectively sets the impact factor by our citation habits. Why do we bother citing Science, Nature and Cell when we publish in our second tier journals? We are psychologically conditioned to believe that an article published in one of these journals will better support our argument. I, for one, am going to make an effort to find quality articles from specialty journals for my future citations.

There are way more citations produced outside of the top tier journal that are propping up these over inflated impact factors.

While this analysis of the peer review system may be appropriate, there is one other consideration that has been missed in this analysis: in the same manner that reviewers for journals (and grant applications) favor work that is fashionable, readers and those who cite papers in their own publications probably follow a similar pattern.
True breakthroughs can usually not be measured at short term levels. Ideas and results that truly challenge established dogmas will have even greater difficulties in getting published and then in being recognized and accepted.

With the spread of open access articles, it's true most of us just run a search and find exactly what relevant article we're looking for. For graduate students and young investigators though trying to get a foot through the door, I think not getting an article published in a "top journal" influences where they can start their careers regardless of research quality and can even influence what type of research they commit to.

I don't see the problem. Nature and Science have very high rejection rates and work on a fairly tight schedule. The papers in question were all published in very high quality journals that would be read by practitioners in these fields.

(Nature386;319 (1997))
Copyright Macmillan Journals Ltd
Robin P Clarke
Sir---Simon Wain-Hobson's correspondence item (385, 384) is by no means the first expression of dissatisfaction with the refereeing of papers, and won't be the last. I second his proposal for reviews to be of a sufficient length. Say 10% of the paper's length for a rejection, 5% otherwise?

There is a change that could not only improve the quality of reviews and consequent decisions, but also make the editorial decisionmaking process more efficient. The idea is for editors not to examine reviews initially but rather to send them immediately to authors for response, and then if, and only if, the authors feel they have a credible defence against the reviews will they then submit their response to the editor for consideration. Reviewers would then know that they cannot get away with sloppy or malicious reviews. And editors would be saved the work of trying to weed out poor papers that can slip through the less controlled review process that is the current usual norm. This could go some way towards reducing the massive volume of poor-quality papers being published (other than in Nature of course!).

Meanwhile authors could copy the trick I once used, of compiling a collection of previous reviews of the paper along with my rebuttals, and sending that with the submitted manuscript to the next journal. It worked for me. The paper involved was described by a reviewer for Personality and Individual Differences as well-written, well-argued, and well-documented, whereas a British Journal of Psychiatry reviewer reckoned it was of lowest grade in all three respects! It is difficult to see how this gross discrepancy can be accounted for except in terms of maliciousness on the part of the latter reviewer, such as I encountered with many previous reviews. Robin P. Clarke [slightly re-edited]

I further strongly recommend this excellent article by Eysenck which is as relevant now as it was in 1992: "Peer review: Advice to referees and contributors" doi:10.1016/0191-8869(92)90066-X

This article raises sound issues. The solution is that all peer-review journals should publish the name of the reviewer, the institution of the reviewer, and the reviewers comments in an open access format.

In this way, I can almost bet that the reviewer would be more inclined to submit largely constructive, encouraging, positive and collegial comments to the open access reviewer forum. Contrst such a style with the unfortunate current problem of petty mindedness, erroneous reviewing where the entire manuscript has never been read and the often quite frankly insulting critiques of the envious or ambitious reviewer attempting to unethically stifle progress and your work.

The result is peer reviewed material, immediately available to all, without worry about impact factor. If the results are important, search engines almost all have will pick up the article. Others will then use the article and prolifically cite it. As for the cost, $1500 is nothing compared with the charge for personnelle and materials required to produce the results.

To my understanding, the problem is not peer review but the way we evaluate the scientific publications based on the impact factor of the journal. Pick any journal and you will find articles of low to high quality in the same issue! Rather than trying to publish in high impact journals, may be we need to concentrate on good science and recognition will follow...Correcting peer review is like correcting human psycology!

There is no denial that top journals usually publish high impact work, but they are also known for publishing some false results. These examples discussed here and many others really show that it is what in an article not journals that matters in the long run. The ugly duckling effect is just sweet for these scientists. For your students and beginning postdocs, the energy would be better spent on thinking about the questions and doing the best you can to answer them. The reward will come to you, no matter which journal you end up publishing.

Although the author implies that the peer-review process is to blame for not publishing these papers, it appears that 2 of the 5 papers described (or perhaps 3 of 6 if the BLAST paper is included) were not even submitted to the higher impact factor journals.

That's like blaming a bouncer for turning away the people who stayed at home.

It is an unfortunate and dark side of the business that the most competent reviewers of a scientific work submitted for publication are often the most fierce competitors. Their reviews can be biased and overly hostile and intended to block publication rather than to give a fair and unprejudiced adjudication of the work.

A paper published in any of these excellent journals such as Molecular and Cellular Biology, Molecular Biology of the Cell, AIDS or Journal of Immunology, etc, reflects the high quality of the work and it is not just a condensed version of it.
On the long term journal impact means absolutely nothing since by then the scientific work stands by itself. Unfortunately on the short term the impact factor does, many times to the detriment of the scientist.

I think that this article shows that we are giving a high value to things that don't deserve it. Why such articles must be published in Nature and Science? Why don?t publish it in other excellent journals of the specific field? Is not only the entire peer-review system that must be reevaluated, but the way that we measure the importance of an article, too. The way how the author wrote this text could lead one to think that only Nature and Science?s publications matters, and this is not true! The impact factor is destroying excellent journals in some fields, because it is a vicious circle, that is encouraged by this article, for example. We must rethink the impact factor.

Journals with high impact factors may have problems in peer reviewing manuscripts, as all journals have. It is really extremely difficult to find a human machine that will determine the ultimate merit of the manuscript. The submitted manuscripts must do rounds before they are trimmed and spruced to fit in the scheme of things. Peer reviewing becomes a blessing in disguise in many cases. In these cases the referee becomes the ultimate guide and judge, and rightly so. But there are instances when evaluation has been biased. Even this can be taken in stride considering the vagaries, susceptibilities, and frailties of some referees. But it is most galling to observe that in some cases, the publication of manuscript is thwarted to buy time so that others (senior ones, friends etc.) working in the same field may get an upper hand and publish results earlier to get the credit of having seen or discovered first.

Nirmal Kumar Mishra
Retd. Professor of Zoology, Patna university,Patna

My conclusions/assessment of the peer review system now even more seriously flawed because of the preliminary screeners' Thursday editorial meetings:

1. The NIH Syndrome (Not invented here or as our departmental secretary said: or not invented at Hammersmith (Clin School)- many academics from that School had arrived at Cambridge in the late 1980s):
I shall not name one of the most discoveries - efforts for cloning a receptor in a fiercely competitive field of enormous importance- theoretical, clinical, pharmaceutical biotechnological- hence $bns worth to international companies was mucked up for 15 yrs. It involved a Nobel laureate, at least 10 members of the elite Academies (no names) getting the exps. (bucket chemistry) done by technicians, without any deep knowledge of and the hazards of artefacts, contamination during manipulations of blood and cells. In contrast, work was done successfully up to a stage by a senior scientist. highly experienced in blood cell work (40 donors blood), producing a 'result' of enormous importance (appreciated by a Nobel laureate friend, and the willingness to clone the receptor by a biochemist - got Nobel prize 7 years later). Judged/refereed (for NATURE, 4 grant giving bodies) by some of these able people- based on the technicians' work- it was derided, rejected. Instead NATURE published a paper relating to just 4 donors blood, seriously flawed- because the result agreed with the views of the referees, presumably!
That flawed paper still remains on NATURE-data base- it has never been withdrawn.

The comeuppance- frustration-embarrassment for the highly distinguished came 15 years later when two labs of perceptive scientists finally cloned the receptor.


2. The tyrannical problem of being judged by the not quite so widely read person(s) without deep, wide knowledge is not new. Think of Pasteur's germ theory of infection (versus Lister's appreciation), Ehrlich's chemotherapy with arsenicals, UK MRC Mellanby and Florey versus Rockefeller Foundation support. Then jump to Stephen Hawking as a young arrival at Cambridge (from Oxford), relatively unknown at a Royal Society meeting: commenting upon Fred Hoyle's exposition: 'This is all wrong'. To Hoyle's rejoinder, Hawking said: "I have worked it out".
In other fields think of Mozart (never had a proper job), Whittle's jet engine and Cockerel?s hovercraft not being judged sufficiently important by the establishment civil servants.

My views a reviewed author and reviewer (MSS, Grant applications, seeing the reports of the two other reviewers appointed sent by the Eds:the problem, sadly, arose from the management of scientific publishing by clever publishers,busy editors (many not practising scientists, trying to cope with a large number of MSS. For this purpose, the MSS instead of being judged by people who know the subject, are being judged by the not quite so widely read.
And therein lies the problem.

In the interest of science many editors have been made aware of this- with no apparent effect.

Re Francis Crick: I think probably Sir Lawrence Bragg and a few others once 'complained' that the 'trouble with Francis' and his sharp, quick mind, and he can't himself, is that he sees other peoples data, interprets their data, and solves it for them and they stand there struggling- or something like that.

J Mehrishi. PhD, FRCPath.

The problem is not precisely that novel ideas would be rejected in top journals, rather novel authors. In Nature one is more likely to be rejected immediately without review process if he has no history with Nature, because this is the easiest criterion for editors... This tends to create a sort of "verified authors" who can get a lot published there, and the rest.

We are all too aware of the imperfect peer-review process, and clearly innovation is sometimes difficult. But a handful of papers that, in reality, eventually got their due, is a very minor aspect of the problem, or perhaps in light of the number of citations they got, a non-problem. And what is wrong with a journal of impact factor 6? The number of high-ranking papers and corresponding careers that are boosted with research that is rightfully forgotten overnight clearly swamps the breakthroughs from the second tier.

Expecting a foolproof review process? sorry but I am not that optimistic!

It is a matter of choice which process the journal follows and when decide they have to take all the good, the bad and the ugly that come with it!

Impact factors are the currency of grant reviews yet they are based on quixotic journal peer review, dependant on the opinion of an infinitesmally small number of referees for each individual article (usually 2-4 per article). In addition they are biased against "specialist" journals serving smaller disciplines.
However they are here to stay: the best that can be said for them is that papers published in high impact factor journals are probably of high quality and likely tot be important, while papers published in lower impact journals should not be considered likely to be unimportant. Can a numerical correlation between Impact Factor and article importance (not just number of citations) be determined?