This month our new column, Thought Experiment, considers whether mathematics can answer the deepest perplexities of science, such as evolution and consciousness. Here’s a corollary: Can metrics point to the great discoveries of science, and reward the discoverers?

Underlying this latter question is the profound truth that scientific careers are largely made or squelched by numbers that measure the import of one’s research—particularly, how often one is published in high-impact-factor (IF) journals and how often (and where) one’s papers are cited. That flaws abound in these metrics, is framed by neurobiologist Bjoern Brembs: “Without a moment’s hesitation I would fail any undergraduate who comes with a project using statistics only half as bad as the IF. But it’s good enough to determine who gets promoted and who doesn’t?”

Fortunately, scientometricians are tackling the problems. As laid out in a recent issue of Nature, alternatives to IF and the citation index are being developed to quantify research performance, including Google’s PageRank, social bookmarking, the h-index, online access, and mapping techniques. Yet it’s important to remember that popularity metrics can easily be skewed or gamed, and that intellectual merit can be lost in the shuffle.

Beneath these computations lies the foundation of the scientific career edifice: peer review, whereby grant proposals and research results are deemed worthy of funding and of bearing the imprimatur of a high-impact journal… or not. On this subject, one need only cede the podium to Richard Smith, former editor of BMJ, who decries the enormous time, money and energy spent with dismal outcomes in detecting fraud and error, and in unearthing the truly great papers: “After 30 years of practicing peer review and 15 years of studying it experimentally, I’m unconvinced of its value.”

More than 300 years since the invention of peer review and 30 years post-Web, it’s time to act. Lest we forget, the Web was originally designed to disrupt scientific publishing, as recently noted by Michael Clarke in the Scholarly Kitchen blog.

The first major disruption has been open access (OA) publishing, a prerequisite for the new metrics, which thrive on increasing numbers of papers and data. And despite its fledgling status, OA has ushered in a second major disruption to the scientific establishment: post-publication peer review (PPPR), in a variety of experiments and formulations, pioneered by BioMedCentral and PubMedCentral.

Before opining on the glories of PPPR, a word from our sponsor: The Scientist and Faculty of 1000 (a PPPR service) have recently joined forces. Our audience and our goals are the same: to identify and amplify the most interesting developments in the life sciences. Magazine content will thus emphasize research deemed by Faculty Members to be game-changing in 31 disciplines and 305 specialties.

In the basic formulation of PPPR, qualified specialists (peers) evaluate papers after they are published. Instead of hiding reviewers’ identities and comments, they become part of the published record and open to community review and response. Renowned educator Paolo Freire once said, “To impede communication is to reduce men to the status of things.” PPPR at its best facilitates ongoing dialogue among authors, peer reviewers, and readers.

Thus, transparency and ongoing scrutiny by a much wider community can minimize the failures of traditional peer review (depicted on the cover of this issue), and can also bring to light innovations and discoveries that may have been ahead of the curve at the time of publication. Robust involvement by the community is required, and proposed “reputation systems” may be the key to ensure rewards for commenting and revising.

The most wonderous disruption of all: the idea that papers and data have lives beyond their original posting, and discoveries may emerge from them via evaluations and iterations months or even years later. Let’s work out the details.

sgreene@the-scientist.com

Correction (August 11): When originally posted, this article misspelled Bjoern Brembs's name. The Scientist regrets the error.

One problem:
Qualification and Distribution
Solution:
Controlled examination
One tool:
Peer review ???

Another problem:
Scientific networking
Solution:
Communication
One tool:
FREE puplication !!!

What ever the problem is the evaluation system depends on what you are looking for! For the scientific exchange you simply need the non commented data to discuss! One question I ask myself is whether we need an improvement or establishment of the exchange...

I'm confused. We seem to be discussing several things at once:
1) High impact factor journals that can be read only by those who have been lucky enough to find a job in a well-endowed institution.
2) Open-access publications for which, apparantly, you have to be lucky enough to find a job in an institution that can pay the publication fees.
3) Peer review, where it is may be advantageous to be lucky enough to have found a job in an institution that has the right kind of reputation.

My previous heading was truncated.

Unfortunately authors of the comments suggesting that "if it ain't broken don't fix it" completely misunderstand the nature of the scientific period we are living in. It is a epoch-changing transitional period, which will be followed (within the next 10-30 years) by radical changes in practically all natural sciences. Looking at peer review from this historical perspective, it contributes enormously to the prolongation of the our transitory period, and in this sense it is very, very costly for our society. The conventional peer review system would be operating much more effectively IF we lived in the period of a considerably more developed science.

The simplest, but obviously not a complete, solution I suggested in the header. I'm sure very soon such journals, if properly organized, would have very high ranking, moving us much faster along the scientific road.

Wikipedia is part of a model of "corrections" and could conceptually be part of a publication process. Also, with the electronic media, references can be hyper-linked to almost any scientific article. Indeed, this idea can have dozens of "references" and layers of debatable comments. We also have search and select tools to find missing links, and to "tag" linkages or limitations in references. This makes many continued communication accessible, and corrections can be identified as "delayed references" and "active rebuttals" all linked together.
I think we need to review the options and collaborate to find some "first trials" of evaluating and identifying relevant information and format. This could be a much better way to find linkages and also to identify plagiarism. A "cutoff" time would be helpful from a practical perspective. In addition, we need to add multimedia to the modes of expression and description.

Unfortunately the present peer review system reflects in part, and is affected by the cut throat competition for the ever dwindling amount of funding dollars.

Any system of evaluation is prone to personal biases, PPPR not withstanding. It is noteworthy that this system is still highly-valued by scientific authors and is generally believed to result in better scientific papers. As reported in the LINK">2009 Peer Review Survey,

"Almost all researchers (91%) believe that their last paper was improved as a result of peer review"

PPPR can only comment on a paper AFTER it was published.

"The failures of traditional peer review" are due to poor execution as opposed to flawed principle. Too many reviewers are either lazy or biased. Editors need to recognize this and remove the reviewers. This will be just as likely to occur in a PPPR system.

Great article. I think peer review isn't broken but it is crumpled and leaking credibility fast.

Peer review has become more and more negative over the years.

I've posted my thoughts on this from the perspective of a stem cell scientist on my blog LINK

Paul

Peer review allows such wildly crackpot papers to
be published so often and people still believe in it! Consider physics and for one example a wildly popular theory with many papers published in the "best" journals. This theory has no rationale, wildly disagrees with reality (requiring a dimensionality of 10 or 11) and is mathematically impossible. It has long been known that a universe is possible only with dimension 3+1, which physicists reject since it agrees with reality so cannot get through peer review. For proof see book
Our Almost Impossible Universe:
Why the laws of nature make the existence of humans extraordinarily unlikely.
But correct, rigorously proven results do not get through peer review.

Oops, there was some mistyping. These are the authors



Tracy L. Bale, Tallie Z. Baram, Alan S. Brown, Jill M. Goldstein, Thomas R. Insel, Margaret M. McCarthy, Charles B. Nemeroff, Teresa M. Reyes, Richard B. Simerly, Ezra S. Susser, Eric J. Nestler

Indeed, I think it is time for NIH to adopt *the basic formulation of PPPR?? Instead of hiding reviewers? identities and comments, they become part of the published record and open to community review and response*. Other than minimizing the failures of traditional peer review, it would help not to cast doubt over people who truly deserve credit for funding and/or publications advancing the field.

Here is a recent example in: Biological Psychiatry 68 (4): 314-319, 2010

Early Life Programming and Neurodevelopmental Disorders

Tracy L. Balea, Tallie Z. Baramc, Alan S. Brownd, Jill M. Goldsteine, Thomas R. Inself, Margaret M. McCarthyg, Charles B. Nemeroffh, Teresa M. Reyesb, Richard B. Simerlyi, Ezra S. Susserd, Eric J. Nestlerj.


The Reporter appears to show that almost half of the authorship received awards from NIMH that were reviewed by Special Emphasis Panels convened by the Institute in 2008 through 2010. Since the Director of NIMH is an author, one can?t help but ask who in that group truly deserved the award and who eat the one corresponding to a small fish. Complete information on the reviews would dissipate doubts on the science and interests beneath it.