Björn Brembs

**tl;dr:** *Evidence suggests that the prestige signal in our current
journals is noisy, expensive and flags unreliable science. There is a
lack of evidence that the supposed filter function of prestigious
journals is not just a biased random selection of already self-selected
input material. As such, massive improvement along several variables can
be expected from a more modern implementation of the prestige signal.*

Some common responses to the proposal to
[replace](https://doi.org/10.5281/zenodo.5526634) the now more than
35,000 peer-reviewed scholarly journals with more
[modern](https://www.coalition-s.org/blog/creating-a-market-to-replace-publisher-monopolies/)
[solutions](https://researchprofessionalnews.com/rr-news-europe-views-of-europe-2022-1-how-to-reclaim-ownership-of-scholarly-publishing/)
are "Why do you want to get rid of peer review?" or "How should we know
what to read without journals?".

Both, of course, imply insufficient engagement with the proposal and its
consequences. As for the first question regarding peer-review, there is
currently very little evidence as to the cost-effectiveness of
peer-review and most comparisons between un-reviewed 'preprint'
manuscripts and the published articles after peer-review, show [few
substantial
differences](https://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.3001285).
Given the [huge costs associated with
peer-review](https://researchintegrityjournal.biomedcentral.com/articles/10.1186/s41073-021-00118-2),
one would expect peer-review effectiveness to increase dramatically by
modernizing the way it is organized and deployed. As this conclusion and
expectation is not new, the
[literature](https://www.nature.com/articles/sj.bdj.2016.131) is
[brimming](https://academic.oup.com/ajhp/article/74/24/2080/5102692)
with
[suggestions](https://www.sciencedirect.com/science/article/abs/pii/S1070324116302814)
for [improving peer-review](https://reimaginereview.asapbio.org/), just
[waiting](https://www.tandfonline.com/doi/abs/10.1080/1047840X.2012.692215)
to be
[tested](https://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.3000116).
The proposal to replace journals specifically includes a more effective
way to leverage the power of peer-review.

The second question implies two implicit assumptions, namely that the
prestige inherent in journal rank carries a useful signal and that this
signal is actually being commonly used when making decisions about which
portion of the scholarly literature to read. Let's see if there is
evidence supporting these assumptions.

There is some evidence in the literature
([review](https://www.frontiersin.org/articles/10.3389/fnhum.2013.00291/full)
of a subsection) that there indeed is a signal conveying more 'novel' or
'breakthrough' discoveries in the more prestigious journals. However,
this literature also emphasizes that the signal to noise ratio is very
poor: it contains many false-positives (articles that turn out to be not
particularly novel or ground-breaking) and false-negatives (many novel
and ground-breaking discoveries are first published in lower ranking
journals). Unfortunately, because of the nature of journal publishing,
the full extent to which the false-negatives contribute the noise in the
signal cannot be known, as it is commonly not known which submissions
have been rejected by any given journal (more about that later). Adding
insult to injury, this already noisy signal is then degraded even
further by the reliability of the signal being no higher (in fact,
[slightly
lower](https://www.frontiersin.org/articles/10.3389/fnhum.2018.00037/full))
than in average journals. This means that more of the published articles
in prestigious journals, even those that are novel or signify a
break-through, turn out to be irreproducible or contain more errors than
articles in average journals. Finally, this weak signal is then bought
at a price that exceeds the costs of producing such a scholarly article
by about a [factor of ten](https://f1000research.com/articles/10-20/v2).
Thus, in conclusion, the prestige signal is noisy, the science it flags
is unreliable and the moneys it draws from the scholarly community
exceed the cost of just producing the articles by almost an order of
magnitude.

It may thus be of little surprise that one can find evidence that this
noisy signal that tends to also flag unreliable science does not seem to
be used as commonly by readers as it is sometimes claimed.
Unfortunately, there is no direct evidence as to how readers use the
scholarly literature, in part, again, because our literature is
fragmented into 35,000 different journals. Therefore, currently, we are
forced to restrict ourselves to citations as a proxy for reading: one
can only cite what one has also read. Using the signal of journal rank
(as routinely and reliably measured by impact factor),
[Lozano](https://onlinelibrary.wiley.com/doi/full/10.1002/asi.22731) *et
al*. predicted subsequent citations to published articles. In other
words, the authors tested the assumption that journal rank signals to
readers what they should read and, consequently, cite. When looking at
it historically (figure below), this predictive power of journal rank
starts surprisingly low, but statistically significant, due to the
authors analyzing millions of articles.

::: {.wp-block-image}
<figure class="aligncenter is-resized">
<img
src="https://onlinelibrary.wiley.com/cms/asset/e998d983-dd22-43a8-bc03-099dd26dd42c/asi22731-fig-0001-m.jpg"
decoding="async" height="400" alt="figure" />
</figure>
:::

As the serials crisis starts to hit in the 1960s/70s and libraries are
forced to use the impact factor to cancel subscriptions to low-ranking
journals, this predictive power, not surprisingly, starts to increase,
if ever so slightly: you really can only cite what you can read. The
advent of keyword searches and online access then abolishes this modest
increase and today, the predictive power of journal rank on
citations/reading appears to be as low as it ever was. Given that
citations to articles in prestigious journals are often used by authors
to signal the importance of their topic, even these low numbers are
probably overestimating the true effect of journal rank on reading
decisions. So there seems to be an effect of prestige on
citations/reading, but it is very weak, indicating that either only very
few readers are using it or each reader is not applying it very
stringently. In other words, the low predictive value of journal rank on
future citations can be considered a reflection of users realizing that
the publication venue contains little useful information when making
choices about what to read.

Noticing an apparent disconnect between these citation data and the fact
that journal rank is partly based on citations, Pedro Beltrao [pointed
to](https://www.evocellnet.com/2013/06/doing-away-with-scientific-journals.html)
a different study, where high impact factor journals, on average, tend
to be enriched in papers that will be highly cited in the future,
comparing the slice of 1% most highly cited papers. However, these
results only superficially seem to be in contradiction to the data
presented by Lozano *et al*. In fact, I would argue that the 1% study
actually supports the interpretation that journal rank is only very
rarely used as a signal of what to read or cite: because there are more
"novel" or "break through" papers in these journals (as detailed above),
people will cite these rare 1% papers more often. But that effect hardly
changes the probability of the remaining 99% of papers to be cited.
These papers are 'regular' papers that could have been published
anywhere. This evidence thus suggests that readers decide what they cite
not based on containers, but based on content: the 1% of articles in
prestigious journals that are actually novel and ground-breaking are
cited more, and the other 99% are cited just as much as articles in any
other journal. In fact, this insight is corroborated when comparing
citation distributions between journals.

All of this evidence supports the interpretation that yes, there is a
signal in journal rank, but it mainly comes from a very low percentage
of the papers (that then, in turn, is less reliable than average) and
hardly translates to the vast majority of the papers in prestigious
journals at all: the effect is almost gone when you include all papers.
Therefore, arguably, the little effect that remains and that can be
detected, is likely not due to the selection of the editors or
reviewers, but due to the selection of submission venue by the authors:
they tend to send their most novel and break-through work to the most
prestigious journals. The fact that what we think is our most novel and
groundbreaking work is too often also our worst work in other aspects,
probably explains why the reliability of publications in prestigious
journals is so low: not even the professional editors and their
prestigious reviewers are capable of weeding out the unreliability.

Taken together, despite the best efforts of the professional editors and
best reviewers the planet has to offer, the input material that
prestigious journals have to deal with, appears to be the dominant
factor for any 'novelty' signal in the stream of publications coming
from these journals. Looking at all articles, the effect of all this
expensive editorial and reviewer work amounts to probably not much more
than a slightly biased random selection, dominated largely by the input
and to probably only a very small degree by the filter properties. In
this perspective, editors and reviewers appear helplessly overtaxed,
being tasked with a job that is humanly impossible to perform correctly
in the antiquated way it is organized now.

Unfortunately, given the current implementation of the prestige signal
by antiquated journals, it remains impossible to access the input
material and test the interpretation above. If this interpretation of
the evidence were correct, much would stand to be gained by replacing
the current implementation of the 'prestige signal' with a more modern
way to organize it.

How could a more modern system support a 'prestige signal' that would
actually deserve the moniker? Obviously, if journals were to be replaced
by a modern information infrastructure, only our imagination is the
limit for which filters the scholarly community may want to implement.
Some general ideas may help guide that brainstorming process: If the use
of such 'prestige' not only in career advancement and funding, but also
in the defense of the current system is anything to go by, there should
be massive demand for a prestige signal that was worth its price. Today,
this prestige arises from selectivity based on expertise (Nature's
slogan always was "the world's best science"). This entails an
expert-based filter that selects only very few ('the best') out of of
the roughly 3 million peer-reviewed articles being published each year.
Importantly, there is no *a priori* need to objectively specify and
determine the criteria for this filter in advance. In a scenario after
all journals had been replaced by a modern infrastructure for text, data
and code, such services (maybe multiple services, competing for our
subscriptions?) need only record not just the articles they selected (as
now) but also those they explicitly did not select in addition to the
rest that wasn't even considered. Users (or the services themselves or
both) would then be able to compute track records of such services
according to criteria that are important to them.

Mimicking current implementations, e.g., the number of citations could
be used to determine which service selected the most highly cited
articles, how many it missed and which it falsely didn't even consider.
But why stop at bare citations? A modern infrastructure allows for
plenty of different markers for scholarly quality. One could just as
well use a (already existing) [citation
typology](https://jbiomedsem.biomedcentral.com/articles/10.1186/2041-1480-1-S1-S6)
to differentiate between different types of citations, one could count
replications, media mentions, anything, really, to derive track records
by which these services may be compared. Given the plentiful demand
indicated by the fervent supporters of prestigious journals, services
would compete with each other using their track records for the
subscriptions of individual and institutional users, providing for
innovation at competitive prices, just like any other service market.
Such an efficient, competitive marketplace of services, however, can
ever only arise, if the current monopoly journals are replaced with a
system that allows for such a market to be designed. If demand was not
as high as expected, but such a signal nevertheless desired by some, a
smaller, more basic implementation could be arranged on a non-profit,
open source basis, funded by the vast savings that replacing journals
would entail. One may also opt to hold competitions for such services,
awarding prizes to the service that best serves the needs of the
scholarly community. The possibilities are endless -- but only once the
scholarly community finds a way to put itself into a position where it
has any power over the implementation of its infrastructure.

Perhaps most importantly, such a modern implementation of the prestige
filter offers the promise of actual consequences arising from the
strategies these prestige services employ. Today, publishing articles
that later turn out to be flawed has virtually no tangible consequences
for the already prestigious journals, just as a less prestigious journal
can hardly increase its prestige by publishing stellar work. With power
comes responsibility and our proposal would allow the scholarly
community to hold such services to account, a feat virtually impossible
to reach today.

P.S.: Much like the ideas for replacing journals have been around for
decades, the ideas for replacing prestige signals have been discussed
fro a long time. [There is a post from Peter Suber from
2002](https://dash.harvard.edu/bitstream/handle/1/4727447/suber_thoughts.htm),
and there is [one of my own from
2012](http://blogarchive.brembs.net/comment.php?comment.news.882).


<strong>
 tl;dr:
</strong>
<em>
 Evidence suggests that the prestige signal in our current journals is noisy, expensive and flags unreliable science. There is a lack of evidence that the supposed filter function of prestigious journals is not just a biased random selection of already self-selected input material.
</em>

Replacing the prestige signal

In
a [paper](https://www.nature.com/neuro/journal/v17/n2/full/nn.3616.html)
published in *Nature Neuroscience* now over a year ago, the authors
claimed to have found a very surprising feature, which was long thought
to be a bug. In my [blog post covering the hype in the paper and that in
an accompanying news-type
article](https://bjoern.brembs.net/2014/02/hiding-the-shoulders-of-giants/),
I wrote that today, it has become ever rarer that scientists admit to
standing on the shoulders of giants, as we are not rewarded for
referring to giants, but only for being giants ourselves. The blog post
has triggered some email correspondence between a number of colleagues
in or close to this field, at the end of which was the decision that two
of us, [Nicolas
Stergiou](https://coe.unomaha.edu/facstaff/nstergiou.php) and I, would
contact the journal and inform them of the missing references in the two
articles.

The first reply was that we ought to write a 'letter to the editor'
instead of informally notifying the journal that there were some crucial
references missing. So we sat down and wrote our letter (submitted
February 10, 2015):

> Dear Sir,
>
> HHMI Investigator [Michael Eisen recently
> wrote](https://www.michaeleisen.org/blog/?p=1619) that we ought not
> only to come down hard on people who cheat but also on the far great
> number of people who overhype their results. He suggests that hyping
> should be punished just as fraud. This letter to the editor represents
> such an effort as we would like to report on such a case of hyping.
>
> "Standing on the shoulders of giants" is what scientists say to
> acknowledge the work they are building on. It is a statement of
> humility and mostly accompanied by citations to the primary literature
> preceding the current work. In today's competitive scientific
> enterprise, however, such humility appears completely misplaced.
> Instead, what many assume to be required in the struggle to survive is
> to convince everyone that they are the giant, the genius, the prodigy
> who is deserving of the research funds, the next position, tenure. The
> *Nature Neuroscience* article "Temporal structure of motor variability
> is dynamically regulated and predicts motor learning ability" by Wu
> *et al*. (doi:[10.1038/nn.3616](https://dx.doi.org/10.1038/nn.3616))
> with its accompanying news-type article "Motor variability is not
> noise, but grist for the learning mill" by Herzfeld and Shadmehr
> (doi:[10.1038/nn.3633](https://dx.doi.org/10.1038/nn.3633){target="null"})
> from earlier this year clearly fall within this category. Both
> articles claim that the researchers have made the game-changing
> discovery that something long thought to be a bug in our movement
> system is actually a spectacular feature. It is argued that this
> discovery is such a huge surprise, because nobody in their right mind
> would have ever thought this "unwanted characteristic" to actually
> serve some purpose.
>
> The problem with this line of argument is that probably most people in
> the field thought it should be obvious, even to be expected -- and not
> surprising at all. Skinner is largely credited with the analogy of
> operant conditioning and evolution. This analogy entails that reward
> and punishment act on behaviors like selection is acting on mutations
> in evolution: an animal behaves variably and encounters a reward after
> it initiated a particular action. This reward will make the action now
> more likely to occur in the future, just as selection will make
> certain alleles more frequent in a population. Already in 1981,
> Skinner called this "Selection by Consequences" (*Science* Vol. 213
> no. 4507 pp. 501-504, DOI:
> [10.1126/science.7244649](https://dx.doi.org/10.1126/science.7244649)).
> Skinner's analogy sparked wide interest, e.g. an entire journal issue
> (Behavioral and Brain Sciences 7(04), 1984), which later appeared in
> book form (The Selection of Behavior: The Operant Behaviorism of B. F.
> Skinner: Comments and Consequences. A. Charles Catania, Stevan R.
> Harnad, Cambridge University Press). Clearly, the idea that
> reinforcement selects from a variation of different behaviors is not a
> novel concept at all, but more than three decades old and rather
> prominent. This analogy cannot have escaped anybody working on any
> kind of operant/motor learning, except those seriously neglecting the
> most relevant literature. This interaction of variability and
> selection is a well-known and not overly complicated concept, based in
> evolutionary biology and psychology/neuroscience. Consequently,
> numerous laboratories have been studying various aspects of this
> interaction for a long time. Skinner's projection was that increased
> behavioral variability leads to increased operant learning rates, just
> like increased mutations rates lead to increased rates of evolutionary
> change. More than a dozen years ago, Allen Neuringer showed this to be
> the case in rats (Psychonomic Bulletin & Review 2002, 9 (2), 250-258,
> doi: [10.3758/BF03196279](https://dx.doi.org/10.3758/BF03196279)), but
> there are studies in humans as well (Shea, J. B., & Morgan, R. B.
> (1979). Contextual interference effects on the acquisition, retention,
> and transfer of a motor skill. Journal of Experimental Psychology:
> Human Learning and Memory, 5, 179--187). That such variability is
> beneficial, rather than detrimental has been shown even in situations
> where the variability is so high, that the acquisition rate is
> reduced, but post-training performance is enhanced (Schmidt RA, Bjork
> RA (1992): New conceptualizations of practice: Common Principles in
> Three Paradigms Suggest New Concepts for training. Psychological
> Science, 3(4): 207-217).
>
> Wu *et al*. confirm both Skinner's conjecture as well as previously
> published reports (some cited above) that indeed the rate of learning
> in operant conditioning is increased in subjects where the initial
> variability in the behavior is higher. However, instead of citing the
> wealth of earlier work, Wu *et al.* claim that their results were
> surprising: "Surprisingly, we found that higher levels of
> task-relevant motor variability predicted faster learning". Herzfeld
> and Shadmehr were similarly stunned: "These results provide intriguing
> evidence that some of the motor variability commonly attributed to
> unwanted noise is in fact exploration in motor command space."
>
> We regard it as highly unlikely that none of the seven authors in
> total should have never heard of Skinner or the work over the last
> four decades by many human movement scientists that have explored the
> temporal structure of human movement variability and its relationship
> with motor learning. The work by senior scientists such as Karl
> Newell, Michael Turvey, Richard Schmidt, and their students published
> in books and hundreds of journal articles is completely ignored, just
> as the work by several younger mid-career scientists such as Nick
> Stergiou, Jeff Hausdorff, Thurmon Lockhart, Didier Dilignieres, and
> many others. After a thorough review of this literature the authors
> may realize that their results are neither new nor novel. If indeed
> the authors were unaware of this entire section of literature so
> relevant to their own research, it would be an indictment in its own
> right.
>
> Hence, in formal corrections of both articles, we would expect all
> mentions throughout both texts of how surprising these findings were,
> to be replaced with references including, but not limited to the works
> cited above.
>
> Yours sincerely,

In hindsight, we probably ought to have mentioned that the research of
course has merit and that there are a lot of valuable and exciting
results in the paper, but that it is one specific aspect that is really
not surprising at all. However, we wanted to make it brief and focused
just on the hype. Maybe that was a mistake. Be that as it may, after
about two months of peer review, our letter was rejected. Perhaps not so
surprisingly, the peer reviewers apparently were the same ones that
reviewed the original manuscript (at least one of them):

> <div>
>
> This is the basis on which I recommended publication, and I do not
> feel it needs a corrigendum.
>
> </div>

<div>

After all, the [data
indicate](https://journal.frontiersin.org/article/10.3389/fnhum.2013.00291/full)
that such high-ranking journals do not use very rigorous peer-review,
leading to the least reliable science being published there. What was
quite surprising, though, was the implicit support and even endorsement
of the hype:

</div>

> <div>
>
> I agree that the article's tone is a little more breathless than
> strictly required, but this is the style presently in vogue
>
> </div>

<div>

and

</div>

> <div>
>
> The letter complains about "over-hyping" about certain claims made in
> the paper related to previous work. I have some sympathy with the
> letter writers on this front. However, I also have some sympathy for
> the authors, who understandably were trying to emphasize the novelty
> of their work.
>
> </div>

<div>

It thus appears as if at least two of the three reviewers agreed with
us, that the articles were hyping the results, but they don't see
anything wrong with hype. This means that hype is not just a problem
of [journalists](https://www.bmj.com/content/349/bmj.g7015) any more.
Hype is not just a problem of journals
and [GlamMagz](https://www.nature.com/ng/journal/v35/n1/full/ng0903-1.html)
any more. Hype has now arrived in the middle of the scientific community
with explicit endorsement from reviewers. It looks like [Michael Eisen's
call](https://www.michaeleisen.org/blog/?p=1619) for hype-punishment
will go largely unheeded.

</div>

<div>

</div>

<div>

[*You can download the complete reviews of all reviewers [here
(PDF)](https://bjoern.brembs.net/wp/?wpdmdl=817), to make sure none of
the quotes above were taken out of context.*]{style="font-size: 10pt;"}

</div>


In a paper published in
<em>
 Nature Neuroscience
</em>
now over a year ago, the authors claimed to have found a very surprising feature, which was long thought to be a bug.

Nature reviewers endorse hype

I really loathe reviewing for GlamMagz for two main reasons. For one,
it's hard to remain neutral: publication of a paper in my field in such
a journal is beneficial both for the field and for the young people who
are authors on this paper. Second, the demands of some of my colleagues
so often make my blood boil. At that point I'm very happy these reviews
are anonymous and I really don't want to know the names of these
colleagues. Here are some of the things the reviewers wrote in this most
recent round:

> "the authors have not convinced me of the conceptual novelty of their
> findings to warrant publication in a very top-tier journal"
>
> "appears standard for a top-tier journal"
>
> "However, currently, this manuscript is more suited for a specialized
> journal"

This seems to suggest to me that these colleagues apply different
standards for different journals. So when the journal sent each reviewer
the reviews of the other reviewers and asked for comments, one of my
comments was the following:

> The intuitive notion of journal rank is a figment of our imagination,
> devoid of any empirical support. So-called "top-tier journals" exist
> in the same reality in which homeopathy cures, dowsing rods find water
> and astrologers correctly predict the future. Asking for additional
> 'interesting' or 'curious to know' experiments merely because of the
> venue the authors chose borders on the unethical, IMHO, as there are
> millions of 'curious to know' experiments to be potentially carried
> out. Reviewers ought to try and avoid multiple standards in their
> reviews and restrain additional experiments to the minimum required by
> the statements of the authors. Professional editors are being paid
> precisely because they can predict whether their readers will find the
> statements of the authors 'interesting' or 'curious to know'.

(Where 'curious to know' was a phrase one of the reviewers used whenever
they wanted some tangentially related experiment to be done)

I doubt the editors at GlamMagz are very interested in these sorts of
comments. If they are, I now have a boilerplate from which to draw
future comments.


I really loathe reviewing for GlamMagz for two main reasons. For one, it’s hard to remain neutral: publication of a paper in my field in such a journal is beneficial both for the field and for the young people who are authors on this paper. Second, the demands of some of my colleagues so often make my blood boil. At that point I’m very happy these reviews are anonymous and I really don’t want to know the names of these colleagues.

Why this GlamMag will likely not ask for my review again

[[![ResearchBlogging.org](https://www.researchblogging.org/public/citation_icons/rb2_mid.png){decoding="async"
style="border: 0;"}](https://www.researchblogging.org)]{style="float: left; padding: 5px;"}[The
data clearly
show](https://journal.frontiersin.org/Journal/10.3389/fnhum.2013.00291/full)
that publications in *Cell*, *Nature* or *Science* (*CNS* for short), on
average, cannot be distinguished from other publications, be it by
methodology, reproducibility or other measures of quality. Even their
citation advantage, while statistically significant, is so small that it
is practically negligible. Regardless of all the data, individual
examples sometimes serve to illuminate the data and drive some facts
home. There are more or less extreme examples of *CNS* publications not
meeting the expectations to which they are commonly held. It's needless
to mention all the examples that went through the media, but I would
instead [cite a paper from my
field](http://blogarchive.brembs.net/news.php?item.519.3) (where I feel
reasonably competent) as an instance of an extreme case of an obviously
flawed paper making in into the journal *Nature*.

However, one doesn't even need to look at the extreme cases. Let's look
at a [paper in *Science
Magazine*](https://www.sciencemag.org/content/344/6186/901.full) I
actually quite like. It's an interesting topic, a beautiful experimental
paradigm and the results in the wildtype flies are truly exciting. Their
mutant and transgenic results, if correct, are tantalizing as they run
counter to essentially all available literature on the gene in question.
I personally know and like the senior author of the paper, Gero
Miesenböck, and appreciate all of his work, including this publication.
All I'll try to do here is to make the case, that this paper is an
average paper in terms of the level of evidence provided to support the
claims made by the authors. Not obviously worse than most papers in our
field, but definitely nowhere near the level of evidence those
colleagues ignorant of the data would assume to be required for a *CNS*
paper. All papers have strengths and weaknesses, some have more
uncertainty associated with their results, others less. The data suggest
that there isn't much difference between journals, on average, and I'll
try to use some details in this *Science* paper to exemplify the data.

The authors state in their title that the gene *dFoxP* influences the
speed of perceptual decision-making in *Drosophila* fruit flies. They
test the flies by first conditioning them to avoid a given odor using
electric shocks. The choice test is then done in a chamber on individual
flies. Two odors enter the chamber from opposite ends and exit the
chamber in the middle. The fly then walks back and forth between the
ends of the chamber and avoidance is measured by where the fly spends
most of the time: in the half with the odor associated with electric
shock, or in the half with the control odor. In the current study, the
authors made the control odor increasingly similar to the one associated
with shock and found that wildtype flies spent more time in the middle
of the chamber, where they had to decide of whether to keep walking or
make a turn, the more similar the odors were. That is definitely an
exciting finding, suggesting that more similar odors require more
processing time for an avoidance decision than less similar odors.

In order to get a handle on the biological processes underlying this
interesting phenomenon, the authors have conducted a screen of
genetically modified flies, looking for mutants with longer decision
times than the wildtype flies. One of the candidates that showed up in
the screen, is the *Drosophila* orthologue of a gene involved in
language in humans, *FoxP*. It seems hard to understand what a gene
generally thought to be involved in motor learning has to do with this
perceptual task, but data are data, unless there are alternative
explanations. There are three main concerns that raise the suspicion
that the involvement of *dFoxP* in this task may not be as
straightforward as suggested in the title of this paper.

1.  It s common practice to homogenize the genetic background of mutants
    with that of the control strain to which the mutants are compared.
    This is done to exclude variations other than in the gene of
    interest to be responsible for any differences. However, the authors
    do not state whether and if so for how many generations the mutant
    flies were outcrossed to the proper wildtype genetic background.
    Given that the flies came from a screen, it is unlikely that all
    strains tested in the screen were outcrossed. For instance, the
    alleles described in the paper show significant lethality which
    disappears when outcrossed for six generations, suggesting that
    their general level of fitness and health appears to be affected by
    their genetic background. It is thus possible that factors in the
    genetic background and not *dFoxP* may be responsible for the
    phenotype. Given the track record of the Miesenböck lab, it is
    unlikely they did not outcross the flies, but at the very least, the
    number of generations for which the outcrossing has taken place
    would have to be listed in the paper.

2.  A common procedure to test for genetic background effects, in
    addition to outcrossing which can reduce but not eliminate genetic
    background effects, is to use an additional method to manipulate the
    gene in question. The authors used RNA interference (RNAi) to
    attempt to down-regulate the expression of *dFoxP*. It is common
    practice to use quantitative PCR to test the effectiveness of this
    method, as a) RNAi usually does not abolish gene expression
    completely and b) it is known to have off-target effects, i.e., it
    may affect the expression of other genes besides/instead of the
    target gene. The line used in this study comes from a collection
    known to have problems with off-target effects, so testing
    effectiveness is especially important. However, no evidence that
    such a test was performed is described. Below is what such data
    would look like, if you were to order all RNAi constructs currently
    available, drive their expression in all neurons (with nSyb-GAL4)
    and then have an undergraduate student run a whole slew of qPCR
    reactions to compare the expression (in fly heads) of all three
    *dFoxP* isoforms to the respective control strain (many thanks Joel
    and postdoc [Axel](https://lab.brembs.net/author/axel/) for all that
    work!): No effect!

    ::: {#attachment_596 .wp-caption .alignnone style="width: 530px"}
    [![relative expression of dFoxP isoforms in six different RNAi
    lines](https://bjoern.brembs.net/wp/wp-content/uploads/2014/06/foxp_qpcr-1024x429.jpg){.wp-image-596
    .size-large decoding="async"
    aria-describedby="caption-attachment-596" width="520" height="217"
    srcset="https://bjoern.brembs.net/wp/wp-content/uploads/2014/06/foxp_qpcr-1024x429.jpg 1024w, https://bjoern.brembs.net/wp/wp-content/uploads/2014/06/foxp_qpcr-300x125.jpg 300w, https://bjoern.brembs.net/wp/wp-content/uploads/2014/06/foxp_qpcr.jpg 1808w"
    sizes="(max-width: 520px) 100vw, 520px"}](https://bjoern.brembs.net/wp/wp-content/uploads/2014/06/foxp_qpcr.jpg)
    relative expression of dFoxP isoforms in six different RNAi lines
    :::

    I cannot emphasize enough that the graph above represents [very
    preliminary data]{style="text-decoration: underline;"}. It's data
    from one undergraduate student who had prior experience with qPCR
    before coming to our lab, but who did not do this work full-time.
    There definitely need to be more biological replicates and some
    additional drivers need to be used (e.g. an actin driver to drive
    RNAi in all cells). On the other hand, the results match my own qPCR
    results with these lines and results from another laboratory that we
    collaborate with. So while I would not exclude that we may see
    significant reduction in *dFoxP* expression with some technical
    tweak, the currently available data suggest that none of the
    available dFoxP RNAi lines significantly knocks down *dFoxP*
    expression, including the one tested by DasGupta *et al*. (15732V).
    Our work also points to a reason why none of the lines seems to have
    a detrimental effect on *dFoxP* expression: we have localized
    polymorphisms in the *dFoxP* gene which could bias the RNAi process
    towards sequestration, rather than degradation of the mRNA,
    potentially explaining some of the very high values in the plot
    above. Thus, it is possible (and we consider it even very likely)
    that some or all of these lines actually do affect *dFoxP*
    expression in the way intended. However, we currently have little
    possibility to ensure it's not an off-target effect after all.  
    I'd thus tentatively conclude that the phenotype DasGupta *et al.*
    have discovered can indeed be ascribed to *dFoxP* action in this
    task. However, at this point, there is insufficient data to make
    that statement with any reasonable certainty (no matter how likely I
    personally find that to be the case!).

3.  Finally, previous work on *FoxP* both in other animals and in flies
    (including our own paper coming out tomorrow, watch this space after
    5pm EST on June 25) suggests an involvement of *dFoxP* in motor
    learning and perhaps also (consequently?) motor coordination. In
    fact, [our](http://blogarchive.brembs.net/comment-n632.html)
    [work](http://blogarchive.brembs.net/comment-n678.html) shows that
    learning about external stimuli is fine in *FoxP* mutant flies. It
    is thus critical to make sure that any animals showing a deficit in
    this task can perform the required movements (walking, turning,
    starting, stopping) accurately. The authors here attempted to ensure
    this by measuring the time spent in sections of the chamber other
    than the decision section in the middle. However, one would assume
    that flies will mostly turn or sit at the ends of the chamber and in
    the decision sector, with fewer turns and pauses in the sectors
    in-between. Thus, by evaluating sectors in which the flies mainly
    walk straight, the authors may be underestimating the contribution
    of turns and no-movement episodes to the decision-making process. If
    that is correct (and the brief explanations in the methods section
    make it very difficult to be entirely sure about how precisely and
    on which sectors they have performed the calculations),
    *dFoxP*-manipulated flies may either have a problem starting to walk
    from a stop, or have difficulty turning in the chamber and the
    authors would interpret this as an increased decision-time, rather
    than a motor problem.

In summary, it may well be that *dFoxP* entails all the functions
reported by DasGupta *et al*. and this would indeed make for a giant
head-scratcher about the (ancestral) functions of *FoxP*, given all the
mountain of work involving it in motor, rather than perceptual
processes. However, at this point, some more work (at least some more
detailed method explanations!) is required to be reasonably certain
about the existence of such a problem. Personally, I do think that the
phenotype is due to the authors' *FoxP* manipulations, but I may be
wrong, the evidence is not strong enough either way. I do have my
doubts, however, about the phenotype not being due to motor defects, but
I could be wrong there as well, as the data is inconclusive. At this
point, we need more experiments and/or data analysis to show who is
wrong. Again, I still like the paper, it's the kind of work that drives
our field forward.

Speaking more generally: there is no need for any researcher to wait
(perhaps indefinitely?) until they have the perfect data set, with the
unambiguous results and the foolproof conclusions. Just send your plain
decent work to *CNS* magazines as well! If they publish it, you will get
a job and nobody but some lone blogger will ever ask about the content
of that paper ever again.

------------------------------------------------------------------------

[DasGupta, S., Ferreira, C., & Miesenbock, G. (2014). FoxP influences
the speed and accuracy of a perceptual decision in Drosophila [Science,
344]{style="font-style: italic;"} (6186), 901-904 DOI:
[10.1126/science.1252114](https://dx.doi.org/10.1126/science.1252114){rev="review"}]{.Z3988
title="ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.jtitle=Science&rft_id=info%3Adoi%2F10.1126%2Fscience.1252114&rfr_id=info%3Asid%2Fresearchblogging.org&rft.atitle=FoxP+influences+the+speed+and+accuracy+of+a+perceptual+decision+in+Drosophila&rft.issn=0036-8075&rft.date=2014&rft.volume=344&rft.issue=6186&rft.spage=901&rft.epage=904&rft.artnum=http%3A%2F%2Fwww.sciencemag.org%2Fcgi%2Fdoi%2F10.1126%2Fscience.1252114&rft.au=DasGupta%2C+S.&rft.au=Ferreira%2C+C.&rft.au=Miesenbock%2C+G.&rfe_dat=bpr3.included=1;bpr3.tags=Anthropology%2CBiology%2CPsychology%2CNeuroscience%2CBehavioral+Biology%2C+Ecology%2C+Evolutionary+Biology%2C+Genetics%2C+Zoology%2C+Biological+Anthropology%2C+Evolutionary+Anthropology%2C+Behavioral+Neuroscience%2C+Cognitive+Neuroscience%2C+Neurolinguistics%2C+Comparative+Psycholog"}


The data clearly show that publications in
<em>
 Cell
</em>
,
<em>
 Nature
</em>
or
<em>
 Science
</em>
(
<em>
 CNS
</em>
for short), on average, cannot be distinguished from other publications, be it by methodology, reproducibility or other measures of quality. Even their citation advantage, while statistically significant, is so small that it is practically negligible.

No need to only send your best work to Science Magazine

Talk about egg on face! *Nature* "the world's best science" *Magazine*
sets out to publish back-to-back papers on -- of all topics -- stem cell
science. The same field that brought *Science Magazine* Who-Suk Hwang
and Elsevier's *Cell* Mitalipov's 'errors'. So *Nature* was warned and,
presumably, they got down to business and did the very best they could
to prevent *Cell*'s and *Science*'s mishaps to happen to them. They
screened the manuscripts for 9 months, probably requiring some extra
experimentation and after this procedure went on to publish the two
papers, showing that an unlikely and trivially easy treatment could
generate stem cells. And then, perhaps not too surprisingly for people
reading this obscure blog, of course [the unthinkable
happened](https://blog.pubpeer.com/?p=117): barely a week after
publication, the first issues were spotted, lanes were spliced in gels,
images duplicated. Later, as failures to replicate accumulated, calls
for retractions were issued -- this time even by one of the papers'
authors who had previously claimed he had reproduced the technique.

What was it that took 'the internet' a week that "the world's best
science" magazine could not detect in 9 months? The wisdom of the crowd.
[There is no
evidence](https://journal.frontiersin.org/Journal/10.3389/fnhum.2013.00291/full)
to justify the standing of "the world's leading journals" and the rising
tide of post-publication review embarrassing legacy review only
corroborates this insight: GlamMagz are undeserving of their status.


Talk about egg on face!
<em>
 Nature
</em>
“the world’s best science”
<em>
 Magazine
</em>
sets out to publish back-to-back papers on – of all topics – stem cell science.

Even the most thorough peer-review at the ‘best’ journals not up to snuff?

*[During my [flyfishing vacation last
year](http://blogarchive.brembs.net/comment-n862.html), pretty much
nothing was happening on this blog. Now that I've migrated the blog to
WordPress, I can actually schedule posts to appear when in fact I'm not
even at the computer. I'm using this functionality to re-blog a few
posts from the [archives](http://blogarchive.brembs.net) during the
month of august while I'm away. This post is from [December 6,
2010](http://blogarchive.brembs.net/comment-n684.html):]{style="font-size: 12px;"}  
*

This morning my friend [Ramy reminded
us](https://friendfeed.com/plosone/087270c2/why-was-plos-one-blahttps://friendfeed.com/plosone/087270c2/why-was-plos-one-blamed-for-media-hype-aboutmed-for-media-hype-about){target="_blank"
rel="external"} of the recent spats over two *[PLoS
One](https://plosone.org){target="_blank" rel="external"}* publications
([Darwinius](https://whyevolutionistrue.wordpress.com/2010/03/06/ida-not-a-missing-link/){target="_blank"},
[Red
Sea](https://whyevolutionistrue.wordpress.com/2010/09/23/parting-the-red-sea/){target="_blank"})
and how they were used to question the 'reputation' of *[PLoS
One](https://plosone.org){target="_blank" rel="external"}* as a journal.
Of course, it is about as meaningful to talk about the reputation of a
journal as it is to talk about the reputation of the cover of a book.
Journals are containers which say very little about their content. But
on to the really relevant point:

Specifically, Ramy pointed out how the current spat about a [publication
in the
journal](https://www.sciencemag.org/content/early/2010/12/01/science.1197258){target="_blank"}
*[Science](https://www.sciencemag.org/content/early/2010/12/01/science.1197258){target="_blank"}*
on a purportedly arsenic-based lifeform (see, e.g.,
[Pharyngula](https://scienceblogs.com/pharyngula/2010/12/its_not_an_arsenic-based_life.php){target="_blank"}
and especially [Rosie
Redfield](https://rrresearch.blogspot.com/2010/12/arsenic-associated-bacteria-nasas.html){target="_blank"})
didn't reflect on *Science* at all, despite the basically identical
story-line of media hype before publication followed by more sober
commentary from the scientific community after publication. Why is
*[PLoS One](https://plosone.org){target="_blank" rel="external"}*
criticized in the first two cases, but nobody questions *Science* in
this (or the numerous other) cases? Clearly, the two GlamMagz *Nature*
and *Science* both have their share of in some cases pretty embarrassing
blunders. My personal favorite is a [paper in *Nature* about fly
thermosensation](http://blogarchive.brembs.net/comment-n519.html){target="_self"},
easily the worst conducted study in this field in quite a few years.
Yet, nobody questions the 'reputation' of Nature. Also in this case,
none of the critical commenters questions the legitimacy of the
gatekeeper function that the GlamMagz are so happy to tout.

Let's be honest about it: there's no journal without fault. Everyone
makes mistakes. Journals are no more gatekeepers than the persons
working there. Any perceived hierarchy among journals is merely that:
perceived. A perception caused by visibility, historical baggage,
group-think and circular reasoning.

It doesn't matter *where* something is published -- what matters is
*what* is being published. Given the obscene subscription rates some of
these journals charge, if anything, they should be held to a *higher*
standard and their 'reputation' (i.e., their justification for charging
these outrageous subscription fees!) being constantly questioned, rather
than this unquestioning dogma that anything published there must be
relevant, because it was published there. If anything, every single
contested paper should be used to question the level of subscription
fees raised by these journals.

In fact, every retraction should lead to an immediate reduction in
subscription fees for the journal in which the retracted paper was
published, because the journal failed to serve its gatekeeper purpose.
If the journals justify, as they do, their obscene subscription
extortion with their outstanding peer-review process, their price needs
to drop every time that process fails. Given the
[hyper-inflation](http://blogarchive.brembs.net/comment-n683.html){target="_self"}
of
[retractions](https://nsaunders.wordpress.com/2010/11/30/analysis-of-retractions-in-pubmed/){target="_blank"},
we should see a precipitous drop in subscription charges immediately,
should such a policy be enforced.

**UPDATE**: Almost simultaneously, pretty much along the same lines is
Egon's post on [trust in
science](https://chem-bla-ics.blogspot.com/2010/12/trust-has-no-place-in-science.html).

------------------------------------------------------------------------

Wolfe-Simon, F., Blum, J., Kulp, T., Gordon, G., Hoeft, S., Pett-Ridge,
J., Stolz, J., Webb, S., Weber, P., Davies, P., Anbar, A., & Oremland,
R. (2010). A Bacterium That Can Grow by Using Arsenic Instead of
Phosphorus Science DOI:
[10.1126/science.1197258](https://dx.doi.org/10.1126/science.1197258){rev="review"}


<em>
 During my flyfishing vacation last year, pretty much nothing was happening on this blog. Now that I’ve migrated the blog to WordPress, I can actually schedule posts to appear when in fact I’m not even at the computer. I’m using this functionality to re-blog a few posts from the archives during the month of august while I’m away.
</em>

Rogue Scholar Posts

Replacing the prestige signal

Nature reviewers endorse hype

Why this GlamMag will likely not ask for my review again

No need to only send your best work to Science Magazine

Even the most thorough peer-review at the ‘best’ journals not up to snuff?

Unquestioning dogma: the gatekeepers of science