Mark Rubin

In this paper (Rubin, 2022), I make two related points: (1) researchers should halve two-sided

 p

values if they wish to use them to make directional claims, and (2) researchers should

 not

halve their alpha level if they’re using two one-sided tests to test two directional null hypotheses.

One-Sided Significance Tests

Researchers often distinguish between:  (1) Exploratory hypothesis tests — unplanned tests of post hoc hypotheses that may be based on the current results, and  (2) Confirmatory hypothesis tests — planned tests of a priori hypotheses that are independent from the current results  This distinction is supposed to be useful because exploratory results are assumed to be more “tentative” and “open to bias” than confirmatory results.

Exploratory hypothesis tests can be more compelling than confirmatory hypothesis tests

In this paper (Rubin, 2017), I consider Gelman and Loken’s (2013, 2014) garden of forking paths problem. Forking paths occur when researchers decide which analyses to perform based on information from their sample.

An Evaluation of Four Solutions to the Garden of Forking Paths Problem

The Costs of HARKing: Does it Matter if Researchers Engage in Undisclosed Hypothesizing After the Results are Known? While no-one’s looking, a Texas sharpshooter fires his gun at a barn wall, walks up to his bullet holes, and paints targets around them. When his friends arrive, he points at the targets and claims he’s a good shot.

The Costs of HARKing: Does it Matter if Researchers Engage in Undisclosed Hypothesizing After the…

In this new article, I consider questionable research practices in the field of metascience. A

 questionable metascience practice

(QMP) is a research practice, assumption, or perspective that’s been questioned by several commentators as being potentially problematic for metascience and/or the science reform movement. I discuss 10 QMPs that relate to criticism, replication, bias, generalization, and the characterization of science.

Questionable Metascience Practices

Preregistration entails researchers registering their planned research hypotheses, methods, and analyses in a time-stamped document before they undertake their data collection and analyses. This document is then made available with the published research report in order to allow readers to identify discrepancies between what the researchers originally planned to do and what they actually ended up doing.

Does Preregistration Improve the Interpretablity and Credibility of Research Findings?

In this new paper (Rubin, 2021), I consider when researchers should adjust their alpha level (significance threshold) during multiple testing and multiple comparisons. I consider three types of multiple testing (disjunction, conjunction, and individual), and I argue that an alpha adjustment is only required for one of these three types.

When to Adjust Alpha During Multiple Testing

In this paper (Rubin, 2021), I consider two types of Type I error probability. The

 Neyman-Pearson Type I error rate

refers to the maximum frequency of incorrectly rejecting a null hypothesis if a test was to be repeatedly reconducted on a series of different random samples that are all drawn from the exact same null population. Hence, the Neyman-Pearson Type I error rate refers to a long run of

 exact

replications.

What Type of Type I Error?

In this paper (Rubin, 2020), I consider Fisher’s criticism that the Neyman-Pearson approach to hypothesis testing relies on the assumption of “repeated sampling from the same population.” This criticism is problematic for the Neyman-Pearson approach because it implies that test users need to know, for sure, what counts as the same or equivalent population as their current population.

Repeated Sampling from the Same Population?

Stories by Mark Rubin on Medium

In this [new article](https://doi.org/10.36850/mr4), I consider
questionable research practices in the field of metascience. A
*questionable metascience practice* (QMP) is a research practice,
assumption, or perspective that's been questioned by several
commentators as being potentially problematic for metascience and/or the
science reform movement. I discuss 10 QMPs that relate to criticism,
replication, bias, generalization, and the characterization of science.
My aim is not to cast aspersions on the field of metascience but to
encourage a deeper consideration of its more questionable research
practices, assumptions, and perspectives.

<figure>
<img
src="https://cdn-images-1.medium.com/max/666/1*gUc7USykxjwSsshmLL51hQ.jpeg" />
</figure>

### **10 QMPs** {#qmps}

(1)️ **Rejecting or ignoring self-criticism**: Rejecting or ignoring
criticisms of metascience and/or science reform

(2)️ **Fast 'n' bropen scientific criticism**: A quick, superficial,
dismissive, and/or mocking style of scientific criticism

(3)️ **Overplaying the role of replication**: Assuming that replication
is essential to science, and that it indexes "the truth"

(4)️ **Unspecified replication rate targets**: Assuming a replication
rate is "too low" without specifying an "acceptable" rate

(5)️ **Metabias**: An unacknowledged bias towards explaining the
replication crisis in terms of researcher bias

(6)️ **A bias reduction assumption**: Focusing on selective reporting as
the primary form of researcher bias and assuming that it can be reduced
without increasing other forms of bias

(7)️ **Devaluing exploratory results**: Devaluing an exploratory result
as being more "tentative" than a confirmatory result without considering
other relevant issues (e.g., quality of associated theory, methods,
analyses, transparency)

(8)️ **Presuming QRPs are problematic**: Presuming that questionable
research practices (e.g., HARKing) are always problematic research
practices

(9)️ **A focus on knowledge accumulation**: Conceiving knowledge
accumulation as the primary objective of science without considering (a)
the role of specified ignorance or (b) different objectives in other
philosophies of science

\(10\) **Homogenizing science**: Focusing on specific approaches (e.g.,
quantitative methods; replicable effects) as "the scientific method"

### **A Caveat**

In my article, I stress that only *some* metascientists engage in *some*
QMPs *some* of the time, and that these QMPs may not *always* be
problematic. Research is required to estimate the prevalence and impact
of QMPs. In the meantime, I think that QMPs should be viewed as
invitations to ask questions about how we go about doing better
metascience.

<figure>
<img
src="https://cdn-images-1.medium.com/max/500/1*wEXB4y5pQWq9xASq52Jrew.jpeg" />
</figure>

### **The Article**

Rubin, M. (2023). Questionable metascience practices. *Journal of Trial
and Error.* <https://doi.org/10.36850/mr4>

### My Other Work in This Area

For my other work in the area of metascience, please see:
[https://sites.google.com/site/markrubinsocialpsychresearch/replication-crisis](https://www.blogger.com/blog/post/edit/7326905493335343666/6062434439681064559#)

![](https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cceefa31ce46){width="1"
height="1"}

Researchers often distinguish between:

\(1\) Exploratory hypothesis tests --- unplanned tests of post hoc
hypotheses that may be based on the current results, and

\(2\) Confirmatory hypothesis tests --- planned tests of a priori
hypotheses that are independent from the current results

This distinction is supposed to be useful because exploratory results
are assumed to be more "tentative" and "open to bias" than confirmatory
results. In this [paper](https://doi.org/10.1080/09515089.2022.2113771),
we challenge this assumption and argue that exploratory results can be
***more*** compelling than confirmatory results.

Our article has three parts. In the first part, we demonstrate that the
same data can be used to generate and test a hypothesis in a
transparently valid manner. We agree that circular reasoning can
invalidate some exploratory hypothesis tests. However, circular
reasoning can be identified by checking the ***contents*** of the
reasoning without knowing the ***timing*** of that reasoning (i.e., a
priori or post hoc).

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/1*bzE3T0N_RJYeBZBUZPIDXw.png" />
<figcaption>Figure 1. An illustration of two ways in which exploratory
data analyses may provide legitimate support for post hoc
hypotheses</figcaption>
</figure>

In the second part of our article, we argue that exploratory hypothesis
tests can have several evidential ***advantages*** over confirmatory
tests and, consequently, they have the potential to deliver more
compelling research conclusions. In particular, exploratory hypothesis
tests:

✅ avoid researcher commitment and prophecy biases

✅ reduce the motive for data fraud

✅ are more appropriate following unplanned deviations

✅ facilitate inference to the best explanation

✅ allow peer reviewers to contribute to exploratory analyses

Finally, in the third part of our article, we consider several potential
\*disadvantages\* of exploratory hypothesis tests and conclude that
these potential disadvantages may not be problematic. In particular,
exploratory hypotheses tests are not necessarily disadvantaged due to:

✅overfitting

✅bias

✅HARKing

✅unacceptable research practices

And they:

✅are usually necessary

✅can be falsified

✅can predict anything but may suffer an evaluative cost in doing so

To be clear, our claim is not that exploratory hypothesis tests are
***always*** more compelling than confirmatory tests or even that they
are ***typically*** more compelling. Our claim is only that exploratory
tests ***can be*** more compelling in specific research situations. More
generally, we encourage researchers to evaluate specific tests and
results on a case-by-case basis rather than to follow simplistic
heuristics such as "exploratory results are more tentative," which
represents a form of methodolatory.

Our paper builds on some of my previous work on
[preregistration](https://doi.org/10.20982/tqmp.16.4.p376) and
[HARKing](https://drive.google.com/file/d/1bGIUjHSEAoJYJke6RWtBphXJjZLr1UeX/view).
And please check out Szollosi and Donkin's (2021)
[paper](https://doi.org/10.1177/1745691620966796) on "the misguided
distinction between exploratory and confirmatory research."

For more info, please see our open access article:

Rubin, M., & Donkin, C. (2022). Exploratory hypothesis tests can be more
compelling than confirmatory hypothesis tests. *Philosophical
Psychology*. <https://doi.org/10.1080/09515089.2022.2113771>

![](https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8f55d6d52d09){width="1"
height="1"}

In this paper ([Rubin,
20](https://www.google.com/url?q=https%3A%2F%2Fdoi.org%2F10.1007%2Fs11229-019-02433-0&sa=D&sntz=1&usg=AFQjCNEVuCuq3elOVD71HY_Ee1DN_5q1Og)21),
I consider two types of Type I error probability. The *Neyman-Pearson
Type I error rate* refers to the maximum frequency of incorrectly
rejecting a null hypothesis if a test was to be repeatedly reconducted
on a series of different random samples that are all drawn from the
exact same null population. Hence, the Neyman-Pearson Type I error rate
refers to a long run of *exact* replications. In contrast, the
*Fisherian Type I error probability* is the probability of incorrectly
rejecting a null hypothesis in relation to a hypothetical population
that reflects the relevant characteristics of the particular sample
under consideration. Hence, the Fisherian Type I error rate refers to a
one-off sample rather than a series of samples that are drawn during a
long run of exact replications.

I argue that social science deals with units of analysis (people, social
groups, and social systems) that change over time. As the Greek
philosopher Heraclitus put it: "No man ever steps in the same river
twice, for it's not the same river and he's not the same man." Rivers
and men are what Schmidt (2009, p. 92) called "irreversible units" in
that they are complex time-sensitive systems that accumulate history.
The scientific investigation of these irreversible units cannot proceed
on the assumption that exact replications are possible. Consequently,
the Neyman-Pearson Type I error rate is irrelevant in social science,
because it relies on a concept of exact replication that cannot take
place in the case of people, social groups, and social systems. Why
should social scientists be interested in an error rate for an
impossible process of resampling from the same fixed and unchanging
population?

<figure>
<img
src="https://cdn-images-1.medium.com/max/554/1*zfDlYm8FEwiXhSQ8QrnI2g.jpeg" />
<figcaption>Source: <a
href="https://images.app.goo.gl/RiEP5tPX75AVbmFy6">https://images.app.goo.gl/RiEP5tPX75AVbmFy6</a></figcaption>
</figure>

I argue that the Fisherian Type I error probability is more appropriate
in social science because it refers to one-off samples from hypothetical
populations. In this case, researchers recognise that every sample comes
from a potentially different population. Hence, researchers can apply
the Fisherian Type I error probability to each sample-specific
provisional decision that they make about rejecting the same substantive
null hypothesis in a series of *direct* replications.

I conclude that the replication crisis may be partly (not wholly) due to
researchers' unrealistic expectations about replicability based on their
consideration of the Neyman-Pearson Type I error rate across a long run
of exact replications.

For further information, please see:

Rubin, M. (2021). What type of Type I error? Contrasting the
Neyman-Pearson and Fisherian approaches in the context of exact and
direct replications. *Synthese, 198,* 5809--5834*. *.
[https://doi.org/10.1007/s11229-019-02433-0](https://www.google.com/url?q=https%3A%2F%2Fdoi.org%2F10.1007%2Fs11229-019-02433-0&sa=D&sntz=1&usg=AFQjCNEVuCuq3elOVD71HY_Ee1DN_5q1Og)
\*[Self-archived
version](https://drive.google.com/file/d/10mEkuIElCGE5lYO-24HCYS8NKfytZBpG/view?usp=sharing)\*

I also discuss the differences between the Fisherian and Neyman-Pearson
approaches to hypothesis
testing [here](https://sites.google.com/site/markrubinsocialpsychresearch/replication-crisis/repeated-sampling).

![](https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a32bac4af348){width="1"
height="1"}