Mark Rubin

In this paper (Rubin, 2022), I make two related points: (1) researchers should halve two-sided

 p

values if they wish to use them to make directional claims, and (2) researchers should

 not

halve their alpha level if they’re using two one-sided tests to test two directional null hypotheses.

One-Sided Significance Tests

Researchers often distinguish between:  (1) Exploratory hypothesis tests — unplanned tests of post hoc hypotheses that may be based on the current results, and  (2) Confirmatory hypothesis tests — planned tests of a priori hypotheses that are independent from the current results  This distinction is supposed to be useful because exploratory results are assumed to be more “tentative” and “open to bias” than confirmatory results.

Exploratory hypothesis tests can be more compelling than confirmatory hypothesis tests

In this paper (Rubin, 2017), I consider Gelman and Loken’s (2013, 2014) garden of forking paths problem. Forking paths occur when researchers decide which analyses to perform based on information from their sample.

An Evaluation of Four Solutions to the Garden of Forking Paths Problem

The Costs of HARKing: Does it Matter if Researchers Engage in Undisclosed Hypothesizing After the Results are Known? While no-one’s looking, a Texas sharpshooter fires his gun at a barn wall, walks up to his bullet holes, and paints targets around them. When his friends arrive, he points at the targets and claims he’s a good shot.

The Costs of HARKing: Does it Matter if Researchers Engage in Undisclosed Hypothesizing After the…

In this new article, I consider questionable research practices in the field of metascience. A

 questionable metascience practice

(QMP) is a research practice, assumption, or perspective that’s been questioned by several commentators as being potentially problematic for metascience and/or the science reform movement. I discuss 10 QMPs that relate to criticism, replication, bias, generalization, and the characterization of science.

Questionable Metascience Practices

Preregistration entails researchers registering their planned research hypotheses, methods, and analyses in a time-stamped document before they undertake their data collection and analyses. This document is then made available with the published research report in order to allow readers to identify discrepancies between what the researchers originally planned to do and what they actually ended up doing.

Does Preregistration Improve the Interpretablity and Credibility of Research Findings?

In this new paper (Rubin, 2021), I consider when researchers should adjust their alpha level (significance threshold) during multiple testing and multiple comparisons. I consider three types of multiple testing (disjunction, conjunction, and individual), and I argue that an alpha adjustment is only required for one of these three types.

When to Adjust Alpha During Multiple Testing

In this paper (Rubin, 2021), I consider two types of Type I error probability. The

 Neyman-Pearson Type I error rate

refers to the maximum frequency of incorrectly rejecting a null hypothesis if a test was to be repeatedly reconducted on a series of different random samples that are all drawn from the exact same null population. Hence, the Neyman-Pearson Type I error rate refers to a long run of

 exact

replications.

What Type of Type I Error?

In this paper (Rubin, 2020), I consider Fisher’s criticism that the Neyman-Pearson approach to hypothesis testing relies on the assumption of “repeated sampling from the same population.” This criticism is problematic for the Neyman-Pearson approach because it implies that test users need to know, for sure, what counts as the same or equivalent population as their current population.

Repeated Sampling from the Same Population?

Stories by Mark Rubin on Medium

In this paper ([Rubin,
2022](https://drive.google.com/file/d/1Qww86RX2l6LSz9jkMvjJ4Rs2hp38mwJp/view?usp=sharing)),
I make two related points: (1) researchers should halve two-sided *p*
values if they wish to use them to make directional claims, and (2)
researchers should *not* halve their alpha level if they're using two
one-sided tests to test two directional null hypotheses.

<figure>
<img
src="https://cdn-images-1.medium.com/max/582/1*Zc7i5746Q9-Uvt7KCytSFQ.jpeg" />
<figcaption>Sometimes, two-sided tests are called “two-tailed” tests!
(I’ll get my coat!) Source: <a
href="https://www.britannica.com/animal/ring-tailed-lemur">https://www.britannica.com/animal/ring-tailed-lemur</a></figcaption>
</figure>

#### (1) Researchers should halve two-sided p values when making directional claims {#researchers-should-halve-two-sided-p-values-when-making-directional-claims}

Researchers sometimes conduct two-sided significance tests and then use
the resulting two-sided *p* values to make directional claims. I argue
that this approach is inappropriate because two-sided *p* values refer
to *non-directional* hypotheses, rather than *directional* hypotheses.

So, for example, if you conduct a two-sided *t* test and obtain a
significant two-sided *p* value, then your significant result refers to
a non-directional null hypothesis (e.g., "men have *the same*
self-esteem as women"), and you should make a corresponding
non-directional claim (e.g., "men and women have significantly
*different* self-esteem"). If you wish to make a *directional* claim
(e.g., "men have significantly *higher* self-esteem than women"), then
you should *halve* your two-sided *p* value to obtain a one-side
*p *value.

This first point is important because, if you use a two-sided *p* value
to make a decision about a directional null hypothesis, then (a) your
evidence will be weaker than it should be (i.e., your *p* value will be
too large), and (b) your Type II error rate will be higher than
necessary. For the same view, please see Georgi Georgiev's onesided.org
website [here](https://www.google.com/url?q=https%3A%2F%2Fwww.onesided.org%2F&sa=D&sntz=1&usg=AOvVaw2JD1rf-0wwN37D48nu3saz).

#### (2) Researchers should not halve their alpha level when using two one-sided tests {#researchers-should-not-halve-their-alpha-level-when-using-two-one-sided-tests}

I also argue that, if you use two one-sided tests to test two
directional null hypotheses, then it's not necessary to adjust your
alpha level to compensate for multiple testing, because your decision
about rejecting each directional hypothesis is based on a *single* test
result, rather than *multiple* test results.

For example, imagine that you use a one-sided test to test the
directional null hypothesis that "men have the same or lower self-esteem
than women." In this case, there's no need to lower your alpha level
(e.g., from .050 to .025), because your Type I error rate only refers to
a single test of a single null hypothesis. It doesn't refer to either
(a) the other directional null hypothesis (i.e., "men have the same or
*higher* self-esteem than women") or (b) the non-directional null
hypothesis (i.e., "men have the *same* self-esteem as women)."
Consequently, no alpha adjustment is required. For similar views, please
see Georgi Georgiev's piece
[here](https://www.google.com/url?q=https%3A%2F%2Fwww.onesided.org%2Farticles%2Fthe-paradox-of-one-sided-v-two-sided-tests-of-significance.php&sa=D&sntz=1&usg=AOvVaw0ijf7GBDf01WIfbBgSoCAR)
and my paper on multiple
testing [here](https://sites.google.com/site/markrubinsocialpsychresearch/replication-crisis/multiple-testing).

For further information, please see:

Rubin, M. (2022). That's not a two-sided test! It's two one-sided tests!
*Significance, 19*(2), 50--53. [**Publisher's
version**](https://www.google.com/url?q=https%3A%2F%2Fdoi.org%2F10.1111%2F1740-9713.01619&sa=D&sntz=1&usg=AOvVaw3zf1JzRR-EQYkn74ob-tyV)**.**
[**Self-archived
version**](https://drive.google.com/file/d/1Qww86RX2l6LSz9jkMvjJ4Rs2hp38mwJp/view?usp=sharing)**.**

![](https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a6947b10385f){width="1"
height="1"}