R Tutorial Series: One-Way ANOVA with Pairwise Comparisons

When we have more than two groups in a one-way ANOVA, we typically want to statistically assess the differences between each group. Whereas a one-way omnibus ANOVA assesses whether a significant difference exists at all amongst the groups, pairwise comparisons can be used to determine which group differences are statistically significant.

Tutorial Files

Before we begin, you may want to download the sample data (.csv) used in this tutorial. Be sure to right-click and save the file to your R working directory. This dataset contains a hypothetical sample of 30 participants who are divided into three stress reduction treatment groups (mental, physical, and medical). The values are represented on a scale that ranges from 1 to 5. This dataset can be conceptualized as a comparison between three stress treatment programs, one using mental methods, one using physical training, and one using medication. The values represent how effective the treatment programs were at reducing participant's stress levels, with higher numbers indicating higher effectiveness.

Beginning Steps

To begin, we need to read our dataset into R and store its contents in a variable.
  1. > #read the dataset into an R variable using the read.csv(file) function
  2. > dataOneWayComparisons <- read.csv("dataset_ANOVA_OneWayComparisons.csv")
  3. > #display the data
  4. > dataOneWayComparisons

The first ten rows of our dataset.

Omnibus Test

One way to begin an ANOVA is to run a general omnibus test. The advantage to starting here is that if the omnibus test comes up insignificant, you can stop your analysis and deem all pairwise comparisons insignificant. If the omnibus test is significant, you should continue with pairwise comparisons.
  1. > #use anova(object) to test the omnibus hypothesis
  2. > #Is there a significant difference amongst the treatment means?
  3. > anova(lm(StressReduction ~ Treatment, dataOneWayComparisons))
The omnibus ANOVA test

Pairwise Comparisons

Since the omnibus test was significant, we are safe to continue with our pairwise comparisons. To make pairwise comparisons between the treatment groups, we will use the pairwise.t.test() function, which has the following major arguments.
  • x: the dependent variable
  • g: the independent variable
  • p.adj: the p-value adjustment method used to control for the family-wise Type I error rate across the comparisons; one of "none", "bonferroni", "holm", "hochberg", "hommel", "BH", or "BY"
The pairwise.t.test() function can be implemented as follows.
  1. > #use pairwise.t.test(x, g, p.adj) to test the pairwise comparisons between the treatment group means
  2. > #What significant differences are present amongst the treatment means?
  3. > pairwise.t.test(dataOneWayComparisons$StressReduction, dataOneWayComparisons$Treatment, p.adj = "none")

Pairwise comparisons of treatment group means

Note that the desired p-adjustment method will vary by researcher, study, etc. Here, we will assume an alpha level of .05 for all tests, effectively making no adjustment for the family-wise Type I error rate.
These results indicate that there is a statistically significant difference between the mental and medical (p = .004) and physical and medical (p = 0.045), but not the mental and physical (p = 0.302) treatments. The treatment means are 3.5 for mental, 3 for physical, and 2 for medical. Subsequently, we are inclined to conclude based on this study that the mental and physical treatments lead to greater stress reduction than the medical method, but that there is insufficient statistical support to determine that either the mental or physical treatment method is superior.

Complete One-Way ANOVA with Pairwise Comparisons Example

To see a complete example of how one-way ANOVA pairwise comparisons can be conducted in R, please download the one-way ANOVA comparisons example (.txt) file.

References

R Documentation (n.d.). Adjust P-values for Multiple Comparisons. Retrieved January 16, 2011 from http://stat.ethz.ch/R-manual/R-devel/library/stats/html/p.adjust.html

2 comments:

  1. what is the advantage of using pairwise.t.test as opposed to a tukey honest significant difference test here. This came up here

    But I am unclear as to when it is appropriate to use a post-hoc integration of the interacting variables to do the t-test

    ReplyDelete
  2. Hi Dave,

    The tests can yield different results, because they use different statistical techniques to calculate significance. I am not qualified to make recommendations about the techniques or explain them from a statistical standpoint (only how to execute them in R and generally apply them in certain contexts). However, I will have a post coming up in a few weeks that demonstrates several methods for conducting pairwise comparisons in R, including HSD.

    I recommend talking to a professional statistician about the merits and circumstances surrounding the different tests. Also, consider that different fields have different standards and that practical application rarely meets the ideals and rigors that statisticians would like to see. Therefore, you should also consult a senior member of your field and the published research to get an idea of what others are doing.

    John

    ReplyDelete