R Tutorial Series: Basic Polynomial Regression

Often times, a scatterplot reveals a pattern that seems not so linear. Polynomial regression can be used to explore a predictor at different levels of curvilinearity. This tutorial will demonstrate how polynomial regression can be used in a hierarchical fashion to best represent a dataset in R.

Tutorial Files

Before we begin, you may want to download the sample data (.csv) used in this tutorial. Be sure to right-click and save the file to your R working directory. Note that all code samples in this tutorial assume that this data has already been read into an R variable and has been attached. This dataset contains hypothetical student data that uses practice exam scores to predict final exam scores.

Scatterplot

The preceding scatterplot demonstrates that these data may not be linear. Notably, no one scored lower than 50 on the practice exam and at approximately the 85 and above practice mark, final exam scores taper off. These suggest that the data is curvilinear. Furthermore, since exam scores range between 0 to 100, it is not possible to observe nor appropriate to predict that an individual with a 150 practice score would have a certain final exam score.

Creating The Higher Order Variables

A two step process, identical to the one used to create interaction variables, can be followed to create higher order variables in R. First, the variables must be centered to mitigate multicollinearity. Second, the predictor must be multiplied by itself a certain number of times to create each higher order variable. In this tutorial, we will explore the a linear, quadratic, and cubic model. Therefore, the predictor will need to be squared to create the quadratic model and cubed to create the cubic model.

Step 1: Centering

To center a variable, simply subtract its mean from each data point and save the result into a new R variable, as demonstrated below.

1. > #center the independent variable
2. > FinalC <- Final - mean(Final)
3. > #center the predictor
4. > PracticeC <- Practice - mean(Practice)

Step 2: Multiplication

Once the input variable has been centered, the higher order terms can be created. Since a higher order variable is formed by the product of a predictor with itself, we can simply multiply our centered term from step one and save the result into a new R variable, as demonstrated below.

1. > #create the quadratic variable
2. > PracticeC2 <- PracticeC * PracticeC
3. > #create the cubic variable
4. > PracticeC3 <- PracticeC * PracticeC * PracticeC

Creating The Models

Now we have all of the pieces necessary to assemble our linear and curvilinear models.

1. > #create the models using lm(FORMULA, DATAVAR)
2. > #linear model
3. > linearModel <- lm(FinalC ~ PracticeC, datavar)
4. > #quadratic model
5. > quadraticModel <- lm(FinalC ~ PracticeC + PracticeC2, datavar)
6. > #cubic model
7. > cubicModel <- lm(FinalC ~ PracticeC + PracticeC2 + PracticeC3, datavar)

Evaluating The Models

As is the case in other forms of regression, it can be helpful to summarize and compare our potential models using the summary(MODEL) and anova(MODEL1, MODEL2,… MODELi) functions.

1. > #display summary information about the models
2. > summary(linearModel)
3. > summary(quadraticModel)
4. > summary(cubicModel)
5. #compare the models using ANOVA
6. anova(linearModel, quadraticModel, cubicModel)

The model summaries and ANOVA comparison chart are displayed below.

At this point we can compare the models. In this case, the quadratic and cubic terms are not statistically significant themselves nor are their models statistically significant beyond the linear model. However, in a real research study, there would be other practical considerations to make before deciding on a final model.

More On Interactions, Polynomials, and HLR

Certainly, much more can be done with these topics than I have covered in my tutorials. What I have provided is a basic discussion with guided examples. The regression topics covered in these tutorials can be mixed and matched to create exceedingly complex models. For example, multiple interactions and higher order variables could be contained in a single model. The good news is that more complex models can be created using the same techniques covered here. The basic principles remain the same.

Complete Polynomial Regression Example

To see a complete example of how polynomial regression models can be created in R, please download the polynomial regression example (.txt) file.

R Tutorial Series: Hierarchical Linear Regression

Regression models can become increasingly complex as more variables are included in an analysis. Furthermore, they can become exceedingly convoluted when things such as polynomials and interactions are explored. Thankfully, once the potential independent variables have been narrowed down through theoretical and practical considerations, a procedure exists to help us identify which predictors make a significant statistical contribution to our model. Hierarchical linear regression (HLR) can be used to compare successive regression models and to determine the significance that each one has above and beyond the others. This tutorial will explore how the basic HLR process can be conducted in R.

Tutorial Files

Before we begin, you may want to download the sample data (.csv) used in this tutorial. Be sure to right-click and save the file to your R working directory. This dataset contains information used to estimate undergraduate enrollment at the University of New Mexico (Office of Institutional Research, 1990). Note that all code samples in this tutorial assume that this data has already been read into an R variable and has been attached.

Pre-Analysis Steps

Before comparing regression models, we must have models to compare. In the segment on multiple linear regression, we created three successive models to estimate the fall undergraduate enrollment at the University of New Mexico. The complete code used to derive these models is provided in that tutorial. This article assumes that you are familiar with these models and how they were created. Therefore, a shorthand method for generating the models is displayed below.

1. > #create three linear models using lm(FORMULA, DATAVAR)
2. > #one predictor model
3. > onePredictorModel <- lm(ROLL ~ UNEM, datavar)
4. > #two predictor model
5. > twoPredictorModel <- lm(ROLL ~ UNEM + HGRAD, datavar)
6. > #three predictor model
7. > threePredictorModel <- lm(ROLL ~ UNEM + HGRAD + INC, datavar)

Comparing Individual Models

The summary(OBJECT) function can be used to ascertain the overall variance explained (R-squared) and statistical significance (F-test) of each individual model, as well as the significance of each predictor to each model (t-test). The following code demonstrates how to generate summaries for each model.

1. > #get summary data for each model using summary(OBJECT)
2. > summary(onePredictorModel)
3. > summary(twoPredictorModel)
4. > summary(threePredictorModel)

The results of the previous functions are displayed below.

From the summary functions, we can infer that all of the models are statistically significant. Moreover, each one explains more of the overall variance than the previous model. We can also assess the significance of the individual predictors to each equation. Note that, if preferred, similar comparisons could be made by using the anova() function on each model.

Comparing Successive Models

The anova(MODEL1, MODEL2,… MODELi) function can be used to compare the significance of each successive model. The code sample below demonstrates how to use ANOVA to accomplish this task.

1. > #compare successive models using anova(MODEL1, MODEL2, MODELi)
2. > anova(onePredictorModel, twoPredictorModel, threePredictorModel)

The table resulting from the preceding function is pictured below.

Here, we can see that each successive model is significant above and beyond the previous one. This suggests that each predictor added along the way is making an important contribution to the overall model.

More HLR

Undoubtedly, HLR is a complex topic that has only been addressed at the most basic level in this tutorial. Further guides in the series will cover related subjects, such as interactions and polynomial regression. However, individuals whose work requires a deeper inspection into the procedures of HLR are encouraged to seek additional resources (and to consider writing a guest tutorial for this series).

Complete Hierarchical Linear Regression Example

To see a complete example of how HLR can be conducted in R, please download the HLR example (.txt) file.

References

Office of Institutional Research (1990). Enrollment Forecast [Data File]. Retrieved November 22, 2009 from http://lib.stat.cmu.edu/DASL/Datafiles/enrolldat.html