Introduction to Regression
Analysis
Correlation is often
but a first step in multivariate analysis. Suppose that we
have some reason to believe that A is a cause of B, and we
would like to predict B when we know A. Correlation can tell
us if there is association, but that is all it can do.
However, regression analysis can take us far beyond
that.
Regression analysis is a way to fit a line through a
scatter of points. For example, in the diagram below some
upwardsloping line would fit through the points better than
a downwardsloping line. However, there are an infinite
number of upwardsloping lines that we could draw. Which one
is the best one?
Regression gives us a line, which will be an equation of
the form Y = a + bX, that has three desirable properties.
First, when we find the errors between what the line
predicts for Y and the actual Ys, and we sum these errors,
they will sum to zero. This is also a property of the mean
when we have only one variable, and it is desirable because
it tells us that the regression line does not on average
predict too high or too low.
A second property is a bit harder to appreciate. If we
look at the correlation between the X values (which are the
independent values that we are using to make the predictions
of Y) and the errors, we find that the correlation is zero.
This means that we have extracted all the information about
Y from the X values. If there were still information
available in the X values, there would be some other line
using this information that would predict better.
A third property of the line is that if we square the
error terms and sum them, we get the smallest possible sum
of squared errors. (For this reason, the procedure is often
called "leastsquares regression.") This is also a property
of the mean when we deal with one variable: the mean also
minimizes the sum of squared errors.
Most textbooks include formulas for finding a and b in
the equation Y = a + bX. However, in practice no one
computes regression lines manually (though it makes a good
exercise for students to do once or twice). The computations
for realistic problems are too laborious and prone to error.
One of the first uses of computers when they entered
universities and businesses in the 1950s and 1960s was for
statistical computation. Without computers most modern
statistical analysis would not be done.
Statistical programs not only compute the regression
line, but a host of other numbers. One of the most important
is the Rsquare. In simple, twovariable regressions, it is
the square of the correlation. It also represents the amount
of variation in Y that the regression line can explain. So
if the Rsquare is .10, then the line explains only 10% of
the original variation, which in most cases is not a lot.
However, if it is .96, then the line explains 96% of the
original variation, which in most cases is considered to be
good. The Rsquared is often the first number that
researchers look at, and it is almost always included with
any published results.
