The least-squares regression line

Lines of best fit, residuals, the least-squares regression line and its formulas in terms of r and the standard deviations, the line passing through the mean point, and interpreting the gradient and intercept in context.

Residuals and least squares

A line of best fit $\hat{y} = a + bx$ gives a predicted value $\hat{y}$ for each $x$. The residual of a data point is $$\text{residual} = y - \hat{y},$$ the vertical distance from the point to the line. The least-squares regression line is the line that makes the sum of the squared residuals as small as possible.

Find the best line

Adjust the gradient b and intercept a. The least-squares line makes the sum of squared residuals as small as possible: about 13.9.

Try this

Try $b = 3.7$ and $a = 48$. How close can you get to $13.9$?

Finding the line

Least-squares regression line

$$\hat{y} = a + bx \qquad b = r\,\frac{s_y}{s_x} \qquad a = \bar{y} - b\bar{x}$$

Here $s_x$ and $s_y$ are the standard deviations of $x$ and $y$. The second formula shows that the line always passes through the mean point $(\bar{x}, \bar{y})$. In practice a calculator gives $a$, $b$ and $r$ directly from the data.

Example 1●●

For the study data, $\bar{x} = 3.5$, $\bar{y} = 61$, $s_x \approx 1.871$, $s_y \approx 7.043$ and $r \approx 0.9715$. Find the least-squares line.

$b = 0.9715 \times \dfrac{7.043}{1.871} \approx 3.66$.

$a = 61 - 3.657 \times 3.5 \approx 48.2$.

So $\hat{y} = 48.2 + 3.66x$.

Interpreting the line

The gradient $b$ is the predicted change in $y$ for each increase of $1$ in $x$. Here, each extra hour of study adds about $3.66$ marks.
The intercept $a$ is the predicted $y$ when $x = 0$. Here, a student who doesn't study is predicted to score about $48$. It only has a sensible meaning when $x = 0$ is a reasonable value.

Flashcards

Residual

$y - \hat{y}$, the actual value minus the predicted value.

Gradient of the least-squares line

$b = r\,\dfrac{s_y}{s_x}$

Intercept of the least-squares line

$a = \bar{y} - b\bar{x}$

The point every least-squares line passes through

$(\bar{x}, \bar{y})$

Practice

$r = 0.8$, $s_x = 5$ and $s_y = 10$. Find the gradient $b$.
$b = 2$, $\bar{x} = 8$ and $\bar{y} = 50$. Find the intercept $a$.
$r = -0.6$, $s_x = 4$ and $s_y = 10$. Find $b$.
A data point is $(3, 61)$, and the line predicts $\hat{y} = 59$ at $x = 3$. Find the residual.
A data point is $(4, 60)$ and the line predicts $63$. Find the residual.
The least-squares line minimises:
For $\hat{y} = 48.2 + 3.66x$ (score against hours of study), the gradient means:
Every least-squares regression line passes through:
If $r$ is negative, the gradient of the least-squares line is: