The least-squares line of best fit

Fitting the least-squares regression line with a calculator or from summary statistics, interpreting its gradient and intercept in context, and understanding residuals.

Once you know there's a strong linear association, the next step is to draw a line through the middle of the points and use it to make predictions. Everyone who draws a line by eye draws a slightly different one. The least-squares line is the one line that everyone (and every calculator) agrees on.

Residuals and "least squares"

For each point, the residual is the vertical gap between the actual $y$-value and the value the line predicts:

Residual

$$\text{residual} = \text{actual } y - \text{predicted } y$$

A point above the line has a positive residual and a point below has a negative one. The least-squares line is the line that makes the sum of the squared residuals as small as possible. Squaring stops the positive and negative gaps from cancelling, and it punishes big misses more than small ones.

Finding the line with a calculator

The line is written $y = mx + c$ in the Standard course. Your calculator may use $y = a + bx$, where $b$ is the gradient and $a$ is the intercept. Watch out for the swap.

Example 1●

Eight students recorded hours of study per week ($x$) and their exam mark out of $100$ ($y$).

Hours ($x$)	2	4	5	7	8	10	12	13
Mark ($y$)	48	55	52	63	66	70	79	81

Find the equation of the least-squares line, with values to two decimal places.

The calculator gives $a \approx 40.72$ and $b \approx 3.09$.

Enter the pairs in stats mode and choose linear regression.

$m \approx 3.09$ and $c \approx 40.72$.

In y = a + bx, b is the gradient, so it becomes m.

$\text{mark} = 3.09 \times \text{hours} + 40.72$

Write the line using the variable names, which makes it easier to use later.

Interpreting the gradient and intercept

The numbers in the equation have meanings, and HSC questions ask for them in words.

The gradient $m$ is the predicted change in $y$ for each increase of $1$ in $x$. Here: each extra hour of study per week is linked to about $3.09$ more marks.
The intercept $c$ is the predicted $y$ when $x = 0$. Here: a student who does no study is predicted to score about $40.72$. That only makes sense if $x = 0$ is close to the data. Here the smallest $x$ is $2$, so it's a small stretch, which is acceptable.

The line from summary statistics

Sometimes you are not given the raw data, only the means, standard deviations and $r$. Then use:

Least-squares line from summary statistics

$$m = r\,\frac{s_y}{s_x} \qquad c = \bar{y} - m\bar{x}$$

The gradient formula makes sense: if $y$ is much more spread out than $x$, the line has to be steeper. And $r$ scales it down when the association is weak. The intercept formula comes straight from the line passing through $(\bar{x}, \bar{y})$.

Example 2●●

For a group of adults, height $x$ (cm) and mass $y$ (kg) have $\bar{x} = 168.4$, $s_x = 7.2$, $\bar{y} = 64.1$, $s_y = 8.5$ and $r = 0.82$. Find the least-squares line and predict the mass of someone $175$ cm tall.

$m = 0.82 \times \dfrac{8.5}{7.2} = 0.9681\ldots \approx 0.968$

Gradient: r times the ratio of the standard deviations, y over x.

$c = 64.1 - 0.9681 \times 168.4 = -98.92\ldots \approx -98.92$

Intercept: substitute the means, since the line passes through them. Keep the unrounded m on your calculator.

$\text{mass} = 0.968 \times \text{height} - 98.92$

Write the line in context.

$\text{mass} \approx 0.9681 \times 175 - 98.92 \approx 70.5$ kg

Substitute x = 175, using the unrounded values.

The negative intercept means a person of height $0$ cm would weigh $-98.92$ kg. That's nonsense, and it is fine. $x = 0$ is far outside the data, so the intercept has no real meaning here.

InteractiveDrag the points

Try this

Drag any point and watch $r$ and the least-squares line update. Turn on Show residuals to see the gaps the line is minimising.

Drag one point far above the others. How much does the line tilt? How much does $r$ drop? Then try to arrange the points so that $r$ is close to $0$.

Your turn

For a set of data, $\bar{x} = 20$, $s_x = 4$, $\bar{y} = 50$, $s_y = 10$ and $r = -0.6$. Find the equation of the least-squares line.

Show the answer

$m = -0.6 \times \dfrac{10}{4} = -1.5$. Then $c = 50 - (-1.5)(20) = 50 + 30 = 80$. The line is $y = -1.5x + 80$.

Flashcards

Residual

Actual $y$ minus predicted $y$. Positive means the point is above the line.

Least-squares line from summary statistics

$m = r\dfrac{s_y}{s_x}$ and $c = \bar{y} - m\bar{x}$

Interpreting the gradient

The predicted change in $y$ for each increase of $1$ in $x$, in context and with units.

A point the line always passes through

$(\bar{x}, \bar{y})$

Practice

The least-squares line is the line that:
A point has actual $y = 70$, and the line predicts $67$. What is the residual?
A calculator in $y = a + bx$ mode gives $a = 12.5$ and $b = -0.8$. In $y = mx + c$ form the line is:
The line $\text{cost} = 45 \times \text{hours} + 80$ models a plumber's charges in dollars. The gradient means:
Find $m$ given $r = -0.75$, $s_x = 5$ and $s_y = 10$.
The least-squares line has gradient $m = 2$, and $\bar{x} = 15$, $\bar{y} = 40$. Find $c$.
For a line of age of a car ($x$, years) against value ($y$), the intercept is $c = 31.2$ (thousands of dollars). The best interpretation is:
Which point must lie on the least-squares line?
Exam style: Using $\text{mark} = 3.09 \times \text{hours} + 40.72$, predict the mark of a student who studies $10$ hours per week. Give your answer to one decimal place.
Exam style: Another student, not in the original data, studied $10$ hours per week and scored $75$. Using the prediction from the previous question ($71.6$), find the residual for this student.