Real data gives more equations than unknowns, and measurement noise means no exact solution exists. Ax = b has no answer at all.
Step 1: Let's Learn
Read it, or press Listen and follow the words.
Asking a better question
Rather than demanding an exact solution, ask for the x making Ax as close to b as possible.
It is a projection
The closest reachable vector is the projection of b onto the column space of A. That is the geometric answer.
The residual
The gap between b and Ax is the residual. Least squares makes the total of the squared residuals as small as possible.
Where you have met it
A line of best fit is least squares. Every regression in statistics is this computation.
Where else it appears
Computer graphics, image compression, page ranking, machine learning and quantum mechanics all rest on the ideas of this course.
When there is no exact answer
An overdetermined system — more equations than unknowns — usually has no solution. Least squares finds the vector minimising the total squared error instead, which is the best available answer.
It is a projection
The best approximation is the projection of b onto the column space of A. The problem has no solution because b lies outside that space, and the projection is the nearest point inside it.
The normal equations
AᵀAx = Aᵀb characterises the least squares solution. The transpose appears because the error must be orthogonal to the column space, which is what minimising requires.
This is linear regression
Fitting a line to data is a least squares problem, and so is every multiple regression. The statistics course computes it with formulas; linear algebra explains where those formulas come from.
Step 2: Try It Yourself
Tap and try it out.
The slope is 1: for every 1 across, the line goes 1 up.
Step 3: Watch an Example
One step at a time.
Watch Bilal Reframe an Impossible Problem
Bilal has 50 data points and a 2-parameter line to fit, so Ax = b has no exact solution.
- Step 1
He notes that 50 equations in 2 unknowns will not be satisfiable exactly.
Step 4: Your Turn
Practice makes it stick.
The Overdetermined System
Problem 1 of 2
40 equations in 3 unknowns. Is an exact solution likely? 1 yes, 0 no.
The Residual
Problem 2 of 2
A prediction of 12 against an observed value of 15. What is the residual?
Best, Not Exact
1 of 8
A prediction of 20 against an observed 18. What is the residual?
2 of 8
Residuals of 3 and 4. What is the total of their squares?
3 of 8
Least squares minimises the total of the squared what? Enter 1 for residuals, 2 for slopes.
4 of 8
Every residual is 0. Was the system solvable exactly? 1 yes, 0 no.
5 of 8
The closest reachable vector to b is its projection onto what? Enter 1 for the column space, 2 for the null space.
6 of 8
Residuals of 5 and 12. What is the total of their squares?
7 of 8
Order the reasoning behind least squares.
- 1Ask instead for the closest reachable Ax
- 2Project b onto the column space of A
- 3Solve for the x that produces that projection
- 4Notice that Ax = b has no exact solution
8 of 8
Is a line of best fit a least squares solution? 1 yes, 0 no.
Step 5: Quick Check
Show what you know.
Question 1 of 2
Residuals of 6 and 8. What is the total of their squares?
Question 2 of 2
What does least squares find?
What You Learned
- Real data gives more equations than unknowns, so Ax = b usually has no exact solution.
- Least squares finds the x making Ax as close to b as possible, by projecting onto the column space.
- Every line of best fit and every regression in statistics is this computation.