(a) Find a formula for the least squares error (5.35) in terms of an orthonormal basis of the su

(a) Find a formula for the least squares error (5.35) in...

Key Concepts

Orthogonal Basis

An orthogonal basis contains vectors that are mutually perpendicular but not necessarily of unit length. When using an orthogonal basis in least squares approximations, the projection coefficients must be adjusted by dividing the inner product of the target vector and each basis vector by the square of the norm of that basis vector. This generalization accounts for the differing lengths of the basis vectors while still taking advantage of their orthogonality.

Orthonormal Basis

An orthonormal basis consists of vectors that are both orthogonal to each other and of unit length. This simplifies many computations, such as finding projections and least squares approximations, because the inner product of distinct basis vectors is zero and the magnitude of each vector is one. This property allows the projection coefficients to be obtained directly via the inner product of the target vector and each basis vector.

Projection

Projection is the operation of mapping a vector onto a subspace such that the projected vector is the closest approximation to the original vector within that subspace. When the subspace is defined by a basis, the projection can be computed by summing the contributions of each basis element scaled by appropriate coefficients, especially simple when the basis is orthonormal as inner products yield the coefficients directly.

Least Squares Approximation

This concept involves approximating a given vector by a vector in a subspace such that the distance between the given vector and its approximation is minimized. In the context of least squares, the error is measured by the squared Euclidean norm. The goal is to project the vector onto the subspace and find the unique vector within that subspace that minimizes the residual—the difference between the original vector and its projection.

Transcript

00:02 In this question, we are given a matrix a, a vector b, and the implied system of linear equations ax equals b, and we're asked to find the least squares error vector of this system and the least squares error.

00:24 We're also then asked to verify that the least squares error vector is orthogonal to the column space of a.

00:38 So first of all, you may recall that the solution, or at least the least square solution, to the system was discussed in a previous video and is the subject of exercise 5 in the textbook, in this particular section of the textbook.

01:03 And from there, you might remember that the least square solution was found to be 12, negative 3, 9.

01:22 Using that, let's find b minus ax, where x is that solution, and that is by definition our least squares error vector.

01:41 So here is what that looks like in terms of the numbers, which means that we have to start by multiplying the matrix a with the vector x.

02:02 So the first entry will be the dot product of the first column, or sorry, the first row of a with x, so that would be 12 minus 9.

02:15 The second entry is the second row of a dotted with x, so 24 minus 3 minus 18.

02:26 The third is 12 minus 3, and finally, the last is 12 minus 3 minus 9.

02:39 Then, finishing off with a subtraction, we get a result of 3 minus 3, 0, 3.

02:52 From that, we can find the least squares error, the magnitude of that vector.

03:03 So i'll start with the squared error, because that's just the sum of the squares of the elements.

03:13 0 squared is 0, so i won't even write that...

Please give Ace some feedback