Minimization of squared norm of a vector sum

Problem statement

Given a vector of the form , where is a matrix and is a vector, we want to find that minimizes the squared norm of the vector.

Find for which is minimized. Here is shown here and the vector sum looks like this. The radius of the circle is the length of the vector sum. The vector with minimum norm is shown in bold.

Solution

We can write the function as:

It then follows that . Setting it to zero, we get the solution:

Note that is the negative projection of onto the column space of .

The optimal value of is one for which is the negative of the projection of onto the column space of .

Ideally, if were equal to , the norm of the sum would be zero. But that may not be possible if is not in the column space of . Simply said, the optimal value of tries to get as close as possible to .