Tiny Volt21. Symmetric matrix-vector product
The projection matrix chapter showed that for any unit vector , the matrix projects vectors onto the 1-dimensional subspace of . This chapter combines that fact with the eigendecomposition of symmetric matrices to give a beautiful picture of what a symmetric matrix does to a vector.
Recall that an -by- symmetric matrix has real eigenvalues and the corresponding eigenvectors are all perpendicular to each other. This observation provides an elegant insight into what happens when a symmetric matrix is multiplied with a vector.
Also recall the eigendecomposition of a matrix where is the matrix of unit eigenvectors of and is a diagonal matrix where each diagonal element is an eigenvalue of . If is symmetric, we have which gives us .
Finally, recall that for any unit vector , the matrix is the projection matrix corresponding to the subspace described by . In other words, for any vector , the projection of on the subspace given by is .
Armed with this knowledge, we rewrite the symmetric matrix in a way that illuminates the relationship between projections and matrix-vector product.
21.1 Yet another point of view of matrix-matrix product
Here we will prove that:
But before we prove it, let's understand the problem clearly:
- The vectors are all column vectors. Thus is a row vector.
- This makes the product an -by- matrix.
- Thus the right hand side is a sum of matrices.
Now let's prove it.
21.1.1 The proof
We will prove it only for two dimensions to avoid messy sums. But the line of reasoning applies to vectors of any dimension.
Let's assume:
Then we have:
The proof for follows the same line of reasoning. It just includes many more sums. Now let's apply this to the eigendecomposition of a symmetric matrix .
21.2 Revisiting the eigendecomposition of a symmetric matrix
We have where:
Note that the column vectors of viz. are all unit vectors. Then we have:
Now recall that is the projection matrix of the subspace of . Thus we can rewrite the above expression as:
21.3 Matrix-vector product in terms of projections
Given a vector , we can write the matrix-vector product as:
In a less technical language, we can write it as:
Thus, the product of a symmetric matrix with a vector is the sum of projections of on the eigenvectors scaled by their respective eigenvalues.
21.4 A visualization of this phenomenon
We use the matrix as an example here. This matrix has eigenvalues and , and the corresponding unit eigenvectors are given by the columns of the matrix .
We can write this matrix as:
Let's see this decomposition act on a vector on the canvas. First draw as grey lines, perpendicular to each other since is symmetric . Then take shown in rose. Now project on the two subspaces: are shown in teal. Next, scale each projection by its eigenvalue and add the results tip-to-tail: in gold. Their sum is , again in rose.
Note how the first projection barely shrinks and flips (since ) while the second one stretches (since ). The tip of moves along , shown in grey, while the tip of stays on , shown in rose. Now and watch the projections, their scaled copies and the output all update together. You can also .
- When lies on one of the subspaces, its projection on the other subspace vanishes, and : the input is an eigenvector.
- Everywhere else, is a weighted mix of the two projections, weighted by the eigenvalues.
In the next chapter we will use this projection picture to understand a special family of symmetric matrices: positive semi-definite matrices.