Tiny Volt16. Eigenvectors of symmetric matrices
When a matrix has a full set of eigenvectors, we can gather them as the columns of and factor , where is diagonal and holds the eigenvalues. The eigenvectors form a basis, and in that basis acts as pure scaling. This chapter looks at a family of matrices where this decomposition becomes especially clean.
A square matrix is symmetric if it is its own transpose, . A symmetric matrix always has real eigenvalues, and the corresponding eigenvectors are all perpendicular to each other. This makes such a matrix simpler to work with, and it gives the matrix-vector product an especially elegant interpretation.
16.1 The eigenvalues of and are the same
First, note that and share the same eigenvalues. The eigenvalues of are the values of for which becomes singular. But whenever is singular, is singular too, because a matrix and its transpose are singular together, and
So the two matrices go singular at exactly the same values of , which means they have identical eigenvalues.
Since the eigenvalues are the same, the number of independent eigenvectors is the same for and as well. This does not mean the eigenvectors themselves are the same. It only means that the matrix whose columns are the eigenvectors of has the same rank as the matrix whose columns are the eigenvectors of .
16.2 A symmetric matrix of rank has real eigenvalues
This proof leans on a little knowledge of imaginary numbers, so feel free to skip it and come back later. Let be an eigenvector of with eigenvalue .
Earlier, when we said an eigenvalue "did not exist", what we really meant was that it was an imaginary number rather than a real one. Here we do not yet know whether is real. But we do know that is symmetric, so . Consider the squared length of for an eigenvector :
Now is a real, non-negative number, and so is . For the equation to hold, must be real and non-negative, which forces itself to be real. This argument applies to every eigenvalue of , so every eigenvalue of a symmetric matrix is real.
16.3 A symmetric matrix of rank has perpendicular eigenvectors
Now that we know a rank- symmetric matrix has real eigenvalues, both and have independent eigenvectors. Here is why those eigenvectors turn out to be perpendicular:
- We can write . It is safe to assume every column of has norm , since an eigenvector only cares about direction, not length. This gives us the next line.
- Taking the transpose of both sides:
- But , so comparing the two factorizations gives . A matrix whose transpose equals its inverse is a rotation matrix.
- The columns of a rotation matrix are all perpendicular to one another. Since is a rotation matrix, its columns, which are the eigenvectors of , are mutually perpendicular.
16.4 Let's visualize eigenvectors of symmetric matrices
Below we watch three different symmetric matrices transform points on a circle into points on an ellipse. As in the previous chapters, the grey unit circle holds the input vectors and the grey ellipse is its image under the matrix. The eigenvectors show up as two directions through the origin, drawn as white lines. Because we only care about their direction, they are lines rather than single points.
The new thing to look for: for a symmetric matrix the two eigenvector lines always meet at a right angle. Sweeping the input vector by exactly carries it from one eigenvector straight onto the other.
Start with
which is symmetric because the off-diagonal entries match.
The matrix . Reveal . They cross at : the eigenvalues are along and along .
To see it in motion, draw . The input starts on the first eigenvector, so the arrows are collinear and is just stretched by . and it lands exactly on the second eigenvector, where the two arrows line up again and the output is scaled by . That the sweep needed a clean quarter-turn is the perpendicularity of the eigenvectors made visible. Its unit eigenvectors are and .
Next take
which is again symmetric, but this time one eigenvalue is negative, so that direction gets flipped as well as scaled.
Watch it and reveal . The eigenvalues are along and along .
Draw : on the first eigenvector they point the same way, with the output stretched by . onto the second eigenvector, where the negative eigenvalue makes the gold output point in the opposite direction along the same white line. Its unit eigenvectors are and .
Finally take
whose eigenvector directions are tilted off the coordinate axes, but which are still perpendicular to each other because is symmetric.
and reveal . Even though neither direction is horizontal or vertical, they still meet at a right angle. The eigenvalues are along and along .
Draw starting on the first eigenvector, where the output is stretched by . onto the perpendicular eigenvector, where the small negative eigenvalue shrinks the output and flips it around. Its unit eigenvectors are and .
In all three cases the two eigenvector directions were perpendicular. That is the defining geometric feature of a symmetric matrix, and the next chapter shows what it buys us.
16.4.1 Tracking the eigenvectors as a matrix becomes symmetric
And it really is the symmetry doing the work. To see the right angle switch on, take a matrix whose entries we can steer:
which is not symmetric, since while .
The matrix as before. Reveal : with the angle between them is visibly not a right angle. Everything on the canvas is computed live from the four entries, so now . As closes in on the ellipse re-shapes, both eigenvector lines swing, and at the moment , when becomes symmetric, the marked angle lands on exactly .
In the next chapter we study the powers of a matrix: raising to higher powers through its eigendecomposition, which turns out to be the engine behind Fibonacci numbers, Markov chains, and more.