Tiny Volt14. Eigenvectors and eigenvalues
To keep a normal vector perpendicular after a transformation , we transform it not by but by the inverse-transpose . The reason came from the dot product: , which collapses to exactly when . In this chapter we ask a different question about a matrix acting on a vector, and it opens a door to some of the most beautiful ideas in linear algebra.
What happens when the input vector points in the same direction as the output vector for a given matrix? This simple question leads to a treasure trove of beautiful mathematical concepts.
14.1 Direction of input vector = direction of output vector
An matrix can be seen as a function whose input vector and output vector are both -dimensional.
If both the input and the output point in the same direction, it means the output is just a scaled copy of the input . In other words,
where is a scalar number. The vector is called an eigenvector of , and the number is called the eigenvalue belonging to that eigenvector.
Note that we only care about the direction of the eigenvector , not its length: if is an eigenvector, so is every vector pointing along the same line, and each of them is scaled by the same .
14.1.1 Visualizing (eigen)vectors for 2-by-2 matrices
Before the algebra, let's build a non-rigorous intuition. We want to see the output vector for an input vector in every possible direction. For a 2-by-2 matrix, we can do this by watching how the points on a circle are transformed:
- Sample many points uniformly on a circle of fixed radius. Each point is an input vector.
- Compute the output vector for each input vector .
- For each pair , draw a line from the input point to the output point.
Where the output lands right on top of (or directly opposite) the input direction, we have found an eigenvector. Let's start with the matrix
The matrix , each colored segment joins an input point to its output point. Notice that most segments are slanted: the output points in a different direction than the input. But along a couple of special directions the segments lie flat against a radial line, meaning input and output are collinear there.
Reveal as two lines through the origin. Along each of these lines the transformation only stretches or shrinks: the eigenvalues are and .
To make this concrete, draw . Right now the input sits on the first eigenvector, so the two arrows are collinear and is simply stretched by . and watch the gold output swing away from the teal input almost everywhere. and the two arrows fall onto the same violet line again, this time with the output shorter than the input ().
So has two eigenvectors. Not every matrix is so lucky.
Next take the shear matrix
which . This vertical shear pushes each point up by an amount equal to its -coordinate, so the only direction that stays put is the y-axis (where ). Reveal , with : every point on the y-axis maps exactly to itself.
Draw : on the y-axis they coincide perfectly. and, apart from that one vertical direction, the output never lines up with the input again. So has exactly one eigenvector.
Finally take a scaled rotation
which . Because a rotation moves every direction, there is no direction left pointing where it started.
Draw and : the gold output is always turned away from the teal input by . There is , so has zero (real) eigenvectors.
14.1.2 Maximum number of unique eigenvalues
In general, an matrix can have at most unique eigenvalues. So a 2-by-2 matrix can have at most two, and as the three examples above show, it may have two, one, or none. To see why the ceiling is , we need a little algebra.
14.2 The algebra of
For any vector we can write , where is the identity matrix:
Let's use this to solve for a 2-by-2 matrix . Moving everything to one side:
Since is a non-zero vector, the matrix must be singular. This means its column vectors point in the same direction. In other words, the vectors and are parallel.
Two parallel vectors share the same perpendicular, so must be perpendicular to (a vector perpendicular to ). Their dot product is therefore zero:
This equation is quadratic in , so it can have at most two solutions. That is exactly why a 2-by-2 matrix has at most two unique eigenvalues. In general, the analogous equation for an matrix is a degree- polynomial, giving at most unique eigenvalues.
Given a 2D vector , the vector (for any number ) is perpendicular to it, because their dot product vanishes: . Rotating a vector by is exactly this swap-and-negate operation.
14.3 The (superficial) geometry of
We just saw that if is an eigenvalue of , then is a singular matrix. So:
- If has two unique eigenvalues, is singular for those two values of .
- If has one unique eigenvalue, it is singular for that one value.
- If has no eigenvalues, is never singular.
We have seen before that a singular 2-by-2 matrix collapses the entire 2D plane onto a 1D line. So let's watch as we vary , and look for the moments where the grid flattens into a line. Starting with our matrix :
The starts at , where it is just itself and the grid fills the plane. and watch the grid squash flat twice on the way: once near and again near . Each flattening is a value of that makes singular, which is precisely an eigenvalue. to reset.
For the shear , the flattens exactly once. and the grid collapses precisely at , its single eigenvalue, then .
Finally the scaled rotation , whose never flattens. and the grid keeps its full 2D spread the entire time: the matrix is never singular, so has no real eigenvalues. .
14.4 Let's focus on the best case
Note that if two eigenvalues of a matrix are different, their eigenvectors point in different directions too. So when an matrix has unique eigenvalues, it also has eigenvectors pointing in different directions.
What if we use these eigenvectors as the basis vectors of a new coordinate system? That single idea unlocks eigendecomposition, and we will follow it in the next chapter.
In the next chapter we study eigendecomposition: using a full set of eigenvectors as a basis to factor a matrix into a diagonal core, where its action is nothing more than scaling along each eigen-direction.