Review

The dot product takes two vectors and returns a single number. It is zero exactly when the vectors are perpendicular, and it lurks inside every matrix-vector multiplication: each entry of is the dot product of a row of with .

Transpose of a matrix is another matrix, denoted by or , such that the rows of are the columns of :

Note that a vector can also be seen as a matrix with only one column. Thus we can define the transpose for a vector as well:

---

11.1 Matrix: array of columns or rows?

When you see a matrix, there are two ways you can look at it: an array of columns and an array of rows. So far we have been treating it as an array of columns only.

11.1.1 Matrix-vector multiplication from the POV of columns

Each of these perspectives provides a way to interpret matrix-vector multiplication. From the first perspective, you can imagine the columns of a matrix being scaled and added up:

11.1.2 Matrix-vector multiplication from the POV of rows

From the second perspective, you can think of the rows of the matrix being fed to the dot product function along with the input vector:

Note that the same matrix is colored by columns in the first equation and by rows in the second, the colors are there to visualize the two perspectives.

---

11.2 We have always been taking weighted sums of rows

Consider the matrix-matrix multiplication where and is a 3-by-3 matrix. The first matrix is 1-by-3 and the second matrix is 3-by-3.

• The first way is to view it as three dot products with the columns of M:

• But note that the result is the same as the weighted sum of the rows of M, where the weights come from the vector v:

We started out with taking weighted sums of only the columns of a matrix. The property of weighted sums of the rows of a matrix emerges simply out of weighted sums of columns.

---

11.3 A useful property

This duality between a sum of weighted columns and a sum of weighted rows can be expressed in a very elegant identity:

• On the left hand side, we take a weighted sum of columns and then convert the resulting column into a row.
• On the right hand side, we convert the columns into rows first and then take a weighted sum of those rows.

It is obvious that both lead to the same result. We can also extend this identity to matrix-matrix multiplication (since a matrix-matrix multiplication is just many matrix-vector multiplications):

---

11.4 Three ways to interpret matrix-matrix multiplication

It is useful to be able to switch between these three points of view when looking at a matrix-matrix multiplication . Let the row vectors of be and let the column vectors of be .

1. Product AB = a matrix of weighted sums of columns of A:

2. Product AB = a matrix of weighted sums of rows of B:

3. Product AB = a matrix of dot products of rows of A with columns of B:

---

11.5 Visualizing the column space and row space

Let's visualize some transformations by a 2-by-2 matrix M again. This time, we want to focus on what the rows of M tell us about the transformation.

11.5.1 Making a case for visualizing the transformation of a circle

If we want to see how a vector is transformed by a matrix, it is sufficient to see how the unit vector is transformed. This is because we can simply scale the output of by the norm of :

Next, to fully understand the transformation by a matrix, we don't want just one unit vector. We would ideally like to view how all the unit vectors are transformed. But the unit vectors are exactly the points on a unit circle. Thus, by understanding how the points on a circle are transformed by a matrix, we get a good idea of what that matrix is doing.

We are given the matrix

whose first row is colored and second row .

The matrix maps every point on the circle to a point on an ellipse - each colored segment joins an input point to its output point. This is nothing new. What's interesting here is the relationship between the shape of the ellipse and the row vectors.

• Let's draw the first row vector

• and the second row vector of M as arrows.

Now pick an input vector on the circle and its output vector.

---

11.6 Exploring connection between row vectors and output vector

Note that each output coordinate is a dot product with a row of M:

To visualize this, let's reduce the clutter.

11.6.1 First row-vector

We want to show that the (signed) projection of first row vector on the input vector is equal to the x-coordinate of the output vector.

• Project the first row vector on the input vector.
• Calculate the length of this projection and draw a point on the x-axis at the same distance. Note that this distance is signed.

• This is exactly the x-coordinate of the output vector.

You can try rotating the input vector and see this setup changes according.

To reiterate, the signed projection of the first row vector is equal to the x-coordinate of the output vector.

---

11.6.2 Second row-vector

We can show the same thing for the second row vector. Let's bring it back into view.

• Project the second row vector on the input vector.
• Calculate the length of this projection and draw a point on the y-axis at the same distance. Note that this distance is signed.

• This is exactly the y-coordinate of the output vector.

You can try rotating the input vector and see this setup changes accordingly.

To reiterate, the signed projection of the second row vector is equal to the y-coordinate of the output vector.

---

11.6.3 Row vector and ellipse bounds

Let's look at the matrix multiplication again:

One can ask - what are the largest (or smallest) x- and y- coordinate values for the output vector? Since these values are the dot products of the row vectors with the input vector, coordinate is maximized when the input vector aligns with the row vector .

One can indeed see this by aligning the input vector with the first row vector. Note how the output vector has the largest x-coordinate on the entire ellipse.

Similarly, by aligning the input vector with the second row vector, one gets the output vector whose y-coordinate is the largest on the entire ellipse.

What about the smallest x and y coordinates?

The same reasoning can be applied to get the smallest coordinates of the output vectors. This happens when the dot product is minimized - which happens when the two vectors point in exactly opposite directions.

This entire reasoning is applicable to matrix multiplication of any dimension, not just two.

In other words,

the output ellipse is bounded by a rectangle whose x-coordinates are ± norm of the first row and whose y-coordinates are ± norm of the second row.

This is because the projection of a row on can never exceed the row's own length.

To confirm, let's change the rows and watch the rectangle follow:

• Scale the first row up: the ellipse and its bounding rectangle stretch horizontally only. Bring it back.
• Scale the second row down: the ellipse flattens vertically. Restore it.

So the columns of a matrix tell us where the basis vectors land, but the rows tell us how the output coordinates are extracted.

In the next chapter we will study rotations, the matrices whose rows and columns are unit vectors perpendicular to each other, for which the transpose plays an especially beautiful role.

---

← 10. Dot product · 12. Rotations →