Loading editor...

3. Matrix as a function

In the previous chapter we saw that a matrix-vector multiplication is nothing more than a weighted sum of the columns of the matrix - the columns are vectors, and the elements of are the weights.

We will adopt a new perspective of a matrix that offers insights which may not be obvious using the weighted-sum-of-columns view.

3.1 Matrix is a function

Consider the matrix-vector product . In this new perspective, you can consider a matrix as a function. The input of this function is the vector and the output is the vector . This shift in perspective can also be expressed programmatically:

3.1.1 A weighted-sum view of matrix-vector multiplication

def matrix_vector_multiplication(matrix, x):
    y = add( scale(column, weight) for (column, weight) in zip(matrix, x) )
    return y

3.1.2 A matrix-as-a-function view of matrix-vector multiplication

def matrix(x):
    # you can think of matrix_columns as an internal state of this function
    y = add( scale(column, weight) for (column, weight) in zip(matrix_columns, x) )
    return y

3.1.3 Input type of this function

The function matrix takes as input a vector (array) x whose number of elements is equal to the number of columns in the matrix.

3.1.4 Output type of this function

The function matrix returns as output a vector (array) y whose number of elements is equal to the number of elements in each column of the matrix.

Another way of saying this is that a matrix of shape (r,c) where r = number of rows and c = number of columns takes as input a vector of shape (c,1) and returns as output a vector of shape (r,1).

3.2 Expanding the perspective of matrix-as-a-function

Note that a vector is just a point in space. In our new perspective, an r-by-c matrix (a row-by-column matrix) maps the (input) point in a -dimensional space to an (output) point in an -dimensional space.

We want to study how a matrix maps a set of points from the input space, not just one point.

Note that if the matrix is 2-by-2, its input is a -dimensional vector and its output is also a -dimensional vector.

3.3 Some transformations done by a matrix

We will see how a matrix maps a set of points arranged in an orderly fashion. We will illustrate this using a 2-by-2 matrix, so that both the input space and the output space fit on the same 2D canvas.

3.3.1 Line → Line

If the input points lie in a straight line, the output points also lie in a straight line. On the canvas we take five points on

(where is the x-intercept and is the y-intercept), and feed each of them to the matrix

The five output points land on . Now modify the input line and watch the output points: no matter where the input line is, the output points continue to stay in a straight line.

  • x-intercept : · ·
  • y-intercept : · ·

Instead of saying "a matrix maps a set of points in a straight line to another set of points in a straight line", we usually say "a matrix maps a straight line to another straight line".

3.3.2 Parallel Lines → Parallel Lines: Scale equivariance

Two (or more) parallel lines remain parallel even after being transformed by a matrix. Changing the scale of an input point (and by extension a set of points like a line, a curve etc) also changes the scale of the output point by the same value. This is obvious algebraically:

where is a constant, is the input vector and is a matrix.

How the scale varies in the input space is the same as (or equal to) how the scale varies in the output space. We call this property scale equivariance.

This begs a few questions:

  • Is the matrix transformation translation equivariant? In other words, does translating the input point by a certain value also translate the output point by the same value?
  • What about rotation? Does rotating the input point by a certain angle also rotate the output point by the same angle?
  • If two parallel lines remain parallel, does it mean the angle between two non-parallel lines also remains the same before and after a matrix transformation (for any matrix)?

We can get the answers to these questions (all the answers are no by the way) and also visualize how parallel lines remain parallel by looking at how .

It is transformed into by the matrix

Note how the parallel sides of the trapezium remain parallel in the output, for any scale and any translation of the input:

  • scale the trapezium: · ·
  • translate it horizontally: · ·
  • translate it vertically: · ·

Note that translating the input does not change the shape of the output - it simply translates the output points too, but in a direction that is different from the direction of translation in the input space

Can you figure out what the directions of the translation of the output are when the input translates along x and y axes?

3.3.3 Circle → Ellipse

If the input points lie on a circle, the output points lie on an ellipse. On the canvas, points on are fed to the matrix

and the output points .

The shape of the ellipse depends only on the matrix, not on the radius of the circle. The radius of the circle only determines the size of the ellipse. See for yourself:

  • change the radius (the ellipse only changes its size, never its shape): · ·
  • change the matrix (now the shape of the ellipse changes): · ·

We can describe this as — "a matrix maps a circle to an ellipse". We will explore this property in detail later on.

3.3.4 Generalizing Point → Point to Curve → Curve and Space → Space

You may have noticed a pattern here - we began with saying that a matrix maps an input point to an output point. But soon we began to say that a matrix maps a curve (line, circle etc) to another curve. The matrix is still mapping points on that curve to points on another curve. But we have abstracted our view into thinking how a curve is mapped to another curve by a matrix. We can generalize this to any area in a 2D plane or a volume in a 3D space (or a higher dimensional space) - we want to see how this area or volume is transformed by a matrix, which provides some beautiful insights. Let us see some examples:

3.3.5 Convex Shape → Convex Shape

Any convex shape remains convex even after a matrix transformation.

A (non-rigorous?) definition of convex shape: a shape is convex if for any two points inside the shape, the entire line connecting those two points is also inside the shape.

On the canvas, is transformed by the matrix

into — which is also convex!

Try it out:

  • move a vertex of the grey pentagon (as long as you keep the input convex, the output stays convex): · ·
  • change the matrix: · ·

3.3.6 Mapping the entire input space!

How does the entire 2D plane get transformed by a matrix? We can visualize this for a 2-by-2 matrix. Our input space is the standard cartesian plane with an x-axis, a y-axis and of horizontal and vertical lines. The canvas shows how all these grid lines are transformed by the matrix

which starts out as the identity matrix (so the transformed grid initially sits exactly on top of the input grid).

Now deform the space and watch the two arrows track the columns of the matrix:

  • : · :
  • : · :

A couple of things to note here:

  1. The vector is mapped to — the arrow.
  2. The vector is mapped to — the arrow.

This also applies to a matrix of any shape (m, n):

  1. The vector is mapped to the first column of the matrix.
  2. The vector is mapped to the second column of the matrix.
  3. ... and so on: the vector is mapped to the -th column of the matrix.
  • back to the identity:

3.4 What happens if the two columns of a matrix point in the same direction?

We will explore this question in detail in the next chapter. For now, let us just look at an example of a 2-by-2 matrix. The canvas again shows the transformed grid along with the two columns

  • the in green and
  • the in pink.

Look at what the transformed space looks like as the two columns of the matrix start pointing in similar directions: — the green column rotates from towards , i.e. towards the pink column, and the entire 2D grid gets squished into (nearly) a single line.

to get the identity matrix back.

In the next chapter we will study this squishing in more detail.

← 2. Geometry of weighted sums · 4. Rank and reversibility →