Tiny VoltLinear operator
Let's start with the central idea of calculus: a small change in output given a small change in input , for a function where can be scalars or vectors:
Canceling the higher order terms involving leaves as a linear function of . As an example, consider :
The term is dropped as higher order, and note . Though not immediately obvious, is a linear function of . For a fixed , is also a (not necessarily linear) function of .
0. Linear operator
A linear operator is a linear map from one vector space to another vector space.
It has a subtle difference from a function: a function can be a map from any set to any set, but here we constrain the input and outputs to be elements of a vector space. The most common example of a linear operator is a matrix (ref: Matrix is a function). In fact, any linear operator can be represented as a matrix. However it is often more efficient to write it in the less obvious way (as in the above example). It will become clearer as we proceed.
For a (smooth) function , we can obtain a linear operator which is a function of (a small change in input) and whose output is a function of (a small change in output at the point ).
First, the carries a point in across to its image in (the grey layer).
Then the sends the small change to the corresponding change (the teal layer), the map that induces at the point .
import torch
def square(X):
return X @ X
def d_square(dX):
def operator(X):
return X @ dX + dX @ X
return operator
X = torch.randn(1000,1000,dtype=torch.float64)
dX = torch.randn_like(X).mul(1e-8)
actual_dy = (X+dX)@(X+dX) - X@X
# The below line is the same as
# dy = X @ dX + dX @ X
dy = d_square(dX)(X)
torch.allclose(actual_dy, dy)
'''
Outputs:
True
'''
Note: is linear in but may not be linear in . For example:
which is not linear in .
1. Product rule and chain rule
Consider two (possibly vector-valued) functions along with their (differential) linear operators:
Then for a function , we have:
This is the product rule for vector valued functions (the last term is dropped as a higher order term).
Now consider another function . Then we have:
This is the chain rule for vector valued functions. Note that is evaluated at and is evaluated at . This will become clear with an example below.
2. Some examples
Below are some interesting examples of differential linear operators. Note that for any matrix or vector .
2.1. where is a vector
This function calculates the squared norm of a vector. It can be re-written as . Now we just use instead of :
where we used the fact that is a scalar and hence equals . Thus, if the change is orthogonal to the vector , then the change in output is zero since . This makes sense because this small change can be seen as a rotation of the vector . The rotation does not change the length of .
2.2.
We already saw this for the case of . To prove this, one can apply the product rule and use induction with the initial condition . Assuming that it is true for , we have:
Note how it is similar to the case of . However since matrix multiplication is not commutative, the order of and is important.
2.3.
One can use the identity and then apply the product rule to it:
where the last line follows from left-multiplying both sides by .
2.4. where is an orthonormal matrix
Using the property of orthonormal matrix, we have . We can use the fact that .
Thus is negative of its transpose, which means is a skew-symmetric matrix. This insight is a motivation for key ideas behind Lie algebra, a quick introduction of which can be found here.
3. Closing example
We end this part with an example of a chain rule for a matrix-valued function. We have:
import torch
x = torch.rand(1000,1000).double()
dx = torch.randn_like(x).mul(1e-8)
def square(x):
return x @ x
def invert(x):
return torch.linalg.inv(x)
def d_square(dx):
def operator(x):
return x @ dx + dx @ x
return operator
def d_invert(dx):
def operator(x):
x_inv = torch.linalg.inv(x)
return -x_inv @ dx @ x_inv
return operator
Finally, we perturb the input by a small value and record the change in the output . Finally we calculate the change in output using the chain rule .
y = invert(square(x))
dy = invert(square(x+dx)) - y
g = square(x)
dg = d_square(dx)(x)
df = d_invert(dg)(g)
torch.allclose(df, dy)
'''
Outputs:
True
'''