Tiny VoltGradient of a function
We start with functions whose output is a scalar. The gradient is defined only for scalar-valued functions; for vector-valued functions, it generalizes to the Jacobian.
0. What is a gradient?
Consider a scalar-valued function . Then a small change in output given a small change in input can be expressed as an inner product with the change in input :
This is because is a linear function of and its output is a scalar. Note that this becomes possible only because the output is a scalar. is the gradient of . It is a vector with the same dimension as the input vector .
Conceptually, the gradient of a scalar-valued function is a vector whose components are the partial derivatives of the function with respect to each of the coordinates of the input vector.
0.1 Gradient is perpendicular to the tangent plane
In the above expression, setting gives us . Since the output does not change, the vector lies in the tangent plane of the function at the point . The gradient is perpendicular to the tangent plane.
For example, consider . Writing and using , we get . So the gradient of the norm is simply the unit vector in the direction of .
Given a function , one can draw its gradient at each point as a vector field, where each vector represents the gradient at that point:
- .
Note how the gradient at each is a unit vector that points as the input point. Since rotation does not change the magnitude of the vector, all these gradients are normal to passing through this point.
1. Gradient and the inner product
The idea of inner product between two vectors can be extended to matrices. This allows us to use matrix identities to compute the gradient of a scalar-valued function. The inner product of two matrices and (of the same shape) is the sum of their elementwise products. It is also known as the Frobenius inner product and denoted by .
1.1 Frobenius inner product and Frobenius norm
The trace of a square matrix , denoted by , is the sum of its diagonal elements. It is connected to the Frobenius inner product like so:
One can also extend the idea of the norm of a vector to the norm of a matrix. The Frobenius norm of a matrix , denoted by , is the square root of the Frobenius inner product of with itself.
2. Some useful identities
It is helpful to remember some identities that are useful in computing the gradient of a scalar-valued function.
2.1 Trace of a matrix
- Linear:
- Invariant under transposition:
- Commutative:
- Invariant under circular shifts:
- Invariant under similarity:
- And finally:
2.2 Kronecker product
More details can be found in this article. Two key properties of the Kronecker product that will come in handy are:
3. Examples
Note that all the functions below are scalar valued.
3.1
The function can be rewritten as . Then:
3.2
Here we use the trick that the trace of a scalar is the scalar itself. Hence .
Note that using the Kronecker identity of the bilinear form, we could also write:
Indeed one can verify that .
3.3
Assume that exists. First we establish that for a small perturbation .
Next we perturb the matrix by a small amount and rewrite it as .
Here is the cofactor matrix of . Thus, the gradient of determinant of a matrix is its cofactor matrix.
Next we will generalize the idea of a gradient to functions whose output is a vector.