Tiny VoltJacobian of a function
Gradient of a function is defined only for scalar-valued functions. What if the output is also vector-valued? The idea is to view a output as number of outputs.
0. Key idea
Let be a vector-valued function. One can actually think of it as two scalar-valued functions . Gradient is defined for the functions since they are scalar-valued.
A vector-valued maps a point in to a point in .
The same map is equally , one per output coordinate.
Thus a small change in the output of can be shown as a small change in the output of for a small change in input .
Thus, the Jacobian matrix can be viewed as a matrix of gradients of scalar-valued functions. However a more abstract view of Jacobian is to look at it in the following way:
It is applicable for small changes in input and output. Thus the shape of the Jacobian matrix is .
1. Properties of Jacobian
1.1 Jacobian of a composition of functions
Given two functions , the Jacobian of is given by:
Some interesting properties of Jacobian of a function (input and output dims are the same) are mentioned below.
1.2 Jacobian determinant
The determinant of a Jacobian matrix provides an idea of how the space around a point is stretched or compressed. If the determinant is non-zero, it means the function is invertible at that point. If it is positive, the orientation of the space is preserved. The absolute value of the determinant is also used extensively in normalizing flows.
1.3 Jacobian inverse
If a function is invertible at a point , then the Jacobian of its inverse is the inverse of the Jacobian of at . It also implies that the determinant of the Jacobian of is the inverse of the determinant of the Jacobian of .