Function is a DAG

Viewing a function as a directed acyclic graph is a powerful tool. It allows one to understand how autodiff systems are designed and used to compute gradients.

---

0. Simple examples

• DAG of :
• DAG of :

Note that the we have left out how the input is mapped to the output. The DAG only shows the structure of the function and how different variables are related to each other.

Consider a function that calculates the variance of variables where .

The DAG of the variance function connects each input both to the mean and (via ) to the sum of squares , which finally produces .

---

1. Mixing vectors and scalars

In the previous example, every node in the DAG of variance function was a scalar. We can consider a DAG where some of the nodes are vectors. This provides a more linear-algebraic view of the function. From this point of view, variance is a function .

Here are vectors while the rest are scalars, giving another DAG of the same variance function.

Here are vectors while the rest are scalars.

---

2. Jacobians as edge representations

Each edge can be seen as a function whose Jacobian is a matrix of shape . We attach this Jacobian matrix to its corresponding edge.

Next we will see how to use this setup to calculate the Jacobian of the function represented by the DAG.

---

← Key matrix derivatives · Autodiff on a DAG →