Finding a square root with gradient descent

This is perhaps the simplest example to showcase how autograd and gradient descent can be used. Here we start with a random guess for square root of 3 and converge towards the correct value by iterating these steps:

1. calculate the loss
2. backpropagate the loss
3. perform gradient descent

First, let us define the scalar x = 3 and scalar y = 3. y is going to be our guess for the square root of 3, which is initialized to 3. We also define the target = 3 which is the target for y squared.

Creating the visualization

Then we move on creating the visual information to see what's going on:

• create a point (x,y): \point x y
• create quadratic bezier curves: this is to simulate the plot y = sqrt(x) so that we can see how far the point y is from the actual value.

Now the fun part begins:

• register the parameter to optimize: \param y
• calculate the mean squared loss: \mean-squared-loss y target - it creates a variable named loss. You can view its value using \highlight loss.
• zero existing gradients: \zero-grad
• backpropagate: \backprop loss
• take a gradient descent step: \gradient-descent-step

The last part uses a variable named \learning-rate with a default value of 1e-2 as the learning rate. You can set it to whatever value you like in the editor.

The command zero-back-step <loss> essentially performs the steps of zeroing the existing grads, backpropagating the loss and taking a gradient descent step. To perform "training", you keep clicking the links below and see the value of y converge to the correct value:

• re-calculate loss: \mean-squared-loss y target
• zero-back-step: \zero-back-step loss
• re-calculate loss: \mean-squared-loss y target
• zero-back-step: \zero-back-step loss
• re-calculate loss: \mean-squared-loss y target
• zero-back-step: \zero-back-step loss
• re-calculate loss: \mean-squared-loss y target
• zero-back-step: \zero-back-step loss

This should obviously be done in a loop for real-world training. Run the \square-root-of-3, \mean-squared-loss y target and \zero-back-step loss commands in the geomatic editor and modify the variables to play with it. You can set the learning rate to a higher value like 0.1 and see how the values converges to the negative square root.