Gradient Descent

I was once described by a colleague as an "armchair mathematician". In retrospect, I am not sure if this was intended as a compliment or not, but I'll embrace it. In any case, it's probably preferable to "the sweet spot between Sheldon Cooper and Alan Carr", which was a description in some student feedback a few years ago.

As an armchair mathematician, I take a spectator-level interest in the awarding of the Fields medal, especially since in 2022 it went to a man from my hometown, and I am glad to see Essex represented on one of the biggest scientific stages. Whilst looking at the Fields medal awards in 2026, I came across the Gauss Prize, which was awarded this year to Yurii Nesterov for his work on mathematical optimisation, which has paved the way for the AI revolution. We will come back to this later, but let's think about optimisation first. Optimisation is familiar to all STEM students through curve fitting, but not many chemistry students want to get into the maths underpinning the methods in my experience. This is a bit of a shame, because it's cool, and in a more practical sense it's important for understanding the limitations of some of these methods. For instance, more than once I've read reports where people quote fitted rate constants like:

Which belies both a basic lack of proof reading on the part of the student, but also an admirable level of trust placed in the results of the optimisation algorithm used to extract the parameter¹. All of this leads me to believe that more education about these methods is necessary, especially when ChatGPT can probably vibe code you a questionably functional, but pretty looking, analysis program for whatever your problem is in less than the time it takes you to get your third americano of the morning. Nesterov at least seems to agree with me, saying in his Lectures on Convex Optimisation:

At first glance, everything looks very simple: many commercial optimisation packages are easily available and any user can get a "solution" to the model just by clicking at an icon on the desktop of a person computer. However, the question is, what do we actually get? How much can we trust the answer?

So let's talk about these algorithms, and especially gradient descent algorithms which are of what Nesterov won the Gauss Prize for, and have an important application in machine learning we will talk about later.

The idea of gradient descent is intuitive and fairly obvious if you think about the problem of finding the minimum point of some surface. If you're standing in Cairngorms (or another mountain range of your choice), and want to get to the bottom of the valley as fast as possible² then the obvious thing to do is take a steps downhill, in the opposite direction to the gradient of the hill you're on. There are then a lot of subtleties about exactly how large a step you should take, and in what direction, but that's the fundamental idea.

We can write it down as follows. If we are standing at a point \(x_n\) on our surface \(f(x)\), then the gradient of the surface at that point is \(\nabla f(x_n)\). As we want to go downhill³, We want to take a step of size \(\eta\) in the the opposite direction to the gradient, so we should step to the new point \(x_{n+1}\) given by:

Or in words:

Bigger steps, or steeper gradients, mean that the difference in position as we step is larger. Presuming that we can evaluate (or approximate) the gradient of our function \(\nabla f\) , then we can take as many steps as we need for us to end up in a minimum point (at which point repeated steps will just lead to us wandering back and forth around the minimum). That's basically the algorithm. Where we end up might not be the global minimum, but we can worry about that later. The main issue is how to avoid this taking a very long time, so that we take as few steps as possible.

One of the problems in the basic gradient descent is that the algorithm is greedy. It doesn't use anything other than the gradient at the current the point to evaluate where to go next, so it throws away all the information in the previous steps. In our hill-walking analogy, this would be like not attempting to look ahead at the best route forward, and stopping to sit down after every step rather than using the momentum you have built up from where you're coming from. Perhaps it's foggy, so we can't look ahead, but we shouldn't be taking a rest break every step. It would be better to incorporate the existing momentum from the previous step in our next step. We can do this by incorporating some fraction \(\beta\) of our previous step \(x_n - x_{n-1}\) into our new step:

It's a bit neater to roll the momentum term into a new variable: \(p_n = \beta(x_n - x_{n-1})\) , so we get:

Or in words:

We can think of this as first taking a step using our existing momentum from the previous step (\(p_n\)), and then taking a step opposite to the gradient at our original position (\(-\eta\nabla f(x_n)\)). This will obviously help us get to the bottom faster, if the hill is smooth and the direction of our previous step is similar to the direction we want to go anyway in the current step (we basically just move further for free). However, if our landscape isn't smooth, this might be a bit cumbersome because the momentum might take us in the wrong direction - but for these problems, there are other algorithms. This method is sometimes called the heavy ball method, as a heavy ball would carry momentum as it rolls down a hill.

To bring things back to Yurii Nesterov, the Nesterov Accelerated Gradient method makes a modification to the method above, where we evaluate the gradient at the new position after we have taken the momentum step:

Or more concisely, using our substitution for \(p_n\):

Or:

Where it's clear that now we evaluate the gradient after the momentum step, which seems reasonable (or at least, not silly). You can think of this as us guessing that the momentum is taking us in the right direction, and computing the gradient there for our next step, which intuitively seems like it would be faster. In addition to this, Nesterov showed that the optimal value of \(\beta\) is given by:

Where \(n\) is the current step. This value of \(\beta\) guarantees under some conditions that the accelerated descent algorithm will converge faster than the others we've described, but the intuition for why this is a faster method is quite subtle and lots of literature has discussed how to interpret this physically (a lot of which is beyond the comprehension of an armchair mathematician). See the notes linked here for some readable discussion, but what's more interesting for us is to play with these methods and see how they look in practice. So below is an interactive plot I've made to demonstrate things, which implements these methods in simple TypeScript.

You can play with finding the minima in the following functions:

In order: a simple quadratic, a more complex quartic, and the Rosenbrock Function that is commonly used for testing optimisation algorithms (the global minimum in the Rosenbrock function used is at the point (1,1)). The methods available are the ones described above - tweak the parameters and see how they perform. Mouseover text on all the options explains a bit more about things, so you can play start points, convergence criteria, adaptive step sizes, and some constraints on steps like using a backtracking line search to prevent any step going uphill, and imposing the Armijo condition on the backtracking search to ensure any step makes a minimum amount of descent. Exploring the options will show you that there's no universal parameter set that will work for all problems - which is the thing I think it is useful to take away from this. Nesterov states in the aforementioned book:

The main fact, which should be known to any person dealing with optimisation models, is that in general, optimisation problems are unsolvable.

It's interesting then, to think about where these kinds of algorithms are used. One case common in academic STEM is modelling data to extract parameters - but anyone who has spoken to me will know that this is a topic I can be very boring about, so let's think about another: machine learning.

Machine learning is the process by which computers learn from data and do things with it, like play Go at a superhuman level, find trends in noisy data, or generate slightly creepy videos of virtual cats. Optimisation is an integral part of all of this, but a concrete place to see gradient descent is in the way that a neural network works. Essentially these are models that take in a large (like, very large) number of parameters, and find the optimum value of them for a specific problem. Gradient descent is a common way that these parameters are optimised, and if you're familiar with neural networks but not gradient descent algorithms, then the step size (\(\eta\) ) we talked about is commonly called the learning rate in the context of neural networks. Large Language Models (LLMs) like ChatGPT, Claude, and Gemini are all fundamentally neural networks optimised using gradient descent or variants of it. So, we can see that this mathematics is underpinning the AI revolution, and while the overall profitability of this 'revolution' remains disputed, it's difficult to argue that it's not bringing in a lot of investment and making a lot of money for private companies⁴. The obvious question is then: where has the knowledge that enables this technology has come from?

The answer is from people like Yurii Nesterov. People who work in publicly funded institutions, on research grants funded by governments and charities, which are ultimately funded by the taxpayer. There has been considerable debate about the ethics of AI companies training LLMs on the entire back catalogue of human creativity, only to sell the result back to us; yet there seems to have been comparatively little debate about the ethics of companies, who develop technology built on public knowledge that only exists due to enormous sustained public investment in knowledge creation, lobbying hard to avoid regulation and disempower the nation states which ultimately facilitate all of their activities. I'm not anti-AI, and I use it a lot for helping to write programs - which it is really good at, and has saved me a lot of time in the last few months - but doesn't it seem obvious that none of this exists in a vacuum, and that it is built on the shoulders of all of human knowledge and creativity, which is owned (and has been paid for) by all of us?

Maybe, then, it is time for us to collectively say that all this lobbying, which essentially seems to follow the standard playbook of complaining about the how the governments of the nation states that provide the infrastructure and knowledge for you to exist also want you to pay a bit back, is a bit silly. It is not very difficult to envisage some fairly dark outcomes from the path we are on. Perhaps it is time for someone to say that the algorithm has converged, or at the very least, impose some hard bounds, or arrest the learning rate, before the descent takes us into a well that no amount of momentum would get us out of.