[0:00]
If we look at how the intelligence of an AI model grows as we increase its compute, we can see that we are limited by our data. It doesn't matter how big we will make the model. If we're using supervised fine tuning, which is the naive training method, the performance is only going to be limited to imitating perfectly what the model is trained on. In this video, I want to show you and to visualize intuitively an algorithm that allows us to surpass the intelligence that exists in our data. What we call 
[0:31]
reinforcement learning and specifically GRPO. If we look at how naive supervised fine-tuning works, we can see that we just take human data and our model and every word split a piece of text between all of the words before it and that word and we train the model that the context of all the words before would make it statistically more likely for that target word to be selected next. And if we do it for trillions of times, the 
[1:02]
model will end up understanding language in a way. But this training doesn't make the model correct about anything. It make it good at imitating what was in our training data. It can look like it reasons and it can look like it thinks about how to get the right answer, but it's just imitating how humans look when they imitating how humans look when they reason. And this is where GRPO for reasoning 
[1:33]
comes into the image. Let's say we have the historical weather data of places and we want to train a model to be able to reason about weather and predict the temperature of a future day. We can take a naively trained model and ask it to predict the temperature tomorrow based on the temperature today. This model might seem like it is reasoning but at this point it just imitate [music] how human reason in the training data. The important thing is that at the end the model produces one verifiable answer. 
[2:08]
But now let's not stop here and let the model generate another reasoning imitation example that converges to another answer and another one. And now we have a group of answers which is where the G in grpo comes from. For training the model with GRPO let's say we have eight different reasoning and eight different answers. We can take each of those answers and compare it to the real true data 
[2:40]
and this will give us the error for each one of the reasoning paths. Now let's sort all of the answers from the smallest error to the highest error. Now we can do something very simple. Take each of those answers and compare it with the average of all answers. This will tell us how relatively good each of the responses are. And we can assume that if the answer is more correct, the reasoning that the model did to get to this answer is more correct. [music] And 
[3:11]
now we can train the model on the responses that it itself produces. For the good responses, it's going to be just like supervised fine-tuning. And for the negative ones, it's going to be the same, just the loss is multiplied by a negative number. So it's less likely to produce similar responses in the future. the future. [music] Now if we repeat this many many times we 
[3:41]
can see that the accuracy will increase as the model [music] learns to actually reason and get more and more accurate reason and get more and more accurate answers. But there are a few failure modes worth discussing because even this approach would fail most of the time. The first failure mode is that because the model is unconstrained to human language, the more we train it, the more it can drift away from being similar to its training data and starting to look like a 
[4:11]
language that is [music] completely different from English. The second failure mode that doesn't always happen with GRPO but can happen is that the diversity becomes so low that the model has no way to learn because the good examples are too similar to the bad examples. Something that can help both of those failure modes is KL divergence loss, which acts as a spring that pulls the model closer to the original one, but it doesn't pull 
[4:43]
it closer parameter-wise. [music] It makes sure that the output distribution is at least x% let's say 5% similar. And this is how we can get essentially unlimited intelligence even if we're limited by our data. as long as we can verify how good the response is. Now, if some of you want to understand it even deeper, I would really recommend you to play with it and to start interacting with problems related to this field. And 
[5:14]
for that, I can highly recommend the sponsor of this video, Brilliant. Watching videos like this one is rarely enough. And in Brilliant, you will be able to learn with interactive lessons. [music] I would specifically recommend you to try the course about clustering and classification as it covers a diverse set of topics directly related to both diffusion and language models. So if you want to become a better problem solver by building skills in many many scientific subjects crafted 
[5:44]
from worldclass teachers from Harvard, Stanford and MIT, you should go to Stanford and MIT, you should go to brilliant.org/galahhat and try it for free for 30 days. And because you used my code, if you do choose to get a [music] subscription one day, you will get 20% off. Something about how GRPO works reminds me of how evolution doesn't really need a clear target to improve organisms. And it's interesting to think about the 
[6:15]
parallels of the failure modes like lack of genetic diversity. Anyway, I would love to hear your thoughts in the comments. [music] And if you like this video, maybe you like other videos I make. So check out my channel. It is still small, so YouTube is probably not going to show you another video if you don't click on it now. And I hope to see you again. See you next time. Bye-bye.