If we look at how the intelligence of an
AI model grows as we increase its
compute, we can see that we are limited
by our data. It doesn't matter how big
we will make the model. If we're using
supervised fine tuning, which is the
naive training method, the performance
is only going to be limited to imitating
perfectly what the model is trained on.
In this video, I want to show you and to
visualize intuitively an algorithm that
allows us to surpass the intelligence
that exists in our data. What we call
reinforcement learning and specifically
GRPO. If we look at how naive supervised
fine-tuning works, we can see that we
just take human data and our model and
every word split a piece of text between
all of the words before it and that word
and we train the model that the context
of all the words before would make it
statistically more likely for that
target word to be selected next. And if
we do it for trillions of times, the
model will end up understanding language
in a way.
But this training doesn't make the model
correct about anything. It make it good
at imitating what was in our training
data. It can look like it reasons and it
can look like it thinks about how to get
the right answer, but it's just
imitating how humans look when they
reason.
And this is where GRPO for reasoning
comes into the image. Let's say we have
the historical weather data of places
and we want to train a model to be able
to reason about weather and predict the
temperature of a future day. We can take
a naively trained model and ask it to
predict the temperature tomorrow based
on the temperature today. This model
might seem like it is reasoning but at
this point it just imitate [music] how
human reason in the training data. The
important thing is that at the end the
model produces one verifiable answer.
But now let's not stop here and let the
model generate another reasoning
imitation example that converges to
another answer and another one.
And now we have a group of answers which
is where the G in grpo comes from. For
training the model with GRPO let's say
we have eight different reasoning and
eight different answers. We can take
each of those answers and compare it to
the real true data
and this will give us the error for each
one of the reasoning paths. Now let's
sort all of the answers from the
smallest error to the highest error. Now
we can do something very simple. Take
each of those answers and compare it
with the average of all answers. This
will tell us how relatively good each of
the responses are. And we can assume
that if the answer is more correct, the
reasoning that the model did to get to
this answer is more correct. [music] And
now we can train the model on the
responses that it itself produces. For
the good responses, it's going to be
just like supervised fine-tuning.
And for the negative ones, it's going to
be the same, just the loss is multiplied
by a negative number. So it's less
likely to produce similar responses in
the future.
[music]
Now if we repeat this many many times we
can see that the accuracy will increase
as the model [music] learns to actually
reason and get more and more accurate
answers.
But there are a few failure modes worth
discussing because even this approach
would fail most of the time. The first
failure mode is that because the model
is unconstrained to human language, the
more we train it, the more it can drift
away from being similar to its training
data and starting to look like a
language that is [music] completely
different from English.
The second failure mode that doesn't
always happen with GRPO but can happen
is that the diversity becomes so low
that the model has no way to learn
because the good examples are too
similar to the bad examples. Something
that can help both of those failure
modes is KL divergence loss, which acts
as a spring that pulls the model closer
to the original one, but it doesn't pull
it closer parameter-wise. [music] It
makes sure that the output distribution
is at least x% let's say 5% similar. And
this is how we can get essentially
unlimited intelligence even if we're
limited by our data. as long as we can
verify how good the response is. Now, if
some of you want to understand it even
deeper, I would really recommend you to
play with it and to start interacting
with problems related to this field. And
for that, I can highly recommend the
sponsor of this video, Brilliant.
Watching videos like this one is rarely
enough. And in Brilliant, you will be
able to learn with interactive lessons.
[music] I would specifically recommend
you to try the course about clustering
and classification as it covers a
diverse set of topics directly related
to both diffusion and language models.
So if you want to become a better
problem solver by building skills in
many many scientific subjects crafted
from worldclass teachers from Harvard,
Stanford and MIT, you should go to
brilliant.org/galahhat
and try it for free for 30 days. And
because you used my code, if you do
choose to get a [music] subscription one
day, you will get 20% off.
Something about how GRPO works reminds
me of how evolution doesn't really need
a clear target to improve organisms. And
it's interesting to think about the
parallels of the failure modes like lack
of genetic diversity. Anyway, I would
love to hear your thoughts in the
comments. [music] And if you like this
video, maybe you like other videos I
make. So check out my channel. It is
still small, so YouTube is probably not
going to show you another video if you
don't click on it now. And I hope to see
you again. See you next time. Bye-bye.