Transformers, the tech behind LLMs | Deep Learning Chapter 5
Log in
YouTube
by
DiegoLerma
The initials GPT stand for Generative Pretrained Transformer.
So that first word is straightforward enough, these are bots that generate new text.
Pretrained refers to how the model went through a process of learning
from a massive amount of data, and the prefix insinuates that there's
more room to fine-tune it on specific tasks with additional training.
But the last word, that's the real key piece.
A transformer is a specific kind of neural network, a machine learning model,
and it's the core invention underlying the current boom in AI.
What I want to do with this video and the following chapters is go through
a visually-driven explanation for what actually happens inside a transformer.
We're going to follow the data that flows through it and go step by step.
There are many different kinds of models that you can build using transformers.
Some models take in audio and produce a transcript.
This sentence comes from a model going the other way around,
producing synthetic speech just from text.
All those tools that took the world by storm in 2022 like DALL-E and Midjourney
that take in a text description and produce an image make use of transformers.
Even if I can't quite get it to understand what a pie creature is supposed to be,
I'm still blown away that this kind of thing is even remotely possible.
And the original transformer introduced in 2017 by Google was invented for
the specific use case of translating text from one language into another.
But the variant that you and I will focus on, which is the type that
underlies tools like ChatGPT, will be a model that's trained to take in a piece of text,
maybe even with some surrounding images or sound accompanying it,
and produce a prediction for what comes next in the passage.
That prediction takes the form of a probability distribution
over many different chunks of text that might follow.
At first glance, you might think that predicting the next
word feels like a very different goal from generating new text.
But once you have a prediction model like this,
a simple thing you generate a longer piece of text is to give it an initial
snippet to work with, have it take a random sample from the distribution
it just generated, append that sample to the text,
and then run the whole process again to make a new prediction based on all the new text,
including what it just added.
I don't know about you, but it really doesn't feel like this should actually work.
In this animation, for example, I'm running GPT-2 on my laptop and having it repeatedly
predict and sample the next chunk of text to generate a story based on the seed text.
The story just doesn't really make that much sense.
But if I swap it out for API calls to GPT-3 instead, which is the same basic model,