Coding a Paper - Ep. 7: Putting it all together
We’re finally ready to put all the pieces of our model together and run it!
We’ll need to implement the connections between our different components, and combine them into a transformer layer, which we’ll define a class for.
We’ll test everything as we go, and Nick will take us through the bugs and issues he encountered with his original implementation (and how to resolve them, of course!).
With the model complete, we’ll initialize a version to train with:
128 embedding size
10 layers
8 attention heads
The context window is a modest size at 512 tokens, but the past information carried forward by XL recurrence and our retrieval of relevant memories with kNN allow us to approximate a much larger context window!
We’ll move the model onto the GPU, run some training steps on a dataset of suitably-long research papers, and watch our training loss go down successfully.
For those of you who have been following along, congrats on making it through to the end!
How does it feel to have built your own LLM from the ground up?!
Links:
Link to Colab Notebook: https://colab.research.google.com/dri...
You can follow me on twitter: / nickcdryan
Check out the membership site for a full course version of the series (coming soon) and lots of other NLP content and code! https://www.chrismccormick.ai/membership
Chapters:
00:00 introduction
00:36 adjust relative position bias
02:48 fix our attention class
05:20 why normalize keys and queries?
06:40 add relative position bias to attention
08:00 how to put it all together
09:40 pseudocode outline
12:02 tips for putting it all together
15:25 build a layer block
19:47 build the model class and put everything together
30:23 fixing bugs and some rewrites
37:10 running our model!
38:12 moving our model onto a GPU
41:47 next steps to test and optimize
43:30 conclusion
Watch video Coding a Paper - Ep. 7: Putting it all together online without registration, duration hours minute second in high quality. This video was added by user ChrisMcCormickAI 07 March 2024, don't forget to share it with your friends and acquaintances, it has been viewed on our site 1,110 once and liked it 32 people.