Meta’s Joe Spisak on Llama 3.1 405B and the Democratization of Frontier Models | Training Data
As head of Product Management for Generative AI at Meta, Joe Spisak leads the team behind Llama, which just released the new 3.1 405B model. We spoke with Joe just two days after the model’s release to ask what’s new, what it enables, and how Meta sees the role of open source in the AI ecosystem.
Joe shares that where Llama 3.1 405B really focused is on pushing scale (it was trained on 15 trillion tokens using 16,000 GPUs) and he’s excited about the zero-shot tool use it will enable, as well as its role in distillation and generating synthetic data to teach smaller models. He tells us why he thinks even frontier models will ultimately commoditize—and why that’s a good thing for the startup ecosystem.
Hosted by: Stephanie Zhan and Sonya Huang, Sequoia Capital
00:00 Introduction
01:28 The Llama 3.1 405B launch
05:02 The open source license
07:01 What's in it for Meta?
10:19 Why not open source?
11:16 Will frontier models commoditize?
12:41 What about startups?
16:29 The Mistral team
19:36 Are all frontier strategies comparable?
22:38 Is model development becoming more like software development?
26:34 Agentic reasoning
29:09 What future levers will unlock reasoning?
31:20 Will coding and math lead to unlocks?
33:09 Small models
34:08 7X more data
37:36 Are we going to hit a wall?
39:49 Lightning round
Смотрите видео Meta’s Joe Spisak on Llama 3.1 405B and the Democratization of Frontier Models | Training Data онлайн без регистрации, длительностью часов минут секунд в хорошем качестве. Это видео добавил пользователь Sequoia Capital 30 Июль 2024, не забудьте поделиться им ссылкой с друзьями и знакомыми, на нашем сайте его посмотрели 6,90 раз и оно понравилось 16 людям.