LLaMA Pro: Progressive LLaMA with Block Expansion (Paper Explained)

Published: 07 January 2024
on channel: Yannic Kilcher

35,162

927

Note: The H800 is a variant of the H100 for the Chinese market

OUTLINE:
0:00 - Introduction
5:30 - Adding new blocks to LLaMA
15:00 - Block expansion
27:40 - Experiments
30:40 - Conclusion

Paper: https://arxiv.org/abs/2401.02415
Other Paper: https://proceedings.mlr.press/v162/sh...

Abstract:
Humans generally acquire new skills without compromising the old; however, the opposite holds for Large Language Models (LLMs), e.g., from LLaMA to CodeLLaMA. To this end, we propose a new post-pretraining method for LLMs with an expansion of Transformer blocks. We tune the expanded blocks using only new corpus, efficiently and effectively improving the model's knowledge without catastrophic forgetting. In this paper, we experiment on the corpus of code and math, yielding LLaMA Pro-8.3B, a versatile foundation model initialized from LLaMA2-7B, excelling in general tasks, programming, and mathematics. LLaMA Pro and its instruction-following counterpart (LLaMA Pro-Instruct) achieve advanced performance among various benchmarks, demonstrating superiority over existing open models in the LLaMA family and the immense potential of reasoning and addressing diverse tasks as an intelligent agent. Our findings provide valuable insights into integrating natural and programming languages, laying a solid foundation for developing advanced language agents that operate effectively in various environments.

Authors: Chengyue Wu, Yukang Gan, Yixiao Ge, Zeyu Lu, Jiahao Wang, Ye Feng, Ping Luo, Ying Shan

Links:
Homepage: https://ykilcher.com
Merch: https://ykilcher.com/merch
YouTube:    / yannickilcher
Twitter:   / ykilcher
Discord: https://ykilcher.com/discord
LinkedIn:   / ykilcher

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannick...
Patreon:   / yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n

Watch video LLaMA Pro: Progressive LLaMA with Block Expansion (Paper Explained) online without registration, duration hours minute second in high quality. This video was added by user Yannic Kilcher 07 January 2024, don't forget to share it with your friends and acquaintances, it has been viewed on our site 35,162 once and liked it 927 people.