Scaling Data-Constrained Language Models
Sasha Rush (Cornell University & Hugging Face)
https://simons.berkeley.edu/talks/sas...
Large Language Models and Transformers
Extrapolating scaling trends suggest that training dataset size for LLMs may soon be limited by the amount of text data available on the internet. In this talk we investigate scaling language models in data-constrained regimes. Specifically, we run a set of empirical experiments varying the extent of data repetition and compute budget. From these experiments we propose and empirically validate a scaling law for compute optimality that accounts for the decreasing value of repeated tokens and excess parameters. Finally, we discuss and experiment with approaches
for mitigating data scarcity.
Смотрите видео Scaling Data-Constrained Language Models онлайн без регистрации, длительностью часов минут секунд в хорошем качестве. Это видео добавил пользователь Simons Institute 01 Январь 1970, не забудьте поделиться им ссылкой с друзьями и знакомыми, на нашем сайте его посмотрели 4,534 раз и оно понравилось 84 людям.