RoBERTa: A Robustly Optimized BERT Pretraining Approach

Опубликовано: 03 Сентябрь 2019
на канале: Yannic Kilcher
25,578
845

This paper shows that the original BERT model, if trained correctly, can outperform all of the improvements that have been proposed lately, raising questions about the necessity and reasoning behind these.

Abstract:
Language model pretraining has led to significant performance gains but careful comparison between different approaches is challenging. Training is computationally expensive, often done on private datasets of different sizes, and, as we will show, hyperparameter choices have significant impact on the final results. We present a replication study of BERT pretraining (Devlin et al., 2019) that carefully measures the impact of many key hyperparameters and training data size. We find that BERT was significantly undertrained, and can match or exceed the performance of every model published after it. Our best model achieves state-of-the-art results on GLUE, RACE and SQuAD. These results highlight the importance of previously overlooked design choices, and raise questions about the source of recently reported improvements. We release our models and code.

Authors: Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, Veselin Stoyanov

https://arxiv.org/abs/1907.11692


YouTube:    / yannickilcher  
Twitter:   / ykilcher  
Minds: https://www.minds.com/ykilcher
BitChute: https://www.bitchute.com/channel/10a5...


Смотрите видео RoBERTa: A Robustly Optimized BERT Pretraining Approach онлайн без регистрации, длительностью часов минут секунд в хорошем качестве. Это видео добавил пользователь Yannic Kilcher 03 Сентябрь 2019, не забудьте поделиться им ссылкой с друзьями и знакомыми, на нашем сайте его посмотрели 25,578 раз и оно понравилось 845 людям.