BigBird Research Ep. 4 - Where Does BigBird Help?
UPDATE: This series was a build-up to a more polished tutorial on BigBird, and it’s available now! Check out our complete guide to BigBird here: https://www.chrismccormick.ai/LongerT...
Weekly Research Group, April 29th, 2021
Sign up for our weekly newsletter for information on our weekly discussion group! https://www.chrismccormick.ai/subscribe
So far, I’ve struggled to get BigBird to outperform the original BERT (using the simple strategy of truncating text for BERT). This week, the group helped me figure out how we might craft a code example that best demonstrates where BigBird might be most applicable or useful.
In the process, we touched on:
- The authors’ recommendation to only use Sparse Attention above 1,024 tokens.
- Why BigBird is valuable for Question Answering.
- Possible strategies for addressing GPU memory concerns with BigBird.
You can view my “Research Journal” doc here: https://docs.google.com/document/d/1f...
Outside of BigBird, we also talked about how to use a classifier to help label a large unlabeled dataset, and strategies for detecting the author of a piece of text.
You can view our notes from the Q&A discussion in the following doc: https://docs.google.com/document/d/1j...
This forum post serves as a place for further discussion and questions from the session: https://discourse.chrismccormick.ai/t...
I’ll be implementing the group’s suggestions this week, and for the next group we’ll see how it went!
Смотрите видео BigBird Research Ep. 4 - Where Does BigBird Help? онлайн без регистрации, длительностью часов минут секунд в хорошем качестве. Это видео добавил пользователь ChrisMcCormickAI 30 Апрель 2021, не забудьте поделиться им ссылкой с друзьями и знакомыми, на нашем сайте его посмотрели 631 раз и оно понравилось 13 людям.