Hugging Face Tokenizers (11.3)
This video discusses how to use HuggingFace tokenizer models. These models break text down into tokens, which might be a single word, or make up multiple words. Tokenizers are the first step in natural language processing.
Code for This Video:
https://github.com/jeffheaton/app_dee...
~~~~~~~~~~~~~~~ COURSE MATERIAL ~~~~~~~~~~~~~~~
📖 Textbook - Coming soon
😸🐙 GitHub - https://github.com/jeffheaton/app_dee...
▶️ Play List - • 2024 PyTorch Version Applications of ...
🏫 WUSTL Course Site - https://sites.wustl.edu/jeffheaton/t8...
~~~~~~~~~~~~~~~ CONNECT ~~~~~~~~~~~~~~~
🖥️ Website: https://www.heatonresearch.com/
🐦 Twitter - / jeffheaton
😸🐙 GitHub - https://github.com/jeffheaton
📸 Instagram - / jeffheatondotcom
🦾 Discord: / discord
▶️ Subscribe: https://www.youtube.com/c/heatonresea...
~~~~~~~~~~~~~~ SUPPORT ME 🙏~~~~~~~~~~~~~~
🅿 Patreon - / jeffheaton
🙏 Other Ways to Support (some free) - https://www.heatonresearch.com/suppor...
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
#PyTorch #nlp #huggingface
Watch video Hugging Face Tokenizers (11.3) online without registration, duration hours minute second in high quality. This video was added by user Jeff Heaton 22 November 2023, don't forget to share it with your friends and acquaintances, it has been viewed on our site 99 once and liked it 3 people.