Behavioral Testing of ML Models (Unit tests for machine learning)
How can we empower machine learning models with powerful software engineering techniques like unit testing?
Evaluating ML models using a single metric (like accuracy or F1-score) produce a low-resolution picture of model performance. Behavioral tests can give us a much higher resolution evaluation of a model's capabilities. By creating tests (which are small targeted test sets), we can better compare models or observe how model performance changes after re-training a model (or fine-tuning it). We discuss the paper 'Beyond Accuracy: Behavioral Testing of NLP Models with CheckList', which was selected as the ACL 2020 Best Paper.
Introduction (0:00)
Comparing models using capabilities (0:33)
Behavioral test of NLP models (3:06)
Test Type 1: Minimum Functionality Tests (4:22)
Test Type 2: Invariance Tests (7:04)
Test Type 3: Directional Expectation Tests (7:32)
Summary and Conclusion (10:00)
------
Paper: Beyond Accuracy: Behavioral Testing of NLP Models with CheckList
https://www.aclweb.org/anthology/2020...
Code:
https://github.com/marcotcr/checklist
------
Twitter: / jayalammar
Blog: https://jalammar.github.io/
Mailing List: https://jayalammar.substack.com/
More videos by Jay:
Language Processing with BERT: The 3 Minute Intro (Deep learning for NLP)
• Language Processing with BERT: The 3 ...
Explainable AI Cheat Sheet - Five Key Categories
• Explainable AI Cheat Sheet - Five Key...
The Narrated Transformer Language Model
• The Narrated Transformer Language Model
Jay's Visual Intro to AI
• Jay's Visual Intro to AI
How GPT-3 Works - Easily Explained with Animations
• How GPT3 Works - Easily Explained wit...
Watch video Behavioral Testing of ML Models (Unit tests for machine learning) online without registration, duration hours minute second in high quality. This video was added by user Jay Alammar 28 June 2021, don't forget to share it with your friends and acquaintances, it has been viewed on our site 10,733 once and liked it 313 people.