The Lottery Ticket Hypothesis and pruning in PyTorch
In this video, we are going to explain how one can do pruning in PyTorch. We will then use this knowledge to implement a paper called "The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks". The paper states that feedforward neural networks have subnetworks (winning tickets) inside of them that perform as good as (or even better than) the original network. It also proposes a recipe how to find them.
Paper: https://arxiv.org/abs/1803.03635
Official code: https://github.com/facebookresearch/o...
Code from this video: https://github.com/jankrepl/mildlyove...
00:00 Intro
00:50 Paper overview: Hypothesis (diagram) [slides]
01:33 Paper overview: Hypothesis (formal) [slides]
02:15 Paper overview: Finding winning tickets [slides]
03:44 Paper overview: Our setup [slides]
05:08 Pruning 101 in PyTorch [code]
10:29 Data - MNIST [code]
12:18 Multilayer perceptron [code]
14:05 Pruning: Linear + MLP [code]
16:45 Randomly initializing: Linear + MLP [code]
18:24 Weight copying: Linear + MLP [code]
19:51 Computing statistics [code]
20:53 Training functions [code]
24:38 CLI and training preparation [code]
27:13 Train-prune loop [code]
30:04 Grid search script [code]
31:01 Results: Actual vs desired pruning [no code]
32:47 Results: Winning tickets (parallel coordinate plots) [no code]
36:02 Results: Winning tickets (standard plots) [no code]
37:12 Outro
If you have any video suggestions or you just wanna chat feel free to join the discord server: / discord
Twitter: / moverfitted
Credits logo animation
Title: Conjungation · Author: Uncle Milk · Source: / unclemilk · License: https://creativecommons.org/licenses/... · Download (9MB): https://auboutdufil.com/?id=600
Watch video The Lottery Ticket Hypothesis and pruning in PyTorch online without registration, duration hours minute second in high quality. This video was added by user mildlyoverfitted 20 December 2021, don't forget to share it with your friends and acquaintances, it has been viewed on our site 8,482 once and liked it 293 people.