Deep Learning for Coders with fastai and PyTorch

Integrating Machine Learning with Product Design
Topics in Data Ethics
Recourse and Accountability
Feedback Loops
Bias
Disinformation
Identifying and Addressing Ethical Issues
Analyze a Project You Are Working On
Processes to Implement
The Power of Diversity
Fairness, Accountability, and Transparency
Role of Policy
The Effectiveness of Regulation
Rights and Policy
Cars: A Historical Precedent
Conclusion
Questionnaire
Further Research
Deep Learning in Practice: That’s a Wrap!

II. Understanding fastai’s Applications
4. Under the Hood: Training a Digit Classifier
Pixels: The Foundations of Computer Vision
First Try: Pixel Similarity
NumPy Arrays and PyTorch Tensors
Computing Metrics Using Broadcasting
Stochastic Gradient Descent
Calculating Gradients
Stepping with a Learning Rate
An End-to-End SGD Example
Summarizing Gradient Descent
The MNIST Loss Function
Sigmoid
SGD and Mini-Batches
Putting It All Together
Creating an Optimizer
Adding a Nonlinearity
Going Deeper
Jargon Recap
Questionnaire
Further Research

5. Image Classification
From Dogs and Cats to Pet Breeds
Presizing
Checking and Debugging a DataBlock
Cross-Entropy Loss
Viewing Activations and Labels
Softmax
Log Likelihood
Taking the log
Model Interpretation
Improving Our Model
The Learning Rate Finder
Unfreezing and Transfer Learning
Discriminative Learning Rates
Selecting the Number of Epochs
Deeper Architectures
Conclusion
Questionnaire
Further Research

6. Other Computer Vision Problems
Multi-Label Classification
The Data
Constructing a DataBlock
Binary Cross Entropy
Regression
Assembling the Data
Training a Model
Conclusion
Questionnaire
Further Research

7. Training a State-of-the-Art Model
Imagenette
Normalization
Progressive Resizing
Test Time Augmentation
Mixup
Label Smoothing
Conclusion
Questionnaire
Further Research

8. Collaborative Filtering Deep Dive
A First Look at the Data
Learning the Latent Factors
Creating the DataLoaders
Collaborative Filtering from Scratch
Weight Decay
Creating Our Own Embedding Module
Interpreting Embeddings and Biases
Using fastai.collab
Embedding Distance
Bootstrapping a Collaborative Filtering Model
Deep Learning for Collaborative Filtering
Conclusion
Questionnaire
Further Research

9. Tabular Modeling Deep Dive
Categorical Embeddings
Beyond Deep Learning
The Dataset
Kaggle Competitions
Look at the Data
Decision Trees
Handling Dates
Using TabularPandas and TabularProc
Creating the Decision Tree
Categorical Variables
Random Forests
Creating a Random Forest
Out-of-Bag Error
Model Interpretation
Tree Variance for Prediction Confidence
Feature Importance
Removing Low-Importance Variables
Removing Redundant Features
Partial Dependence
Data Leakage
Tree Interpreter
Extrapolation and Neural Networks
The Extrapolation Problem
Finding Out-of-Domain Data
Using a Neural Network
Ensembling
Boosting
Combining Embeddings with Other Methods
Conclusion
Questionnaire
Further Research

10. NLP Deep Dive: RNNs
Text Preprocessing
Tokenization
Word Tokenization with fastai
Subword Tokenization
Numericalization with fastai
Putting Our Texts into Batches for a Language Model
Training a Text Classifier
Language Model Using DataBlock
Fine-Tuning the Language Model
Saving and Loading Models
Text Generation
Creating the Classifier DataLoaders
Fine-Tuning the Classifier
Disinformation and Language Models
Conclusion
Questionnaire
Further Research

11. Data Munging with fastai’s Mid-Level API
Going Deeper into fastai’s Layered API
Transforms
Writing Your Own Transform
Pipeline
TfmdLists and Datasets: Transformed Collections
TfmdLists
Datasets
Applying the Mid-Level Data API: SiamesePair
Conclusion
Questionnaire
Further Research
Understanding fastai’s Applications: Wrap Up

III. Foundations of Deep Learning
12. A Language Model from Scratch
The Data
Our First Language Model from Scratch
Our Language Model in PyTorch
Our First Recurrent Neural Network
Improving the RNN
Maintaining the State of an RNN
Creating More Signal
Multilayer RNNs
The Model
Exploding or Disappearing Activations
LSTM
Building an LSTM from Scratch
Training a Language Model Using LSTMs
Regularizing an LSTM
Dropout
Activation Regularization and Temporal Activation Regularization
Training a Weight-Tied Regularized LSTM
Conclusion
Questionnaire
Further Research

13. Convolutional Neural Networks
The Magic of Convolutions
Mapping a Convolutional Kernel
Convolutions in PyTorch
Strides and Padding
Understanding the Convolution Equations
Our First Convolutional Neural Network
Creating the CNN
Understanding Convolution Arithmetic
Receptive Fields
A Note About Twitter
Color Images
Improving Training Stability
A Simple Baseline
Increase Batch Size
1cycle Training
Batch Normalization
Conclusion
Questionnaire
Further Research

14. ResNets
Going Back to Imagenette
Building a Modern CNN: ResNet
Skip Connections
A State-of-the-Art ResNet
Bottleneck Layers
Conclusion
Questionnaire
Further Research

15. Application Architectures Deep Dive
Computer Vision
cnn_learner
unet_learner
A Siamese Network
Natural Language Processing
Tabular
Conclusion
Questionnaire
Further Research

16. The Training Process
Establishing a Baseline
A Generic Optimizer
Momentum
RMSProp
Adam
Decoupled Weight Decay
Callbacks
Creating a Callback
Callback Ordering and Exceptions
Conclusion
Questionnaire
Further Research
Foundations of Deep Learning: Wrap Up

IV. Deep Learning from Scratch
17. A Neural Net from the Foundations
Building a Neural Net Layer from Scratch
Modeling a Neuron
Matrix Multiplication from Scratch
Elementwise Arithmetic
Broadcasting
Einstein Summation
The Forward and Backward Passes
Defining and Initializing a Layer
Gradients and the Backward Pass
Refactoring the Model
Going to PyTorch
Conclusion
Questionnaire
Further Research

18. CNN Interpretation with CAM
CAM and Hooks
Gradient CAM
Conclusion
Questionnaire
Further Research

19. A fastai Learner from Scratch
Data
Dataset
Module and Parameter
Simple CNN
Loss
Learner
Callbacks
Scheduling the Learning Rate
Conclusion
Questionnaire
Further Research

20. Concluding Thoughts

A. Creating a Blog
Blogging with GitHub Pages
Creating the Repository
Setting Up Your Home Page
Creating Posts
Synchronizing GitHub and Your Computer
Jupyter for Blogging

B. Data Project Checklist
Data Scientists
Strategy
Data
Analytics
Implementation
Maintenance
Constraints

Index