Deep Learning at Scale

Preface
Why Scaling Matters
Who This Book Is For
How This Book Is Organized
Introduction
Part I: Foundational Concepts of Deep Learning
Part II: Distributed Training
Part III: Extreme Scaling
What You Need to Use This Book
Setting Up Your Environment for Hands-on Exercises
Using Code Examples
Conventions Used in This Book
O’Reilly Online Learning
How to Contact Us
Acknowledgments

1. What Nature and History Have Taught Us About Scale
The Philosophy of Scaling
The General Law of Scaling
History of Scaling Law
Scalable Systems
Nature as a Scalable System
Our Visual System: A Biological Inspiration
Artificial Intelligence: The Evolution of Learnable Systems
It Takes Four to Tango
Evolving Deep Learning Trends
Scale in the Context of Deep Learning
Six Development Considerations
Scaling Considerations
Summary

I. Foundational Concepts of Deep Learning
2. Deep Learning
The Role of Data in Deep Learning
Data Flow in Deep Learning
Hands-On Exercise #1: Implementing Minimalistic Deep Learning
Developing the Model
The Embedded/Latent Space
A Word of Caution
The Learning Rate and Loss Landscape
Scaling Consideration
Profiling
Hands-On Exercise #2: Getting Complex with PyTorch
Model Input Data and Pipeline
Model
Auxiliary Utilities
Putting It All Together
Computation Graphs
Inference
Summary

3. The Computational Side of Deep Learning
The Higgs Boson of the Digital World
Floating-Point Numbers: The Faux Continuous Numbers
Units of Data Measurement
Data Storage Formats: The Trade-off of Latency and Throughput
Computer Architecture
The Birth of the Electromechanical Engine
Memory and Persistence
Computation and Memory Combined
The Scaling Laws of Electronics
Scaling Out Computation with Parallelization
Threads Versus Processes: The Unit of Parallelization
Hardware-Optimized Libraries for Acceleration
Parallel Computer Architectures: Flynn’s and Duncan’s Taxonomies
Accelerated Computing
Popular Accelerated Devices for Deep Learning
CUDA
Accelerator Benchmarking
Summary

4. Putting It All Together: Efficient Deep Learning
Hands-On Exercise #1: GPT-2
Exercise Objectives
Model Architecture
Implementation
Running the Example
Experiment Tracking
Measuring to Understand the Limitations and Scale Out
Transitioning from Language to Vision
Hands-On Exercise #2: Vision Model with Convolution
Model Architecture
Running the Example
Observations
Graph Compilation Using PyTorch 2.0
New Components of PyTorch 2.0
Graph Execution in PyTorch 2.0
Modeling Techniques to Scale Training on a Single Device
Graph Compilation
Reduced- and Mixed-Precision Training
Memory Tricks for Efficiency
Optimizer Efficiencies
Model Input Pipeline Tricks
Writing Custom Kernels in PyTorch 2.0 with Triton
Summary

II. Distributed Training
5. Distributed Systems and Communications
Distributed Systems
The Eight Fallacies of Distributed Computing
The Consistency, Availability, and Partition Tolerance (CAP) Theorem
The Scaling Law of Distributed Systems
Types of Distributed Systems
Communication in Distributed Systems
Communication Paradigm
Communication Patterns
Communication Technologies
MPI
Communication Initialization: Rendezvous
Hands-On Exercise
Scaling Compute Capacity
Infrastructure Setup Options
Provisioning of Accelerated Devices
Workload Management
Deep Learning Infrastructure Review
Overview of Leading Deep Learning Clusters
Similarities Between Today’s Most Powerful Systems
Summary

6. Theoretical Foundations of Distributed Deep Learning
Distributed Deep Learning
Centralized DDL
Decentralized DDL
Dimensions of Scaling Distributed Deep Learning
Partitioning Dimensions of Distributed Deep Learning
Types of Distributed Deep Learning Techniques
Choosing a Scaling Technique
Measuring Scale
End-to-End Metrics and Benchmarks
Measuring Incrementally in a Reproducible Environment
Summary

7. Data Parallelism
Data Partitioning
Implications of Data Sampling Strategies
Working with Remote Datasets
Introduction to Data Parallel Techniques
Hands-On Exercise #1: Centralized Parameter Server Using RCP
Hands-On Exercise #2: Centralized Gradient-Partitioned Joint Worker/Server Distributed Training
Hands-On Exercise #3: Decentralized Asynchronous Distributed Training
Centralized Synchronous Data Parallel Strategies
Data Parallel (DP)
Distributed Data Parallel (DDP)
Zero Redundancy Optimizer–Powered Data Parallelism (ZeRO-DP)
Fault-Tolerant Training
Hands-On Exercise #4: Scene Parsing with DDP
Hands-On Exercise #5: Distributed Sharded DDP (ZeRO)
Building Efficient Pipelines
Dataset Format
Local Versus Remote
Staging
Threads Versus Processes: Scaling Your Pipelines
Memory Tricks
Data Augmentations: CPU Versus GPU
JIT Acceleration
Hands-On Exercise #6: Pipeline Efficiency with FFCV
Summary

8. Scaling Beyond Data Parallelism: Model, Pipeline, Tensor, and Hybrid Parallelism
Questions to Ask Before Scaling Vertically
Theoretical Foundations of Vertical Scaling
Revisiting the Dimensions of Scaling
Operators’ Perspective of Parallelism Dimensions
Data Flow and Communications in Vertical Scaling
Basic Building Blocks for Scaling Beyond DP
PyTorch Primitives for Vertical Scaling
Working with Larger Models
Distributed Checkpointing: Saving the Partitioned Model
Summary

9. Gaining Practical Expertise with Scaling Across All Dimensions
Hands-On Exercises: Model, Tensor, Pipeline, and Hybrid Parallelism
The Dataset
Hands-On Exercise #1: Baseline DeepFM
Hands-On Exercise #2: Model Parallel DeepFM
Hands-On Exercise #3: Pipeline Parallel DeepFM
Hands-On Exercise #4: Pipeline Parallel DeepFM with RPC
Hands-On Exercise #5: Tensor Parallel DeepFM
Hands-On Exercise #6: Hybrid Parallel DeepFM
Tools and Libraries for Vertical Scaling
OneFlow
FairScale
DeepSpeed
FSDP
Overview and Comparison
Hands-On Exercise #7: Automatic Vertical Scaling with DeepSpeed
Observations
Summary

III. Extreme Scaling
10. Data-Centric Scaling
The Seven Vs of Data Through a Deep Learning Lens
The Scaling Law of Data
Data Quality
Validity
Variety
Veracity
Value and Volume
The Data Engine and Continual Learning
Volatility
Velocity
Summary

11. Scaling Experiments: Effective Planning and Management
Model Development Is Iterative
Planning for Experiments and Execution
Simplify the Complex
Fast Iteration for Fast Feedback
Decoupled Iterations
Feasibility Testing
Developing and Scaling a Minimal Viable Solution
Setting Up for Iterative Execution
Techniques to Scale Your Experiments
Accelerating Model Convergence
Accelerating Learning Via Optimization and Automation
Accelerating Learning by Increasing Expertise
Learning with Scarce Supervision
Hands-On Exercises
Hands-On Exercise #1: Transfer Learning
Hands-On Exercise #2: Hyperparameter Optimization
Hands-On Exercise #3: Knowledge Distillation
Hands-On Exercise #4: Mixture of Experts
Hands-On Exercise #5: Contrastive Learning
Hands-On Exercise #6: Meta-Learning
Summary

12. Efficient Fine-Tuning of Large Models
Review of Fine-Tuning Techniques
Standard Fine Tuning
Meta-Learning (Zero-/Few-Shot Learning)
Adapter-Based Fine Tuning
Low-Rank Tuning
LoRA—Parameter-Efficient Fine Tuning
Quantized LoRA (QLoRA)
Hands-on Exercise: QLoRA-Based Fine Tuning
Implementation Details
Inference
Exercise Summary
Summary

13. Foundation Models
What Are Foundation Models?
The Evolution of Foundation Models
Challenges Involved in Developing Foundation Models
Measurement Complexity
Deployment Challenges
Propagation of Defects to All Downstream Models
Legal and Ethical Considerations
Ensuring Consistency and Coherency
Multimodal Large Language Models
Projection
Gated Cross-Attention
Query-Based Encoding
Further Exploration
Summary

Index
About the Author