Introducing MLOps

Preface
Who This Book Is For
How This Book Is Organized
Conventions Used in This Book
O’Reilly Online Learning
How to Contact Us
Acknowledgments

I. MLOps: What and Why
1. Why Now and Challenges
Defining MLOps and Its Challenges
MLOps to Mitigate Risk
Risk Assessment
Risk Mitigation
MLOps for Responsible AI
MLOps for Scale
Closing Thoughts

2. People of MLOps
Subject Matter Experts
Data Scientists
Data Engineers
Software Engineers
DevOps
Model Risk Manager/Auditor
Machine Learning Architect
Closing Thoughts

3. Key MLOps Features
A Primer on Machine Learning
Model Development
Establishing Business Objectives
Data Sources and Exploratory Data Analysis
Feature Engineering and Selection
Training and Evaluation
Reproducibility
Responsible AI
Productionalization and Deployment
Model Deployment Types and Contents
Model Deployment Requirements
Monitoring
DevOps Concerns
Data Scientist Concerns
Business Concerns
Iteration and Life Cycle
Iteration
The Feedback Loop
Governance
Data Governance
Process Governance
Closing Thoughts

II. MLOps: How
4. Developing Models
What Is a Machine Learning Model?
In Theory
In Practice
Required Components
Different ML Algorithms, Different MLOps Challenges
Data Exploration
Feature Engineering and Selection
Feature Engineering Techniques
How Feature Selection Impacts MLOps Strategy
Experimentation
Evaluating and Comparing Models
Choosing Evaluation Metrics
Cross-Checking Model Behavior
Impact of Responsible AI on Modeling
Version Management and Reproducibility
Closing Thoughts

5. Preparing for Production
Runtime Environments
Adaptation from Development to Production Environments
Data Access Before Validation and Launch to Production
Final Thoughts on Runtime Environments
Model Risk Evaluation
The Purpose of Model Validation
The Origins of ML Model Risk
Quality Assurance for Machine Learning
Key Testing Considerations
Reproducibility and Auditability
Machine Learning Security
Adversarial Attacks
Other Vulnerabilities
Model Risk Mitigation
Changing Environments
Interactions Between Models
Model Misbehavior
Closing Thoughts

6. Deploying to Production
CI/CD Pipelines
Building ML Artifacts
What’s in an ML Artifact?
The Testing Pipeline
Deployment Strategies
Categories of Model Deployment
Considerations When Sending Models to Production
Maintenance in Production
Containerization
Scaling Deployments
Requirements and Challenges
Closing Thoughts

7. Monitoring and Feedback Loop
How Often Should Models Be Retrained?
Understanding Model Degradation
Ground Truth Evaluation
Input Drift Detection
Drift Detection in Practice
Example Causes of Data Drift
Input Drift Detection Techniques
The Feedback Loop
Logging
Model Evaluation
Online Evaluation
Closing Thoughts

8. Model Governance
Who Decides What Governance the Organization Needs?
Matching Governance with Risk Level
Current Regulations Driving MLOps Governance
Pharmaceutical Regulation in the US: GxP
Financial Model Risk Management Regulation
GDPR and CCPA Data Privacy Regulations
The New Wave of AI-Specific Regulations
The Emergence of Responsible AI
Key Elements of Responsible AI
Element 1: Data
Element 2: Bias
Element 3: Inclusiveness
Element 4: Model Management at Scale
Element 5: Governance
A Template for MLOps Governance
Step 1: Understand and Classify the Analytics Use Cases
Step 2: Establish an Ethical Position
Step 3: Establish Responsibilities
Step 4: Determine Governance Policies
Step 5: Integrate Policies into the MLOps Process
Step 6: Select the Tools for Centralized Governance Management
Step 7: Engage and Educate
Step 8: Monitor and Refine
Closing Thoughts

III. MLOps: Real-World Examples
9. MLOps in Practice: Consumer Credit Risk Management
Background: The Business Use Case
Model Development
Model Bias Considerations
Prepare for Production
Deploy to Production
Closing Thoughts

10. MLOps in Practice: Marketing Recommendation Engines
The Rise of Recommendation Engines
The Role of Machine Learning
Push or Pull?
Data Preparation
Design and Manage Experiments
Model Training and Deployment
Scalability and Customizability
Monitoring and Retraining Strategy
Real-Time Scoring
Ability to Turn Recommendations On and Off
Pipeline Structure and Deployment Strategy
Monitoring and Feedback
Retraining Models
Updating Models
Runs Overnight, Sleeps During Daytime
Option to Manually Control Models
Option to Automatically Control Models
Monitoring Performance
Closing Thoughts

11. MLOps in Practice: Consumption Forecast
Power Systems
Data Collection
Problem Definition: Machine Learning, or Not Machine Learning?
Spatial and Temporal Resolution
Implementation
Modeling
Deployment
Monitoring
Closing Thoughts

Index