Effective Machine Learning Teams

Preface
Who This Book Is For
How This Book Is Organized
Part I: Product and Delivery
Part II: Engineering
Part III: Teams
Additional Thoughts
Conventions Used in This Book
Using Code Examples
O’Reilly Online Learning
How to Contact Us
Acknowledgments
From David Tan
From Ada Leung
From David Colls

1. Challenges and Better Paths in Delivering ML Solutions
ML: Promises and Disappointments
Continued Optimism in ML
Why ML Projects Fail
Is There a Better Way? How Systems Thinking and Lean Can Help
You Can’t “MLOps” Your Problems Away
See the Whole: A Systems Thinking Lens for Effective ML Delivery
The Five Disciplines Required for Effective ML Delivery
Conclusion

Part I. Product and Delivery
2. Product and Delivery Practices for ML Teams
ML Product Discovery
Discovering Product Opportunities
Canvases to Define Product Opportunities
Techniques for Rapidly Designing, Delivering, and Testing Solutions
Inception: Setting Teams Up for Success
Inception: What Is It and How Do We Do It?
How to Plan and Run an Inception
User Stories: Building Blocks of an MVP
Product Delivery
Cadence of Delivery Activities
Measuring Product and Delivery
Conclusion

Part II. Engineering
3. Effective Dependency Management: Principles and Tools
What If Our Code Worked Everywhere, Every Time?
A Better Way: Check Out and Go
Principles for Effective Dependency Management
Tools for Dependency Management
A Crash Course on Docker and batect
What Are Containers?
Reduce the Number of Moving Parts in Docker with batect
Conclusion

4. Effective Dependency Management in Practice
In Context: ML Development Workflow
Identifying What to Containerize
Hands-On Exercise: Reproducible Development Environments, Aided by Containers
Secure Dependency Management
Remove Unnecessary Dependencies
Automate Checks for Security Vulnerabilities
Conclusion

5. Automated Testing: Move Fast Without Breaking Things
Automated Tests: The Foundation for Iterating Quickly and Reliably
Starting with Why: Benefits of Test Automation
If Automated Testing Is So Important, Why Aren’t We Doing It?
Building Blocks for a Comprehensive Test Strategy for ML Systems
The What: Identifying Components For Testing
Characteristics of a Good Test and Pitfalls to Avoid
The How: Structure of a Test
Software Tests
Unit Tests
Training Smoke Tests
API Tests
Post-deployment Tests
Conclusion

6. Automated Testing: ML Model Tests
Model Tests
The Necessity of Model Tests
Challenges of Testing ML Models
Fitness Functions for ML Models
Model Metrics Tests (Global and Stratified)
Behavioral Tests
Testing Large Language Models: Why and How
Essential Complementary Practices for Model Tests
Error Analysis and Visualization
Learn from Production by Closing the Data Collection Loop
Open-Closed Test Design
Exploratory Testing
Means to Improve the Model
Designing to Minimize the Cost of Failures
Monitoring in Production
Bringing It All Together
Next Steps: Applying What You’ve Learned
Make Incremental Improvements
Demonstrate Value
Conclusion

7. Supercharging Your Code Editor with Simple Techniques
The Benefits (and Surprising Simplicity) of Knowing Our IDE
Why Should We Care About IDEs?
If IDEs Are So Important, Why Haven’t I Learned About Them Yet?
The Plan: Getting Productive in Two Stages
Stage 1: Configuring Your IDE
Stage 2: The Star of the Show—Keyboard Shortcuts
You Did It!
Conclusion

8. Refactoring and Technical Debt Management
Technical Debt: The Sand in Our Gears
Getting to a Healthy Level of Debt Through Tests, Design, and Refactoring
Refactoring 101
How to Refactor a Notebook (or a Problematic Codebase)
The Map: Planning Your Journey
The Journey: Hitting the Road
Looking Back at What We’ve Achieved
Technical Debt Management in the Real World
Technical Debt Management Techniques
A Positive Lens on Debt: Systems Health Ratings
Conclusion: Make Good Easy

9. MLOps and Continuous Delivery for ML (CD4ML)
MLOps: Strengths and Missing Puzzle Pieces
MLOps 101
Smells: Hints That We Missed Something
Continuous Delivery for Machine Learning
Benefits of CD4ML
A Crash Course on Continuous Delivery Principles
Building Blocks of CD4ML: Creating a Production-Ready ML System
How CD4ML Supports ML Governance and Responsible AI
Conclusion

Part III. Teams
10. Building Blocks of Effective ML Teams
Common Challenges Faced by ML Teams
Effective Team Internals
Trust as the Foundational Building Block
Communication
Diverse Membership
Purposeful, Shared Progress
Internal Tactics to Build Effective Teams
Improving Flow with Engineering Effectiveness
Feedback Loops
Cognitive Load
Flow State
Conclusion

11. Effective ML Organizations
Common Challenges Faced by ML Organizations
Effective Organizations as Teams of Teams
The Role of Value-Driven Portfolio Management
Team Topologies Model
Team Topologies for ML Teams
Organizational Tactics to Build Effective Teams
Intentional Leadership
Create Structures and Systems for Effective Teams
Engage Stakeholders and Coordinate Organizational Resources
Cultivate Psychological Safety
Champion Continuous Improvement
Embrace Failure as a Learning Opportunity
Build the Culture We Wish We Had
Encourage Teams to Play at Work
Conclusion
Epilogue: Dana’s Journey

Index
About the Authors