Security Chaos Engineering

The Effort Investment Portfolio
Allocating Your Effort Investment Portfolio
Investing Effort Based on Local Context
The Four Failure Modes Resulting from System Design
The Two Key Axes of Resilient Design: Coupling and Complexity
Designing to Preserve Possibilities
Coupling in Complex Systems
The Tight Coupling Trade-Off
The Dangers of Tight Coupling: Taming the Forest
Investing in Loose Coupling in Software Systems
Chaos Experiments Expose Coupling
Complexity in Complex Systems
Understanding Complexity: Essential and Accidental
Complexity and Mental Models
Introducing Linearity into Our Systems
Designing for Interactivity: Identity and Access Management
Navigating Flawed Mental Models
Chapter Takeaways

4. Building and Delivering
Mental Models When Developing Software
Who Owns Application Security (and Resilience)?
Lessons We Can Learn from Database Administration Going DevOps
Decisions on Critical Functionality Before Building
Defining System Goals and Guidelines on “What to Throw Out the Airlock”
Code Reviews and Mental Models
“Boring” Technology Is Resilient Technology
Standardization of Raw Materials
Developing and Delivering to Expand Safety Boundaries
Anticipating Scale and SLOs
Automating Security Checks via CI/CD
Standardization of Patterns and Tools
Dependency Analysis and Prioritizing Vulnerabilities
Observe System Interactions Across Space-Time (or Make More Linear)
Configuration as Code
Fault Injection During Development
Integration Tests, Load Tests, and Test Theater
Beware Premature and Improper Abstractions
Fostering Feedback Loops and Learning During Build and Deliver
Test Automation
Documenting Why and When
Distributed Tracing and Logging
Refining How Humans Interact with Build and Delivery Practices
Flexibility and Willingness to Change
Iteration to Mimic Evolution
Modularity: Humanity’s Ancient Tool for Resilience
Feature Flags and Dark Launches
Preserving Possibilities for Refactoring: Typing
The Strangler Fig Pattern
Chapter Takeaways

5. Operating and Observing
What Does Operating and Observing Involve?
Operational Goals in SCE
The Overlap of SRE and Security
Measuring Operational Success
Crafting Success Metrics like Attackers
The DORA Metrics
SLOs, SLAs, and Principled Performance Analytics
Embracing Confidence-Based Security
Observability for Resilience and Security
Thresholding to Uncover Safety Boundaries
Attack Observability
Scalable Is Safer
Navigating Scalability
Automating Away Toil
Chapter Takeaways

6. Responding and Recovering
Responding to Surprises in Complex Systems
Incident Response and the Effort Investment Portfolio
Action Bias in Incident Response
Practicing Response Activities
Recovering from Surprises
Blameless Culture
Blaming Human Error
Hindsight Bias and Outcome Bias
The Just-World Hypothesis
Neutral Practitioner Questions
Chapter Takeaways

7. Platform Resilience Engineering
Production Pressures and How They Influence System Behavior
What Is Platform Engineering?
Defining a Vision
Defining a User Problem
Local Context Is Critical
User Personas, Stories, and Journeys
Understanding How Humans Make Trade-Offs Under Pressure
Designing a Solution
The Ice Cream Cone Hierarchy of Security Solutions
System Design and Redesign to Eliminate Hazards
Substitute Less Hazardous Methods or Materials
Incorporate Safety Devices and Guards
Provide Warning and Awareness Systems
Apply Administrative Controls Including Guidelines and Training
Two Paths: The Control Strategy or the Resilience Strategy
Experimentation and Feedback Loops for Solution Design
Implementing a Solution
Fostering Consensus
Planning for Migration
Success Metrics
Chapter Takeaways

8. Security Chaos Experiments
Lessons Learned from Early Adopters
Lesson #1. Start in Nonproduction Environments; You Can Still Learn a Lot
Lesson #2. Use Past Incidents as a Source of Experiments
Lesson #3. Publish and Evangelize Experimental Findings
Setting Experiments Up for Success
Designing a Hypothesis
Designing an Experiment
Experiment Design Specifications
Conducting Experiments
Collecting Evidence
Analyzing and Documenting Evidence
Capturing Knowledge for Feedback Loops
Document Experiment Release Notes
Automating Experiments
Easing into Chaos: Game Days
Example Security Chaos Experiments
Security Chaos Experiments for Production Infrastructure
Security Chaos Experiments for Build Pipelines
Security Chaos Experiments in Cloud Native Environments
Security Chaos Experiments in Windows Environments
Chapter Takeaways

9. Security Chaos Engineering in the Wild
Experience Report: The Existence of Order Through Chaos (UnitedHealth Group)
The Story of ChaoSlingr
Step-by-Step Example: PortSlingr
Experience Report: A Quest for Stronger Reliability (Verizon)
The Bigger They Are…
All Hands on Deck Means No Hands on the Helm
Assert Your Hypothesis
Reliability Experiments
Cost Experiments
Performance Experiments
Risk Experiments
More Traditionally Known Experiments
Changing the Paradigm to Continuous
Lessons Learned
Experience Report: Security Monitoring (OpenDoor)
Experience Report: Applied Security (Cardinal Health)
Building the SCE Culture
The Mission of Applied Security
The Method: Continuous Verification and Validation (CVV)
The CVV Process Includes Four Steps
Experience Report: Balancing Reliability and Security via SCE (Accenture Global)
Our Roadmap to SCE Enterprise Capability
Our Process for Adoption
Experience Report: Cyber Chaos Engineering (Capital One)
What Does All This Have to Do with SCE?
What Is Secure Today May Not Be Secure Tomorrow
How We Started
How We Did This in Ye Olden Days
Things I’ve Learned Along the Way
A Reduction of Guesswork
Driving Value
Conclusion
Chapter Takeaways

Index
About the Authors