Data Mesh

4. Principle of the Self-Serve Data Platform
Data Mesh Platform: Compare and Contrast
Serving Autonomous Domain-Oriented Teams
Managing Autonomous and Interoperable Data Products
A Continuous Platform of Operational and Analytical Capabilities
Designed for a Generalist Majority
Favoring Decentralized Technologies
Domain Agnostic
Data Mesh Platform Thinking
Enable Autonomous Teams to Get Value from Data
Exchange Value with Autonomous and Interoperable Data Products
Accelerate Exchange of Value by Lowering the Cognitive Load
Scale Out Data Sharing
Support a Culture of Embedded Innovation
Transition to a Self-Serve Data Mesh Platform
Design the APIs and Protocols First
Prepare for Generalist Adoption
Do an Inventory and Simplify
Create Higher-Level APIs to Manage Data Products
Build Experiences, Not Mechanisms
Begin with the Simplest Foundation, Then Harvest to Evolve
Recap

5. Principle of Federated Computational Governance
Apply Systems Thinking to Data Mesh Governance
Maintain Dynamic Equilibrium Between Domain Autonomy and Global Interoperability
Embrace Dynamic Topology as a Default State
Utilize Automation and the Distributed Architecture
Apply Federation to the Governance Model
Federated Team
Guiding Values
Policies
Incentives
Apply Computation to the Governance Model
Standards as Code
Policies as Code
Automated Tests
Automated Monitoring
Transition to Federated Computational Governance
Delegate Accountability to Domains
Embed Policy Execution in Each Data Product
Automate Enablement and Monitoring over Interventions
Model the Gaps
Measure the Network Effect
Embrace Change over Constancy
Recap

Part II. Why Data Mesh?
6. The Inflection Point
Great Expectations of Data
The Great Divide of Data
Scale: Encounter of a New Kind
Beyond Order
Approaching the Plateau of Return
Recap

7. After the Inflection Point
Respond Gracefully to Change in a Complex Business
Align Business, Tech, and Now Analytical Data
Close the Gap Between Analytical and Operational Data
Localize Data Changes to Business Domains
Reduce Accidental Complexity of Pipelines and Copying Data
Sustain Agility in the Face of Growth
Remove Centralized and Monolithic Bottlenecks
Reduce Coordination of Data Pipelines
Reduce Coordination of Data Governance
Enable Autonomy
Increase the Ratio of Value from Data to Investment
Abstract Technical Complexity with a Data Platform
Embed Product Thinking Everywhere
Go Beyond the Boundaries
Recap

8. Before the Inflection Point
Evolution of Analytical Data Architectures
First Generation: Data Warehouse Architecture
Second Generation: Data Lake Architecture
Third Generation: Multimodal Cloud Architecture
Characteristics of Analytical Data Architecture
Monolithic
Centralized Data Ownership
Technology Oriented
Recap

Part III. How to Design the Data Mesh Architecture
9. The Logical Architecture
Domain-Oriented Analytical Data Sharing Interfaces
Operational Interface Design
Analytical Data Interface Design
Interdomain Analytical Data Dependencies
Data Product as an Architecture Quantum
A Data Product’s Structural Components
Data Product Data Sharing Interactions
Data Discovery and Observability APIs
The Multiplane Data Platform
A Platform Plane
Data Infrastructure (Utility) Plane
Data Product Experience Plane
Mesh Experience Plane
Example
Embedded Computational Policies
Data Product Sidecar
Data Product Computational Container
Control Port
Recap

10. The Multiplane Data Platform Architecture
Design a Platform Driven by User Journeys
Data Product Developer Journey
Incept, Explore, Bootstrap, and Source
Build, Test, Deploy, and Run
Maintain, Evolve, and Retire
Data Product Consumer Journey
Incept, Explore, Bootstrap, Source
Build, Test, Deploy, Run
Maintain, Evolve, and Retire
Recap

Part IV. How to Design the Data Product Architecture
11. Design a Data Product by Affordances
Data Product Affordances
Data Product Architecture Characteristics
Design Influenced by the Simplicity of Complex Adaptive Systems
Emergent Behavior from Simple Local Rules
No Central Orchestrator
Recap

12. Design Consuming, Transforming, and Serving Data
Serve Data
The Needs of Data Users
Serve Data Design Properties
Serve Data Design
Consume Data
Archetypes of Data Sources
Locality of Data Consumption
Data Consumption Design
Transform Data
Programmatic Versus Nonprogrammatic Transformation
Dataflow-Based Transformation
ML as Transformation
Time-Variant Transformation
Transformation Design
Recap

13. Design Discovering, Understanding, and Composing Data
Discover, Understand, Trust, and Explore
Begin Discovery with Self-Registration
Discover the Global URI
Understand Semantic and Syntax Models
Establish Trust with Data Guarantees
Explore the Shape of Data
Learn with Documentation
Discover, Explore, and Understand Design
Compose Data
Consume Data Design Properties
Traditional Approaches to Data Composability
Compose Data Design
Recap

14. Design Managing, Governing, and Observing Data
Manage the Life Cycle
Manage Life-Cycle Design
Data Product Manifest Components
Govern Data
Govern Data Design
Standardize Policies
Data and Policy Integration
Linking Policies
Observe, Debug, and Audit
Observability Design
Recap

Part V. How to Get Started
15. Strategy and Execution
Should You Adopt Data Mesh Today?
Data Mesh as an Element of Data Strategy
Data Mesh Execution Framework
Business-Driven Execution
End-to-End and Iterative Execution
Evolutionary Execution
Recap

16. Organization and Culture
Change
Culture
Values
Reward
Intrinsic Motivations
Extrinsic Motivations
Structure
Organization Structure Assumptions
Discover Data Product Boundaries
People
Roles
Skillset Development
Process
Key Process Changes
Recap

Index
About the Author