Architecting Data and Machine Learning Platforms

Step 5: Leapfrog with Packaged AI Solutions
Predictive Analytics
Understanding and Generating Unstructured Data
Personalization
Packaged Solutions
Step 6: Operationalize AI-Driven Workflows
Identifying the Right Balance of Automation and Assistance
Building a Data Culture
Populating Your Data Science Team
Step 7: Product Management for Data
Applying Product Management Principles to Data
1. Understand and Maintain a Map of Data Flows in the Enterprise
2. Identify Key Metrics
3. Agreed Criteria, Committed Roadmap, and Visionary Backlog
4. Build for the Customers You Have
5. Don’t Shift the Burden of Change Management
6. Interview Customers to Discover Their Data Needs
7. Whiteboard and Prototype Extensively
8. Build Only What Will Be Used Immediately
9. Standardize Common Entities and KPIs
10. Provide Self-Service Capabilities in Your Data Platform
Summary

3. Designing for Your Data Team
Classifying Data Processing Organizations
Data Analysis–Driven Organization
The Vision
The Personas
The Technological Framework
Data Engineering–Driven Organization
The Vision
The Personas
The Technological Framework
Data Science–Driven Organization
The Vision
The Personas
The Technological Framework
Summary

4. A Migration Framework
Modernize Data Workflows
Holistic View
Modernize Workflows
Transform the Workflow Itself
A Four-Step Migration Framework
Prepare and Discover
Assess and Plan
Execute
Optimize
Estimating the Overall Cost of the Solution
Audit of the Existing Infrastructure
Request for Information/Proposal and Quotation
Proof of Concept/Minimum Viable Product
Setting Up Security and Data Governance
Framework
Artifacts
Governance over the Life of the Data
Schema, Pipeline, and Data Migration
Schema Migration
Pipeline Migration
Data Migration
Migration Stages
Summary

5. Architecting a Data Lake
Data Lake and the Cloud—A Perfect Marriage
Challenges with On-Premises Data Lakes
Benefits of Cloud Data Lakes
Design and Implementation
Batch and Stream
Data Catalog
Hadoop Landscape
Cloud Data Lake Reference Architecture
Integrating the Data Lake: The Real Superpower
APIs to Extend the Lake
The Evolution of Data Lake with Apache Iceberg, Apache Hudi, and Delta Lake
Interactive Analytics with Notebooks
Democratizing Data Processing and Reporting
Build Trust in the Data
Data Ingestion Is Still an IT Matter
ML in the Data Lake
Training on Raw Data
Predicting in the Data Lake
Summary

6. Innovating with an Enterprise Data Warehouse
A Modern Data Platform
Organizational Goals
Technological Challenges
Technology Trends and Tools
Hub-and-Spoke Architecture
Data Ingest
Business Intelligence
Transformations
Organizational Structure
DWH to Enable Data Scientists
Query Interface
Storage API
ML Without Moving Your Data
Summary

7. Converging to a Lakehouse
The Need for a Unique Architecture
User Personas
Antipattern: Disconnected Systems
Antipattern: Duplicated Data
Converged Architecture
Two Forms
Lakehouse on Cloud Storage
SQL-First Lakehouse
The Benefits of Convergence
Summary

8. Architectures for Streaming
The Value of Streaming
Industry Use Cases
Streaming Use Cases
Streaming Ingest
Streaming ETL
Streaming ELT
Streaming Insert
Streaming from Edge Devices (IoT)
Streaming Sinks
Real-Time Dashboards
Live Querying
Materialize Some Views
Stream Analytics
Time-Series Analytics
Clickstream Analytics
Anomaly Detection
Resilient Streaming
Continuous Intelligence Through ML
Training Model on Streaming Data
Streaming ML Inference
Automated Actions
Summary

9. Extending a Data Platform Using Hybrid and Edge
Why Multicloud?
A Single Cloud Is Simpler and Cost-Effective
Multicloud Is Inevitable
Multicloud Could Be Strategic
Multicloud Architectural Patterns
Single Pane of Glass
Write Once, Run Anywhere
Bursting from On Premises to Cloud
Pass-Through from On Premises to Cloud
Data Integration Through Streaming
Adopting Multicloud
Framework
Time Scale
Define a Target Multicloud Architecture
Why Edge Computing?
Bandwidth, Latency, and Patchy Connectivity
Use Cases
Benefits
Challenges
Edge Computing Architectural Patterns
Smart Devices
Smart Gateways
ML Activation
Adopting Edge Computing
The Initial Context
The Project
The Final Outcomes and Next Steps
Summary

10. AI Application Architecture
Is This an AI/ML Problem?
Subfields of AI
Generative AI
Problems Fit for ML
Buy, Adapt, or Build?
Data Considerations
When to Buy
What Can You Buy?
How Adapting Works
AI Architectures
Understanding Unstructured Data
Generating Unstructured Data
Predicting Outcomes
Forecasting Values
Anomaly Detection
Personalization
Automation
Responsible AI
AI Principles
ML Fairness
Explainability
Summary

11. Architecting an ML Platform
ML Activities
Developing ML Models
Labeling Environment
Development Environment
User Environment
Preparing Data
Training ML Models
Deploying ML Models
Deploying to an Endpoint
Evaluate Model
Hybrid and Multicloud
Training-Serving Skew
Automation
Automate Training and Deployment
Orchestration with Pipelines
Continuous Evaluation and Training
Choosing the ML Framework
Team Skills
Task Considerations
User-Centric
Summary

12. Data Platform Modernization: A Model Case
New Technology for a New Era
The Need for Change
It Is Not Only a Matter of Technology
The Beginning of the Journey
The Current Environment
The Target Environment
The PoC Use Case
The RFP Responses Proposed by Cloud Vendors
The Target Environment
The Approach on Migration
The RFP Evaluation Process
The Scope of the PoC
The Execution of the PoC
The Final Decision
Peroration
Summary

Index
About the Authors