Building Medallion Architectures

Designing with Delta Lake and Spark

Paperback Engels 2025 1e druk 9781098178833
€ 69,78
Levertijd ± 16 werkdagen
Gratis verzonden
Past door de brievenbus

Samenvatting

To deliver the insights that give them a competitive advantage, organizations increasingly turn to the proven Medallion architecture. Yet implementing a robust data architecture can be difficult, particularly when it comes to using the Medallion architecture's Bronze, Silver, and Gold layers—done wrong, it can hamper your ability to make data-driven decisions. This practical guide helps you build a Medallion architecture the right way with Azure Databricks and Microsoft Fabric.

Drawing on hands-on experience from the field, Piethein Strengholt demystifies common assumptions and complex problems you'll face when embarking on a new data architecture. Architects and engineers of all stripes will find answers to the most typical questions along with insights from real organizations about what's worked, what hasn't, and why. You'll learn:Learn how to build a Medallion architecture with Azure Databricks and Microsoft FabricGain insights from three real case studies that illustrate practical field experience and lessons learnedExplore scaling considerations, including governance, security, generative AI, and moreMake informed decisions when designing or implementing new data architecturesGet proven patterns for success that align with broader organizational objectives

Lezersrecensies

Wees de eerste die een lezersrecensie schrijft!

Over Piethein Strengholt

Piethein Strengholt likes to find practical and lasting solutions to complex problems. After working for more than a decade as a strategy consultant and freelance application developer, he joined ABN AMRO as a principal architect to accelerate subjects like data management, cloud, and integration. In this exciting role, he oversees the company’s data strategy and its impact on the organization. He lives in the Netherlands with his family.

Andere boeken door Piethein Strengholt

Inhoudsopgave

Foreword
Preface
Who Should Read This Book
Navigating This Book
Conventions Used in This Book
Using Code Examples
O’Reilly Online Learning
How to Contact Us
Acknowledgments

I. Understanding the Medallion Framework
1. The Evolution of Data Architecture
What Is a Medallion Architecture?
A Brief History of Data Warehouse Architecture
OLTP Systems
Data Warehouses
The Staging Area
Inmon Methodology
Kimball Methodology
Key Takeaways from Traditional Data Warehouses
A Brief History of Data Lakes
Hadoop’s Distributed File System
MapReduce
Apache Hive
Spark Project
Moving Forward with Data Lakes
A Brief History of Lakehouse Architecture
Founders of Spark
Emergence of Open Table Formats
The Rise of Lakehouse Architectures
Medallion Architecture and Its Practical Challenges
Conclusion

2. Laying the Groundwork
Foundational Preconditions
Extra Landing Zones
Raw Data
Batch Processing
Real-Time Data Processing
Spark Structured Streaming
Change Data Feed
Change Data Capture
Considerations and Learning Resources
ETL and Orchestration Tools
Managing Delta Tables
Z-Ordering
V-Ordering
Table Partitioning
Liquid Clustering
Compaction and Optimized Writes
DeltaLog
Conclusion

3. Demystifying the Medallion Architecture
The Three-Layered Design
Bronze Layer
Processing Hierarchy
Processing Full Data Loads
Processing Incremental Data Loads
Data Historization Within the Bronze Layer
Afgekapt op 60 regels

Managementboek Top 100

€ 69,78
Levertijd ongeveer 16 werkdagen
Gratis verzonden

Rubrieken

Producten

    Auteurs

      Trefwoorden

        Artikelen

          Building Medallion Architectures