Fabian Hueske,
Vasiliki Kalavri
Stream Processing with Apache Flink
Fundamentals, Implementation, and Operation of Streaming Applications
Paperback Engels 2019 1e druk 9781491974292Levertijd ± 16 werkdagen
Gratis verzonden
Samenvatting
Get started with Apache Flink, the open source framework that powers some of the world’s largest stream processing applications. With this practical book, you’ll explore the fundamental concepts of parallel stream processing and discover how this technology differs from traditional batch data processing.
Longtime Apache Flink committers Fabian Hueske and Vasia Kalavri show you how to implement scalable streaming applications with Flink’s DataStream API and continuously run and maintain these applications in operational environments.
Stream processing is ideal for many use cases, including low-latency ETL, streaming analytics, and real-time dashboards as well as fraud detection, anomaly detection, and alerting. You can process continuous data of any kind, including user interactions, financial transactions, and IoT data, as soon as you generate them.
- Learn concepts and challenges of distributed stateful stream processing
- Explore Flink’s system architecture, including its event-time processing mode and fault-tolerance model
- Understand the fundamentals and building blocks of the DataStream API, including its time-based and statefuloperators
- Read data from and write data to external systems with exactly-once consistency
- Deploy and configure Flink clusters
- Operate continuously running streaming applications
Specificaties
ISBN13:9781491974292
Trefwoorden:Programmeren, applicatieontwikkeling, Programmeertalen, Apache Flink, stream processing
Taal:Engels
Bindwijze:paperback
Aantal pagina's:312
Uitgever:O'Reilly
Druk:1
Verschijningsdatum:23-4-2019
Hoofdrubriek:IT-management / ICT
Lezersrecensies
Wees de eerste die een lezersrecensie schrijft!
Inhoudsopgave
Preface
What You Will Learn in This Book
Conventions Used in This Book
Using Code Examples
O’Reilly Online Learning
How to Contact Us
Acknowledgments
1. Introduction to Stateful Stream Processing
Traditional Data Infrastructures
Transactional Processing
Analytical Processing
Stateful Stream Processing
Event-Driven Applications
Data Pipelines
Streaming Analytics
The Evolution of Open Source Stream Processing
A Bit of History
A Quick Look at Flink
Running Your First Flink Application
Summary
2. Stream Processing Fundamentals
Introduction to Dataflow Programming
Dataflow Graphs
Data Parallelism and Task Parallelism
Data Exchange Strategies
Processing Streams in Parallel
Latency and Throughput
Operations on Data Streams
Time Semantics
What Does One Minute Mean in Stream Processing?
Processing Time
Event Time
Watermarks
Processing Time Versus Event Time
State and Consistency Models
Task Failures
Result Guarantees
Summary
3. The Architecture of Apache Flink
System Architecture
Components of a Flink Setup
Application Deployment
Task Execution
Highly Available Setup
Data Transfer in Flink
Credit-Based Flow Control
Task Chaining
Event-Time Processing
Timestamps
Watermarks
Watermark Propagation and Event Time
Timestamp Assignment and Watermark Generation
State Management
Operator State
Keyed State
State Backends
Scaling Stateful Operators
What You Will Learn in This Book
Conventions Used in This Book
Using Code Examples
O’Reilly Online Learning
How to Contact Us
Acknowledgments
1. Introduction to Stateful Stream Processing
Traditional Data Infrastructures
Transactional Processing
Analytical Processing
Stateful Stream Processing
Event-Driven Applications
Data Pipelines
Streaming Analytics
The Evolution of Open Source Stream Processing
A Bit of History
A Quick Look at Flink
Running Your First Flink Application
Summary
2. Stream Processing Fundamentals
Introduction to Dataflow Programming
Dataflow Graphs
Data Parallelism and Task Parallelism
Data Exchange Strategies
Processing Streams in Parallel
Latency and Throughput
Operations on Data Streams
Time Semantics
What Does One Minute Mean in Stream Processing?
Processing Time
Event Time
Watermarks
Processing Time Versus Event Time
State and Consistency Models
Task Failures
Result Guarantees
Summary
3. The Architecture of Apache Flink
System Architecture
Components of a Flink Setup
Application Deployment
Task Execution
Highly Available Setup
Data Transfer in Flink
Credit-Based Flow Control
Task Chaining
Event-Time Processing
Timestamps
Watermarks
Watermark Propagation and Event Time
Timestamp Assignment and Watermark Generation
State Management
Operator State
Keyed State
State Backends
Scaling Stateful Operators
Checkpoints, Savepoints, and State Recovery
Consistent Checkpoints
Recovery from a Consistent Checkpoint
Flink’s Checkpointing Algorithm
Performace Implications of Checkpointing
Savepoints
Summary
4. Setting Up a Development Environment for Apache Flink
Required Software
Run and Debug Flink Applications in an IDE
Import the Book’s Examples in an IDE
Run Flink Applications in an IDE
Debug Flink Applications in an IDE
Bootstrap a Flink Maven Project
Summary
5. The DataStream API (v1.7)
Hello, Flink!
Set Up the Execution Environment
Read an Input Stream
Apply Transformations
Output the Result
Execute
Transformations
Basic Transformations
KeyedStream Transformations
Multistream Transformations
Distribution Transformations
Setting the Parallelism
Types
Supported Data Types
Creating Type Information for Data Types
Explicitly Providing Type Information
Defining Keys and Referencing Fields
Field Positions
Field Expressions
Key Selectors
Implementing Functions
Function Classes
Lambda Functions
Rich Functions
Including External and Flink Dependencies
Summary
6. Time-Based and Window Operators
Configuring Time Characteristics
Assigning Timestamps and Generating Watermarks
Watermarks, Latency, and Completeness
Process Functions
TimerService and Timers
Emitting to Side Outputs
CoProcessFunction
Window Operators
Defining Window Operators
Built-in Window Assigners
Applying Functions on Windows
Customizing Window Operators
Joining Streams on Time
Interval Join
Window Join
Handling Late Data
Dropping Late Events
Redirecting Late Events
Updating Results by Including Late Events
Summary
7. Stateful Operators and Applications
Implementing Stateful Functions
Declaring Keyed State at RuntimeContext
Implementing Operator List State with the ListCheckpointed Interface
Using Connected Broadcast State
Using the CheckpointedFunction Interface
Receiving Notifications About Completed Checkpoints
Enabling Failure Recovery for Stateful Applications
Ensuring the Maintainability of Stateful Applications
Specifying Unique Operator Identifiers
Defining the Maximum Parallelism of Keyed State Operators
Performance and Robustness of Stateful Applications
Choosing a State Backend
Choosing a State Primitive
Preventing Leaking State
Evolving Stateful Applications
Updating an Application without Modifying Existing State
Removing State from an Application
Modifying the State of an Operator
Queryable State
Architecture and Enabling Queryable State
Exposing Queryable State
Querying State from External Applications
Summary
8. Reading from and Writing to External Systems
Application Consistency Guarantees
Idempotent Writes
Transactional Writes
Provided Connectors
Apache Kafka Source Connector
Apache Kafka Sink Connector
Filesystem Source Connector
Filesystem Sink Connector
Apache Cassandra Sink Connector
Implementing a Custom Source Function
Resettable Source Functions
Source Functions, Timestamps, and Watermarks
Implementing a Custom Sink Function
Idempotent Sink Connectors
Transactional Sink Connectors
Asynchronously Accessing External Systems
Summary
9. Setting Up Flink for Streaming Applications
Deployment Modes
Standalone Cluster
Docker
Apache Hadoop YARN
Kubernetes
Highly Available Setups
HA Standalone Setup
HA YARN Setup
HA Kubernetes Setup
Integration with Hadoop Components
Filesystem Configuration
System Configuration
Java and Classloading
CPU
Main Memory and Network Buffers
Disk Storage
Checkpointing and State Backends
Security
Summary
10. Operating Flink and Streaming Applications
Running and Managing Streaming Applications
Savepoints
Managing Applications with the Command-Line Client
Managing Applications with the REST API
Bundling and Deploying Applications in Containers
Controlling Task Scheduling
Controlling Task Chaining
Defining Slot-Sharing Groups
Tuning Checkpointing and Recovery
Configuring Checkpointing
Configuring State Backends
Configuring Recovery
Monitoring Flink Clusters and Applications
Flink Web UI
Metric System
Monitoring Latency
Configuring the Logging Behavior
Summary
11. Where to Go from Here?
The Rest of the Flink Ecosystem
The DataSet API for Batch Processing
Table API and SQL for Relational Analysis
FlinkCEP for Complex Event Processing and Pattern Matching
Gelly for Graph Processing
A Welcoming Community
Index
Consistent Checkpoints
Recovery from a Consistent Checkpoint
Flink’s Checkpointing Algorithm
Performace Implications of Checkpointing
Savepoints
Summary
4. Setting Up a Development Environment for Apache Flink
Required Software
Run and Debug Flink Applications in an IDE
Import the Book’s Examples in an IDE
Run Flink Applications in an IDE
Debug Flink Applications in an IDE
Bootstrap a Flink Maven Project
Summary
5. The DataStream API (v1.7)
Hello, Flink!
Set Up the Execution Environment
Read an Input Stream
Apply Transformations
Output the Result
Execute
Transformations
Basic Transformations
KeyedStream Transformations
Multistream Transformations
Distribution Transformations
Setting the Parallelism
Types
Supported Data Types
Creating Type Information for Data Types
Explicitly Providing Type Information
Defining Keys and Referencing Fields
Field Positions
Field Expressions
Key Selectors
Implementing Functions
Function Classes
Lambda Functions
Rich Functions
Including External and Flink Dependencies
Summary
6. Time-Based and Window Operators
Configuring Time Characteristics
Assigning Timestamps and Generating Watermarks
Watermarks, Latency, and Completeness
Process Functions
TimerService and Timers
Emitting to Side Outputs
CoProcessFunction
Window Operators
Defining Window Operators
Built-in Window Assigners
Applying Functions on Windows
Customizing Window Operators
Joining Streams on Time
Interval Join
Window Join
Handling Late Data
Dropping Late Events
Redirecting Late Events
Updating Results by Including Late Events
Summary
7. Stateful Operators and Applications
Implementing Stateful Functions
Declaring Keyed State at RuntimeContext
Implementing Operator List State with the ListCheckpointed Interface
Using Connected Broadcast State
Using the CheckpointedFunction Interface
Receiving Notifications About Completed Checkpoints
Enabling Failure Recovery for Stateful Applications
Ensuring the Maintainability of Stateful Applications
Specifying Unique Operator Identifiers
Defining the Maximum Parallelism of Keyed State Operators
Performance and Robustness of Stateful Applications
Choosing a State Backend
Choosing a State Primitive
Preventing Leaking State
Evolving Stateful Applications
Updating an Application without Modifying Existing State
Removing State from an Application
Modifying the State of an Operator
Queryable State
Architecture and Enabling Queryable State
Exposing Queryable State
Querying State from External Applications
Summary
8. Reading from and Writing to External Systems
Application Consistency Guarantees
Idempotent Writes
Transactional Writes
Provided Connectors
Apache Kafka Source Connector
Apache Kafka Sink Connector
Filesystem Source Connector
Filesystem Sink Connector
Apache Cassandra Sink Connector
Implementing a Custom Source Function
Resettable Source Functions
Source Functions, Timestamps, and Watermarks
Implementing a Custom Sink Function
Idempotent Sink Connectors
Transactional Sink Connectors
Asynchronously Accessing External Systems
Summary
9. Setting Up Flink for Streaming Applications
Deployment Modes
Standalone Cluster
Docker
Apache Hadoop YARN
Kubernetes
Highly Available Setups
HA Standalone Setup
HA YARN Setup
HA Kubernetes Setup
Integration with Hadoop Components
Filesystem Configuration
System Configuration
Java and Classloading
CPU
Main Memory and Network Buffers
Disk Storage
Checkpointing and State Backends
Security
Summary
10. Operating Flink and Streaming Applications
Running and Managing Streaming Applications
Savepoints
Managing Applications with the Command-Line Client
Managing Applications with the REST API
Bundling and Deploying Applications in Containers
Controlling Task Scheduling
Controlling Task Chaining
Defining Slot-Sharing Groups
Tuning Checkpointing and Recovery
Configuring Checkpointing
Configuring State Backends
Configuring Recovery
Monitoring Flink Clusters and Applications
Flink Web UI
Metric System
Monitoring Latency
Configuring the Logging Behavior
Summary
11. Where to Go from Here?
The Rest of the Flink Ecosystem
The DataSet API for Batch Processing
Table API and SQL for Relational Analysis
FlinkCEP for Complex Event Processing and Pattern Matching
Gelly for Graph Processing
A Welcoming Community
Index
Afgekapt op 60 regels
Rubrieken
- advisering
- algemeen management
- coaching en trainen
- communicatie en media
- economie
- financieel management
- inkoop en logistiek
- internet en social media
- it-management / ict
- juridisch
- leiderschap
- marketing
- mens en maatschappij
- non-profit
- ondernemen
- organisatiekunde
- personal finance
- personeelsmanagement
- persoonlijke effectiviteit
- projectmanagement
- psychologie
- reclame en verkoop
- strategisch management
- verandermanagement
- werk en loopbaan