Kafka: Real-Time Data and Stream Processing at Scale

✍ Scribed by Narkhede, Neha;Palino, Todd;Shapira, Gwen

• Publisher: O'Reilly Media, Incorporated
• Year: 2017
• Tongue: English
• Category: Library

✦ Synopsis

Table of Contents; Foreword; Preface; Who Should Read This Book; Conventions Used in This Book; Using Code Examples; O'Reilly Safari; How to Contact Us; Acknowledgments; Chapter 1. Meet Kafka; Publish/Subscribe Messaging; How It Starts; Individual Queue Systems; Enter Kafka; Messages and Batches; Schemas; Topics and Partitions; Producers and Consumers; Brokers and Clusters; Multiple Clusters; Why Kafka?; Multiple Producers; Multiple Consumers; Disk-Based Retention; Scalable; High Performance; The Data Ecosystem; Use Cases; Kafka's Origin; LinkedIn's Problem; The Birth of Kafka; Open Source.

✦ Table of Contents

Table of Contents
Foreword
Preface
Who Should Read This Book
Conventions Used in This Book
Using Code Examples
O'Reilly Safari
How to Contact Us
Acknowledgments
Chapter 1. Meet Kafka
Publish/Subscribe Messaging
How It Starts
Individual Queue Systems
Enter Kafka
Messages and Batches
Schemas
Topics and Partitions
Producers and Consumers
Brokers and Clusters
Multiple Clusters
Why Kafka?
Multiple Producers
Multiple Consumers
Disk-Based Retention
Scalable
High Performance
The Data Ecosystem
Use Cases
Kafka's Origin
LinkedIn's Problem
The Birth of Kafka
Open Source. The NameGetting Started with Kafka
Chapter 2. Installing Kafka
First Things First
Choosing an Operating System
Installing Java
Installing Zookeeper
Installing a Kafka Broker
Broker Configuration
General Broker
Topic Defaults
Hardware Selection
Disk Throughput
Disk Capacity
Memory
Networking
CPU
Kafka in the Cloud
Kafka Clusters
How Many Brokers?
Broker Configuration
OS Tuning
Production Concerns
Garbage Collector Options
Datacenter Layout
Colocating Applications on Zookeeper
Summary
Chapter 3. Kafka Producers: Writing Messages to Kafka
Producer Overview. Constructing a Kafka ProducerSending a Message to Kafka
Sending a Message Synchronously
Sending a Message Asynchronously
Configuring Producers
Serializers
Custom Serializers
Serializing Using Apache Avro
Using Avro Records with Kafka
Partitions
Old Producer APIs
Summary
Chapter 4. Kafka Consumers: Reading Data from Kafka
Kafka Consumer Concepts
Consumers and Consumer Groups
Consumer Groups and Partition Rebalance
Creating a Kafka Consumer
Subscribing to Topics
The Poll Loop
Configuring Consumers
Commits and Offsets
Automatic Commit
Commit Current Offset. Asynchronous CommitCombining Synchronous and Asynchronous Commits
Commit Specified Offset
Rebalance Listeners
Consuming Records with Specific Offsets
But How Do We Exit?
Deserializers
Standalone Consumer: Why and How to Use a Consumer Without a Group
Older Consumer APIs
Summary
Chapter 5. Kafka Internals
Cluster Membership
The Controller
Replication
Request Processing
Produce Requests
Fetch Requests
Other Requests
Physical Storage
Partition Allocation
File Management
File Format
Indexes
Compaction
How Compaction Works
Deleted Events
When Are Topics Compacted?
Summary. Chapter 6. Reliable Data DeliveryReliability Guarantees
Replication
Broker Configuration
Replication Factor
Unclean Leader Election
Minimum In-Sync Replicas
Using Producers in a Reliable System
Send Acknowledgments
Configuring Producer Retries
Additional Error Handling
Using Consumers in a Reliable System
Important Consumer Configuration Properties for Reliable Processing
Explicitly Committing Offsets in Consumers
Validating System Reliability
Validating Configuration
Validating Applications
Monitoring Reliability in Production
Summary
Chapter 7. Building Data Pipelines.

✦ Subjects

Electronic books