Building Data Streaming Applications with Apache Kafka.

✍ Scribed by Kumar, Manish; Singh, Chanchal

• Publisher: Packt Publishing
• Year: 2017
• Tongue: English
• Leaves: 269
• Category: Library

✦ Table of Contents

Cover
Title Page
Copyright
Credits
About the Authors
About the Reviewer
www.PacktPub.com
Customer Feedback
Table of Contents
Preface
Chapter 1: Introduction to Messaging Systems
Understanding the principles of messaging systems
Understanding messaging systems
Peeking into a point-to-point messaging system
Publish-subscribe messaging system
Advance Queuing Messaging Protocol
Using messaging systems in big data streaming applications
Summary
Chapter 2: Introducing Kafka the Distributed Messaging Platform
Kafka origins
Kafka's architecture
Message topics
Message partitions. Replication and replicated logsMessage producers
Message consumers
Role of Zookeeper
Summary
Chapter 3: Deep Dive into Kafka Producers
Kafka producer internals
Kafka Producer APIs
Producer object and ProducerRecord object
Custom partition
Additional producer configuration
Java Kafka producer example
Common messaging publishing patterns
Best practices
Summary
Chapter 4: Deep Dive into Kafka Consumers
Kafka consumer internals
Understanding the responsibilities of Kafka consumers
Kafka consumer APIs
Consumer configuration
Subscription and polling
Committing and polling. Additional configurationJava Kafka consumer
Scala Kafka consumer
Rebalance listeners
Common message consuming patterns
Best practices
Summary
Chapter 5: Building Spark Streaming Applications with Kafka
Introduction to Spark
Spark architecture
Pillars of Spark
The Spark ecosystem
Spark Streaming
Receiver-based integration
Disadvantages of receiver-based approach
Java example for receiver-based integration
Scala example for receiver-based integration
Direct approach
Java example for direct approach
Scala example for direct approach. Use case log processing --
fraud IP detectionMaven
Producer
Property reader
Producer code
Fraud IP lookup
Expose hive table
Streaming code
Summary
Chapter 6: Building Storm Applications with Kafka
Introduction to Apache Storm
Storm cluster architecture
The concept of a Storm application
Introduction to Apache Heron
Heron architecture
Heron topology architecture
Integrating Apache Kafka with Apache Storm --
Java
Example
Integrating Apache Kafka with Apache Storm --
Scala
Use case --
log processing in Storm, Kafka, Hive
Producer
Producer code
Fraud IP lookup. Running the projectSummary
Chapter 7: Using Kafka with Confluent Platform
Introduction to Confluent Platform
Deep driving into Confluent architecture
Understanding Kafka Connect and Kafka Stream
Kafka Streams
Playing with Avro using Schema Registry
Moving Kafka data to HDFS
Camus
Running Camus
Gobblin
Gobblin architecture
Kafka Connect
Flume
Summary
Chapter 8: Building ETL Pipelines Using Kafka
Considerations for using Kafka in ETL pipelines
Introducing Kafka Connect
Deep dive into Kafka Connect
Introductory examples of using Kafka Connect
Kafka Connect common use cases.

✦ Subjects

Apache Kafka;Real-time data processing;Application software -- Development;COMPUTERS -- General