Kafka: The Definitive Guide Confluent Kafka

• Cover
• Confluent
• Copyright
• Table of Contents
• Foreword
• Preface
  • Who Should Read This Book
  • Conventions Used in This Book
  • Using Code Examples
  • O’Reilly Safari
  • How to Contact Us
  • Acknowledgments
• Chapter 1. Meet Kafka
  • Publish/Subscribe Messaging
    • How It Starts
    • Individual Queue Systems
  • Enter Kafka
    • Messages and Batches
    • Schemas
    • Topics and Partitions
    • Producers and Consumers
    • Brokers and Clusters
    • Multiple Clusters
  • Why Kafka?
    • Multiple Producers
    • Multiple Consumers
    • Disk-Based Retention
    • Scalable
    • High Performance
  • The Data Ecosystem
    • Use Cases
  • Kafka’s Origin
    • LinkedIn’s Problem
    • The Birth of Kafka
    • Open Source
    • The Name
  • Getting Started with Kafka
• Chapter 2. Installing Kafka
  • First Things First
    • Choosing an Operating System
    • Installing Java
    • Installing Zookeeper
  • Installing a Kafka Broker
  • Broker Configuration
    • General Broker
    • Topic Defaults
  • Hardware Selection
    • Disk Throughput
    • Disk Capacity
    • Memory
    • Networking
    • CPU
  • Kafka in the Cloud
  • Kafka Clusters
    • How Many Brokers?
    • Broker Configuration
    • OS Tuning
  • Production Concerns
    • Garbage Collector Options
    • Datacenter Layout
    • Colocating Applications on Zookeeper
  • Summary
• Chapter 3. Kafka Producers: Writing Messages to Kafka
  • Producer Overview
  • Constructing a Kafka Producer
  • Sending a Message to Kafka
    • Sending a Message Synchronously
    • Sending a Message Asynchronously
  • Configuring Producers
  • Serializers
    • Custom Serializers
    • Serializing Using Apache Avro
    • Using Avro Records with Kafka
  • Partitions
  • Old Producer APIs
  • Summary
• Chapter 4. Kafka Consumers: Reading Data from Kafka
  • Kafka Consumer Concepts
    • Consumers and Consumer Groups
    • Consumer Groups and Partition Rebalance
  • Creating a Kafka Consumer
  • Subscribing to Topics
  • The Poll Loop
  • Configuring Consumers
  • Commits and Offsets
    • Automatic Commit
    • Commit Current Offset
    • Asynchronous Commit
    • Combining Synchronous and Asynchronous Commits
    • Commit Specified Offset
  • Rebalance Listeners
  • Consuming Records with Specific Offsets
  • But How Do We Exit?
  • Deserializers
  • Standalone Consumer: Why and How to Use a Consumer Without a Group
  • Older Consumer APIs
  • Summary
• Chapter 5. Kafka Internals
  • Cluster Membership
  • The Controller
  • Replication
  • Request Processing
    • Produce Requests
    • Fetch Requests
    • Other Requests
  • Physical Storage
    • Partition Allocation
    • File Management
    • File Format
    • Indexes
    • Compaction
    • How Compaction Works
    • Deleted Events
    • When Are Topics Compacted?
  • Summary
• Chapter 6. Reliable Data Delivery
  • Reliability Guarantees
  • Replication
  • Broker Configuration
    • Replication Factor
    • Unclean Leader Election
    • Minimum In-Sync Replicas
  • Using Producers in a Reliable System
    • Send Acknowledgments
    • Configuring Producer Retries
    • Additional Error Handling
  • Using Consumers in a Reliable System
    • Important Consumer Configuration Properties for Reliable Processing
    • Explicitly Committing Offsets in Consumers
  • Validating System Reliability
    • Validating Configuration
    • Validating Applications
    • Monitoring Reliability in Production
  • Summary
• Chapter 7. Building Data Pipelines
  • Considerations When Building Data Pipelines
    • Timeliness
    • Reliability
    • High and Varying Throughput
    • Data Formats
    • Transformations
    • Security
    • Failure Handling
    • Coupling and Agility
  • When to Use Kafka Connect Versus Producer and Consumer
  • Kafka Connect
    • Running Connect
    • Connector Example: File Source and File Sink
    • Connector Example: MySQL to Elasticsearch
    • A Deeper Look at Connect
  • Alternatives to Kafka Connect
    • Ingest Frameworks for Other Datastores
    • GUI-Based ETL Tools
    • Stream-Processing Frameworks
  • Summary
• Chapter 8. Cross-Cluster Data Mirroring
  • Use Cases of Cross-Cluster Mirroring
  • Multicluster Architectures
    • Some Realities of Cross-Datacenter Communication
    • Hub-and-Spokes Architecture
    • Active-Active Architecture
    • Active-Standby Architecture
    • Stretch Clusters
  • Apache Kafka’s MirrorMaker
    • How to Configure
    • Deploying MirrorMaker in Production
    • Tuning MirrorMaker
  • Other Cross-Cluster Mirroring Solutions
    • Uber uReplicator
    • Confluent’s Replicator
  • Summary
• Chapter 9. Administering Kafka
  • Topic Operations
    • Creating a New Topic
    • Adding Partitions
    • Deleting a Topic
    • Listing All Topics in a Cluster
    • Describing Topic Details
  • Consumer Groups
    • List and Describe Groups
    • Delete Group
    • Offset Management
  • Dynamic Configuration Changes
    • Overriding Topic Configuration Defaults
    • Overriding Client Configuration Defaults
    • Describing Configuration Overrides
    • Removing Configuration Overrides
  • Partition Management
    • Preferred Replica Election
    • Changing a Partition’s Replicas
    • Changing Replication Factor
    • Dumping Log Segments
    • Replica Verification
  • Consuming and Producing
    • Console Consumer
    • Console Producer
  • Client ACLs
  • Unsafe Operations
    • Moving the Cluster Controller
    • Killing a Partition Move
    • Removing Topics to Be Deleted
    • Deleting Topics Manually
  • Summary
• Chapter 10. Monitoring Kafka
  • Metric Basics
    • Where Are the Metrics?
    • Internal or External Measurements
    • Application Health Checks
    • Metric Coverage
  • Kafka Broker Metrics
    • Under-Replicated Partitions
    • Broker Metrics
    • Topic and Partition Metrics
    • JVM Monitoring
    • OS Monitoring
    • Logging
  • Client Monitoring
    • Producer Metrics
    • Consumer Metrics
    • Quotas
  • Lag Monitoring
  • End-to-End Monitoring
  • Summary
• Chapter 11. Stream Processing
  • What Is Stream Processing?
  • Stream-Processing Concepts
    • Time
    • State
    • Stream-Table Duality
    • Time Windows
  • Stream-Processing Design Patterns
    • Single-Event Processing
    • Processing with Local State
    • Multiphase Processing/Repartitioning
    • Processing with External Lookup: Stream-Table Join
    • Streaming Join
    • Out-of-Sequence Events
    • Reprocessing
  • Kafka Streams by Example
    • Word Count
    • Stock Market Statistics
    • Click Stream Enrichment
  • Kafka Streams: Architecture Overview
    • Building a Topology
    • Scaling the Topology
    • Surviving Failures
  • Stream Processing Use Cases
  • How to Choose a Stream-Processing Framework
  • Summary
• Appendix A. Installing Kafka on Other Operating Systems
  • Installing on Windows
    • Using Windows Subsystem for Linux
    • Using Native Java
  • Installing on MacOS
    • Using Homebrew
    • Installing Manually
• Index
• About the Authors
• Colophon