Big Data SMACK A Guide To Apache Spark, Mesos, Akka, Cassandra, And Kafka

• Contents at a Glance
• Contents
• About the Authors
• About the Technical Reviewer
• Acknowledgments
• Introduction
• Part I: Introduction
  • Chapter 1: Big Data, Big Challenges
    • Big Data Problems
    • Infrastructure Needs
    • ETL
    • Lambda Architecture
    • Hadoop
    • Data Center Operation
      • The Open Source Reign
      • The Data Store Diversification
    • Is SMACK the Solution?
  • Chapter 2: Big Data, Big Solutions
    • Traditional vs. Modern (Big) Data
    • SMACK in a Nutshell
    • Apache Spark vs. MapReduce
    • The Engine
    • The Model
    • The Broker
    • The Storage
    • The Container
    • Summary
• Part II: Playing SMACK
  • Chapter 3: The Language: Scala
    • Functional Programming
      • Predicate
      • Literal Functions
      • Implicit Loops
    • Collections Hierarchy
      • Sequences
      • Maps
      • Sets
    • Choosing Collections
      • Sequences
        • Immutable Sequences
          • Mutable Sequences
      • Maps
      • Sets
    • Traversing
      • foreach
      • for
      • Iterators
    • Mapping
    • Flattening
    • Filtering
    • Extracting
    • Splitting
    • Unicity
    • Merging
    • Lazy Views
    • Sorting
    • Streams
    • Arrays
    • ArrayBuffers
    • Queues
    • Stacks
    • Ranges
    • Summary
  • Chapter 4: The Model: Akka
    • The Actor Model
      • Threads and Labyrinths
      • Actors 101
    • Installing Akka
    • Akka Actors
      • Actors
      • Actor System
      • Actor Reference
      • Actor Communication
      • Actor Lifecycle
      • Starting Actors
      • Stopping Actors
      • Killing Actors
      • Shutting down the Actor System
      • Actor Monitoring
      • Looking up Actors
      • Actor Code of Conduct
        • Actors Do Not Block Each Other
        • Communication is only via Messages
        • Messages must be Immutable
        • Messages must be Self-Contained
    • Summary
  • Chapter 5: Storage: Apache Cassandra
    • Once Upon a Time...
      • Modern Cassandra
    • NoSQL Everywhere
    • The Memory Value
      • Key-Value and Column
        • Key-Value
        • Column-Family
    • Why Cassandra?
      • The Data Model
    • Cassandra 101
      • Installation
        • Prerequisites
        • File Download
        • Start
        • Validation
        • CQL
          • CQL Shell
          • CQL Commands
    • Beyond the Basics
      • Client-Server
      • Other Clients
        • Drivers
          • Java
            • Get Dependence
            • Snippet
          • Ruby
            • Get Dependence
            • Snippet
          • Node
            • Get Dependence
            • Snippet
      • Apache Spark-Cassandra Connector
      • Installing the Connector
      • Establishing the Connection
    • More Than One Is Better
      • cassandra.yaml
      • Setting the Cluster
        • Machine01
        • Machine02
        • Machine03
        • Booting the Cluster
    • Putting It All Together
  • Chapter 6: The Engine: Apache Spark
    • Introducing Spark
      • Apache Spark Download
      • Let’s Kick the Tires
      • Loading a Data File
      • Loading Data from S3
    • Spark Architecture
      • SparkContext
      • Creating a SparkContext
      • SparkContext Metadata
      • SparkContext Methods
    • Working with RDDs
      • Standalone Apps
        • Initializing the SparkContext
        • Standalone Programs
        • Run the Program
      • RDD Operations
        • Transformations
        • Actions
        • RDD Persistence (Caching)
    • Spark in Cluster Mode
      • Runtime Architecture
      • Driver
        • Divide a Program into Tasks
        • Scheduling Tasks on Executors
      • Executor
      • Cluster Manager
      • Program Execution
      • Application Deployment
      • Running in Cluster Mode
      • Spark Standalone Mode
      • Running Spark on EC2
      • Running Spark on Mesos
      • Submitting Our Application
      • Configuring Resources
      • High Availability
    • Spark Streaming
      • Spark Streaming Architecture
      • Transformations
        • Stateless Transformations
        • Stateful Transformations
          • Windowed Operations
          • Update State by Key
        • Output Operations
      • 24/7 Spark Streaming
      • Checkpointing
      • Spark Streaming Performance
        • Parallelism Techniques
        • Window Size and Batch Size
        • Garbage Collector
    • Summary
  • Chapter 7: The Manager: Apache Mesos
    • Divide et Impera (Divide and Rule)
    • Distributed Systems
      • Why Are They Important?
    • It Is Difficult to Have a Distributed System
    • Ta-dah!! Apache Mesos
    • Mesos Framework
      • Architecture
    • Mesos 101
      • Installation
        • Get the Installer
        • Missing Dependencies
        • Start Mesos
          • Master Server
          • Slave Server
      • Teaming
        • ZooKeeper
          • Installation
        • Chronos
          • Installation
          • Run
        • Marathon
          • Installation
        • Aurora
          • Overview
    • Let’s Talk About Clusters
      • Apache Mesos and Apache Kafka
        • JDK Validation
        • Gradle Validation
        • Export libmesos Location
      • Mesos and Apache Spark
      • The Best Is Yet to Come
      • Summary
  • Chapter 8: The Broker: Apache Kafka
    • Kafka Introduction
      • Born in the Fast Data Era
      • Use Cases
    • Kafka Installation
      • Installing Java
      • Installing Kafka
      • Importing Kafka
    • Kafka in Cluster
      • Single Node–Single Broker Cluster
        • Starting the Broker
        • Creating a Topic
        • Starting a Producer
        • Starting a Consumer
      • Single Node–Multiple Broker Cluster
        • Starting the Brokers
        • Creating a Topic
        • Starting a Producer
        • Starting a Consumer
      • Multiple Node–Multiple Broker Cluster
      • Broker Properties
    • Kafka Architecture
      • Log Compaction
      • Kafka Design
      • Message Compression
      • Replication
    • Kafka Producers
      • Producer API
      • Scala Producers
        • Step 1. Import Classes
        • Step 2. Define Properties
        • Step 3. Build and Send the Message
        • Step 4. Create the Topic
        • Step 5. Compile the Producer
        • Step 6. Run the Producer
        • Step 7. Run a Consumer
      • Producers with Custom Partitioning
        • Step 1. Import Classes
        • Step 2. Define Properties
        • Step 3. Implement the Partitioner class
        • Step 4. Build and Send the Message
        • Step 5. Create the Topic
        • Step 6. Compile the Programs
        • Step 7. Run the Producer
        • Step 8. Run a Consumer
      • Producer Properties
    • Kafka Consumers
      • Consumer API
      • Simple Scala Consumers
        • Step 1. Import Classes
        • Step 2. Define Properties
        • Step 3. Code the SimpleConsumer
        • Step 4. Create the Topic
        • Step 5. Compile the Program
        • Step 6. Run the Producer
        • Step 7. Run the Consumer
      • Multithread Scala Consumers
        • Step 1. Import Classes
        • Step 2. Define Properties
        • Step 3. Code the MultiThreadConsumer
        • Step 4. Create the Topic
        • Step 5. Compile the Program
        • Step 6. Run the Producer
        • Step 7. Run the Consumer
      • Consumer Properties
    • Kafka Integration
      • Integration with Apache Spark
    • Kafka Administration
      • Cluster Tools
      • Adding Servers
        • Kafka Topic Tools
        • Cluster Mirroring
    • Summary
• Part III: Improving SMACK
  • Chapter 9: Fast Data Patterns
    • Fast Data
      • Fast Data at a Glance
      • Beyond Big Data
      • Fast Data Characteristics
        • Fast Ingestion
        • Analysis Streaming
        • Per Event Transactions
      • Fast Data and Hadoop
      • Data Enrichment
      • Queries
    • ACID vs. CAP
      • ACID Properties
      • CAP Theorem
      • Consistency
      • CRDT
    • Integrating Streaming and Transactions
      • Pattern 1: Reject Requests Beyond a Threshold
      • Pattern 2: Alerting on Predicted Trends Variation
      • When Not to Integrate Streaming and Transactions
      • Aggregation Techniques
    • Streaming Transformations
      • Pattern 3: Use Streaming Transformations to Avoid ETL
      • Pattern 4: Connect Big Data Analytics to Real-Time Stream Processing
      • Pattern 5: Use Loose Coupling to Improve Reliability
      • Points to Consider
    • Fault Recovery Strategies
      • Pattern 6: At-Most-Once Delivery
      • Pattern 7: At-Least-Once Delivery
      • Pattern 8: Exactly-Once Delivery
    • Tag Data Identifiers
      • Pattern 9: Use Upserts over Inserts
      • Pattern 10: Tag Data with Unique Identifiers
        • Subpattern: Fine-Grained Timestamps
        • Subpattern: Unique IDs at the Event Source
      • Pattern 11: Use Kafka Offsets as Unique Identifiers
      • When to Avoid Idempotency
      • Example: Switch Processing
        • Case 1: Ordered Requests
        • Case 2: Unordered Requests
    • Summary
  • Chapter 10: Data Pipelines
    • Data Pipeline Strategies and Principles
      • Asynchronous Message Passing
      • Consensus and Gossip
      • Data Locality
      • Failure Detection
      • Fault Tolerance/No Single Point of Failure
      • Isolation
      • Location Transparency
      • Parallelism
      • Partition for Scale
      • Replay for Any Point of Failure
      • Replicate for Resiliency
      • Scalable Infrastructure
      • Share Nothing/Masterless
      • Dynamo Systems Principles
    • Spark and Cassandra
      • Spark Streaming with Cassandra
        • Setting up Spark Streaming
        • Setting up Cassandra
        • Setting up SparkContext
        • Create a Streaming Context
        • Creating a Stream
          • Kafka Stream
          • Actor Stream
        • Enable Spark Streaming with Cassandra
      • Saving Data
        • Reading the StreamingContext from Cassandra
        • Loading and Analyzing Data from Cassandra
        • Saving data from a RDD to Cassandra
      • Saving Datasets to Cassandra
      • Saving a Collection of Tuples
      • Saving a Collection of Objects
      • Modifying CQL Collections
      • Saving Objects of Cassandra User-Defined Types
      • Converting Scala Options to Cassandra Options
      • Saving RDDs as New Tables
    • Akka and Kafka
    • Akka and Cassandra
      • Writing to Cassandra
      • Reading from Cassandra
      • Connecting to Cassandra
      • Scanning Tweets
      • Testing TweetScannerActor
    • Akka and Spark
    • Kafka and Cassandra
      • CQL Types Supported
      • Cassandra Sink
    • Summary
  • Chapter 11: Glossary
    • ACID
    • agent
    • API
    • BI
    • big data
    • CAP
    • CEP
    • client-server
    • cloud
    • cluster
    • column family
    • coordinator
    • CQL
    • CQLS
    • concurrency
    • commutative operations
    • CRDTs
    • dashboard
    • data feed
    • DBMS
    • determinism
    • dimension data
    • distributed computing.
    • driver
    • ETL
    • exabyte
    • exponential backoff
    • failover
    • fast data
    • gossip
    • graph database
    • HDSF
    • HTAP
    • IaaS
    • idempotence
    • IMDG
    • IoT
    • key-value
    • keyspace
    • latency
    • master-slave
    • metadata
    • NoSQL
    • operational analytics
    • RDBMS
    • real-time analytics
    • replication
    • PaaS
    • probabilistic data structures
    • SaaS
    • scalability
    • shared nothing
    • Spark-Cassandra Connector
    • streaming analytics
    • synchronization
    • unstructured data
• Index