Spark The Definitive Guide Big Data Processing Made Simple

User Manual:

Open the PDF directly: View PDF .
Page Count: 600 [warning: Documents this large are best viewed by clicking the View PDF Link!]

Scroll down to view the document on your mobile browser.

Preface
- About the Authors
- Who This Book Is For
- Conventions Used in This Book
- Using Code Examples
- O’Reilly Safari
- How to Contact Us
- Acknowledgments
I. Gentle Overview of Big Data and Spark
1. What Is Apache Spark?
- Apache Spark’s Philosophy
- Context: The Big Data Problem
- History of Spark
- The Present and Future of Spark
- Running Spark
2. A Gentle Introduction to Spark
- Spark’s Basic Architecture
  - Spark Applications
- Spark’s Language APIs
- Spark’s APIs
- Starting Spark
- The SparkSession
- DataFrames
  - Partitions
- Transformations
  - Lazy Evaluation
- Actions
- Spark UI
- An End-to-End Example
  - DataFrames and SQL
- Conclusion
3. A Tour of Spark’s Toolset
- Running Production Applications
- Datasets: Type-Safe Structured APIs
- Structured Streaming
- Machine Learning and Advanced Analytics
- Lower-Level APIs
- SparkR
- Spark’s Ecosystem and Packages
- Conclusion
II. Structured APIs—DataFrames, SQL, and Datasets
4. Structured API Overview
- DataFrames and Datasets
- Schemas
- Overview of Structured Spark Types
- Overview of Structured API Execution
- Conclusion
5. Basic Structured Operations
- Schemas
- Columns and Expressions
  - Columns
    - Explicit column references
  - Expressions
    - Columns as expressions
    - Accessing a DataFrame’s columns
- Records and Rows
  - Creating Rows
- DataFrame Transformations
- Conclusion
6. Working with Different Types of Data
- Where to Look for APIs
- Converting to Spark Types
- Working with Booleans
- Working with Numbers
- Working with Strings
  - Regular Expressions
- Working with Dates and Timestamps
- Working with Nulls in Data
- Ordering
- Working with Complex Types
- Working with JSON
- User-Defined Functions
- Conclusion
7. Aggregations
- Aggregation Functions
- Grouping
  - Grouping with Expressions
  - Grouping with Maps
- Window Functions
- Grouping Sets
- User-Defined Aggregation Functions
- Conclusion
8. Joins
- Join Expressions
- Join Types
- Inner Joins
- Outer Joins
- Left Outer Joins
- Right Outer Joins
- Left Semi Joins
- Left Anti Joins
- Natural Joins
- Cross (Cartesian) Joins
- Challenges When Using Joins
  - Joins on Complex Types
  - Handling Duplicate Column Names
- How Spark Performs Joins
  - Communication Strategies
- Conclusion
9. Data Sources
- The Structure of the Data Sources API
- CSV Files
- JSON Files
- Parquet Files
  - Reading Parquet Files
    - Parquet options
  - Writing Parquet Files
- ORC Files
  - Reading Orc Files
  - Writing Orc Files
- SQL Databases
- Text Files
  - Reading Text Files
  - Writing Text Files
- Advanced I/O Concepts
- Conclusion
10. Spark SQL
- What Is SQL?
- Big Data and SQL: Apache Hive
- Big Data and SQL: Spark SQL
  - Spark’s Relationship to Hive
    - The Hive metastore
- How to Run Spark SQL Queries
- Catalog
- Tables
- Views
  - Creating Views
  - Dropping Views
- Databases
- Select Statements
  - case…when…then Statements
- Advanced Topics
- Miscellaneous Features
  - Configurations
  - Setting Configuration Values in SQL
- Conclusion
11. Datasets
- When to Use Datasets
- Creating Datasets
  - In Java: Encoders
  - In Scala: Case Classes
- Actions
- Transformations
  - Filtering
  - Mapping
- Joins
- Grouping and Aggregations
- Conclusion
III. Low-Level APIs
12. Resilient Distributed Datasets (RDDs)
- What Are the Low-Level APIs?
  - When to Use the Low-Level APIs?
  - How to Use the Low-Level APIs?
- About RDDs
- Creating RDDs
- Manipulating RDDs
- Transformations
- Actions
- Saving Files
- Caching
- Checkpointing
- Pipe RDDs to System Commands
- Conclusion
13. Advanced RDDs
- Key-Value Basics (Key-Value RDDs)
- Aggregations
- CoGroups
- Joins
  - Inner Join
  - zips
- Controlling Partitions
- Custom Serialization
- Conclusion
14. Distributed Shared Variables
- Broadcast Variables
- Accumulators
  - Basic Example
  - Custom Accumulators
- Conclusion
IV. Production Applications
15. How Spark Runs on a Cluster
- The Architecture of a Spark Application
  - Execution Modes
- The Life Cycle of a Spark Application (Outside Spark)
- The Life Cycle of a Spark Application (Inside Spark)
- Execution Details
  - Pipelining
  - Shuffle Persistence
- Conclusion
16. Developing Spark Applications
- Writing Spark Applications
- Testing Spark Applications
- The Development Process
- Launching Applications
  - Application Launch Examples
- Configuring Applications
- Conclusion
17. Deploying Spark
- Where to Deploy Your Cluster to Run Spark Applications
  - On-Premises Cluster Deployments
  - Spark in the Cloud
- Cluster Managers
- Miscellaneous Considerations
- Conclusion
18. Monitoring and Debugging
- The Monitoring Landscape
- What to Monitor
  - Driver and Executor Processes
  - Queries, Jobs, Stages, and Tasks
- Spark Logs
- The Spark UI
- Debugging and Spark First Aid
- Conclusion
19. Performance Tuning
- Indirect Performance Enhancements
- Direct Performance Enhancements
- Conclusion
V. Streaming
20. Stream Processing Fundamentals
- What Is Stream Processing?
- Stream Processing Design Points
- Spark’s Streaming APIs
  - The DStream API
  - Structured Streaming
- Conclusion
21. Structured Streaming Basics
- Structured Streaming Basics
- Core Concepts
- Structured Streaming in Action
- Transformations on Streams
- Input and Output
- Streaming Dataset API
- Conclusion
22. Event-Time and Stateful Processing
- Event Time
- Stateful Processing
- Arbitrary Stateful Processing
- Event-Time Basics
- Windows on Event Time
  - Tumbling Windows
    - Sliding windows
  - Handling Late Data with Watermarks
- Dropping Duplicates in a Stream
- Arbitrary Stateful Processing
- Conclusion
23. Structured Streaming in Production
- Fault Tolerance and Checkpointing
- Updating Your Application
- Metrics and Monitoring
- Alerting
- Advanced Monitoring with the Streaming Listener
- Conclusion
VI. Advanced Analytics and Machine Learning
24. Advanced Analytics and Machine Learning Overview
- A Short Primer on Advanced Analytics
- Spark’s Advanced Analytics Toolkit
  - What Is MLlib?
    - When and why should you use MLlib (versus scikit-learn, TensorFlow, or foo package)
- High-Level MLlib Concepts
  - Low-level data types
- MLlib in Action
- Deployment Patterns
- Conclusion
25. Preprocessing and Feature Engineering
- Formatting Models According to Your Use Case
- Transformers
- Estimators for Preprocessing
  - Transformer Properties
- High-Level Transformers
- Working with Continuous Features
- Working with Categorical Features
- Text Data Transformers
- Feature Manipulation
- Feature Selection
  - ChiSqSelector
- Advanced Topics
  - Persisting Transformers
- Writing a Custom Transformer
- Conclusion
26. Classification
- Use Cases
- Types of Classification
- Classification Models in MLlib
  - Model Scalability
- Logistic Regression
- Decision Trees
- Random Forest and Gradient-Boosted Trees
- Naive Bayes
- Evaluators for Classification and Automating Model Tuning
- Detailed Evaluation Metrics
- One-vs-Rest Classifier
- Multilayer Perceptron
- Conclusion
27. Regression
- Use Cases
- Regression Models in MLlib
  - Model Scalability
- Linear Regression
- Generalized Linear Regression
- Decision Trees
- Random Forests and Gradient-Boosted Trees
- Advanced Methods
  - Survival Regression (Accelerated Failure Time)
  - Isotonic Regression
- Evaluators and Automating Model Tuning
- Metrics
- Conclusion
28. Recommendation
- Use Cases
- Collaborative Filtering with Alternating Least Squares
- Evaluators for Recommendation
- Metrics
  - Regression Metrics
  - Ranking Metrics
- Frequent Pattern Mining
- Conclusion
29. Unsupervised Learning
- Use Cases
- Model Scalability
- k-means
- Bisecting k-means
- Gaussian Mixture Models
- Latent Dirichlet Allocation
- Conclusion
30. Graph Analytics
- Building a Graph
- Querying the Graph
  - Subgraphs
- Motif Finding
- Graph Algorithms
- Conclusion
31. Deep Learning
- What Is Deep Learning?
- Ways of Using Deep Learning in Spark
- Deep Learning Libraries
- A Simple Example with Deep Learning Pipelines
- Conclusion
VII. Ecosystem
32. Language Specifics: Python (PySpark) and R (SparkR and sparklyr)
- PySpark
  - Fundamental PySpark Differences
  - Pandas Integration
- R on Spark
  - SparkR
  - sparklyr
- Conclusion
33. Ecosystem and Community
- Spark Packages
- Community
  - Spark Summit
  - Local Meetups
- Conclusion
Index

Navigation menu