java/

test/ resources scala/ java/ We put the source code in the Scala and Java directories. In this case, we put something like the following in a file; this initializes the SparkSession, runs the application, and then exits: object DataFrameExample extends Serializable { def main(args: Array[String]) = { val pathToDataFolder = args(0) // start up the SparkSession // along with explicitly setting a given config val spark = SparkSession.builder().appName("Spark Example") .config("spark.sql.warehouse.dir", "/user/hive/warehouse") .getOrCreate() // udf registration spark.udf.register("myUDF", someUDF(_:String):String) val df = spark.read.json(pathToDataFolder + "data.json") val manipulated = df.groupBy(expr("myUDF(group)")).sum().collect() .foreach(x => println(x)) } } Notice how we defined a main class that we can run from the command line when we use spark-submit to submit it to our cluster for execution. Now that we have our project set up and have added some code to it, it’s time to build it. We can use sbt assemble to build an “uber-jar” or “fat-jar” that contains all of the dependencies in one JAR. This can be simple for some deployments but cause complications (especially dependency conflicts) for others. A lighter-weight approach is to run sbt package, which will gather all of your dependencies into the target folder but will not package all of them into one big JAR. Running the application The target folder contains the JAR that we can use as an argument to spark-submit. After building the Scala package, you end up with something that you can spark-submit on your local machine by using the following code (this snippet takes advantage of aliasing to create the $SPARK_HOME variable; you could replace $SPARK_HOME with the exact directory that contains your downloaded version of Spark): $SPARK_HOME/bin/spark-submit \ --class com.databricks.example.DataFrameExample \ --master local \ target/scala-2.11/example_2.11-0.1-SNAPSHOT.jar "hello" Writing Python Applications Writing PySpark Applications is really no different than writing normal Python applications or packages. It’s quite similar to writing command-line applications in particular. Spark doesn’t have a build concept, just Python scripts, so to run an application, you simply execute the script against the cluster. To facilitate code reuse, it is common to package multiple Python files into egg or ZIP files of Spark code. To include those files, you can use the --py-files argument of spark-submit to add .py, .zip, or .egg files to be distributed with your application. When it’s time to run your code, you create the equivalent of a “Scala/Java main class” in Python. Specify a certain script as an executable script that builds the SparkSession. This is the one that we will pass as the main argument to spark-submit: # in Python from __future__ import print_function if __name__ == '__main__': from pyspark.sql import SparkSession spark = SparkSession.builder \ .master("local") \ .appName("Word Count") \ .config("spark.some.config.option", "some-value") \ .getOrCreate() print(spark.range(5000).where("id > 500").selectExpr("sum(id)").collect()) When you do this, you’re going to get a SparkSession that you can pass around your application. It is best practice to pass around this variable at runtime rather than instantiating it within every Python class. One helpful tip when developing in Python is to use pip to specify PySpark as a dependency. You can do this by running the command pip install pyspark. This allows you to use it in a way that you might use other Python packages. This makes for very helpful code completion in many editors, as well. This is brand new in Spark 2.2, so it might take a version or two to be completely production ready, but Python is very popular in the Spark community, and it’s sure to be a cornerstone of Spark’s future. Running the application After you’ve written your code, it’s time to submit it for execution. (We’re executing the same code that we have in the project template.) You just need to call spark-submit with that information: $SPARK_HOME/bin/spark-submit --master local pyspark_template/main.py Writing Java Applications Writing Java Spark Applications is, if you squint, the same as writing Scala applications. The core differences involve how you specify your dependencies. This example assumes that you are using Maven to specify your dependencies. In this case, you’ll use the following format. In Maven, you must add the Spark Packages repository so that you can fetch dependencies from those locations: org.apache.spark spark-core_2.11 2.1.0 org.apache.spark spark-sql_2.11 2.1.0 graphframes graphframes 0.4.0-spark2.1-s_2.11 SparkPackagesRepo http://dl.bintray.com/spark-packages/maven Naturally, you follow the same directory structure as in the Scala project version (seeing as they both conform to the Maven specification). We then just follow the relevant Java examples to actually build and execute the code. Now we can create a simple example that specifies a main class for us to execute against (more on this at the end of the chapter): import org.apache.spark.sql.SparkSession; public class SimpleExample { public static void main(String[] args) { SparkSession spark = SparkSession .builder() .getOrCreate(); spark.range(1, 2000).count(); } } We then package it by using mvn package (you need to have Maven installed to do so). Running the application This operation is going to be the exact same as running the Scala application (or the Python application, for that matter). Simply use spark-submit: $SPARK_HOME/bin/spark-submit \ --class com.databricks.example.SimpleExample \ --master local \ target/spark-example-0.1-SNAPSHOT.jar "hello" Testing Spark Applications You now know what it takes to write and run a Spark Application, so let’s move on to a less exciting but still very important topic: testing. Testing Spark Applications relies on a couple of key principles and tactics that you should keep in mind as you’re writing your applications. Strategic Principles Testing your data pipelines and Spark Applications is just as important as actually writing them. This is because you want to ensure that they are resilient to future change, in data, logic, and output. In this section, we’ll first discuss what you might want to test in a typical Spark Application, then discuss how to organize your code for easy testing. Input data resilience Being resilient to different kinds of input data is something that is quite fundamental to how you write your data pipelines. The data will change because the business needs will change. Therefore your Spark Applications and pipelines should be resilient to at least some degree of change in the input data or otherwise ensure that these failures are handled in a graceful and resilient way. For the most part this means being smart about writing your tests to handle those edge cases of different inputs and making sure that the pager only goes off when it’s something that is truly important. Business logic resilience and evolution The business logic in your pipelines will likely change as well as the input data. Even more importantly, you want to be sure that what you’re deducing from the raw data is what you actually think that you’re deducing. This means that you’ll need to do robust logical testing with realistic data to ensure that you’re actually getting what you want out of it. One thing to be wary of here is trying to write a bunch of “Spark Unit Tests” that just test Spark’s functionality. You don’t want to be doing that; instead, you want to be testing your business logic and ensuring that the complex business pipeline that you set up is actually doing what you think it should be doing. Resilience in output and atomicity Assuming that you’re prepared for departures in the structure of input data and that your business logic is well tested, you now want to ensure that your output structure is what you expect. This means you will need to gracefully handle output schema resolution. It’s not often that data is simply dumped in some location, never to be read again—most of your Spark pipelines are probably feeding other Spark pipelines. For this reason you’re going to want to make certain that your downstream consumers understand the “state” of the data—this could mean how frequently it’s updated as well as whether the data is “complete” (e.g., there is no late data) or that there won’t be any last-minute corrections to the data. All of the aforementioned issues are principles that you should be thinking about as you build your data pipelines (actually, regardless of whether you’re using Spark). This strategic thinking is important for laying down the foundation for the system that you would like to build. Tactical Takeaways Although strategic thinking is important, let’s talk a bit more in detail about some of the tactics that you can actually use to make your application easy to test. The highest value approach is to verify that your business logic is correct by employing proper unit testing and to ensure that you’re resilient to changing input data or have structured it so that schema evolution will not become unwielding in the future. The decision for how to do this largely falls on you as the developer because it will vary according to your business domain and domain expertise. Managing SparkSessions Testing your Spark code using a unit test framework like JUnit or ScalaTest is relatively easy because of Spark’s local mode—just create a local mode SparkSession as part of your test harness to run it. However, to make this work well, you should try to perform dependency injection as much as possible when managing SparkSessions in your code. That is, initialize the SparkSession only once and pass it around to relevant functions and classes at runtime in a way that makes it easy to substitute during testing. This makes it much easier to test each individual function with a dummy SparkSession in unit tests. Which Spark API to Use? Spark offers several choices of APIs, ranging from SQL to DataFrames and Datasets, and each of these can have different impacts for maintainability and testability of your application. To be perfectly honest, the right API depends on your team and its needs: some teams and projects will need the less strict SQL and DataFrame APIs for speed of development, while others will want to use type-safe Datasets or RDDs. In general, we recommend documenting and testing the input and output types of each function regardless of which API you use. The type-safe API automatically enforces a minimal contract for your function that makes it easy for other code to build on it. If your team prefers to use DataFrames or SQL, then spend some time to document and test what each function returns and what types of inputs it accepts to avoid surprises later, as in any dynamically typed programming language. While the lower-level RDD API is also statically typed, we recommend going into it only if you need low-level features such as partitioning that are not present in Datasets, which should not be very common; the Dataset API allows more performance optimizations and is likely to provide even more of them in the future. A similar set of considerations applies to which programming language to use for your application: there certainly is no right answer for every team, but depending on your needs, each language will provide different benefits. We generally recommend using statically typed languages like Scala and Java for larger applications or those where you want to be able to drop into low-level code to fully control performance, but Python and R may be significantly better in other cases—for example, if you need to use some of their other libraries. Spark code should easily be testable in the standard unit testing frameworks in every language. Connecting to Unit Testing Frameworks To unit test your code, we recommend using the standard frameworks in your langage (e.g., JUnit or ScalaTest), and setting up your test harnesses to create and clean up a SparkSession for each test. Different frameworks offer different mechanisms to do this, such as “before” and “after” methods. We have included some sample unit testing code in the application templates for this chapter. Connecting to Data Sources As much as possible, you should make sure your testing code does not connect to production data sources, so that developers can easily run it in isolation if these data sources change. One easy way to make this happen is to have all your business logic functions take DataFrames or Datasets as input instead of directly connecting to various sources; after all, subsequent code will work the same way no matter what the data source was. If you are using the structured APIs in Spark, another way to make this happen is named tables: you can simply register some dummy datasets (e.g., loaded from small text file or from in-memory objects) as various table names and go from there. The Development Process The development process with Spark Applications is similar to development workflows that you have probably already used. First, you might maintain a scratch space, such as an interactive notebook or some equivalent thereof, and then as you build key components and algorithms, you move them to a more permanent location like a library or package. The notebook experience is one that we often recommend (and are using to write this book) because of its simplicity in experimentation. There are also some tools, such as Databricks, that allow you to run notebooks as production applications as well. When running on your local machine, the spark-shell and its various language-specific implementations are probably the best way to develop applications. For the most part, the shell is for interactive applications, whereas spark-submit is for production applications on your Spark cluster. You can use the shell to interactively run Spark, just as we showed you at the beginning of this book. This is the mode with which you will run PySpark, Spark SQL, and SparkR. In the bin folder, when you download Spark, you will find the various ways of starting these shells. Simply run spark-shell(for Scala), spark-sql, pyspark, and sparkR. After you’ve finished your application and created a package or script to run, spark-submit will become your best friend to submit this job to a cluster. Launching Applications The most common way for running Spark Applications is through spark-submit. Previously in this chapter, we showed you how to run spark-submit; you simply specify your options, the application JAR or script, and the relevant arguments: ./bin/spark-submit \ --class \ --master \ --deploy-mode \ --conf = \ ... # other options \ [application-arguments] You can always specify whether to run in client or cluster mode when you submit a Spark job with spark-submit. However, you should almost always favor running in cluster mode (or in client mode on the cluster itself) to reduce latency between the executors and the driver. When submitting applciations, pass a .py file in the place of a .jar, and add Python .zip, .egg, or .py to the search path with --py-files. For reference, Table 16-1 lists all of the available spark-submit options, including those that are particular to some cluster managers. To enumerate all these options yourself, run sparksubmit with --help. Table 16-1. Spark submit help text Parameter Description --master MASTER_URL spark://host:port, mesos://host:port, yarn, or local --deploymode DEPLOY_MODE Whether to launch the driver program locally (“client”) or on one of the worker machines inside the cluster (“cluster”) (Default: client) --class CLASS_NAME Your application’s main class (for Java / Scala apps). --name NAME A name of your application. --jars JARS Comma-separated list of local JARs to include on the driver and executor classpaths. --packages Comma-separated list of Maven coordinates of JARs to include on the driver and executor classpaths. Will search the local Maven repo, then Maven Central and any additional remote repositories given by --repositories. The format for the coordinates should be groupId:artifactId:version. --excludepackages Comma-separated list of groupId:artifactId, to exclude while resolving the dependencies provided in --packages to avoid dependency conflicts. Comma-separated list of additional remote repositories to search for the Maven coordinates -repositories given with --packages. --py-files PY_FILES Comma-separated list of .zip, .egg, or .py files to place on the PYTHONPATH for Python apps. --files FILES Comma-separated list of files to be placed in the working directory of each executor. --conf PROP=VALUE Arbitrary Spark configuration property. -propertiesfile FILE Path to a file from which to load extra properties. If not specified, this will look for conf/spark-defaults.conf. --drivermemory MEM Memory for driver (e.g., 1000M, 2G) (Default: 1024M). --driverExtra Java options to pass to the driver. java-options --driverExtra library path entries to pass to the driver. library-path --driverclass-path Extra class path entries to pass to the driver. Note that JARs added with --jars are automatically included in the classpath. --executormemory MEM Memory per executor (e.g., 1000M, 2G) (Default: 1G). --proxy-user User to impersonate when submitting the application. This argument does not work with -NAME principal / --keytab. --help, -h Show this help message and exit. --verbose, Print additional debug output. v --version Print the version of current Spark. There are some deployment-specific configurations as well (see Table 16-2). Table 16-2. Deployment Specific Configurations Table 16-2. Deployment Specific Configurations Cluster Managers Modes Conf Standalone Cluster --drivercores NUM Standalone/Mesos Cluster --supervise Description Cores for driver (Default: 1). If given, restarts the driver on failure. Standalone/Mesos Cluster --kill If given, kills the driver specified. SUBMISSION_ID Standalone/Mesos Cluster --status If given, requests the status of the driver specified. SUBMISSION_ID Standalone/Mesos Either --totalexecutorcores NUM Total cores for all executors. Standalone/YARN Either --executorcores NUM1 Number of cores per executor. (Default: 1 in YARN mode or all available cores on the worker in standalone mode) YARN Either --drivercores NUM Number of cores used by the driver, only in cluster mode (Default: 1). YARN Either queue QUEUE_NAME The YARN queue to submit to (Default: “default”). YARN Either --numallocation is enabled, the initial number of executors will be at executors NUM least NUM. YARN Either --archives ARCHIVES Comma-separated list of archives to be extracted into the working directory of each executor. YARN Either --principal PRINCIPAL Principal to be used to log in to KDC, while running on secure HDFS. --keytab KEYTAB The full path to the file that contains the keytab for the principal specified above. This keytab will be copied to the node running the Application Master via the Secure Distributed Cache, for renewing the login tickets and the delegation tokens periodically. Number of executors to launch (Default: 2). If dynamic YARN Either Application Launch Examples We already covered some local-mode application examples previously in this chapter, but it’s worth looking at how we use some of the aforementioned options, as well. Spark also includes several examples and demonstration applications in the examples directory that is included when you download Spark. If you’re stuck on how to use certain parameters, simply try them first on your local machine and use the SparkPi class as the main class: ./bin/spark-submit \ --class org.apache.spark.examples.SparkPi \ --master spark://207.184.161.138:7077 \ --executor-memory 20G \ --total-executor-cores 100 \ replace/with/path/to/examples.jar \ 1000 The following snippet does the same for Python. You run it from the Spark directory and this will allow you to submit a Python application (all in one script) to the standalone cluster manager. You can also set the same executor limits as in the preceding example: ./bin/spark-submit \ --master spark://207.184.161.138:7077 \ examples/src/main/python/pi.py \ 1000 You can change this to run in local mode as well by setting the master to local or local[*] to run on all the cores on your machine. You will also need to change the /path/to/examples.jar to the relevant Scala and Spark versions you are running. Configuring Applications Spark includes a number of different configurations, some of which we covered in Chapter 15. There are many different configurations, depending on what you’re hoping to achieve. This section covers those very details. For the most part, this information is included for reference and is probably worth skimming only, unless you’re looking for something in particular. The majority of configurations fall into the following categories: Application properties Runtime environment Shuffle behavior Spark UI Compression and serialization Memory management Execution behavior Networking Scheduling Dynamic allocation Security Encryption Spark SQL Spark streaming SparkR Spark provides three locations to configure the system: Spark properties control most application parameters and can be set by using a SparkConf object Java system properties Hardcoded configuration files There are several templates that you can use, which you can find in the /conf directory available in the root of the Spark home folder. You can set these properties as hardcoded variables in your applications or by specifying them at runtime. You can use environment variables to set permachine settings, such as the IP address, through the conf/spark-env.sh script on each node. Lastly, you can configure logging through log4j.properties. The SparkConf The SparkConf manages all of our application configurations. You create one via the import statement, as shown in the example that follows. After you create it, the SparkConf is immutable for that specific Spark Application: // in Scala import org.apache.spark.SparkConf val conf = new SparkConf().setMaster("local[2]").setAppName("DefinitiveGuide") .set("some.conf", "to.some.value") # in Python from pyspark import SparkConf conf = SparkConf().setMaster("local[2]").setAppName("DefinitiveGuide")\ .set("some.conf", "to.some.value") You use the SparkConf to configure individual Spark Applications with Spark properties. These Spark properties control how the Spark Application runs and how the cluster is configured. The example that follows configures the local cluster to have two threads and specifies the application name that shows up in the Spark UI. You can configure these at runtime, as you saw previously in this chapter through command-line arguments. This is helpful when starting a Spark Shell that will automatically include a basic Spark Application for you; for instance: ./bin/spark-submit --name "DefinitiveGuide" --master local[4] ... Of note is that when setting time duration-based properties, you should use the following format: 25ms (milliseconds) 5s (seconds) 10m or 10min (minutes) 3h (hours) 5d (days) 1y (years) Application Properties Application properties are those that you set either from spark-submit or when you create your Spark Application. They define basic application metadata as well as some execution characteristics. Table 16-3 presents a list of current application properties. Table 16-3. Application properties Property name Default Meaning spark.app.name (none) The name of your application. This will appear in the UI and in log data. spark.driver.cores 1 Number of cores to use for the driver process, only in cluster mode. spark.driver.maxResultSize 1g Limit of total size of serialized results of all partitions for each Spark action (e.g., collect). Should be at least 1M, or 0 for unlimited. Jobs will be aborted if the total size exceeds this limit. Having a high limit can cause OutOfMemoryErrors in the driver (depends on spark.driver.memory and memory overhead of objects in JVM). Setting a proper limit can protect the driver from OutOfMemoryErrors. spark.driver.memory 1g Amount of memory to use for the driver process, where SparkContext is initialized. (e.g. 1g, 2g). Note: in client mode, this must not be set through the SparkConf directly in your application, because the driver JVM has already started at that point. Instead, set this through the --driver-memory command-line option or in your default properties file. spark.executor.memory 1g Amount of memory to use per executor process (e.g., 2g, 8g). spark.extraListeners (none) A comma-separated list of classes that implement SparkListener; when initializing SparkContext, instances of these classes will be created and registered with Spark’s listener bus. If a class has a single-argument constructor that accepts a SparkConf, that constructor will be called; otherwise, a zero-argument constructor will be called. If no valid constructor can be found, the SparkContext creation will fail with an exception. spark.logConf FALSE Logs the effective SparkConf as INFO when a SparkContext is started. spark.master (none) The cluster manager to connect to. See the list of allowed master URLs. spark.submit.deployMode (none) The deploy mode of the Spark driver program, either “client” or “cluster,” which means to launch driver program locally (“client”) or remotely (“cluster”) on one of the nodes inside the cluster. spark.log.callerContext (none) Application information that will be written into Yarn RM log/HDFS audit log when running on Yarn/HDFS. Its length depends on the Hadoop configuration hadoop.caller.context.max.size. It should be concise, and typically can have up to 50 characters. spark.driver.supervise FALSE If true, restarts the driver automatically if it fails with a non-zero exit status. Only has effect in Spark standalone mode or Mesos cluster deploy mode. You can ensure that you’ve correctly set these values by checking the application’s web UI on port 4040 of the driver on the “Environment” tab. Only values explicitly specified through sparkdefaults.conf, SparkConf, or the command line will appear. For all other configuration properties, you can assume the default value is used. Runtime Properties Although less common, there are times when you might also need to configure the runtime environment of your application. Due to space limitations, we cannot include the entire configuration set here. Refer to the relevant table on the Runtime Environment in the Spark documentation. These properties allow you to configure extra classpaths and python paths for both drivers and executors, Python worker configurations, as well as miscellaneous logging properties. Execution Properties These configurations are some of the most relevant for you to configure because they give you finer-grained control on actual execution. Due to space limitations, we cannot include the entire configuration set here. Refer to the relevant table on Execution Behavior in the Spark documentation. The most common configurations to change are spark.executor.cores (to control the number of available cores) and spark.files.maxPartitionBytes (maximum partition size when reading files). Configuring Memory Management There are times when you might need to manually manage the memory options to try and optimize your applications. Many of these are not particularly relevant for end users because they involve a lot of legacy concepts or fine-grained controls that were obviated in Spark 2.X because of automatic memory management. Due to space limitations, we cannot include the entire configuration set here. Refer to the relevant table on Memory Management in the Spark documentation. Configuring Shuffle Behavior We’ve emphasized how shuffles can be a bottleneck in Spark jobs because of their high communication overhead. Therefore there are a number of low-level configurations for controlling shuffle behavior. Due to space limitations, we cannot include the entire configuration set here. Refer to the relevant table on Shuffle Behavior in the Spark documentation. Environmental Variables You can configure certain Spark settings through environment variables, which are read from the conf/spark-env.sh script in the directory where Spark is installed (or conf/spark-env.cmd on Windows). In Standalone and Mesos modes, this file can give machine-specific information such as hostnames. It is also sourced when running local Spark Applications or submission scripts. Note that conf/spark-env.sh does not exist by default when Spark is installed. However, you can copy conf/spark-env.sh.template to create it. Be sure to make the copy executable. The following variables can be set in spark-env.sh: JAVA_HOME Location where Java is installed (if it’s not on your default PATH). PYSPARK_PYTHON Python binary executable to use for PySpark in both driver and workers (default is python2.7 if available; otherwise, python). Property spark.pyspark.python takes precedence if it is set. PYSPARK_DRIVER_PYTHON Python binary executable to use for PySpark in driver only (default is PYSPARK_PYTHON). Property spark.pyspark.driver.python takes precedence if it is set. SPARKR_DRIVER_R R binary executable to use for SparkR shell (default is R). Property spark.r.shell.command takes precedence if it is set. SPARK_LOCAL_IP IP address of the machine to which to bind. SPARK_PUBLIC_DNS Hostname your Spark program will advertise to other machines. In addition to the variables ust listed, there are also options for setting up the Spark standalone cluster scripts, such as number of cores to use on each machine and maximum memory. Because spark-env.sh is a shell script, you can set some of these programmatically; for example, you might compute SPARK_LOCAL_IP by looking up the IP of a specific network interface. NOTE When running Spark on YARN in cluster mode, you need to set environment variables by using the spark.yarn.appMasterEnv.[EnvironmentVariableName] property in your conf/spark-defaults.conf file. Environment variables that are set in spark-env.sh will not be reflected in the YARN Application Master process in cluster mode. See the YARN-related Spark Properties for more information. Job Scheduling Within an Application Within a given Spark Application, multiple parallel jobs can run simultaneously if they were submitted from separate threads. By job, in this section, we mean a Spark action and any tasks that need to run to evaluate that action. Spark’s scheduler is fully thread-safe and supports this use case to enable applications that serve multiple requests (e.g., queries for multiple users). By default, Spark’s scheduler runs jobs in FIFO fashion. If the jobs at the head of the queue don’t need to use the entire cluster, later jobs can begin to run right away, but if the jobs at the head of the queue are large, later jobs might be delayed significantly. It is also possible to configure fair sharing between jobs. Under fair sharing, Spark assigns tasks between jobs in a round-robin fashion so that all jobs get a roughly equal share of cluster resources. This means that short jobs submitted while a long job is running can begin receiving resources right away and still achieve good response times without waiting for the long job to finish. This mode is best for multiuser settings. To enable the fair scheduler, set the spark.scheduler.mode property to FAIR when configuring a SparkContext. The fair scheduler also supports grouping jobs into pools, and setting different scheduling options, or weights, for each pool. This can be useful to create a high-priority pool for more important jobs or to group the jobs of each user together and give users equal shares regardless of how many concurrent jobs they have instead of giving jobs equal shares. This approach is modeled after the Hadoop Fair Scheduler. Without any intervention, newly submitted jobs go into a default pool, but jobs pools can be set by adding the spark.scheduler.pool local property to the SparkContext in the thread that’s submitting them. This is done as follows (assuming sc is your SparkContext: sc.setLocalProperty("spark.scheduler.pool", "pool1") After setting this local property, all jobs submitted within this thread will use this pool name. The setting is per-thread to make it easy to have a thread run multiple jobs on behalf of the same user. If you’d like to clear the pool that a thread is associated with, set it to null. Conclusion This chapter covered a lot about Spark Applications; we learned how to write, test, run, and configure them in all of Spark’s languages. In Chapter 17, we talk about deploying and the cluster management options you have when it comes to running Spark Applications. Chapter 17. Deploying Spark This chapter explores the infrastructure you need in place for you and your team to be able to run Spark Applications: Cluster deployment choices Spark’s different cluster managers Deployment considerations and configuring deployments For the most, part Spark should work similarly with all the supported cluster managers; however, customizing the setup means understanding the intricacies of each of the cluster management systems. The hard part is deciding on the cluster manager (or choosing a managed service). Although we would be happy to include all the minute details about how you can configure different cluster with different cluster managers, it’s simply impossible for this book to provide hyper-specific details for every situation in every single enviroment. The goal of this chapter, therefore, is not to discuss each of the cluster managers in full detail, but rather to look at their fundamental differences and to provide a reference for a lot of the material already available on the Spark website. Unfortunately, there is no easy answer to “which is the easiest cluster manager to run” because it varies so much by use case, experience, and resources. The Spark documentation site offers a lot of detail about deploying Spark with actionable examples. We do our best to discuss the most relevant points. As of this writing, Spark has three officially supported cluster managers: Standalone mode Hadoop YARN Apache Mesos These cluster managers maintain a set of machines onto which you can deploy Spark Applications. Naturally, each of these cluster managers has an opinionated view toward management, and so there are trade-offs and semantics that you will need to keep in mind. However, they all run Spark applications the same way (as covered in Chapter 16). Let’s begin with the first point: where to deploy your cluster. Where to Deploy Your Cluster to Run Spark Applications There are two high-level options for where to deploy Spark clusters: deploy in an on-premises cluster or in the public cloud. This choice is consequential and is therefore worth discussing. On-Premises Cluster Deployments Deploying Spark to an on-premises cluster is sometimes a reasonable option, especially for organizations that already manage their own datacenters. As with everything else, there are tradeoffs to this approach. An on-premises cluster gives you full control over the hardware used, meaning you can optimize performance for your specific workload. However, it also introduces some challenges, especially when it comes to data analytics workloads like Spark. First, with onpremises deployment, your cluster is fixed in size, whereas the resource demands of data analytics workloads are often elastic. If you make your cluster too small, it will be hard to launch the occasional very large analytics query or training job for a new machine learning model, whereas if you make it large, you will have resources sitting idle. Second, for on-premises clusters, you need to select and operate your own storage system, such as a Hadoop file system or scalable key-value store. This includes setting up georeplication and disaster recovery if required. If you are going to deploy on-premises, the best way to combat the resource utilization problem is to use a cluster manager that allows you to run many Spark applications and dynamically reassign resources between them, or even allows non-Spark applications on the same cluster. All of Spark’s supported cluster managers allow multiple concurrent applications, but YARN and Mesos have better support for dynamic sharing and also additionally support non-Spark workloads. Handling resource sharing is likely going to be the biggest difference your users see day to day with Spark on-premise versus in the cloud: in public clouds, it’s easy to give each application its own cluster of exactly the required size for just the duration of that job. For storage, you have several different options, but covering all the trade-offs and operational details in depth would probably require its own book. The most common storage systems used for Spark are distributed file systems such as Hadoop’s HDFS and key-value stores such as Apache Cassandra. Streaming message bus systems such as Apache Kafka are also often used for ingesting data. All these systems have varying degrees of support for management, backup, and georeplication, sometimes built into the system and sometimes only through third-party commercial tools. Before choosing a storage option, we recommend evaluating the performance of its Spark connector and evaluating the available management tools. Spark in the Cloud While early big data systems were designed for on-premises deployment, the cloud is now an increasingly common platform for deploying Spark. The public cloud has several advantages when it comes to big data workloads. First, resources can be launched and shut down elastically, so you can run that occasional “monster” job that takes hundreds of machines for a few hours without having to pay for them all the time. Even for normal operation, you can choose a different type of machine and cluster size for each application to optimize its cost performance— for example, launch machines with Graphics Processing Units (GPUs) just for your deep learning jobs. Second, public clouds include low-cost, georeplicated storage that makes it easier to manage large amounts of data. Many companies looking to migrate to the cloud imagine they’ll run their applications in the same way that they run their on-premises clusters. All the major cloud providers (Amazon Web Services [AWS], Microsoft Azure, Google Cloud Platform [GCP], and IBM Bluemix) include managed Hadoop clusters for their customers, which provide HDFS for storage as well as Apache Spark. This is actually not a great way to run Spark in the cloud, however, because by using a fixed-size cluster and file system, you are not going to be able to take advantage of elasticity. Instead, it is generally a better idea to use global storage systems that are decoupled from a specific cluster, such as Amazon S3, Azure Blob Storage, or Google Cloud Storage and spin up machines dynamically for each Spark workload. With decoupled compute and storage, you will be able to pay for computing resources only when needed, scale them up dynamically, and mix different hardware types. Basically, keep in mind that running Spark in the cloud need not mean migrating an on-premises installation to virtual machines: you can run Spark natively against cloud storage to take full advantage of the cloud’s elasticity, cost-saving benefit, and management tools without having to manage an on-premise computing stack within your cloud environment. Several companies provide “cloud-native” Spark-based services, and all installations of Apache Spark can of course connect to cloud storage. Databricks, the company started by the Spark team from UC Berkeley, is one example of a service provider built specifically for Spark in the cloud. Databricks provides a simple way to run Spark workloads without the heavy baggage of a Hadoop installation. The company provides a number of features for running Spark more efficiently in the cloud, such as auto-scaling, auto-termination of clusters, and optimized connectors to cloud storage, as well as a collaborative environment for working on notebooks and standalone jobs. The company also provides a free Community Edition for learning Spark where you can run notebooks on a small cluster and share them live with others. A fun fact is that this entire book was written using the free Community Edition of Databricks, because we found the integrated Spark notebooks, live collaboration, and cluster management the easiest way to produce and test this content. If you run Spark in the cloud, much of the content in this chapter might not be relevant because you can often create a separate, short-lived Spark cluster for each job you execute. In that case, the standalone cluster manager is likely the easiest to use. However, you may still want to read this content if you’d like to share a longer-lived cluster among many applications, or to install Spark on virtual machines yourself. Cluster Managers Unless you are using a high-level managed service, you will have to decide on the cluster manager to use for Spark. Spark supports three aforementioned cluster managers: standalone clusters, Hadoop YARN, and Mesos. Let’s review each of these. Standalone Mode Spark’s standalone cluster manager is a lightweight platform built specifically for Apache Spark workloads. Using it, you can run multiple Spark Applications on the same cluster. It also provides simple interfaces for doing so but can scale to large Spark workloads. The main disadvantage of the standalone mode is that it’s more limited than the other cluster managers—in particular, your cluster can only run Spark. It’s probably the best starting point if you just want to quickly get Spark running on a cluster, however, and you do not have experience using YARN or Mesos. Starting a standalone cluster Starting a standalone cluster requires provisioning the machines for doing so. That means starting them up, ensuring that they can talk to one another over the network, and getting the version of Spark you would like to run on those sets of machines. After that, there are two ways to start the cluster: by hand or using built-in launch scripts. Let’s first launch a cluster by hand. The first step is to start the master process on the machine that we want that to run on, using the following command: $SPARK_HOME/sbin/start-master.sh When we run this command, the cluster manager master process will start up on that machine. Once started, the master prints out a spark://HOST:PORT URI. You use this when you start each of the worker nodes of the cluster, and you can use it as the master argument to your SparkSession on application initialization. You can also find this URI on the master’s web UI, which is http://master-ip-address:8080 by default. With that URI, start the worker nodes by logging in to each machine and running the following script using the URI you just received from the master node. The master machine must be available on the network of the worker nodes you are using, and the port must be open on the master node, as well: $SPARK_HOME/sbin/start-slave.sh As soon as you’ve run that on another machine, you have a Spark cluster running! This process is naturally a bit manual; thankfully there are scripts that can help to automate this process. Cluster launch scripts You can configure cluster launch scripts that can automate the launch of standalone clusters. To do this, create a file called conf/slaves in your Spark directory that will contain the hostnames of all the machines on which you intend to start Spark workers, one per line. If this file does not exist, everything will launch locally. When you go to actually start the cluster, the master machine will access each of the worker machines via Secure Shell (SSH). By default, SSH is run in parallel and requires that you configure password-less (using a private key) access. If you do not have a password-less setup, you can set the environment variable SPARK_SSH_FOREGROUND and serially provide a password for each worker. After you set up this file, you can launch or stop your cluster by using the following shell scripts, based on Hadoop’s deploy scripts, and available in $SPARK_HOME/sbin: $SPARK_HOME/sbin/start-master.sh Starts a master instance on the machine on which the script is executed. $SPARK_HOME/sbin/start-slaves.sh Starts a slave instance on each machine specified in the conf/slaves file. $SPARK_HOME/sbin/start-slave.sh Starts a slave instance on the machine on which the script is executed. $SPARK_HOME/sbin/start-all.sh Starts both a master and a number of slaves as described earlier. $SPARK_HOME/sbin/stop-master.sh Stops the master that was started via the bin/start-master.sh script. $SPARK_HOME/sbin/stop-slaves.sh Stops all slave instances on the machines specified in the conf/slaves file. $SPARK_HOME/sbin/stop-all.sh Stops both the master and the slaves as described earlier. Standalone cluster configurations Standalone clusters have a number of configurations that you can use to tune your application. These control everything from what happens to old files on each worker for terminated applications to the worker’s core and memory resources. These are controlled via environment variables or via application properties. Due to space limitations, we cannot include the entire configuration set here. Refer to the relevant table on Standalone Environment Variables in the Spark documentation. Submitting applications After you create the cluster, you can submit applications to it using the spark:// URI of the master. You can do this either on the master node itself or another machine using spark-submit. There are some specific command-line arguments for standalone mode, which we covered in “Launching Applications”. Spark on YARN Hadoop YARN is a framework for job scheduling and cluster resource management. Even though Spark is often (mis)classified as a part of the “Hadoop Ecosystem,” in reality, Spark has little to do with Hadoop. Spark does natively support the Hadoop YARN cluster manager but it requires nothing from Hadoop itself. You can run your Spark jobs on Hadoop YARN by specifying the master as YARN in the spark-submit command-line arguments. Just like with standalone mode, there are a number of knobs that you are able to tune according to what you would like the cluster to do. The number of knobs is naturally larger than that of Spark’s standalone mode because Hadoop YARN is a generic scheduler for a large number of different execution frameworks. Setting up a YARN cluster is beyond the scope of this book, but there are some great books on the topic as well as managed services that can simplify this experience. Submitting applications When submitting applications to YARN, the core difference from other deployments is that -master will become yarn as opposed the master node IP, as it is in standalone mode. Instead, Spark will find the YARN configuration files using the environment variable HADOOP_CONF_DIR or YARN_CONF_DIR. Once you have set those environment variables to your Hadoop installation’s configuration directory, you can just run spark-submit like we saw in Chapter 16. NOTE There are two deployment modes that you can use to launch Spark on YARN. As discussed in previous chapters, cluster mode has the spark driver as a process managed by the YARN cluster, and the client can exit after creating the application. In client mode, the driver will run in the client process and therefore YARN will be responsible only for granting executor resources to the application, not maintaining the master node. Also of note is that in cluster mode, Spark doesn’t necessarily run on the same machine on which you’re executing. Therefore libraries and external jars must be distributed manually or through the --jars command-line argument. There are a few YARN-specific properties that you can set by using spark-submit. These allow you to control priority queues and things like keytabs for security. We covered these in “Launching Applications” in Chapter 16. Configuring Spark on YARN Applications Deploying Spark as YARN applications requires you to understand the variety of different configurations and their implications for your Spark applications. This section covers some best practices for basic configurations and includes references to some of the important configuration for running your Spark applications. Hadoop configurations If you plan to read and write from HDFS using Spark, you need to include two Hadoop configuration files on Spark’s classpath: hdfs-site.xml, which provides default behaviors for the HDFS client; and core-site.xml, which sets the default file system name. The location of these configuration files varies across Hadoop versions, but a common location is inside of /etc/hadoop/conf. Some tools create these configurations on the fly, as well, so it’s important to understand how your managed service might be deploying these, as well. To make these files visible to Spark, set HADOOP_CONF_DIR in $SPARK_HOME/spark-env.sh to a location containing the configuration files or as an environment variable when you go to sparksubmit your application. Application properties for YARN There are a number of Hadoop-related configurations and things that come up that largely don’t have much to do with Spark, just running or securing YARN in a way that influences how Spark runs. Due to space limitations, we cannot include the configuration set here. Refer to the relevant table on YARN Configurations in the Spark documentation. Spark on Mesos Apache Mesos is another clustering system that Spark can run on. A fun fact about Mesos is that the project was also started by many of the original authors of Spark, including one of the authors of this book. In the Mesos project’s own words: Apache Mesos abstracts CPU, memory, storage, and other compute resources away from machines (physical or virtual), enabling fault-tolerant and elastic distributed systems to easily be built and run effectively. For the most part, Mesos intends to be a datacenter scale-cluster manager that manages not just short-lived applications like Spark, but long-running applications like web applications or other resource interfaces. Mesos is the heaviest-weight cluster manager, simply because you might choose this cluster manager only if your organization already has a large-scale deployment of Mesos, but it makes for a good cluster manager nonetheless. Mesos is a large piece of infrastructure, and unfortunately there’s simply too much information for us to cover how to deploy and maintain Mesos clusters. There are many great books on the subject for that, including Dipa Dubhashi and Akhil Das’s Mastering Mesos (O’Reilly, 2016). The goal here is to bring up some of the considerations that you’ll need to think about when running Spark Applications on Mesos. For instance, one common thing you will hear about Spark on Mesos is fine-grained versus coarse-grained mode. Historically Mesos supported a variety of different modes (fine-grained and coarse-grained), but at this point, it supports only coarse-grained scheduling (fine-grained has been deprecated). Coarse-grained mode means that each Spark executor runs as a single Mesos task. Spark executors are sized according to the following application properties: spark.executor.memory spark.executor.cores spark.cores.max/spark.executor.cores Submitting applications Submitting applications to a Mesos cluster is similar to doing so for Spark’s other cluster managers. For the most part you should favor cluster mode when using Mesos. Client mode requires some extra configuration on your part, especially with regard to distributing resources around the cluster. For instance, in client mode, the driver needs extra configuration information in spark-env.sh to work with Mesos. In spark-env.sh set some environment variables: export MESOS_NATIVE_JAVA_LIBRARY= This path is typically /lib/libmesos.so where the prefix is /usr/local by default. On Mac OS X, the library is called libmesos.dylib instead of libmesos.so: export SPARK_EXECUTOR_URI= Finally, set the Spark Application property spark.executor.uri to . Now, when starting a Spark application against the cluster, pass a mesos:// URL as the master when creating a SparkContex, and set that property as a parameter in your SparkConf variable or the initialization of a SparkSession: // in Scala import org.apache.spark.sql.SparkSession val spark = SparkSession.builder .master("mesos://HOST:5050") .appName("my app") .config("spark.executor.uri", "") .getOrCreate() Submitting cluster mode applications is fairly straightforward and follows the same sparksubmit structure you read about before. We covered these in “Launching Applications”. Configuring Mesos Just like any other cluster manager, there are a number of ways that we can configure our Spark Applications when they’re running on Mesos. Due to space limitations, we cannot include the entire configuration set here. Refer to the relevant table on Mesos Configurations in the Spark documentation. Secure Deployment Configurations Spark also provides some low-level ability to make your applications run more securely, especially in untrusted environments. Note that the majority of this setup will happen outside of Spark. These configurations are primarily network-based to help Spark run in a more secure manner. This means authentication, network encryption, and setting TLS and SSL configurations. Due to space limitations, we cannot include the entire configuration set here. Refer to the relevant table on Security Configurations in the Spark documentation. Cluster Networking Configurations Just as shuffles are important, there can be some things worth tuning on the network. This can also be helpful when performing custom deployment configurations for your Spark clusters when you need to use proxies in between certain nodes. If you’re looking to increase Spark’s performance, these should not be the first configurations you go to tune, but may come up in custom deployment scenarios. Due to space limitations, we cannot include the entire configuration set here. Refer to the relevant table on Networking Configurations in the Spark documentation. Application Scheduling Spark has several facilities for scheduling resources between computations. First, recall that, as described earlier in the book, each Spark Application runs an independent set of executor processes. Cluster managers provide the facilities for scheduling across Spark applications. Second, within each Spark application, multiple jobs (i.e., Spark actions) may be running concurrently if they were submitted by different threads. This is common if your application is serving requests over the network. Spark includes a fair scheduler to schedule resources within each application. We introduced this topic in the previous chapter. If multiple users need to share your cluster and run different Spark Applications, there are different options to manage allocation, depending on the cluster manager. The simplest option, available on all cluster managers, is static partitioning of resources. With this approach, each application is given a maximum amount of resources that it can use, and holds onto those resources for the entire duration. In spark-submit there are a number of properties that you can set to control the resource allocation of a particular application. Refer to Chapter 16 for more information. In addition, dynamic allocation (described next) can be turned on to let applications scale up and down dynamically based on their current number of pending tasks. If, instead, you want users to be able to share memory and executor resources in a fine-grained manner, you can launch a single Spark Application and use thread scheduling within it to serve multiple requests in parallel. Dynamic allocation If you would like to run multiple Spark Applications on the same cluster, Spark provides a mechanism to dynamically adjust the resources your application occupies based on the workload. This means that your application can give resources back to the cluster if they are no longer used, and request them again later when there is demand. This feature is particularly useful if multiple applications share resources in your Spark cluster. This feature is disabled by default and available on all coarse-grained cluster managers; that is, standalone mode, YARN mode, and Mesos coarse-grained mode. There are two requirements for using this feature. First, your application must set spark.dynamicAllocation.enabled to true. Second, you must set up an external shuffle service on each worker node in the same cluster and set spark.shuffle.service.enabled to true in your application. The purpose of the external shuffle service is to allow executors to be removed without deleting shuffle files written by them. This is set up differently for each cluster manager and is described in the job scheduling configuration. Due to space limitations, we cannot include the configuration set for dynamic allocation. Refer to the relevant table on Dynamic Allocation Configurations. Miscellaneous Considerations There several other topics to consider when deploying Spark applications that may affect your choice of cluster manager and its setup. These are just things that you should think about when comparing different deployment options. One of the more important considerations is the number and type of applications you intend to be running. For instance, YARN is great for HDFS-based applications but is not commonly used for much else. Additionally, it’s not well designed to support the cloud, because it expects information to be available on HDFS. Also, compute and storage is largely coupled together, meaning that scaling your cluster involves scaling both storage and compute instead of just one or the other. Mesos does improve on this a bit conceptually, and it supports a wide range of application types, but it still requires pre-provisioning machines and, in some sense, requires buy-in at a much larger scale. For instance, it doesn’t really make sense to have a Mesos cluster for only running Spark Applications. Spark standalone mode is the lightest-weight cluster manager and is relatively simple to understand and take advantage of, but then you’re going to be building more application management infrastructure that you could get much more easily by using YARN or Mesos. Another challenge is managing different Spark versions. Your hands are largely tied if you want to try to run a variety of different applications running different Spark versions, and unless you use a well-managed service, you’re going to need to spend a fair amount of time either managing different setup scripts for different Spark services or removing the ability for your users to use a variety of different Spark applications. Regardless of the cluster manager that you choose, you’re going to want to consider how you’re going to set up logging, store logs for future reference, and allow end users to debug their applications. These are more “out of the box” for YARN or Mesos and might need some tweaking if you’re using standalone. One thing you might want to consider—or that might influence your decision making—is maintaining a metastore in order to maintain metadata about your stored datasets, such as a table catalog. We saw how this comes up in Spark SQL when we are creating and maintaining tables. Maintaining an Apache Hive metastore, a topic beyond the scope of this book, might be something that’s worth doing to facilitate more productive, cross-application referencing to the same datasets. Depending on your workload, it might be worth considering using Spark’s external shuffle service. Typically Spark stores shuffle blocks (shuffle output) on a local disk on that particular node. An external shuffle service allows for storing those shuffle blocks so that they are available to all executors, meaning that you can arbitrarily kill executors and still have their shuffle outputs available to other applications. Finally, you’re going to need to configure at least some basic monitoring solution and help users debug their Spark jobs running on their clusters. This is going to vary across cluster management options and we touch on some of the things that you might want to set up in Chapter 18. Conclusion This chapter looked at the world of configuration options that you have when choosing how to deploy Spark. Although most of the information is irrelevant to the majority of users, it is worth mentioning if you’re performing more advanced use cases. It might seem fallacious, but there are other configurations that we have omitted that control even lower-level behavior. You can find these in the Spark documentation or in the Spark source code. Chapter 18 talks about some of the options that we have when monitoring Spark Applications. Chapter 18. Monitoring and Debugging This chapter covers the key details you need to monitor and debug your Spark Applications. To do this, we will walk through the Spark UI with an example query designed to help you understand how to trace your own jobs through the execution life cycle. The example we’ll look at will also help you understand how to debug your jobs and where errors are likely to occur. The Monitoring Landscape At some point, you’ll need to monitor your Spark jobs to understand where issues are occuring in them. It’s worth reviewing the different things that we can actually monitor and outlining some of the options for doing so. Let’s review the components we can monitor (see Figure 18-1). Spark Applications and Jobs The first thing you’ll want to begin monitoring when either debugging or just understanding better how your application executes against the cluster is the Spark UI and the Spark logs. These report information about the applications currently running at the level of concepts in Spark, such as RDDs and query plans. We talk in detail about how to use these Spark monitoring tools throughout this chapter. JVM Spark runs the executors in individual Java Virtual Machines (JVMs). Therefore, the next level of detail would be to monitor the individual virtual machines (VMs) to better understand how your code is running. JVM utilities such as jstack for providing stack traces, jmap for creating heap-dumps, jstat for reporting time–series statistics, and jconsole for visually exploring various JVM properties are useful for those comfortable with JVM internals. You can also use a tool like jvisualvm to help profile Spark jobs. Some of this information is provided in the Spark UI, but for very low-level debugging, the aforementioned tools can come in handy. OS/Machine The JVMs run on a host operating system (OS) and it’s important to monitor the state of those machines to ensure that they are healthy. This includes monitoring things like CPU, network, and I/O. These are often reported in cluster-level monitoring solutions; however, there are more specific tools that you can use, including dstat, iostat, and iotop. Cluster Naturally, you can monitor the cluster on which your Spark Application(s) will run. This might be a YARN, Mesos, or standalone cluster. Usually it’s important to have some sort of monitoring solution here because, somewhat obviously, if your cluster is not working, you should probably know pretty quickly. Some popular cluster-level monitoring tools include Ganglia and Prometheus. Figure 18-1. Components of a Spark application that you can monitor What to Monitor After that brief tour of the monitoring landscape, let’s discuss how we can go about monitoring and debugging our Spark Applications. There are two main things you will want to monitor: the processes running your application (at the level of CPU usage, memory usage, etc.), and the query execution inside it (e.g., jobs and tasks). Driver and Executor Processes When you’re monitoring a Spark application, you’re definitely going to want to keep an eye on the driver. This is where all of the state of your application lives, and you’ll need to be sure it’s running in a stable manner. If you could monitor only one machine or a single JVM, it would definitely be the driver. With that being said, understanding the state of the executors is also extremely important for monitoring individual Spark jobs. To help with this challenge, Spark has a configurable metrics system based on the Dropwizard Metrics Library. The metrics system is configured via a configuration file that Spark expects to be present at $SPARK_HOME/conf/metrics.properties. A custom file location can be specified by changing the spark.metrics.conf configuration property. These metrics can be output to a variety of different sinks, including cluster monitoring solutions like Ganglia. Queries, Jobs, Stages, and Tasks Although the driver and executor processes are important to monitor, sometimes you need to debug what’s going on at the level of a specific query. Spark provides the ability to dive into queries, jobs, stages, and tasks. (We learned about these in Chapter 15.) This information allows you to know exactly what’s running on the cluster at a given time. When looking for performance tuning or debugging, this is where you are most likely to start. Now that we know what we want to monitor, let’s look at the two most common ways of doing so: the Spark logs and the Spark UI. Spark Logs One of the most detailed ways to monitor Spark is through its log files. Naturally, strange events in Spark’s logs, or in the logging that you added to your Spark Application, can help you take note of exactly where jobs are failing or what is causing that failure. If you use the application template provided with the book, the logging framework we set up in the template will allow your application logs to show up along Spark’s own logs, making them very easy to correlate. One challenge, however, is that Python won’t be able to integrate directly with Spark’s Javabased logging library. Using Python’s logging module or even simple print statements will still print the results to standard error, however, and make them easy to find. To change Spark’s log level, simply run the following command: spark.sparkContext.setLogLevel("INFO") This will allow you to read the logs, and if you use our application template, you can log your own relevant information along with these logs, allowing you to inspect both your own application and Spark. The logs themselves will be printed to standard error when running a local mode application, or saved to files by your cluster manager when running Spark on a cluster. Refer to each cluster manager’s documentation about how to find them—typically, they are available through the cluster manager’s web UI. You won’t always find the answer you need simply by searching logs, but it can help you pinpoint the given problem that you’re encountering and possibly add new log statements in your application to better understand it. It’s also convenient to collect logs over time in order to reference them in the future. For instance, if your application crashes, you’ll want to debug why, without access to the now-crashed application. You may also want to ship logs off the machine they were written on to hold onto them if a machine crashes or gets shut down (e.g., if running in the cloud). The Spark UI The Spark UI provides a visual way to monitor applications while they are running as well as metrics about your Spark workload, at the Spark and JVM level. Every SparkContext running launches a web UI, by default on port 4040, that displays useful information about the application. When you run Spark in local mode, for example, just navigate to http://localhost:4040 to see the UI when running a Spark Application on your local machine. If you’re running multiple applications, they will launch web UIs on increasing port numbers (4041, 4042, …). Cluster managers will also link to each application’s web UI from their own UI. Figure 18-2 shows all of the tabs available in the Spark UI. Figure 18-2. Spark UI tabs These tabs are accessible for each of the things that we’d like to monitor. For the most part, each of these should be self-explanatory: The Jobs tab refers to Spark jobs. The Stages tab pertains to individual stages (and their relevant tasks). The Storage tab includes information and the data that is currently cached in our Spark Application. The Environment tab contains relevant information about the configurations and current settings of the Spark application. The SQL tab refers to our Structured API queries (including SQL and DataFrames). The Executors tab provides detailed information about each executor running our application. Let’s walk through an example of how you can drill down into a given query. Open a new Spark shell, run the following code, and we will trace its execution through the Spark UI: # in Python spark.read\ .option("header", "true")\ .csv("/data/retail-data/all/online-retail-dataset.csv")\ .repartition(2)\ .selectExpr("instr(Description, 'GLASS') >= 1 as is_glass")\ .groupBy("is_glass")\ .count()\ .collect() This results in three rows of various values. The code kicks off a SQL query, so let’s navigate to the SQL tab, where you should see something similar to Figure 18-3. Figure 18-3. The SQL tab The first thing you see is aggregate statistics about this query: Submitted Time: 2017/04/08 16:24:41 Duration: 2 s Succeeded Jobs: 2 These will become important in a minute, but first let’s take a look at the Directed Acyclic Graph (DAG) of Spark stages. Each blue box in these tabs represent a stage of Spark tasks. The entire group of these stages represent our Spark job. Let’s take a look at each stage in detail so that we can better understand what is going on at each level, starting with Figure 18-4. Figure 18-4. Stage one The box on top, labeled WholeStateCodegen, represents a full scan of the CSV file. The box below that represents a shuffle that we forced when we called repartition. This turned our original dataset (of a yet to be specified number of partitions) into two partitions. The next step is our projection (selecting/adding/filtering columns) and the aggregation. Notice that in Figure 18-5 the number of output rows is six. This convienently lines up with the number of output rows multiplied by the number of partitions at aggregation time. This is because Spark performs an aggregation for each partition (in this case a hash-based aggregation) before shuffling the data around in preparation for the final stage. Figure 18-5. Stage two The last stage is the aggregation of the subaggregations that we saw happen on a per-partition basis in the previous stage. We combine those two partitions in the final three rows that are the output of our total query (Figure 18-6). Figure 18-6. Stage three Let’s look further into the job’s execution. On the Jobs tab, next to Succeeded Jobs, click 2. As Figure 18-7 demonstrates, our job breaks down into three stages (which corresponds to what we saw on the SQL tab). Figure 18-7. The Jobs tab These stages have more or less the same information as what’s shown in Figure 18-6, but clicking the label for one of them will show the details for a given stage. In this example, three stages ran, with eight, two, and then two hundred tasks each. Before diving into the stage detail, let’s review why this is the case. The first stage has eight tasks. CSV files are splittable, and Spark broke up the work to be distributed relatively evenly between the different cores on the machine. This happens at the cluster level and points to an important optimization: how you store your files. The following stage has two tasks because we explicitly called a repartition to move the data into two partitions. The last stage has 200 tasks because the default shuffle partitions value is 200. Now that we reviewed how we got here, click the stage with eight tasks to see the next level of detail, as shown in Figure 18-8. Figure 18-8. Spark tasks Spark provides a lot of detail about what this job did when it ran. Toward the top, notice the Summary Metrics section. This provides a synopsis of statistics regarding various metrics. What you want to be on the lookout for is uneven distributions of the values (we touch on this in Chapter 19). In this case, everything looks very consistent; there are no wide swings in the distribution of values. In the table at the bottom, we can also examine on a per-executor basis (one for every core on this particular machine, in this case). This can help identify whether a particular executor is struggling with its workload. Spark also makes available a set of more detailed metrics, as shown in Figure 18-8, which are probably not relevant to the large majority of users. To view those, click Show Additional Metrics, and then either choose (De)select All or select individual metrics, depending on what you want to see. You can repeat this basic analysis for each stage that you want to analyze. We leave that as an exercise for the reader. Other Spark UI tabs The remaining Spark tabs, Storage, Environment, and Executors, are fairly self-explanatory. The Storage tab shows information about the cached RDDs/DataFrames on the cluster. This can help you see if certain data has been evicted from the cache over time. The Environment tab shows you information about the Runtime Environment, including information about Scala and Java as well as the various Spark Properties that you configured on your cluster. Configuring the Spark user interface There are a number of configurations that you can set regarding the Spark UI. Many of them are networking configurations such as enabling access control. Others let you configure how the Spark UI will behave (e.g., how many jobs, stages, and tasks are stored). Due to space limitations, we cannot include the entire configuration set here. Consult the relevant table on Spark UI Configurations in the Spark documentation. Spark REST API In addition to the Spark UI, you can also access Spark’s status and metrics via a REST API. This is is available at http://localhost:4040/api/v1 and is a way of building visualizations and monitoring tools on top of Spark itself. For the most part this API exposes the same information presented in the web UI, except that it doesn’t include any of the SQL-related information. This can be a useful tool if you would like to build your own reporting solution based on the information available in the Spark UI. Due to space limitations, we cannot include the list of API endpoints here. Consult the relevant table on REST API Endpoints in the Spark documentation. Spark UI History Server Normally, the Spark UI is only available while a SparkContext is running, so how can you get to it after your application crashes or ends? To do this, Spark includes a tool called the Spark History Server that allows you to reconstruct the Spark UI and REST API, provided that the application was configured to save an event log. You can find up-to-date information about how to use this tool in the Spark documentation. To use the history server, you first need to configure your application to store event logs to a certain location. You can do this by by enabling spark.eventLog.enabled and the event log location with the configuration spark.eventLog.dir. Then, once you have stored the events, you can run the history server as a standalone application, and it will automatically reconstruct the web UI based on these logs. Some cluster managers and cloud services also configure logging automatically and run a history server by default. There are a number of other configurations for the history server. Due to space limitations, we cannot include the entire configuration set here. Refer to the relevant table on Spark History Server Configurations in the Spark documentation. Debugging and Spark First Aid The previous sections defined some core “vital signs”—that is, things that we can monitor to check the health of a Spark Application. For the remainder of the chapter we’re going to take a “first aid” approach to Spark debugging: We’ll review some signs and symptoms of problems in your Spark jobs, including signs that you might observe (e.g., slow tasks) as well as symptoms from Spark itself (e.g., OutOfMemoryError). There are many issues that may affect Spark jobs, so it’s impossible to cover everything. But we will discuss some of the more common Spark issues you may encounter. In addition to the signs and symptoms, we’ll also look at some potential treatments for these issues. Most of the recommendations about fixing issues refer to the configuration tools discussed in Chapter 16. Spark Jobs Not Starting This issue can arise frequently, especially when you’re just getting started with a fresh deployment or environment. Signs and symptoms Spark jobs don’t start. The Spark UI doesn’t show any nodes on the cluster except the driver. The Spark UI seems to be reporting incorrect information. Potential treatments This mostly occurs when your cluster or your application’s resource demands are not configured properly. Spark, in a distributed setting, does make some assumptions about networks, file systems, and other resources. During the process of setting up the cluster, you likely configured something incorrectly, and now the node that runs the driver cannot talk to the executors. This might be because you didn’t specify what IP and port is open or didn’t open the correct one. This is most likely a cluster level, machine, or configuration issue. Another option is that your application requested more resources per executor than your cluster manager currently has free, in which case the driver will be waiting forever for executors to be launched. Ensure that machines can communicate with one another on the ports that you expect. Ideally, you should open up all ports between the worker nodes unless you have more stringent security constraints. Ensure that your Spark resource configurations are correct and that your cluster manager is properly set up for Spark. Try running a simple application first to see if that works. One common issue may be that you requested more memory per executor than the cluster manager has free to allocate, so check how much it is reporting free (in its UI) and your spark-submit memory configuration. Errors Before Execution This can happen when you’re developing a new application and have previously run code on this cluster, but now some new code won’t work. Signs and symptoms Commands don’t run at all and output large error messages. You check the Spark UI and no jobs, stages, or tasks seem to run. Potential treatments After checking and confirming that the Spark UI environment tab shows the correct information for your application, it’s worth double-checking your code. Many times, there might be a simple typo or incorrect column name that is preventing the Spark job from compiling into its underlying Spark plan (when using the DataFrame API). You should take a look at the error returned by Spark to confirm that there isn’t an issue in your code, such as providing the wrong input file path or field name. Double-check to verify that the cluster has the network connectivity that you expect between your driver, your workers, and the storage system you are using. There might be issues with libraries or classpaths that are causing the wrong version of a library to be loaded for accessing storage. Try simplifying your application until you get a smaller version that reproduces the issue (e.g., just reading one dataset). Errors During Execution This kind of issue occurs when you already are working on a cluster or parts of your Spark Application run before you encounter an error. This can be a part of a scheduled job that runs at some interval or a part of some interactive exploration that seems to fail after some time. Signs and symptoms One Spark job runs successfully on the entire cluster but the next one fails. A step in a multistep query fails. A scheduled job that ran yesterday is failing today. Difficult to parse error message. Potential treatments Check to see if your data exists or is in the format that you expect. This can change over time or some upstream change may have had unintended consequences on your application. If an error quickly pops up when you run a query (i.e., before tasks are launched), it is most likely an analysis error while planning the query. This means that you likely misspelled a column name referenced in the query or that a column, view, or table you referenced does not exist. Read through the stack trace to try to find clues about what components are involved (e.g., what operator and stage it was running in). Try to isolate the issue by progressively double-checking input data and ensuring the data conforms to your expectations. Also try removing logic until you can isolate the problem in a smaller version of your application. If a job runs tasks for some time and then fails, it could be due to a problem with the input data itself, wherein the schema might be specified incorrectly or a particular row does not conform to the expected schema. For instance, sometimes your schema might specify that the data contains no nulls but your data does actually contain nulls, which can cause certain transformations to fail. It’s also possible that your own code for processing the data is crashing, in which case Spark will show you the exception thrown by your code. In this case, you will see a task marked as “failed” on the Spark UI, and you can also view the logs on that machine to understand what it was doing when it failed. Try adding more logs inside your code to figure out which data record was being processed. Slow Tasks or Stragglers This issue is quite common when optimizing applications, and can occur either due to work not being evenly distributed across your machines (“skew”), or due to one of your machines being slower than the others (e.g., due to a hardware problem). Signs and symptoms Any of the following are appropriate symptoms of the issue: Spark stages seem to execute until there are only a handful of tasks left. Those tasks then take a long time. These slow tasks show up in the Spark UI and occur consistently on the same dataset(s). These occur in stages, one after the other. Scaling up the number of machines given to the Spark Application doesn’t really help— some tasks still take much longer than others. In the Spark metrics, certain executors are reading and writing much more data than others. Potential treatments Slow tasks are often called “stragglers.” There are many reasons they may occur, but most often the source of this issue is that your data is partitioned unevenly into DataFrame or RDD partitions. When this happens, some executors might need to work on much larger amounts of work than others. One particularly common case is that you use a group-by-key operation and one of the keys just has more data than others. In this case, when you look at the Spark UI, you might see that the shuffle data for some nodes is much larger than for others. Try increasing the number of partitions to have less data per partition. Try repartitioning by another combination of columns. For example, stragglers can come up when you partition by a skewed ID column, or a column where many values are null. In the latter case, it might make sense to first filter out the null values. Try increasing the memory allocated to your executors if possible. Monitor the executor that is having trouble and see if it is the same machine across jobs; you might also have an unhealthy executor or machine in your cluster—for example, one whose disk is nearly full. If this issue is associated with a join or an aggregation, see “Slow Joins” or “Slow Aggregations”. Check whether your user-defined functions (UDFs) are wasteful in their object allocation or business logic. Try to convert them to DataFrame code if possible. Ensure that your UDFs or User-Defined Aggregate Functions (UDAFs) are running on a small enough batch of data. Oftentimes an aggregation can pull a lot of data into memory for a common key, leading to that executor having to do a lot more work than others. Turning on speculation, which we discuss in “Slow Reads and Writes”, will have Spark run a second copy of tasks that are extremely slow. This can be helpful if the issue is due to a faulty node because the task will get to run on a faster one. Speculation does come at a cost, however, because it consumes additional resources. In addition, for some storage systems that use eventual consistency, you could end up with duplicate output data if your writes are not idempotent. (We discussed speculation configurations in Chapter 17.) Another common issue can arise when you’re working with Datasets. Because Datasets perform a lot of object instantiation to convert records to Java objects for UDFs, they can cause a lot of garbage collection. If you’re using Datasets, look at the garbage collection metrics in the Spark UI to see if they’re consistent with the slow tasks. Stragglers can be one of the most difficult issues to debug, simply because there are so many possible causes. However, in all likelihood, the cause will be some kind of data skew, so definitely begin by checking the Spark UI for imbalanced amounts of data across tasks. Slow Aggregations If you have a slow aggregation, start by reviewing the issues in the “Slow Tasks” section before proceeding. Having tried those, you might continue to see the same problem. Signs and symptoms Slow tasks during a groupBy call. Jobs after the aggregation are slow, as well. Potential treatments Unfortunately, this issue can’t always be solved. Sometimes, the data in your job just has some skewed keys, and the operation you want to run on them needs to be slow. Increasing the number of partitions, prior to an aggregation, might help by reducing the number of different keys processed in each task. Increasing executor memory can help alleviate this issue, as well. If a single key has lots of data, this will allow its executor to spill to disk less often and finish faster, although it may still be much slower than executors processing other keys. If you find that tasks after the aggregation are also slow, this means that your dataset might have remained unbalanced after the aggregation. Try inserting a repartition call to partition it randomly. Ensuring that all filters and SELECT statements that can be are above the aggregation can help to ensure that you’re working only on the data that you need to be working on and nothing else. Spark’s query optimizer will automatically do this for the structured APIs. Ensure null values are represented correctly (using Spark’s concept of null) and not as some default value like " " or "EMPTY". Spark often optimizes for skipping nulls early in the job when possible, but it can’t do so for your own placeholder values. Some aggregation functions are also just inherently slower than others. For instance, collect_list and collect_set are very slow aggregation functions because they must return all the matching objects to the driver, and should be avoided in performance-critical code. Slow Joins Joins and aggregations are both shuffles, so they share some of the same general symptoms as well as treatments. Signs and symptoms A join stage seems to be taking a long time. This can be one task or many tasks. Stages before and after the join seem to be operating normally. Potential treatments Many joins can be optimized (manually or automatically) to other types of joins. We covered how to select different join types in Chapter 8. Experimenting with different join orderings can really help speed up jobs, especially if some of those joins filter out a large amount of data; do those first. Partitioning a dataset prior to joining can be very helpful for reducing data movement across the cluster, especially if the same dataset will be used in multiple join operations. It’s worth experimenting with different prejoin partitioning. Keep in mind, again, that this isn’t “free” and does come at the cost of a shuffle. Slow joins can also be caused by data skew. There’s not always a lot you can do here, but sizing up the Spark application and/or increasing the size of executors can help, as described in earlier sections. Ensuring that all filters and select statements that can be are above the join can help to ensure that you’re working only on the data that you need for the join. Ensure that null values are handled correctly (that you’re using null) and not some default value like " " or "EMPTY", as with aggregations. Sometimes Spark can’t properly plan for a broadcast join if it doesn’t know any statistics about the input DataFrame or table. If you know that one of the tables that you are joining is small, you can try to force a broadcast (as discussed in Chapter 8), or use Spark’s statistics collection commands to let it analyze the table. Slow Reads and Writes Slow I/O can be difficult to diagnose, especially with networked file systems. Signs and symptoms Slow reading of data from a distributed file system or external system. Slow writes from network file systems or blob storage. Potential treatments Turning on speculation (set spark.speculation to true) can help with slow reads and writes. This will launch additional tasks with the same operation in an attempt to see whether it’s just some transient issue in the first task. Speculation is a powerful tool and works well with consistent file systems. However, it can cause duplicate data writes with some eventually consistent cloud services, such as Amazon S3, so check whether it is supported by the storage system connector you are using. Ensuring sufficient network connectivity can be important—your Spark cluster may simply not have enough total network bandwidth to get to your storage system. For distributed file systems such as HDFS running on the same nodes as Spark, make sure Spark sees the same hostnames for nodes as the file system. This will enable Spark to do locality-aware scheduling, which you will be able to see in the “locality” column in the Spark UI. We’ll talk about locality a bit more in the next chapter. Driver OutOfMemoryError or Driver Unresponsive This is usually a pretty serious issue because it will crash your Spark Application. It often happens due to collecting too much data back to the driver, making it run out of memory. Signs and symptoms Spark Application is unresponsive or crashed. OutOfMemoryErrors or garbage collection messages in the driver logs. Commands take a very long time to run or don’t run at all. Interactivity is very low or non-existent. Memory usage is high for the driver JVM. Potential treatments There are a variety of potential reasons for this happening, and diagnosis is not always straightforward. Your code might have tried to collect an overly large dataset to the driver node using operations such as collect. You might be using a broadcast join where the data to be broadcast is too big. Use Spark’s maximum broadcast join configuration to better control the size it will broadcast. A long-running application generated a large number of objects on the driver and is unable to release them. Java’s jmap tool can be useful to see what objects are filling most of the memory of your driver JVM by printing a histogram of the heap. However, take note that jmap will pause that JVM while running. Increase the driver’s memory allocation if possible to let it work with more data. Issues with JVMs running out of memory can happen if you are using another language binding, such as Python, due to data conversion between the two requiring too much memory in the JVM. Try to see whether your issue is specific to your chosen language and bring back less data to the driver node, or write it to a file instead of bringing it back as in-memory objects. If you are sharing a SparkContext with other users (e.g., through the SQL JDBC server and some notebook environments), ensure that people aren’t trying to do something that might be causing large amounts of memory allocation in the driver (like working overly large arrays in their code or collecting large datasets). Executor OutOfMemoryError or Executor Unresponsive Spark applications can sometimes recover from this automatically, depending on the true underlying issue. Signs and symptoms OutOfMemoryErrors or garbage collection messages in the executor logs. You can find these in the Spark UI. Executors that crash or become unresponsive. Slow tasks on certain nodes that never seem to recover. Potential treatments Try increasing the memory available to executors and the number of executors. Try increasing PySpark worker size via the relevant Python configurations. Look for garbage collection error messages in the executor logs. Some of the tasks that are running, especially if you’re using UDFs, can be creating lots of objects that need to be garbage collected. Repartition your data to increase parallelism, reduce the amount of records per task, and ensure that all executors are getting the same amount of work. Ensure that null values are handled correctly (that you’re using null) and not some default value like " " or "EMPTY", as we discussed earlier. This is more likely to happen with RDDs or with Datasets because of object instantiations. Try using fewer UDFs and more of Spark’s structured operations when possible. Use Java monitoring tools such as jmap to get a histogram of heap memory usage on your executors, and see which classes are taking up the most space. If executors are being placed on nodes that also have other workloads running on them, such as a key-value store, try to isolate your Spark jobs from other jobs. Unexpected Nulls in Results Signs and symptoms Unexpected null values after transformations. Scheduled production jobs that used to work no longer work, or no longer produce the right results. Potential treatments It’s possible that your data format has changed without adjusting your business logic. This means that code that worked before is no longer valid. Use an accumulator to try to count records or certain types, as well as parsing or processing errors where you skip a record. This can be helpful because you might think that you’re parsing data of a certain format, but some of the data doesn’t. Most often, users will place the accumulator in a UDF when they are parsing their raw data into a more controlled format and perform the counts there. This allows you to count valid and invalid records and then operate accordingly after the fact. Ensure that your transformations actually result in valid query plans. Spark SQL sometimes does implicit type coercions that can cause confusing results. For instance, the SQL expression SELECT 5*"23" results in 115 because the string “25” converts to an the value 25 as an integer, but the expression SELECT 5 * " " results in null because casting the empty string to an integer gives null. Make sure that your intermediate datasets have the schema you expect them to (try using printSchema on them), and look for any CAST operations in the final query plan. No Space Left on Disk Errors Signs and symptoms You see “no space left on disk” errors and your jobs fail. Potential treatments The easiest way to alleviate this, of course, is to add more disk space. You can do this by sizing up the nodes that you’re working on or attaching external storage in a cloud environment. If you have a cluster with limited storage space, some nodes may run out first due to skew. Repartitioning the data as described earlier may help here. There are also a number of storage configurations with which you can experiment. Some of these determine how long logs should be kept on the machine before being removed. For more information, see the Spark executor logs rolling configurations in Chapter 16. Try manually removing some old log files or old shuffle files from the machine(s) in question. This can help alleviate some of the issue although obviously it’s not a permanent fix. Serialization Errors Signs and symptoms You see serialization errors and your jobs fail. Potential treatments This is very uncommon when working with the Structured APIs, but you might be trying to perform some custom logic on executors with UDFs or RDDs and either the task that you’re trying to serialize to these executors or the data you are trying to share cannot be serialized. This often happens when you’re working with either some code or data that cannot be serialized into a UDF or function, or if you’re working with strange data types that cannot be serialized. If you are using (or intend to be using Kryo serialization), verify that you’re actually registering your classes so that they are indeed serialized. Try not to refer to any fields of the enclosing object in your UDFs when creating UDFs inside a Java or Scala class. This can cause Spark to try to serialize the whole enclosing object, which may not be possible. Instead, copy the relevant fields to local variables in the same scope as closure and use those. Conclusion This chapter covered some of the main tools that you can use to monitor and debug your Spark jobs and applications, as well as the most common issues we see and their resolutions. As with debugging any complex software, we recommend taking a principled, step-by-step approach to debug issues. Add logging statements to figure out where your job is crashing and what type of data arrives at each stage, try to isolate the problem to the smallest piece of code possible, and work up from there. For data skew issues, which are unique to parallel computing, use Spark’s UI to get a quick overview of how much work each task is doing. In Chapter 19, we discuss performance tuning in particular and various tools you can use for that. Chapter 19. Performance Tuning Chapter 18 covered the Spark user interface (UI) and basic first-aid for your Spark Application. Using the tools outlined in that chapter, you should be able to ensure that your jobs run reliably. However, sometimes you’ll also need them to run faster or more efficiently for a variety of reasons. That’s what this chapter is about. Here, we present a discussion of some of the performance choices that are available to make your jobs run faster. Just as with monitoring, there are a number of different levels that you can try to tune at. For instance, if you had an extremely fast network, that would make many of your Spark jobs faster because shuffles are so often one of the costlier steps in a Spark job. Most likely, you won’t have much ability to control such things; therefore, we’re going to discuss the things you can control through code choices or configuration. There are a variety of different parts of Spark jobs that you might want to optimize, and it’s valuable to be specific. Following are some of the areas: Code-level design choices (e.g., RDDs versus DataFrames) Data at rest Joins Aggregations Data in flight Individual application properties Inside of the Java Virtual Machine (JVM) of an executor Worker nodes Cluster and deployment properties This list is by no means exhaustive, but it does at least ground the conversation and the topics that we cover in this chapter. Additionally, there are two ways of trying to achieve the execution characteristics that we would like out of Spark jobs. We can either do so indirectly by setting configuration values or changing the runtime environment. These should improve things across Spark Applications or across Spark jobs. Alternatively, we can try to directly change execution characteristic or design choices at the individual Spark job, stage, or task level. These kinds of fixes are very specific to that one area of our application and therefore have limited overall impact. There are numerous things that lie on both sides of the indirect versus direct divide, and we will draw lines in the sand accordingly. One of the best things you can do to figure out how to improve performance is to implement good monitoring and job history tracking. Without this information, it can be difficult to know whether you’re really improving job performance. Indirect Performance Enhancements As discussed, there are a number of indirect enhancements that you can perform to help your Spark jobs run faster. We’ll skip the obvious ones like “improve your hardware” and focus more on the things within your control. Design Choices Although good design choices seem like a somewhat obvious way to optimize performance, we often don’t prioritize this step in the process. When designing your applications, making good design choices is very important because it not only helps you to write better Spark applications but also to get them to run in a more stable and consistent manner over time and in the face of external changes or variations. We’ve already discussed some of these topics earlier in the book, but we’ll summarize some of the fundamental ones again here. Scala versus Java versus Python versus R This question is nearly impossible to answer in the general sense because a lot will depend on your use case. For instance, if you want to perform some single-node machine learning after performing a large ETL job, we might recommend running your Extract, Transform, and Load (ETL) code as SparkR code and then using R’s massive machine learning ecosystem to run your single-node machine learning algorithms. This gives you the best of both worlds and takes advantage of the strength of R as well as the strength of Spark without sacrifices. As we mentioned numerous times, Spark’s Structured APIs are consistent across languages in terms of speed and stability. That means that you should code with whatever language you are most comfortable using or is best suited for your use case. Things do get a bit more complicated when you need to include custom transformations that cannot be created in the Structured APIs. These might manifest themselves as RDD transformations or user-defined functions (UDFs). If you’re going to do this, R and Python are not necessarily the best choice simply because of how this is actually executed. It’s also more difficult to provide stricter guarantees of types and manipulations when you’re defining functions that jump across languages. We find that using Python for the majority of the application, and porting some of it to Scala or writing specific UDFs in Scala as your application evolves, is a powerful technique—it allows for a nice balance between overall usability, maintainability, and performance. DataFrames versus SQL versus Datasets versus RDDs This question also comes up frequently. The answer is simple. Across all languages, DataFrames, Datasets, and SQL are equivalent in speed. This means that if you’re using DataFrames in any of these languages, performance is equal. However, if you’re going to be defining UDFs, you’ll take a performance hit writing those in Python or R, and to some extent a lesser performance hit in Java and Scala. If you want to optimize for pure performance, it would behoove you to try and get back to DataFrames and SQL as quickly as possible. Although all DataFrame, SQL, and Dataset code compiles down to RDDs, Spark’s optimization engine will write “better” RDD code than you can manually and certainly do it with orders of magnitude less effort. Additionally, you will lose out on new optimizations that are added to Spark’s SQL engine every release. Lastly, if you want to use RDDs, we definitely recommend using Scala or Java. If that’s not possible, we recommend that you restrict the “surface area” of RDDs in your application to the bare minimum. That’s because when Python runs RDD code, it’s serializes a lot of data to and from the Python process. This is very expensive to run over very big data and can also decrease stability. Although it isn’t exactly relevant to performance tuning, it’s important to note that there are also some gaps in what functionality is supported in each of Spark’s languages. We discussed this in Chapter 16. Object Serialization in RDDs In Part III, we briefly discussed the serialization libraries that can be used within RDD transformations. When you’re working with custom data types, you’re going to want to serialize them using Kryo because it’s both more compact and much more efficient than Java serialization. However, this does come at the inconvenience of registering the classes that you will be using in your application. You can use Kryo serialization by setting spark.serializer to org.apache.spark.serializer.KryoSerializer. You will also need to explicitly register the classes that you would like to register with the Kryo serializer via the spark.kryo.classesToRegister configuration. There are also a number of advanced parameters for controlling this in greater detail that are described in the Kryo documentation. To register your classes, use the SparkConf that you just created and pass in the names of your classes: conf.registerKryoClasses(Array(classOf[MyClass1], classOf[MyClass2])) Cluster Configurations This area has huge potential benefits but is probably one of the more difficult to prescribe because of the variation across hardware and use cases. In general, monitoring how the machines themselves are performing will be the most valuable approach toward optimizing your cluster configurations, especially when it comes to running multiple applications (whether they are Spark or not) on a single cluster. Cluster/application sizing and sharing This somewhat comes down to a resource sharing and scheduling problem; however, there are a lot of options for how you want to share resources at the cluster level or at the application level. Take a look at the configurations listed at the end of Chapter 16 as well as some configurations in Chapter 17. Dynamic allocation Spark provides a mechanism to dynamically adjust the resources your application occupies based on the workload. This means that your application can give resources back to the cluster if they are no longer used, and request them again later when there is demand. This feature is particularly useful if multiple applications share resources in your Spark cluster. This feature is disabled by default and available on all coarse-grained cluster managers; that is, standalone mode, YARN mode, and Mesos coarse-grained mode. If you’d like to enable this feature, you should set spark.dynamicAllocation.enabled to true. The Spark documentation presents a number of individual parameters that you can tune. Scheduling Over the course of the previous chapters, we discussed a number of different potential optimizations that you can take advantage of to either help Spark jobs run in parallel with scheduler pools or help Spark applications run in parallel with something like dynamic allocation or setting max-executor-cores. Scheduling optimizations do involve some research and experimentation, and unfortunately there are not super-quick fixes beyond setting spark.scheduler.mode to FAIR to allow better sharing of resources across multiple users, or setting --max-executor-cores, which specifies the maximum number of executor cores that your application will need. Specifying this value can ensure that your application does not take up all the resources on the cluster. You can also change the default, depending on your cluster manager, by setting the configuration spark.cores.max to a default of your choice. Cluster managers also provide some scheduling primitives that can be helpful when optimizing multiple Spark Applications, as discussed in Chapters 16 and 17. Data at Rest More often that not, when you’re saving data it will be read many times as other folks in your organization access the same datasets in order to run different analyses. Making sure that you’re storing your data for effective reads later on is absolutely essential to successful big data projects. This involves choosing your storage system, choosing your data format, and taking advantage of features such as data partitioning in some storage formats. File-based long-term data storage There are a number of different file formats available, from simple comma-separated values (CSV) files and binary blobs, to more sophisticated formats like Apache Parquet. One of the easiest ways to optimize your Spark jobs is to follow best practices when storing data and choose the most efficient storage format possible. Generally you should always favor structured, binary types to store your data, especially when you’ll be accessing it frequently. Although files like “CSV” seem well-structured, they’re very slow to parse, and often also full of edge cases and pain points. For instance, improperly escaped new-line characters can often cause a lot of trouble when reading a large number of files. The most efficient file format you can generally choose is Apache Parquet. Parquet stores data in binary files with column-oriented storage, and also tracks some statistics about each file that make it possible to quickly skip data not needed for a query. It is well integrated with Spark through the built-in Parquet data source. Splittable file types and compression Whatever file format you choose, you should make sure it is “splittable”, which means that different tasks can read different parts of the file in parallel. We saw why this is important in Chapter 18. When we read in the file, all cores were able to do part of the work. That’s because the file was splittable. If we didn’t use a splittable file type—say something like a malformed JSON file—we’re going to need to read in the entire file on a single machine, greatly reducing parallelism. The main place splittability comes in is compression formats. A ZIP file or TAR archive cannot be split, which means that even if we have 10 files in a ZIP file and 10 cores, only one core can read in that data because we cannot parallelize access to the ZIP file. This is a poor use of resources. In contrast, files compressed using gzip, bzip2, or lz4 are generally splittable if they were written by a parallel processing framework like Hadoop or Spark. For your own input data, the simplest way to make it splittable is to upload it as separate files, ideally each no larger than a few hundred megabytes. Table partitioning We discussed table partitioning in Chapter 9, and will only use this section as a reminder. Table partitioning refers to storing files in separate directories based on a key, such as the date field in the data. Storage managers like Apache Hive support this concept, as do many of Spark’s built-in data sources. Partitioning your data correctly allows Spark to skip many irrelevant files when it only requires data with a specific range of keys. For instance, if users frequently filter by “date” or “customerId” in their queries, partition your data by those columns. This will greatly reduce the amount of data that end users must read by most queries, and therefore dramatically increase speed. The one downside of partitioning, however, is that if you partition at too fine a granularity, it can result in many small files, and a great deal of overhead trying to list all the files in the storage system. Bucketing We also discussed bucketing in Chapter 9, but to recap, the essense is that bucketing your data allows Spark to “pre-partition” data according to how joins or aggregations are likely to be performed by readers. This can improve performance and stability because data can be consistently distributed across partitions as opposed to skewed into just one or two. For instance, if joins are frequently performed on a column immediately after a read, you can use bucketing to ensure that the data is well partitioned according to those values. This can help prevent a shuffle before a join and therefore help speed up data access. Bucketing generally works hand-in-hand with partitioning as a second way of physically splitting up data. The number of files In addition to organizing your data into buckets and partitions, you’ll also want to consider the number of files and the size of files that you’re storing. If there are lots of small files, you’re going to pay a price listing and fetching each of those individual files. For instance, if you’re reading a data from Hadoop Distributed File System (HDFS), this data is managed in blocks that are up to 128 MB in size (by default). This means if you have 30 files, of 5 MB each, you’re going to have to potentially request 30 blocks, even though the same data could have fit into 2 blocks (150 MB total). Although there is not necessarily a panacea for how you want to store your data, the trade-off can be summarized as such. Having lots of small files is going to make the scheduler work much harder to locate the data and launch all of the read tasks. This can increase the network and scheduling overhead of the job. Having fewer large files eases the pain off the scheduler but it will also make tasks run longer. In this case, though, you can always launch more tasks than there are input files if you want more parallelism—Spark will split each file across multiple tasks assuming you are using a splittable format. In general, we recommend sizing your files so that they each contain at least a few tens of megatbytes of data. One way of controlling data partitioning when you write your data is through a write option introduced in Spark 2.2. To control how many records go into each file, you can specify the maxRecordsPerFile option to the write operation. Data locality Another aspect that can be important in shared cluster environments is data locality. Data locality basically specifies a preference for certain nodes that hold certain data, rather than having to exchange these blocks of data over the network. If you run your storage system on the same nodes as Spark, and the system supports locality hints, Spark will try to schedule tasks close to each input block of data. For example HDFS storage provides this option. There are several configurations that affect locality, but it will generally be used by default if Spark detects that it is using a local storage system. You will also see data-reading tasks marked as “local” in the Spark web UI. Statistics collection Spark includes a cost-based query optimizer that plans queries based on the properties of the input data when using the structured APIs. However, to allow the cost-based optimizer to make these sorts of decisions, you need to collect (and maintain) statistics about your tables that it can use. There are two kinds of statistics: table-level and column-level statistics. Statistics collection is available only on named tables, not on arbitrary DataFrames or RDDs. To collect table-level statistics, you can run the following command: ANALYZE TABLE table_name COMPUTE STATISTICS To collect column-level statistics, you can name the specific columns: ANALYZE TABLE table_name COMPUTE STATISTICS FOR COLUMNS column_name1, column_name2, ... Column-level statistics are slower to collect, but provide more information for the cost-based optimizer to use about those data columns. Both types of statistics can help with joins, aggregations, filters, and a number of other potential things (e.g., automatically choosing when to do a broadcast join). This is a fast-growing part of Spark, so different optimizations based on statistics will likely be added in the future. NOTE You can follow the progress of cost-based optimization on its JIRA issue. You can also read through the design document on SPARK-16026 to learn more about this feature. This is an active area of development in Spark at the time of writing. Shuffle Configurations Configuring Spark’s external shuffle service (discussed in Chapters 16 and 17) can often increase performance because it allows nodes to read shuffle data from remote machines even when the executors on those machines are busy (e.g., with garbage collection). This does come at the cost of complexity and maintenance, however, so it might not be worth it in your deployment. Beyond configuring this external service, there are also a number of configurations for shuffles, such as the number of concurrent connections per executor, although these usually have good defaults. In addition, for RDD-based jobs, the serialization format has a large impact on shuffle performance—always prefer Kryo over Java serialization, as described in “Object Serialization in RDDs”. Furthermore, for all jobs, the number of partitions of a shuffle matters. If you have too few partitions, then too few nodes will be doing work and there may be skew, but if you have too many partitions, there is an overhead to launching each one that may start to dominate. Try to aim for at least a few tens of megabytes of data per output partition in your shuffle. Memory Pressure and Garbage Collection During the course of running Spark jobs, the executor or driver machines may struggle to complete their tasks because of a lack of sufficient memory or “memory pressure.” This may occur when an application takes up too much memory during execution or when garbage collection runs too frequently or is slow to run as large numbers of objects are created in the JVM and subsequently garbage collected as they are no longer used. One strategy for easing this issue is to ensure that you’re using the Structured APIs as much as possible. These will not only increase the efficiency with which your Spark jobs will execute, but it will also greatly reduce memory pressure because JVM objects are never realized and Spark SQL simply performs the computation on its internal format. The Spark documentation includes some great pointers on tuning garbage collection for RDD and UDF based applications, and we paraphrase the following sections from that information. Measuring the impact of garbage collection The first step in garbage collection tuning is to gather statistics on how frequently garbage collection occurs and the amount of time it takes. You can do this by adding -verbose:gc XX:+PrintGCDetails -XX:+PrintGCTimeStamps to Spark’s JVM options using the spark.executor.extraJavaOptions configuration parameter. The next time you run your Spark job, you will see messages printed in the worker’s logs each time a garbage collection occurs. These logs will be on your cluster’s worker nodes (in the stdout files in their work directories), not in the driver. Garbage collection tuning To further tune garbage collection, you first need to understand some basic information about memory management in the JVM: Java heap space is divided into two regions: Young and Old. The Young generation is meant to hold short-lived objects whereas the Old generation is intended for objects with longer lifetimes. The Young generation is further divided into three regions: Eden, Survivor1, and Survivor2. Here’s a simplified description of the garbage collection procedure: 1. When Eden is full, a minor garbage collection is run on Eden and objects that are alive from Eden and Survivor1 are copied to Survivor2. 2. The Survivor regions are swapped. 3. If an object is old enough or if Survivor2 is full, that object is moved to Old. 4. Finally, when Old is close to full, a full garbage collection is invoked. This involves tracing through all the objects on the heap, deleting the unreferenced ones, and moving the others to fill up unused space, so it is generally the slowest garbage collection operation. The goal of garbage collection tuning in Spark is to ensure that only long-lived cached datasets are stored in the Old generation and that the Young generation is sufficiently sized to store all short-lived objects. This will help avoid full garbage collections to collect temporary objects created during task execution. Here are some steps that might be useful. Gather garbage collection statistics to determine whether it is being run too often. If a full garbage collection is invoked multiple times before a task completes, it means that there isn’t enough memory available for executing tasks, so you should decrease the amount of memory Spark uses for caching (spark.memory.fraction). If there are too many minor collections but not many major garbage collections, allocating more memory for Eden would help. You can set the size of the Eden to be an over-estimate of how much memory each task will need. If the size of Eden is determined to be E, you can set the size of the Young generation using the option -Xmn=4/3*E. (The scaling up by 4/3 is to account for space used by survivor regions, as well.) As an example, if your task is reading data from HDFS, the amount of memory used by the task can be estimated by using the size of the data block read from HDFS. Note that the size of a decompressed block is often two or three times the size of the block. So if you want to have three or four tasks’ worth of working space, and the HDFS block size is 128 MB, we can estimate size of Eden to be 43,128 MB. Try the G1GC garbage collector with -XX:+UseG1GC. It can improve performance in some situations in which garbage collection is a bottleneck and you don’t have a way to reduce it further by sizing the generations. Note that with large executor heap sizes, it can be important to increase the G1 region size with -XX:G1HeapRegionSize. Monitor how the frequency and time taken by garbage collection changes with the new settings. Our experience suggests that the effect of garbage collection tuning depends on your application and the amount of memory available. There are many more tuning options described online, but at a high level, managing how frequently full garbage collection takes place can help in reducing the overhead. You can specify garbage collection tuning flags for executors by setting spark.executor.extraJavaOptions in a job’s configuration. Direct Performance Enhancements In the previous section, we touched on some general performance enhancements that apply to all jobs. Be sure to skim the previous couple of pages before jumping to this section and the solutions here. These solutions here are intended as “band-aids” of sorts for issues with specific stages or jobs, but they require inspecting and optimizing each stage or job separately. Parallelism The first thing you should do whenever trying to speed up a specific stage is to increase the degree of parallelism. In general, we recommend having at least two or three tasks per CPU core in your cluster if the stage processes a large amount of data. You can set this via the spark.default.parallelism property as well as tuning the spark.sql.shuffle.partitions according to the number of cores in your cluster. Improved Filtering Another frequent source of performance enhancements is moving filters to the earliest part of your Spark job that you can. Sometimes, these filters can be pushed into the data sources themselves and this means that you can avoid reading and working with data that is irrelevant to your end result. Enabling partitioning and bucketing also helps achieve this. Always look to be filtering as much data as you can early on, and you’ll find that your Spark jobs will almost always run faster. Repartitioning and Coalescing Repartition calls can incur a shuffle. However, doing some can optimize the overall execution of a job by balancing data across the cluster, so they can be worth it. In general, you should try to shuffle the least amount of data possible. For this reason, if you’re reducing the number of overall partitions in a DataFrame or RDD, first try coalesce method, which will not perform a shuffle but rather merge partitions on the same node into one partition. The slower repartition method will also shuffle data across the network to achieve even load balancing. Repartitions can be particularly helpful when performing joins or prior to a cache call. Remember that repartitioning is not free, but it can improve overall application performance and parallelism of your jobs. Custom partitioning If your jobs are still slow or unstable, you might want to explore performing custom partitioning at the RDD level. This allows you to define a custom partition function that will organize the data across the cluster to a finer level of precision than is available at the DataFrame level. This is very rarely necessary, but it is an option. For more information, see Part III. User-Defined Functions (UDFs) In general, avoiding UDFs is a good optimization opportunity. UDFs are expensive because they force representing data as objects in the JVM and sometimes do this multiple times per record in a query. You should try to use the Structured APIs as much as possible to perform your manipulations simply because they are going to perform the transformations in a much more efficient manner than you can do in a high-level language. There is also ongoing work to make data available to UDFs in batches, such as the Vectorized UDF extension for Python that gives your code multiple records at once using a Pandas data frame. We discussed UDFs and their costs in Chapter 18. Temporary Data Storage (Caching) In applications that reuse the same datasets over and over, one of the most useful optimizations is caching. Caching will place a DataFrame, table, or RDD into temporary storage (either memory or disk) across the executors in your cluster, and make subsequent reads faster. Although caching might sound like something we should do all the time, it’s not always a good thing to do. That’s because caching data incurs a serialization, deserialization, and storage cost. For example, if you are only going to process a dataset once (in a later transformation), caching it will only slow you down. The use case for caching is simple: as you work with data in Spark, either within an interactive session or a standalone application, you will often want to reuse a certain dataset (e.g., a DataFrame or RDD). For example, in an interactive data science session, you might load and clean your data and then reuse it to try multiple statistical models. Or in a standalone application, you might run an iterative algorithm that reuses the same dataset. You can tell Spark to cache a dataset using the cache method on DataFrames or RDDs. Caching is a lazy operation, meaning that things will be cached only as they are accessed. The RDD API and the Structured API differ in how they actually perform caching, so let’s review the gory details before going over the storage levels. When we cache an RDD, we cache the actual, physical data (i.e., the bits). The bits. When this data is accessed again, Spark returns the proper data. This is done through the RDD reference. However, in the Structured API, caching is done based on the physical plan. This means that we effectively store the physical plan as our key (as opposed to the object reference) and perform a lookup prior to the execution of a Structured job. This can cause confusion because sometimes you might be expecting to access raw data but because someone else already cached the data, you’re actually accessing their cached version. Keep that in mind when using this feature. There are different storage levels that you can use to cache your data, specifying what type of storage to use. Table 19-1 lists the levels. Table 19-1. Data cache storage levels Storage level Meaning MEMORY_ONLY Store RDD as deserialized Java objects in the JVM. If the RDD does not fit in memory, some partitions will not be cached and will be recomputed on the fly each time they’re needed. This is the default level. MEMORY_AND_DISK Store RDD as deserialized Java objects in the JVM. If the RDD does not fit in memory, store the partitions that don’t fit on disk, and read them from there when they’re needed. MEMORY_ONLY_SER (Java and Scala) Store RDD as serialized Java objects (one byte array per partition). This is generally more space-efficient than deserialized objects, especially when using a fast serializer, but more CPU-intensive to read. MEMORY_AND_DISK_SER Similar to MEMORY_ONLY_SER, but spill partitions that don’t fit in memory to disk (Java and Scala) instead of recomputing them on the fly each time they’re needed. DISK_ONLY Store the RDD partitions only on disk. MEMORY_ONLY_2, MEMORY_AND_DISK_2, Same as the previous levels, but replicate each partition on two cluster nodes. etc. OFF_HEAP (experimental) Similar to MEMORY_ONLY_SER, but store the data in off-heap memory. This requires offheap memory to be enabled. For more information on these options, take a look at “Configuring Memory Management”. Figure 19-1 presents a simple illustrations of the process. We load an initial DataFrame from a CSV file and then derive some new DataFrames from it using transformations. We can avoid having to recompute the original DataFrame (i.e., load and parse the CSV file) many times by adding a line to cache it along the way. Figure 19-1. A cached DataFrame Now let’s walk through the code: # in Python # Original loading code that does *not* cache DataFrame DF1 = spark.read.format("csv")\ .option("inferSchema", "true")\ .option("header", "true")\ .load("/data/flight-data/csv/2015-summary.csv") DF2 = DF1.groupBy("DEST_COUNTRY_NAME").count().collect() DF3 = DF1.groupBy("ORIGIN_COUNTRY_NAME").count().collect() DF4 = DF1.groupBy("count").count().collect() You’ll see here that we have our “lazily” created DataFrame (DF1), along with three other DataFrames that access data in DF1. All of our downstream DataFrames share that common parent (DF1) and will repeat the same work when we perform the preceding code. In this case, it’s just reading and parsing the raw CSV data, but that can be a fairly intensive process, especially for large datasets. On my machine, those commands take a second or two to run. Luckily caching can help speed things up. When we ask for a DataFrame to be cached, Spark will save the data in memory or on disk the first time it computes it. Then, when any other queries come along, they’ll just refer to the one stored in memory as opposed to the original file. You do this using the DataFrame’s cache method: DF1.cache() DF1.count() We used the count above to eagerly cache the data (basically perform an action to force Spark to store it in memory), because caching itself is lazy—the data is cached only on the first time you run an action on the DataFrame. Now that the data is cached, the previous commands will be faster, as we can see by running the following code: # in Python DF2 = DF1.groupBy("DEST_COUNTRY_NAME").count().collect() DF3 = DF1.groupBy("ORIGIN_COUNTRY_NAME").count().collect() DF4 = DF1.groupBy("count").count().collect() When we ran this code, it cut the time by more than half! This might not seem that wild, but picture a large dataset or one that requires a lot of computation to create (not just reading in a file). The savings can be immense. It’s also great for iterative machine learning workloads because they’ll often need to access the same data a number of times, which we’ll see shortly. The cache command in Spark always places data in memory by default, caching only part of the dataset if the cluster’s total memory is full. For more control, there is also a persist method that takes a StorageLevel object to specify where to cache the data: in memory, on disk, or both. Joins Joins are a common area for optimization. The biggest weapon you have when it comes to optimizing joins is simply educating yourself about what each join does and how it’s performed. This will help you the most. Additionally, equi-joins are the easiest for Spark to optimize at this point and therefore should be preferred wherever possible. Beyond that, simple things like trying to use the filtering ability of inner joins by changing join ordering can yield large speedups. Additionally, using broadcast join hints can help Spark make intelligent planning decisions when it comes to creating query plans, as described in Chapter 8. Avoiding Cartesian joins or even full outer joins is often low-hanging fruit for stability and optimizations because these can often be optimized into different filtering style joins when you look at the entire data flow instead of just that one particular job area. Lastly, following some of the other sections in this chapter can have a significant effect on joins. For example, collecting statistics on tables prior to a join will help Spark make intelligent join decisions. Additionally, bucketing your data appropriately can also help Spark avoid large shuffles when joins are performed. Aggregations For the most part, there are not too many ways that you can optimize specific aggregations beyond filtering data before the aggregation having a sufficiently high number of partitions. However, if you’re using RDDs, controlling exactly how these aggregations are performed (e.g., using reduceByKey when possible over groupByKey) can be very helpful and improve the speed and stability of your code. Broadcast Variables We touched on broadcast joins and variables in previous chapters, and these are a good option for optimization. The basic premise is that if some large piece of data will be used across multiple UDF calls in your program, you can broadcast it to save just a single read-only copy on each node and avoid re-sending this data with each job. For example, broadcast variables may be useful to save a lookup table or a machine learning model. You can also broadcast arbitrary objects by creating broadcast variables using your SparkContext, and then simply refer to those variables in your tasks, as we discussed in Chapter 14. Conclusion There are many different ways to optimize the performance of your Spark Applications and make them run faster and at a lower cost. In general, the main things you’ll want to prioritize are (1) reading as little data as possible through partitioning and efficient binary formats, (2) making sure there is sufficient parallellism and no data skew on the cluster using partitioning, and (3) using high-level APIs such as the Structured APIs as much as possible to take already optimized code. As with any other software optimization work, you should also make sure you are optimizing the right operations for your job: the Spark monitoring tools described in Chapter 18 will let you see which stages are taking the longest time and focus your efforts on those. Once you have identified the work that you believe can be optimized, the tools in this chapter will cover the most important performance optimization opportunities for the majority of users. Part V. Streaming Chapter 20. Stream Processing Fundamentals Stream processing is a key requirement in many big data applications. As soon as an application computes something of value—say, a report about customer activity, or a new machine learning model—an organization will want to compute this result continuously in a production setting. As a result, organizations of all sizes are starting to incorporate stream processing, often even in the first version of a new application. Luckily, Apache Spark has a long history of high-level support for streaming. In 2012, the project incorporated Spark Streaming and its DStreams API, one of the first APIs to enable stream processing using high-level functional operators like map and reduce. Hundreds of organizations now use DStreams in production for large real-time applications, often processing terabytes of data per hour. Much like the Resilient Distributed Dataset (RDD) API, however, the DStreams API is based on relatively low-level operations on Java/Python objects that limit opportunities for higher-level optimization. Thus, in 2016, the Spark project added Structured Streaming, a new streaming API built directly on DataFrames that supports both rich optimizations and significantly simpler integration with other DataFrame and Dataset code. The Structured Streaming API was marked as stable in Apache Spark 2.2, and has also seen swift adoption throughout the Spark community. In this book, we will focus only on the Structured Streaming API, which integrates directly with the DataFrame and Dataset APIs we discussed earlier in the book and is the framework of choice for writing new streaming applications. If you are interested in DStreams, many other books cover that API, including several dedicated books on Spark Streaming only, such as Learning Spark Streaming by Francois Garillot and Gerard Maas (O’Reilly, 2017). Much as with RDDs versus DataFrames, however, Structured Streaming offers a superset of the majority of the functionality of DStreams, and will often perform better due to code generation and the Catalyst optimizer. Before we discuss the streaming APIs in Spark, let’s more formally define streaming and batch processing. This chapter will discuss some of the core concepts in this area that we will need throughout this part of the book. It won’t be a dissertation on this topic, but will cover enough of the concepts to let you make sense of systems in this space. What Is Stream Processing? Stream processing is the act of continuously incorporating new data to compute a result. In stream processing, the input data is unbounded and has no predetermined beginning or end. It simply forms a series of events that arrive at the stream processing system (e.g., credit card transactions, clicks on a website, or sensor readings from Internet of Things [IoT] devices). User applications can then compute various queries over this stream of events (e.g., tracking a running count of each type of event or aggregating them into hourly windows). The application will output multiple versions of the result as it runs, or perhaps keep it up to date in an external “sink” system such as a key-value store. Naturally, we can compare streaming to batch processing, in which the computation runs on a fixed-input dataset. Oftentimes, this might be a large-scale dataset in a data warehouse that contains all the historical events from an application (e.g., all website visits or sensor readings for the past month). Batch processing also takes a query to compute, similar to stream processing, but only computes the result once. Although streaming and batch processing sound different, in practice, they often need to work together. For example, streaming applications often need to join input data against a dataset written periodically by a batch job, and the output of streaming jobs is often files or tables that are queried in batch jobs. Moreover, any business logic in your applications needs to work consistently across streaming and batch execution: for example, if you have a custom code to compute a user’s billing amount, it would be harmful to get a different result when running it in a streaming versus batch fashion! To handle these needs, Structured Streaming was designed from the beginning to interoperate easily with the rest of Spark, including batch applications. Indeed, the Structured Streaming developers coined the term continuous applications to capture end-toend applications that consist of streaming, batch, and interactive jobs all working on the same data to deliver an end product. Structured Streaming is focused on making it simple to build such applications in an end-to-end fashion instead of only handling stream-level per-record processing. Stream Processing Use Cases We defined stream processing as the incremental processing of unbounded datasets, but that’s a strange way to motivate a use case. Before we get into advantages and disadvantages of streaming, let’s explain why you might want to use streaming. We’ll describe six common use cases with varying requirements from the underlying stream processing system. Notifications and alerting Probably the most obvious streaming use case involves notifications and alerting. Given some series of events, a notification or alert should be triggered if some sort of event or series of events occurs. This doesn’t necessarily imply autonomous or preprogrammed decision making; alerting can also be used to notify a human counterpart of some action that needs to be taken. An example might be driving an alert to an employee at a fulfillment center that they need to get a certain item from a location in the warehouse and ship it to a customer. In either case, the notification needs to happen quickly. Real-time reporting Many organizations use streaming systems to run real-time dashboards that any employee can look at. For example, this book’s authors leverage Structured Streaming every day to run real- time reporting dashboards throughout Databricks (where both authors of this book work). We use these dashboards to monitor total platform usage, system load, uptime, and even usage of new features as they are rolled out, among other applications. Incremental ETL One of the most common streaming applications is to reduce the latency companies must endure while retreiving information into a data warehouse—in short, “my batch job, but streaming.” Spark batch jobs are often used for Extract, Transform, and Load (ETL) workloads that turn raw data into a structured format like Parquet to enable efficient queries. Using Structured Streaming, these jobs can incorporate new data within seconds, enabling users to query it faster downstream. In this use case, it is critical that data is processed exactly once and in a fault-tolerant manner: we don’t want to lose any input data before it makes it to the warehouse, and we don’t want to load the same data twice. Moreover, the streaming system needs to make updates to the data warehouse transactionally so as not to confuse the queries running on it with partially written data. Update data to serve in real time Streaming systems are frequently used to compute data that gets served interactively by another application. For example, a web analytics product such as Google Analytics might continuously track the number of visits to each page, and use a streaming system to keep these counts up to date. When users interact with the product’s UI, this web application queries the latest counts. Supporting this use case requires that the streaming system can perform incremental updates to a key–value store (or other serving system) as a sync, and often also that these updates are transactional, as in the ETL case, to avoid corrupting the data in the application. Real-time decision making Real-time decision making on a streaming system involves analyzing new inputs and responding to them automatically using business logic. An example use case would be a bank that wants to automatically verify whether a new transaction on a customer’s credit card represents fraud based on their recent history, and deny the transaction if the charge is determined fradulent. This decision needs to be made in real-time while processing each transaction, so developers could implement this business logic in a streaming system and run it against the stream of transactions. This type of application will likely need to maintain a significant amount of state about each user to track their current spending patterns, and automatically compare this state against each new transaction. Online machine learning A close derivative of the real-time decision-making use case is online machine learning. In this scenario, you might want to train a model on a combination of streaming and historical data from multiple users. An example might be more sophisticated than the aforementioned credit card transaction use case: rather than reacting with hardcoded rules based on one customer’s behavior, the company may want to continuously update a model from all customers’ behavior and test each transaction against it. This is the most challenging use case of the bunch for stream processing systems because it requires aggregation across multiple customers, joins against static datasets, integration with machine learning libraries, and low-latency response times. Advantages of Stream Processing Now that we’ve seen some use cases for streaming, let’s crystallize some of the advantages of stream processing. For the most part, batch is much simpler to understand, troubleshoot, and write applications in for the majority of use cases. Additionally, the ability to process data in batch allows for vastly higher data processing throughput than many streaming systems. However, stream processing is essential in two cases. First, stream processing enables lower latency: when your application needs to respond quickly (on a timescale of minutes, seconds, or milliseconds), you will need a streaming system that can keep state in memory to get acceptable performance. Many of the decision making and alerting use cases we described fall into this camp. Second, stream processing can also be more efficient in updating a result than repeated batch jobs, because it automatically incrementalizes the computation. For example, if we want to compute web traffic statistics over the past 24 hours, a naively implemented batch job might scan all the data each time it runs, always processing 24 hours’ worth of data. In contrast, a streaming system can remember state from the previous computation and only count the new data. If you tell the streaming system to update your report every hour, for example, it would only need to process 1 hour’s worth of data each time (the new data since the last report). In a batch system, you would have to implement this kind of incremental computation by hand to get the same performance, resulting in a lot of extra work that the streaming system will automatically give you out of the box. Challenges of Stream Processing We discussed motivations and advantages of stream processing, but as you likely know, there’s never a free lunch. Let’s discuss some of the challenges of operating on streams. To ground this example, let’s imagine that our application receives input messages from a sensor (e.g., inside a car) that report its value at different times. We then want to search within this stream for certain values, or certain patterns of values. One specific challenge is that the input records might arrive to our application out-of-order: due to delays and retransmissions, for example, we might receive the following sequence of updates in order, where the time field shows the time when the value was actually measured: {value: {value: {value: {value: {value: 1, time: "2017-04-07T00:00:00"} 2, time: "2017-04-07T01:00:00"} 5, time: "2017-04-07T02:00:00"} 10, time: "2017-04-07T01:30:00"} 7, time: "2017-04-07T03:00:00"} In any data processing system, we can construct logic to perform some action based on receiving the single value of “5.” In a streaming system, we can also respond to this individual event quickly. However, things become more complicated if you want only to trigger some action based on a specific sequence of values received, say, 2 then 10 then 5. In the case of batch processing, this is not particularly difficult because we can simply sort all the events we have by time field to see that 10 did come between 2 and 5. However, this is harder for stream processing systems. The reason is that the streaming system is going to receive each event individually, and will need to track some state across events to remember the 2 and 5 events and realize that the 10 event was between them. The need to remember such state over the stream creates more challenges. For instance, what if you have a massive data volume (e.g., millions of sensor streams) and the state itself is massive? What if a machine in the sytem fails, losing some state? What if the load is imbalanced and one machine is slow? And how can your application signal downstream consumers when analysis for some event is “done” (e.g., the pattern 2-10-5 did not occur)? Should it wait a fixed amount of time or remember some state indefinitely? All of these challenges and others—such as making the input and the output of the system transactional —can come up when you want to deploy a streaming application. To summarize, the challenges we described in the previous paragraph and a couple of others, are as follows: Processing out-of-order data based on application timestamps (also called event time) Maintaining large amounts of state Supporting high-data throughput Processing each event exactly once despite machine failures Handling load imbalance and stragglers Responding to events at low latency Joining with external data in other storage systems Determining how to update output sinks as new events arrive Writing data transactionally to output systems Updating your application’s business logic at runtime Each of these topics are an active area of research and development in large-scale streaming systems. To understand how different streaming systems have tackled these challenges, we describe a few of the most common design concepts you will see across them. Stream Processing Design Points To support the stream processing challenges we described, including high throughput, low latency, and out-of-order data, there are multiple ways to design a streaming system. We describe the most common design options here, before describing Spark’s choices in the next section. Record-at-a-Time Versus Declarative APIs The simplest way to design a streaming API would be to just pass each event to the application and let it react using custom code. This is the approach that many early streaming systems, such as Apache Storm, implemented, and it has an important place when applications need full control over the processing of data. Streaming that provide this kind of record-at-a-time API just give the user a collection of “plumbing” to connect together into an application. However, the downside of these systems is that most of the complicating factors we described earlier, such as maintaining state, are solely governed by the application. For example, with a record-at-a-time API, you are responsible for tracking state over longer time periods, dropping it after some time to clear up space, and responding differently to duplicate events after a failure. Programming these systems correctly can be quite challenging. At its core, low-level APIs require deep expertise to be develop and maintain. As a result, many newer streaming systems provide declarative APIs, where your application specifies what to compute but not how to compute it in response to each new event and how to recover from failure. Spark’s original DStreams API, for example, offered functional API based on operations like map, reduce and filter on streams. Internally, the DStream API automatically tracked how much data each operator had processed, saved any relevant state reliably, and recovered the computation from failure when needed. Systems such as Google Dataflow and Apache Kafka Streams provide similar, functional APIs. Spark’s Structured Streaming actually takes this concept even further, switching from functional operations to relational (SQL-like) ones that enable even richer automatic optimization of the execution without programming effort. Event Time Versus Processing Time For the systems with declarative APIs, a second concern is whether the system natively supports event time. Event time is the idea of processing data based on timestamps inserted into each record at the source, as opposed to the time when the record is received at the streaming application (which is called processing time). In particular, when using event time, records may arrive to the system out of order (e.g., if they traveled back on different network paths), and different sources may also be out of sync with each other (some records may arrive later than other records for the same event time). If your application collects data from remote sources that may be delayed, such as mobile phones or IoT devices, event-time processing is crucial: without it, you will miss important patterns when some data is late. In contrast, if your application only processes local events (e.g., ones generated in the same datacenter), you may not need sophisticated event-time processing. When using event-time, several issues become common concerns across applications, including tracking state in a manner that allows the system to incorporate late events, and determining when it is safe to output a result for a given time window in event time (i.e., when the system is likely to have received all the input up to that point). Because of this, many declarative systems, including Structured Streaming, have “native” support for event time integrated into all their APIs, so that these concerns can be handled automatically across your whole program. Continuous Versus Micro-Batch Execution The final design decision you will often see come up is about continuous versus micro-batch execution. In continuous processing-based systems, each node in the system is continually listening to messages from other nodes and outputting new updates to its child nodes. For example, suppose that your application implements a map-reduce computation over several input streams. In a continuous processing system, each of the nodes implementing map would read records one by one from an input source, compute its function on them, and send them to the appropriate reducer. The reducer would then update its state whenever it gets a new record. The key idea is that this happens on each individual record, as illustrated in Figure 20-1. Figure 20-1. Continuous processing Continuous processing has the advantage of offering the lowest possible latency when the total input rate is relatively low, because each node responds immediately to a new message. However, continuous processing systems generally have lower maximum throughput, because they incur a significant amount of overhead per-record (e.g., calling the operating system to send a packet to a downstream node). In addition, continous systems generally have a fixed topology of operators that cannot be moved at runtime without stopping the whole system, which can introduce load balancing issues. In contrast, micro-batch systems wait to accumulate small batches of input data (say, 500 ms’ worth), then process each batch in parallel using a distributed collection of tasks, similar to the execution of a batch job in Spark. Micro-batch systems can often achieve high throughput per node because they leverage the same optimizations as batch systems (e.g., vectorized processing), and do not incur any extra per-record overhead, as illustrated in Figure 20-2. Figure 20-2. Micro-batch Thus, they need fewer nodes to process the same rate of data. Micro-batch systems can also use dynamic load balancing techniques to handle changing workloads (e.g., increasing or decreasing the number of tasks). The downside, however, is a higher base latency due to waiting to accumulate a micro-batch. In practice, the streaming applications that are large-scale enough to need to distribute their computation tend to prioritize throughput, so Spark has traditionally implemented micro-batch processing. In Structured Streaming, however, there is an active development effort to also support a continuous processing mode beneath the same API. When choosing between these two execution modes, the main factors you should keep in mind are your desired latency and total cost of operation (TCO). Micro-batch systems can comfortably deliver latencies from 100 ms to a second, depending on the application. Within this regime, they will generally require fewer nodes to achieve the same throughput, and hence lower operational cost (including lower maintenance cost due to less frequent node failures). For much lower latencies, you should consider a continuous processing system, or using a micro-batch system in conjunction with a fast serving layer to provide low-latency queries (e.g., outputting data into MySQL or Apache Cassandra, where it can be served to clients in milliseconds). Spark’s Streaming APIs We covered some high-level design approaches to stream processing, but thus far we have not discussed Spark’s APIs in detail. Spark includes two streaming APIs, as we discussed at the beginning of this chapter. The earlier DStream API in Spark Streaming is purely micro-batch oriented. It has a declarative (functional-based) API but no support for event time. The newer Structured Streaming API adds higher-level optimizations, event time, and support for continuous processing. The DStream API Spark’s original DStream API has been used broadly for stream processing since its first release in 2012. For example, DStreams was the most widely used processing engine in Datanami’s 2016 survey. Many companies use and operate Spark Streaming at scale in production today due to its high-level API interface and simple exactly-once semantics. Interactions with RDD code, such as joins with static data, are also natively supported in Spark Streaming. Operating Spark Streaming isn’t much more difficult than operating a normal Spark cluster. However, the DStreams API has several limitations. First, it is based purely on Java/Python objects and functions, as opposed to the richer concept of structured tables in DataFrames and Datasets. This limits the engine’s opportunity to perform optimizations. Second, the API is purely based on processing time—to handle event-time operations, applications need to implement them on their own. Finally, DStreams can only operate in a micro-batch fashion, and exposes the duration of micro-batches in some parts of its API, making it difficult to support alternative execution modes. Structured Streaming Structured Streaming is a higher-level streaming API built from the ground up on Spark’s Structured APIs. It is available in all the environments where structured processing runs, including Scala, Java, Python, R, and SQL. Like DStreams, it is a declarative API based on highlevel operations, but by building on the structured data model introduced in the previous part of the book, Structured Streaming can perform more types of optimizations automatically. However, unlike DStreams, Structured Streaming has native support for event time data (all of its the windowing operators automatically support it). As of Apache Spark 2.2, the system only runs in a micro-batch model, but the Spark team at Databricks has announced an effort called Continuous Processing to add a continuous execution mode. This should become an option for users in Spark 2.3. More fundamentally, beyond simplifying stream processing, Structured Streaming is also designed to make it easy to build end-to-end continuous applications using Apache Spark that combine streaming, batch, and interactive queries. For example, Structured Streaming does not use a separate API from DataFrames: you simply write a normal DataFrame (or SQL) computation and launch it on a stream. Structured Streaming will automatically update the result of this computation in an incremental fashion as data arrives. This is a major help when writing end-to-end data applications: developers do not need to maintain a separate streaming version of their batch code, possibly for a different execution system, and risk having these two versions of the code fall out of sync. As another example, Structured Streaming can output data to standard sinks usable by Spark SQL, such as Parquet tables, making it easy to query your stream state from another Spark applications. In future versions of Apache Spark, we expect more and more components of the project to integrate with Structured Streaming, including online learning algorithms in MLlib. In general, Structured Streaming is meant to be an easier-to-use and higher-performance evolution of Spark Streaming’s DStream API, so we will focus solely on this new API in this book. Many of the concepts, such as building a computation out of a graph of transformations, also apply to DStreams, but we leave the exposition of that to other books. Conclusion This chapter covered the basic concepts and ideas that you’re going to need to understand stream processing. The design approaches introduced in this chapter should clarify how you can evaluate streaming systems for a given application. You should also feel comfortable understanding what trade-offs the authors of DStreams and Structured Streaming have made, and why the direct support for DataFrame programs is a big help when using Structured Streaming: there is no need to duplicate your application logic. In the upcoming chapters, we’ll dive right into Structured Streaming to understand how to use it. Chapter 21. Structured Streaming Basics Now that we have covered a brief overview of stream processing, let’s dive right into Structured Streaming. In this chapter, we will, again, state some of the key concepts behind Structured Streaming and then apply them with some code examples that show how easy the system is to use. Structured Streaming Basics Structured Streaming, as we discussed at the end of Chapter 20, is a stream processing framework built on the Spark SQL engine. Rather than introducing a separate API, Structured Streaming uses the existing structured APIs in Spark (DataFrames, Datasets, and SQL), meaning that all the operations you are familiar with there are supported. Users express a streaming computation in the same way they’d write a batch computation on static data. Upon specifying this, and specifying a streaming destination, the Structured Streaming engine will take care of running your query incrementally and continuously as new data arrives into the system. These logical instructions for the computation are then executed using the same Catalyst engine discussed in Part II of this book, including query optimization, code generation, etc. Beyond the core structured processing engine, Structured Streaming includes a number of features specifically for streaming. For instance, Structured Streaming ensures end-to-end, exactly-once processing as well as fault-tolerance through checkpointing and write-ahead logs. The main idea behind Structured Streaming is to treat a stream of data as a table to which data is continuously appended. The job then periodically checks for new input data, process it, updates some internal state located in a state store if needed, and updates its result. A cornerstone of the API is that you should not have to change your query’s code when doing batch or stream processing—you should have to specify only whether to run that query in a batch or streaming fashion. Internally, Structured Streaming will automatically figure out how to “incrementalize” your query, i.e., update its result efficiently whenever new data arrives, and will run it in a faulttolerant fashion. Figure 21-1. Structured streaming input In simplest terms, Structured Streaming is “your DataFrame, but streaming.” This makes it very easy to get started using streaming applications. You probably already have the code for them! There are some limits to the types of queries Structured Streaming will be able to run, however, as well as some new concepts you have to think about that are specific to streaming, such as event-time and out-of-order data. We will discuss these in this and the following chapters. Finally, by integrating with the rest of Spark, Structured Streaming enables users to build what we call continuous applications. A continous application is an end-to-end application that reacts to data in real time by combining a variety of tools: streaming jobs, batch jobs, joins between streaming and offline data, and interactive ad-hoc queries. Because most streaming jobs today are deployed within the context of a larger continuous application, the Spark developers sought to make it easy to specify the whole application in one framework and get consistent results across these different portions of it. For example, you can use Structured Streaming to continuously update a table that users query interactively with Spark SQL, serve a machine learning model trained by MLlib, or join streams with offline data in any of Spark’s data sources —applications that would be much more complex to build using a mix of different tools. Core Concepts Now that we introduced the high-level idea, let’s cover some of the important concepts in a Structured Streaming job. One thing you will hopefully find is that there aren’t many. That’s because Structured Streaming is designed to be simple. Read some other big data streaming books and you’ll notice that they begin by introducing terminology like distributed stream processing topologies for skewed data reducers (a caricature, but accurate) and other complex verbiage. Spark’s goal is to handle these concerns automatically and give users a simple way to run any Spark computation on a stream. Transformations and Actions Structured Streaming maintains the same concept of transformations and actions that we have seen throughout this book. The transformations available in Structured Streaming are, with a few restrictions, the exact same transformations that we saw in Part II. The restrictions usually involve some types of queries that the engine cannot incrementalize yet, although some of the limitations are being lifted in new versions of Spark. There is generally only one action available in Structured Streaming: that of starting a stream, which will then run continuously and output results. Input Sources Structured Streaming supports several input sources for reading in a streaming fashion. As of Spark 2.2, the supported input sources are as follows: Apache Kafka 0.10 Files on a distributed file system like HDFS or S3 (Spark will continuously read new files in a directory) A socket source for testing We discuss these in depth later in this chapter, but it’s worth mentioning that the authors of Spark are working on a stable source API so that you can build your own streaming connectors. Sinks Just as sources allow you to get data into Structured Streaming, sinks specify the destination for the result set of that stream. Sinks and the execution engine are also responsible for reliably tracking the exact progress of data processing. Here are the supported output sinks as of Spark 2.2: Apache Kafka 0.10 Almost any file format A foreach sink for running arbitary computation on the output records A console sink for testing A memory sink for debugging We discuss these in more detail later in the chapter when we discuss sources. Output Modes Defining a sink for our Structured Streaming job is only half of the story. We also need to define how we want Spark to write data to that sink. For instance, do we only want to append new information? Do we want to update rows as we receive more information about them over time (e.g., updating the click count for a given web page)? Do we want to completely overwrite the result set every single time (i.e. always write a file with the complete click counts for all pages)? To do this, we define an output mode, similar to how we define output modes in the static Structured APIs. The supported output modes are as follows: Append (only add new records to the output sink) Update (update changed records in place) Complete (rewrite the full output) One important detail is that certain queries, and certain sinks, only support certain output modes, as we will discuss later in the book. For example, suppose that your job is just performing a map on a stream. The output data will grow indefinitely as new records arrive, so it would not make sense to use Complete mode, which requires writing all the data to a new file at once. In contrast, if you are doing an aggregation into a limited number of keys, Complete and Update modes would make sense, but Append would not, because the values of some keys’ need to be updated over time. Triggers Whereas output modes define how data is output, triggers define when data is output—that is, when Structured Streaming should check for new input data and update its result. By default, Structured Streaming will look for new input records as soon as it has finished processing the last group of input data, giving the lowest latency possible for new results. However, this behavior can lead to writing many small output files when the sink is a set of files. Thus, Spark also supports triggers based on processing time (only look for new data at a fixed interval). In the future, other types of triggers may also be supported. Event-Time Processing Structured Streaming also has support for event-time processing (i.e., processing data based on timestamps included in the record that may arrive out of order). There are two key ideas that you will need to understand here for the moment; we will talk about both of these in much more depth in the next chapter, so don’t worry if you’re not perfectly clear on them at this point. Event-time data Event-time means time fields that are embedded in your data. This means that rather than processing data according to the time it reaches your system, you process it according to the time that it was generated, even if records arrive out of order at the streaming application due to slow uploads or network delays. Expressing event-time processing is simple in Structured Streaming. Because the system views the input data as a table, the event time is just another field in that table, and your application can do grouping, aggregation, and windowing using standard SQL operators. However, under the hood, Structured Streaming can take some special actions when it knows that one of your columns is an event-time field, including optimizing query execution or determining when it is safe to forget state about a time window. Many of these actions can be controlled using watermarks. Watermarks Watermarks are a feature of streaming systems that allow you to specify how late they expect to see data in event time. For example, in an application that processes logs from mobile devices, one might expect logs to be up to 30 minutes late due to upload delays. Systems that support event time, including Structured Streaming, usually allow setting watermarks to limit how long they need to remember old data. Watermarks can also be used to control when to output a result for a particular event time window (e.g., waiting until the watermark for it has passed). Structured Streaming in Action Let’s get to an applied example of how you might use Structured Streaming. For our examples, we’re going to be working with the Heterogeneity Human Activity Recognition Dataset. The data consists of smartphone and smartwatch sensor readings from a variety of devices— specifically, the accelerometer and gyroscope, sampled at the highest possible frequency supported by the devices. Readings from these sensors were recorded while users performed activities like biking, sitting, standing, walking, and so on. There are several different smartphones and smartwatches used, and nine total users. You can download the data here, in the activity data folder. TIP This Dataset is fairly large. If it’s too large for your machine, you can remove some of the files and it will work just fine. Let’s read in the static version of the dataset as a DataFrame: // in Scala val static = spark.read.json("/data/activity-data/") val dataSchema = static.schema # in Python static = spark.read.json("/data/activity-data/") dataSchema = static.schema Here’s the schema: root |-- Arrival_Time: long (nullable = true) |-- Creation_Time: long (nullable = true) |-|-|-|-|-|-|-|-|-- Device: string (nullable = true) Index: long (nullable = true) Model: string (nullable = true) User: string (nullable = true) _corrupt_record: string (nullable = true) gt: string (nullable = true) x: double (nullable = true) y: double (nullable = true) z: double (nullable = true) Here’s a sample of the DataFrame: +-------------+------------------+--------+-----+------+----+--------+-----+----| Arrival_Time| Creation_Time| Device|Index| Model|User|_c...ord|. gt| x |1424696634224|142469663222623685|nexus4_1| 62|nexus4| a| null|stand|-0... ... |1424696660715|142469665872381726|nexus4_1| 2342|nexus4| a| null|stand|-0... +-------------+------------------+--------+-----+------+----+--------+-----+----- You can see in the preceding example, which includes a number of timestamp columns, models, user, and device information. The gt field specifies what activity the user was doing at that time. Next, let’s create a streaming version of the same Dataset, which will read each input file in the dataset one by one as if it was a stream. Streaming DataFrames are largely the same as static DataFrames. We create them within Spark applications and then perform transformations on them to get our data into the correct format. Basically, all of the transformations that are available in the static Structured APIs apply to Streaming DataFrames. However, one small difference is that Structured Streaming does not let you perform schema inference without explicitly enabling it. You can enable schema inference for this by setting the configuration spark.sql.streaming.schemaInference to true. Given that fact, we will read the schema from one file (that we know has a valid schema) and pass the dataSchema object from our static DataFrame to our streaming DataFrame. As mentioned, you should avoid doing this in a production scenario where your data may (accidentally) change out from under you: // in Scala val streaming = spark.readStream.schema(dataSchema) .option("maxFilesPerTrigger", 1).json("/data/activity-data") # in Python streaming = spark.readStream.schema(dataSchema).option("maxFilesPerTrigger", 1)\ .json("/data/activity-data") NOTE We discuss maxFilesPerTrigger a little later on in this chapter but essentially it allows you to control how quickly Spark will read all of the files in the folder. By specifying this value lower, we’re artificially limiting the flow of the stream to one file per trigger. This helps us demonstrate how Structured Streaming runs incrementally in our example, but probably isn’t something you’d use in production. Just like with other Spark APIs, streaming DataFrame creation and execution is lazy. In particular, we can now specify transformations on our streaming DataFrame before finally calling an action to start the stream. In this case, we’ll show one simple transformation—we will group and count data by the gt column, which is the activity being performed by the user at that point in time: // in Scala val activityCounts = streaming.groupBy("gt").count() # in Python activityCounts = streaming.groupBy("gt").count() Because this code is being written in local mode on a small machine, we are going to set the shuffle partitions to a small value to avoid creating too many shuffle partitions: spark.conf.set("spark.sql.shuffle.partitions", 5) Now that we set up our transformation, we need only to specify our action to start the query. As mentioned previously in the chapter, we will specify an output destination, or output sink for our result of this query. For this basic example, we are going to write to a memory sink which keeps an in-memory table of the results. In the process of specifying this sink, we’re going to need to define how Spark will output that data. In this example, we use the complete output mode. This mode rewrites all of the keys along with their counts after every trigger: // in Scala val activityQuery = activityCounts.writeStream.queryName("activity_counts") .format("memory").outputMode("complete") .start() # in Python activityQuery = activityCounts.writeStream.queryName("activity_counts")\ .format("memory").outputMode("complete")\ .start() We are now writing out our stream! You’ll notice that we set a unique query name to represent this stream, in this case activity_counts. We specified our format as an in-memory table and we set the output mode. When we run the preceding code, we also want to include the following line: activityQuery.awaitTermination() After this code is executed, the streaming computation will have started in the background. The query object is a handle to that active streaming query, and we must specify that we would like to wait for the termination of the query using activityQuery.awaitTermination() to prevent the driver process from exiting while the query is active. We will omit this from our future parts of the book for readability, but it must be included in your production applications; otherwise, your stream won’t be able to run. Spark lists this stream, and other active ones, under the active streams in our SparkSession. We can see a list of those streams by running the following: spark.streams.active Spark also assigns each stream a UUID, so if need be you could iterate through the list of running streams and select the above one. In this case, we assigned it to a variable, so that’s not necessary. Now that this stream is running, we can experiment with the results by querying the in-memory table it is maintaining of the current output of our streaming aggregation. This table will be called activity_counts, the same as the stream. To see the current data in this output table, we simply need to query it! We’ll do this in a simple loop that will print the results of the streaming query every second: // in Scala for( i <- 1 to 5 ) { spark.sql("SELECT * FROM activity_counts").show() Thread.sleep(1000) } # in Python from time import sleep for x in range(5): spark.sql("SELECT * FROM activity_counts").show() sleep(1) As the preceding queries run, you should see the counts for each activity change over time. For instance, the first show call displays the following result (because we queried it while the stream was reading the first file): +---+-----+ | gt|count| +---+-----+ +---+-----+ The previous show call shows the following result—note that the result will probably vary when you’re running this code personally because you will likely start it at a different time: +----------+-----+ | gt|count| +----------+-----+ | sit| 8207| ... | null| 6966| | bike| 7199| +----------+-----+ With this simple example, the power of Structured Streaming should become clear. You can take the same operations that you use in batch and run them on a stream of data with very few code changes (essentially just specifying that it’s a stream). The rest of this chapter touches on some of the details about the various manipulations, sources, and sinks that you can use with Structured Streaming. Transformations on Streams Streaming transformations, as we mentioned, include almost all static DataFrame transformations that you already saw in Part II. All select, filter, and simple transformations are supported, as are all DataFrame functions and individual column manipulations. The limitations arise on transformations that do not make sense in context of streaming data. For example, as of Apache Spark 2.2, users cannot sort streams that are not aggregated, and cannot perform multiple levels of aggregation without using Stateful Processing (covered in the next chater). These limitations may be lifted as Structured Streaming continues to develop, so we encourage you to check the documentation of your version of Spark for updates. Selections and Filtering All select and filter transformations are supported in Structured Streaming, as are all DataFrame functions and individual column manipulations. We show a simple example using selections and filtering below. In this case, because we are not updating any keys over time, we will use the Append output mode, so that new results are appended to the output table: // in Scala import org.apache.spark.sql.functions.expr val simpleTransform = streaming.withColumn("stairs", expr("gt like '%stairs%'")) .where("stairs") .where("gt is not null") .select("gt", "model", "arrival_time", "creation_time") .writeStream .queryName("simple_transform") .format("memory") .outputMode("append") .start() # in Python from pyspark.sql.functions import expr simpleTransform = streaming.withColumn("stairs", expr("gt like '%stairs%'"))\ .where("stairs")\ .where("gt is not null")\ .select("gt", "model", "arrival_time", "creation_time")\ .writeStream\ .queryName("simple_transform")\ .format("memory")\ .outputMode("append")\ .start() Aggregations Structured Streaming has excellent support for aggregations. You can specify arbitrary aggregations, as you saw in the Structured APIs. For example, you can use a more exotic aggregation, like a cube, on the phone model and activity and the average x, y, z accelerations of our sensor (jump back to Chapter 7 in order to see potential aggregations that you can run on your stream): // in Scala val deviceModelStats = streaming.cube("gt", "model").avg() .drop("avg(Arrival_time)") .drop("avg(Creation_Time)") .drop("avg(Index)") .writeStream.queryName("device_counts").format("memory").outputMode("complete") .start() # in Python deviceModelStats = streaming.cube("gt", "model").avg()\ .drop("avg(Arrival_time)")\ .drop("avg(Creation_Time)")\ .drop("avg(Index)")\ .writeStream.queryName("device_counts").format("memory")\ .outputMode("complete")\ .start() Querying that table allows us to see the results: SELECT * FROM device_counts +----------+------+------------------+--------------------+--------------------+ | gt| model| avg(x)| avg(y)| avg(z)| +----------+------+------------------+--------------------+--------------------+ | sit| null|-3.682775300344...|1.242033094787975...|-4.22021191297611...| | stand| null|-4.415368069618...|-5.30657295890281...|2.264837548081631...| ... | walk|nexus4|-0.007342235359...|0.004341030525168...|-6.01620400184307...| |stairsdown|nexus4|0.0309175199508...|-0.02869185568293...| 0.11661923308518365| ... +----------+------+------------------+--------------------+--------------------+ In addition to these aggregations on raw columns in the dataset, Structured Streaming has special support for columns that represent event time, including watermark support and windowing. We will discuss these in more detail in Chapter 22. NOTE As of Spark 2.2, the one limitation of aggregations is that multiple “chained” aggregations (aggregations on streaming aggregations) are not supported at this time. However, you can achieve this by writing out to an intermediate sink of data, like Kafka or a file sink. This will change in the future as the Structured Streaming community adds this functionality. Joins As of Apache Spark 2.2, Structured Streaming supports joining streaming DataFrames to static DataFrames. Spark 2.3 will add the ability to join multiple streams together. You can do multiple column joins and supplement streaming data with that from static data sources: // in Scala val historicalAgg = static.groupBy("gt", "model").avg() val deviceModelStats = streaming.drop("Arrival_Time", "Creation_Time", "Index") .cube("gt", "model").avg() .join(historicalAgg, Seq("gt", "model")) .writeStream.queryName("device_counts").format("memory").outputMode("complete") .start() # in Python historicalAgg = static.groupBy("gt", "model").avg() deviceModelStats = streaming.drop("Arrival_Time", "Creation_Time", "Index")\ .cube("gt", "model").avg()\ .join(historicalAgg, ["gt", "model"])\ .writeStream.queryName("device_counts").format("memory")\ .outputMode("complete")\ .start() In Spark 2.2, full outer joins, left joins with the stream on the right side, and right joins with the stream on the left are not supported. Structured Streaming also does not yet support stream-tostream joins, but this is also a feature under active development. Input and Output This section dives deeper into the details of how sources, sinks, and output modes work in Structured Streaming. Specifically, we discuss how, when, and where data flows into and out of the system. As of this writing, Structured Streaming supports several sources and sinks, including Apache Kafka, files, and several sources and sinks for testing and debugging. More sources may be added over time, so be sure to check the documentation for the most up-to-date information. We discuss the source and sink for a particular storage system together in this chapter, but in reality you can mix and match them (e.g., use a Kafka input source with a file sink). Where Data Is Read and Written (Sources and Sinks) Structured Streaming supports several production sources and sinks (files and Apache Kafka), as well as some debugging tools like the memory table sink. We mentioned these at the beginning of the chapter, but now let’s cover the details of each one. File source and sink Probably the simplest source you can think of is the simple file source. It’s easy to reason about and understand. While essentially any file source should work, the ones that we see in practice are Parquet, text, JSON, and CSV. The only difference between using the file source/sink and Spark’s static file source is that with streaming, we can control the number of files that we read in during each trigger via the maxFilesPerTrigger option that we saw earlier. Keep in mind that any files you add into an input directory for a streaming job need to appear in it atomically. Otherwise, Spark will process partially written files before you have finished. On file systems that show partial writes, such as local files or HDFS, this is best done by writing the file in an external directory and moving it into the input directory when finished. On Amazon S3, objects normally only appear once fully written. Kafka source and sink Apache Kafka is a distributed publish-and-subscribe system for streams of data. Kafka lets you publish and subscribe to streams of records like you might do with a message queue—these are stored as streams of records in a fault-tolerant way. Think of Kafka like a distributed buffer. Kafka lets you store streams of records in categories that are referred to as topics. Each record in Kafka consists of a key, a value, and a timestamp. Topics consist of immutable sequences of records for which the position of a record in a sequence is called an offset. Reading data is called subscribing to a topic and writing data is as simple as publishing to a topic. Spark allows you to read from Kafka with both batch and streaming DataFrames. As of Spark 2.2, Structured Streaming supports Kafka version 0.10. This too is likely to expand in the future, so be sure to check the documentation for more information about the Kafka versions available. There are only a few options that you need to specify when you read from Kafka. Reading from the Kafka Source To read, you first need to choose one of the following options: assign, subscribe, or subscribePattern. Only one of these can be present as an option when you go to read from Kafka. Assign is a fine-grained way of specifying not just the topic but also the topic partitions from which you would like to read. This is specified as a JSON string {"topicA": [0,1],"topicB":[2,4]}. subscribe and subscribePattern are ways of subscribing to one or more topics either by specifying a list of topics (in the former) or via a pattern (via the latter). Second, you will need to specify the kafka.bootstrap.servers that Kafka provides to connect to the service. After you have specified your options, you have several other options to specify: startingOffsets and endingOffsets The start point when a query is started, either earliest, which is from the earliest offsets; latest, which is just from the latest offsets; or a JSON string specifying a starting offset for each TopicPartition. In the JSON, -2 as an offset can be used to refer to earliest, -1 to latest. For example, the JSON specification could be {"topicA": {"0":23,"1":-1},"topicB":{"0":-2}}. This applies only when a new Streaming query is started, and that resuming will always pick up from where the query left off. Newly discovered partitions during a query will start at earliest. The ending offsets for a given query. failOnDataLoss Whether to fail the query when it’s possible that data is lost (e.g., topics are deleted, or offsets are out of range). This might be a false alarm. You can disable it when it doesn’t work as you expected. The default is true. maxOffsetsPerTrigger The total number of offsets to read in a given trigger. There are also options for setting Kafka consumer timeouts, fetch retries, and intervals. To read from Kafka, do the following in Structured Streaming: // in Scala // Subscribe to 1 topic val ds1 = spark.readStream.format("kafka") .option("kafka.bootstrap.servers", "host1:port1,host2:port2") .option("subscribe", "topic1") .load() // Subscribe to multiple topics val ds2 = spark.readStream.format("kafka") .option("kafka.bootstrap.servers", "host1:port1,host2:port2") .option("subscribe", "topic1,topic2") .load() // Subscribe to a pattern of topics val ds3 = spark.readStream.format("kafka") .option("kafka.bootstrap.servers", "host1:port1,host2:port2") .option("subscribePattern", "topic.*") .load() Python is quite similar: # in Python # Subscribe to 1 topic df1 = spark.readStream.format("kafka")\ .option("kafka.bootstrap.servers", "host1:port1,host2:port2")\ .option("subscribe", "topic1")\ .load() # Subscribe to multiple topics df2 = spark.readStream.format("kafka")\ .option("kafka.bootstrap.servers", "host1:port1,host2:port2")\ .option("subscribe", "topic1,topic2")\ .load() # Subscribe to a pattern df3 = spark.readStream.format("kafka")\ .option("kafka.bootstrap.servers", "host1:port1,host2:port2")\ .option("subscribePattern", "topic.*")\ .load() Each row in the source will have the following schema: key: binary value: binary topic: string partition: int offset: long timestamp: long Each message in Kafka is likely to be serialized in some way. Using native Spark functions in the Structured APIs, or a User-Defined Function (UDF), you can parse the message into a more structured format analysis. A common pattern is to use JSON or Avro to read and write to Kafka. Writing to the Kafka Sink Writing to Kafka queries is largely the same as reading from them except for fewer parameters. You’ll still need to specify the Kafka bootstrap servers, but the only other option you will need to supply is either a column with the topic specification or supply that as an option. For example, the following writes are equivalent: // in Scala ds1.selectExpr("topic", "CAST(key AS STRING)", "CAST(value AS STRING)") .writeStream.format("kafka") .option("checkpointLocation", "/to/HDFS-compatible/dir") .option("kafka.bootstrap.servers", "host1:port1,host2:port2") .start() ds1.selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)") .writeStream.format("kafka") .option("kafka.bootstrap.servers", "host1:port1,host2:port2") .option("checkpointLocation", "/to/HDFS-compatible/dir")\ .option("topic", "topic1") .start() # in Python df1.selectExpr("topic", "CAST(key AS STRING)", "CAST(value AS STRING)")\ .writeStream\ .format("kafka")\ .option("kafka.bootstrap.servers", "host1:port1,host2:port2")\ .option("checkpointLocation", "/to/HDFS-compatible/dir")\ .start() df1.selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)")\ .writeStream\ .format("kafka")\ .option("kafka.bootstrap.servers", "host1:port1,host2:port2")\ .option("checkpointLocation", "/to/HDFS-compatible/dir")\ .option("topic", "topic1")\ .start() Foreach sink The foreach sink is akin to foreachPartitions in the Dataset API. This operation allows arbitrary operations to be computed on a per-partition basis, in parallel. This is available in Scala and Java initially, but it will likely be ported to other languages in the future. To use the foreach sink, you must implement the ForeachWriter interface, which is available in the Scala/Java documents, which contains three methods: open, process, and close. The relevant methods will be called whenever there is a sequence of rows generated as output after a trigger. Here are some important details: The writer must be Serializable, as it were a UDF or a Dataset map function. The three methods (open, process, close) will be called on each executor. The writer must do all its initialization, like opening connections or starting transactions only in the open method. A common source of errors is that if initialization occurs outside of the open method (say in the class that you’re using), that happens on the driver instead of the executor. Because the Foreach sink runs arbitrary user code, one key issue you must consider when using it is fault tolerance. If Structured Streaming asked your sink to write some data, but then crashed, it cannot know whether your original write succeeded. Therefore, the API provides some additional parameters to help you achieve exactly-once processing. First, the open call on your ForeachWriter receives two parameters that uniquely identify the set of rows that need to be acted on. The version parameter is a monotonically increasing ID that increases on a per-trigger basis, and partitionId is the ID of the partition of the output in your task. Your open method should return whether to process this set of rows. If you track your sink’s output externally and see that this set of rows was already output (e.g., you wrote the last version and partitionId written in your storage system), you can return false from open to skip processing this set of rows. Otherwise, return true. Your ForeachWriter will be opened again for each trigger’s worth of data to write. Next, the process method will be called for each record in the data, assuming your open method returned true. This is fairly straightforward—just process or write your data. Finally, whenever open is called, the close method is also called (unless the node crashed before that), regardless of whether open returned true. If Spark witnessed an error during processing, the close method receives that error. It is your responsibility to clean up any open resources during close. Together, the ForeachWriter interface effectively lets you implement your own sink, including your own logic for tracking which triggers’ data has been written or safely overwriting it on failures. We show an example of passing a ForeachWriter below: //in Scala datasetOfString.write.foreach(new ForeachWriter[String] { def open(partitionId: Long, version: Long): Boolean = { // open a database connection } def process(record: String) = { // write string to connection } def close(errorOrNull: Throwable): Unit = { // close the connection } }) Sources and sinks for testing Spark also includes several test sources and sinks that you can use for prototyping or debugging your streaming queries (these should be used only during development and not in production scenarios, because they do not provide end-to-end fault tolerance for your application): Socket source The socket source allows you to send data to your Streams via TCP sockets. To start one, specify a host and port to read data from. Spark will open a new TCP connection to read from that address. The socket source should not be used in production because the socket sits on the driver and does not provide end-to-end fault-tolerance guarantees. Here is a short example of setting up this source to read from localhost:9999: // in Scala val socketDF = spark.readStream.format("socket") .option("host", "localhost").option("port", 9999).load() # in Python socketDF = spark.readStream.format("socket")\ .option("host", "localhost").option("port", 9999).load() If you’d like to actually write data to this application, you will need to run a server that listens on port 9999. On Unix-like systems, you can do this using the NetCat utility, which will let you type text into the first connection that is opened to port 9999. Run the command below before starting your Spark application, then write into it: nc -lk 9999 The socket source will return a table of text strings, one per line in the input data. Console sink The console sink allows you to write out some of your streaming query to the console. This is useful for debugging but is not fault-tolerant. Writing out to the console is simple and only prints some rows of your streaming query to the console. This supports both append and complete output modes: activityCounts.format("console").write() Memory sink The memory sink is a simple source for testing your streaming system. It’s similar to the console sink except that rather than printing to the console, it collects the data to the driver and then makes the data available as an in-memory table that is available for interactive querying. This sink is not fault tolerant, and you shouldn’t use it in production, but is great for testing and querying your stream during development. This supports both append and complete output modes: // in Scala activityCounts.writeStream.format("memory").queryName("my_device_table") If you do want to output data to a table for interactive SQL queries in production, the authors recommend using the Parquet file sink on a distributed file system (e.g., S3). You can then query the data from any Spark application. How Data Is Output (Output Modes) Now that you know where your data can go, let’s discuss how the result Dataset will look when it gets there. This is what we call the output mode. As we mentioned, they’re the same concept as save modes on static DataFrames. There are three modes supported by Structured Streaming. Let’s look at each of them. Append mode Append mode is the default behavior and the simplest to understand. When new rows are added to the result table, they will be output to the sink based on the trigger (explained next) that you specify. This mode ensures that each row is output once (and only once), assuming that you have a fault-tolerant sink. When you use append mode with event-time and watermarks (covered in Chapter 22), only the final result will output to the sink. Complete mode Complete mode will output the entire state of the result table to your output sink. This is useful when you’re working with some stateful data for which all rows are expected to change over time or the sink you are writing does not support row-level updates. Think of it like the state of a stream at the time the previous batch had run. Update mode Update mode is similar to complete mode except that only the rows that are different from the previous write are written out to the sink. Naturally, your sink must support row-level updates to support this mode. If the query doesn’t contain aggregations, this is equivalent to append mode. When can you use each mode? Structured Streaming limits your use of each mode to queries where it makes sense. For example, if your query just does a map operation, Structured Streaming will not allow complete mode, because this would require it to remember all input records since the start of the job and rewrite the whole output table. This requirement is bound to get prohibitively expensive as the job runs. We will discuss when each mode is supported in more detail in the next chapter, once we also cover event-time processing and watermarks. If your chosen mode is not available, Spark Streaming will throw an exception when you start your stream. Here’s a handy table from the documentation that lays all of this out. Keep in mind that this will change in the future, so you’ll want to check the documentation for the most up-to-date version. Table 21-1 shows when you can use each output mode. Table 21-1. Structured streaming output modes as of Spark 2.2 Query Type Supported Query type Output Notes (continued) Modes Aggregation Append, on event-time Queries with aggregation Update, with Complete watermark Append mode uses watermark to drop old aggregation state. This means that as new rows are brought into the table, Spark will only keep around rows that are below the “watermark”. Update mode also uses the watermark to remove old aggregation state. By definition, complete mode does not drop old aggregation state since this mode preserves all data in the Result Table. Other Complete, aggregations Update Since no watermark is defined (only defined in other category), old aggregation state is not dropped. Append mode is not supported as aggregates can update thus violating the semantics of this mode. Queries with mapGroupsWithState Update Append Queries with operation flatMapGroupsWithState mode Append Aggregations are allowed after flatMapGroupsWithState. Update operation mode Update Aggregations not allowed after flatMapGroupsWithState. Append, Update Complete mode not supported as it is infeasible to keep all unaggregated data in the Result Table. Other queries When Data Is Output (Triggers) To control when data is output to our sink, we set a trigger. By default, Structured Streaming will start data as soon as the previous trigger completes processing. You can use triggers to ensure that you do not overwhelm your output sink with too many updates or to try and control file sizes in the output. Currently, there is one periodic trigger type, based on processing time, as well as a “once” trigger to manually run a processing step once. More triggers will likely be added in the future. Processing time trigger For the processing time trigger, we simply specify a duration as a string (you may also use a Duration in Scala or TimeUnit in Java). We’ll show the string format below. // in Scala import org.apache.spark.sql.streaming.Trigger activityCounts.writeStream.trigger(Trigger.ProcessingTime("100 seconds")) .format("console").outputMode("complete").start() # in Python activityCounts.writeStream.trigger(processingTime='5 seconds')\ .format("console").outputMode("complete").start() The ProcessingTime trigger will wait for multiples of the given duration in order to output data. For example, with a trigger duration of one minute, the trigger will fire at 12:00, 12:01, 12:02, and so on. If a trigger time is missed because the previous processing has not yet completed, then Spark will wait until the next trigger point (i.e., the next minute), rather than firing immediately after the previous processing completes. Once trigger You can also just run a streaming job once by setting that as the trigger. This might seem like a weird case, but it’s actually extremely useful in both development and production. During development, you can test your application on just one trigger’s worth of data at a time. During production, the Once trigger can be used to run your job manually at a low rate (e.g., import new data into a summary table just occasionally). Because Structured Streaming still fully tracks all the input files processed and the state of the computation, this is easier than writing your own custom logic to track this in a batch job, and saves a lot of resources over running a continuous job 24/7: // in Scala import org.apache.spark.sql.streaming.Trigger activityCounts.writeStream.trigger(Trigger.Once()) .format("console").outputMode("complete").start() # in Python activityCounts.writeStream.trigger(once=True)\ .format("console").outputMode("complete").start() Streaming Dataset API One final thing to note about Structured Streaming is that you are not limited to just the DataFrame API for streaming. You can also use Datasets to perform the same computation but in type-safe manner. You can turn a streaming DataFrame into a Dataset the same way you did with a static one. As before, the Dataset’s elements need to be Scala case classes or Java bean classes. Other than that, the DataFrame and Dataset operators work as they did in a static setting, and will also turn into a streaming execution plan when run on a stream. Here’s an example using the same dataset that we used in Chapter 11: // in Scala case class Flight(DEST_COUNTRY_NAME: String, ORIGIN_COUNTRY_NAME: String, count: BigInt) val dataSchema = spark.read .parquet("/data/flight-data/parquet/2010-summary.parquet/") .schema val flightsDF = spark.readStream.schema(dataSchema) .parquet("/data/flight-data/parquet/2010-summary.parquet/") val flights = flightsDF.as[Flight] def originIsDestination(flight_row: Flight): Boolean = { return flight_row.ORIGIN_COUNTRY_NAME == flight_row.DEST_COUNTRY_NAME } flights.filter(flight_row => originIsDestination(flight_row)) .groupByKey(x => x.DEST_COUNTRY_NAME).count() .writeStream.queryName("device_counts").format("memory").outputMode("complete") .start() Conclusion It should be clear that Structured Streaming presents a powerful way to write streaming applications. Taking a batch job you already run and turning it into a streaming job with almost no code changes is both simple and extremely helpful from an engineering standpoint if you need to have this job interact closely with the rest of your data processing application. Chapter 22 dives into two advanced streaming-related concepts: event-time processing and stateful processing. Then, after that, Chapter 23 addresses what you need to do to run Structured Streaming in production. Chapter 22. Event-Time and Stateful Processing Chapter 21 covered the core concepts and basic APIs; this chapter dives into event-time and stateful processing. Event-time processing is a hot topic because we analyze information with respect to the time that it was created, not processed. The key idea between this style of processing is that over the lifetime of the job, Spark will maintain relevant state that it can update over the course of the job before outputting it to the sink. Let’s cover these concepts in greater detail before we begin working with code to show they work. Event Time Event time is an important topic to cover discretely because Spark’s DStream API does not support processing information with respect to event-time. At a higher level, in streamprocessing systems there are effectively two relevant times for each event: the time at which it actually occurred (event time), and the time that it was processed or reached the streamprocessing system (processing time). Event time Event time is the time that is embedded in the data itself. It is most often, though not required to be, the time that an event actually occurs. This is important to use because it provides a more robust way of comparing events against one another. The challenge here is that event data can be late or out of order. This means that the stream processing system must be able to handle out-of-order or late data. Processing time Processing time is the time at which the stream-processing system actually receives data. This is usually less important than event time because when it’s processed is largely an implementation detail. This can’t ever be out of order because it’s a property of the streaming system at a certain time (not an external system like event time). Those explanations are nice and abstract, so let’s use a more tangible example. Suppose that we have a datacenter located in San Francisco. An event occurs in two places at the same time: one in Ecuador, the other in Virginia (see Figure 22-1). Figure 22-1. Event Time Across the World Due to the location of the datacenter, the event in Virginia is likely to show up in our datacenter before the event in Ecuador. If we were to analyze this data based on processing time, it would appear that the event in Virginia occurred before the event in Ecuador: something that we know to be wrong. However, if we were to analyze the data based on event time (largely ignoring the time at which it’s processed), we would see that these events occurred at the same time. As we mentioned, the fundamental idea is that the order of the series of events in the processing system does not guarantee an ordering in event time. This can be somewhat unintuitive, but is worth reinforcing. Computer networks are unreliable. That means that events can be dropped, slowed down, repeated, or be sent without issue. Because individual events are not guaranteed to suffer one fate or the other, we must acknowledge that any number of things can happen to these events on the way from the source of the information to our stream processing system. For this reason, we need to operate on event time and look at the overall stream with reference to this information contained in the data rather than on when it arrives in the system. This means that we hope to compare events based on the time at which those events occurred. Stateful Processing The other topic we need to cover in this chapter is stateful processing. Actually, we already demonstrated this many times in Chapter 21. Stateful processing is only necessary when you need to use or update intermediate information (state) over longer periods of time (in either a microbatch or a record-at-a-time approach). This can happen when you are using event time or when you are performing an aggregation on a key, whether that involves event time or not. For the most part, when you’re performing stateful operations. Spark handles all of this complexity for you. For example, when you specify a grouping, Structured Streaming maintains and updates the information for you. You simply specify the logic. When performing a stateful operation, Spark stores the intermediate information in a state store. Spark’s current state store implementation is an in-memory state store that is made fault tolerant by storing intermediate state to the checkpoint directory. Arbitrary Stateful Processing The stateful processing capabilities described above are sufficient to solve many streaming problems. However, there are times when you need fine-grained control over what state should be stored, how it is updated, and when it should be removed, either explicitly or via a time-out. This is called arbitrary (or custom) stateful processing and Spark allows you to essentially store whatever information you like over the course of the processing of a stream. This provides immense flexibility and power and allows for some complex business logic to be handled quite easily. Just as we did before, let’s ground this with some examples: You’d like to record information about user sessions on an ecommerce site. For instance, you might want to track what pages users visit over the course of this session in order to provide recommendations in real time during their next session. Naturally, these sessions have completely arbitrary start and stop times that are unique to that user. Your company would like to report on errors in the web application but only if five events occur during a user’s session. You could do this with count-based windows that only emit a result if five events of some type occur. You’d like to deduplicate records over time. To do so, you’re going to need to keep track of every record that you see before deduplicating it. Now that we’ve explained the core concepts that we’re going to need in this chapter, let’s cover all of this with some examples that you can follow along with and explain some of the important caveats that you need to consider when processing in this manner. Event-Time Basics Let’s begin with the same dataset from the previous chapter. When working with event time, it’s just another column in our dataset, and that’s really all we need to concern ourselves with; we simply use that column, as demonstrated here: // in Scala spark.conf.set("spark.sql.shuffle.partitions", 5) val static = spark.read.json("/data/activity-data") val streaming = spark .readStream .schema(static.schema) .option("maxFilesPerTrigger", 10) .json("/data/activity-data") # in Python spark.conf.set("spark.sql.shuffle.partitions", 5) static = spark.read.json("/data/activity-data") streaming = spark\ .readStream\ .schema(static.schema)\ .option("maxFilesPerTrigger", 10)\ .json("/data/activity-data") streaming.printSchema() root |-|-|-|-|-|-|-|-|-|-- Arrival_Time: long (nullable = true) Creation_Time: long (nullable = true) Device: string (nullable = true) Index: long (nullable = true) Model: string (nullable = true) User: string (nullable = true) gt: string (nullable = true) x: double (nullable = true) y: double (nullable = true) z: double (nullable = true) In this dataset, there are two time-based columns. The Creation_Time column defines when an event was created, whereas the Arrival_Time defines when an event hit our servers somewhere upstream. We will use Creation_Time in this chapter. This example reads from a file but, as we saw in the previous chapter, it would be simple to change it to Kafka if you already have a cluster up and running. Windows on Event Time The first step in event-time analysis is to convert the timestamp column into the proper Spark SQL timestamp type. Our current column is unixtime nanoseconds (represented as a long), therefore we’re going to have to do a little manipulation to get it into the proper format: // in Scala val withEventTime = streaming.selectExpr( "*", "cast(cast(Creation_Time as double)/1000000000 as timestamp) as event_time") # in Python withEventTime = streaming\.selectExpr( "*", "cast(cast(Creation_Time as double)/1000000000 as timestamp) as event_time") We’re now prepared to do arbitrary operations on event time! Note how this experience is just like we’d do in batch operations—there’s no special API or DSL. We simply use columns, just like we might in batch, the aggregation, and we’re working with event time. Tumbling Windows The simplest operation is simply to count the number of occurrences of an event in a given window. Figure 22-2 depicts the process when performing a simple summation based on the input data and a key. Figure 22-2. Tumbling Windows We’re performing an aggregation of keys over a window of time. We update the result table (depending on the output mode) when every trigger runs, which will operate on the data received since the last trigger. In the case of our actual dataset (and Figure 22-2), we’ll do so in 10-minute windows without any overlap between them (each, and only one event can fall into one window). This will update in real time, as well, meaning that if new events were being added upstream to our system, Structured Streaming would update those counts accordingly. This is the complete output mode, Spark will output the entire result table regardless of whether we’ve seen the entire dataset: // in Scala import org.apache.spark.sql.functions.{window, col} withEventTime.groupBy(window(col("event_time"), "10 minutes")).count() .writeStream .queryName("events_per_window") .format("memory") .outputMode("complete") .start() # in Python from pyspark.sql.functions import window, col withEventTime.groupBy(window(col("event_time"), "10 minutes")).count()\ .writeStream\ .queryName("pyevents_per_window")\ .format("memory")\ .outputMode("complete")\ .start() Now we’re writing out to the in-memory sink for debugging, so we can query it with SQL after we have the stream running: spark.sql("SELECT * FROM events_per_window").printSchema() SELECT * FROM events_per_window This shows us something like the following result, depending on the amount of data processed when you had run the query: +---------------------------------------------+-----+ |window |count| +---------------------------------------------+-----+ |[2015-02-23 10:40:00.0,2015-02-23 10:50:00.0]|11035| |[2015-02-24 11:50:00.0,2015-02-24 12:00:00.0]|18854| ... |[2015-02-23 13:40:00.0,2015-02-23 13:50:00.0]|20870| |[2015-02-23 11:20:00.0,2015-02-23 11:30:00.0]|9392 | +---------------------------------------------+-----+ For reference, here’s the schema we get from the previous query: root |-- window: struct (nullable = false) | |-- start: timestamp (nullable = true) | |-- end: timestamp (nullable = true) |-- count: long (nullable = false) Notice how window is actually a struct (a complex type). Using this we can query this struct for the start and end times of a particular window. Of importance is the fact that we can also perform an aggregation on multiple columns, including the event time column. Just like we saw in the previous chapter, we can even perform these aggregations using methods like cube. While we won’t repeat the fact that we can perform the multi-key aggregation below, this does apply to any window-style aggregation (or stateful computation) we would like: // in Scala import org.apache.spark.sql.functions.{window, col} withEventTime.groupBy(window(col("event_time"), "10 minutes"), "User").count() .writeStream .queryName("events_per_window") .format("memory") .outputMode("complete") .start() # in Python from pyspark.sql.functions import window, col withEventTime.groupBy(window(col("event_time"), "10 minutes"), "User").count()\ .writeStream\ .queryName("pyevents_per_window")\ .format("memory")\ .outputMode("complete")\ .start() Sliding windows The previous example was simple counts in a given window. Another approach is that we can decouple the window from the starting time of the window. Figure 22-3 illustrates what we mean. Figure 22-3. Sliding Windows In the figure, we are running a sliding window through which we look at an hour increment, but we’d like to get the state every 10 minutes. This means that we will update the values over time and will include the last hours of data. In this example, we have 10-minute windows, starting every five minutes. Therefore each event will fall into two different windows. You can tweak this further according to your needs: // in Scala import org.apache.spark.sql.functions.{window, col} withEventTime.groupBy(window(col("event_time"), "10 minutes", "5 minutes")) .count() .writeStream .queryName("events_per_window") .format("memory") .outputMode("complete") .start() # in Python from pyspark.sql.functions import window, col withEventTime.groupBy(window(col("event_time"), "10 minutes", "5 minutes"))\ .count()\ .writeStream\ .queryName("pyevents_per_window")\ .format("memory")\ .outputMode("complete")\ .start() Naturally, we can query the in-memory table: SELECT * FROM events_per_window This query gives us the following result. Note that the starting times for each window are now in 5-minute intervals instead of 10, like we saw in the previous query: +---------------------------------------------+-----+ |window |count| +---------------------------------------------+-----+ |[2015-02-23 14:15:00.0,2015-02-23 14:25:00.0]|40375| |[2015-02-24 11:50:00.0,2015-02-24 12:00:00.0]|56549| ... |[2015-02-24 11:45:00.0,2015-02-24 11:55:00.0]|51898| |[2015-02-23 10:40:00.0,2015-02-23 10:50:00.0]|33200| +---------------------------------------------+-----+ Handling Late Data with Watermarks The preceding examples are great, but they have a flaw. We never specified how late we expect to see data. This means that Spark is going to need to store that intermediate data forever because we never specified a watermark, or a time at which we don’t expect to see any more data. This applies to all stateful processing that operates on event time. We must specify this watermark in order to age-out data in the stream (and, therefore, state) so that we don’t overwhelm the system over a long period of time. Concretely, a watermark is an amount of time following a given event or set of events after which we do not expect to see any more data from that time. We know this can happen due to delays on the network, devices that lose a connection, or any number of other issues. In the DStreams API, there was no robust way to handle late data in this way—if an event occurred at a certain time but did not make it to the processing system by the time the batch for a given window started, it would show up in other processing batches. Structured Streaming remedies this. In event time and stateful processing, a given window’s state or set of data is decoupled from a processing window. That means that as more events come in, Structured Streaming will continue to update a window with more information. Let’s return back to our event time example from the beginning of the chapter, shown now in Figure 22-4. Figure 22-4. Event Time Watermarking In this example, let’s imagine that we frequently see some amount of delay from our customers in Latin America. Therefore, we specify a watermark of 10 minutes. When doing this, we instruct Spark that any event that occurs more than 10 “event-time” minutes past a previous event should be ignored. Conversely, this also states that we expect to see every event within 10 minutes. After that, Spark should remove intermediate state and, depending on the output mode, do something with the result. As mentioned at the beginning of the chapter, we need to specify watermarks because if we did not, we’d need to keep all of our windows around forever, expecting them to be updated forever. This brings us to the core question when working with event-time: “how late do I expect to see data?” The answer to this question will be the watermark that you’ll configure for your data. Returning to our dataset, if we know that we typically see data as produced downstream in minutes but we have seen delays in events up to five hours after they occur (perhaps the user lost cell phone connectivity), we’d specify the watermark in the following way: // in Scala import org.apache.spark.sql.functions.{window, col} withEventTime .withWatermark("event_time", "5 hours") .groupBy(window(col("event_time"), "10 minutes", "5 minutes")) .count() .writeStream .queryName("events_per_window") .format("memory") .outputMode("complete") .start() # in Python from pyspark.sql.functions import window, col withEventTime\ .withWatermark("event_time", "30 minutes")\ .groupBy(window(col("event_time"), "10 minutes", "5 minutes"))\ .count()\ .writeStream\ .queryName("pyevents_per_window")\ .format("memory")\ .outputMode("complete")\ .start() It’s pretty amazing, but almost nothing changed about our query. We essentially just added another configuration. Now, Structured Streaming will wait until 30 minutes after the final timestamp of this 10-minute rolling window before it finalizes the result of that window. We can query our table and see the intermediate results because we’re using complete mode—they’ll be updated over time. In append mode, this information won’t be output until the window closes. SELECT * FROM events_per_window +---------------------------------------------+-----+ |window |count| +---------------------------------------------+-----+ |[2015-02-23 14:15:00.0,2015-02-23 14:25:00.0]|9505 | |[2015-02-24 11:50:00.0,2015-02-24 12:00:00.0]|13159| ... |[2015-02-24 11:45:00.0,2015-02-24 11:55:00.0]|12021| |[2015-02-23 10:40:00.0,2015-02-23 10:50:00.0]|7685 | +---------------------------------------------+-----+ At this point, you really know all that you need to know about handling late data. Spark does all of the heavy lifting for you. Just to reinforce the point, if you do not specify how late you think you will see data, then Spark will maintain that data in memory forever. Specifying a watermark allows it to free those objects from memory, allowing your stream to continue running for a long time. Dropping Duplicates in a Stream One of the more difficult operations in record-at-a-time systems is removing duplicates from the stream. Almost by definition, you must operate on a batch of records at a time in order to find duplicates—there’s a high coordination overhead in the processing system. Deduplication is an important tool in many applications, especially when messages might be delivered multiple times by upstream systems. A perfect example of this are Internet of Things (IoT) applications that have upstream producers generating messages in nonstable network environments, and the same message might end up being sent multiple times. Your downstream applications and aggregations should be able to assume that there is only one of each message. Essentially, Structured Streaming makes it easy to take message systems that provide at-leastonce semantics, and convert them into exactly-once by dropping duplicate messages as they come in, based on arbitrary keys. To de-duplicate data, Spark will maintain a number of user specified keys and ensure that duplicates are ignored. WARNING Just like other stateful processing applications, you need to specify a watermark to ensure that the maintained state does not grow infinitely over the course of your stream. Let’s begin the de-duplication process. The goal here will be to de-duplicate the number of events per user by removing duplicate events. Notice how you need to specify the event time column as a duplicate column along with the column you should de-duplicate. The core assumption is that duplicate events will have the same timestamp as well as identifier. In this model, rows with two different timestamps are two different records: // in Scala import org.apache.spark.sql.functions.expr withEventTime .withWatermark("event_time", "5 seconds") .dropDuplicates("User", "event_time") .groupBy("User") .count() .writeStream .queryName("deduplicated") .format("memory") .outputMode("complete") .start() # in Python from pyspark.sql.functions import expr withEventTime\ .withWatermark("event_time", "5 seconds")\ .dropDuplicates(["User", "event_time"])\ .groupBy("User")\ .count()\ .writeStream\ .queryName("pydeduplicated")\ .format("memory")\ .outputMode("complete")\ .start() The result will be similar to the following and will continue to update over time as more data is read by your stream: +----+-----+ |User|count| +----+-----+ | a| 8085| | b| 9123| | c| 7715| | g| 9167| | h| 7733| | e| 9891| | f| 9206| | d| 8124| | i| 9255| +----+-----+ Arbitrary Stateful Processing The first section if this chapter demonstrates how Spark maintains information and updates windows based on our specifications. But things differ when you have more complex concepts of windows; this is, where arbitrary stateful processing comes in. This section includes several examples of different use cases along with examples that show you how you might go about setting up your business logic. Stateful processing is available only in Scala in Spark 2.2. This will likely change in the future. When performing stateful processing, you might want to do the following: Create window based on counts of a given key Emit an alert if there is a number of events within a certain time frame Maintain user sessions of an undetermined amount of time and save those sessions to perform some analysis on later. At the end of the day, there are two things you will want to do when performing this style of processing: Map over groups in your data, operate on each group of data, and generate at most a single row for each group. The relevant API for this use case is mapGroupsWithState. Map over groups in your data, operate on each group of data, and generate one or more rows for each group. The relevant API for this use case is flatMapGroupsWithState. When we say “operate” on each group of data, that means that you can arbitrarily update each group independent of any other group of data. This means that you can define arbitrary window types that don’t conform to tumbling or sliding windows like we saw previously in the chapter. One important benefit that we get when we perform this style of processing is control over configuring time-outs on state. With windows and watermarks, it’s very simple: you simply time-out a window when the watermark passes the window start. This doesn’t apply to arbitrary stateful processing, because you manage the state based on user-defined concepts. Therefore, you need to properly time-out your state. Let’s discuss this a bit more. Time-Outs As mentioned in Chapter 21, a time-out specifies how long you should wait before timing-out some intermediate state. A time-out is a global parameter across all groups that is configured on a per-group basis. Time-outs can be either based on processing time (GroupStateTimeout.ProcessingTimeTimeout) or event time (GroupStateTimeout.EventTimeTimeout). When using time-outs, check for time-out first before processing the values. You can get this information by checking the state.hasTimedOut flag or checking whether the values iterator is empty. You need to set some state (i.e., state must be defined, not removed) for time-outs to be set. With a time-out based on processing time, you can set the time-out duration by calling GroupState.setTimeoutDuration (we’ll see code examples of this later in this section of the chapter). The time-out will occur when the clock has advanced by the set duration. Guarantees provided by this time-out with a duration of D ms are as follows: Time-out will never occur before the clock time has advanced by D ms Time-out will occur eventually when there is a trigger in the query (i.e., after D ms). So there is a no strict upper bound on when the time-out would occur. For example, the trigger interval of the query will affect when the time-out actually occurs. If there is no data in the stream (for any group) for a while, there won’t be any trigger and the timeout function call will not occur until there is data. Because the processing time time-out is based on the clock time, it is affected by the variations in the system clock. This means that time zone changes and clock skew are important variables to consider. With a time-out based on event time, the user also must specify the event-time watermark in the query using watermarks. When set, data older than the watermark is filtered out. As the developer, you can set the timestamp that the watermark should reference by setting a time-out timestamp using the GroupState.setTimeoutTimestamp(...) API. The time-out would occur when the watermark advances beyond the set timestamp. Naturally, you can control the time-out delay by either specifying longer watermarks or simply updating the time-out as you process your stream. Because you can do this in arbitrary code, you can do it on a per-group basis. The guarantee provided by this time-out is that it will never occur before the watermark has exceeded the set time-out. Similar to processing-time time-outs, there is a no strict upper bound on the delay when the timeout actually occurs. The watermark can advance only when there is data in the stream, and the event time of the data has actually advanced. NOTE We mentioned this a few moments ago, but it’s worth reinforcing. Although time-outs are important, they might not always function as you expect. For instance, as of this writing, Structured Streaming does not have asynchronous job execution, which means that Spark will not output data (or time-out data) between the time that a epoch finishes and the next one starts, because it is not processing any data at that time. Also, if a processing batch of data has no records (keep in mind this is a batch, not a group), there are no updates and there cannot be an event-time time-out. This might change in future versions. Output Modes One last “gotcha” when working with this sort of arbitrary stateful processing is the fact that not all output modes discussed in Chapter 21 are supported. This is sure to change as Spark continues to change, but, as of this writing, mapGroupsWithState supports only the update output mode, whereas flatMapGroupsWithState supports append and update. append mode means that only after the time-out (meaning the watermark has passed) will data show up in the result set. This does not happen automatically, it is your responsibility to output the proper row or rows. Please see Table 21-1 to see which output modes can be used when. mapGroupsWithState Our first example of stateful processing uses a feature called mapGroupsWithState. This is similar to a user-defined aggregation function that takes as input an update set of data and then resolves it down to a specific key with a set of values. There are several things you’re going to need to define along the way: Three class definitions: an input definition, a state definition, and optionally an output definition. A function to update the state based on a key, an iterator of events, and a previous state. A time-out parameter (as described in the time-outs section). With these objects and definitions, you can control arbitrary state by creating it, updating it over time, and removing it. Let’s begin with a example of simply updating the key based on a certain amount of state, and then move onto more complex things like sessionization. Because we’re working with sensor data, let’s find the first and last timestamp that a given user performed one of the activities in the dataset. This means that the key we will be grouping on (and mapping on) is a user and activity combination. NOTE When you use mapGroupsWithState, the output of the dream will contain only one row per key (or group) at all times. If you would like each group to have multiple outputs, you should use flatMapGroupsWithState (covered shortly). Let’s establish the input, state, and output definitions: case class InputRow(user:String, timestamp:java.sql.Timestamp, activity:String) case class UserState(user:String, var activity:String, var start:java.sql.Timestamp, var end:java.sql.Timestamp) For readability, set up the function that defines how you will update your state based on a given row: def updateUserStateWithEvent(state:UserState, input:InputRow):UserState = { if (Option(input.timestamp).isEmpty) { return state } if (state.activity == input.activity) { if (input.timestamp.after(state.end)) { state.end = input.timestamp } if (input.timestamp.before(state.start)) { state.start = input.timestamp } } else { if (input.timestamp.after(state.end)) { state.start = input.timestamp state.end = input.timestamp state.activity = input.activity } } state } Now, write the function that defines the way state is updated based on an epoch of rows: import org.apache.spark.sql.streaming.{GroupStateTimeout, OutputMode, GroupState} def updateAcrossEvents(user:String, inputs: Iterator[InputRow], oldState: GroupState[UserState]):UserState = { var state:UserState = if (oldState.exists) oldState.get else UserState(user, "", new java.sql.Timestamp(6284160000000L), new java.sql.Timestamp(6284160L) ) // we simply specify an old date that we can compare against and // immediately update based on the values in our data for (input <- inputs) { state = updateUserStateWithEvent(state, input) oldState.update(state) } state } When we have that, it’s time to start your query by passing in the relevant information. The one thing that you’re going to have to add when you specify mapGroupsWithState is whether you need to time-out a given group’s state. This just gives you a mechanism to control what should be done with state that receives no update after a certain amount of time. In this case, you want to maintain state indefinitely, so specify that Spark should not time-out. Use the update output mode so that you get updates on the user activity: import org.apache.spark.sql.streaming.GroupStateTimeout withEventTime .selectExpr("User as user", "cast(Creation_Time/1000000000 as timestamp) as timestamp", "gt as activity") .as[InputRow] .groupByKey(_.user) .mapGroupsWithState(GroupStateTimeout.NoTimeout)(updateAcrossEvents) .writeStream .queryName("events_per_window") .format("memory") .outputMode("update") .start() SELECT * FROM events_per_window order by user, start Here’s a sample of our result set: +----+--------+--------------------+--------------------+ |user|activity| start| end| +----+--------+--------------------+--------------------+ | a| bike|2015-02-23 13:30:...|2015-02-23 14:06:...| | a| bike|2015-02-23 13:30:...|2015-02-23 14:06:...| ... | d| bike|2015-02-24 13:07:...|2015-02-24 13:42:...| +----+--------+--------------------+--------------------+ An interesting aspect of our data is that the last activity performed at any given time is “bike.” This is related to how the experiment was likely run, in which they had each participant perform the same activities in order. EXAMPLE: COUNT-BASED WINDOWS Typical window operations are built from start and end times for which all events that fall in between those two points contribute to the counting or summation that you’re performing. However, there are times when instead of creating windows based on time, you’d rather create them based on a number of events regardless of state and event times, and perform some aggregation on that window of data. For example, we may want to compute a value for every 500 events received, regardless of when they are received. The next example analyzes the activity dataset from this chapter and outputs the average reading of each device periodically, creating a window based on the count of events and outputting it each time it has accumulated 500 events for that device. You define two case classes for this task: the input row format (which is simply a device and a timestamp); and the state and output rows (which contain the current count of records collected, device ID, and an array of readings for the events in the window). Here are our various, self-describing case class definitions: case class InputRow(device: String, timestamp: java.sql.Timestamp, x: Double) case class DeviceState(device: String, var values: Array[Double], var count: Int) case class OutputRow(device: String, previousAverage: Double) Now, you can define the function to update the individual state based on a single input row. You could write this inline or in a number of other ways, but this example makes it easy to see exactly how you update based on a given row: def updateWithEvent(state:DeviceState, input:InputRow):DeviceState = { state.count += 1 // maintain an array of the x-axis values state.values = state.values ++ Array(input.x) state } Now it’s time to define the function that updates across a series of input rows. Notice in the example that follows that we have a specific key, the iterator of inputs, and the old state, and we update that old state over time as we receive new events. This, in turn, will return our output rows with the updates on a per-device level based on the number of counts it sees. This case is quite straightforward, after a given number of events, you update the state and reset it. You then create an output row. You can see this row in the output table: import org.apache.spark.sql.streaming.{GroupStateTimeout, OutputMode, GroupState} def updateAcrossEvents(device:String, inputs: Iterator[InputRow], oldState: GroupState[DeviceState]):Iterator[OutputRow] = { inputs.toSeq.sortBy(_.timestamp.getTime).toIterator.flatMap { input => val state = if (oldState.exists) oldState.get else DeviceState(device, Array(), 0) val newState = updateWithEvent(state, input) if (newState.count >= 500) { // One of our windows is complete; replace our state with an empty // DeviceState and output the average for the past 500 items from // the old state oldState.update(DeviceState(device, Array(), 0)) Iterator(OutputRow(device, newState.values.sum / newState.values.length.toDouble)) } else { // Update the current DeviceState object in place and output no // records oldState.update(newState) Iterator() } } } Now you can run your stream. You will notice that you need to explicitly state the output mode, which is append. You also need to set a GroupStateTimeout. This time-out specifies the amount of time you want to wait before a window should be output as complete (even if it did not reach the required count). In that case, set an infinite time-out, meaning if a device never gets to that required 500 count threshold, it will maintain that state forever as “incomplete” and not output it to the result table. By specifying both of those parameters you can pass in the updateAcrossEvents function and start the stream: import org.apache.spark.sql.streaming.GroupStateTimeout withEventTime .selectExpr("Device as device", "cast(Creation_Time/1000000000 as timestamp) as timestamp", "x") .as[InputRow] .groupByKey(_.device) .flatMapGroupsWithState(OutputMode.Append, GroupStateTimeout.NoTimeout)(updateAcrossEvents) .writeStream .queryName("count_based_device") .format("memory") .outputMode("append") .start() After you start the stream, it’s time to query it. Here are the results: SELECT * FROM count_based_device +--------+--------------------+ | device| previousAverage| +--------+--------------------+ |nexus4_1| 4.660034012E-4| |nexus4_1|0.001436279298199...| ... |nexus4_1|1.049804683999999...| |nexus4_1|-0.01837188737960...| +--------+--------------------+ You can see the values change over each of those windows as you append new data to the result set. flatMapGroupsWithState Our second example of stateful processing will use a feature called flatMapGroupsWithState. This is quite similar to mapGroupsWithState except that rather than just having a single key with at most one output, a single key can have many outputs. This can provide us a bit more flexibility and the same fundamental structure as mapGroupsWithState applies. Here’s what we’ll need to define. Three class definitions: an input definition, a state definition, and optionally an output definition. A function to update the state based on a key, an iterator of events, and a previous state. A time-out parameter (as described in the time-outs section). With these objects and definitions, we can control arbitrary state by creating it, updating it over time, and removing it. Let’s start with an example of sessionization. EXAMPLE: SESSIONIZATION Sessions are simply unspecified time windows with a series of events that occur. Typically, you want to record these different events in an array in order to compare these sessions to other sessions in the future. In a session, you will likely have arbitrary logic to maintain and update your state over time as well as certain actions to define when state ends (like a count) or a simple time-out. Let’s build on the previous example and define it a bit more strictly as a session. At times, you might have an explicit session ID that you can use in your function. This obviously makes it much easier because you can just perform a simple aggregation and might not even need your own stateful logic. In this case, you’re creating sessions on the fly from a user ID and some time information and if you see no new event from that user in five seconds, the session terminates. You’ll also notice that this code uses time-outs differently than we have in other examples. You can follow the same process of creating your classes, defining our single event update function and then the multievent update function: case class InputRow(uid:String, timestamp:java.sql.Timestamp, x:Double, activity:String) case class UserSession(val uid:String, var timestamp:java.sql.Timestamp, var activities: Array[String], var values: Array[Double]) case class UserSessionOutput(val uid:String, var activities: Array[String], var xAvg:Double) def updateWithEvent(state:UserSession, input:InputRow):UserSession = { // handle malformed dates if (Option(input.timestamp).isEmpty) { return state } state.timestamp = input.timestamp state.values = state.values ++ Array(input.x) if (!state.activities.contains(input.activity)) { state.activities = state.activities ++ Array(input.activity) } state } import org.apache.spark.sql.streaming.{GroupStateTimeout, OutputMode, GroupState} def updateAcrossEvents(uid:String, inputs: Iterator[InputRow], oldState: GroupState[UserSession]):Iterator[UserSessionOutput] = { inputs.toSeq.sortBy(_.timestamp.getTime).toIterator.flatMap { input => val state = if (oldState.exists) oldState.get else UserSession( uid, new java.sql.Timestamp(6284160000000L), Array(), Array()) val newState = updateWithEvent(state, input) if (oldState.hasTimedOut) { val state = oldState.get oldState.remove() Iterator(UserSessionOutput(uid, state.activities, newState.values.sum / newState.values.length.toDouble)) } else if (state.values.length > 1000) { val state = oldState.get oldState.remove() Iterator(UserSessionOutput(uid, state.activities, newState.values.sum / newState.values.length.toDouble)) } else { oldState.update(newState) oldState.setTimeoutTimestamp(newState.timestamp.getTime(), "5 seconds") Iterator() } } } You’ll see in this one that we only expect to see an event at most five seconds late. Anything other than that and we will ignore it. We will use an EventTimeTimeout to set that we want to time-out based on the event time in this stateful operation: import org.apache.spark.sql.streaming.GroupStateTimeout withEventTime.where("x is not null") .selectExpr("user as uid", "cast(Creation_Time/1000000000 as timestamp) as timestamp", "x", "gt as activity") .as[InputRow] .withWatermark("timestamp", "5 seconds") .groupByKey(_.uid) .flatMapGroupsWithState(OutputMode.Append, GroupStateTimeout.EventTimeTimeout)(updateAcrossEvents) .writeStream .queryName("count_based_device") .format("memory") .start() Querying this table will show you the output rows for each user over this time period: SELECT * FROM count_based_device +---+--------------------+--------------------+ |uid| activities| xAvg| +---+--------------------+--------------------+ | a| [stand, null, sit]|-9.10908533566433...| | a| [sit, null, walk]|-0.00654280428601...| ... | c|[null, stairsdown...|-0.03286657789999995| +---+--------------------+--------------------+ As you might expect, sessions that have a number of activities in them have a higher x-axis gyroscope value than ones that have fewer activities. It should be trivial to extend this example to problem sets more relevant to your own domain, as well. Conclusion This chapter covered some of the more advanced topics in Structured Streaming, including event time and stateful processing. This is effectively the user guide to help you actually build out your application logic and turn it into something that provides value. Next, we will discuss what we’ll need to do in order to take this application to production and maintain and update it over time. Chapter 23. Structured Streaming in Production The previous chapters of this part of the book have covered Structured Streaming from a user’s perspective. Naturally this is the core of your application. This chapter covers some of the operational tools needed to run Structured Streaming robustly in production after you’ve developed an application. Structured Streaming was marked as production-ready in Apache Spark 2.2.0, meaning that this release has all the features required for production use and stabilizes the API. Many organizations are already using the system in production because, frankly, it’s not much different from running other production Spark applications. Indeed, through features such as transactional sources/sinks and exactly-once processing, the Structured Streaming designers sought to make it as easy to operate as possible. This chapter will walk you through some of the key operational tasks specific to Structured Streaming. This should supplement everything we saw and learned about Spark operations in Part II. Fault Tolerance and Checkpointing The most important operational concern for a streaming application is failure recovery. Faults are inevitable: you’re going to lose a machine in the cluster, a schema will change by accident without a proper migration, or you may even intentionally restart the cluster or application. In any of these cases, Structured Streaming allows you to recover an application by just restarting it. To do this, you must configure the application to use checkpointing and write-ahead logs, both of which are handled automatically by the engine. Specifically, you must configure a query to write to a checkpoint location on a reliable file system (e.g., HDFS, S3, or any compatible filesystem). Structured Streaming will then periodically save all relevant progress information (for instance, the range of offsets processed in a given trigger) as well as the current intermediate state values to the checkpoint location. In a failure scenario, you simply need to restart your application, making sure to point to the same checkpoint location, and it will automatically recover its state and start processing data where it left off. You do not have to manually manage this state on behalf of the application—Structured Streaming does it for you. To use checkpointing, specify your checkpoint location before starting your application through the checkpointLocation option on writeStream. You can do this as follows: // in Scala val static = spark.read.json("/data/activity-data") val streaming = spark .readStream .schema(static.schema) .option("maxFilesPerTrigger", 10) .json("/data/activity-data") .groupBy("gt") .count() val query = streaming .writeStream .outputMode("complete") .option("checkpointLocation", "/some/location/") .queryName("test_stream") .format("memory") .start() # in Python static = spark.read.json("/data/activity-data") streaming = spark\ .readStream\ .schema(static.schema)\ .option("maxFilesPerTrigger", 10)\ .json("/data/activity-data")\ .groupBy("gt")\ .count() query = streaming\ .writeStream\ .outputMode("complete")\ .option("checkpointLocation", "/some/python/location/")\ .queryName("test_python_stream")\ .format("memory")\ .start() If you lose your checkpoint directory or the information inside of it, your application will not be able to recover from failures and you will have to restart your stream from scratch. Updating Your Application Checkpointing is probably the most important thing to enable in order to run your applications in production. This is because the checkpoint will store all of the information about what your stream has processed thus far and what the intermediate state it may be storing is. However, checkpointing does come with a small catch—you’re going to have to reason about your old checkpoint data when you update your streaming application. When you update your application, you’re going to have to ensure that your update is not a breaking change. Let’s cover these in detail when we review the two types of updates: either an update to your application code or running a new Spark version. Updating Your Streaming Application Code Structured Streaming is designed to allow certain types of changes to the application code between application restarts. Most importantly, you are allowed to change user-defined functions (UDFs) as long as they have the same type signature. This feature can be very useful for bug fixes. For example, imagine that your application starts receiving a new type of data, and one of the data parsing functions in your current logic crashes. With Structured Streaming, you can recompile the application with a new version of that function and pick up at the same point in the stream where it crashed earlier. While small adjustments like adding a new column or changing a UDF are not breaking changes and do not require a new checkpoint directory, there are larger changes that do require an entirely new checkpoint directory. For example, if you update your streaming application to add a new aggregation key or fundamentally change the query itself, Spark cannot construct the required state for the new query from an old checkpoint directory. In these cases, Structured Streaming will throw an exception saying it cannot begin from a checkpoint directory, and you must start from scratch with a new (empty) directory as your checkpoint location. Updating Your Spark Version Structured Streaming applications should be able to restart from an old checkpoint directory across patch version updates to Spark (e.g., moving from Spark 2.2.0 to 2.2.1 to 2.2.2). The checkpoint format is designed to be forward-compatible, so the only way it may be broken is due to critical bug fixes. If a Spark release cannot recover from old checkpoints, this will be clearly documented in its release notes. The Structured Streaming developers also aim to keep the format compatible across minor version updates (e.g., Spark 2.2.x to 2.3.x), but you should check the release notes to see whether this is supported for each upgrade. In either case, if you cannot start from a checkpoint, you will need to start your application again using a new checkpoint directory. Sizing and Rescaling Your Application In general, the size of your cluster should be able to comfortably handle bursts above your data rate. The key metrics you should be monitoring in your application and cluster are discussed as follows. In general, if you see that your input rate is much higher than your processing rate (elaborated upon momentarily), it’s time to scale up your cluster or application. Depending on your resource manager and deployment, you may just be able to dynamically add executors to your application. When it comes time, you can scale-down your application in the same way— remove executors (potentially through your cloud provider) or restart your application with lower resource counts. These changes will likely incur some processing delay (as data is recomputed or partitions are shuffled around when executors are removed). In the end, it’s a business decision as to whether it’s worthwhile to create a system with more sophisticated resource management capabilities. While making underlying infrastructure changes to the cluster or application are sometimes necessary, other times a change may only require a restart of the application or stream with a new configuration. For instance, changing spark.sql.shuffle.partitions is not supported while a stream is currently running (it won’t actually change the number of shuffle partitions). This requires restarting the actual stream, not necessarily the entire application. Heavier weight changes, like changing arbitrary Spark application configurations, will likely require an application restart. Metrics and Monitoring Metrics and monitoring in streaming applications is largely the same as for general Spark applications using the tools described in Chapter 18. However, Structured Streaming does add several more specifics in order to help you better understand the state of your application. There are two key APIs you can leverage to query the status of a streaming query and see its recent execution progress. With these two APIs, you can get a sense of whether or not your stream is behaving as expected. Query Status The query status is the most basic monitoring API, so it’s a good starting point. It aims to answer the question, “What processing is my stream performing right now?” This information is reported in the status field of the query object returned by startStream. For example, you might have a simple counts stream that provides counts of IOT devices defined by the following query (here we’re just using the same query from the previous chapter without the initialization code): query.status To get the status of a given query, simply running the command query.status will return the current status of the stream. This gives us details about what is happening at that point in time in the stream. Here’s a sample of what you’ll get back when querying this status: { "message" : "Getting offsets from ...", "isDataAvailable" : true, "isTriggerActive" : true } The above snippet describes getting the offsets from a Structured Streaming data source (hence the message describing getting offsets). There are a variety of messages to describe the stream’s status. NOTE We have shown the status command inline here the way you would call it in a Spark shell. However, for a standalone application, you may not have a shell attached to run arbitrary code inside your process. In that case, you can expose its status by implementing a monitoring server, such as a small HTTP server that listens on a port and returns query.status when it gets a request. Alternatively, you can use the richer StreamingQueryListener API described later to listen to more events. Recent Progress While the query’s current status is useful to see, equally important is an ability to view the query’s progress. The progress API allows us to answer questions like “At what rate am I processing tuples?” or “How fast are tuples arriving from the source?” By running query.recentProgress, you’ll get access to more time-based information like the processing rate and batch durations. The streaming query progress also includes information about the input sources and output sinks behind your stream. query.recentProgress Here’s the result of the Scala version after we ran the code from before; the Python one will be similar: Array({ "id" : "d9b5eac5-2b27-4655-8dd3-4be626b1b59b", "runId" : "f8da8bc7-5d0a-4554-880d-d21fe43b983d", "name" : "test_stream", "timestamp" : "2017-08-06T21:11:21.141Z", "numInputRows" : 780119, "processedRowsPerSecond" : 19779.89350912779, "durationMs" : { "addBatch" : 38179, "getBatch" : 235, "getOffset" : 518, "queryPlanning" : 138, "triggerExecution" : 39440, "walCommit" : 312 }, "stateOperators" : [ { "numRowsTotal" : 7, "numRowsUpdated" : 7 } ], "sources" : [ { "description" : "FileStreamSource[/some/stream/source/]", "startOffset" : null, "endOffset" : { "logOffset" : 0 }, "numInputRows" : 780119, "processedRowsPerSecond" : 19779.89350912779 } ], "sink" : { "description" : "MemorySink" } }) As you can see from the output just shown, this includes a number of details about the state of the stream. It is important to note that this is a snapshot in time (according to when we asked for the query progress). In order to consistently get output about the state of the stream, you’ll need to query this API for the updated state repeatedly. The majority of the fields in the previous output should be self-explanatory. However, let’s review some of the more consequential fields in detail. Input rate and processing rate The input rate specifies how much data is flowing into Structured Streaming from our input source. The processing rate is how quickly the application is able to analyze that data. In the ideal case, the input and processing rates should vary together. Another case might be when the input rate is much greater than the processing rate. When this happens, the stream is falling behind and you will need to scale the cluster up to handle the larger load. Batch duration Nearly all streaming systems utilize batching to operate at any reasonable throughput (some have an option of high latency in exchange for lower throughput). Structured Streaming achieves both. As it operates on the data, you will likely see batch duration oscillate as Structured Streaming processes varying numbers of events over time. Naturally, this metric will have little to no relevance when the continuous processing engine is made an execution option. TIP Generally it’s a best practice to visualize the changes in batch duration and input and processing rates. It’s much more helpful than simply reporting changes over time. Spark UI The Spark web UI, covered in detail in Chapter 18, also shows tasks, jobs, and data processing metrics for Structured Streaming applications. On the Spark UI, each streaming application will appear as a sequence of short jobs, one for each trigger. However, you can use the same UI to see metrics, query plans, task durations, and logs from your application. One departure of note from the DStream API is that the Streaming Tab is not used by Structured Streaming. Alerting Understanding and looking at the metrics for your Structured Streaming queries is an important first step. However, this involves constantly watching a dashboard or the metrics in order to discover potential issues. You’re going to need robust automatic alerting to notify you when your jobs are failing or not keeping up with the input data rate without monitoring them manually. There are several ways to integrate existing alerting tools with Spark, generally building on the recent progress API we covered before. For example, you may directly feed the metrics to a monitoring system such as the open source Coda Hale Metrics library or Prometheus, or you may simply log them and use a log aggregation system like Splunk. In addition to monitoring and alerting on queries, you’re also going to want to monitor and alert on the state of the cluster and the overall application (if you’re running multiple queries together). Advanced Monitoring with the Streaming Listener We already touched on some of the high-level monitoring tools in Structured Streaming. With a bit of glue logic, you can use the status and queryProgress APIs to output monitoring events into your organization’s monitoring platform of choice (e.g., a log aggregation system or Prometheus dashboard). Beyond these approaches, there is also a lower-level but more powerful way to observe an application’s execution: the StreamingQueryListener class. The StreamingQueryListener class will allow you to receive asynchronous updates from the streaming query in order to automatically output this information to other systems and implement robust monitoring and alerting mechanisms. You start by developing your own object to extend StreamingQueryListener, then attach it to a running SparkSession. Once you attach your custom listener with sparkSession.streams.addListener(), your class will receive notifications when a query is started or stopped, or progress is made on an active query. Here’s a simple example of a listener from the Structured Streaming documentation: val spark: SparkSession = ... spark.streams.addListener(new StreamingQueryListener() { override def onQueryStarted(queryStarted: QueryStartedEvent): Unit = { println("Query started: " + queryStarted.id) } override def onQueryTerminated( queryTerminated: QueryTerminatedEvent): Unit = { println("Query terminated: " + queryTerminated.id) } override def onQueryProgress(queryProgress: QueryProgressEvent): Unit = { println("Query made progress: " + queryProgress.progress) } }) Streaming listeners allow you to process each progress update or status change using custom code and pass it to external systems. For example, the following code for a StreamingQueryListener that will forward all query progress information to Kafka. You’ll have to parse this JSON string once you read data from Kafka in order to access the actual metrics: class KafkaMetrics(servers: String) extends StreamingQueryListener { val kafkaProperties = new Properties() kafkaProperties.put( "bootstrap.servers", servers) kafkaProperties.put( "key.serializer", "kafkashaded.org.apache.kafka.common.serialization.StringSerializer") kafkaProperties.put( "value.serializer", "kafkashaded.org.apache.kafka.common.serialization.StringSerializer") val producer = new KafkaProducer[String, String](kafkaProperties) import org.apache.spark.sql.streaming.StreamingQueryListener import org.apache.kafka.clients.producer.KafkaProducer override def onQueryProgress(event: StreamingQueryListener.QueryProgressEvent): Unit = { producer.send(new ProducerRecord("streaming-metrics", event.progress.json)) } override def onQueryStarted(event: StreamingQueryListener.QueryStartedEvent): Unit = {} override def onQueryTerminated(event: StreamingQueryListener.QueryTerminatedEvent): Unit = {} } Using the StreamingQueryListener interface, you can even monitor Structured Streaming applications on one cluster by running a Structured Streaming application on that same (or another) cluster. You could also manage multiple streams in this way. Conclusion In this chapter, we covered the main tools needed to run Structured Streaming in production: checkpoints for fault tolerance and various monitoring APIs that let you observe how your application is running. Lucky for you, if you’re running Spark in production already, many of the concepts and tools are similar, so you should be able to reuse a lot of your existing knowledge. Be sure to check Part IV to see some other helpful tools for monitoring Spark Applications. Part VI. Advanced Analytics and Machine Learning Chapter 24. Advanced Analytics and Machine Learning Overview Thus far, we have covered fairly general data flow APIs. This part of the book will dive deeper into some of the more specific advanced analytics APIs available in Spark. Beyond large-scale SQL analysis and streaming, Spark also provides support for statistics, machine learning, and graph analytics. These encompass a set of workloads that we will refer to as advanced analytics. This part of the book will cover advanced analytics tools in Spark, including: Preprocessing your data (cleaning data and feature engineering) Supervised learning Recommendation learning Unsupervised engines Graph analytics Deep learning This chapter offers a basic overview of advanced analytics, some example use cases, and a basic advanced analytics workflow. Then we’ll cover the analytics tools just listed and teach you how to apply them. WARNING This book is not intended to teach you everything you need to know about machine learning from scratch. We won’t go into strict mathematical definitions and formulations—not for lack of importance but simply because it’s too much information to include. This part of the book is not an algorithm guide that will teach you the mathematical underpinnings of every available algorithm nor the in-depth implementation strategies used. The chapters included here serve as a guide for users, with the purpose of outlining what you need to know to use Spark’s advanced analytics APIs. A Short Primer on Advanced Analytics Advanced analytics refers to a variety of techniques aimed at solving the core problem of deriving insights and making predictions or recommendations based on data. The best ontology for machine learning is structured based on the task that you’d like to perform. The most common tasks include: Supervised learning, including classification and regression, where the goal is to predict a label for each data point based on various features. Recommendation engines to suggest products to users based on behavior. Unsupervised learning, including clustering, anomaly detection, and topic modeling, where the goal is to discover structure in the data. Graph analytics tasks such as searching for patterns in a social network. Before discussing Spark’s APIs in detail, let’s review each of these tasks along with some common machine learning and advanced analytics use cases. While we have certainly tried to make this introduction as accessible as possible, at times you may need to consult other resources in order to fully understand the material. O’Reilly should we link to or mention any specific ones? Additionally, we will cite the following books throughout the next few chapters because they are great resources for learning more about the individual analytics (and, as a bonus, they are freely available on the web): An Introduction to Statistical Learning by Gareth James, Daniela Witten, Trevor Hastie, and Robert Tibshirani. We refer to this book as “ISL.” Elements of Statistical Learning by Trevor Hastie, Robert Tibshirani, and Jerome Friedman. We refer to this book as “ESL.” Deep Learning by Ian Goodfellow, Yoshua Bengio, and Aaron Courville. We refer to this book as “DLB.” Supervised Learning Supervised learning is probably the most common type of machine learning. The goal is simple: using historical data that already has labels (often called the dependent variables), train a model to predict the values of those labels based on various features of the data points. One example would be to predict a person’s income (the dependent variable) based on age (a feature). This training process usually proceeds through an iterative optimization algorithm such as gradient descent. The training algorithm starts with a basic model and gradually improves it by adjusting various internal parameters (coefficients) during each training iteration. The result of this process is a trained model that you can use to make predictions on new data. There are a number of different tasks we’ll need to complete as part of the process of training and making predictions, such as measuring the success of trained models before using them in the field, but the fundamental principle is simple: train on historical data, ensure that it generalizes to data we didn’t train on, and then make predictions on new data. We can further organize supervised learning based on the type of variable we’re looking to predict. We’ll get to that next. Classification One common type of supervised learning is classification. Classification is the act of training an algorithm to predict a dependent variable that is categorical (belonging to a discrete, finite set of values). The most common case is binary classification, where our resulting model will make a prediction that a given item belongs to one of two groups. The canonical example is classifying email spam. Using a set of historical emails that are organized into groups of spam emails and not spam emails, we train an algorithm to analyze the words in, and any number of properties of, the historical emails and make predictions about them. Once we are satisfied with the algorithm’s performance, we use that model to make predictions about future emails the model has never seen before. When we classify items into more than just two categories, we call this multiclass classification. For example, we may have four different categories of email (as opposed to the two categories in the previous paragraph): spam, personal, work related, and other. There are many use cases for classification, including: Predicting disease A doctor or hospital might have a historical dataset of behavioral and physiological attributes of a set of patients. They could use this dataset to train a model on this historical data (and evaluate its success and ethical implications before applying it) and then leverage it to predict whether or not a patient has heart disease or not. This is an example of binary classification (healthy heart, unhealthy heart) or multiclass classification (healthly heart, or one of several different diseases). Classifying images There are a number of applications from companies like Apple, Google, or Facebook that can predict who is in a given photo by running a classification model that has been trained on historical images of people in your past photos. Another common use case is to classify images or label the objects in images. Predicting customer churn A more business-oriented use case might be predicting customer churn—that is, which customers are likely to stop using a service. You can do this by training a binary classifier on past customers that have churned (and not churned) and using it to try and predict whether or not current customers will churn. Buy or won’t buy Companies often want to predict whether visitors of their website will purchase a given product. They might use information about users’ browsing pattern or attributes such as location in order to drive this prediction. There are many more use cases for classification beyond these examples. We will introduce more use cases, as well as Spark’s classification APIs, in Chapter 26. Regression In classification, our dependent variable is a set of discrete values. In regression, we instead try to predict a continuous variable (a real number). In simplest terms, rather than predicting a category, we want to predict a value on a number line. The rest of the process is largely the same, which is why they’re both forms of supervised learning. We will train on historical data to make predictions about data we have never seen. Here are some typical examples: Predicting sales A store may want to predict total product sales on given data using historical sales data. There are a number of potential input variables, but a simple example might be using last week’s sales data to predict the next day’s data. Predicting height Based on the heights of two individuals, we might want to predict the heights of their potential children. Predicting the number of viewers of a show A media company like Netflix might try to predict how many of their subscribers will watch a particular show. We will introduce more use cases, as well as Spark’s methods for regression, in Chapter 27. Recommendation Recommendation is one of the most intuitive applications of advanced analytics. By studying people’s explicit preferences (through ratings) or implicit ones (through observed behavior) for various products or items, an algorithm can make recommendations on what a user may like by drawing similarities between the users or items. By looking at these similarities, the algorithm makes recommendations to users based on what similar users liked, or what other products resemble the ones the user already purchased. Recommendation is a common use case for Spark and well suited to big data. Here are some example use cases: Movie recommendations Netflix uses Spark, although not necessarily its built-in libraries, to make large-scale movie recommendations to its users. It does this by studying what movies users watch and do not watch in the Netflix application. In addition, Netflix likely takes into consideration how similar a given user’s ratings are to other users’. Product recommendations Amazon uses product recommendations as one of its main tools to increase sales. For instance, based on the items in our shopping cart, Amazon may recommend other items that were added to similar shopping carts in the past. Likewise, on every product page, Amazon shows similar products purchased by other users. We will introduce more recommendation use cases, as well as Spark’s methods for generating recommendations, in Chapter 28. Unsupervised Learning Unsupervised learning is the act of trying to find patterns or discover the underlying structure in a given set of data. This differs from supervised learning because there is no dependent variable (label) to predict. Some example use cases for unsupervised learning include: Anomaly detection Given some standard event type often occuring over time, we might want to report when a nonstandard type of event occurs. For example, a security officer might want to receive notifications when a strange object (think vehicle, skater, or bicyclist) is observed on a pathway. User segmentation Given a set of user behaviors, we might want to better understand what attributes certain users share with other users. For instance, a gaming company might cluster users based on properties like the number of hours played in a given game. The algorithm might reveal that casual players have very different behavior than hardcore gamers, for example, and allow the company to offer different recommendations or rewards to each player. Topic modeling Given a set of documents, we might analyze the different words contained therein to see if there is some underlying relation between them. For example, given a number of web pages on data analytics, a topic modeling algorithm can cluster them into pages about machine learning, SQL, streaming, and so on based on groups of words that are more common in one topic than in others. Intuitively, it is easy to see how segmenting customers could help a platform cater better to each set of users. However, it may be hard to discover whether or not this set of user segments is “correct”. For this reason, it can be difficult to determine whether a particular model is good or not. We will discuss unsupervised learning in detail in Chapter 29. Graph Analytics While less common than classification and regression, graph analytics is a powerful tool. Fundamentally, graph analytics is the study of structures in which we specify vertices (which are objects) and edges (which represent the relationships between those objects). For example, the vertices might represent people and products, and edges might represent a purchase. By looking at the properties of vertices and edges, we can better understand the connections between them and the overall structure of the graph. Since graphs are all about relationships, anything that specifies a relationship is a great use case for graph analytics. Some examples include: Fraud prediction Capital One uses Spark’s graph analytics capabilities to better understand fraud networks. By using historical fraudulent information (like phone numbers, addresses, or names) they discover fraudulent credit requests or transactions. For instance, any user accounts within two hops of a fraudulent phone number might be considered suspicious. Anomaly detection By looking at how networks of individuals connect with one another, outliers and anomalies can be flagged for manual analysis. For instance, if typically in our data each vertex has ten edges associated with it and a given vertex only has one edge, that might be worth investigating as something strange. Classification Given some facts about certain vertices in a network, you can classify other vertices according to their connection to the original node. For instance, if a certain individual is labeled as an influencer in a social network, we could classify other individuals with similar network structures as influencers. Recommendation Google’s original web recommendation algorithm, PageRank, is a graph algorithm that analyzes website relationships in order to rank the importance of web pages. For example, a web page that has a lot of links to it is ranked as more important than one with no links to it. We’ll discuss more examples of graph analytics in Chapter 30. The Advanced Analytics Process You should have a firm grasp of some fundamental use cases for machine learning and advanced analytics. However, finding a use case is only a small part of the actual advanced analytics process. There is a lot of work in preparing your data for analysis, testing different ways of modeling it, and evaluating these models. This section will provide structure to the overall anaytics process and the steps we have to take to not just perform one of the tasks just outlined, but actually evaluate success objectively in order to understand whether or not we should apply our model to the real world (Figure 24-1). Figure 24-1. The machine learning workflow The overall process involves, the following steps (with some variation): 1. Gathering and collecting the relevant data for your task. 2. Cleaning and inspecting the data to better understand it. 3. Performing feature engineering to allow the algorithm to leverage the data in a suitable form (e.g., converting the data to numerical vectors). 4. Using a portion of this data as a training set to train one or more algorithms to generate some candidate models. 5. Evaluating and comparing models against your success criteria by objectively measuring results on a subset of the same data that was not used for training. This allows you to better understand how your model may perform in the wild. 6. Leveraging the insights from the above process and/or using the model to make predictions, detect anomalies, or solve more general business challenges. These steps won’t be the same for every advanced analytics task. However, this workflow does serve as a general framework for what you’re going to need to be successful with advanced analytics. Just as we did with the various advanced analytics tasks earlier in the chapter, let’s break down the process to better understand the overall objective of each step. Data collection Naturally it’s hard to create a training set without first collecting data. Typically this means at least gathering the datasets you’ll want to leverage to train your algorithm. Spark is an excellent tool for this because of its ability to speak to a variety of data sources and work with data big and small. Data cleaning After you’ve gathered the proper data, you’re going to need to clean and inspect it. This is typically done as part of a process called exploratory data analysis, or EDA. EDA generally means using interactive queries and visualization methods in order to better understand distributions, correlations, and other details in your data. During this process you may notice you need to remove some values that may have been misrecorded upstream or that other values may be missing. Whatever the case, it’s always good to know what is in your data to avoid mistakes down the road. The multitude of Spark functions in the structured APIs will provide a simple way to clean and report on your data. Feature engineering Now that you collected and cleaned your dataset, it’s time to convert it to a form suitable for machine learning algorithms, which generally means numerical features. Proper feature engineering can often make or break a machine learning application, so this is one task you’ll want to do carefully. The process of feature engineering includes a variety of tasks, such as normalizing data, adding variables to represent the interactions of other variables, manipulating categorical variables, and converting them to the proper format to be input into our machine learning model. In MLlib, Spark’s machine learning library, all variables will usually have to be input as vectors of doubles (regardless of what they actually represent). We cover the process of feature engineering in great depth in Chapter 25. As you will see in that chapter, Spark provides the essentials you’ll need to manipulate your data using a variety of machine learning statistical techniques. NOTE The following few steps (training models, model tuning, and evaluation) are not relevant to all use cases. This is a general workflow that may vary significantly based on the end objective you would like to achieve. Training models At this point in the process we have a dataset of historical information (e.g., spam or not spam emails) and a task we would like to complete (e.g., classifying spam emails). Next, we will want to train a model to predict the correct output, given some input. During the training process, the parameters inside of the model will change according to how well the model performed on the input data. For instance, to classify spam emails, our algorithm will likely find that certain words are better predictors of spam than others and therefore weight the parameters associated with those words higher. In the end, the trained model will find that certain words should have more influence (because of their consistent association with spam emails) than others. The output of the training process is what we call a model. Models can then be used to gain insights or to make future predictions. To make predictions, you will give the model an input and it will produce an output based on a mathematical manipulation of these inputs. Using the classification example, given the properties of an email, it will predict whether that email is spam or not by comparing to the historical spam and not spam emails that it was trained on. However, just training a model isn’t the objective—we want to leverage our model to produce insights. Thus, we must answer the question: how do we know our model is any good at what it’s supposed to do? That’s where model tuning and evaluation come in. Model tuning and evaluation You likely noticed earlier that we mentioned that you should split your data into multiple portions and use only one for training. This is an essential step in the machine learning process because when you build an advanced analytics model you want that model to generalize to data it has not seen before. Splitting our dataset into multiple portions allows us to objectively test the effectiveness of the trained model against a set of data that it has never seen before. The objective is to see if your model understands something fundamental about this data process or whether or not it just noticed the things particular to only the training set (sometimes called overfitting). That’s why it is called a test set. In the process of training models, we also might take another, separate subset of data and treat that as another type of test set, called a validation set, in order to try out different hyperparameters (parameters that affect the training process) and compare different variations of the same model without overfitting to the test set. WARNING Following proper training, validation, and test set best practices is essential to successfully using machine learning. It’s easy to end up overfitting (training a model that does not generalize well to new data) if we do not properly isolate these sets of data. We cannot cover this problem in depth in this book, but almost any machine learning book will cover this topic. To continue with the classification example we referenced previously, we have three sets of data: a training set for training models, a validation set for testing different variations of the models that we’re training, and lastly, a test set we will use for the final evaluation of our different model variations to see which one performed the best. Leveraging the model and/or insights After running the model through the training process and ending up with a well-performing model, you are now ready to use it! Taking your model to production can be a significant challenge in and of itself. We will discuss some tactics later on in this chapter. Spark’s Advanced Analytics Toolkit The previous overview is just an example workflow and doesn’t encompass all use cases or potential workflows. In addition, you probably noticed that we did not discuss Spark almost at all. This section will discuss Spark’s advanced analytics capabilities. Spark includes several core packages and many external packages for performing advanced analytics. The primary package is MLlib, which provides an interface for building machine learning pipelines. What Is MLlib? MLlib is a package, built on and included in Spark, that provides interfaces for gathering and cleaning data, feature engineering and feature selection, training and tuning large-scale supervised and unsupervised machine learning models, and using those models in production. WARNING MLlib actually consists of two packages that leverage different core data structures. The package org.apache.spark.ml includes an interface for use with DataFrames. This package also offers a high-level interface for building machine learning pipelines that help standardize the way in which you perform the preceding steps. The lower-level package, org.apache.spark.mllib, includes interfaces for Spark’s low-level RDD APIs. This book will focus exclusively on the DataFrame API. The RDD API is the lower-level interface, which is in maintenance mode (meaning it will only receive bug fixes, not new features) at this time. It has also been covered fairly extensively in older books on Spark and is therefore omitted here. When and why should you use MLlib (versus scikit-learn, TensorFlow, or foo package) At a high level, MLlib might sound like a lot of other machine learning packages you’ve probably heard of, such as scikit-learn for Python or the variety of R packages for performing similar tasks. So why should you bother with MLlib at all? There are numerous tools for performing machine learning on a single machine, and while there are several great options to choose from, these single machine tools do have their limits either in terms of the size of data you can train on or the processing time. This means single-machine tools are usually complementary to MLlib. When you hit those scalability issues, take advantage of Spark’s abilities. There are two key use cases where you want to leverage Spark’s ability to scale. First, you want to leverage Spark for preprocessing and feature generation to reduce the amount of time it might take to produce training and test sets from a large amount of data. Then you might leverage single-machine learning libraries to train on those given data sets. Second, when your input data or model size become too difficult or inconvenient to put on one machine, use Spark to do the heavy lifting. Spark makes distributed machine learning very simple. An important caveat to all of this is that while training and data preparation are made simple, there are still some complexities you will need to keep in mind, especially when it comes to deploying a trained model. For example, Spark does not provide a built-in way to serve lowlatency predictions from a model, so you may want to export the model to another serving system or a custom application to do that. MLlib is generally designed to allow inspecting and exporting models to other tools where possible. High-Level MLlib Concepts In MLlib there are several fundamental “structural” types: transformers, estimators, evaluators, and pipelines. By structural, we mean you will think in terms of these types when you define an end-to-end machine learning pipeline. They’ll provide the common language for defining what belongs in what part of the pipeline. Figure 24-2 illustrates the overall workflow that you will follow when developing machine learning models in Spark. Figure 24-2. The machine learning workflow, in Spark Transformers are functions that convert raw data in some way. This might be to create a new interaction variable (from two other variables), normalize a column, or simply change an Integer into a Double type to be input into a model. An example of a transformer is one that converts string categorical variables into numerical values that can be used in MLlib. Transformers are primarily used in preprocessing and feature engineering. Transformers take a DataFrame as input and produce a new DataFrame as output, as illustrated in Figure 24-3. Figure 24-3. A standard transformer Estimators are one of two kinds of things. First, estimators can be a kind of transformer that is initialized with data. For instance, to normalize numerical data we’ll need to initialize our transformation with some information about the current values in the column we would like to normalize. This requires two passes over our data—the initial pass generates the initialization values and the second actually applies the generated function over the data. In the Spark’s nomenclature, algorithms that allow users to train a model from data are also referred to as estimators. An evaluator allows us to see how a given model performs according to criteria we specify like a receiver operating characteristic (ROC) curve. After we use an evaluator to select the best model from the ones we tested, we can then use that model to make predictions. From a high level we can specify each of the transformations, estimations, and evaluations one by one, but it is often easier to specify our steps as stages in a pipeline. This pipeline is similar to scikit-learn’s pipeline concept. Low-level data types In addition to the structural types for building pipelines, there are also several lower-level data types you may need to work with in MLlib (Vector being the most common). Whenever we pass a set of features into a machine learning model, we must do it as a vector that consists of Doubles. This vector can be either sparse (where most of the elements are zero) or dense (where there are many unique values). Vectors are created in different ways. To create a dense vector, we can specify an array of all the values. To create a sparse vector, we can specify the total size and the indices and values of the non-zero elements. Sparse is the best format, as you might have guessed, when the majority of values are zero as this is a more compressed representation. Here is an example of how to manually create a Vector: // in Scala import org.apache.spark.ml.linalg.Vectors val denseVec = Vectors.dense(1.0, 2.0, 3.0) val size = 3 val idx = Array(1,2) // locations of non-zero elements in vector val values = Array(2.0,3.0) val sparseVec = Vectors.sparse(size, idx, values) sparseVec.toDense denseVec.toSparse # in Python from pyspark.ml.linalg import Vectors denseVec = Vectors.dense(1.0, 2.0, 3.0) size = 3 idx = [1, 2] # locations of non-zero elements in vector values = [2.0, 3.0] sparseVec = Vectors.sparse(size, idx, values) WARNING Confusingly, there are similar datatypes that refer to ones that can be used in DataFrames and others that can only be used in RDDs. The RDD implementations fall under the mllib package while the DataFrame implementations fall under ml. MLlib in Action Now that we have described some of the core pieces you can expect to come across, let’s create a simple pipeline to demonstrate each of the components. We’ll use a small synthetic dataset that will help illustrate our point. Let’s read the data in and see a sample before talking about it further: // in Scala var df = spark.read.json("/data/simple-ml") df.orderBy("value2").show() # in Python df = spark.read.json("/data/simple-ml") df.orderBy("value2").show() Here’s a sample of the data: +-----+----+------+------------------+ |color| lab|value1| value2| +-----+----+------+------------------+ |green|good| 1|14.386294994851129| ... | red| bad| 16|14.386294994851129| |green|good| 12|14.386294994851129| +-----+----+------+------------------+ This dataset consists of a categorical label with two values (good or bad), a categorical variable (color), and two numerical variables. While the data is synthetic, let’s imagine that this dataset represents a company’s customer health. The “color” column represents some categorical health rating made by a customer service representative. The “lab” column represents the true customer health. The other two values are some numerical measures of activity within an application (e.g., minutes spent on site and purchases). Suppose that we want to train a classification model where we hope to predict a binary variable—the label—from the other values. TIP Apart from JSON, there are some specific data formats commonly used for supervised learning, including LIBSVM. These formats have real valued labels and sparse input data. Spark can read and write for these formats using its data source API. Here’s an example of how to read in data from a libsvm file using that Data Source API. spark.read.format("libsvm").load( "/data/sample_libsvm_data.txt") For more information on LIBSVM, see the documentation. Feature Engineering with Transformers As already mentioned, transformers help us manipulate our current columns in one way or another. Manipulating these columns is often in pursuit of building features (that we will input into our model). Transformers exist to either cut down the number of features, add more features, manipulate current ones, or simply to help us format our data correctly. Transformers add new columns to DataFrames. When we use MLlib, all inputs to machine learning algorithms (with several exceptions discussed in later chapters) in Spark must consist of type Double (for labels) and Vector[Double] (for features). The current dataset does not meet that requirement and therefore we need to transform it to the proper format. To achieve this in our example, we are going to specify an RFormula. This is a declarative language for specifying machine learning transformations and is simple to use once you understand the syntax. RFormula supports a limited subset of the R operators that in practice work quite well for simple models and manipulations (we demonstrate the manual approach to this problem in Chapter 25). The basic RFormula operators are: ~ Separate target and terms + Concat terms; “+ 0” means removing the intercept (this means that the y-intercept of the line that we will fit will be 0) Remove a term; “- 1” means removing the intercept (this means that the y-intercept of the line that we will fit will be 0—yes, this does the same thing as “+ 0” : Interaction (multiplication for numeric values, or binarized categorical values) . All columns except the target/dependent variable In order to specify transformations with this syntax, we need to import the relevant class. Then we go through the process of defining our formula. In this case we want to use all available variables (the .) and also add in the interactions between value1 and color and value2 and color, treating those as new features: // in Scala import org.apache.spark.ml.feature.RFormula val supervised = new RFormula() .setFormula("lab ~ . + color:value1 + color:value2") # in Python from pyspark.ml.feature import RFormula supervised = RFormula(formula="lab ~ . + color:value1 + color:value2") At this point, we have declaratively specified how we would like to change our data into what we will train our model on. The next step is to fit the RFormula transformer to the data to let it discover the possible values of each column. Not all transformers have this requirement but because RFormula will automatically handle categorical variables for us, it needs to determine which columns are categorical and which are not, as well as what the distinct values of the categorical columns are. For this reason, we have to call the fit method. Once we call fit, it returns a “trained” version of our transformer we can then use to actually transform our data. NOTE We’re using the RFormula transformer because it makes performing several transformations extremely easy to do. In Chapter 25, we’ll show other ways to specify a similar set of transformations and outline the component parts of the RFormula when we cover the specific transformers in MLlib. Now that we covered those details, let’s continue on and prepare our DataFrame: // in Scala val fittedRF = supervised.fit(df) val preparedDF = fittedRF.transform(df) preparedDF.show() # in Python fittedRF = supervised.fit(df) preparedDF = fittedRF.transform(df) preparedDF.show() Here’s the output from the training and transformation process: +-----+----+------+------------------+--------------------+-----+ |color| lab|value1| value2| features|label| +-----+----+------+------------------+--------------------+-----+ |green|good| 1|14.386294994851129|(10,[1,2,3,5,8],[...| 1.0| ... | red| bad| 2|14.386294994851129|(10,[0,2,3,4,7],[...| 0.0| +-----+----+------+------------------+--------------------+-----+ In the output we can see the result of our transformation—a column called features that has our previously raw data. What’s happening behind the scenes is actually pretty simple. RFormula inspects our data during the fit call and outputs an object that will transform our data according to the specified formula, which is called an RFormulaModel. This “trained” transformer always has the word Model in the type signature. When we use this transformer, Spark automatically converts our categorical variable to Doubles so that we can input it into a (yet to be specified) machine learning model. In particular, it assigns a numerical value to each possible color category, creates additional features for the interaction variables between colors and value1/value2, and puts them all into a single vector. We then call transform on that object in order to transform our input data into the expected output data. Thus far you (pre)processed the data and added some features along the way. Now it is time to actually train a model (or a set of models) on this dataset. In order to do this, you first need to prepare a test set for evaluation. TIP Having a good test set is probably the most important thing you can do to ensure you train a model you can actually use in the real world (in a dependable way). Not creating a representative test set or using your test set for hyperparameter tuning are surefire ways to create a model that does not perform well in real-world scenarios. Don’t skip creating a test set—it’s a requirement to know how well your model actually does! Let’s create a simple test set based off a random split of the data now (we’ll be using this test set throughout the remainder of the chapter): // in Scala val Array(train, test) = preparedDF.randomSplit(Array(0.7, 0.3)) # in Python train, test = preparedDF.randomSplit([0.7, 0.3]) Estimators Now that we have transformed our data into the correct format and created some valuable features, it’s time to actually fit our model. In this case we will use a classification algorithm called logistic regression. To create our classifier we instantiate an instance of LogisticRegression, using the default configuration or hyperparameters. We then set the label columns and the feature columns; the column names we are setting—label and features—are actually the default labels for all estimators in Spark MLlib, and in later chapters we omit them: // in Scala import org.apache.spark.ml.classification.LogisticRegression val lr = new LogisticRegression().setLabelCol("label").setFeaturesCol("features") # in Python from pyspark.ml.classification import LogisticRegression lr = LogisticRegression(labelCol="label",featuresCol="features") Before we actually go about training this model, let’s inspect the parameters. This is also a great way to remind yourself of the options available for each particular model: // in Scala println(lr.explainParams()) # in Python print lr.explainParams() While the output is too large to reproduce here, it shows an explanation of all of the parameters for Spark’s implementation of logistic regression. The explainParams method exists on all algorithms available in MLlib. Upon instantiating an untrained algorithm, it becomes time to fit it to data. In this case, this returns a LogisticRegressionModel: // in Scala val fittedLR = lr.fit(train) # in Python fittedLR = lr.fit(train) This code will kick off a Spark job to train the model. As opposed to the transformations that you saw throughout the book, the fitting of a machine learning model is eager and performed immediately. Once complete, you can use the model to make predictions. Logically this means tranforming features into labels. We make predictions with the transform method. For example, we can transform our training dataset to see what labels our model assigned to the training data and how those compare to the true outputs. This, again, is just another DataFrame we can manipulate. Let’s perform that prediction with the following code snippet: fittedLR.transform(train).select("label", "prediction").show() This results in: +-----+----------+ |label|prediction| +-----+----------+ | 0.0| 0.0| ... | 0.0| 0.0| +-----+----------+ Our next step would be to manually evaluate this model and calculate performance metrics like the true positive rate, false negative rate, and so on. We might then turn around and try a different set of parameters to see if those perform better. However, while this is a useful process, it can also be quite tedious. Spark helps you avoid manually trying different models and evaluation criteria by allowing you to specify your workload as a declarative pipeline of work that includes all your transformations as well as tuning your hyperparameters. A REVIEW OF HYPERPARAMETERS Although we mentioned them previously, let’s more formally define hyperparameters. Hyperparameters are configuration parameters that affect the training process, such as model architecture and regularization. They are set prior to starting training. For instance, logistic regression has a hyperparameter that determines how much regularization should be performed on our data through the training phase (regularization is a technique that pushes models against overfitting data). You’ll see in the next couple of pages that we can set up our pipeline to try different hyperparameter values (e.g., different regularization values) in order to compare different variations of the same model against one another. Pipelining Our Workflow As you probably noticed, if you are performing a lot of transformations, writing all the steps and keeping track of DataFrames ends up being quite tedious. That’s why Spark includes the Pipeline concept. A pipeline allows you to set up a dataflow of the relevant transformations that ends with an estimator that is automatically tuned according to your specifications, resulting in a tuned model ready for use. Figure 24-4 illustrates this process. Figure 24-4. Pipelining the ML workflow Note that it is essential that instances of transformers or models are not reused across different pipelines. Always create a new instance of a model before creating another pipeline. In order to make sure we don’t overfit, we are going to create a holdout test set and tune our hyperparameters based on a validation set (note that we create this validation set based on the original dataset, not the preparedDF used in the previous pages): // in Scala val Array(train, test) = df.randomSplit(Array(0.7, 0.3)) # in Python train, test = df.randomSplit([0.7, 0.3]) Now that you have a holdout set, let’s create the base stages in our pipeline. A stage simply represents a transformer or an estimator. In our case, we will have two estimators. The RFomula will first analyze our data to understand the types of input features and then transform them to create new features. Subsequently, the LogisticRegression object is the algorithm that we will train to produce a model: // in Scala val rForm = new RFormula() val lr = new LogisticRegression().setLabelCol("label").setFeaturesCol("features") # in Python rForm = RFormula() lr = LogisticRegression().setLabelCol("label").setFeaturesCol("features") We will set the potential values for the RFormula in the next section. Now instead of manually using our transformations and then tuning our model we just make them stages in the overall pipeline, as in the following code snippet: // in Scala import org.apache.spark.ml.Pipeline val stages = Array(rForm, lr) val pipeline = new Pipeline().setStages(stages) # in Python from pyspark.ml import Pipeline stages = [rForm, lr] pipeline = Pipeline().setStages(stages) Training and Evaluation Now that you arranged the logical pipeline, the next step is training. In our case, we won’t train just one model (like we did previously); we will train several variations of the model by specifying different combinations of hyperparameters that we would like Spark to test. We will then select the best model using an Evaluator that compares their predictions on our validation data. We can test different hyperparameters in the entire pipeline, even in the RFormula that we use to manipulate the raw data. This code shows how we go about doing that: // in Scala import org.apache.spark.ml.tuning.ParamGridBuilder val params = new ParamGridBuilder() .addGrid(rForm.formula, Array( "lab ~ . + color:value1", "lab ~ . + color:value1 + color:value2")) .addGrid(lr.elasticNetParam, Array(0.0, 0.5, 1.0)) .addGrid(lr.regParam, Array(0.1, 2.0)) .build() # in Python from pyspark.ml.tuning import ParamGridBuilder params = ParamGridBuilder()\ .addGrid(rForm.formula, [ "lab ~ . + color:value1", "lab ~ . + color:value1 + color:value2"])\ .addGrid(lr.elasticNetParam, [0.0, 0.5, 1.0])\ .addGrid(lr.regParam, [0.1, 2.0])\ .build() In our current paramter grid, there are three hyperparameters that will diverge from the defaults: Two different versions of the RFormula Three different options for the ElasticNet parameter Two different options for the regularization parameter This gives us a total of 12 different combinations of these parameters, which means we will be training 12 different versions of logistic regression. We explain the ElasticNet parameter as well as the regularization options in Chapter 26. Now that the grid is built, it’s time to specify our evaluation process. The evaluator allows us to automatically and objectively compare multiple models to the same evaluation metric. There are evaluators for classification and regression, covered in later chapters, but in this case we will use the BinaryClassificationEvaluator, which has a number of potential evaluation metrics, as we’ll discuss in Chapter 26. In this case we will use areaUnderROC, which is the total area under the receiver operating characteristic, a common measure of classification performance: // in Scala import org.apache.spark.ml.evaluation.BinaryClassificationEvaluator val evaluator = new BinaryClassificationEvaluator() .setMetricName("areaUnderROC") .setRawPredictionCol("prediction") .setLabelCol("label") # in Python from pyspark.ml.evaluation import BinaryClassificationEvaluator evaluator = BinaryClassificationEvaluator()\ .setMetricName("areaUnderROC")\ .setRawPredictionCol("prediction")\ .setLabelCol("label") Now that we have a pipeline that specifies how our data should be transformed, we will perform model selection to try out different hyperparameters in our logistic regression model and measure success by comparing their performance using the areaUnderROC metric. As we discussed, it is a best practice in machine learning to fit hyperparameters on a validation set (instead of your test set) to prevent overfitting. For this reason, we cannot use our holdout test set (that we created before) to tune these parameters. Luckily, Spark provides two options for performing hyperparameter tuning automatically. We can use TrainValidationSplit, which will simply perform an arbitrary random split of our data into two different groups, or CrossValidator, which performs K-fold cross-validation by splitting the dataset into k nonoverlapping, randomly partitioned folds: // in Scala import org.apache.spark.ml.tuning.TrainValidationSplit val tvs = new TrainValidationSplit() .setTrainRatio(0.75) // also the default. .setEstimatorParamMaps(params) .setEstimator(pipeline) .setEvaluator(evaluator) # in Python from pyspark.ml.tuning import TrainValidationSplit tvs = TrainValidationSplit()\ .setTrainRatio(0.75)\ .setEstimatorParamMaps(params)\ .setEstimator(pipeline)\ .setEvaluator(evaluator) Let’s run the entire pipeline we constructed. To review, running this pipeline will test out every version of the model against the validation set. Note the type of tvsFitted is TrainValidationSplitModel. Any time we fit a given model, it outputs a “model” type: // in Scala val tvsFitted = tvs.fit(train) # in Python tvsFitted = tvs.fit(train) And of course evaluate how it performs on the test set! evaluator.evaluate(tvsFitted.transform(test)) // 0.9166666666666667 We can also see a training summary for some models. To do this we extract it from the pipeline, cast it to the proper type, and print our results. The metrics available on each model are discussed throughout the next several chapters. Here’s how we can see the results: // in Scala import org.apache.spark.ml.PipelineModel import org.apache.spark.ml.classification.LogisticRegressionModel val trainedPipeline = tvsFitted.bestModel.asInstanceOf[PipelineModel] val TrainedLR = trainedPipeline.stages(1).asInstanceOf[LogisticRegressionModel] val summaryLR = TrainedLR.summary summaryLR.objectiveHistory // 0.6751425885789243, 0.5543659647777687, 0.473776... The objective history shown here provides details related to how our algorithm performed over each training iteration. This can be helpful because we can note the progress our algorithm is making toward the best model. Large jumps are typically expected at the beginning, but over time the values should become smaller and smaller, with only small amounts of variation between the values. Persisting and Applying Models Now that we trained this model, we can persist it to disk to use it for prediction purposes later on: tvsFitted.write.overwrite().save("/tmp/modelLocation") After writing out the model, we can load it into another Spark program to make predictions. To do this, we need to use a “model” version of our particular algorithm to load our persisted model from disk. If we were to use CrossValidator, we’d have to read in the persisted version as the CrossValidatorModel, and if we were to use LogisticRegression manually we would have to use LogisticRegressionModel. In this case, we use TrainValidationSplit, which outputs TrainValidationSplitModel: // in Scala import org.apache.spark.ml.tuning.TrainValidationSplitModel val model = TrainValidationSplitModel.load("/tmp/modelLocation") model.transform(test) Deployment Patterns In Spark there are several different deployment patterns for putting machine learning models into production. Figure 24-5 illustrates common workflows. Figure 24-5. The productionization process Here are the various options for how you might go about deploying a Spark model. These are the general options you should be able to link to the process illustrated in Figure 24-5. Train your machine learning (ML) model offline and then supply it with offline data. In this context, we mean offline data to be data that is stored for analysis, and not data that you need to get an answer from quickly. Spark is well suited to this sort of deployment. Train your model offline and then put the results into a database (usually a key-value store). This works well for something like recommendation but poorly for something like classification or regression where you cannot just look up a value for a given user but must calculate one based on the input. Train your ML algorithm offline, persist the model to disk, and then use that for serving. This is not a low-latency solution if you use Spark for the serving part, as the overhead of starting up a Spark job can be high, even if you’re not running on a cluster. Additionally this does not parallelize well, so you’ll likely have to put a load balancer in front of multiple model replicas and build out some REST API integration yourself. There are some interesting potential solutions to this problem, but no standards currently exist for this sort of model serving. Manually (or via some other software) convert your distributed model to one that can run much more quickly on a single machine. This works well when there is not too much manipulation of the raw data in Spark but can be hard to maintain over time. Again, there are several solutions in progress. For example, MLlib can export some models to PMML, a common model interchange format. Train your ML algorithm online and use it online. This is possible when used in conjunction with Structured Streaming, but can be complex for some models. While these are some of the options, there are many other ways of performing model deployment and management. This is an area under heavy development and many potential innovations are currently being worked on. Conclusion In this chapter we covered the core concepts behind advanced analytics and MLlib. We also showed you how to use them. The next chapter will discuss preprocessing in depth, including Spark’s tools for feature engineering and data cleaning. Then we’ll move into detailed descriptions of each algorithm available in MLlib along with some tools for graph analytics and deep learning. Chapter 25. Preprocessing and Feature Engineering Any data scientist worth her salt knows that one of the biggest challenges (and time sinks) in advanced analytics is preprocessing. It’s not that it’s particularly complicated programming, but rather that it requires deep knowledge of the data you are working with and an understanding of what your model needs in order to successfully leverage this data. This chapter covers the details of how you can use Spark to perform preprocessing and feature engineering. We’ll walk through the core requirements you’ll need to meet in order to train an MLlib model in terms of how your data is structured. We will then discuss the different tools Spark makes available for performing this kind of work. Formatting Models According to Your Use Case To preprocess data for Spark’s different advanced analytics tools, you must consider your end objective. The following list walks through the requirements for input data structure for each advanced analytics task in MLlib: In the case of most classification and regression algorithms, you want to get your data into a column of type Double to represent the label and a column of type Vector (either dense or sparse) to represent the features. In the case of recommendation, you want to get your data into a column of users, a column of items (say movies or books), and a column of ratings. In the case of unsupervised learning, a column of type Vector (either dense or sparse) is needed to represent the features. In the case of graph analytics, you will want a DataFrame of vertices and a DataFrame of edges. The best way to get your data in these formats is through transformers. Transformers are functions that accept a DataFrame as an argument and return a new DataFrame as a response. This chapter will focus on what transformers are relevant for particular use cases rather than attempting to enumerate every possible transformer. NOTE Spark provides a number of transformers as part of the org.apache.spark.ml.feature package. The corresponding package in Python is pyspark.ml.feature. New transformers are constantly popping up in Spark MLlib and therefore it is impossible to include a definitive list in this book. The most upto-date information can be found on the Spark documentation site. Before we proceed, we’re going to read in several different sample datasets, each of which has different properties we will manipulate in this chapter: // in Scala val sales = spark.read.format("csv") .option("header", "true") .option("inferSchema", "true") .load("/data/retail-data/by-day/*.csv") .coalesce(5) .where("Description IS NOT NULL") val fakeIntDF = spark.read.parquet("/data/simple-ml-integers") var simpleDF = spark.read.json("/data/simple-ml") val scaleDF = spark.read.parquet("/data/simple-ml-scaling") # in Python sales = spark.read.format("csv")\ .option("header", "true")\ .option("inferSchema", "true")\ .load("/data/retail-data/by-day/*.csv")\ .coalesce(5)\ .where("Description IS NOT NULL") fakeIntDF = spark.read.parquet("/data/simple-ml-integers") simpleDF = spark.read.json("/data/simple-ml") scaleDF = spark.read.parquet("/data/simple-ml-scaling") In addition to this realistic sales data, we’re going to use several simple synthetic datasets as well. FakeIntDF, simpleDF, and scaleDF all have very few rows. This will give you the ability to focus on the exact data manipulation we are performing instead of the various inconsistencies of any particular dataset. Because we’re going to be accessing the sales data a number of times, we’re going to cache it so we can read it efficiently from memory as opposed to reading it from disk every time we need it. Let’s also check out the first several rows of data in order to better understand what’s in the dataset: sales.cache() sales.show() +---------+---------+--------------------+--------+-------------------+--------|InvoiceNo|StockCode| Description|Quantity| InvoiceDate|UnitPr... +---------+---------+--------------------+--------+-------------------+--------| 580538| 23084| RABBIT NIGHT LIGHT| 48|2011-12-05 08:38:00| 1... ... | 580539| 22375|AIRLINE BAG VINTA...| 4|2011-12-05 08:39:00| 4... +---------+---------+--------------------+--------+-------------------+--------- NOTE It is important to note that we filtered out null values here. MLlib does not always play nicely with null values at this point in time. This is a frequent cause for problems and errors and a great first step when you are debugging. Improvements are also made with every Spark release to improve algorithm handling of null values. Transformers We discussed transformers in the previous chapter, but it’s worth reviewing them again here. Transformers are functions that convert raw data in some way. This might be to create a new interaction variable (from two other variables), to normalize a column, or to simply turn it into a Double to be input into a model. Transformers are primarily used in preprocessing or feature generation. Spark’s transformer only includes a transform method. This is because it will not change based on the input data. Figure 25-1 is a simple illustration. On the left is an input DataFrame with the column to be manipulated. On the right is the input DataFrame with a new column representing the output transformation. Figure 25-1. A Spark transformer The Tokenizer is an example of a transformer. It tokenizes a string, splitting on a given character, and has nothing to learn from our data; it simply applies a function. We’ll discuss the tokenizer in more depth later in this chapter, but here’s a small code snippet showing how a tokenizer is built to accept the input column, how it transforms the data, and then the output from that transformation: // in Scala import org.apache.spark.ml.feature.Tokenizer val tkn = new Tokenizer().setInputCol("Description") tkn.transform(sales.select("Description")).show(false) +-----------------------------------+------------------------------------------+ |Description |tok_7de4dfc81ab7__output | +-----------------------------------+------------------------------------------+ |RABBIT NIGHT LIGHT |[rabbit, night, light] | |DOUGHNUT LIP GLOSS |[doughnut, lip, gloss] | ... |AIRLINE BAG VINTAGE WORLD CHAMPION |[airline, bag, vintage, world, champion] | |AIRLINE BAG VINTAGE JET SET BROWN |[airline, bag, vintage, jet, set, brown] | +-----------------------------------+------------------------------------------+ Estimators for Preprocessing Another tool for preprocessing are estimators. An estimator is necessary when a transformation you would like to perform must be initialized with data or information about the input column (often derived by doing a pass over the input column itself). For example, if you wanted to scale the values in our column to have mean zero and unit variance, you would need to perform a pass over the entire data in order to calculate the values you would use to normalize the data to mean zero and unit variance. In effect, an estimator can be a transformer configured according to your particular input data. In simplest terms, you can either blindly apply a transformation (a “regular” transformer type) or perform a transformation based on your data (an estimator type). Figure 252 is a simple illustration of an estimator fitting to a particular input dataset, generating a transformer that is then applied to the input dataset to append a new column (of the transformed data). Figure 25-2. A Spark estimator An example of this type of estimator is the StandardScaler, which scales your input column according to the range of values in that column to have a zero mean and a variance of 1 in each dimension. For that reason it must first perform a pass over the data to create the transformer. Here’s a sample code snippet showing the entire process, as well as the output: // in Scala import org.apache.spark.ml.feature.StandardScaler val ss = new StandardScaler().setInputCol("features") ss.fit(scaleDF).transform(scaleDF).show(false) +---+--------------+------------------------------------------------------------+ |id |features |stdScal_d66fbeac10ea__output | +---+--------------+------------------------------------------------------------+ |0 |[1.0,0.1,-1.0]|[1.1952286093343936,0.02337622911060922,-0.5976143046671968]| ... |1 |[3.0,10.1,3.0]|[3.5856858280031805,2.3609991401715313,1.7928429140015902] | +---+--------------+------------------------------------------------------------+ We will use both estimators and transformers throughout and cover more about these particular estimators (and add examples in Python) later on in this chapter. Transformer Properties All transformers require you to specify, at a minimum, the inputCol and the outputCol, which represent the column name of the input and output, respectively. You set these with setInputCol and setOutputCol. There are some defaults (you can find these in the documentation), but it is a best practice to manually specify them yourself for clarity. In addition to input and output columns, all transformers have different parameters that you can tune (whenever we mention a parameter in this chapter you must set it with a set() method). In Python, we also have another method to set these values with keyword arguments to the object’s constructor. We exclude these from the examples in the next chapter for consistency. Estimators require you to fit the transformer to your particular dataset and then call transform on the resulting object. NOTE Spark MLlib stores metadata about the columns it uses in each DataFrame as an attribute on the column itself. This allows it to properly store (and annotate) that a column of Doubles may actually represent a series of categorical variables instead of continuous values. However, metadata won’t show up when you print the schema or the DataFrame. High-Level Transformers High-level transformers, such as the RFormula we saw in the previous chapter, allow you to concisely specify a number of transformations in one. These operate at a “high level”, and allow you to avoid doing data manipulations or transformations one by one. In general, you should try to use the highest level transformers you can, in order to minimize the risk of error and help you focus on the business problem instead of the smaller details of implementation. While this is not always possible, it’s a good objective. RFormula The RFormula is the easiest transfomer to use when you have “conventionally” formatted data. Spark borrows this transformer from the R language to make it simple to declaratively specify a set of transformations for your data. With this transformer, values can be either numerical or categorical and you do not need to extract values from strings or manipulate them in any way. The RFormula will automatically handle categorical inputs (specified as strings) by performing something called one-hot encoding. In brief, one-hot encoding converts a set of values into a set of binary columns specifying whether or not the data point has each particular value (we’ll discuss one-hot encoding in more depth later in the chapter). With the RFormula, numeric columns will be cast to Double but will not be one-hot encoded. If the label column is of type String, it will be first transformed to Double with StringIndexer. WARNING Automatic casting of numeric columns to Double without one-hot encoding has some important implications. If you have numerically valued categorical variables, they will only be cast to Double, implicitly specifying an order. It is important to ensure the input types correspond to the expected conversion. If you have categorical variables that really have no order relation, they should be cast to String. You can also manually index columns (see “Working with Categorical Features”). The RFormula allows you to specify your transformations in declarative syntax. It is simple to use once you understand the syntax. Currently, RFormula supports a limited subset of the R operators that in practice work quite well for simple transformations. The basic operators are: ~ Separate target and terms + Concatenate terms; “+ 0” means removing the intercept (this means the y-intercept of the line that we will fit will be 0) Remove a term; “- 1” means removing intercept (this means the y-intercept of the line that we will fit will be 0) : Interaction (multiplication for numeric values, or binarized categorical values) . All columns except the target/dependent variable RFormula also uses default columns of label and features to label, you guessed it, the label and the set of features that it outputs (for supervised machine learning). The models covered later on in this chapter by default require those column names, making it easy to pass the resulting transformed DataFrame into a model for training. If this doesn’t make sense yet, don’t worry— it’ll become clear once we actually start using models in later chapters. Let’s use RFormula in an example. In this case, we want to use all available variables (the .) and then specify an interaction between value1 and color and value2 and color as additional features to generate: // in Scala import org.apache.spark.ml.feature.RFormula val supervised = new RFormula() .setFormula("lab ~ . + color:value1 + color:value2") supervised.fit(simpleDF).transform(simpleDF).show() # in Python from pyspark.ml.feature import RFormula supervised = RFormula(formula="lab ~ . + color:value1 + color:value2") supervised.fit(simpleDF).transform(simpleDF).show() +-----+----+------+------------------+--------------------+-----+ |color| lab|value1| value2| features|label| +-----+----+------+------------------+--------------------+-----+ |green|good| 1|14.386294994851129|(10,[1,2,3,5,8],[...| 1.0| | blue| bad| 8|14.386294994851129|(10,[2,3,6,9],[8....| 0.0| ... | red| bad| 1| 38.97187133755819|(10,[0,2,3,4,7],[...| 0.0| | red| bad| 2|14.386294994851129|(10,[0,2,3,4,7],[...| 0.0| +-----+----+------+------------------+--------------------+-----+ SQL Transformers A SQLTransformer allows you to leverage Spark’s vast library of SQL-related manipulations just as you would a MLlib transformation. Any SELECT statement you can use in SQL is a valid transformation. The only thing you need to change is that instead of using the table name, you should just use the keyword TH​IS. You might want to use SQLTransformer if you want to formally codify some DataFrame manipulation as a preprocessing step, or try different SQL expressions for features during hyperparameter tuning. Also note that the output of this transformation will be appended as a column to the output DataFrame. You might want to use an SQLTransformer in order to represent all of your manipulations on the very rawest form of your data so you can version different variations of manipulations as transformers. This gives you the benefit of building and testing varying pipelines, all by simply swapping out transformers. The following is a basic example of using SQLTransformer: // in Scala import org.apache.spark.ml.feature.SQLTransformer val basicTransformation = new SQLTransformer() .setStatement(""" SELECT sum(Quantity), count(*), CustomerID FROM __THIS__ GROUP BY CustomerID """) basicTransformation.transform(sales).show() # in Python from pyspark.ml.feature import SQLTransformer basicTransformation = SQLTransformer()\ .setStatement(""" SELECT sum(Quantity), count(*), CustomerID FROM __THIS__ GROUP BY CustomerID """) basicTransformation.transform(sales).show() Here’s a sample of the output: -------------+--------+----------+ |sum(Quantity)|count(1)|CustomerID| +-------------+--------+----------+ | 119| 62| 14452.0| ... | 138| 18| 15776.0| +-------------+--------+----------+ For extensive samples of these transformations, refer back to Part II. VectorAssembler The VectorAssembler is a tool you’ll use in nearly every single pipeline you generate. It helps concatenate all your features into one big vector you can then pass into an estimator. It’s used typically in the last step of a machine learning pipeline and takes as input a number of columns of Boolean, Double, or Vector. This is particularly helpful if you’re going to perform a number of manipulations using a variety of transformers and need to gather all of those results together. The output from the following code snippet will make it clear how this works: // in Scala import org.apache.spark.ml.feature.VectorAssembler val va = new VectorAssembler().setInputCols(Array("int1", "int2", "int3")) va.transform(fakeIntDF).show() # in Python from pyspark.ml.feature import VectorAssembler va = VectorAssembler().setInputCols(["int1", "int2", "int3"]) va.transform(fakeIntDF).show() +----+----+----+--------------------------------------------+ |int1|int2|int3|VectorAssembler_403ab93eacd5585ddd2d__output| +----+----+----+--------------------------------------------+ | 1| 2| 3| [1.0,2.0,3.0]| | 4| 5| 6| [4.0,5.0,6.0]| | 7| 8| 9| [7.0,8.0,9.0]| +----+----+----+--------------------------------------------+ Working with Continuous Features Continuous features are just values on the number line, from positive infinity to negative infinity. There are two common transformers for continuous features. First, you can convert continuous features into categorical features via a process called bucketing, or you can scale and normalize your features according to several different requirements. These transformers will only work on Double types, so make sure you’ve turned any other numerical values to Double: // in Scala val contDF = spark.range(20).selectExpr("cast(id as double)") # in Python contDF = spark.range(20).selectExpr("cast(id as double)") Bucketing The most straightforward approach to bucketing or binning is using the Bucketizer. This will split a given continuous feature into the buckets of your designation. You specify how buckets should be created via an array or list of Double values. This is useful because you may want to simplify the features in your dataset or simplify their representations for interpretation later on. For example, imagine you have a column that represents a person’s weight and you would like to predict some value based on this information. In some cases, it might be simpler to create three buckets of “overweight,” “average,” and “underweight.” To specify the bucket, set its borders. For example, setting splits to 5.0, 10.0, 250.0 on our contDF will actually fail because we don’t cover all possible input ranges. When specifying your bucket points, the values you pass into splits must satisfy three requirements: The minimum value in your splits array must be less than the minimum value in your DataFrame. The maximum value in your splits array must be greater than the maximum value in your DataFrame. You need to specify at a minimum three values in the splits array, which creates two buckets. WARNING The Bucketizer can be confusing because we specify bucket borders via the splits method, but these are not actually splits. To cover all possible ranges, scala.Double.NegativeInfinity might be another split option, with scala.Double.PositiveInfinity to cover all possible ranges outside of the inner splits. In Python we specify this in the following way: float("inf"), float("-inf"). In order to handle null or NaN values, we must specify the handleInvalid parameter as a certain value. We can either keep those values (keep), error or null, or skip those rows. Here’s an example of using bucketing: // in Scala import org.apache.spark.ml.feature.Bucketizer val bucketBorders = Array(-1.0, 5.0, 10.0, 250.0, 600.0) val bucketer = new Bucketizer().setSplits(bucketBorders).setInputCol("id") bucketer.transform(contDF).show() # in Python from pyspark.ml.feature import Bucketizer bucketBorders = [-1.0, 5.0, 10.0, 250.0, 600.0] bucketer = Bucketizer().setSplits(bucketBorders).setInputCol("id") bucketer.transform(contDF).show() +----+---------------------------------------+ | id|Bucketizer_4cb1be19f4179cc2545d__output| +----+---------------------------------------+ | 0.0| 0.0| ... |10.0| 2.0| |11.0| 2.0| ... +----+---------------------------------------+ In addition to splitting based on hardcoded values, another option is to split based on percentiles in our data. This is done with QuantileDiscretizer, which will bucket the values into userspecified buckets with the splits being determined by approximate quantiles values. For instance, the 90th quantile is the point in your data at which 90% of the data is below that value. You can control how finely the buckets should be split by setting the relative error for the approximate quantiles calculation using setRelativeError. Spark does this is by allowing you to specify the number of buckets you would like out of the data and it will split up your data accordingly. The following is an example: // in Scala import org.apache.spark.ml.feature.QuantileDiscretizer val bucketer = new QuantileDiscretizer().setNumBuckets(5).setInputCol("id") val fittedBucketer = bucketer.fit(contDF) fittedBucketer.transform(contDF).show() # in Python from pyspark.ml.feature import QuantileDiscretizer bucketer = QuantileDiscretizer().setNumBuckets(5).setInputCol("id") fittedBucketer = bucketer.fit(contDF) fittedBucketer.transform(contDF).show() +----+----------------------------------------+ | id|quantileDiscretizer_cd87d1a1fb8e__output| +----+----------------------------------------+ | 0.0| 0.0| ... | 6.0| 1.0| | 7.0| 2.0| ... |14.0| 3.0| |15.0| 4.0| ... +----+----------------------------------------+ Advanced bucketing techniques The techniques descriubed here are the most common ways of bucketing data, but there are a number of other ways that exist in Spark today. All of these processes are the same from a data flow perspective: start with continuous data and place them in buckets so that they become categorical. Differences arise depending on the algorithm used to compute these buckets. The simple examples we just looked at are easy to intepret and work with, but more advanced techniques such as locality sensitivity hashing (LSH) are also available in MLlib. Scaling and Normalization We saw how we can use bucketing to create groups out of continuous variables. Another common task is to scale and normalize continuous data. While not always necessary, doing so is usually a best practice. You might want to do this when your data contains a number of columns based on different scales. For instance, say we have a DataFrame with two columns: weight (in ounces) and height (in feet). If you don’t scale or normalize, the algorithm will be less sensitive to variations in height because height values in feet are much lower than weight values in ounces. That’s an example where you should scale your data. An example of normalization might involve transforming the data so that each point’s value is a representation of its distance from the mean of that column. Using the same example from before, we might want to know how far a given individual’s height is from the mean height. Many algorithms assume that their input data is normalized. As you might imagine, there are a multitude of algorithms we can apply to our data to scale or normalize it. Enumerating them all is unnecessary here because they are covered in many other texts and machine learning libraries. If you’re unfamiliar with the concept in detail, check out any of the books referenced in the previous chapter. Just keep in mind the fundamental goal—we want our data on the same scale so that values can easily be compared to one another in a sensible way. In MLlib, this is always done on columns of type Vector. MLlib will look across all the rows in a given column (of type Vector) and then treat every dimension in those vectors as its own particular column. It will then apply the scaling or normalization function on each dimension separately. A simple example might be the following vectors in a column: 1,2 3,4 When we apply our scaling (but not normalization) function, the “3” and the “1” will be adjusted according to those two values while the “2” and the “4” will be adjusted according to one another. This is commonly referred to as component-wise comparisons. StandardScaler The StandardScaler standardizes a set of features to have zero mean and a standard deviation of 1. The flag withStd will scale the data to unit standard deviation while the flag withMean (false by default) will center the data prior to scaling it. WARNING Centering can be very expensive on sparse vectors because it generally turns them into dense vectors, so be careful before centering your data. Here’s an example of using a StandardScaler: // in Scala import org.apache.spark.ml.feature.StandardScaler val sScaler = new StandardScaler().setInputCol("features") sScaler.fit(scaleDF).transform(scaleDF).show() # in Python from pyspark.ml.feature import StandardScaler sScaler = StandardScaler().setInputCol("features") sScaler.fit(scaleDF).transform(scaleDF).show() The output is shown below: +---+--------------+------------------------------------------------------------+ |id |features |StandardScaler_41aaa6044e7c3467adc3__output | +---+--------------+------------------------------------------------------------+ |0 |[1.0,0.1,-1.0]|[1.1952286093343936,0.02337622911060922,-0.5976143046671968]| ... |1 |[3.0,10.1,3.0]|[3.5856858280031805,2.3609991401715313,1.7928429140015902] | +---+--------------+------------------------------------------------------------+ MinMaxScaler The MinMaxScaler will scale the values in a vector (component wise) to the proportional values on a scale from a given min value to a max value. If you specify the minimum value to be 0 and the maximum value to be 1, then all the values will fall in between 0 and 1: // in Scala import org.apache.spark.ml.feature.MinMaxScaler val minMax = new MinMaxScaler().setMin(5).setMax(10).setInputCol("features") val fittedminMax = minMax.fit(scaleDF) fittedminMax.transform(scaleDF).show() # in Python from pyspark.ml.feature import MinMaxScaler minMax = MinMaxScaler().setMin(5).setMax(10).setInputCol("features") fittedminMax = minMax.fit(scaleDF) fittedminMax.transform(scaleDF).show() +---+--------------+-----------------------------------------+ | id| features|MinMaxScaler_460cbafafbe6b9ab7c62__output| +---+--------------+-----------------------------------------+ | 0|[1.0,0.1,-1.0]| [5.0,5.0,5.0]| ... | 1|[3.0,10.1,3.0]| [10.0,10.0,10.0]| +---+--------------+-----------------------------------------+ MaxAbsScaler The max absolute scaler (MaxAbsScaler) scales the data by dividing each value by the maximum absolute value in this feature. All values therefore end up between −1 and 1. This transformer does not shift or center the data at all in the process: // in Scala import org.apache.spark.ml.feature.MaxAbsScaler val maScaler = new MaxAbsScaler().setInputCol("features") val fittedmaScaler = maScaler.fit(scaleDF) fittedmaScaler.transform(scaleDF).show() # in Python from pyspark.ml.feature import MaxAbsScaler maScaler = MaxAbsScaler().setInputCol("features") fittedmaScaler = maScaler.fit(scaleDF) fittedmaScaler.transform(scaleDF).show() +---+--------------+----------------------------------------------------------+ |id |features |MaxAbsScaler_402587e1d9b6f268b927__output | +---+--------------+----------------------------------------------------------+ |0 |[1.0,0.1,-1.0]|[0.3333333333333333,0.009900990099009901,-0.3333333333333]| ... |1 |[3.0,10.1,3.0]|[1.0,1.0,1.0] | +---+--------------+----------------------------------------------------------+ ElementwiseProduct The ElementwiseProduct allows us to scale each value in a vector by an arbitrary value. For example, given the vector below and the row “1, 0.1, -1” the output will be “10, 1.5, -20.” Naturally the dimensions of the scaling vector must match the dimensions of the vector inside the relevant column: // in Scala import org.apache.spark.ml.feature.ElementwiseProduct import org.apache.spark.ml.linalg.Vectors val scaleUpVec = Vectors.dense(10.0, 15.0, 20.0) val scalingUp = new ElementwiseProduct() .setScalingVec(scaleUpVec) .setInputCol("features") scalingUp.transform(scaleDF).show() # in Python from pyspark.ml.feature import ElementwiseProduct from pyspark.ml.linalg import Vectors scaleUpVec = Vectors.dense(10.0, 15.0, 20.0) scalingUp = ElementwiseProduct()\ .setScalingVec(scaleUpVec)\ .setInputCol("features") scalingUp.transform(scaleDF).show() +---+--------------+-----------------------------------------------+ | id| features|ElementwiseProduct_42b29ea5a55903e9fea6__output| +---+--------------+-----------------------------------------------+ | 0|[1.0,0.1,-1.0]| [10.0,1.5,-20.0]| ... | 1|[3.0,10.1,3.0]| [30.0,151.5,60.0]| +---+--------------+-----------------------------------------------+ Normalizer The normalizer allows us to scale multidimensional vectors using one of several power norms, set through the parameter “p”. For example, we can use the Manhattan norm (or Manhattan distance) with p = 1, Euclidean norm with p = 2, and so on. The Manhattan distance is a measure of distance where you can only travel from point to point along the straight lines of an axis (like the streets in Manhattan). Here’s an example of using the Normalizer: // in Scala import org.apache.spark.ml.feature.Normalizer val manhattanDistance = new Normalizer().setP(1).setInputCol("features") manhattanDistance.transform(scaleDF).show() # in Python from pyspark.ml.feature import Normalizer manhattanDistance = Normalizer().setP(1).setInputCol("features") manhattanDistance.transform(scaleDF).show() +---+--------------+-------------------------------+ | id| features|normalizer_1bf2cd17ed33__output| +---+--------------+-------------------------------+ | 0|[1.0,0.1,-1.0]| [0.47619047619047...| | 1| [2.0,1.1,1.0]| [0.48780487804878...| | 0|[1.0,0.1,-1.0]| [0.47619047619047...| | 1| [2.0,1.1,1.0]| [0.48780487804878...| | 1|[3.0,10.1,3.0]| [0.18633540372670...| +---+--------------+-------------------------------+ Working with Categorical Features The most common task for categorical features is indexing. Indexing converts a categorical variable in a column to a numerical one that you can plug into machine learning algorithms. While this is conceptually simple, there are some catches that are important to keep in mind so that Spark can do this in a stable and repeatable manner. In general, we recommend re-indexing every categorical variable when pre-processing just for consistency’s sake. This can be helpful in maintaining your models over the long run as your encoding practices may change over time. StringIndexer The simplest way to index is via the StringIndexer, which maps strings to different numerical IDs. Spark’s StringIndexer also creates metadata attached to the DataFrame that specify what inputs correspond to what outputs. This allows us later to get inputs back from their respective index values: // in Scala import org.apache.spark.ml.feature.StringIndexer val lblIndxr = new StringIndexer().setInputCol("lab").setOutputCol("labelInd") val idxRes = lblIndxr.fit(simpleDF).transform(simpleDF) idxRes.show() # in Python from pyspark.ml.feature import StringIndexer lblIndxr = StringIndexer().setInputCol("lab").setOutputCol("labelInd") idxRes = lblIndxr.fit(simpleDF).transform(simpleDF) idxRes.show() +-----+----+------+------------------+--------+ |color| lab|value1| value2|labelInd| +-----+----+------+------------------+--------+ |green|good| 1|14.386294994851129| 1.0| ... | red| bad| 2|14.386294994851129| 0.0| +-----+----+------+------------------+--------+ We can also apply StringIndexer to columns that are not strings, in which case, they will be converted to strings before being indexed: // in Scala val valIndexer = new StringIndexer() .setInputCol("value1") .setOutputCol("valueInd") valIndexer.fit(simpleDF).transform(simpleDF).show() # in Python valIndexer = StringIndexer().setInputCol("value1").setOutputCol("valueInd") valIndexer.fit(simpleDF).transform(simpleDF).show() +-----+----+------+------------------+--------+ |color| lab|value1| value2|valueInd| +-----+----+------+------------------+--------+ |green|good| 1|14.386294994851129| 1.0| ... | red| bad| 2|14.386294994851129| 0.0| +-----+----+------+------------------+--------+ Keep in mind that the StringIndexer is an estimator that must be fit on the input data. This means it must see all inputs to select a mapping of inputs to IDs. If you train a StringIndexer on inputs “a,” “b,” and “c” and then go to use it against input “d,” it will throw an error by default. Another option is to skip the entire row if the input value was not a value seen during training. Going along with the previous example, an input value of “d” would cause that row to be skipped entirely. We can set this option before or after training the indexer or pipeline. More options may be added to this feature in the future but as of Spark 2.2, you can only skip or throw an error on invalid inputs. valIndexer.setHandleInvalid("skip") valIndexer.fit(simpleDF).setHandleInvalid("skip") Converting Indexed Values Back to Text When inspecting your machine learning results, you’re likely going to want to map back to the original values. Since MLlib classification models make predictions using the indexed values, this conversion is useful for converting model predictions (indices) back to the original categories. We can do this with IndexToString. You’ll notice that we do not have to input our value to the String key; Spark’s MLlib maintains this metadata for you. You can optionally specify the outputs. // in Scala import org.apache.spark.ml.feature.IndexToString val labelReverse = new IndexToString().setInputCol("labelInd") labelReverse.transform(idxRes).show() # in Python from pyspark.ml.feature import IndexToString labelReverse = IndexToString().setInputCol("labelInd") labelReverse.transform(idxRes).show() +-----+----+------+------------------+--------+--------------------------------+ |color| lab|value1| value2|labelInd|IndexToString_415...2a0d__output| +-----+----+------+------------------+--------+--------------------------------+ |green|good| 1|14.386294994851129| 1.0| good| ... | red| bad| 2|14.386294994851129| 0.0| bad| +-----+----+------+------------------+--------+--------------------------------+ Indexing in Vectors VectorIndexer is a helpful tool for working with categorical variables that are already found inside of vectors in your dataset. This tool will automatically find categorical features inside of your input vectors and convert them to categorical features with zero-based category indices. For example, in the following DataFrame, the first column in our Vector is a categorical variable with two different categories while the rest of the variables are continuous. By setting maxCategories to 2 in our VectorIndexer, we are instructing Spark to take any column in our vector with two or less distinct values and convert it to a categorical variable. This can be helpful when you know how many unique values there are in your largest category because you can specify this and it will automatically index the values accordingly. Conversely, Spark changes the data based on this parameter, so if you have continuous variables that don’t appear particularly continuous (lots of repeated values) these can be unintentionally converted to categorical variables if there are too few unique values. // in Scala import org.apache.spark.ml.feature.VectorIndexer import org.apache.spark.ml.linalg.Vectors val idxIn = spark.createDataFrame(Seq( (Vectors.dense(1, 2, 3),1), (Vectors.dense(2, 5, 6),2), (Vectors.dense(1, 8, 9),3) )).toDF("features", "label") val indxr = new VectorIndexer() .setInputCol("features") .setOutputCol("idxed") .setMaxCategories(2) indxr.fit(idxIn).transform(idxIn).show # in Python from pyspark.ml.feature import VectorIndexer from pyspark.ml.linalg import Vectors idxIn = spark.createDataFrame([ (Vectors.dense(1, 2, 3),1), (Vectors.dense(2, 5, 6),2), (Vectors.dense(1, 8, 9),3) ]).toDF("features", "label") indxr = VectorIndexer()\ .setInputCol("features")\ .setOutputCol("idxed")\ .setMaxCategories(2) indxr.fit(idxIn).transform(idxIn).show() +-------------+-----+-------------+ | features|label| idxed| +-------------+-----+-------------+ |[1.0,2.0,3.0]| 1|[0.0,2.0,3.0]| |[2.0,5.0,6.0]| 2|[1.0,5.0,6.0]| |[1.0,8.0,9.0]| 3|[0.0,8.0,9.0]| +-------------+-----+-------------+ One-Hot Encoding Indexing categorical variables is only half of the story. One-hot encoding is an extremely common data transformation performed after indexing categorical variables. This is because indexing does not always represent our categorical variables in the correct way for downstream models to process. For instance, when we index our “color” column, you will notice that some colors have a higher value (or index number) than others (in our case, blue is 1 and green is 2). This is incorrect because it gives the mathematical appearance that the input to the machine learning algorithm seems to specify that green > blue, which makes no sense in the case of the current categories. To avoid this, we use OneHotEncoder, which will convert each distinct value to a Boolean flag (1 or 0) as a component in a vector. When we encode the color value, then we can see these are no longer ordered, making them easier for downstream models (e.g., a linear model) to process: // in Scala import org.apache.spark.ml.feature.{StringIndexer, OneHotEncoder} val lblIndxr = new StringIndexer().setInputCol("color").setOutputCol("colorInd") val colorLab = lblIndxr.fit(simpleDF).transform(simpleDF.select("color")) val ohe = new OneHotEncoder().setInputCol("colorInd") ohe.transform(colorLab).show() # in Python from pyspark.ml.feature import OneHotEncoder, StringIndexer lblIndxr = StringIndexer().setInputCol("color").setOutputCol("colorInd") colorLab = lblIndxr.fit(simpleDF).transform(simpleDF.select("color")) ohe = OneHotEncoder().setInputCol("colorInd") ohe.transform(colorLab).show() +-----+--------+------------------------------------------+ |color|colorInd|OneHotEncoder_46b5ad1ef147bb355612__output| +-----+--------+------------------------------------------+ |green| 1.0| (2,[1],[1.0])| | blue| 2.0| (2,[],[])| ... | red| 0.0| (2,[0],[1.0])| | red| 0.0| (2,[0],[1.0])| +-----+--------+------------------------------------------+ Text Data Transformers Text is always tricky input because it often requires lots of manipulation to map to a format that a machine learning model will be able to use effectively. There are generally two kinds of texts you’ll see: free-form text and string categorical variables. This section primarily focuses on freeform text because we already discussed categorical variables. Tokenizing Text Tokenization is the process of converting free-form text into a list of “tokens” or individual words. The easiest way to do this is by using the Tokenizer class. This transformer will take a string of words, separated by whitespace, and convert them into an array of words. For example, in our dataset we might want to convert the Description field into a list of tokens. // in Scala import org.apache.spark.ml.feature.Tokenizer val tkn = new Tokenizer().setInputCol("Description").setOutputCol("DescOut") val tokenized = tkn.transform(sales.select("Description")) tokenized.show(false) # in Python from pyspark.ml.feature import Tokenizer tkn = Tokenizer().setInputCol("Description").setOutputCol("DescOut") tokenized = tkn.transform(sales.select("Description")) tokenized.show(20, False) +-----------------------------------+------------------------------------------+ |Description DescOut | +-----------------------------------+------------------------------------------+ |RABBIT NIGHT LIGHT |[rabbit, night, light] | |DOUGHNUT LIP GLOSS |[doughnut, lip, gloss] | ... |AIRLINE BAG VINTAGE WORLD CHAMPION |[airline, bag, vintage, world, champion] | |AIRLINE BAG VINTAGE JET SET BROWN |[airline, bag, vintage, jet, set, brown] | +-----------------------------------+------------------------------------------+ We can also create a Tokenizer that is not just based white space but a regular expression with the RegexTokenizer. The format of the regular expression should conform to the Java Regular Expression (RegEx) syntax: // in Scala import org.apache.spark.ml.feature.RegexTokenizer val rt = new RegexTokenizer() .setInputCol("Description") .setOutputCol("DescOut") .setPattern(" ") // simplest expression .setToLowercase(true) rt.transform(sales.select("Description")).show(false) # in Python from pyspark.ml.feature import RegexTokenizer rt = RegexTokenizer()\ .setInputCol("Description")\ .setOutputCol("DescOut")\ .setPattern(" ")\ .setToLowercase(True) rt.transform(sales.select("Description")).show(20, False) +-----------------------------------+------------------------------------------+ |Description DescOut | +-----------------------------------+------------------------------------------+ |RABBIT NIGHT LIGHT |[rabbit, night, light] | |DOUGHNUT LIP GLOSS |[doughnut, lip, gloss] | ... |AIRLINE BAG VINTAGE WORLD CHAMPION |[airline, bag, vintage, world, champion] | |AIRLINE BAG VINTAGE JET SET BROWN |[airline, bag, vintage, jet, set, brown] | +-----------------------------------+------------------------------------------+ Another way of using the RegexTokenizer is to use it to output values matching the provided pattern instead of using it as a gap. We do this by setting the gaps parameter to false. Doing this with a space as a pattern returns all the spaces, which is not too useful, but if we made our pattern capture individual words, we could return those: // in Scala import org.apache.spark.ml.feature.RegexTokenizer val rt = new RegexTokenizer() .setInputCol("Description") .setOutputCol("DescOut") .setPattern(" ") .setGaps(false) .setToLowercase(true) rt.transform(sales.select("Description")).show(false) # in Python from pyspark.ml.feature import RegexTokenizer rt = RegexTokenizer()\ .setInputCol("Description")\ .setOutputCol("DescOut")\ .setPattern(" ")\ .setGaps(False)\ .setToLowercase(True) rt.transform(sales.select("Description")).show(20, False) +-----------------------------------+------------------+ |Description DescOut | +-----------------------------------+------------------+ |RABBIT NIGHT LIGHT |[ , ] | |DOUGHNUT LIP GLOSS |[ , , ] | ... |AIRLINE BAG VINTAGE WORLD CHAMPION |[ , , , , ] | |AIRLINE BAG VINTAGE JET SET BROWN |[ , , , , ] | +-----------------------------------+------------------+ Removing Common Words A common task after tokenization is to filter stop words, common words that are not relevant in many kinds of analysis and should thus be removed. Frequently occurring stop words in English include “the,” “and,” and “but.” Spark contains a list of default stop words you can see by calling the following method, which can be made case insensitive if necessary (as of Spark 2.2, supported languages for stopwords are “danish,” “dutch,” “english,” “finnish,” “french,” “german,” “hungarian,” “italian,” “norwegian,” “portuguese,” “russian,” “spanish,” “swedish,” and “turkish”): // in Scala import org.apache.spark.ml.feature.StopWordsRemover val englishStopWords = StopWordsRemover.loadDefaultStopWords("english") val stops = new StopWordsRemover() .setStopWords(englishStopWords) .setInputCol("DescOut") stops.transform(tokenized).show() # in Python from pyspark.ml.feature import StopWordsRemover englishStopWords = StopWordsRemover.loadDefaultStopWords("english") stops = StopWordsRemover()\ .setStopWords(englishStopWords)\ .setInputCol("DescOut") stops.transform(tokenized).show() The following output shows how this works: +--------------------+--------------------+------------------------------------+ | Description| DescOut|StopWordsRemover_4ab18...6ed__output| +--------------------+--------------------+------------------------------------+ ... |SET OF 4 KNICK KN...|[set, of, 4, knic...| [set, 4, knick, k...| ... +--------------------+--------------------+------------------------------------+ Notice how the word of is removed in the output column. That’s because it’s such a common word that it isn’t relevant to any downstream manipulation and simply adds noise to our dataset. Creating Word Combinations Tokenizing our strings and filtering stop words leaves us with a clean set of words to use as features. It is often of interest to look at combinations of words, usually by looking at colocated words. Word combinations are technically referred to as n-grams—that is, sequences of words of length n. An n-gram of length 1 is called a unigrams; those of length 2 are called bigrams, and those of length 3 are called trigrams (anything above those are just four-gram, five-gram, etc.), Order matters with n-gram creation, so converting a sentence with three words into bigram representation would result in two bigrams. The goal when creating n-grams is to better capture sentence structure and more information than can be gleaned by simply looking at all words individually. Let’s create some n-grams to illustrate this concept. The bigrams of “Big Data Processing Made Simple” are: “Big Data” “Data Processing” “Processing Made” “Made Simple” While the trigrams are: “Big Data Processing” “Data Processing Made” “Procesing Made Simple” With n-grams, we can look at sequences of words that commonly co-occur and use them as inputs to a machine learning algorithm. These can create better features than simply looking at all of the words individually (say, tokenized on a space character): // in Scala import org.apache.spark.ml.feature.NGram val unigram = new NGram().setInputCol("DescOut").setN(1) val bigram = new NGram().setInputCol("DescOut").setN(2) unigram.transform(tokenized.select("DescOut")).show(false) bigram.transform(tokenized.select("DescOut")).show(false) # in Python from pyspark.ml.feature import NGram unigram = NGram().setInputCol("DescOut").setN(1) bigram = NGram().setInputCol("DescOut").setN(2) unigram.transform(tokenized.select("DescOut")).show(False) bigram.transform(tokenized.select("DescOut")).show(False) +-----------------------------------------+------------------------------------DescOut |ngram_104c4da6a01b__output ... +-----------------------------------------+------------------------------------|[rabbit, night, light] |[rabbit, night, light] ... |[doughnut, lip, gloss] |[doughnut, lip, gloss] ... ... |[airline, bag, vintage, world, champion] |[airline, bag, vintage, world, cha... |[airline, bag, vintage, jet, set, brown] |[airline, bag, vintage, jet, set, ... +-----------------------------------------+------------------------------------- And the result for bigrams: +------------------------------------------+-----------------------------------DescOut |ngram_6e68fb3a642a__output ... +------------------------------------------+-----------------------------------|[rabbit, night, light] |[rabbit night, night light] ... |[doughnut, lip, gloss] |[doughnut lip, lip gloss] ... ... |[airline, bag, vintage, world, champion] |[airline bag, bag vintage, vintag... |[airline, bag, vintage, jet, set, brown] |[airline bag, bag vintage, vintag... +------------------------------------------+------------------------------------ Converting Words into Numerical Representations Once you have word features, it’s time to start counting instances of words and word combinations for use in our models. The simplest way is just to include binary counts of a word in a given document (in our case, a row). Essentially, we’re measuring whether or not each row contains a given word. This is a simple way to normalize for document sizes and occurrence counts and get numerical features that allow us to classify documents based on content. In addition, we can count words using a CountVectorizer, or reweigh them according to the prevalence of a given word in all the documents using a TF–IDF transformation (discussed next). A CountVectorizer operates on our tokenized data and does two things: 1. During the fit process, it finds the set of words in all the documents and then counts the occurrences of those words in those documents. 2. It then counts the occurrences of a given word in each row of the DataFrame column during the transformation process and outputs a vector with the terms that occur in that row. Conceptually this tranformer treats every row as a document and every word as a term and the total collection of all terms as the vocabulary. These are all tunable parameters, meaning we can set the minimum term frequency (minTF) for the term to be included in the vocabulary (effectively removing rare words from the vocabulary); minimum number of documents a term must appear in (minDF) before being included in the vocabulary (another way to remove rare words from the vocabulary); and finally, the total maximum vocabulary size (vocabSize). Lastly, by default the CountVectorizer will output the counts of a term in a document. To just return whether or not a word exists in a document, we can use setBinary(true). Here’s an example of using CountVectorizer: // in Scala import org.apache.spark.ml.feature.CountVectorizer val cv = new CountVectorizer() .setInputCol("DescOut") .setOutputCol("countVec") .setVocabSize(500) .setMinTF(1) .setMinDF(2) val fittedCV = cv.fit(tokenized) fittedCV.transform(tokenized).show(false) # in Python from pyspark.ml.feature import CountVectorizer cv = CountVectorizer()\ .setInputCol("DescOut")\ .setOutputCol("countVec")\ .setVocabSize(500)\ .setMinTF(1)\ .setMinDF(2) fittedCV = cv.fit(tokenized) fittedCV.transform(tokenized).show(False) While the output looks a little complicated, it’s actually just a sparse vector that contains the total vocabulary size, the index of the word in the vocabulary, and then the counts of that particular word: +---------------------------------+--------------------------------------------+ DescOut |countVec | +---------------------------------+--------------------------------------------+ |[rabbit, night, light] |(500,[150,185,212],[1.0,1.0,1.0]) | |[doughnut, lip, gloss] |(500,[462,463,492],[1.0,1.0,1.0]) | ... |[airline, bag, vintage, world,...|(500,[2,6,328],[1.0,1.0,1.0]) | |[airline, bag, vintage, jet, s...|(500,[0,2,6,328,405],[1.0,1.0,1.0,1.0,1.0]) | +---------------------------------+--------------------------------------------+ Term frequency–inverse document frequency Another way to approach the problem of converting text into a numerical representation is to use term frequency–inverse document frequency (TF–IDF). In simplest terms, TF–IDF measures how often a word occurs in each document, weighted according to how many documents that word occurs in. The result is that words that occur in a few documents are given more weight than words that occur in many documents. In practice, a word like “the” would be weighted very low because of its prevalence while a more specialized word like “streaming” would occur in fewer documents and thus would be weighted higher. In a way, TF–IDF helps find documents that share similar topics. Let’s take a look at an example—first, we’ll inspect some of the documents in our data containing the word “red”: // in Scala val tfIdfIn = tokenized .where("array_contains(DescOut, 'red')") .select("DescOut") .limit(10) tfIdfIn.show(false) # in Python tfIdfIn = tokenized\ .where("array_contains(DescOut, 'red')")\ .select("DescOut")\ .limit(10) tfIdfIn.show(10, False) +---------------------------------------+ DescOut | +---------------------------------------+ |[gingham, heart, , doorstop, red] | ... |[red, retrospot, oven, glove] | |[red, retrospot, plate] | +---------------------------------------+ We can see some overlapping words in these documents, but these words provide at least a rough topic-like representation. Now let’s input that into TF–IDF. To do this, we’re going to hash each word and convert it to a numerical representation, and then weigh each word in the voculary according to the inverse document frequency. Hashing is a similar process as CountVectorizer, but is irreversible—that is, from our output index for a word, we cannot get our input word (multiple words might map to the same output index): // in Scala import org.apache.spark.ml.feature.{HashingTF, IDF} val tf = new HashingTF() .setInputCol("DescOut") .setOutputCol("TFOut") .setNumFeatures(10000) val idf = new IDF() .setInputCol("TFOut") .setOutputCol("IDFOut") .setMinDocFreq(2) # in Python from pyspark.ml.feature import HashingTF, IDF tf = HashingTF()\ .setInputCol("DescOut")\ .setOutputCol("TFOut")\ .setNumFeatures(10000) idf = IDF()\ .setInputCol("TFOut")\ .setOutputCol("IDFOut")\ .setMinDocFreq(2) // in Scala idf.fit(tf.transform(tfIdfIn)).transform(tf.transform(tfIdfIn)).show(false) # in Python idf.fit(tf.transform(tfIdfIn)).transform(tf.transform(tfIdfIn)).show(10, False) While the output is too large to include here, notice that a certain value is assigned to “red” and that this value appears in every document. Also note that this term is weighted extremely low because it appears in every document. The output format is a sparse Vector we can subsequently input into a machine learning model in a form like this: (10000,[2591,4291,4456],[1.0116009116784799,0.0,0.0]) This vector is represented using three different values: the total vocabulary size, the hash of every word appearing in the document, and the weighting of each of those terms. This is similar to the CountVectorizer output. Word2Vec Word2Vec is a deep learning–based tool for computing a vector representation of a set of words. The goal is to have similar words close to one another in this vector space, so we can then make generalizations about the words themselves. This model is easy to train and use, and has been shown to be useful in a number of natural language processing applications, including entity recognition, disambiguation, parsing, tagging, and machine translation. Word2Vec is notable for capturing relationships between words based on their semantics. For example, if v~king, v~queen, v~man, and v~women represent the vectors for those four words, then we will often get a representation where v~king − v~man + v~woman ~= v~queen. To do this, Word2Vec uses a technique called “skip-grams” to convert a sentence of words into a vector representation (optionally of a specific size). It does this by building a vocabulary, and then for every sentence, it removes a token and trains the model to predict the missing token in the "n-gram” representation. Word2Vec works best with continuous, free-form text in the form of tokens. Here’s a simple example from the documentation: // in Scala import org.apache.spark.ml.feature.Word2Vec import org.apache.spark.ml.linalg.Vector import org.apache.spark.sql.Row // Input data: Each row is a bag of words from a sentence or document. val documentDF = spark.createDataFrame(Seq( "Hi I heard about Spark".split(" "), "I wish Java could use case classes".split(" "), "Logistic regression models are neat".split(" ") ).map(Tuple1.apply)).toDF("text") // Learn a mapping from words to Vectors. val word2Vec = new Word2Vec() .setInputCol("text") .setOutputCol("result") .setVectorSize(3) .setMinCount(0) val model = word2Vec.fit(documentDF) val result = model.transform(documentDF) result.collect().foreach { case Row(text: Seq[_], features: Vector) => println(s"Text: [${text.mkString(", ")}] => \nVector: $features\n") } # in Python from pyspark.ml.feature import Word2Vec # Input data: Each row is a bag of words from a sentence or document. documentDF = spark.createDataFrame([ ("Hi I heard about Spark".split(" "), ), ("I wish Java could use case classes".split(" "), ), ("Logistic regression models are neat".split(" "), ) ], ["text"]) # Learn a mapping from words to Vectors. word2Vec = Word2Vec(vectorSize=3, minCount=0, inputCol="text", outputCol="result") model = word2Vec.fit(documentDF) result = model.transform(documentDF) for row in result.collect(): text, vector = row print("Text: [%s] => \nVector: %s\n" % (", ".join(text), str(vector))) Text: [Hi, I, heard, about, Spark] => Vector: [-0.008142343163490296,0.02051363289356232,0.03255096450448036] Text: [I, wish, Java, could, use, case, classes] => Vector: [0.043090314205203734,0.035048123182994974,0.023512658663094044] Text: [Logistic, regression, models, are, neat] => Vector: [0.038572299480438235,-0.03250147425569594,-0.01552378609776497] Spark’s Word2Vec implementation includes a variety of tuning parameters that can be found in the documentation. Feature Manipulation While nearly every transformer in ML manipulates the feature space in some way, the following algorithms and tools are automated means of either expanding the input feature vectors or reducing them to a lower number of dimensions. PCA Principal Components Analysis (PCA) is a mathematical technique for finding the most important aspects of our data (the principal components). It changes the feature representation of our data by creating a new set of features (“aspects”). Each new feature is a combination of the original features. The power of PCA is that it can create a smaller set of more meaningful features to be input into your model, at the potential cost of interpretability. You’d want to use PCA if you have a large input dataset and want to reduce the total number of features you have. This frequently comes up in text analysis where the entire feature space is massive and many of the features are largely irrelevant. Using PCA, we can find the most important combinations of features and only include those in our machine learning model. PCA takes a parameter , specifying the number of output features to create. Generally, this should be much smaller than your input vectors’ dimension. NOTE Picking the right is nontrivial and there’s no prescription we can give. Check out the relevant chapters in ESL and ISL for more information. Let’s train PCA with a of 2: // in Scala import org.apache.spark.ml.feature.PCA val pca = new PCA().setInputCol("features").setK(2) pca.fit(scaleDF).transform(scaleDF).show(false) # in Python from pyspark.ml.feature import PCA pca = PCA().setInputCol("features").setK(2) pca.fit(scaleDF).transform(scaleDF).show(20, False) +---+--------------+------------------------------------------+ |id |features |pca_7c5c4aa7674e__output | +---+--------------+------------------------------------------+ |0 |[1.0,0.1,-1.0]|[0.0713719499248418,-0.4526654888147822] | ... |1 |[3.0,10.1,3.0]|[-10.872398139848944,0.030962697060150646]| +---+--------------+------------------------------------------+ Interaction In some cases, you might have domain knowledge about specific variables in your dataset. For example, you might know that a certain interaction between the two variables is an important variable to include in a downstream estimator. The feature transformer Interaction allows you to create an interaction between two variables manually. It just multiplies the two features together—something that a typical linear model would not do for every possible pair of features in your data. This transformer is currently only available directly in Scala but can be called from any language using the RFormula. We recommend users just use RFormula instead of manually creating interactions. Polynomial Expansion Polynomial expansion is used to generate interaction variables of all the input columns. With polynomial expansion, we specify to what degree we would like to see various interactions. For example, for a degree-2 polynomial, Spark takes every value in our feature vector, multiplies it by every other value in the feature vector, and then stores the results as features. For instance, if we have two input features, we’ll get four output features if we use a second degree polynomial (2x2). If we have three input features, we’ll get nine output features (3x3). If we use a thirddegree polynomial, we’ll get 27 output features (3x3x3) and so on. This transformation is useful when you want to see interactions between particular features but aren’t necessarily sure about which interactions to consider. WARNING Polynomial expansion can greatly increase your feature space, leading to both high computational costs and overfitting. Use it with caution, especially for higher degrees. Here’s an example of a second degree polynomial: // in Scala import org.apache.spark.ml.feature.PolynomialExpansion val pe = new PolynomialExpansion().setInputCol("features").setDegree(2) pe.transform(scaleDF).show(false) # in Python from pyspark.ml.feature import PolynomialExpansion pe = PolynomialExpansion().setInputCol("features").setDegree(2) pe.transform(scaleDF).show() +---+--------------+-----------------------------------------------------------+ |id |features |poly_9b2e603812cb__output | +---+--------------+-----------------------------------------------------------+ |0 |[1.0,0.1,-1.0]|[1.0,1.0,0.1,0.1,0.010000000000000002,-1.0,-1.0,-0.1,1.0] | ... |1 |[3.0,10.1,3.0]|[3.0,9.0,10.1,30.299999999999997,102.00999999999999,3.0... | +---+--------------+-----------------------------------------------------------+ Feature Selection Often, you will have a large range of possible features and want to select a smaller subset to use for training. For example, many features might be correlated, or using too many features might lead to overfitting. This process is called feature selection. There are a number of ways to evaluate feature importance once you’ve trained a model but another option is to do some rough filtering beforehand. Spark has some simple options for doing that, such as ChiSqSelector. ChiSqSelector ChiSqSelector leverages a statistical test to identify features that are not independent from the label we are trying to predict, and drop the uncorrelated features. It’s often used with categorical data in order to reduce the number of features you will input into your model, as well as to reduce the dimensionality of text data (in the form of frequencies or counts). Since this method is based on the Chi-Square test, there are several different ways we can pick the “best” features. The methods are numTopFeatures, which is ordered by p-value; percentile, which takes a proportion of the input features (instead of just the top N features); and fpr, which sets a cut off p-value. We will demonstrate this with the output of the CountVectorizer created earlier in this chapter: // in Scala import org.apache.spark.ml.feature.{ChiSqSelector, Tokenizer} val tkn = new Tokenizer().setInputCol("Description").setOutputCol("DescOut") val tokenized = tkn .transform(sales.select("Description", "CustomerId")) .where("CustomerId IS NOT NULL") val prechi = fittedCV.transform(tokenized) val chisq = new ChiSqSelector() .setFeaturesCol("countVec") .setLabelCol("CustomerId") .setNumTopFeatures(2) chisq.fit(prechi).transform(prechi) .drop("customerId", "Description", "DescOut").show() # in Python from pyspark.ml.feature import ChiSqSelector, Tokenizer tkn = Tokenizer().setInputCol("Description").setOutputCol("DescOut") tokenized = tkn\ .transform(sales.select("Description", "CustomerId"))\ .where("CustomerId IS NOT NULL") prechi = fittedCV.transform(tokenized)\ .where("CustomerId IS NOT NULL") chisq = ChiSqSelector()\ .setFeaturesCol("countVec")\ .setLabelCol("CustomerId")\ .setNumTopFeatures(2) chisq.fit(prechi).transform(prechi)\ .drop("customerId", "Description", "DescOut").show() Advanced Topics There are several advanced topics surrounding transformers and estimators. Here we touch on the two most common, persisting transformers as well as writing custom ones. Persisting Transformers Once you’ve used an estimator to configure a transformer, it can be helpful to write it to disk and simply load it when necessary (e.g., for use in another Spark session). We saw this in the previous chapter when we persisted an entire pipeline. To persist a transformer individually, we use the write method on the fitted transformer (or the standard transformer) and specify the location: // in Scala val fittedPCA = pca.fit(scaleDF) fittedPCA.write.overwrite().save("/tmp/fittedPCA") # in Python fittedPCA = pca.fit(scaleDF) fittedPCA.write().overwrite().save("/tmp/fittedPCA") We can then load it back in: // in Scala import org.apache.spark.ml.feature.PCAModel val loadedPCA = PCAModel.load("/tmp/fittedPCA") loadedPCA.transform(scaleDF).show() # in Python from pyspark.ml.feature import PCAModel loadedPCA = PCAModel.load("/tmp/fittedPCA") loadedPCA.transform(scaleDF).show() Writing a Custom Transformer Writing a custom transformer can be valuable when you want to encode some of your own business logic in a form that you can fit into an ML Pipeline, pass on to hyperparameter search, and so on. In general you should try to use the built-in modules (e.g., SQLTransformer) as much as possible because they are optimized to run efficiently. But sometimes we do not have that luxury. Let’s create a simple tokenizer to demonstrate: import org.apache.spark.ml.UnaryTransformer import org.apache.spark.ml.util.{DefaultParamsReadable, DefaultParamsWritable, Identifiable} import org.apache.spark.sql.types.{ArrayType, StringType, DataType} import org.apache.spark.ml.param.{IntParam, ParamValidators} class MyTokenizer(override val uid: String) extends UnaryTransformer[String, Seq[String], MyTokenizer] with DefaultParamsWritable { def this() = this(Identifiable.randomUID("myTokenizer")) val maxWords: IntParam = new IntParam(this, "maxWords", "The max number of words to return.", ParamValidators.gtEq(0)) def setMaxWords(value: Int): this.type = set(maxWords, value) def getMaxWords: Integer = $(maxWords) override protected def createTransformFunc: String => Seq[String] = ( inputString: String) => { inputString.split("\\s").take($(maxWords)) } override protected def validateInputType(inputType: DataType): Unit = { require( inputType == StringType, s"Bad input type: $inputType. Requires String.") } override protected def outputDataType: DataType = new ArrayType(StringType, true) } // this will allow you to read it back in by using this object. object MyTokenizer extends DefaultParamsReadable[MyTokenizer] val myT = new MyTokenizer().setInputCol("someCol").setMaxWords(2) myT.transform(Seq("hello world. This text won't show.").toDF("someCol")).show() It is also possible to write a custom estimator where you must customize the transformation based on the actual input data. However, this isn’t as common as writing a standalone transformer and is therefore not included in this book. A good way to do this is to look at one of the simple estimators we saw before and modify the code to suit your use case. A good place to start might be the StandardScaler. Conclusion This chapter gave a whirlwind tour of many of the most common preprocessing transformations Spark has available. There are several domain-specific ones we did not have enough room to cover (e.g., Discrete Cosine Transform), but you can find more information in the documentation. This area of Spark is also constantly growing as the community develops new ones. Another important aspect of this feature engineering toolkit is consistency. In the previous chapter we covered the pipeline concept, an essential tool to package and train end-to-end ML workflows. In the next chapter we will start going through the variety of machine learning tasks you may have and what algorithms are available for each one. Chapter 26. Classification Classification is the task of predicting a label, category, class, or discrete variable given some input features. The key difference from other ML tasks, such as regression, is that the output label has a finite set of possible values (e.g., three classes). Use Cases Classification has many use cases, as we discussed in Chapter 24. Here are a few more to consider as a reinforcement of the multitude of ways classification can be used in the real world. Predicting credit risk A financing company might look at a number of variables before offering a loan to a company or individual. Whether or not to offer the loan is a binary classification problem. News classification An algorithm might be trained to predict the topic of a news article (sports, politics, business, etc.). Classifying human activity By collecting data from sensors such as a phone accelerometer or smart watch, you can predict the person’s activity. The output will be one of a finite set of classes (e.g., walking, sleeping, standing, or running). Types of Classification Before we continue, let’s review several different types of classification. Binary Classification The simplest example of classification is binary classification, where there are only two labels you can predict. One example is fraud analytics, where a given transaction can be classified as fraudulent or not; or email spam, where a given email can be classified as spam or not spam. Multiclass Classification Beyond binary classification lies multiclass classification, where one label is chosen from more than two distinct possible labels. A typical example is Facebook predicting the people in a given photo or a meterologist predicting the weather (rainy, sunny, cloudy, etc.). Note how there is always a finite set of classes to predict; it’s never unbounded. This is also called multinomial classification. Multilabel Classification Finally, there is multilabel classification, where a given input can produce multiple labels. For example, you might want to predict a book’s genre based on the text of the book itself. While this could be multiclass, it’s probably better suited for multilabel because a book may fall into multiple genres. Another example of multilabel classification is identifying the number of objects that appear in an image. Note that in this example, the number of output predictions is not necessarily fixed, and could vary from image to image. Classification Models in MLlib Spark has several models available for performing binary and multiclass classification out of the box. The following models are available for classification in Spark: Logistic regression Decision trees Random forests Gradient-boosted trees Spark does not support making multilabel predictions natively. In order to train a multilabel model, you must train one model per label and combine them manually. Once manually constructed, there are built-in tools that support measuring these kinds of models (discussed at the end of the chapter). This chapter will cover the basics of each of these models by providing: A simple explanation of the model and the intuition behind it Model hyperparameters (the different ways we can initialize the model) Training parameters (parameters that affect how the model is trained) Prediction parameters (parameters that affect how predictions are made) You can set the hyperparameters and training parameters in a ParamGrid as we saw in Chapter 24. Model Scalability Model scalability is an important consideration when choosing your model. In general, Spark has great support for training large-scale machine learning models (note, these are large scale; on single-node workloads there are a number of other tools that also perform well). Table 26-1 is a simple model scalability scorecard to use to find the best model for your particular task (if scalability is your core consideration). The actual scalability will depend on your configuration, machine size, and other specifics but should make for a good heuristic. Table 26-1. Model scalability reference Model Features count Training examples Output classes Logistic regression 1 to 10 million No limit Features x Classes < 10 million Decision trees 1,000s No limit Features x Classes < 10,000s Random forest 10,000s No limit Features x Classes < 100,000s No limit Features x Classes < 10,000s Gradient-boosted trees 1,000s We can see that nearly all these models scale to large collections of input data and there is ongoing work to scale them even further. The reason no limit is in place for the number of training examples is because these are trained using methods like stochastic gradient descent and L-BFGS. These methods are optimized specifically for working with massive datasets and to remove any constraints that might exist on the number of training examples you would hope to learn on. Let’s start looking at the classification models by loading in some data: // in Scala val bInput = spark.read.format("parquet").load("/data/binary-classification") .selectExpr("features", "cast(label as double) as label") # in Python bInput = spark.read.format("parquet").load("/data/binary-classification")\ .selectExpr("features", "cast(label as double) as label") NOTE Like our other advanced analytics chapters, this one cannot teach you the mathematical underpinnings of every model. See Chapter 4 in ISL and ESL for a review of classification. Logistic Regression Logistic regression is one of the most popular methods of classification. It is a linear method that combines each of the individual inputs (or features) with specific weights (these weights are generated during the training process) that are then combined to get a probability of belonging to a particular class. These weights are helpful because they are good representations of feature importance; if you have a large weight, you can assume that variations in that feature have a significant effect on the outcome (assuming you performed normalization). A smaller weight means the feature is less likely to be important. See ISL 4.3 and ESL 4.4 for more information. Model Hyperparameters Model hyperparameters are configurations that determine the basic structure of the model itself. The following hyperparameters are available for logistic regression: family Can be multinomial (two or more distinct labels; multiclass classification) or binary (only two distinct labels; binary classification). elasticNetParam A floating-point value from 0 to 1. This parameter specifies the mix of L1 and L2 regularization according to elastic net regularization (which is a linear combination of the two). Your choice of L1 or L2 depends a lot on your particular use case but the intuition is as follows: L1 regularization (a value of 1) will create sparsity in the model because certain feature weights will become zero (that are of little consequence to the output). For this reason, it can be used as a simple feature-selection method. On the other hand, L2 regularization (a value of 0) does not create sparsity because the corresponding weights for particular features will only be driven toward zero, but will never completely reach zero. ElasticNet gives us the best of both worlds—we can choose a value between 0 and 1 to specify a mix of L1 and L2 regularization. For the most part, you should be tuning this by testing different values. fitIntercept Can be true or false. This hyperparameter determines whether or not to fit the intercept or the arbitrary number that is added to the linear combination of inputs and weights of the model. Typically you will want to fit the intercept if we haven’t normalized our training data. regParam A value ≥ 0. that determines how much weight to give to the regularization term in the objective function. Choosing a value here is again going to be a function of noise and dimensionality in our dataset. In a pipeline, try a wide range of values (e.g., 0, 0.01, 0.1, 1). standardization Can be true or false, whether or not to standardize the inputs before passing them into the model. See Chapter 25 for more information. Training Parameters Training parameters are used to specify how we perform our training. Here are the training parameters for logistic regression. maxIter Total number of iterations over the data before stopping. Changing this parameter probably won’t change your results a ton, so it shouldn’t be the first parameter you look to adjust. The default is 100. tol This value specifies a threshold by which changes in parameters show that we optimized our weights enough, and can stop iterating. It lets the algorithm stop before maxIter iterations. The default value is 1.0E-6. This also shouldn’t be the first parameter you look to tune. weightCol The name of a weight column used to weigh certain rows more than others. This can be a useful tool if you have some other measure of how important a particular training example is and have a weight associated with it. For example, you might have 10,000 examples where you know that some labels are more accurate than others. You can weigh the labels you know are correct more than the ones you don’t. Prediction Parameters These parameters help determine how the model should actually be making predictions at prediction time, but do not affect training. Here are the prediction parameters for logistic regression: threshold A Double in the range of 0 to 1. This parameter is the probability threshold for when a given class should be predicted. You can tune this parameter according to your requirements to balance between false positives and false negatives. For instance, if a mistaken prediction would be costly—you might want to make its prediction threshold very high. thresholds This parameter lets you specify an array of threshold values for each class when using multiclass classification. It works similarly to the single threshold parameter described previously. Example Here’s a simple example using the LogisticRegression model. Notice how we didn’t specify any parameters because we’ll leverage the defaults and our data conforms to the proper column naming. In practice, you probably won’t need to change many of the parameters: // in Scala import org.apache.spark.ml.classification.LogisticRegression val lr = new LogisticRegression() println(lr.explainParams()) // see all parameters val lrModel = lr.fit(bInput) # in Python from pyspark.ml.classification import LogisticRegression lr = LogisticRegression() print lr.explainParams() # see all parameters lrModel = lr.fit(bInput) Once the model is trained you can get information about the model by taking a look at the coefficients and the intercept. The coefficients correspond to the individual feature weights (each feature weight is multiplied by each respective feature to compute the prediction) while the intercept is the value of the italics-intercept (if we chose to fit one when specifying the model). Seeing the coefficients can be helpful for inspecting the model that you built and comparing how features affect the prediction: // in Scala println(lrModel.coefficients) println(lrModel.intercept) # in Python print lrModel.coefficients print lrModel.intercept For a multinomial model (the current one is binary), lrModel.coefficientMatrix and lrModel.interceptVector can be used to get the coefficients and intercept. These will return Matrix and Vector types representing the values or each of the given classes. Model Summary Logistic regression provides a model summary that gives you information about the final, trained model. This is analogous to the same types of summaries we see in many R language machine learning packages. The model summary is currently only available for binary logistic regression problems, but multiclass summaries will likely be added in the future. Using the binary summary, we can get all sorts of information about the model itself including the area under the ROC curve, the f measure by threshold, the precision, the recall, the recall by thresholds, and the ROC curve. Note that for the area under the curve, instance weighting is not taken into account, so if you wanted to see how you performed on the values you weighed more highly, you’d have to do that manually. This will probably change in future Spark versions. You can see the summary using the following APIs: // in Scala import org.apache.spark.ml.classification.BinaryLogisticRegressionSummary val summary = lrModel.summary val bSummary = summary.asInstanceOf[BinaryLogisticRegressionSummary] println(bSummary.areaUnderROC) bSummary.roc.show() bSummary.pr.show() # in Python summary = lrModel.summary print summary.areaUnderROC summary.roc.show() summary.pr.show() The speed at which the model descends to the final result is shown in the objective history. We can access this through the objective history on the model summary: summary.objectiveHistory This is an array of doubles that specify how, over each training iteration, we are performing with respect to our objective function. This information is helpful to see if we have sufficient iterations or need to be tuning other parameters. Decision Trees Decision trees are one of the more friendly and interpretable models for performing classification because they’re similar to simple decision models that humans use quite often. For example, if you have to predict whether or not someone will eat ice cream when offered, a good feature might be whether or not that individual likes ice cream. In pseudocode, if person.likes(“ice_cream”), they will eat ice cream; otherwise, they won’t eat ice cream. A decision tree creates this type of structure with all the inputs and follows a set of branches when it comes time to make a prediction. This makes it a great starting point model because it’s easy to reason about, easy to inspect, and makes very few assumptions about the structure of the data. In short, rather than trying to train coeffiecients in order to model a function, it simply creates a big tree of decisions to follow at prediction time. This model also supports multiclass classification and provides outputs as predictions and probabilities in two different columns. While this model is usually a great start, it does come at a cost. It can overfit data extremely quickly. By that we mean that, unrestrained, the decision tree will create a pathway from the start based on every single training example. That means it encodes all of the information in the training set in the model. This is bad because then the model won’t generalize to new data (you will see poor test set prediction performance). However, there are a number of ways to try and rein in the model by limiting its branching structure (e.g., limiting its height) to get good predictive power. See ISL 8.1 and ESL 9.2 for more information. Model Hyperparameters There are many different ways to configure and train decision trees. Here are the hyperparameters that Spark’s implementation supports: maxDepth Since we’re training a tree, it can be helpful to specify a max depth in order to avoid overfitting to the dataset (in the extreme, every row ends up as its own leaf node). The default is 5. maxBins In decision trees, continuous features are converted into categorical features and maxBins determines how many bins should be created from continous features. More bins gives a higher level of granularity. The value must be greater than or equal to 2 and greater than or equal to the number of categories in any categorical feature in your dataset. The default is 32. impurity To build up a “tree” you need to configure when the model should branch. Impurity represents the metric (information gain) to determine whether or not the model should split at a particular leaf node. This parameter can be set to either be “entropy” or “gini” (default), two commonly used impurity metrics. minInfoGain This parameter determines the minimum information gain that can be used for a split. A higher value can prevent overfitting. This is largely something that needs to be determined from testing out different variations of the decision tree model. The default is zero. minInstancePerNode This parameter determines the minimum number of training instances that need to end in a particular node. Think of this as another manner of controlling max depth. We can prevent overfitting by limiting depth or we can prevent it by specifying that at minimum a certain number of training values need to end up in a particular leaf node. If it’s not met we would “prune” the tree until that requirement is met. A higher value can prevent overfitting. The default is 1, but this can be any value greater than 1. Training Parameters These are configurations we specify in order to manipulate how we perform our training. Here is the training parameter for decision trees: checkpointInterval Checkpointing is a way to save the model’s work over the course of training so that if nodes in the cluster crash for some reason, you don’t lose your work. A value of 10 means the model will get checkpointed every 10 iterations. Set this to -1 to turn off checkpointing. This parameter needs to be set together with a checkpointDir (a directory to checkpoint to) and with useNodeIdCache=true. Consult the Spark documentation for more information on checkpointing. Prediction Parameters There is only one prediction parameter for decision trees: thresholds. Refer to the explanation for thresholds under “Logistic Regression”. Here’s a minimal but complete example of using a decision tree classifier: // in Scala import org.apache.spark.ml.classification.DecisionTreeClassifier val dt = new DecisionTreeClassifier() println(dt.explainParams()) val dtModel = dt.fit(bInput) # in Python from pyspark.ml.classification import DecisionTreeClassifier dt = DecisionTreeClassifier() print dt.explainParams() dtModel = dt.fit(bInput) Random Forest and Gradient-Boosted Trees These methods are extensions of the decision tree. Rather than training one tree on all of the data, you train multiple trees on varying subsets of the data. The intuition behind doing this is that various decision trees will become “experts” in that particular domain while others become experts in others. By combining these various experts, you then get a “wisdom of the crowds” effect, where the group’s performance exceeds any individual. In addition, these methods can help prevent overfitting. Random forests and gradient-boosted trees are two distinct methods for combining decision trees. In random forests, we simply train a lot of trees and then average their response to make a prediction. With gradient-boosted trees, each tree makes a weighted prediction (such that some trees have more predictive power for some classes than others). They have largely the same parameters, which we note below. One current limitation is that gradient-boosted trees currently only support binary labels. NOTE There are several popular tools for learning tree-based models. For example, the XGBoost library provides an integration package for Spark that can be used to run it on Spark. See ISL 8.2 and ESL 10.1 for more information on these tree ensemble models. Model Hyperparameters Random forests and gradient-boosted trees provide all of the same model hyperparameters supported by decision trees. In addition, they add several of their own. Random forest only numTrees The total number of trees to train. featureSubsetStrategy This parameter determines how many features should be considered for splits. This can be a variety of different values including “auto”, “all”, “sqrt”, “log2”, or a number “n.” When your input is “n” the model will use n * number of features during training. When n is in the range (1, number of features), the model will use n features during training. There’s no onesize-fits-all solution here, so it’s worth experimenting with different values in your pipeline. Gradient-boosted trees (GBT) only lossType This is the loss function for gradient-boosted trees to minimize during training. Currently, only logistic loss is supported. maxIter Total number of iterations over the data before stopping. Changing this probably won’t change your results a ton, so it shouldn’t be the first parameter you look to adjust. The default is 100. stepSize This is the learning rate for the algorithm. A larger step size means that larger jumps are made between training iterations. This can help in the optimization process and is something that should be tested in training. The default is 0.1 and this can be any value from 0 to 1. Training Parameters There is only one training parameter for these models, checkpointInterval. Refer back to the explanation under “Decision Trees” for details on checkpointing. Prediction Parameters These models have the same prediction parameters as decision trees. Consult the prediction parameters under that model for more information. Here’s a short code example of using each of these classifiers: // in Scala import org.apache.spark.ml.classification.RandomForestClassifier val rfClassifier = new RandomForestClassifier() println(rfClassifier.explainParams()) val trainedModel = rfClassifier.fit(bInput) // in Scala import org.apache.spark.ml.classification.GBTClassifier val gbtClassifier = new GBTClassifier() println(gbtClassifier.explainParams()) val trainedModel = gbtClassifier.fit(bInput) # in Python from pyspark.ml.classification import RandomForestClassifier rfClassifier = RandomForestClassifier() print rfClassifier.explainParams() trainedModel = rfClassifier.fit(bInput) # in Python from pyspark.ml.classification import GBTClassifier gbtClassifier = GBTClassifier() print gbtClassifier.explainParams() trainedModel = gbtClassifier.fit(bInput) Naive Bayes Naive Bayes classifiers are a collection of classifiers based on Bayes’ theorem. The core assumption behind the models is that all features in your data are independent of one another. Naturally, strict independence is a bit naive, but even if this is violated, useful models can still be produced. Naive Bayes classifiers are commonly used in text or document classification, although it can be used as a more general-purpose classifier as well. There are two different model types: either a multivariate Bernoulli model, where indicator variables represent the existence of a term in a document; or the multinomial model, where the total counts of terms are used. One important note when it comes to Naive Bayes is that all input features must be non-negative. See ISL 4.4 and ESL 6.6 for more background on these models. Model Hyperparameters These are configurations we specify to determine the basic structure of the models: modelType Either “bernoulli” or “multinomial.” See the previous section for more information on this choice. weightCol Allows weighing different data points differently. Refer back to “Training Parameters” for the explanation of this hyperparameter. Training Parameters These are configurations that specify how we perform our training: smoothing This determines the amount of regularization that should take place using additive smoothing. This helps smooth out categorical data and avoid overfitting on the training data by changing the expected probability for certain classes. The default value is 1. Prediction Parameters Naive Bayes shares the same prediction parameter, thresholds, as all of our other models. Refer back to the previous explanation for threshold to see how to use this. Here’s an example of using a Naive Bayes classifier. // in Scala import org.apache.spark.ml.classification.NaiveBayes val nb = new NaiveBayes() println(nb.explainParams()) val trainedModel = nb.fit(bInput.where("label != 0")) # in Python from pyspark.ml.classification import NaiveBayes nb = NaiveBayes() print nb.explainParams() trainedModel = nb.fit(bInput.where("label != 0")) WARNING Note that in this example dataset, we have features that have negative values. In this case, the rows with negative features correspond to rows with label “0”. Therefore we’re just going to filter them out (via the label) instead of processing them further to demonstrate the naive bayes API. Evaluators for Classification and Automating Model Tuning As we saw in Chapter 24, evaluators allow us to specify the metric of success for our model. An evaluator doesn’t help too much when it stands alone; however, when we use it in a pipeline, we can automate a grid search of our various parameters of the models and transformers—trying all combinations of the parameters to see which ones perform the best. Evaluators are most useful in this pipeline and parameter grid context. For classification, there are two evaluators, and they expect two columns: a predicted label from the model and a true label. For binary classification we use the BinaryClassificationEvaluator. This supports optimizing for two different metrics “areaUnderROC” and areaUnderPR.” For multiclass classification, we need to use the MulticlassClassificationEvaluator, which supports optimizing for “f1”, “weightedPrecision”, “weightedRecall”, and “accuracy”. To use evaluators, we build up our pipeline, specify the parameters we would like to test, and then run it and see the results. See Chapter 24 for a code example. Detailed Evaluation Metrics MLlib also contains tools that let you evaluate multiple classification metrics at once. Unfortunately, these metrics classes have not been ported over to Spark’s DataFrame-based ML package from the underlying RDD framework. So, at the time of this writing, you still have to create an RDD to use these. In the future, this functionality will likely be ported to DataFrames and the following may no longer be the best way to see metrics (although you will still be able to use these APIs). There are three different classification metrics we can use: Binary classification metrics Multiclass classification metrics Multilabel classification metrics All of these measures follow the same approximate style. We’ll compare generated outputs with true values and the model calculates all of the relevant metrics for us. Then we can query the object for the values for each of the metrics: // in Scala import org.apache.spark.mllib.evaluation.BinaryClassificationMetrics val out = model.transform(bInput) .select("prediction", "label") .rdd.map(x => (x(0).asInstanceOf[Double], x(1).asInstanceOf[Double])) val metrics = new BinaryClassificationMetrics(out) # in Python from pyspark.mllib.evaluation import BinaryClassificationMetrics out = model.transform(bInput)\ .select("prediction", "label")\ .rdd.map(lambda x: (float(x[0]), float(x[1]))) metrics = BinaryClassificationMetrics(out) Once we’ve done that, we can see typical classification success metrics on this metric’s object using a similar API to the one we saw with logistic regression: // in Scala metrics.areaUnderPR metrics.areaUnderROC println("Receiver Operating Characteristic") metrics.roc.toDF().show() # in Python print metrics.areaUnderPR print metrics.areaUnderROC print "Receiver Operating Characteristic" metrics.roc.toDF().show() One-vs-Rest Classifier There are some MLlib models that don’t support multiclass classification. In these cases, users can leverage a one-vs-rest classifier in order to perform multiclass classification given only a binary classifier. The intuition behind this is that for every class you hope to predict, the one-vsrest classifier will turn the problem into a binary classification problem by isolating one class as the target class and grouping all of the other classes into one. Thus the prediction of the class becomes binary (is it this class or not this class?). One-vs-rest is implemented as an estimator. For the base classifier it takes instances of the classifier and creates a binary classification problem for each of the classes. The classifier for class i is trained to predict whether the label is i or not, distinguishing class i from all other classes. Predictions are done by evaluating each binary classifier and the index of the most confident classifier is output as the label. See the Spark documentation for a nice example of the use of one-vs-rest. Multilayer Perceptron The multilayer perceptron is a classifier based on neural networks with a configurable number of layers (and layer sizes). We will discuss it in Chapter 31. Conclusion In this chapter we covered the majority of tools Spark provides for classification: predicting one of a finite set of labels for each data point based on its features. In the next chapter, we’ll look at regression, where the required output is continuous instead of categorical. Chapter 27. Regression Regression is a logical extension of classification. Rather than just predicting a single value from a set of values, regression is the act of predicting a real number (or continuous variable) from a set of features (represented as numbers). Regression can be harder than classification because, from a mathematical perspective, there are an infinite number of possible output values. Furthermore, we aim to optimize some metric of error between the predicted and true value, as opposed to an accuracy rate. Aside from that, regression and classification are fairly similar. For this reason, we will see a lot of the same underlying concepts applied to regression as we did with classification. Use Cases The following is a small set of regression use cases that can get you thinking about potential regression problems in your own domain: Predicting movie viewership Given information about a movie and the movie-going public, such as how many people have watched the trailer or shared it on social media, you might want to predict how many people are likely to watch the movie when it comes out. Predicting company revenue Given a current growth trajectory, the market, and seasonality, you might want to predict how much revenue a company will gain in the future. Predicting crop yield Given information about the particular area in which a crop is grown, as well as the current weather throughout the year, you might want to predict the total crop yield for a particular plot of land. Regression Models in MLlib There are several fundamental regression models in MLlib. Some of these models are carryovers from Chapter 26. Others are only relevant to the regression problem domain. This list is current as of Spark 2.2 but will grow: Linear regression Generalized linear regression Isotonic regression Decision trees Random forest Gradient-boosted trees Survival regression This chapter will cover the basics of each of these particular models by providing: A simple explanation of the model and the intuition behind the algorithm Model hyperparameters (the different ways that we can initialize the model) Training parameters (parameters that affect how the model is trained) Prediction parameters (parameters that affect how predictions are made) You can search over the hyperparameters and training parameters using a ParamGrid, as we saw in Chapter 24. Model Scalability The regression models in MLlib all scale to large datasets. Table 27-1 is a simple model scalability scorecard that will help you in choosing the best model for your particular task (if scalability is your core consideration). These will depend on your configuration, machine size, and other factors. Table 27-1. Regression scalability reference Model Number features Training examples Linear regression 1 to 10 million No limit Generalized linear regression 4,096 No limit Isotonic regression N/A Millions Decision trees 1,000s No limit Random forest 10,000s No limit Gradient-boosted trees 1,000s No limit Survival regression 1 to 10 million No limit NOTE Like our other advanced analytics chapters, this one cannot teach you the mathematical underpinnings of every model. See Chapter 3 in ISL and ESL for a review of regression. Let’s read in some sample data that we will use throughout the chapter: // in Scala val df = spark.read.load("/data/regression") # in Python df = spark.read.load("/data/regression") Linear Regression Linear regression assumes that a linear combination of your input features (the sum of each feature multiplied by a weight) results along with an amount of Gaussian error in the output. This linear assumption (along with Gaussian error) does not always hold true, but it does make for a simple, interpretable model that’s hard to overfit. Like logistic regression, Spark implements ElasticNet regularization for this, allowing you to mix L1 and L2 regularization. See ISL 3.2 and ESL 3.2 for more information. Model Hyperparameters Linear regression has the same model hyperparameters as logistic regression. See Chapter 26 for more information. Training Parameters Linear regression also shares all of the same training parameters from logistic regression. Refer back to Chapter 26 for more on this topic. Example Here’s a short example of using linear regression on our sample dataset: // in Scala import org.apache.spark.ml.regression.LinearRegression val lr = new LinearRegression().setMaxIter(10).setRegParam(0.3)\ .setElasticNetParam(0.8) println(lr.explainParams()) val lrModel = lr.fit(df) # in Python from pyspark.ml.regression import LinearRegression lr = LinearRegression().setMaxIter(10).setRegParam(0.3).setElasticNetParam(0.8) print lr.explainParams() lrModel = lr.fit(df) Training Summary Just as in logistic regression, we get detailed training information back from our model. The code font method is a simple shorthand for accessing these metrics. It reports several conventional metrics for measuring the success of a regression model, allowing you to see how well your model is actually fitting the line. The summary method returns a summary object with several fields. Let’s go through these in turn. The residuals are simply the weights for each of the features that we input into the model. The objective history shows how our training is going at every iteration. The root mean squared error is a measure of how well our line is fitting the data, determined by looking at the distance between each predicted value and the actual value in the data. The R-squared variable is a measure of the proportion of the variance of the predicted variable that is captured by the model. There are a number of metrics and summary information that may be relevant to your use case. This section demonstrates the API, but does not comprehensively cover every metric (consult the API documentation for more information). Here are some of the attributes of the model summary for linear regression: // in Scala val summary = lrModel.summary summary.residuals.show() println(summary.objectiveHistory.toSeq.toDF.show()) println(summary.rootMeanSquaredError) println(summary.r2) # in Python summary = lrModel.summary summary.residuals.show() print summary.totalIterations print summary.objectiveHistory print summary.rootMeanSquaredError print summary.r2 Generalized Linear Regression The standard linear regression that we saw in this chapter is actually a part of a family of algorithms called generalized linear regression. Spark has two implementations of this algorithm. One is optimized for working with very large sets of features (the simple linear regression covered previously in this chapter), while the other is more general, includes support for more algorithms, and doesn’t currently scale to large numbers of features. The generalized form of linear regression gives you more fine-grained control over what kind of regression model you use. For instance, these allow you to select the expected noise distribution from a variety of families, including Gaussian (linear regression), binomial (logistic regression), poisson (poisson regression), and gamma (gamma regression). The generalized models also support setting a link function that specifies the relationship between the linear predictor and the mean of the distribution function. Table 27-2 shows the available link functions for each family. Table 27-2. Regression families, response types, and link functions Family Response type Supported links Gaussian Continuous Identity*, Log, Inverse Binomial Binary Logit*, Probit, CLogLog Poisson Count Log*, Identity, Sqrt Gamma Continuous Inverse*, Idenity, Log Tweedie Zero-inflated continuous Power link function The asterisk signifies the canonical link function for each family. See ISL 3.2 and ESL 3.2 for more information on generalized linear models. WARNING A fundamental limitation as of Spark 2.2 is that generalized linear regression only accepts a maximum of 4,096 features for inputs. This will likely change for later versions of Spark, so be sure to refer to the documentation. Model Hyperparameters These are configurations that we specify to determine the basic structure of the model itself. In addition to fitIntercept and regParam (mentioned in “Regression”), generalized linear regression includes several other hyperparameters: family A description of the error distribution to be used in the model. Supported options are Poisson, binomial, gamma, Gaussian, and tweedie. link The name of link function which provides the relationship between the linear predictor and the mean of the distribution function. Supported options are cloglog, probit, logit, inverse, sqrt, identity, and log (default: identity). solver The solver algorithm to be used for optimization. The only currently supported solver is irls (iteratively reweighted least squares). variancePower The power in the variance function of the Tweedie distribution, which characterizes the relationship between the variance and mean of the distribution. Only applicable to the Tweedie family. Supported values are 0 and [1, Infinity). The default is 0. linkPower The index in the power link function for the Tweedie family. Training Parameters The training parameters are the same that you will find for logistic regression. Consult Chapter 26 for more information. Prediction Parameters This model adds one prediction parameter: linkPredictionCol A column name that will hold the output of our link function for each prediction. Example Here’s an example of using GeneralizedLinearRegression: // in Scala import org.apache.spark.ml.regression.GeneralizedLinearRegression val glr = new GeneralizedLinearRegression() .setFamily("gaussian") .setLink("identity") .setMaxIter(10) .setRegParam(0.3) .setLinkPredictionCol("linkOut") println(glr.explainParams()) val glrModel = glr.fit(df) # in Python from pyspark.ml.regression import GeneralizedLinearRegression glr = GeneralizedLinearRegression()\ .setFamily("gaussian")\ .setLink("identity")\ .setMaxIter(10)\ .setRegParam(0.3)\ .setLinkPredictionCol("linkOut") print glr.explainParams() glrModel = glr.fit(df) Training Summary As for the simple linear model in the previous section, the training summary provided by Spark for the generalized linear model can help you ensure that your model is a good fit for the data that you used as the training set. It is important to note that this does not replace running your algorithm against a proper test set, but it can provide more information. This information includes a number of different potential metrics for analyzing the fit of your algorithm, including some of the most common success metrics: R squared The coefficient of determination; a measure of fit. The residuals The difference between the label and the predicted value. Be sure to inspect the summary object on the model to see all the available methods. Decision Trees Decision trees as applied to regression work fairly similarly to decision trees applied to classification. The main difference is that decision trees for regression output a single number per leaf node instead of a label (as we saw with classification). The same interpretability properties and model structure still apply. In short, rather than trying to train coeffiecients to model a function, decision tree regression simply creates a tree to predict the numerical outputs. This is of significant consequence because unlike generalized linear regression, we can predict nonlinear functions in the input data. This also creates a significant risk of overfitting the data, so we need to be careful when tuning and evaluating these models. We also covered decision trees in Chapter 26 (refer to “Decision Trees”). For more information on this topic, consult ISL 8.1 and ESL 9.2. Model Hyperparameters The model hyperparameters that apply decision trees for regression are the same as those for classification except for a slight change to the impurity parameter. See Chapter 26 for more information on the other hyperparameters: impurity The impurity parameter represents the metric (information gain) for whether or not the model should split at a particular leaf node with a particular value or keep it as is. The only metric currently supported for regression trees is “variance.” Training Parameters In addition to hyperparameters, classification and regression trees also share the same training parameters. See “Training Parameters” for these parameters. Example Here’s a short example of using a decision tree regressor: // in Scala import org.apache.spark.ml.regression.DecisionTreeRegressor val dtr = new DecisionTreeRegressor() println(dtr.explainParams()) val dtrModel = dtr.fit(df) # in Python from pyspark.ml.regression import DecisionTreeRegressor dtr = DecisionTreeRegressor() print dtr.explainParams() dtrModel = dtr.fit(df) Random Forests and Gradient-Boosted Trees The random forest and gradient-boosted tree models can be applied to both classification and regression. As a review, these both follow the same basic concept as the decision tree, except rather than training one tree, many trees are trained to perform a regression. In the random forest model, many de-correlated trees are trained and then averaged. With gradient-boosted trees, each tree makes a weighted prediction (such that some trees have more predictive power for some classes over others). Random forest and gradient-boosted tree regression have the same model hyperparameters and training parameters as the corresponding classification models, except for the purity measure (as is the case with DecisionTreeRegressor). See ISL 8.2 and ESL 10.1 for more information on tree ensembles. Model Hyperparameters These models share many of the same parameters as we saw in the previous chapter as well as for regression decision trees. Refer back to “Model Hyperparameters” for a thorough explanation of these parameters. As for a single regression tree, however, the only impurity metric currently supported is variance. Training Parameters These models support the same checkpointInterval parameter as classification trees, as described in Chapter 26. Example Here’s a small example of how to use these two models to perform a regression: // in Scala import org.apache.spark.ml.regression.RandomForestRegressor import org.apache.spark.ml.regression.GBTRegressor val rf = new RandomForestRegressor() println(rf.explainParams()) val rfModel = rf.fit(df) val gbt = new GBTRegressor() println(gbt.explainParams()) val gbtModel = gbt.fit(df) # in Python from pyspark.ml.regression import RandomForestRegressor from pyspark.ml.regression import GBTRegressor rf = RandomForestRegressor() print rf.explainParams() rfModel = rf.fit(df) gbt = GBTRegressor() print gbt.explainParams() gbtModel = gbt.fit(df) Advanced Methods The preceding methods are highly general methods for performing a regression. The models are by no means exhaustive, but do provide the essential regression types that many folks use. This next section will cover some of the more specialized regression models that Spark includes. We omit code examples simply because they follow the same patterns as the other algorithms. Survival Regression (Accelerated Failure Time) Statisticians use survival analysis to understand the survival rate of individuals, typically in controlled experiments. Spark implements the accelerated failure time model, which, rather than describing the actual survival time, models the log of the survival time. This variation of survival regression is implemented in Spark because the more well-known Cox Proportional Hazard’s model is semi-parametric and does not scale well to large datasets. By contrast, accelerated failure time does because each instance (row) contributes to the resulting model independently. Accelerated failure time does have different assumptions than the Cox survival model and therefore one is not necessarily a drop-in replacement for the other. Covering these differing assumptions is outside of the scope of this book. See L. J. Wei’s paper on accelerated failure time for more information. The requirement for input is quite similar to that of other regressions. We will tune coefficients according to feature values. However, there is one departure, and that is the introduction of a censor variable column. A test subject censors during a scientific study when that individual drops out of a study, since their state at the end of the experiment may be unknown. This is important because we cannot assume an outcome for an individual that censors (doesn’t report that state to the researchers) at some intermediate point in a study. See more about survival regression with AFT in the documentation. Isotonic Regression Isotonic regression is another specialized regression model, with some unique requirements. Essentially, isotonic regression specifies a piecewise linear function that is always monotonically increasing. It cannot decrease. This means that if your data is going up and to the right in a given plot, this is an appropriate model. If it varies over the course of input values, then this is not appropriate. The illustration of isotonic regression’s behavior in Figure 27-1 makes it much easier to understand. Figure 27-1. Isotonic regression line Notice how this gets a better fit than the simple linear regression. See more about how to use this model in the Spark documentation. Evaluators and Automating Model Tuning Regression has the same core model tuning functionality that we saw with classification. We can specify an evaluator, pick a metric to optimize for, and then train our pipeline to perform that parameter tuning on our part. The evaluator for regression, unsurprisingly, is called the RegressionEvaluator and allows us to optimize for a number of common regression success metrics. Just like the classification evaluator, RegressionEvaluator expects two columns, a column representing the prediction and another representing the true label. The supported metrics 2 to optimize for are the root mean squared error (“rmse”), the mean squared error (“mse”), the r2 metric (“r2”), and the mean absolute error (“mae”). To use RegressionEvaluator, we build up our pipeline, specify the parameters we would like to test, and then run it. Spark will automatically select the model that performs best and return this to us: // in Scala import org.apache.spark.ml.evaluation.RegressionEvaluator import org.apache.spark.ml.regression.GeneralizedLinearRegression import org.apache.spark.ml.Pipeline import org.apache.spark.ml.tuning.{CrossValidator, ParamGridBuilder} val glr = new GeneralizedLinearRegression() .setFamily("gaussian") .setLink("identity") val pipeline = new Pipeline().setStages(Array(glr)) val params = new ParamGridBuilder().addGrid(glr.regParam, Array(0, 0.5, 1)) .build() val evaluator = new RegressionEvaluator() .setMetricName("rmse") .setPredictionCol("prediction") .setLabelCol("label") val cv = new CrossValidator() .setEstimator(pipeline) .setEvaluator(evaluator) .setEstimatorParamMaps(params) .setNumFolds(2) // should always be 3 or more but this dataset is small val model = cv.fit(df) # in Python from pyspark.ml.evaluation import RegressionEvaluator from pyspark.ml.regression import GeneralizedLinearRegression from pyspark.ml import Pipeline from pyspark.ml.tuning import CrossValidator, ParamGridBuilder glr = GeneralizedLinearRegression().setFamily("gaussian").setLink("identity") pipeline = Pipeline().setStages([glr]) params = ParamGridBuilder().addGrid(glr.regParam, [0, 0.5, 1]).build() evaluator = RegressionEvaluator()\ .setMetricName("rmse")\ .setPredictionCol("prediction")\ .setLabelCol("label") cv = CrossValidator()\ .setEstimator(pipeline)\ .setEvaluator(evaluator)\ .setEstimatorParamMaps(params)\ .setNumFolds(2) # should always be 3 or more but this dataset is small model = cv.fit(df) Metrics Evaluators allow us to evaluate and fit a model according to one specific metric, but we can also access a number of regression metrics via the RegressionMetrics object. As for the classification metrics in the previous chapter, RegressionMetrics operates on RDDs of (prediction, label) pairs. For instance, let’s see how we can inspect the results of the previously trained model. // in Scala import org.apache.spark.mllib.evaluation.RegressionMetrics val out = model.transform(df) .select("prediction", "label") .rdd.map(x => (x(0).asInstanceOf[Double], x(1).asInstanceOf[Double])) val metrics = new RegressionMetrics(out) println(s"MSE = ${metrics.meanSquaredError}") println(s"RMSE = ${metrics.rootMeanSquaredError}") println(s"R-squared = ${metrics.r2}") println(s"MAE = ${metrics.meanAbsoluteError}") println(s"Explained variance = ${metrics.explainedVariance}") # in Python from pyspark.mllib.evaluation import RegressionMetrics out = model.transform(df)\ .select("prediction", "label").rdd.map(lambda x: (float(x[0]), float(x[1]))) metrics = RegressionMetrics(out) print "MSE: " + str(metrics.meanSquaredError) print "RMSE: " + str(metrics.rootMeanSquaredError) print "R-squared: " + str(metrics.r2) print "MAE: " + str(metrics.meanAbsoluteError) print "Explained variance: " + str(metrics.explainedVariance) Consult the Spark documentation for the latest methods. Conclusion In this chapter, we covered the basics of regression in Spark, including how we train models and how we measure success. In the next chapter, we’ll take a look at recommendation engines, one of the more popular applications of MLlib. Chapter 28. Recommendation The task of recommendation is one of the most intuitive. By studying people’s explicit preferences (through ratings) or implicit preferences (through observed behavior), you can make recommendations on what one user may like by drawing similarities between the user and other users, or between the products they liked and other products. Using the underlying similarities, recommendation engines can make new recommendations to other users. Use Cases Recommendation engines are one of the best use cases for big data. It’s fairly easy to collect training data about users’ past preferences at scale, and this data can be used in many domains to connect users with new content. Spark is an open source tool of choice used across a variety of companies for large-scale recommendations: Movie recommendations Amazon, Netflix, and HBO all want to provide relevant film and TV content to their users. Netflix utilizes Spark, to make large scale movie recommendations to their users. Course recommendations A school might want to recommend courses to students by studying what courses similar students have liked or taken. Past enrollment data makes for a very easy to collect training dataset for this task. In Spark, there is one workhorse recommendation algorithm, Alternating Least Squares (ALS). This algorithm leverages a technique called collaborative filtering, which makes recommendations based only on which items users interacted with in the past. That is, it does not require or use any additional features about the users or the items. It supports several ALS variants (e.g., explicit or implicit feedback). Apart from ALS, Spark provides Frequent Pattern Mining for finding association rules in market basket analysis. Finally, Spark’s RDD API also includes a lower-level matrix factorization method that will not be covered in this book. Collaborative Filtering with Alternating Least Squares ALS finds a -dimensional feature vector for each user and item such that the dot product of each user’s feature vector with each item’s feature vector approximates the user’s rating for that item. Therefore this only requires an input dataset of existing ratings between user-item pairs, with three columns: a user ID column, an item ID column (e.g., a movie), and a rating column. The ratings can either be explicit—a numerical rating that we aim to predict directly—or implicit —in which case each rating represents the strength of interactions observed between a user and item (e.g., number of visits to a particular page), which measures our level of confidence in the user’s preference for that item. Given this input DataFrame, the model will produce feature vectors that you can use to predict users’ ratings for items they have not yet rated. One issue to note in practice is that this algorithm does have a preference for serving things that are very common or that it has a lot of information on. If you’re introducing a new product that no users have expressed a preference for, the algorithm isn’t going to recommend it to many people. Additionally, if new users are onboarding onto the platform, they may not have any ratings in the training set. Therefore, the algorithm won’t know what to recommend them. These are examples of what we call the cold start problem, which we discuss later on in the chapter. In terms of scalability, one reason for Spark’s popularity for this task is that the algorithm and implementation in MLlib can scale to millions of users, millions of items, and billions of ratings. Model Hyperparameters These are configurations that we can specify to determine the structure of the model as well as the specific collaborative filtering problem we wish to solve: rank The rank term determines the dimension of the feature vectors learned for users and items. This should normally be tuned through experimentation. The core trade-off is that by specifying too high a rank, the algorithm may overfit the training data; but by specifying a low rank, then it may not make the best possible predictions. The default value is 10. alpha When training on implicit feedback (behavioral observations), the alpha sets a baseline confidence for preference. This has a default of 1.0 and should be driven through experimentation. regParam Controls regularization to prevent overfitting. You should test out different values for the regularization parameter to find the optimal value for your problem. The default is 0.1. implicitPrefs This Boolean value specifies whether you are training on implicit (true) or explicit (false) (refer back to the preceding discussion for an explanation of the difference between explicit and implicit). This value should be set based on the data that you’re using as input to the model. If the data is based off passive endorsement of a product (say, via a click or page visit), then you should use implicit preferences. In contrast, if the data is an explicit rating (e.g., the user gave this restaurant 4/5 stars), you should use explicit preferences. Explicit preferences are the default. nonnegative If set to true, this parameter configures the model to place non-negative constraints on the least-squares problem it solves and only return non-negative feature vectors. This can improve performance in some applications. The default value is false. Training Parameters The training parameters for alternating least squares are a bit different from those that we have seen in other models. That’s because we’re going to get more low-level control over how the data is distributed across the cluster. The groups of data that are distributed around the cluster are called blocks. Determining how much data to place in each block can have a significant impact on the time it takes to train the algorithm (but not the final result). A good rule of thumb is to aim for approximately one to five million ratings per block. If you have less data than that in each block, more blocks will not improve the algorithm’s performance. numUserBlocks This determines how many blocks to split the users into. The default is 10. numItemBlocks This determines how many blocks to split the items into. The default is 10. maxIter Total number of iterations over the data before stopping. Changing this probably won’t change your results a ton, so this shouldn’t be the first parameter you adjust. The default is 10. An example of when you might want to increase this is that after inspecting your objective history and noticing that it doesn’t flatline after a certain number of training iterations. checkpointInterval Checkpointing allows you to save model state during training to more quickly recover from node failures. You can set a checkpoint directory using SparkContext.setCheckpointDir. seed Specifying a random seed can help you replicate your results. Prediction Parameters Prediction parameters determine how a trained model should actually make predictions. In our case, there’s one parameter: the cold start strategy (set through coldStartStrategy). This setting determines what the model should predict for users or items that did not appear in the training set. The cold start challenge commonly arises when you’re serving a model in production, and new users and/or items have no ratings history, and therefore the model has no recommendation to make. It can also occur when using simple random splits as in Spark’s CrossValidator or TrainValidationSplit, where it is very common to encounter users and/or items in the evaluation set that are not in the training set. By default, Spark will assign NaN prediction values when it encounters a user and/or item that is not present in the actual model. This can be useful because you design your overall system to fall back to some default recommendation when a new user or item is in the system. However, this is undesirable during training because it will ruin the ability for your evaluator to properly measure the success of your model. This makes model selection impossible. Spark allows users to set the coldStartStrategy parameter to drop in order to drop any rows in the DataFrame of predictions that contain NaN values. The evaluation metric will then be computed over the nonNaN data and will be valid. drop and nan (the default) are the only currently supported cold-start strategies. Example This example will make use of a dataset that we have not used thus far in the book, the MovieLens movie rating dataset. This dataset, naturally, has information relevant for making movie recommendations. We will first use this dataset to train a model: // in Scala import org.apache.spark.ml.recommendation.ALS val ratings = spark.read.textFile("/data/sample_movielens_ratings.txt") .selectExpr("split(value , '::') as col") .selectExpr( "cast(col[0] as int) as userId", "cast(col[1] as int) as movieId", "cast(col[2] as float) as rating", "cast(col[3] as long) as timestamp") val Array(training, test) = ratings.randomSplit(Array(0.8, 0.2)) val als = new ALS() .setMaxIter(5) .setRegParam(0.01) .setUserCol("userId") .setItemCol("movieId") .setRatingCol("rating") println(als.explainParams()) val alsModel = als.fit(training) val predictions = alsModel.transform(test) # in Python from pyspark.ml.recommendation import ALS from pyspark.sql import Row ratings = spark.read.text("/data/sample_movielens_ratings.txt")\ .rdd.toDF()\ .selectExpr("split(value , '::') as col")\ .selectExpr( "cast(col[0] as int) as userId", "cast(col[1] as int) as movieId", "cast(col[2] as float) as rating", "cast(col[3] as long) as timestamp") training, test = ratings.randomSplit([0.8, 0.2]) als = ALS()\ .setMaxIter(5)\ .setRegParam(0.01)\ .setUserCol("userId")\ .setItemCol("movieId")\ .setRatingCol("rating") print als.explainParams() alsModel = als.fit(training) predictions = alsModel.transform(test) We can now output the top recommendations for each user or movie. The model’s recommendForAllUsers method returns a DataFrame of a userId, an array of recommendations, as well as a rating for each of those movies. recommendForAllItems returns a DataFrame of a movieId, as well as the top users for that movie: // in Scala alsModel.recommendForAllUsers(10) .selectExpr("userId", "explode(recommendations)").show() alsModel.recommendForAllItems(10) .selectExpr("movieId", "explode(recommendations)").show() # in Python alsModel.recommendForAllUsers(10)\ .selectExpr("userId", "explode(recommendations)").show() alsModel.recommendForAllItems(10)\ .selectExpr("movieId", "explode(recommendations)").show() Evaluators for Recommendation When covering the cold-start strategy, we can set up an automatic model evaluator when working with ALS. One thing that may not be immediately obvious is that this recommendation problem is really just a kind of regression problem. Since we’re predicting values (ratings) for given users, we want to optimize for reducing the total difference between our users’ ratings and the true values. We can do this using the same RegressionEvaluator that we saw in Chapter 27. You can place this in a pipeline to automate the training process. When doing this, you should also set the cold-start strategy to be drop instead of NaN and then switch it back to NaN when it comes time to actually make predictions in your production system: // in Scala import org.apache.spark.ml.evaluation.RegressionEvaluator val evaluator = new RegressionEvaluator() .setMetricName("rmse") .setLabelCol("rating") .setPredictionCol("prediction") val rmse = evaluator.evaluate(predictions) println(s"Root-mean-square error = $rmse") # in Python from pyspark.ml.evaluation import RegressionEvaluator evaluator = RegressionEvaluator()\ .setMetricName("rmse")\ .setLabelCol("rating")\ .setPredictionCol("prediction") rmse = evaluator.evaluate(predictions) print("Root-mean-square error = %f" % rmse) Metrics Recommendation results can be measured using both the standard regression metrics and some recommendation-specific metrics. It should come as no surprise that there are more sophisticated ways of measuring recommendation success than simply evaluating based on regression. These metrics are particularly useful for evaluating your final model. Regression Metrics We can recycle the regression metrics for recommendation. This is because we can simply see how close each prediction is to the actual rating for that user and item: // in Scala import org.apache.spark.mllib.evaluation.{ RankingMetrics, RegressionMetrics} val regComparison = predictions.select("rating", "prediction") .rdd.map(x => (x.getFloat(0).toDouble,x.getFloat(1).toDouble)) val metrics = new RegressionMetrics(regComparison) # in Python from pyspark.mllib.evaluation import RegressionMetrics regComparison = predictions.select("rating", "prediction")\ .rdd.map(lambda x: (x(0), x(1))) metrics = RegressionMetrics(regComparison) Ranking Metrics More interestingly, we also have another tool: ranking metrics. A RankingMetric allows us to compare our recommendations with an actual set of ratings (or preferences) expressed by a given user. RankingMetric does not focus on the value of the rank but rather whether or not our algorithm recommends an already ranked item again to a user. This does require some data preparation on our part. You may want to refer to Part II for a refresher on some of the methods. First, we need to collect a set of highly ranked movies for a given user. In our case, we’re going to use a rather low threshold: movies ranked above 2.5. Tuning this value will largely be a business decision: // in Scala import org.apache.spark.mllib.evaluation.{RankingMetrics, RegressionMetrics} import org.apache.spark.sql.functions.{col, expr} val perUserActual = predictions .where("rating > 2.5") .groupBy("userId") .agg(expr("collect_set(movieId) as movies")) # in Python from pyspark.mllib.evaluation import RankingMetrics, RegressionMetrics from pyspark.sql.functions import col, expr perUserActual = predictions\ .where("rating > 2.5")\ .groupBy("userId")\ .agg(expr("collect_set(movieId) as movies")) At this point, we have a collection of users, along with a truth set of previously ranked movies for each user. Now we will get our top 10 recommendations from our algorithm on a per-user basis. We will then see if the top 10 recommendations show up in our truth set. If we have a well-trained model, it will correctly recommend the movies a user already liked. If it doesn’t, it may not have learned enough about each particular user to successfully reflect their preferences: // in Scala val perUserPredictions = predictions .orderBy(col("userId"), col("prediction").desc) .groupBy("userId") .agg(expr("collect_list(movieId) as movies")) # in Python perUserPredictions = predictions\ .orderBy(col("userId"), expr("prediction DESC"))\ .groupBy("userId")\ .agg(expr("collect_list(movieId) as movies")) Now we have two DataFrames, one of predictions and another the top-ranked items for a particular user. We can pass them into the RankingMetrics object. This object accepts an RDD of these combinations, as you can see in the following join and RDD conversion: // in Scala val perUserActualvPred = perUserActual.join(perUserPredictions, Seq("userId")) .map(row => ( row(1).asInstanceOf[Seq[Integer]].toArray, row(2).asInstanceOf[Seq[Integer]].toArray.take(15) )) val ranks = new RankingMetrics(perUserActualvPred.rdd) # in Python perUserActualvPred = perUserActual.join(perUserPredictions, ["userId"]).rdd\ .map(lambda row: (row[1], row[2][:15])) ranks = RankingMetrics(perUserActualvPred) Now we can see the metrics from that ranking. For instance, we can see how precise our algorithm is with the mean average precision. We can also get the precision at certain ranking points, for instance, to see where the majority of the positive recommendations fall: // in Scala ranks.meanAveragePrecision ranks.precisionAt(5) # in Python ranks.meanAveragePrecision ranks.precisionAt(5) Frequent Pattern Mining In addition to ALS, another tool that MLlib provides for creating recommendations is frequent pattern mining. Frequent pattern mining, sometimes referred to as market basket analysis, looks at raw data and finds association rules. For instance, given a large number of transactions it might identify that users who buy hot dogs almost always purchase hot dog buns. This technique can be applied in the recommendation context, especially when people are filling shopping carts (either on or offline). Spark implements the FP-growth algorithm for frequent pattern mining. See the Spark documentation and ESL 14.2 for more information about this algorithm. Conclusion In this chapter, we discussed one of Spark’s most popular machine learning algorithms in practice—alternating least squares for recommendation. We saw how we can train, tune, and evaluate this model. In the next chapter, we’ll move to unsupervised learning and discuss clustering. Chapter 29. Unsupervised Learning This chapter will cover the details of Spark’s available tools for unsupervised learning, focusing specifically on clustering. Unsupervised learning is, generally speaking, used less often than supervised learning because it’s usually harder to apply and measure success (from an end-result perspective). These challenges can become exacerbated at scale. For instance, clustering in highdimensional space can create odd clusters simply because of the properties of high-dimensional spaces, something referred to as the curse of dimensionality. The curse of dimensionality describes the fact that as a feature space expands in dimensionality, it becomes increasingly sparse. This means that the data needed to fill this space for statistically meaningful results increases rapidly with any increase in dimensionality. Additionally, with high dimensions comes more noise in the data. This, in turn, may cause your model to hone in on noise instead of the true factors causing a particular result or grouping. Therefore in the model scalability table, we include computational limits, as well as a set of statistical recommendations. These are heuristics and should be helpful guides, not requirements. At its core, unsupervised learning is trying to discover patterns or derive a concise representation of the underlying structure of a given dataset. Use Cases Here are some potential use cases. At its core, these patterns might reveal topics, anomalies, or groupings in our data that may not have been obvious beforehand: Finding anomalies in data If the majority of values in a dataset cluster into a larger group with several small groups on the outside, those groups might warrant further investigation. Topic modeling By looking at large bodies of text, it is possible to find topics that exist across those different documents. Model Scalability Just like with our other models, it’s important to mention the basic model scalability requirements along with statistical recommendations. Table 29-1. Clustering model scalability reference Model Statistical recommendation Computation limits Training examples k-means 50 to 100 maximum Bisecting kmeans 50 to 100 maximum Features x clusters < 10 million No limit GMM 50 to 100 maximum Features x clusters < 10 million No limit LDA An interpretable number 1,000s of topics No limit Features x clusters < 10 million No limit Let’s get started by loading some example numerical data: // in Scala import org.apache.spark.ml.feature.VectorAssembler val va = new VectorAssembler() .setInputCols(Array("Quantity", "UnitPrice")) .setOutputCol("features") val sales = va.transform(spark.read.format("csv") .option("header", "true") .option("inferSchema", "true") .load("/data/retail-data/by-day/*.csv") .limit(50) .coalesce(1) .where("Description IS NOT NULL")) sales.cache() # in Python from pyspark.ml.feature import VectorAssembler va = VectorAssembler()\ .setInputCols(["Quantity", "UnitPrice"])\ .setOutputCol("features") sales = va.transform(spark.read.format("csv") .option("header", "true") .option("inferSchema", "true") .load("/data/retail-data/by-day/*.csv") .limit(50) .coalesce(1) .where("Description IS NOT NULL")) sales.cache() k-means -means is one of the most popular clustering algorithms. In this algorithm, a user-specified number of clusters ( ) are randomly assigned to different points in the dataset. The unassigned points are then “assigned” to a cluster based on their proximity (measured in Euclidean distance) to the previously assigned point. Once this assignment happens, the center of this cluster (called the centroid) is computed, and the process repeats. All points are assigned to a particular centroid, and a new centroid is computed. We repeat this process for a finite number of iterations or until convergence (i.e., when our centroid locations stop changing). This does not, however, mean that our clusters are always sensical. For instance, a given “logical” cluster of data might be split right down the middle simply because of the starting points of two distinct clusters. Thus, it is often a good idea to perform multiple runs of -means starting with different initializations. Choosing the right value for is an extremely important aspect of using this algorithm successfully, as well as a hard task. There’s no real prescription for the number of clusters you need, so you’ll likely have to experiment with different values and consider what you would like the end result to be. For more information on -means, see ISL 10.3 and ESL 14.3. Model Hyperparameters These are configurations that we specify to determine the basic structure of the model: This is the number of clusters that you would like to end up with. Training Parameters initMode The initialization mode is the algorithm that determines the starting locations of the centroids. The supported options are random and -means|| (the default). The latter is a parallelized variant of the -means|| method. While the details are not within the scope of this book, the thinking behind the latter method is that rather than simply choosing random initialization locations, the algorithm chooses cluster centers that are already well spread out to generate a better clustering. initSteps The number of steps for -means|| initialization mode. Must be greater than 0. (The default value is 2.) maxIter Total number of iterations over the data before stopping. Changing this probably won’t change your results a ton, so don’t make this the first parameter you look to adjust. The default is 20. tol Specifies a threshold by which changes in centroids show that we optimized our model enough, and can stop iterating early, before maxIter iterations. The default value is 0.0001. This algorithm is generally robust to these parameters, and the main trade-off is that running more initialization steps and iterations may lead to a better clustering at the expense of longer training time: Example // in Scala import org.apache.spark.ml.clustering.KMeans val km = new KMeans().setK(5) println(km.explainParams()) val kmModel = km.fit(sales) # in Python from pyspark.ml.clustering import KMeans km = KMeans().setK(5) print km.explainParams() kmModel = km.fit(sales) k-means Metrics Summary -means includes a summary class that we can use to evaluate our model. This class provides some common measures for -means success (whether these apply to your problem set is another question). The -means summary includes information about the clusters created, as well as their relative sizes (number of examples). We can also compute the within set sum of squared errors, which can help measure how close our values are from each cluster centroid, using computeCost. The implicit goal in -means is that we want to minimize the within set sum of squared error, subject to the given number of clusters: // in Scala val summary = kmModel.summary summary.clusterSizes // number of points kmModel.computeCost(sales) println("Cluster Centers: ") kmModel.clusterCenters.foreach(println) # in Python summary = kmModel.summary print summary.clusterSizes # number of points kmModel.computeCost(sales) centers = kmModel.clusterCenters() print("Cluster Centers: ") for center in centers: print(center) Bisecting k-means Bisecting -means is a variant of -means. The core difference is that instead of clustering points by starting “bottom-up” and assigning a bunch of different groups in the data, this is a top-down clustering method. This means that it will start by creating a single group and then splitting that group into smaller groups in order to end up with the number of clusters specified by the user. This is usually a faster method than -means and will yield different results. Model Hyperparameters These are configurations that we specify to determine the basic structure of the model: This is the number of clusters that you would like to end up with. Training Parameters minDivisibleClusterSize The minimum number of points (if greater than or equal to 1.0) or the minimum proportion of points (if less than 1.0) of a divisible cluster. The default is 1.0, meaning that there must be at least one point in each cluster. maxIter Total number of iterations over the data before stopping. Changing this probably won’t change your results a ton, so don’t make this the first parameter you look to adjust. The default is 20. Most of the parameters in this model should be tuned in order to find the best result. There’s no rule that applies to all datasets. Example // in Scala import org.apache.spark.ml.clustering.BisectingKMeans val bkm = new BisectingKMeans().setK(5).setMaxIter(5) println(bkm.explainParams()) val bkmModel = bkm.fit(sales) # in Python from pyspark.ml.clustering import BisectingKMeans bkm = BisectingKMeans().setK(5).setMaxIter(5) bkmModel = bkm.fit(sales) Bisecting k-means Summary Bisecting -means includes a summary class that we can use to evaluate our model, that is largely the same as the -means summary. This includes information about the clusters created, as well as their relative sizes (number of examples): // in Scala val summary = bkmModel.summary summary.clusterSizes // number of points kmModel.computeCost(sales) println("Cluster Centers: ") kmModel.clusterCenters.foreach(println) # in Python summary = bkmModel.summary print summary.clusterSizes # number of points kmModel.computeCost(sales) centers = kmModel.clusterCenters() print("Cluster Centers: ") for center in centers: print(center) Gaussian Mixture Models Gaussian mixture models (GMM) are another popular clustering algorithm that makes different assumptions than bisecting -means or -means do. Those algorithms try to group data by reducing the sum of squared distances from the center of the cluster. Gaussian mixture models, on the other hand, assume that each cluster produces data based upon random draws from a Gaussian distribution. This means that clusters of data should be less likely to have data at the edge of the cluster (reflected in the Guassian distribution) and much higher probability of having data in the center. Each Gaussian cluster can be of arbitrary size with its own mean and standard deviation (and hence a possibly different, ellipsoid shape). There are still user-specified clusters that will be created during training. A simplified way of thinking about Gaussian mixture models is that they’re like a soft version of -means. -means creates very rigid clusters—each point is only within one cluster. GMMs allow for a more nuanced cluster associated with probabilities, instead of rigid boundaries. For more information, see ESL 14.3. Model Hyperparameters These are configurations that we specify to determine the basic structure of the model: This is the number of clusters that you would like to end up with. Training Parameters maxIter Total number of iterations over the data before stopping. Changing this probably won’t change your results a ton, so don’t make this the first parameter you look to adjust. The default is 100. tol This value simply helps us specify a threshold by which changes in parameters show that we optimized our weights enough. A smaller value can lead to higher accuracy at the cost of performing more iterations (although never more than maxIter). The default value is 0.01. As with our -means model, these training parameters are less likely to have an impact than the number of clusters, . Example // in Scala import org.apache.spark.ml.clustering.GaussianMixture val gmm = new GaussianMixture().setK(5) println(gmm.explainParams()) val model = gmm.fit(sales) # in Python from pyspark.ml.clustering import GaussianMixture gmm = GaussianMixture().setK(5) print gmm.explainParams() model = gmm.fit(sales) Gaussian Mixture Model Summary Like our other clustering algorithms, Gaussian mixture models include a summary class to help with model evaluation. This includes information about the clusters created, like the weights, the means, and the covariance of the Gaussian mixture, which can help us learn more about the underlying structure inside of our data: // in Scala val summary = model.summary model.weights model.gaussiansDF.show() summary.cluster.show() summary.clusterSizes summary.probability.show() # in Python summary = model.summary print model.weights model.gaussiansDF.show() summary.cluster.show() summary.clusterSizes summary.probability.show() Latent Dirichlet Allocation Latent Dirichlet Allocation (LDA) is a hierarchical clustering model typically used to perform topic modelling on text documents. LDA tries to extract high-level topics from a series of documents and keywords associated with those topics. It then interprets each document as having a variable number of contributions from multiple input topics. There are two implementations that you can use: online LDA and expectation maximization. In general, online LDA will work better when there are more examples, and the expectation maximization optimizer will work better when there is a larger input vocabulary. This method is also capable of scaling to hundreds or thousands of topics. To input our text data into LDA, we’re going to have to convert it into a numeric format. You can use the CountVectorizer to achieve this. Model Hyperparameters These are configurations that we specify to determine the basic structure of the model: The total number of topics to infer from the data. The default is 10 and must be a positive number. docConcentration Concentration parameter (commonly named “alpha”) for the prior placed on documents’ distributions over topics (“theta”). This is the parameter to a Dirichlet distribution, where larger values mean more smoothing (more regularization). If not set by the user, then docConcentration is set automatically. If set to singleton vector [alpha], then alpha is replicated to a vector of length k in fitting. Otherwise, the docConcentration vector must be length . topicConcentration The concentration parameter (commonly named “beta” or “eta”) for the prior placed on a topic’s distributions over terms. This is the parameter to a symmetric Dirichlet distribution. If not set by the user, then topicConcentration is set automatically. Training Parameters These are configurations that specify how we perform training: maxIter Total number of iterations over the data before stopping. Changing this probably won’t change your results a ton, so don’t make this the first parameter you look to adjust. The default is 20. optimizer This determines whether to use EM or online training optimization to determine the LDA model. The default is online. learningDecay Learning rate, set as an exponential decay rate. This should be between (0.5, 1.0] to guarantee asymptotic convergence. The default is 0.51 and only applies to the online optimizer. learningOffset A (positive) learning parameter that downweights early iterations. Larger values make early iterations count less. The default is 1,024.0 and only applies to the online optimizer. optimizeDocConcentration Indicates whether the docConcentration (Dirichlet parameter for document-topic distribution) will be optimized during training. The default is true but only applies to the online optimizer. subsamplingRate The fraction of the corpus to be sampled and used in each iteration of mini-batch gradient descent, in range (0, 1]. The default is 0.5 and only applies to the online optimizer. seed This model also supports specifying a random seed for reproducibility. checkpointInterval This is the same checkpoint feature that we saw in Chapter 26. Prediction Parameters topicDistributionCol The column that will hold the output of the topic mixture distribution for each document. Example // in Scala import org.apache.spark.ml.feature.{Tokenizer, CountVectorizer} val tkn = new Tokenizer().setInputCol("Description").setOutputCol("DescOut") val tokenized = tkn.transform(sales.drop("features")) val cv = new CountVectorizer() .setInputCol("DescOut") .setOutputCol("features") .setVocabSize(500) .setMinTF(0) .setMinDF(0) .setBinary(true) val cvFitted = cv.fit(tokenized) val prepped = cvFitted.transform(tokenized) # in Python from pyspark.ml.feature import Tokenizer, CountVectorizer tkn = Tokenizer().setInputCol("Description").setOutputCol("DescOut") tokenized = tkn.transform(sales.drop("features")) cv = CountVectorizer()\ .setInputCol("DescOut")\ .setOutputCol("features")\ .setVocabSize(500)\ .setMinTF(0)\ .setMinDF(0)\ .setBinary(True) cvFitted = cv.fit(tokenized) prepped = cvFitted.transform(tokenized) // in Scala import org.apache.spark.ml.clustering.LDA val lda = new LDA().setK(10).setMaxIter(5) println(lda.explainParams()) val model = lda.fit(prepped) # in Python from pyspark.ml.clustering import LDA lda = LDA().setK(10).setMaxIter(5) print lda.explainParams() model = lda.fit(prepped) After we train the model, you will see some of the top topics. This will return the term indices, and we’ll have to look these up using the CountVectorizerModel that we trained in order to find out the true words. For instance, when we trained on the data our top 3 topics were hot, home, and brown after looking them up in our vocabulary: // in Scala model.describeTopics(3).show() cvFitted.vocabulary # in Python model.describeTopics(3).show() cvFitted.vocabulary These methods result in detailed information about the vocabulary used as well as the emphasis on particular terms. These can be helpful for better understanding the underlying topics. Due to space constraints, we can’t show this output. Using similar APIs, we can get some more technical measures like the log likelihood and perplexity. The goal of these tools is to help you optimize the number of topics, based on your data. When using perplexity in your success criteria, you should apply these metrics to a holdout set to reduce the overall perplexity of the model. Another option is to optimize to increase the log likelihood value on the holdout set. We can calculate each of these by passing a dataset into the following functions: model.logLikelihood and model.logPerplexity. Conclusion This chapter covered the most popular algorithms that Spark includes for unsupervised learning. The next chapter will bring us out of MLlib and talk about some of the advanced analytics ecosystem that has grown outside of Spark. Chapter 30. Graph Analytics The previous chapter covered some conventional unsupervised techniques. This chapter is going to dive into a more specialized toolset: graph processing. Graphs are data structures composed of nodes, or vertices, which are arbitrary objects, and edges that define relationships between these nodes. Graph analytics is the process of analyzing these relationships. An example graph might be your friend group. In the context of graph analytics, each vertex or node would represent a person, and each edge would represent a relationship. Figure 30-1 shows a sample graph. Figure 30-1. A sample graph with seven nodes and seven edges This particular graph is undirected, in that the edges do not have a specified “start” and “end” vertex. There are also directed graphs that specify a start and end. Figure 30-2 shows a directed graph where the edges are directional. Figure 30-2. A directed graph Edges and vertices in graphs can also have data associated with them. In our friend example, the weight of the edge might represent the intimacy between different friends; acquaintances would have low-weight edges between them, while married individuals would have edges with large weights. We could set this value by looking at communication frequency between nodes and weighting the edges accordingly. Each vertex (person) might also have data such as a name. Graphs are a natural way of describing relationships and many different problem sets, and Spark provides several ways of working in this analytics paradigm. Some business use cases could be detecting credit card fraud, motif finding, determining importance of papers in bibliographic networks (i.e., which papers are most referenced), and ranking web pages, as Google famously used the PageRank algorithm to do. Spark has long contained an RDD-based library for performing graph processing: GraphX. This provided a very low-level interface that was extremely powerful, but just like RDDs, wasn’t easy to use or optimize. GraphX remains a core part of Spark. Companies continue to build production applications on top of it, and it still sees some minor feature development. The GraphX API is well documented simply because it hasn’t changed much since its creation. However, some of the developers of Spark (including some of the original authors of GraphX) have recently created a next-generation graph analytics library on Spark: GraphFrames. GraphFrames extends GraphX to provide a DataFrame API and support for Spark’s different language bindings so that users of Python can take advantage of the scalability of the tool. In this book, we will focus on GraphFrames. GraphFrames is currently available as a Spark package, an external package that you need to load when you start up your Spark application, but may be merged into the core of Spark in the future. For the most part, there should be little difference in performance between the two (except for a huge user experience improvement in GraphFrames). There is some small overhead when using GraphFrames, but for the most part it tries to call down to GraphX where appropriate; and for most, the user experience gains greatly outweigh this minor overhead. HOW DOES GRAPHFRAMES COMPARE TO GRAPH DATABASES? Spark is not a database. Spark is a distributed computation engine, but it does not store data long-term or perform transactions. You can build a graph computation on top of Spark, but that’s fundamentally different from a database. GraphFrames can scale to much larger workloads than many graph databases and performs well for analytics but does not support transactional processing and serving. The goal of this chapter is to show you how to use GraphFrames to perform graph analytics on Spark. We are going to be doing this with publicly available bike data from the Bay Area Bike Share portal. TIP During the course of writing this book, this map and data have changed dramatically (even the naming!). We include a copy of the dataset inside the data folder of this book’s repository. Be sure to use that dataset to replicate the following results; and when you’re feeling adventurous, expand to the whole dataset! To get set up, you’re going to need to point to the proper package. To do this from the command line, you’ll run: ./bin/spark-shell --packages graphframes:graphframes:0.5.0-spark2.2-s_2.11 // in Scala val bikeStations = spark.read.option("header","true") .csv("/data/bike-data/201508_station_data.csv") val tripData = spark.read.option("header","true") .csv("/data/bike-data/201508_trip_data.csv") # in Python bikeStations = spark.read.option("header","true")\ .csv("/data/bike-data/201508_station_data.csv") tripData = spark.read.option("header","true")\ .csv("/data/bike-data/201508_trip_data.csv") Building a Graph The first step is to build the graph. To do this we need to define the vertices and edges, which are DataFrames with some specifically named columns. In our case, we’re creating a directed graph. This graph will point from the source to the location. In the context of this bike trip data, this will point from a trip’s starting location to a trip’s ending location. To define the graph, we use the naming conventions for columns presented in the GraphFrames library. In the vertices table we define our identifier as id (in our case this is of string type), and in the edges table we label each edge’s source vertex ID as src and the destination ID as dst: // in Scala val stationVertices = bikeStations.withColumnRenamed("name", "id").distinct() val tripEdges = tripData .withColumnRenamed("Start Station", "src") .withColumnRenamed("End Station", "dst") # in Python stationVertices = bikeStations.withColumnRenamed("name", "id").distinct() tripEdges = tripData\ .withColumnRenamed("Start Station", "src")\ .withColumnRenamed("End Station", "dst") We can now build a GraphFrame object, which represents our graph, from the vertex and edge DataFrames we have so far. We will also leverage caching because we’ll be accessing this data frequently in later queries: // in Scala import org.graphframes.GraphFrame val stationGraph = GraphFrame(stationVertices, tripEdges) stationGraph.cache() # in Python from graphframes import GraphFrame stationGraph = GraphFrame(stationVertices, tripEdges) stationGraph.cache() Now we can see the basic statistics about graph (and query our original DataFrame to ensure that we see the expected results): // in Scala println(s"Total Number of Stations: ${stationGraph.vertices.count()}") println(s"Total Number of Trips in Graph: ${stationGraph.edges.count()}") println(s"Total Number of Trips in Original Data: ${tripData.count()}") # in Python print "Total Number of Stations: " + str(stationGraph.vertices.count()) print "Total Number of Trips in Graph: " + str(stationGraph.edges.count()) print "Total Number of Trips in Original Data: " + str(tripData.count()) This returns the following results: Total Number of Stations: 70 Total Number of Trips in Graph: 354152 Total Number of Trips in Original Data: 354152 Querying the Graph The most basic way of interacting with the graph is simply querying it, performing things like counting trips and filtering by given destinations. GraphFrames provides simple access to both vertices and edges as DataFrames. Note that our graph retained all the additional columns in the data in addition to IDs, sources, and destinations, so we can also query those if needed: // in Scala import org.apache.spark.sql.functions.desc stationGraph.edges.groupBy("src", "dst").count().orderBy(desc("count")).show(10) # in Python from pyspark.sql.functions import desc stationGraph.edges.groupBy("src", "dst").count().orderBy(desc("count")).show(10) +--------------------+--------------------+-----+ | src| dst|count| +--------------------+--------------------+-----+ |San Francisco Cal...| Townsend at 7th| 3748| |Harry Bridges Pla...|Embarcadero at Sa...| 3145| ... | Townsend at 7th|San Francisco Cal...| 2192| |Temporary Transba...|San Francisco Cal...| 2184| +--------------------+--------------------+-----+ We can also filter by any valid DataFrame expression. In this instance, I want to look at one specific station and the count of trips in and out of that station: // in Scala stationGraph.edges .where("src = 'Townsend at 7th' OR dst = 'Townsend at 7th'") .groupBy("src", "dst").count() .orderBy(desc("count")) .show(10) # in Python stationGraph.edges\ .where("src = 'Townsend at 7th' OR dst = 'Townsend at 7th'")\ .groupBy("src", "dst").count()\ .orderBy(desc("count"))\ .show(10) +--------------------+--------------------+-----+ | src| dst|count| +--------------------+--------------------+-----+ |San Francisco Cal...| Townsend at 7th| 3748| | Townsend at 7th|San Francisco Cal...| 2734| ... | Steuart at Market| Townsend at 7th| 746| | Townsend at 7th|Temporary Transba...| 740| +--------------------+--------------------+-----+ Subgraphs Subgraphs are just smaller graphs within the larger one. We saw in the last section how we can query a given set of edges and vertices. We can use this query ability to create subgraphs: // in Scala val townAnd7thEdges = stationGraph.edges .where("src = 'Townsend at 7th' OR dst = 'Townsend at 7th'") val subgraph = GraphFrame(stationGraph.vertices, townAnd7thEdges) # in Python townAnd7thEdges = stationGraph.edges\ .where("src = 'Townsend at 7th' OR dst = 'Townsend at 7th'") subgraph = GraphFrame(stationGraph.vertices, townAnd7thEdges) We can then apply the following algorithms to either the original graph or the subgraph. Motif Finding Motifs are a way of expresssing structural patterns in a graph. When we specify a motif, we are querying for patterns in the data instead of actual data. In GraphFrames, we specify our query in a domain-specific language similar to Neo4J’s Cypher language. This language lets us specify combinations of vertices and edges and assign then names. For example, if we want to specify that a given vertex a connects to another vertex b through an edge ab, we would specify (a)[ab]->(b). The names inside parentheses or brackets do not signify values but instead what the columns for matching vertices and edges should be named in the resulting DataFrame. We can omit the names (e.g., (a)-[]->()) if we do not intend to query the resulting values. Let’s perform a query on our bike data. In plain English, let’s find all the rides that form a “triangle” pattern between three stations. We express this with the following motif, using the find method to query our GraphFrame for that pattern. (a) signifies the starting station, and [ab] represents an edge from (a) to our next station (b). We repeat this for stations (b) to (c) and then from (c) to (a): // in Scala val motifs = stationGraph.find("(a)-[ab]->(b); (b)-[bc]->(c); (c)-[ca]->(a)") # in Python motifs = stationGraph.find("(a)-[ab]->(b); (b)-[bc]->(c); (c)-[ca]->(a)") Figure 30-3 presents a visual representation of this query. Figure 30-3. Triangle motif in our triangle query The DataFrame we get from running this query contains nested fields for vertices a, b, and c, as well as the respective edges. We can now query this as we would a DataFrame. For example, given a certain bike, what is the shortest trip the bike has taken from station a, to station b, to station c, and back to station a? The following logic will parse our timestamps, into Spark timestamps and then we’ll do comparisons to make sure that it’s the same bike, traveling from station to station, and that the start times for each trip are correct: // in Scala import org.apache.spark.sql.functions.expr motifs.selectExpr("*", "to_timestamp(ab.`Start Date`, 'MM/dd/yyyy HH:mm') as abStart", "to_timestamp(bc.`Start Date`, 'MM/dd/yyyy HH:mm') as bcStart", "to_timestamp(ca.`Start Date`, 'MM/dd/yyyy HH:mm') as caStart") .where("ca.`Bike #` = bc.`Bike #`").where("ab.`Bike #` = bc.`Bike #`") .where("a.id != b.id").where("b.id != c.id") .where("abStart < bcStart").where("bcStart < caStart") .orderBy(expr("cast(caStart as long) - cast(abStart as long)")) .selectExpr("a.id", "b.id", "c.id", "ab.`Start Date`", "ca.`End Date`") .limit(1).show(false) # in Python from pyspark.sql.functions import expr motifs.selectExpr("*", "to_timestamp(ab.`Start Date`, 'MM/dd/yyyy HH:mm') as abStart", "to_timestamp(bc.`Start Date`, 'MM/dd/yyyy HH:mm') as bcStart", "to_timestamp(ca.`Start Date`, 'MM/dd/yyyy HH:mm') as caStart")\ .where("ca.`Bike #` = bc.`Bike #`").where("ab.`Bike #` = bc.`Bike #`")\ .where("a.id != b.id").where("b.id != c.id")\ .where("abStart < bcStart").where("bcStart < caStart")\ .orderBy(expr("cast(caStart as long) - cast(abStart as long)"))\ .selectExpr("a.id", "b.id", "c.id", "ab.`Start Date`", "ca.`End Date`") .limit(1).show(1, False) We see the fastest trip is approximately 20 minutes. Pretty fast for three different people (we assume) using the same bike! Note also that we had to filter the triangles returned by our motif query in this example. In general, different vertex IDs used in the query will not be forced to match distinct vertices, so you should perform this type of filtering if you want distinct vertices. One of the most powerful features of GraphFrames is that you can combine motif finding with DataFarme queries over the resulting tables to further narrow down, sort, or aggregate the patterns found. Graph Algorithms A graph is just a logical representation of data. Graph theory provides numerous algorithms for analyzing data in this format, and GraphFrames allows us to leverage many algorithms out of the box. Development continues as new algorithms are added to GraphFrames, so this list will most likely continue to grow. PageRank One of the most prolific graph algorithms is PageRank. Larry Page, cofounder of Google, created PageRank as a research project for how to rank web pages. Unfortunately, a complete explanation of how PageRank works is outside the scope of this book. However, to quote Wikipedia, the high-level explanation is as follows: PageRank works by counting the number and quality of links to a page to determine a rough estimate of how important the website is. The underlying assumption is that more important websites are likely to receive more links from other websites. PageRank generalizes quite well outside of the web domain. We can apply this right to our own data and get a sense for important bike stations (specifically, those that receive a lot of bike traffic). In this example, important bike stations will be assigned large PageRank values: // in Scala import org.apache.spark.sql.functions.desc val ranks = stationGraph.pageRank.resetProbability(0.15).maxIter(10).run() ranks.vertices.orderBy(desc("pagerank")).select("id", "pagerank").show(10) # in Python from pyspark.sql.functions import desc ranks = stationGraph.pageRank(resetProbability=0.15, maxIter=10) ranks.vertices.orderBy(desc("pagerank")).select("id", "pagerank").show(10) +--------------------+------------------+ | id| pagerank| +--------------------+------------------+ |San Jose Diridon ...| 4.051504835989922| |San Francisco Cal...|3.3511832964279518| ... | Townsend at 7th| 1.568456580534273| |Embarcadero at Sa...|1.5414242087749768| +--------------------+------------------+ GRAPH ALGORITHM APIS: PARAMETERS AND RETURN VALUES Most algorithms in GraphFrames are accessed as methods which take parameters (e.g., resetProbability in this PageRank example). Most algorithms return either a new GraphFrame or a single DataFrame. The results of the algorithm are stored as one or more columns in the GraphFrame’s vertices and/or edges or the DataFrame. For PageRank, the algorithm returns a GraphFrame, and we can extract the estimated PageRank values for each vertex from the new pagerank column. WARNING Depending on the resources available on your machine, this may take some time. You can always try a smaller set of data before running this to see the results. On Databricks Community Edition, this takes about 20 seconds to run, although some reviewers found it to take much longer on their machines. Interestingly, we see that Caltrain stations rank quite highly. This makes sense because these are natural connection points where a lot of bike trips might end up. Either as commuters move from home to the Caltrain station for their commute or from the Caltrain station to home. In-Degree and Out-Degree Metrics Our graph is a directed graph. This is due to the bike trips being directional, starting in one location and ending in another. One common task is to count the number of trips into or out of a given station. To measure trips in and out of stations, we will use a metric called in-degree and out-degree, respectively, as seen in Figure 30-4. Figure 30-4. In-degree and out-degree This is particularly applicable in the context of social networking because certain users may have many more inbound connections (i.e., followers) than outbound connections (i.e., people they follow). Using the following query, you can find interesting people in the social network who might have more influence than others. GraphFrames provides a simple way to query our graph for this information: // in Scala val inDeg = stationGraph.inDegrees inDeg.orderBy(desc("inDegree")).show(5, false) # in Python inDeg = stationGraph.inDegrees inDeg.orderBy(desc("inDegree")).show(5, False) The result of querying for the stations sorted by the highest in-degree: +----------------------------------------+--------+ |id |inDegree| +----------------------------------------+--------+ |San Francisco Caltrain (Townsend at 4th)|34810 | |San Francisco Caltrain 2 (330 Townsend) |22523 | |Harry Bridges Plaza (Ferry Building) |17810 | |2nd at Townsend |15463 | |Townsend at 7th |15422 | +----------------------------------------+--------+ We can query the out degrees in the same fashion: // in Scala val outDeg = stationGraph.outDegrees outDeg.orderBy(desc("outDegree")).show(5, false) # in Python outDeg = stationGraph.outDegrees outDeg.orderBy(desc("outDegree")).show(5, False) +---------------------------------------------+---------+ |id |outDegree| +---------------------------------------------+---------+ |San Francisco Caltrain (Townsend at 4th) |26304 | |San Francisco Caltrain 2 (330 Townsend) |21758 | |Harry Bridges Plaza (Ferry Building) |17255 | |Temporary Transbay Terminal (Howard at Beale)|14436 | |Embarcadero at Sansome |14158 | +---------------------------------------------+---------+ The ratio of these two values is an interesting metric to look at. A higher ratio value will tell us where a large number of trips end (but rarely begin), while a lower value tells us where trips often begin (but infrequently end): // in Scala val degreeRatio = inDeg.join(outDeg, Seq("id")) .selectExpr("id", "double(inDegree)/double(outDegree) as degreeRatio") degreeRatio.orderBy(desc("degreeRatio")).show(10, false) degreeRatio.orderBy("degreeRatio").show(10, false) # in Python degreeRatio = inDeg.join(outDeg, "id")\ .selectExpr("id", "double(inDegree)/double(outDegree) as degreeRatio") degreeRatio.orderBy(desc("degreeRatio")).show(10, False) degreeRatio.orderBy("degreeRatio").show(10, False) Those queries result in the following data: +----------------------------------------+------------------+ |id |degreeRatio | +----------------------------------------+------------------+ |Redwood City Medical Center |1.5333333333333334| |San Mateo County Center |1.4724409448818898| ... |Embarcadero at Vallejo |1.2201707365495336| |Market at Sansome |1.2173913043478262| +----------------------------------------+------------------+ +-------------------------------+------------------+ |id |degreeRatio | +-------------------------------+------------------+ |Grant Avenue at Columbus Avenue|0.5180520570948782| |2nd at Folsom |0.5909488686085761| ... |San Francisco City Hall |0.7928849902534113| |Palo Alto Caltrain Station |0.8064516129032258| +-------------------------------+------------------+ Breadth-First Search Breadth-first search will search our graph for how to connect two sets of nodes, based on the edges in the graph. In our context, we might want to do this to find the shortest paths to different stations, but the algorithm also works for sets of nodes specified through a SQL expression. We can specify the maximum of edges to follow with the maxPathLength, and we can also specify an edgeFilter to filter out edges that do not meet a requirement, like trips during nonbusiness hours. We’ll choose two fairly close stations so that this does not run too long. However, you can do interesting graph traversals when you have sparse graphs that have distant connections. Feel free to play around with the stations (especially those in other cities) to see if you can get distant stations to connect: // in Scala stationGraph.bfs.fromExpr("id = 'Townsend at 7th'") .toExpr("id = 'Spear at Folsom'").maxPathLength(2).run().show(10) # in Python stationGraph.bfs(fromExpr="id = 'Townsend at 7th'", toExpr="id = 'Spear at Folsom'", maxPathLength=2).show(10) +--------------------+--------------------+--------------------+ | from| e0| to| +--------------------+--------------------+--------------------+ |[65,Townsend at 7...|[913371,663,8/31/...|[49,Spear at Fols...| |[65,Townsend at 7...|[913265,658,8/31/...|[49,Spear at Fols...| ... |[65,Townsend at 7...|[903375,850,8/24/...|[49,Spear at Fols...| |[65,Townsend at 7...|[899944,910,8/21/...|[49,Spear at Fols...| +--------------------+--------------------+--------------------+ Connected Components A connected component defines an (undirected) subgraph that has connections to itself but does not connect to the greater graph, as illustrated in Figure 30-5. Figure 30-5. A connected component The connected components algorithm does not directly relate to our current problem because they assume an undirected graph. However, we can still run the algorithm, which just assumes that there are is no directionality associated with our edges. In fact, if we look at the bike share map, we assume that we would get two distinct connected components (Figure 30-6). Figure 30-6. A map of Bay Area bike share locations WARNING To run this algorithm, you will need to set a checkpoint directory which will store the state of the job at every iteration. This allows you to continue where you left off if the job crashes. This is probably one of the most expensive algorithms currently in GraphFrames, so expect delays. One thing you will likely have to do to run this algorithm on your local machine is take a sample of the data, just as we do in the following code example (taking a sample can help you get to a result without crashing the Spark application with garbage collection issues): // in Scala spark.sparkContext.setCheckpointDir("/tmp/checkpoints") # in Python spark.sparkContext.setCheckpointDir("/tmp/checkpoints") // in Scala val minGraph = GraphFrame(stationVertices, tripEdges.sample(false, 0.1)) val cc = minGraph.connectedComponents.run() # in Python minGraph = GraphFrame(stationVertices, tripEdges.sample(False, 0.1)) cc = minGraph.connectedComponents() From this query we get two connected components but not necessarily the ones we might expect. Our sample may not have all of the correct data or information so we’d probably need more compute resources to investigate further: // in Scala cc.where("component != 0").show() # in Python cc.where("component != 0").show() +----------+------------------+---------+-----------+---------+------------+----|station_id| id| lat| long|dockcount| landmark|in... +----------+------------------+---------+-----------+---------+------------+----| 47| Post at Kearney|37.788975|-122.403452| 19|San Franc...| ... | 46|Washington at K...|37.795425|-122.404767| 15|San Franc...| ... +----------+------------------+---------+-----------+---------+------------+----- Strongly Connected Components GraphFrames includes another related algorithm that relates to directed graphs: strongly connected components, which takes directionality into account. A strongly connected component is a subgraph that has paths between all pairs of vertices inside it. // in Scala val scc = minGraph.stronglyConnectedComponents.maxIter(3).run() # in Python scc = minGraph.stronglyConnectedComponents(maxIter=3) scc.groupBy("component").count().show() Advanced Tasks This is just a short selection of some of the features of GraphFrames. The GraphFrames library also includes features such as writing your own algorithms via a message-passing interface, triangle counting, and converting to and from GraphX. You can find more information in the GraphFrames documentation. Conclusion In this chapter, we took a tour of GraphFrames, a library for performing graph analysis on Apache Spark. We took a more tutorial-based approach, since this processing technique is not necessarily the first tool that people use when performing advanced analytics. It is nonetheless a powerful tool for analyzing relationships between different objects, and critical in many domains. The next chapter will talk about more cutting-edge functionality—specifically, deep learning. Chapter 31. Deep Learning Deep learning is one of the most exciting areas of development around Spark due to its ability to solve several previously difficult machine learning problems, especially those involving unstructured data such as images, audio, and text. This chapter will cover how Spark works in tandem with deep learning, and some of the different approaches you can use to work with Spark and deep learning together. Because deep learning is still a new field, many of the newest tools are implemented in external libraries. This chapter will not focus on packages that are necessarily core to Spark but rather on the massive amount of innovation in libraries built on top of Spark. We will start with several high-level ways to use deep learning on Spark, discuss when to use each one, and then go over the libraries available for them. As usual, we will include end-to-end examples. NOTE To make the most of this chapter you should know at least the basics of deep learning as well as the basics of Spark. With that being said, we point to an excellent resource at the beginning of this part of the book called the Deep Learning Book, by some of the top researchers in this area. What Is Deep Learning? To define deep learning, we must first define neural networks. A neural network is a graph of nodes with weights and activation functions. These nodes are organized into layers that are stacked on top of one another. Each layer is connected, either partially or completely, to the previous layer in the network. By stacking layers one after the other, these simple functions can learn to recognize more and more complex signals in the input: simple lines with one layer, circles and squares with the next layer, complex textures in another, and finally the full object or output you hope to identify. The goal is to train the network to associate certain inputs with certain outputs by tuning the weights associated with each connection and the values of each node in the network. Figure 31-1 shows the simple neural network. Figure 31-1. A neural network Deep learning, or deep neural networks, stack many of these layers together into various different architectures. Neural networks themselves have existed for decades, and have waxed and waned in terms of popularity for various machine learning problems. Recently, however, a combination of much larger datasets (e.g., the ImageNet corpus for object recognition), powerful hardware (clusters and GPUs), and new training algorithms have enabled training much larger neural networks that outperform previous approaches in many machine learning tasks. Typical machine learning techniques typically cannot continue to perform well as more data is added; their performance hits a ceiling. Deep learning can benefit from enormous amounts of data and information and it is not uncommon for deep learning datasets to be orders of magnitude larger than other machine learning datasets. Deep neural networks have now become the standard in computer vision, speech processing, and some natural language tasks, where they often “learn” better features than previous hand-tuned models. They are also actively being applied in other areas of machine learning. Apache Spark’s strength as a big data and parallel computing system makes it a natural framework to use with deep learning. Researchers and engineers have put a lot of effort into speeding up these neural network-like calculations. Nowadays, the most popular way to use neural networks or deep learning is to use a framework, implemented by a research institute or corporation. The most popular as of the time of this writing are TensorFlow, MXNet, Keras, and PyTorch. This area is rapidly evolving so it’s always worth searching around for others. Ways of Using Deep Learning in Spark For the most part, regardless of which application you are targeting, there are three major ways to use deep learning in Spark: Inference The simplest way to use deep learning is to take a pretrained model and apply it to large datasets in parallel using Spark. For example, you could use an image classification model, trained using a standard dataset like ImageNet, and apply it to your own image collection to identify pandas, flowers, or cars. Many organizations publish large, pretrained models on common datasets (e.g., Faster R-CNN and YOLO for object detection), so you can often take a model from your favorite deep learning framework and apply it in parallel using a Spark function. Using PySpark, you could simply call a framework such as TensorFlow or PyTorch in a map function to get distributed inference, though some of the libraries we discuss for it make further optimizations beyond simply calling these libraries in a map function. Featurization and transfer learning The next level of complexity is to use an existing model as a featurizer instead of taking its final output. Many deep learning models learn useful feature representations in their lower layers as they get trained for an end-to-end task. For example, a classifier trained on the ImageNet dataset will also learn low-level features present in all natural images, such as edges and textures. We can then use these features to learn models for a new problem not covered by the original dataset. This method is called transfer learning, and generally involves the last few layers of a pretrained model and retraining them with the data of interest. Transfer learning is also especially useful if you do not have a large amount of training data: training a full-blown network from scratch requires a dataset of hundreds of thousands of images, like ImageNet, to avoid overfitting, which will not be available in many business contexts. In contrast, transfer learning can work even with a few thousand images because it updates fewer parameters. Model training Spark can also be used to train a new deep learning model from scratch. There are two common methods here. First, you can use a Spark cluster to parallelize the training of a single model over multiple servers, communicating updates between them. Alternatively, some libraries let the user train multiple instances of similar models in parallel to try various model architectures and hyperparameters, accelerating the model search and tuning process. In both cases, Spark’s deep learning libraries make it simple to pass data from RDDs and DataFrames to deep learning algorithms. Finally, even if you do not wish to train your model in parallel, these libraries can be used to extract data from a cluster and export it to a singlemachine training script using the native data format of frameworks like TensorFlow. In all three cases, the deep learning code typically runs as part of a larger application that includes Extract, Transform, and Load (ETL) steps to parse the input data, I/O from various sources, and potentially batch or streaming inference. For these other parts of the application, you can simply use the DataFrame, RDD, and MLlib APIs described earlier in this book. One of Spark’s strengths is the ease of combining these steps into a single parallel workflow. Deep Learning Libraries In this section, we’ll survey a few of the most popular libraries available for deep learning in Spark. We will describe the main use cases of the library and link them to references or examples when possible. This list is not meant to be exhaustive, because the field is rapidly evolving. We encourage you to check each library’s website and the Spark documentation for the latest updates. MLlib Neural Network Support Spark’s MLlib currently has native support for a single deep learning algorithm: the ml.classification.MultilayerPerceptronClassifier class’s multilayer perceptron classifier. This class is limited to training relatively shallow networks containing fully connected layers with the sigmoid activation function and an output layer with a softmax activation function. This class is most useful for training the last few layers of a classification model when using transfer learning on top of an existing deep learning–based featurizer. For example, it can be added on top of the Deep Learning Pipelines library we describe later in this chapter to quickly perform transfer learning over Keras and TensorFlow models. TensorFrames TensorFrames is an inference and transfer learning-oriented library that makes it easy to pass data between Spark DataFrames and TensorFlow. It supports Python and Scala interfaces and focuses on providing a simple but optimized interface to pass data from TensorFlow to Spark and back. In particular, using TensorFrames to apply a model over Spark DataFrames is generally more efficient than calling a Python map function that directly invokes the TensorFlow model, due to faster data transfer and amortization of the startup cost. TensorFrames is most useful for inference, in both streaming and batch settings, and for transfer learning, where you can apply an existing model over raw data to featurize it, then learn the last layers using a MultilayerPerceptronClassifier or even a simpler logistic regression or random forest classifier over the data. BigDL BigDL is a distributed deep learning framework for Apache Spark primarily developed by Intel. It aims to support distributed training of large models as well as fast applications of these models using inference. One key advantage of BigDL over the other libraries described here is that it is primarily optimized to use CPUs instead of GPUs, making it efficient to run on an existing, CPU-based cluster (e.g., an Apache Hadoop deployment). BigDL provides high-level APIs to build neural networks from scratch and automatically distributes all operations by default. It can also train models described with the Keras DL library. TensorFlowOnSpark TensorFlowOnSpark is a widely used library that can train TensorFlow models in a parallel fashion on Spark clusters. TensorFlow includes some foundations to do distributed training, but it still needs to rely on a cluster manager for managing the hardware and data communications. It does not come with a cluster manager or a distributed I/O layer out of the box. TensorFlowOnSpark launches TensorFlow’s existing distributed mode inside a Spark job, and automatically feeds data from Spark RDDs or DataFrames into the TensorFlow job. If you already know how to use TensorFlow’s distributed mode, TensorFlowOnSpark makes it easy to launch your job inside a Spark cluster and pass it data processed with other Spark libraries (e.g., DataFrame transformations) from any input source Spark supports. TensorFlowOnSpark was originally developed at Yahoo! and is also used in production at other large organizations. The project also integrates with Spark’s ML Pipelines API. DeepLearning4J DeepLearning4j is an open-source, distributed deep learning project in Java and Scala that provides both single-node and distributed training options. One of its advantages over Pythonbased deep learning frameworks is that it was primarily designed for the JVM, making it more convenient for groups that do not wish to add Python to their development process. It includes a wide variety of training algorithms and support for CPUs as well as GPUs. Deep Learning Pipelines Deep Learning Pipelines is an open source package from Databricks that integrates deep learning functionality into Spark’s ML Pipelines API. The package existing deep learning frameworks (TensorFlow and Keras at the time of writing), but focuses on two goals: Incorporating these frameworks into standard Spark APIs (such as ML Pipelines and Spark SQL) to make them very easy to use Distributing all computation by default For example, Deep Learning Pipelines provides a DeepImageFeaturizer class that acts as a transformer in the Spark ML Pipeline API, allowing you to build a transfer learning pipeline in just a few lines of code (e.g., by adding a perceptron or logistic regression classifier on top). Likewise, the library supports parallel grid search over multiple model parameters using MLlib’s grid search and cross-validation API. Finally, users can export an ML model as a Spark SQL user-defined function and make it available to analysts using SQL or streaming applications. At the time of writing (summer 2017), Deep Learning Pipelines is under heavy development, so we encourage you to check its website for the latest updates. Table 31-1 summarizes the various deep learning libraries and the main use cases they support: Table 31-1. Deep learning libraries Library Underlying DL framework Use cases BigDL BigDL Distributed training, inference, ML Pipeline integration DeepLearning4J DeepLearning4J Inference, transfer learning, distributed training Deep Learning Pipelines TensorFlow, Keras Inference, transfer learning, multi-model training, ML Pipeline and Spark SQL integration MLlib Perceptron Spark Distributed training, ML Pipeline integration TensorFlowOnSpark TensorFlow Distributed training, ML Pipeline integration TensorFrames Inference, transfer learning, DataFrame integration TensorFlow While there are several approaches different companies have taken to integrating Spark and deep learning libraries, the one currently aiming for the closest integration with MLlib and DataFrames is Deep Learning Pipelines. This library aims to improve Spark’s support for image and tensor data (which will be integrated into the core Spark codebase in Spark 2.3), and to make all deep learning functionality available in the ML Pipeline API. Its friendly API makes it the simplest way to run deep learning on Spark today and will be the focus of the remaining sections in this chapter. A Simple Example with Deep Learning Pipelines As we described, Deep Learning Pipelines provides high-level APIs for scalable deep learning by integrating popular deep learning frameworks with ML Pipelines and Spark SQL. Deep Learning Pipelines builds on Spark’s ML Pipelines for training and on Spark DataFrames and SQL for deploying models. It includes high-level APIs for common aspects of deep learning so they can be done efficiently in a few lines of code: Working with images in Spark DataFrames; Applying deep learning models at scale, whether they are your own or standard popular models, to image and tensor data; Transfer learning using common pretrained deep learning models; Exporting models as Spark SQL functions to make it simple for all kinds of users to take advantage of deep learning; and Distributed deep learning hyperparameter tuning via ML Pipelines. Deep Learning Pipelines currently only offers an API in Python, which is designed to work closely with existing Python deep learning packages such as TensorFlow and Keras. Setup Deep Learning Pipelines is a Spark Package, so we’ll load it just like we loaded GraphFrames. Deep Learning Pipelines works on Spark 2.x and the package can be found here. You’re going to need to install a few Python dependencies, including TensorFrames, TensorFlow, Keras, and h5py. Make sure these are installed across both your driver and worker machines. We’ll use the flowers dataset from the TensorFlow retraining tutorial. Now if you’re running this on a cluster of machines, you’re going to need a way to put these files on a distributed file system once you download them. We include a sample of these images in the book’s GitHub Repository. Images and DataFrames One of the historical challenges when working with images in Spark is that getting them into a DataFrame was difficult and tedious. Deep Learning Pipelines includes utility functions that make loading and decoding images in a distributed fashion easy. This is an area that’s changing rapidly. Currently, this is a part of Deep Learning Pipelines. Basic image loading and representation will be included in Spark 2.3. While it is not released yet, all of the examples in this chapter should be compatible with this upcoming version of Spark: from sparkdl import readImages img_dir = '/data/deep-learning-images/' image_df = readImages(img_dir) The resulting DataFrame contains the path and then the image along with some associated metadata: image_df.printSchema() root |-|-| | | | | filePath: string (nullable = false) image: struct (nullable = true) |-- mode: string (nullable = false) |-- height: integer (nullable = false) |-- width: integer (nullable = false) |-- nChannels: integer (nullable = false) |-- data: binary (nullable = false) Transfer Learning Now that we have some data, we can get started with some simple transfer learning. Remember, this means leveraging a model that someone else created and modifying it to better suit our own purposes. First, we will load the data for each type of flower and create a training and test set: from sparkdl import readImages from pyspark.sql.functions import lit tulips_df = readImages(img_dir + "/tulips").withColumn("label", lit(1)) daisy_df = readImages(img_dir + "/daisy").withColumn("label", lit(0)) tulips_train, tulips_test = tulips_df.randomSplit([0.6, 0.4]) daisy_train, daisy_test = daisy_df.randomSplit([0.6, 0.4]) train_df = tulips_train.unionAll(daisy_train) test_df = tulips_test.unionAll(daisy_test) In the next step we will leverage a transformer called the DeepImageFeaturizer. This will allow us to leverage a pretrained model called Inception, a powerful neural network successfully used to identify patterns in images. The version we are using is pretrained to work well with images of various common objects and animals. This is one of the standard pretrained models that ship with the Keras library. However, this particular neural network is not trained to recognize daisies and roses. So we’re going to use transfer learning in order to make it into something useful for our own purposes: distinguishing different flower types. Note that we can use the same ML Pipeline concepts we learned about throughout this part of the book and leverage them with Deep Learning Pipelines: DeepImageFeaturizer is just an ML transformer. Additionally, all that we’ve done to extend this model is add on a logistic regression model in order to facilitate the training of our end model. We could use another classifier in its place. The following code snippet demonstrates adding this model (note this may take time to complete as it’s a fairly resource intensive process): from pyspark.ml.classification import LogisticRegression from pyspark.ml import Pipeline from sparkdl import DeepImageFeaturizer featurizer = DeepImageFeaturizer(inputCol="image", outputCol="features", modelName="InceptionV3") lr = LogisticRegression(maxIter=1, regParam=0.05, elasticNetParam=0.3, labelCol="label") p = Pipeline(stages=[featurizer, lr]) p_model = p.fit(train_df) Once we’ve trained the model, we can use the same classification evaluator we used in Chapter 25. We can specify the metric we’d like to test and then evaluate it: from pyspark.ml.evaluation import MulticlassClassificationEvaluator tested_df = p_model.transform(test_df) evaluator = MulticlassClassificationEvaluator(metricName="accuracy") print("Test set accuracy = " + str(evaluator.evaluate(tested_df.select( "prediction", "label")))) With our DataFrame of examples, we can inspect the rows and images in which we made mistakes in the previous training: from pyspark.sql.types import DoubleType from pyspark.sql.functions import expr # a simple UDF to convert the value to a double def _p1(v): return float(v.array[1]) p1 = udf(_p1, DoubleType()) df = tested_df.withColumn("p_1", p1(tested_df.probability)) wrong_df = df.orderBy(expr("abs(p_1 - label)"), ascending=False) wrong_df.select("filePath", "p_1", "label").limit(10).show() Applying deep learning models at scale Spark DataFrames are a natural construct for applying deep learning models to a large-scale dataset. Deep Learning Pipelines provides a set of Transformers for applying TensorFlow graphs and TensorFlow-backed Keras models at scale. In addition, popular image models can be applied out of the box, without requiring any TensorFlow or Keras code. The transformers, backed by the Tensorframes library, efficiently handle the distribution of models and data to Spark tasks. Applying Popular Models There are many standard deep learning models for images. If the task at hand is very similar to what the models provide (e.g., object recognition with ImageNet classes), or merely for exploration, you can use the transformer DeepImagePredictor by simply specifying the model name. Deep Learning Pipelines supports a variety of standard models included in Keras, which are listed on its website. The following is an example of using DeepImagePredictor: from sparkdl import readImages, DeepImagePredictor image_df = readImages(img_dir) predictor = DeepImagePredictor( inputCol="image", outputCol="predicted_labels", modelName="InceptionV3", decodePredictions=True, topK=10) predictions_df = predictor.transform(image_df) Notice that the predicted_labels column shows “daisy” as a high probability class for all sample flowers using this base model. However, as can be seen from the differences in the probability values, the neural network has the information to discern the two flower types. As we can see, our transfer learning example was able to properly learn the differences between daisies and tulips starting from the base model: df = p_model.transform(image_df) Applying custom Keras models Deep Learning Pipelines also allows us to apply a Keras model in a distributed manner using Spark. To do this, check the user guide on the KerasImageFileTransformer. This loads a Keras model and applies it to a DataFrame column. Applying TensorFlow models Deep Learning Pipelines, through its integration with TensorFlow, can be used to create custom transformers that manipulate images using TensorFlow. For instance, you could create a transformer to change the size of an image or modify the color spectrum. To do this, use the TFImageTransformer class. Deploying models as SQL functions Another option is to deploy a model as a SQL function allowing any user who knows SQL to be able to use a deep learning model. Once this function is used, the resulting UDF function takes a column and produces the output of the particular model. For instance, you could apply Inception v3 to a variety of images by using the registerKeraImageUDF class: from keras.applications import InceptionV3 from sparkdl.udf.keras_image_model import registerKerasImageUDF from keras.applications import InceptionV3 registerKerasImageUDF("my_keras_inception_udf", InceptionV3(weights="imagenet")) This way, the power of deep learning is available to any Spark user, not just the specialist who built the model. Conclusion This chapter discussed several common approaches to using deep learning in Spark. We covered a variety of available libraries and then worked through some basic examples of common tasks. This area of Spark is under very active development and will continue to advance as time moves on so it’s worth checking in on the libraries to learn more as time goes on! Over time, the authors of this book hope to keep this chapter up to date with current developments. Part VII. Ecosystem Chapter 32. Language Specifics: Python (PySpark) and R (SparkR and sparklyr) This chapter will cover some of the more nuanced language specifics of Apache Spark. We’ve seen a huge number of PySpark examples throughout the book. In Chapter 1, we discussed at a high level how Spark runs code from other languages. Let’s talk through some of the more specific integrations: PySpark SparkR sparklyr As a reminder, Figure 32-1 shows the fundamental architecture for these specific languages. Figure 32-1. The Spark Driver Now let’s cover each of these in depth. PySpark We covered a ton of PySpark throughout this book. In fact, PySpark is included alongside Scala and SQL in nearly every chapter in this book. Therefore, this section will be short and sweet, covering only the details that are relevant to Spark itself. As we discussed in Chapter 1, Spark 2.2 included a way to install PySpark with pip. Simply, pip install pyspark will make it available as a package on your local machine. This is new, so there may be some bugs to fix, but it is something that you can leverage in your projects today. Fundamental PySpark Differences If you’re using the structured APIs, your code should run just about as fast as if you had written it in Scala, except if you’re not using UDFs in Python. If you’re using a UDF, you may have a performance impact. Refer back to Chapter 6 for more information on why this is the case. If you’re using the unstructured APIs, specifically RDDs, then your performance is going to suffer (at the cost of a bit more flexibility). We touch on this reasoning in Chapter 12, but the fundamental idea is that Spark is going to have to work a lot harder converting information from something that Spark and the JVM can understand to Python and back again. This includes both functions as well as data and is a process known as serialization. We’re not saying it never makes sense to use them; it’s just something to be aware of when doing so. Pandas Integration One of the powers of PySpark is its ability to work across programming models. For instance, a common pattern is to perform very large-scale ETL work with Spark and then collect the (singlemachine-sized) result to the driver and then leverage Pandas to manipulate it further. This allows you to use a best-in-class tool for the best task at hand—Spark for big data and Pandas for small data: import pandas as pd df = pd.DataFrame({"first":range(200), "second":range(50,250)}) sparkDF = spark.createDataFrame(df) newPDF = sparkDF.toPandas() newPDF.head() These niceties make working with data big and small easy with Spark. Spark’s community continues to focus on improving this interoperability with various other projects, so the integration between Spark and Python will continue to improve. For example, at the time of writing, the community is actively working on Vectorized UDFs (SPARK-21190), which add a mapBatches API to let you process a Spark DataFrame as a series of Pandas data frames in Python instead of converting each individual row to a Python object. This feature is targeted to appear in Spark 2.3. R on Spark The rest of this chapter will cover R, Spark’s newest officially supported language. R is a language and environment for statistical computing and graphics. It is similar to the S language and environment developed at Bell Laboratories by John Chambers (of no relation to one of the authors of this book) and colleagues. The R langauge has been around for decades and is consistently popular among statisticians and those doing research in numerical computing. R is steadily becoming a first-class citizen in Spark and provides the simplest open source interface for distributed computation to the R language. The popularity of R for performing single-machine data analysis and advanced analytics makes it an excellent complement to Spark. There are two core initiatives to making this partnership a reality: SparkR and sparklyr. These packages take slightly different approaches to provide similar functionality. SparkR provides a DataFrame API similar to R’s data.frame, while sparklyr is based on the popular dplyr package for accessing structured data. You can use whichever you prefer in your code, but over time we expect that the community might converge toward a single integrated package. We will cover both packages here to let you choose which API you prefer. For the most part, both of these projects are mature and well supported, albeit by slightly different communities. They both support Spark’s structured APIs and allow for machine learning. We will elaborate on their differences in the next sections. SparkR SparkR is an R package (originating as a collaborative research project between UC Berkeley, Databricks, and MIT CSAIL) that provides a frontend to Apache Spark based on familiar R APIs. SparkR is conceptually similar to R’s built-in data.frame API, except for some departures from the API semantics, such as lazy evaluation. SparkR is a part of the official Spark project and is supported as such. See the documentation for SparkR for more information. Pros and cons of using SparkR instead of other languages The reasons we would recommend that you use SparkR as opposed to PySpark are the following. You are familiar with R and want to take the smallest step to leverage the capabilities of Spark: You want to leverage R-specific functionality or libraries (say the excellent ggplot2 library) and would like to work with big data in the process. R is a powerful programming language that provides a lot of advantages over other languages when it comes to certain tasks. However, it has its share of shortcomings like natively working with distributed data. SparkR aims to fill this gap and does a great job enabling users to be successful on both small and large data, in a conceptual way similar to PySpark and Pandas. Setup Let’s take a look at how to use SparkR. Naturally, you will need to have R installed on your system to follow along in this chapter. To start up the shell, in your Spark home folder, run ./bin/sparkR to start SparkR. This will automatically create a SparkSession for you. If you were to run SparkR from RStudio, you would have to do something like the following: library(SparkR) spark <- sparkR.session() Once we’ve started the shell, we can run Spark commands. For instance, we can read in a CSV file like we saw in Chapter 9: retail.data <- read.df( "/data/retail-data/all/", "csv", header="true", inferSchema="true") print(str(retail.data)) We can take some rows from this SparkDataFrame and convert them to a standard R data.frame type: local.retail.data <- take(retail.data, 5) print(str(local.retail.data)) Key Concepts Now that we saw some very basic code, let’s reiterate key concepts. First, SparkR is still Spark. Basically, all the tools that you have seen across the entire book apply directly to SparkR. It runs according to the same principles as PySpark and has almost all of the same functionality available as PySpark. As shown in Figure 32-1, there is a gateway that connects the R process to the JVM that contains a SparkSession, and SparkR converts user code into structured Spark manipulations across the cluster. This makes its efficiency on par with Python and Scala when using the structured APIs. SparkR has no support for RDDs or other low-level APIs. While SparkR is used less than PySpark or Scala, it’s still popular and continues to grow. For those that want to know enough Spark to leverage SparkR effectively, we recommend reading the following section, along with Parts I and II of this book. When working through those other chapters, feel free to try and use SparkR in place of Python or Scala. You’ll see that once you get the hang of it, it’s easy to translate between the various languages. The rest of this chapter will explain the most important differences between SparkR and “standard” R to make it easier to be productive with SparkR faster. The first thing we should cover is the different between local types and Spark types. A data.frame type’s core difference with the Spark version is that it is available in memory and is usually directly available in that particular process. A SparkDataFrame is just a logical representation of a series of manipulations. Therefore when we manipulate a data.frame, we’ll see our results right away. On a SparkDataFrame, we are going to logically manipulate the data using the same transformation and action concepts that we saw throughout the book. Once we have a SparkDataFrame, we can collect it to a data.frame similar to how we can read in data using Spark. We can also collect it into a local data.frame with the following code (using the SparkDataFrame we created in “Setup”): # collect brings it from Spark to your local environment collect(count(groupBy(retail.data, "country"))) # createDataFrame comverts a data.frame # from your local environment to Spark This difference is of consequence for end users. Certain functions or assumptions that apply to local data.frames do not apply in Spark. For instance, we cannot index a SparkDataFrame according to a particular row. Additionally, we cannot change point values in a SparkDataFrame but can do that in a local data.frame. Function masking One frequent “gotcha” when users come to SparkR is that certain functions are masked by SparkR. When I imported SparkR, I received the following message: The following objects are masked from ‘package:stats’: cov, filter, lag, na.omit, predict, sd, var, window The following objects are masked from ‘package:base’: as.data.frame, colnames, ... This means that if we wish to call these masked functions, we need to be explicit about the package that we’re calling them from or at least understand which function masks another. The ? can be helpful in determining these conflicts: ?na.omit # refers to SparkR due to package loading order ?stats::na.omit # refers explicitly to stats ?SparkR::na.omit # refers explicitly to sparkR's null value filtering SparkR functions only apply to SparkDataFrames One implication of function masking is that functions that worked on objects previously may no longer work on them after you bring in the SparkR package. This is because SparkR functions only apply on Spark objects. For instance, we cannot use the sample function on a standard data.frame because Spark takes that function name: sample(mtcars) # fails What you have to do instead is explicitly use the base sample function. Additionally the function signatures differ between the two functions, which means that even if you are familiar with the syntax and argument order for one particular library, it does not necessarily mean it’s the same order for SparkR: base::sample(some.r.data.frame) # some.r.data.frame = R data.frame type Data manipulation Data manipulation in SparkR is conceptually the same as Spark’s DataFrame API in other languages. The core difference is in the syntax, largely due to us running R code and not another language. Aggregations, filtering, and many of the functions that you can find in the other chapters throughout this book are also available in R. For the most part, you can look at the names of functions or manipulations that you find throughout this book and find out if they are available in SparkR by running ?. This should work the vast majority of the time, as there is good coverage of structured SQL functions: ?to_date # to Data DataFrame column manipulation SQL is largely the same. We can specify SQL commands that we can then manipulate as DataFrames. For instance, we can find all tables that contain the word “production” in them: tbls <- sql("SHOW TABLES") collect( select( filter(tbls, like(tbls$tableName, "%production%")), "tableName", "isTemporary")) We can also use the popular magrittr package to make this code more readable, leveraging the piping operator to chain our transformations in a more functional and readable syntax: library(magrittr) tbls %>% filter(like(tbls$tableName, "%production%")) %>% select("tableName", "isTemporary") %>% collect() Data sources SparkR supports all of the data sources that Spark supports, including third-party packages. We can see in the following snippet that we simply specify the options using a slightly different syntax: retail.data <- read.df( "/data/retail-data/all/", "csv", header="true", inferSchema="true") flight.data <- read.df( "/data/flight-data/parquet/2010-summary.parquet", "parquet") Refer back to Chapter 9 for more information. Machine learning Machine learning is a fundamental part of the R language, as well as of Spark. From SparkR there is a decent availability of Spark MLlib algorithms. Typically they arrive in R one or two versions after they are introduced in Scala or Python. As of Spark 2.1, the following algorithms are supported in SparkR: spark.glm or glm: Generalized linear model spark.survreg: Accelerated failure time (AFT) survival regression model spark.naiveBayes: Naive Bayes model spark.kmeans: -means model spark.logit: Logistic regression model spark.isoreg: Isotonic regression model spark.gaussianMixture: Gaussian mixture model spark.lda: Latent Dirichlet allocation (LDA) model spark.mlp: Multilayer perceptron classification model spark.gbt: Gradient boosted tree model for regression and classification spark.randomForest: Random forest model for regression and classification spark.als: Alternating least squares (ALS) matrix factorization model spark.kstest: Kolmogorov-Smirnov test Under the hood, SparkR uses MLlib to train the model, which means that most everything covered in Part VI is relevant for SparkR users. Users can call summary to print a summary of the fitted model, predict to make predictions on new data, and write.ml/read.ml to save/load fitted models. SparkR supports a subset of the available R formula operators for model fitting, including ~, ., :, +, and -. Here’s an example of running a simple regression on the retail dataset: model <- spark.glm(retail.data, Quantity ~ UnitPrice + Country, family='gaussian') summary(model) predict(model, retail.data) write.ml(model, "/tmp/myModelOutput", overwrite=T) newModel <- read.ml("/tmp/myModelOutput") The API is consistent across models, although not all models support detailed summary outputs like we saw with glm. For more information about specific models or preprocessing techniques, see the corresponding chapters in Part VI. While this pales in comparison to R’s extensive collection of statistical algorithms and analysis libraries, many users do not require the scale that Spark provides for the actual training and usage of their machine learning algorithms. Users have the opportunity to build training sets on large data using Spark and then collect that dataset to their local environment for training on a local data.frame. User-defined functions In SparkR, there are several ways of running user-defined functions. A user-defined function is one that is created in the native language and run on the server in that same native langauge. These run, for the most part, in the same way that a Python UDF runs, by performing serialization into and out of the JVM of the function. The different kinds of UDFs you can define are as follows: First, spark.lapply lets you run multiple instances of a function in Spark on different parameter values provided in an R collection. This is a great way of performing grid search and comparing the results: families <- c("gaussian", "poisson") train <- function(family) { model <- glm(Sepal.Length ~ Sepal.Width + Species, iris, family = family) summary(model) } # Return a list of model's summaries model.summaries <- spark.lapply(families, train) # Print the summary of each model print(model.summaries) Second, dapply and dapplyCollect let you process SparkDataFrame data using custom code. In particular, these functions will take each partition of the SparkDataFrame, convert it to an R data.frame inside of an executor, and then call your R code over that partition (represented as an R data.frame). They will then return the results: a SparkDataFrame for dapply or a local data.frame for dapplyCollect. To use dapply, which returns a SparkDataFrame, you must specify the output schema that will result from the transformation so that Spark understands what kind of data you will return. For example, the following code will allow you to train a local R model per partition in your SparkDataFrame, assuming you partition your data according to the correct keys: df <- withColumnRenamed(createDataFrame(as.data.frame(1:100)), "1:100", "col") outputSchema <- structType( structField("col", "integer"), structField("newColumn", "double")) udfFunc <- function (remote.data.frame) { remote.data.frame['newColumn'] = remote.data.frame$col * 2 remote.data.frame } # outputs SparkDataFrame, so it requires a schema take(dapply(df, udfFunc, outputSchema), 5) # collects all results to a, so no schema required. # however this will fail if the result is large dapplyCollect(df, udfFunc) Finally, the gapply and gapplyCollect functions apply a UDF to a group of data in a fashion similar to dapply. In fact, these two methods are largely the same, except that one operates on a generic SparkDataFrame, and the other applies to a grouped DataFrame. The gapply function will apply this function on a per-group basis and by passing in the key as the first parameter to the function that you define. In this way, you can be sure to have a function customized according to each particular group: local <- as.data.frame(1:100) local['groups'] <- c("a", "b") df <- withColumnRenamed(createDataFrame(local), "1:100", "col") outputSchema <- structType( structField("col", "integer"), structField("groups", "string"), structField("newColumn", "double")) udfFunc <- function (key, remote.data.frame) { if (key == "a") { remote.data.frame['newColumn'] = remote.data.frame$col * 2 } else if (key == "b") { remote.data.frame['newColumn'] = remote.data.frame$col * 3 } else if (key == "c") { remote.data.frame['newColumn'] = remote.data.frame$col * 4 } remote.data.frame } # outputs SparkDataFrame, so it requires a schema take(gapply(df, "groups", udfFunc, outputSchema), 50) gapplyCollect(df, "groups", udfFunc) SparkR will continue to grow as a part of Spark; and if you’re familiar with R and a little bit of Spark, this can be a very powerful tool. sparklyr sparklyr is a newer package from the RStudio team based on the popular dplyr package for structured data. This package is fundamentally different from SparkR and its authors take a more opinionated stance toward what the integration between Spark and R should do. This means that sparklyr sheds some of the Spark concepts that are available throughout this book, like the SparkSession, and uses its own ideas instead. For some, this means that sparklyr takes a R-first approach instead of SparkR’s approach of closely matching Python and Scala APIs. That approach speaks to its origins as a framework; sparklyr was created within the R community by the folks at RStudio (the popular R IDE), rather than being created by the Spark community. Whether sparklyr’s or SparkR’s approach is better or worse completely depends on the end user’s preference. In short, sparklyr provides an improved experience for R users familiar with dplyr, with slightly less overall functionality than SparkR (which may change over time). Specifically, sparklyr provides a complete dplyr backend to Spark, making it easy to take the dplyr code that you run today on your local machine and make it distributed. The implication of the dplyr backend architecture is that the same functions you use on local data.frame objects apply in a distributed manner to distributed Spark DataFrames. In essence, scaling up requires no code changes. Since functions apply to both single node and distributed DataFrames, this architecture addresses one of the core challenges with SparkR today, where function masking can lead to strange debugging scenarios. In addition, this architectural choice makes sparklyr an easier transition than simply using SparkR. Like SparkR, sparklyr is an evolving project; and when this book is published, the sparklyr project will have evolved further. For the most up-to-date reference, you should see the sparklyr website. The following sections provide a lightweight comparison and won’t go into depth on this particular project. Let’s get started with some handson examples of sparklyr. The first thing we need to do is install the package: install.packages("sparklyr") library(sparklyr) Key concepts sparklyr ignores some of the fundamental concepts that Spark has and that we discussed throughout this book. We posit that this is because these concepts are unfamiliar (and potentially irrelevant) to the typical R user. For instance, rather than a SparkSession, there’s simply spark_connect, which allows you to connect to a Spark cluster: sc <- spark_connect(master = "local") The returned variable is a remote dplyr data source. This connection, even though it resembles a SparkContext, is not the same SparkContext we mentioned in this book. This is a purely sparklyr concept that represents a Spark cluster connect. This function is largely the entire interface for how you will define configurations that you would like to use in your spark environment. Through this interface, you can specify initialization configurations for the spark cluster as a whole: spark_connect(master = "local", config = spark_config()) This works by using the config package in R to specify the configurations you would like to set on your Spark cluster. These details are covered in the sparklyr deployment documentation. Using this variable, we can manipulate remote Spark data from a local R process, thus the result of spark_connect performs roughly the same administrative role for end users as a SparkContext. No DataFrames sparklyr ignores the concept of a unique SparkDataFrame type. Instead it leverages tables (which are still mapped to DataFrames inside Spark) similar to other dplyr data sources and allows you to manipulate those. This aligns more with the typical R workflow, which is to use dplyr and magrittr to functionally define transformations from a source table. However, it means that some of Spark’s built-in functions and APIs may not be accessible unless dplyr also supports them. Data manipulation Once we connect to our cluster, we can run all the available dplyr functions and manipulations as if they were a local dplyr data.frame. This architectural choice gives those familiar with R the ability to do the same transformations using the same code, at scale. This means there’s no new syntax or concepts for R users to learn. While sparklyr does improve the R end-user experience, it comes at a cost of reducing the overall power available to sparklyr users, since the concepts are R concepts, not necessarily Spark concepts. For instance, sparklyr does not support user-defined functions that you can create and apply in SparkR using dapply, gapply, and lapply. As sparklyr continues to mature, it may add this sort of functionality, but at the time of this writing this capability does not exist. sparklyr is under very active development and more functionality is being added so refer to the sparklyr homepage for more information. Executing SQL While there is less direct Spark integration, users can execute arbitrary SQL code against the cluster using the DBI library corresponding to almost the same SQL interface we have seen in previous chapters: library(DBI) allTables <- dbGetQuery(sc, "SHOW TABLES") This SQL interface provides a convenient lower-level interface to the SparkSession. For instance, users can use DBI’s interface to set Spark SQL specific properties on the Spark cluster: setShufflePartitions <- dbGetQuery(sc, "SET spark.sql.shuffle.partitions=10") Unfortunately, neither DBI nor spark_connect does not give you an interface for setting Sparkspecific properties, which you are going to have to specify when you connect to your cluster. Data sources Users can leverage many of the same data sources available in Spark using sparklyr. For example, you should be able to create table statements using arbitrary data sources. However, only CSV, JSON, and Parquet formats are supported as first-class citizens using the following function definitions: spark_write_csv(tbl_name, location) spark_write_json(tbl_name, location) spark_write_parquet(tbl_name, location) Machine learning sparklyr also has support for some of the core machine learning algorithms that we saw in previous chapters. A list of the supported algorithms (at the time of this writing) includes: ml_kmeans: -means clustering ml_linear_regression: Linear regression ml_logistic_regression: Logistic regression ml_survival_regression: Survival regression ml_generalized_linear_regression: Generalized linear regression ml_decision_tree: Decision trees ml_random_forest: Random forests ml_gradient_boosted_trees: Gradient-boosted trees ml_pca: Principal components analysis ml_naive_bayes: Naive-Bayes ml_multilayer_perceptron: Multilayer perceptron ml_lda: Latent Dirichlet allocation ml_one_vs_rest: One versus rest (allowing you to make a binary classifier into a multiclass classifier) However, development does continue, so check MLlib for more information. Conclusion SparkR and sparklyr are areas of rapid growth in the Spark project, so visit their websites to find out the latest updates about each one. Moreover, the entire Spark project continues to grow as new members, tools, integrations, and packages join the community. The next chapter will discuss the Spark community and some of the other resources available to you. Chapter 33. Ecosystem and Community One of Spark’s biggest selling points is the sheer volume of resources, tools, and contributors. At the time of this writing, there are over 1,000 contributors to the Spark codebase. This is orders of magnitude more than most other projects dream of achieving and a testament to Spark’s amazing community—both in terms of contributors and stewards. The Spark project shows no sign of slowing down, as companies large and small seek to join the community. This environment has stimulated a large number of projects that complement and extend Spark’s features, including formal Spark packages and informal extensions that users can use in Spark. Spark Packages Spark has a package repository for packages specific to Spark: Spark Packages. These packages were discussed in Chapters 9 and 24. Spark packages are libraries for Spark applications that can easily be shared with the community. GraphFrames is a perfect example; it makes graph analysis available on Spark’s structured APIs in ways much easier to use than the lower-level (GraphX) API built into Spark. There are numerous other packages, including many machine learning and deep learning ones, that leverage Spark as the core and extend its functionality. Beyond these advanced analytics packages, others exist to solve problems in particular verticals. Healthcare and genomics have seen a surge in opportunity for big data applications. For example, the ADAM Project leverages unique, internal optimizations to Spark’s Catalyst engine to provide a scalable API & CLI for genome processing. Another package, Hail, is an open source, scalable framework for exploring and analyzing genomic data. Starting from sequencing or microarray data in VCF and other formats, Hail provides scalable algorithms to enable statistical analysis of gigabyte-scale data on a laptop or terabyte-scale data on cluster. At the time of this writing, there are nearly 400 different packages to choose. As a user, you can specify Spark packages as dependencies in your build files (as seen in this book’s book GitHub repository). You can also download the pre-built jars and include them in your class path without explicitly adding them to your build file. Spark packages can also be included at runtime by passing a parameter to the spark-shell or spark-submit command-line tools. An Abridged List of Popular Packages As mentioned, there are nearly 400 Spark packages. Including all of these is not relevant to you as a user because you can search for specific packages on the Spark package website. However, it is worth mentioning some of the more popular packages: Spark Cassandra Connector This connector helps you get data in and out of the Cassandra database. Spark Redshift Connector This connector helps you get data in and out of the Redshift database. Spark bigquery This connector helps you get data in and out of Google’s BigQuery. Spark Avro This package allows you to read and write Avro files. Elasticsearch This package allows you to get data into and out of Elasticsearch. Magellan Allows you to perform geo-spatial data analytics on top of Spark. GraphFrames Allows you to perform graph analysis with DataFrames. Spark Deep Learning Allows you to leverage Deep Learning and Spark together. Using Spark Packages There are two core ways you can include Spark Packages in your projects. In Scala or Java, you can include it as a build dependency, or you can also specify your packages at runtime (for Python or R). Let’s review the ways in which you can include this information. In Scala Including the following resolver in your build.sbt file will allow you to include Spark packages as dependencies. For example, we can add this resolver: // allows us to include spark packages resolvers += "bintray-spark-packages" at "https://dl.bintray.com/spark-packages/maven/" Now that we added this line, we can include a library dependency for our Spark package: libraryDependencies ++= Seq( ... // spark packages "graphframes" % "graphframes" % "0.4.0-spark2.1-s_2.11", ) This is to include the GraphFrames library. There are slight versioning differences between packages, but you can always find this information on the Spark packages website. In Python At the time of this writing , there is no explicit way to include a Spark package as a dependency in a Python package. These sorts of dependencies must be set at runtime. At runtime We saw how we can specify Spark packages in Scala packages, but we can also include these packages at runtime. This is as simple as including a new argument to the spark-shell and spark-submit that you would use to run your code. For example, to include the magellan library: $SPARK_HOME/bin/spark-shell --packages harsha2010:magellan:1.0.4-s_2.11 External Packages In addition to the formal Spark Packages, there are a number of informal packages that are built on or leverage Spark’s capabilities. A prime example is the popular gradient-boosted, decisiontree framework XGBoost, which makes use of Spark for scheduling distributed training on individual partitions. A number of these are liberally licensed, public projects available on GitHub. Using your favorite search engine is a great way to discover projects that may already exist, rather than having to write your own. Community Spark has a large, robust community. It is so much larger than the packages and direct contributions. The ecosystem of end users who build Spark into their products and write tutorials is an ever-growing group. As of this writing, there are over 1,000 contributors to the repository on Github. The official Spark website maintains the most up-to-date community information, including mailing lists, improvement proposals, and project committers. This website also includes many resources about new Spark versions, documentation, and release notes for the community. Spark Summit Spark Summits are events that occur across the globe at various times a year. This is the canonical event for Spark-related talks, where thousands of end users and developers attend these summits to learn about the cutting edge in Spark and hear about use cases. There are hundreds of tracks and training courses over the course of several days. In 2016, there were three events: New York (Spark Summit East), San Francisco (Spark Summit West), and Amsterdam (Spark Summit Europe). In 2017, there were Spark Summits in Boston, San Francisco, and Dublin. Coming in 2018—and beyond—there will be even more events. Find out more at at the Spark Summit website. There are hundreds of freely available Spark Summit videos for learning about use cases, Spark’s development, and strategies and tactics that you can use to get the most out of Spark. You can browse historical Spark Summit talks and videos on the website. Local Meetups There are many Spark-related meetup groups on meetup.com. Figure 33-1 shows a map of Spark-related meetups on Meetup.com. Figure 33-1. Spark meetup map Spark’s “official meetup group” in the Bay Area (founded by one of the authors of this book), can be found here. However, there are over 600 Spark-related meetups around the world, totaling nearly 350,000 members. These meetups continue to spring up and grow, so be sure to find one in your area. Conclusion This whirlwind chapter discussed nontechnical resources that Spark makes available. One important fact is that one of Spark’s greatest assets is the Spark community. We are extremely proud of the community’s involvement in the development of Spark and love to hear about what companies, academic institutions, and individuals build with Spark. We sincerely hope that you’ve enjoyed this book and we look forward to seeing you at a Spark Summit! Index Symbols --jars command-line argument, Submitting applications ./bin/pyspark, Launching the Python console, Starting Spark ./bin/spark-shell, Launching the Scala console, Starting Spark =!= operator, Concatenating and Appending Rows (Union), Working with Booleans == (equal to) expression, Working with Booleans ` (backtick) character, Reserved Characters and Keywords A Accelerated Failure Time (AFT), Survival Regression (Accelerated Failure Time) accumulators basic example, Basic Example-Basic Example custom, Custom Accumulators overview of, Distributed Shared Variables, Accumulators acknowledgements, Acknowledgments actions, Actions ADAM Project, Spark Packages advanced analytics (see machine learning and advanced analytics) aggregate function, aggregate AggregateByKey function, aggregateByKey AggregationBuffer, User-Defined Aggregation Functions aggregations aggregate function, aggregate AggregateByKey function, aggregateByKey aggregation functions, Aggregation Functions-Aggregating to Complex Types CombineByKey function, combineByKey countByKey, countByKey debugging, Slow Aggregations foldByKey function, foldByKey groupByKey, groupByKey grouping, Grouping-Grouping with Maps grouping sets, Grouping Sets-Pivot on complex types, Aggregating to Complex Types overview of, Aggregations-Aggregations performance tuning, Aggregations on RDDs, Aggregations-foldByKey reduceByKey, reduceByKey in Structured Streaming API, Aggregations User-Defined Aggregation Functions (UDAFs), User-Defined Aggregation Functions window functions, Window Functions-Grouping Sets alerting and notifications, Notifications and alerting, Alerting Alternating Least Squares (ALS), Use Cases-Ranking Metrics analytics (see machine learning and advanced analytics) analyzer phase in Spark SQL, Logical Planning anomaly detection through graph analysis, Graph Analytics through unsupervised learning, Unsupervised Learning anti joins, Left Anti Joins Apache Hadoop, Apache Spark’s Philosophy, Splittable File Types and Compression, Hadoop Files Apache Hive, Big Data and SQL: Apache Hive, The SparkContext, Miscellaneous Considerations Apache Maven, A Simple Scala-Based App Apache Mesos, The Architecture of a Spark Application, Deploying Spark, Spark on Mesos Apache Spark (see also Spark applications; Spark SQL) actions, Actions API selection, Which Spark API to Use? architecture of, Spark’s Basic Architecture benefits of, What Is Apache Spark?, Context: The Big Data Problem, Conclusion, Ecosystem and Community building Spark from source, Building Spark from source case sensitivity, Case Sensitivity cloud deployment of, Spark in the Cloud (see also deployment) DataFrames, DataFrames downloading, Downloading Spark Locally ecosystem of packages and tools, Spark’s Ecosystem and Packages, Spark Packages-External Packages focus on computation, Apache Spark’s Philosophy functional programming model underlying, History of Spark, An End-to-End Example fundamental APIs of, Spark’s APIs history of, Preface, History of Spark interactive nature of, History of Spark internal type representations, Spark Types-Spark Types language APIs, Spark’s Language APIs launching interactive consoles, Launching Spark’s Interactive Consoles libraries supported by, Apache Spark’s Philosophy managing Spark versions, Miscellaneous Considerations, Updating Your Spark Version philosophy of, Apache Spark’s Philosophy recent improvements to, The Present and Future of Spark reserved characters and keywords, Reserved Characters and Keywords running, Running Spark, Running Spark in the Cloud, How Spark Runs on a ClusterConclusion Spark UI, Spark UI starting, Starting Spark toolkit components and libraries, What Is Apache Spark?, A Tour of Spark’s Toolset-Spark’s Ecosystem and Packages topics covered, Preface transformation end-to-end example, An End-to-End Example-DataFrames and SQL transformations basics, Transformations unified nature of, Apache Spark’s Philosophy append output mode, Append mode application properties, Application Properties, Application properties for YARN applications (Structured Streaming) (see also production applications; Spark applications; Structured Streaming API) alerting, Alerting monitoring, Metrics and Monitoring-Spark UI sizing and rescaling, Sizing and Rescaling Your Application Stream Listener monitoring, Advanced Monitoring with the Streaming Listener updating, Updating Your Application approximations, Aggregations approxQuantile method, Working with Numbers approx_count_distinct function, approx_count_distinct arrays, Arrays-explode array_contains, array_contains asynchronous job execution, Time-Outs atomicity, Resilience in output and atomicity attributions, Using Code Examples automatic model tuning, Evaluators for Classification and Automating Model Tuning, Evaluators and Automating Model Tuning average, calculating, avg Avro, An Abridged List of Popular Packages B backtick (`) character, Reserved Characters and Keywords batch processing, What Is Stream Processing?, Batch duration Bay Area Bike Share data, Graph Analytics BigDL, BigDL bigquery package, An Abridged List of Popular Packages binary classification, Classification, Binary Classification binning, Bucketing (see also bucketing) bisecting -means, Bisecting k-means Summary Booleans, Working with Booleans breadth-first search, Breadth-First Search broadcast joins, Communication Strategies broadcast variables, Distributed Shared Variables-Broadcast Variables, Broadcast Variables bucketing, Bucketing, Bucketing, Bucketing-Advanced bucketing techniques business logic, Business logic resilience and evolution ByKey, Key-Value Basics (Key-Value RDDs) C caching, Temporary Data Storage (Caching)-Temporary Data Storage (Caching) calendar dates, Working with Dates and Timestamps-Working with Dates and Timestamps capitalization, Working with Strings Cartesian products, Cross (Cartesian) Joins case classes, In Scala: Case Classes, Datasets and RDDs of Case Classes case sensitivity, Case Sensitivity case…when…then…end style statements, case…when…then Statements Cassandra Connector, Conclusion, On-Premises Cluster Deployments, An Abridged List of Popular Packages casting, Changing a Column’s Type (cast) catalog, Logical Planning, Miscellaneous Considerations Catalog (Spark SQL), Catalog, Miscellaneous Considerations Catalyst computation engine, Overview of Structured Spark Types, Spark Packages categorical features, preprocessing, Working with Categorical Features-Text Data Transformers centroid, Machine Learning and Advanced Analytics, k-means (see also -means) checkpointing, Checkpointing, Fault Tolerance and Checkpointing, Connected Components Chi-Square Selector, Feature Selection classification decision trees, Decision Trees-Prediction Parameters evaluators and automating model tuning, Evaluators for Classification and Automating Model Tuning logistic regression, Logistic Regression-Model Summary metrics, Detailed Evaluation Metrics models in MLlib, Classification Models in MLlib-Model Scalability multilayer perceptron classifier, MLlib Neural Network Support Naive Bayes, Naive Bayes One-vs-Rest, One-vs-Rest Classifier random forests and gradient boosted trees, Random Forest and Gradient-Boosted TreesPrediction Parameters through graph analysis, Graph Analytics types of, Types of Classification use cases for, Classification, Use Cases client mode, Client mode client requests, Client Request cloud deployment, Spark in the Cloud (see also deployment) cluster managers (see also clusters) cluster managers, Spark on YARN-Application properties for YARN Mesos, Spark on Mesos overview of, The Architecture of a Spark Application purpose of, Spark’s Basic Architecture selecting, Deploying Spark standalone, Cluster Managers-Submitting applications cluster mode, Cluster mode clusters (see also cluster managers) cluster networking configurations, Cluster Networking Configurations creating, Launch defined, Spark’s Basic Architecture monitoring, The Monitoring Landscape on-premises clusters, On-Premises Cluster Deployments performance tuning, Cluster Configurations sizing and sharing configuration, Cluster/application sizing and sharing coalesce, Repartition and Coalesce, Coalesce, coalesce, Repartitioning and Coalescing code examples, obtaining and using, Using Code Examples, Data Used in This Book CoGroups, CoGroups col function, Columns cold start problem, Collaborative Filtering with Alternating Least Squares, Prediction Parameters collaborative filtering, Use Cases-Ranking Metrics collect method, Datasets: Type-Safe Structured APIs, Collecting Rows to the Driver, Actions collect_list function, Lists collect_set function, Lists column function, Columns columns accessing, Accessing a DataFrame’s columns adding, Adding Columns case sensitivity, Case Sensitivity changing type (cast), Changing a Column’s Type (cast) converting rows to, Pivot exploding, explode instantiating, Spark Types locating methods, Where to Look for APIs manipulating with Select and SelectExpr, select and selectExpr MLlib column metadata, Transformer Properties overview of, Columns removing, Removing Columns renaming, Renaming Columns reserved characters and keywords, Reserved Characters and Keywords working with in Spark, Columns and Expressions CombineByKey function, combineByKey comments and questions, How to Contact Us common words, removing, Removing Common Words community, Spark’s Ecosystem and Packages comparison (=!=) operator, Concatenating and Appending Rows (Union), Working with Booleans complete output mode, Complete mode complex types, Columns, Writing Complex Types, Complex Types-Lists compression formats, Splittable File Types and Compression, Splittable file types and compression computing engines, What Is Apache Spark?, Apache Spark’s Philosophy concatenation, Concatenating and Appending Rows (Union) conf/slaves file, Cluster launch scripts configuration options application properties, Application Properties environmental variables, Environmental Variables execution properties, Execution Properties job scheduling, Job Scheduling Within an Application memory management, Configuring Memory Management overview of, Configuring Applications runtime properties, Runtime Properties shuffle behavior, Configuring Shuffle Behavior SparkConf, The SparkConf connected components algorithm, Connected Components console sink, Sources and sinks for testing console, launching interactive, Launching Spark’s Interactive Consoles continuous applications, What Is Stream Processing?, Structured Streaming Basics continuous features, preprocessing of, Working with Continuous Features-Normalizer continuous processing-based systems, Continuous Versus Micro-Batch Execution correlated predicated subqueries, Correlated predicate subqueries correlated subqueries, Subqueries correlation, computing, Working with Numbers, Covariance and Correlation cost-based optimizations, Statistics collection (see also performance tuning) count action, Aggregations, Actions, count count-based windows, mapGroupsWithState-mapGroupsWithState countApprox, countApprox countApproxDistinct, countApproxDistinct countByKey, Aggregations countByValue, countByValue countByValueApprox, countByValueApprox countDistinct function, countDistinct counting, Working with Numbers CountVectorizer, Converting Words into Numerical Representations covariance, calculating, Covariance and Correlation CREATE EXTERNAL TABLE statement, Creating External Tables cross (Cartesian) joins, Cross (Cartesian) Joins CSV (comma-separated values) files CSV reader options, CSV Files-Reading CSV Files reading, Reading CSV Files writing, Writing CSV Files cube operator, Cube curse of dimensionality, Unsupervised Learning D data locality, Data locality data sources community-created, Data Sources CSV files, CSV Files-Writing CSV Files data source APIs structure, The Structure of the Data Sources API-Save modes downloading data used in this book, Data Used in This Book JSON files, JSON Files-Parquet Files managing file size, Managing File Size ORC files, ORC Files Parquet files, Parquet Files-Writing Parquet Files reading data in parallel, Reading Data in Parallel Spark's core, Data Sources splittable file formats, Splittable File Types and Compression SQL databases, SQL Databases-Text Files in Structured Streaming API, Input Sources, Where Data Is Read and Written (Sources and Sinks)-Reading from the Kafka Source, Sources and sinks for testing text files, Text Files writing complex types, Writing Complex Types writing data in parallel, Writing Data in Parallel data, cleaning, Data cleaning data, reading basics of, Basics of Reading Data core API structure, Read API Structure CSV files, Reading CSV Files debugging, Slow Reads and Writes in parallel, Reading Data in Parallel read mode, Read modes data, storing bucketing, Bucketing collecting statistics, Statistics collection data locality, Data locality file-based long-term, File-based long-term data storage number of files, The number of files splittable file types and compression, Splittable file types and compression table partitioning, Table partitioning temporary (caching), Temporary Data Storage (Caching)-Temporary Data Storage (Caching) data, types of arrays, Arrays-explode Booleans, Working with Booleans converting to Spark types, Converting to Spark Types dates and timestamps, Working with Dates and Timestamps-Working with Dates and Timestamps JSON data, Working with JSON locating transformations, Where to Look for APIs maps, Maps null values, Working with Nulls in Data-Ordering, Signs and symptoms, Formatting Models According to Your Use Case numbers, Working with Numbers-Working with Numbers ordering null values, Ordering strings, Working with Strings-Regular Expressions structs, Structs user-defined functions (UDFs), User-Defined Functions-User-Defined Functions, Aggregating to Complex Types, User-defined functions data, updating in real time, Update data to serve in real time data, writing basics of, Basics of Writing Data core API structure, Write API Structure debugging, Slow Reads and Writes in parallel, Writing Data in Parallel save mode, Save modes databases (Spark SQL) (see also SQL databases) creating, Creating Databases dropping, Dropping Databases OLTP vs. OLAP, Big Data and SQL: Spark SQL overview of, Databases setting, Setting the Database Databricks Community Edition, Running Spark, Running Spark in the Cloud, The Development Process, Spark in the Cloud DataFrameNaFunctions submodule, Where to Look for APIs DataFrameReader, An End-to-End Example, Basics of Reading Data DataFrames basic structured operations, Basic Structured Operations-Conclusion basics of, DataFrames, DataFrames and Datasets components of, Basic Structured Operations creating, Creating DataFrames creating from RDDs, Interoperating Between DataFrames, Datasets, and RDDs vs. Datasets, DataFrames Versus Datasets, DataFrames versus SQL versus Datasets versus RDDs locating methods and functions, Where to Look for APIs manipulating, Creating DataFrames-Conclusion streaming, Structured Streaming in Action DataFrameStatFunctions submodule, Where to Look for APIs DataFrameWriter, Basics of Writing Data Datasets actions, Actions benefits of, Datasets: Type-Safe Structured APIs, When to Use Datasets creating, Creating Datasets creating from RDDs, Interoperating Between DataFrames, Datasets, and RDDs vs. DataFrames, DataFrames Versus Datasets filtering, Filtering grouping and aggregations, Grouping and Aggregations joins, Joins locating methods, Where to Look for APIs mapping, Mapping overview of, Datasets: Type-Safe Structured APIs, DataFrames and Datasets, Datasets streaming, Streaming Dataset API transformations, Transformations when to use, When to Use Datasets, Which Spark API to Use?, DataFrames versus SQL versus Datasets versus RDDs dates and timestamps, Working with Dates and Timestamps-Working with Dates and Timestamps de-duplication, Dropping Duplicates in a Stream debugging (see monitoring and debugging) decision making, real-time, Real-time decision making decision trees applied to classification, Decision Trees-Prediction Parameters applied to regression, Decision Trees example, Prediction Parameters model hyperparameters, Model Hyperparameters overview of, Decision Trees prediction parameters, Prediction Parameters training parameters, Training Parameters declarative APIs, Record-at-a-Time Versus Declarative APIs deep learning BigDL, BigDL Deep Learning Pipelines, Deep Learning Pipelines-Conclusion DeepLearning4j, DeepLearning4J MLlib neural network support, MLlib Neural Network Support overview of, What Is Deep Learning? in Spark, Ways of Using Deep Learning in Spark-Ways of Using Deep Learning in Spark TensorFlowOnSpark, TensorFlowOnSpark TensorFrames, TensorFrames Deep Learning (Goodfellow), A Short Primer on Advanced Analytics Deep Learning Pipelines, Deep Learning Pipelines-Conclusion deep neural networks, What Is Deep Learning? DeepLearning4j, DeepLearning4J default stop words, Removing Common Words dependencies, Transformations deployment application scheduling, Application Scheduling cluster networking configurations, Cluster Networking Configurations external shuffle service, Miscellaneous Considerations logging considerations, Miscellaneous Considerations, The Monitoring Landscape managing Spark versions, Miscellaneous Considerations Mesos, Spark on Mesos metastores, Miscellaneous Considerations monitoring, Miscellaneous Considerations number and type of applications, Miscellaneous Considerations overview of, Deploying Spark secure deployment configurations, Secure Deployment Configurations standalone cluster manager, Cluster Managers-Submitting applications where to deploy, Where to Deploy Your Cluster to Run Spark Applications-Spark in the Cloud YARN framework, Spark on YARN-Application properties for YARN describe method, Working with Numbers development process, The Development Process development template, Developing Spark Applications, Spark Logs (see also Spark applications) dimensionality, curse of, Unsupervised Learning directed acyclic graph (DAG), DataFrames and SQL, Columns as expressions, The Spark UI directed graphs, Graph Analytics, Building a Graph disks, no space left errors, Signs and symptoms distinct method, distinct distributed collections, The SparkSession distributed shared variables accumulators, Accumulators-Custom Accumulators broadcast variables, Broadcast Variables-Broadcast Variables distributed stream processing, Core Concepts DLB (see Deep Learning (Goodfellow)) dplyr data sources, sparklyr driver processes, Spark Applications, Collecting Rows to the Driver, The Architecture of a Spark Application, Client Request, Driver and Executor Processes, Driver OutOfMemoryError or Driver Unresponsive, Language Specifics: Python (PySpark) and R (SparkR and sparklyr) drop function, drop Dropwizard Metrics Library, Driver and Executor Processes dstat utility, The Monitoring Landscape DStreams API, Stream Processing Fundamentals, The DStream API, Event Time, Handling Late Data with Watermarks duplicates, removing, Dropping Duplicates in a Stream dynamic allocation, Application Scheduling, Dynamic allocation E edge nodes, Client mode edges, Graph Analytics ElasticNet model, Linear Regression Elasticsearch, An Abridged List of Popular Packages Elements of Statistical Learning (Hastie), A Short Primer on Advanced Analytics ElementwiseProduct, ElementwiseProduct empty data, Working with Nulls in Data end-to-end applications, What Is Stream Processing?, Structured Streaming, Structured Streaming Basics environmental variables, Environmental Variables eponymous functions, skewness and kurtosis equal to (==) expression, Working with Booleans errors (see also monitoring and debugging) before execution, Errors Before Execution during execution, Errors During Execution no space left on disk errors, Signs and symptoms OutOfMemoryError, Driver OutOfMemoryError or Driver Unresponsive-Potential treatments serialization errors, Signs and symptoms ESL (see Elements of Statistical Learning (Hastie)) estimators, High-Level MLlib Concepts, Estimators, Estimators for Preprocessing evaluators, High-Level MLlib Concepts, Training and Evaluation-Training and Evaluation, Evaluators for Classification and Automating Model Tuning, Evaluators and Automating Model Tuning, Evaluators for Recommendation event logs, Spark UI History Server event time, Event Time Versus Processing Time event-time processing basics of, Event-Time Basics benefits of Spark for, Event-Time and Stateful Processing defined, Event-Time and Stateful Processing dropping duplicates, Dropping Duplicates in a Stream event time defined, Event Time example, Event Time key ideas, Event-Time Processing late data handling, Handling Late Data with Watermarks-Handling Late Data with Watermarks processing time defined, Event Time windows, Windows on Event Time-Handling Late Data with Watermarks execution modes, Execution Modes execution plan, DataFrames and SQL, Logical Instructions execution properties, Execution Properties execution, of Spark applications, Execution executor processes, Spark Applications, The Architecture of a Spark Application, Driver and Executor Processes, Executor OutOfMemoryError or Executor Unresponsive explain plan, An End-to-End Example explode function, explode exploratory data analysis (EDA), Data cleaning expr function, Columns as expressions expressions accessing DataFrame columns, Accessing a DataFrame’s columns building, Working with Different Types of Data-Conclusion (see also data, types of) columns as expressions, Columns as expressions defined, Columns and Expressions, Expressions grouping with, Grouping with Expressions joins, Join Expressions-Conclusion external packages, External Packages external shuffle service, Miscellaneous Considerations, Shuffle Configurations external tables, Creating External Tables extract, transform, and load (ETL), Schemas, When to Use Datasets, Scala versus Java versus Python versus R, Incremental ETL, Ways of Using Deep Learning in Spark F failure recovery, Fault Tolerance and Checkpointing fair scheduler, Application Scheduling, Scheduling fault tolerance, Fault Tolerance and Checkpointing feature engineering, Feature engineering feature generation, Feature Manipulation-Polynomial Expansion feature selection, Feature Selection featurization, Ways of Using Deep Learning in Spark file size, managing, Managing File Size fill function, fill filter method, Filtering Rows, Working with Booleans, Improved Filtering, Selections and Filtering first function, first and last first method, first fit method, Feature Engineering with Transformers flatMap, flatMap, Mapping over Values flatMapGroupsWithState, When can you use each mode?, flatMapGroupsWithStateflatMapGroupsWithState foldByKey function, foldByKey foreach sink, Foreach sink foreachPartition, foreachPartition fraud prediction, Graph Analytics frequent item pairs, Working with Numbers frequent pattern mining, Frequent Pattern Mining function (Spark SQL), Functions G gamma regression, Generalized Linear Regression Ganglia, The Monitoring Landscape garbage collection, Memory Pressure and Garbage Collection-Garbage collection tuning gateway machines, Client mode Gaussian (linear regression), Generalized Linear Regression Gaussian mixture models (GMM), Gaussian Mixture Models generalized linear regression example, Example model hyperparameters, Model Hyperparameters overview of, Generalized Linear Regression prediction parameters, Prediction Parameters training parameters, Training Parameters training summary, Training Summary geo-spatial data analytics, An Abridged List of Popular Packages ggplot library, SparkR glom function, glom gradient boosted trees (GBT) applied to classification, Random Forest and Gradient-Boosted Trees-Prediction Parameters applied to regression, Random Forests and Gradient-Boosted Trees example, Prediction Parameters model hyperparameters, Model Hyperparameters overview of, Random Forest and Gradient-Boosted Trees prediction parameters, Prediction Parameters training parameters, Training Parameters graph analysis breadth-first search, Breadth-First Search building graphs, Building a Graph connected components, Connected Components GraphFrames algorithms, Graph Algorithms in-degree and out-degree metrics, In-Degree and Out-Degree Metrics-In-Degree and OutDegree Metrics motif finding, Motif Finding-Motif Finding overview of, Graph Analytics-Graph Analytics PageRank algorithm, PageRank querying graphs, Querying the Graph strongly connected components, Strongly Connected Components subgraphs, Subgraphs use cases for, Graph Analytics graph databases, Graph Analytics GraphFrames, Graph Analytics-Conclusion, Graph Algorithms, Spark Packages GraphX, Graph Analytics group-by function, DataFrames and SQL, Window Functions groupByKey, groupByKey grouping, Grouping-Grouping with Maps grouping sets, Grouping Sets-Pivot grouping_id operator, Grouping Metadata gzip compression, Splittable File Types and Compression H Hadoop Distributed File System (HDFS), Apache Spark’s Philosophy, Splittable File Types and Compression, Hadoop Files, On-Premises Cluster Deployments, Hadoop configurations Hadoop YARN, The Architecture of a Spark Application, Deploying Spark, Spark on YARNApplication properties for YARN Hail, Spark Packages Heterogeneity Human Activity Recognition Dataset, Structured Streaming in Action History Server, Spark UI History Server Hive, Big Data and SQL: Apache Hive Hive metastore, The Hive metastore, Miscellaneous Considerations HiveContext, The SparkContext HiveQL, Creating External Tables hyperparameters, Model tuning and evaluation, Estimators I ifnull function, ifnull, nullIf, nvl, and nvl2 image models, Applying Popular Models immutability, Transformations incremental data updates, Update data to serve in real time incremental ETL, Incremental ETL IndexToString, Converting Indexed Values Back to Text inference, Ways of Using Deep Learning in Spark informal packages, External Packages initcap function, Working with Strings inner joins, Inner Joins input data resilience, Input data resilience input/output (Structured Streaming API) file source and sink, File source and sink foreach sink, Foreach sink-Foreach sink input rate monitoring, Input rate and processing rate Kafka source and sink, Kafka source and sink output modes, How Data Is Output (Output Modes) reading from Kafka source, Reading from the Kafka Source sources and sinks for testing, Sources and sinks for testing triggers, When Data Is Output (Triggers) writing to Kafka sink, Writing to the Kafka Sink Interaction feature transformer, Interaction interactive consoles, launching, Launching Spark’s Interactive Consoles iostat utility, The Monitoring Landscape iotop utility, The Monitoring Landscape isotonic regression, Isotonic Regression J Java Encoders, In Java: Encoders SimpleDateFormat, Working with Dates and Timestamps TimeZone format, Working with Dates and Timestamps type reference, Spark Types writing Spark applications in, Writing Java Applications Java Database Connectivity (JDBC), SQL Databases, SparkSQL Thrift JDBC/ODBC Server Java Virtual Machine (JVM), The Hive metastore, Datasets, The Monitoring Landscape jconsole utility, The Monitoring Landscape jmap utility, The Monitoring Landscape joins broadcast join, Communication Strategies challenges when using, Challenges When Using Joins-Approach 3: Renaming a column before the join cross (Cartesian) joins, Cross (Cartesian) Joins debugging, Slow Joins how Spark performs joins, How Spark Performs Joins-Little table–to–little table inner joins, Inner Joins, Inner Join left anti joins, Left Anti Joins left outer joins, Left Outer Joins left semi joins, Left Semi Joins natural joins, Natural Joins outer joins, Outer Joins overview of, Join Expressions performance tuning, Joins in RDDs, Joins right outer joins, Right Outer Joins shuffle join, Communication Strategies in Structured Streaming API, Joins types available, Join Types zips, zips joinWith method, Joins JSON data line-delimited JSON files, JSON Files options available, JSON Options reading JSON files, Reading JSON Files working with, Working with JSON writing JSON files, Writing JSON Files jstack utility, The Monitoring Landscape jstat utility, The Monitoring Landscape JUnit, Managing SparkSessions jvisualvm utility, The Monitoring Landscape K -means algorithm, Machine Learning and Advanced Analytics example, Training Parameters model hyperparameters, Model Hyperparameters overview of, k-means summary class, k-means Metrics Summary training parameters, Training Parameters Kafka overview of, Kafka source and sink reading from, Reading from the Kafka Source writing to, Writing to the Kafka Sink Keras, Applying Popular Models Key–Value RDDs aggregate function, aggregate AggregateByKey function, aggregateByKey aggregations, Aggregations-foldByKey CombineByKey function, combineByKey creating, keyBy extracting keys and values, Extracting Keys and Values foldByKey function, foldByKey groupByKey, groupByKey mapping over values, Mapping over Values reduceByKey, reduceByKey when to use, Key-Value Basics (Key-Value RDDs), DataFrames versus SQL versus Datasets versus RDDs Kryo serialization, Custom Serialization Kryo Serialization, Object Serialization in RDDs kurtosis, calculating, skewness and kurtosis L L-BFGS (Limited memory Broyden-Fletcher-Goldfarb-Shanno), Model Scalability language APIs Java, Spark’s Language APIs overview of, Spark’s Language APIs Python, Spark’s Language APIs, PySpark R, Spark’s Language APIs, R on Spark-Machine learning Scala, Spark’s Language APIs selecting, Which Spark API to Use?, Scala versus Java versus Python versus R SQL, Spark’s Language APIs last function, first and last late data, handling, Handling Late Data with Watermarks-Handling Late Data with Watermarks Latent Dirichlet Allocation (LDA) example, Training Parameters model hyperparameters, Model Hyperparameters overview of, Latent Dirichlet Allocation prediction parameters, Prediction Parameters training parameters, Training Parameters lazy evaluation, Lazy Evaluation left anti joins, Left Anti Joins left outer joins, Left Outer Joins left semi joins, Left Semi Joins libraries, supported by Spark, Apache Spark’s Philosophy LIBSVM data format, MLlib in Action limit method, Limit line-delimited JSON files, JSON Files linear regression, Linear Regression lists, Lists lit function, Converting to Spark Types literals, Converting to Spark Types (Literals) local mode, Spark Applications, Local mode logging, Miscellaneous Considerations, The Monitoring Landscape, Spark Logs logical plan, An End-to-End Example, Logical Planning, Logical Instructions logistic regression example, Example model hyperparameters, Model Hyperparameters model summary, Model Summary overview of, Logistic Regression prediction parameters, Prediction Parameters training parameters, Training Parameters lookup function, lookup lower-level APIs defined, What Are the Low-Level APIs? distributed shared variables, Distributed Shared Variables-Conclusion how to use, How to Use the Low-Level APIs? overview of, Lower-Level APIs RDD advanced applications, Advanced RDDs-Conclusion RDD basics, About RDDs-Conclusion when to use, When to Use the Low-Level APIs?, Which Spark API to Use?, DataFrames versus SQL versus Datasets versus RDDs lowercase, Working with Strings M machine learning and advanced analytics advanced analytics process, The Advanced Analytics Process-Leveraging the model and/or insights classification, Classification-Conclusion data cleaning, Data cleaning data collection, Data collection deep learning, Deep Learning-Conclusion deployment patterns, Deployment Patterns feature engineering, Feature engineering, Feature Manipulation-ChiSqSelector graph analysis, Graph Analytics, Graph Analytics-Conclusion MLlib, Spark’s Advanced Analytics Toolkit-Persisting and Applying Models model training, Feature engineering model tuning and evaluation, Model tuning and evaluation online machine learning, Online machine learning overview of, Machine Learning and Advanced Analytics-Machine Learning and Advanced Analytics, Advanced Analytics and Machine Learning Overview-A Short Primer on Advanced Analytics persisting and applying models, Persisting and Applying Models Pipeline concept, Pipelining Our Workflow preprocessing, Preprocessing and Feature Engineering-Word2Vec recommendation, Recommendation, Recommendation-Conclusion regression, Regression-Conclusion supervised learning, Supervised Learning-Regression unsupervised learning, Unsupervised Learning, Unsupervised Learning-Conclusion Magellan, An Abridged List of Popular Packages magrittr library, SparkR main() function, Spark Applications mapGroupsWithState, When can you use each mode?, mapGroupsWithStatemapGroupsWithState mapPartitions, mapPartitions MapPartitionsRDD, mapPartitions maps, Maps, Grouping with Maps market basket analysis, Frequent Pattern Mining Maven, A Simple Scala-Based App max function, min and max, max and min MaxAbsScaler, MaxAbsScaler maxFilesPerTrigger, Structured Streaming in Action, File source and sink meetup groups, Local Meetups memory management configuring, Configuring Memory Management garbage collection, Memory Pressure and Garbage Collection-Garbage collection tuning OutOfMemoryError, Driver OutOfMemoryError or Driver Unresponsive-Potential treatments temporary data storage (caching), Temporary Data Storage (Caching)-Temporary Data Storage (Caching) memory sinks, Structured Streaming in Action, Sources and sinks for testing Mesos (see Apache Mesos) metadata describing, Describing Table Metadata grouping, Grouping Metadata Hive metastore, Miscellaneous Considerations MLlib column metadata, Transformer Properties refreshing, Refreshing Table Metadata tables (Spark SQL), Spark-Managed Tables metastores, Miscellaneous Considerations metrics, Driver and Executor Processes, Metrics and Monitoring-Spark UI, Detailed Evaluation Metrics, Metrics, Metrics, In-Degree and Out-Degree Metrics-In-Degree and Out-Degree Metrics (see also monitoring and debugging) micro-batch systems, Continuous Versus Micro-Batch Execution min function, min and max, max and min minDF, Converting Words into Numerical Representations minimum term frequency (minTF), Converting Words into Numerical Representations MinMaxScaler, MinMaxScaler missing data, Working with Nulls in Data MLlib benefits of, Machine Learning and Advanced Analytics, When and why should you use MLlib (versus scikit-learn, TensorFlow, or foo package) classification models in, Classification Models in MLlib-Model Scalability estimators, High-Level MLlib Concepts evaluators, High-Level MLlib Concepts low-level data types, Low-level data types neural network support, MLlib Neural Network Support overview of, Machine Learning and Advanced Analytics-Machine Learning and Advanced Analytics, What Is MLlib? packages included in, What Is MLlib? persisting and applying models, Persisting and Applying Models pipeline example, MLlib in Action-Persisting and Applying Models regression models in, Regression Models in MLlib-Conclusion transformers, High-Level MLlib Concepts models (see also individual models; machine learning and advanced analytics; MLlib) automatic model tuning, Evaluators for Classification and Automating Model Tuning, Evaluators and Automating Model Tuning classification models, Classification Models in MLlib-Conclusion deep learning models, Deep Learning-Conclusion formatting according to use case, Formatting Models According to Your Use Case-Formatting Models According to Your Use Case image models, Applying Popular Models regression models, Regression-Conclusion scalability of, Model Scalability, Model Scalability, Collaborative Filtering with Alternating Least Squares, Model Scalability training, Feature engineering training deep learning models, Ways of Using Deep Learning in Spark tuning and evaluation, Model tuning and evaluation monitoring and debugging column metadata, Transformer Properties components to monitor, The Monitoring Landscape-The Monitoring Landscape deployment decisions, Miscellaneous Considerations driver issues, Driver OutOfMemoryError or Driver Unresponsive errors before execution, Errors Before Execution errors during execution, Errors During Execution executor issues, Executor OutOfMemoryError or Executor Unresponsive no space left on disk errors, Signs and symptoms processes to monitor, What to Monitor role in performance tuning, Performance Tuning serialization errors, Signs and symptoms slow aggregation, Slow Aggregations slow joins, Slow Joins slow reads and writes, Slow Reads and Writes slow tasks or stragglers, Slow Tasks or Stragglers Spark jobs not starting, Spark Jobs Not Starting Spark logs, Spark Logs Spark UI, The Spark UI-Spark UI History Server Structured Streaming API, Metrics and Monitoring-Spark UI unexpected nulls in results, Signs and symptoms, Formatting Models According to Your Use Case monotonically_increasing_id function, Working with Numbers motif finding algorithms, Motif Finding-Motif Finding multiclass classification, Multiclass Classification multilabel classification, Multilabel Classification multilayer perceptron classifier, MLlib Neural Network Support multinomial models, Naive Bayes multivariate Bernoulli models, Naive Bayes N n-grams, Creating Word Combinations Naive Bayes classifiers, Naive Bayes narrow dependencies, Transformations natural joins, Natural Joins Netflix, Use Cases node-to-node communication strategy, How Spark Performs Joins nodes, Graph Analytics normalization and scaling, Scaling and Normalization-Normalizer notifications and alerting, Notifications and alerting null values, Columns, Working with Nulls in Data-Ordering, Signs and symptoms, Formatting Models According to Your Use Case nullIf function, ifnull, nullIf, nvl, and nvl2 numbers, Working with Numbers-Working with Numbers nvl function, ifnull, nullIf, nvl, and nvl2 nvl2 function, ifnull, nullIf, nvl, and nvl2 O object serialization, Custom Serialization, Signs and symptoms, Object Serialization in RDDs official meetup group, Local Meetups on-premises clusters, On-Premises Cluster Deployments once trigger, Once trigger One-vs-Rest, One-vs-Rest Classifier OneHotEncoder, Machine Learning and Advanced Analytics, One-Hot Encoding online analytic processing (OLAP), Big Data and SQL: Spark SQL online machine learning, Online machine learning online transaction processing (OLTP), Big Data and SQL: Spark SQL optimization, cost-based, Statistics collection (see also performance tuning) ORC files, ORC Files org.apache.spark.sql.functions package, Creating DataFrames, Where to Look for APIs, Aggregation Functions out-degree metric, In-Degree and Out-Degree Metrics-In-Degree and Out-Degree Metrics outer joins, Outer Joins OutOfMemoryError drivers, Driver OutOfMemoryError or Driver Unresponsive executors, Executor OutOfMemoryError or Executor Unresponsive output modes, Output Modes, How Data Is Output (Output Modes), Output Modes output schema resolution, Resilience in output and atomicity output sinks, Sinks (see also sinks) P packages, Spark’s Ecosystem and Packages, Spark Packages-External Packages PageRank algorithm, Graph Analytics, PageRank Pandas, Pandas Integration parallelism, Parallelism parallelize method, From a Local Collection ParamGrid, Training and Evaluation Parquet files benefits of, Parquet Files options available, Parquet options reading, Reading Parquet Files writing, Writing Parquet Files Partitioner, Types of RDDs, Custom Partitioning partitions based on sliding windows, Partitioning based on a sliding window controlling with RDDs, Controlling Partitions-Custom Partitioning custom partitioning, Custom Partitioning-Custom Partitioning defined, Partitions partitioning schemes, Basic Structured Operations performance tuning, Table partitioning, Repartitioning and Coalescing purpose of, Partitioning repartitioning, Repartition and Coalesce role in application lifecycle, Stages Pearson Correlation Coefficient, Working with Numbers, Covariance and Correlation per node computation strategy, How Spark Performs Joins performance tuning aggregations, Aggregations automatic model tuning, Evaluators for Classification and Automating Model Tuning, Evaluators and Automating Model Tuning broadcast variables, Broadcast Variables cluster configurations, Cluster Configurations cluster networking configurations, Cluster Networking Configurations data at rest, Data at Rest-Statistics collection design choices, Design Choices direct vs. indirect approaches, Performance Tuning improved filtering, Improved Filtering joins, Joins memory pressure and garbage collection, Memory Pressure and Garbage Collection-Garbage collection tuning object serialization in RDDs, Object Serialization in RDDs overview of, Performance Tuning parallelism, Parallelism repartitioning and coalescing, Repartitioning and Coalescing role of monitoring in, Performance Tuning scheduling, Scheduling shuffle configurations, Shuffle Configurations temporary data storage (caching), Temporary Data Storage (Caching)-Temporary Data Storage (Caching) User-Defined Functions (UDFs), User-Defined Functions (UDFs) physical plan, Lazy Evaluation, Physical Planning, Logical instructions to physical execution, Temporary Data Storage (Caching) pip install pyspark, Downloading Spark for a Hadoop cluster pipe method, Pipe RDDs to System Commands Pipeline concept, Pipelining Our Workflow pipelining, Transformations, Pipelining pivots, Pivot poisson regression, Generalized Linear Regression polynomial expansion, Polynomial Expansion predicate pushdown, Lazy Evaluation predicate subqueries, Uncorrelated predicate subqueries preprocessing bucketing, Bucketing-Advanced bucketing techniques categorical features, Working with Categorical Features-Text Data Transformers continuous features, Working with Continuous Features-Normalizer converting indexed values back to text, Converting Indexed Values Back to Text converting words into numbers, Converting Words into Numerical Representations-Term frequency–inverse document frequency creating word combinations, Creating Word Combinations estimators, Estimators for Preprocessing feature generation, Feature Manipulation-Polynomial Expansion formatting models according to use case, Formatting Models According to Your Use CaseFormatting Models According to Your Use Case high-level transformers, High-Level Transformers-VectorAssembler indexing in vectors, Indexing in Vectors one-hot encoding, One-Hot Encoding removing common words, Removing Common Words scaling and normalization, Scaling and Normalization SQLTransformers, SQL Transformers StringIndexer, StringIndexer text data transformers, Text Data Transformers-Term frequency–inverse document frequency tokenizing text, Tokenizing Text transformers, Transformers, Persisting Transformers-Writing a Custom Transformer VectorAssembler, VectorAssembler Word2Vec, Word2Vec Principal component analysis (PCA), PCA processing time, Event Time Versus Processing Time, Event Time, Input rate and processing rate processing time trigger, Processing time trigger production applications benefits of Spark for, Running Production Applications deploying, Deploying Spark-Conclusion developing, Developing Spark Applications-Conclusion how Spark runs on clusters, How Spark Runs on a Cluster-Conclusion monitoring and debugging, Monitoring and Debugging-Conclusion performance tuning, Performance Tuning-Conclusion Structured Streaming API, Structured Streaming in Production-Conclusion PushedFilters, Query Pushdown PySpark, Downloading Spark for a Hadoop cluster, PySpark Python launching the console, Launching the Python console PySpark, PySpark type reference, Spark Types writing Spark applications in, Writing Python Applications Q query execution, monitoring of, What to Monitor, Query Status-Batch duration query optimizer, Statistics collection query pushdown, Query Pushdown-Partitioning based on a sliding window questions and comments, How to Contact Us R R overview of, Spark’s Language APIs, R on Spark sparklyr, sparklyr-Machine learning SparkR, SparkR-User-defined functions random forests applied to classification, Random Forest and Gradient-Boosted Trees-Prediction Parameters applied to regression, Random Forests and Gradient-Boosted Trees example, Prediction Parameters model hyperparameters, Model Hyperparameters overview of, Random Forest and Gradient-Boosted Trees prediction parameters, Prediction Parameters training parameters, Training Parameters random splits, Random Splits, Random Splits rdd method, Interoperating Between DataFrames, Datasets, and RDDs read attribute, Basics of Reading Data reading data basics of, Basics of Reading Data core API structure, Read API Structure debugging, Slow Reads and Writes read mode, Read modes real-time decision making, Real-time decision making real-time reporting, Real-time reporting recommendation collaborative filtering with alternating least squares, Collaborative Filtering with Alternating Least Squares evaluators, Evaluators for Recommendation example, Example frequent pattern mining, Frequent Pattern Mining metrics, Metrics-Ranking Metrics model hyperparameters, Model Hyperparameters prediction parameters, Prediction Parameters through graph analysis, Graph Analytics training parameters, Training Parameters use cases for, Recommendation, Use Cases record-at-a-time APIs, Record-at-a-Time Versus Declarative APIs records (see also columns; rows) random samples of, Random Samples random splits of, Random Splits repartition and coalesce, Repartition and Coalesce restricting extraction of, Limit vs. rows, Records and Rows recovery, Fault Tolerance and Checkpointing Redshift Connector, An Abridged List of Popular Packages reduce method, reduce reduceByKey, reduceByKey REFRESH TABLE, Refreshing Table Metadata RegexTokenizer, Tokenizing Text regression decision trees, Decision Trees evaluators and automating model tuning, Evaluators and Automating Model Tuning generalized linear regression, Generalized Linear Regression-Training Summary isotonic regression, Isotonic Regression linear regression, Linear Regression metrics, Metrics models in MLlib, Regression Models in MLlib-Model Scalability random forests and gradient boosted trees, Random Forests and Gradient-Boosted Trees survival regression (accelerated failure time), Survival Regression (Accelerated Failure Time) use cases for, Regression, Use Cases Regular Expressions (RegExes), Regular Expressions-Regular Expressions, Tokenizing Text RelationalGroupedDataset, DataFrames and SQL, Aggregations repartition, Repartition and Coalesce, repartition, Stages, Repartitioning and Coalescing repartitionAndSortWithinPartitions, repartitionAndSortWithinPartitions replace function, replace reports, real-time, Real-time reporting requests, Client Request rescaling applications, Sizing and Rescaling Your Application reserved characters, Reserved Characters and Keywords resilience of business logic, Business logic resilience and evolution of output data, Resilience in output and atomicity resilience, of input data, Input data resilience Resilient Distributed Datasets (RDDs) accessing values in, first actions, Actions-take aggregations, Aggregations-foldByKey caching, Caching checkpointing, Checkpointing CoGroups, CoGroups counting, count creating, Creating RDDs-From Data Sources filtering, filter joins, Joins Key–Value RDDs, Key-Value Basics (Key-Value RDDs)-foldByKey manipulating, Manipulating RDDs mapping, map object serialization in, Object Serialization in RDDs overview of, Lower-Level APIs, About RDDs partitions, Controlling Partitions-Custom Partitioning pipe RDDs to system commands, Pipe RDDs to System Commands-Conclusion random splits, Random Splits RDDs of Case Classes vs. Datasets, Datasets and RDDs of Case Classes reducing, reduce removing duplicates from, distinct saving files, Saving Files serialization, Custom Serialization sorting, sort transformations, Transformations-Random Splits types of, Types of RDDs when to use, When to Use RDDs?, Which Spark API to Use?, DataFrames versus SQL versus Datasets versus RDDs resource utilization problem, On-Premises Cluster Deployments, Cluster/application sizing and sharing REST API Endpoints, Spark REST API RFormula benefits of, RFormula column labels, RFormula example, RFormula operators, Feature Engineering with Transformers, RFormula right outer joins, Right Outer Joins rollup operator, Rollups rounding numbers, Working with Numbers Row type, DataFrames Versus Datasets, Records and Rows, Datasets rows accessing data in, Creating Rows collecting to the driver, Collecting Rows to the Driver concatenating and appending, Concatenating and Appending Rows (Union) converting to columns, Pivot creating, Creating Rows extracting unique, Getting Unique Rows filtering, Filtering Rows generating unique IDs for, Working with Numbers sorting, Sorting Rows runtime properties, Runtime Properties S sample method, Random Samples sampleByKey function, sampleByKey save modes, Save modes saveAsTextFile, saveAsTextFile sbt tool, A Simple Scala-Based App Scala benefits of, A Simple Scala-Based App case classes, In Scala: Case Classes column creation in, Columns comparison operators in, Working with Booleans launching the console, Launching the Scala console symbols in, Columns type reference, Spark Types scalability, Model Scalability, Model Scalability, Collaborative Filtering with Alternating Least Squares, Model Scalability scalar queries, Uncorrelated scalar queries ScalaTest, Managing SparkSessions scaling and normalization, Scaling and Normalization-Normalizer scheduling, Job Scheduling Within an Application, Application Scheduling, Scheduling schema inference, An End-to-End Example, Structured Streaming in Action schema-on-read, Schemas, Schemas schemas components of, Schemas defined, DataFrames defining, Schemas enforcing, Schemas secure deployment configurations, Secure Deployment Configurations Select method, select and selectExpr, Window Functions, Selections and Filtering select statements, Select Statements SelectExpr method, select and selectExpr semi joins, Left Semi Joins sequenceFiles, SequenceFiles serialization, Custom Serialization, Signs and symptoms, Object Serialization in RDDs sessionization, flatMapGroupsWithState-flatMapGroupsWithState sets, grouping, Grouping Sets-Pivot SHOW FUNCTIONS statement, Functions shuffle configuring behavior, Configuring Shuffle Behavior defined, Transformations, Stages efficiency of, Conclusion external service, Miscellaneous Considerations performance tuning, Shuffle Configurations shuffle joins, Communication Strategies shuffle persistence, Shuffle Persistence simple types, Columns sinks console sink, Sources and sinks for testing defined, Sinks files, File source and sink for testing, Sources and sinks for testing memory sinks, Structured Streaming in Action, Sources and sinks for testing sizing applications, Sizing and Rescaling Your Application skewness, calculating, skewness and kurtosis sliding windows, Sliding windows-Sliding windows social networking, In-Degree and Out-Degree Metrics socket source, Sources and sinks for testing sort action, An End-to-End Example sortBy method, sort spaces, removing, Working with Strings Spark (see Apache Spark) Spark applications application properties, Application Properties architecture and components of, The Architecture of a Spark Application-Completion basics of, Spark Applications configuring, Configuring Applications-Job Scheduling Within an Application development process, The Development Process development template, Developing Spark Applications, Spark Logs execution details, Execution Details execution modes, Execution Modes launching, Launching Applications-Application Launch Examples lifecycle of inside of Spark, The Life Cycle of a Spark Application (Inside Spark)-Tasks lifecycle of outside of Spark, The Life Cycle of a Spark Application (Outside Spark)Completion monitoring and debugging, Monitoring and Debugging-Conclusion performance tuning, Performance Tuning-Conclusion scheduling, Application Scheduling, Scheduling (see also deployment) testing, Testing Spark Applications-Connecting to Data Sources writing, Writing Spark Applications-Running the application Spark community, Spark’s Ecosystem and Packages, Data Sources, Community-Local Meetups Spark Deep Learning, An Abridged List of Popular Packages Spark jobs asynchronous job execution, Time-Outs debugging, Debugging and Spark First Aid-Potential treatments defined, Spark UI execution order, The Life Cycle of a Spark Application (Inside Spark) improving speed and execution of, Aggregations monitoring, Queries, Jobs, Stages, and Tasks scheduling, Job Scheduling Within an Application, Application Scheduling, Scheduling stages and tasks in, Logical instructions to physical execution-Tasks Spark logs, Spark Logs Spark Packages, Spark’s Ecosystem and Packages, Spark Packages-External Packages Spark plan, Physical Planning (see also physical plan) Spark REST API, Spark REST API Spark SQL application configurations, Miscellaneous Features benefits of, Big Data and SQL: Spark SQL Catalog, Catalog, Miscellaneous Considerations complex types, Complex Types-Lists databases, Databases vs. DataFrames and Datasets, DataFrames versus SQL versus Datasets versus RDDs functions, Functions history of, Big Data and SQL: Apache Hive launching the console, Launching the SQL console lists, Lists overview of, Spark SQL relationship to Hive, Spark’s Relationship to Hive running queries, How to Run Spark SQL Queries-SparkSQL Thrift JDBC/ODBC Server select statement, Select Statements structs, Structs subqueries, Subqueries-Uncorrelated scalar queries tables, Tables-Caching Tables views, Views-Dropping Views Spark Streaming, Stream Processing Fundamentals Spark Summits, Spark Summit Spark UI benefits of, Spark UI configuring, Configuring the Spark user interface History Server, Spark UI History Server job execution investigation, The Spark UI-The Spark UI overview of, The Spark UI query investigation, The Spark UI-The Spark UI Spark REST API, Spark REST API Structured Streaming API and, Spark UI tabs in, The Spark UI, Other Spark UI tabs Spark Unti Tests, Business logic resilience and evolution spark variable, The SparkSession spark-packages.org, Apache Spark’s Philosophy spark-shell, The Development Process spark-submit, Running Production Applications, The Life Cycle of a Spark Application (Outside Spark), A Simple Scala-Based App, Launching Applications-Launching Applications spark.sql function, DataFrames and SQL spark.sql.hive.metastore.jars, The Hive metastore spark.sql.hive.metastore.version, The Hive metastore spark.sql.shuffle.partitions, Stages SparkConf, The SparkConf SparkContext, How to Use the Low-Level APIs?, The SparkSession-The SparkContext sparklyr data manipulation, Data manipulation data sources, Data sources executing SQL, Executing SQL key concepts, Key concepts lack of DataFrames in, No DataFrames machine learning, Machine learning overview of, sparklyr SparkR benefits and drawbacks of, Pros and cons of using SparkR instead of other languages data manipulation, Data manipulation data sources, Data sources function masking, Function masking key concepts, Key Concepts-User-defined functions machine learning, Machine learning overview of, SparkR, SparkR setup, Setup SparkR functions, SparkR functions only apply to SparkDataFrames user-defined functions (UDFs), User-defined functions-User-defined functions SparkSession instances creating, Starting Spark, The SparkSession in Python, The SparkSession in Scala, The SparkSession managing, Managing SparkSessions purpose of, The SparkSession role in launch process, Launch SparkContext and, The SparkContext-The SparkContext split function, split splittable file formats, Splittable File Types and Compression, Splittable file types and compression SQL (Structured Query Language), What Is SQL? (see also Spark SQL; SQL databases) SQL databases Java Database Connectivity (JDBC) driver for, SQL Databases JDBC data source options, SQL Databases query pushdown, Query Pushdown-Partitioning based on a sliding window reading from, Reading from SQL Databases-Reading from SQL Databases reading in parallel, Reading from databases in parallel, Reading Data in Parallel SQLite, SQL Databases systems available, SQL Databases writing to, Writing to SQL Databases SQLContext, The SparkSession SQLite, SQL Databases SQLTransformers, SQL Transformers stages, Logical instructions to physical execution-Tasks, Queries, Jobs, Stages, and Tasks standalone cluster manager, Cluster Managers-Submitting applications standard deviation, calculating, Variance and Standard Deviation StandardScaler, Estimators for Preprocessing, StandardScaler stateful processing arbitrary, Arbitrary Stateful Processing benefits of Spark for, Event-Time and Stateful Processing considerations for, Arbitrary Stateful Processing flatMapGroupsWithState, flatMapGroupsWithState-flatMapGroupsWithState mapGroupsWithState, mapGroupsWithState-mapGroupsWithState output modes, Output Modes overview of, Stateful Processing time-outs, Time-Outs StatFunctions Package, Working with Numbers statistical functions, Working with Numbers statistics, collecting and maintaining, Statistics collection stddev function, Variance and Standard Deviation stochastic gradient descent, Model Scalability stop words, removing, Removing Common Words STORED AS, Creating Tables stragglers, Slow Tasks or Stragglers stream processing advantages of, Advantages of Stream Processing API selection, Spark’s Streaming APIs basics of, What Is Stream Processing? challenges of, Challenges of Stream Processing continuous vs. micro-batch execution, Continuous Versus Micro-Batch Execution design points, Stream Processing Design Points-Continuous Versus Micro-Batch Execution event time vs. processing time, Event Time Versus Processing Time history of, Stream Processing Fundamentals record-at-a-time vs. declarative APIs, Record-at-a-Time Versus Declarative APIs use cases for, Stream Processing Use Cases-Online machine learning Streaming Listener, Advanced Monitoring with the Streaming Listener string, Working with Strings-Regular Expressions StringIndexer, Machine Learning and Advanced Analytics, StringIndexer strongly connected components algorithm, Strongly Connected Components StructFields, Schemas structs, Structs, Structs StructType, Schemas Structured APIs basic structured operations, Basic Structured Operations-Conclusion code execution, Overview of Structured API Execution-Execution DataFrames, DataFrames and Datasets Datasets, Datasets: Type-Safe Structured APIs, DataFrames and Datasets overview of, Structured API Overview schemas and, Schemas selecting over RDDs, Lower-Level APIs Spark fundamental concepts, Structured API Overview structured Spark types and, Overview of Structured Spark Types-Spark Types, Converting to Spark Types (Literals) Structured Streaming API alerting, Alerting application updates, Updating Your Application-Sizing and Rescaling Your Application applied example, Structured Streaming in Action-Structured Streaming in Action benefits of, What Is Stream Processing?, Structured Streaming, Structured Streaming Basics core concepts, Core Concepts-Watermarks fault tolerance and checkpointing, Fault Tolerance and Checkpointing history of, Stream Processing Fundamentals input and output, Input and Output-Once trigger lack of asynchronous job execution, Time-Outs metrics and monitoring, Metrics and Monitoring-Spark UI overview of, Structured Streaming-Structured Streaming, Structured Streaming Basics as production-ready, Structured Streaming in Production sizing and rescaling applications, Sizing and Rescaling Your Application streaming Dataset API, Streaming Dataset API Streaming Listener monitoring, Advanced Monitoring with the Streaming Listener transformations on streams, Transformations on Streams-Joins subgraphs, Subgraphs subqueries, Subqueries subqueries (Spark SQL), Subqueries-Uncorrelated scalar queries sum aggregation method, DataFrames and SQL sum function, sum sumDistinct function, sumDistinct summary statistics, computing, Working with Numbers supervised learning classification, Classification, Classification-Conclusion goal of, Supervised Learning regression, Regression, Regression-Conclusion survival regression (Accelerated Failure Time), Survival Regression (Accelerated Failure Time) T tables (Spark SQL) caching, Caching Tables creating, Creating Tables creating external, Creating External Tables dropping, Dropping Tables inserting into, Inserting into Tables managed vs. unmanaged tables, Spark-Managed Tables metadata, Describing Table Metadata overview of, Tables take action, An End-to-End Example, Datasets: Type-Safe Structured APIs, Actions, take tasks, Logical instructions to physical execution-Tasks, Queries, Jobs, Stages, and Tasks, Slow Tasks or Stragglers template, Developing Spark Applications, Spark Logs temporary data storage (caching), Temporary Data Storage (Caching)-Temporary Data Storage (Caching) TensorFlowOnSpark, TensorFlowOnSpark TensorFrames, TensorFrames testing connecting to data sources, Connecting to Data Sources connecting to unit testing frameworks, Connecting to Unit Testing Frameworks key principles and tactics, Strategic Principles managing SparkSessions, Managing SparkSessions Spark API selection, Which Spark API to Use? tactical considerations, Tactical Takeaways text data transformers, Text Data Transformers-Term frequency–inverse document frequency text files, Text Files TF-IDF (term frequency-inverse document frequency), Term frequency–inverse document frequency-Term frequency–inverse document frequency the curse of dimensionality, Unsupervised Learning Thrift JDBC/Open Database Connectivity (ODBC) server, SparkSQL Thrift JDBC/ODBC Server time-outs, Arbitrary Stateful Processing, Time-Outs time-related information, Working with Dates and Timestamps-Working with Dates and Timestamps timestamps, Event-Time Processing TimestampType class, Working with Dates and Timestamps toDF method, Interoperating Between DataFrames, Datasets, and RDDs Tokenizer transformer, Transformers, Tokenizing Text toLocalIterator method, Collecting Rows to the Driver toolkit components Datasets API, Datasets: Type-Safe Structured APIs ecosystem of packages and tools, Spark’s Ecosystem and Packages lower-level APIs, Lower-Level APIs machine learning and advanced analytics, Machine Learning and Advanced AnalyticsMachine Learning and Advanced Analytics overview of, What Is Apache Spark?, A Tour of Spark’s Toolset running production applications, Running Production Applications SparkR, SparkR Structured Streaming, Structured Streaming topic modeling, Unsupervised Learning to_date function, Working with Dates and Timestamps to_timestamp function, Working with Dates and Timestamps transfer learning, Ways of Using Deep Learning in Spark, Transfer Learning transformations basics of, Transformations core operations, DataFrame Transformations custom, User-Defined Functions-User-Defined Functions DataFrame creation, Creating DataFrames on Datasets, Transformations end-to-end example, An End-to-End Example-DataFrames and SQL locating APIs, Where to Look for APIs in Structured Streaming API, Transformations and Actions, Transformations on Streams-Joins working with different data types, Working with Booleans-User-Defined Functions transformers example, Feature Engineering with Transformers-Feature Engineering with Transformers formatting models according to use case, Formatting Models According to Your Use Case Interaction feature transformer, Interaction locating, Formatting Models According to Your Use Case persisting, Persisting Transformers properties, Transformer Properties purpose of, High-Level MLlib Concepts, Transformers RFormula, RFormula text data transformers, Text Data Transformers-Word2Vec Tokenizer transformer, Transformers, Tokenizing Text writing custom, Writing a Custom Transformer treeAggregate method, aggregate triggers, Structured Streaming, Triggers, When Data Is Output (Triggers) tumbling windows, Tumbling Windows-Tumbling Windows tuning (see performance tuning) type safety, Datasets: Type-Safe Structured APIs, Which Spark API to Use? typographical conventions, Conventions Used in This Book U uncorrelated predicate subqueries, Uncorrelated predicate subqueries uncorrelated scalar queries, Uncorrelated scalar queries uncorrelated subqueries, Subqueries undirected graphs, Graph Analytics unified software platforms, Apache Spark’s Philosophy unions, Concatenating and Appending Rows (Union) unit tests, Business logic resilience and evolution-Connecting to Unit Testing Frameworks unresolved logical plan, Logical Planning unsupervised learning bisecting -means, Bisecting k-means Summary Gaussian mixture models, Gaussian Mixture Models Latent Dirichlet Allocation, Latent Dirichlet Allocation-Example model scalability, Model Scalability use cases for, Unsupervised Learning, Use Cases -means, k-means-k-means Metrics Summary update output mode, Update mode updates, real-time, Update data to serve in real time updating applications, Updating Your Application-Sizing and Rescaling Your Application uppercase, Working with Strings user segmentation, Unsupervised Learning User-Defined Aggregation Functions (UDAFs), User-Defined Aggregation Functions user-defined functions (UDFs), User-Defined Functions-User-Defined Functions, Aggregating to Complex Types, User-defined functions, Types of RDDs, User-Defined Functions (UDFs), Updating Your Streaming Application Code, User-defined functions-User-defined functions USING, Creating Tables V variance, calculating, Variance and Standard Deviation Vector data type (MLlib), Low-level data types, Scaling and Normalization VectorAssembler, VectorAssembler VectorIndexer, Indexing in Vectors Vectorized UDF, User-Defined Functions (UDFs), Pandas Integration versions, updating, Updating Your Spark Version vertices, Graph Analytics views (Spark SQL) creating, Creating Views dropping, Dropping Views purpose of, Views vocabulary size (vocabSize), Converting Words into Numerical Representations W watermarks, Watermarks, Handling Late Data with Watermarks-Handling Late Data with Watermarks where method, Filtering Rows whitespace, removing, Working with Strings wide dependencies, Transformations windows count-based, mapGroupsWithState-mapGroupsWithState over time-series columns, Structured Streaming partitioning based on sliding, Partitioning based on a sliding window sliding windows, Sliding windows-Sliding windows timestamp conversion, Windows on Event Time tumbling windows, Tumbling Windows-Tumbling Windows unique aggregations using, Window Functions-Grouping Sets withColumnRenamed method, DataFrames and SQL word combinations, creating, Creating Word Combinations Word2Vec, Word2Vec words, converting into numbers, Converting Words into Numerical Representations-Term frequency–inverse document frequency write method, Persisting Transformers writing data basics of, Basics of Writing Data core API structure, Write API Structure debugging, Slow Reads and Writes save mode, Save modes X XGBoost, Random Forest and Gradient-Boosted Trees, External Packages Y YARN (see Hadoop YARN) Z zips, zips About the Authors Bill Chambers is a product manager at Databricks focused on helping customers succeed with their large scale data science and analytics initiatives using Spark and Databricks. Bill also regularly blogs about data science and big data and presents at conferences and meetups. He has a Master’s degree in Information Systems from the UC Berkeley School of Information, where he focused on data science. Matei Zaharia is an assistant professor of computer science at Stanford University and chief technologist at Databricks. He started the Spark project at UC Berkeley in 2009, where he was a PhD student, and he continues to serve as its vice president at Apache. Matei also co-started the Apache Mesos project and is a committer on Apache Hadoop. Matei’s research work was recognized through the 2014 ACM Doctoral Dissertation Award and the VMware Systems Research Award. Colophon The animal on the cover of Spark: The Definitive Guide is the swallow-tailed kite (Elanoides forficatus). Found in woodland and wetland locations ranging from southern Brazil to the southeastern United States, these raptors subsist on small reptiles, ambhibians, and mammals, as well as large insects. They build their nests near water. Swallow-tailed kites tend to be 20–27 inches in length, and coast through the air on wings spanning around 4 feet, using their sharply forked tails to steer. Their plumage grows in a strikingly contrasting black and white, and they spend most of their time in the air, even grazing the surface of bodies of water to drink rather than staying put on the ground. Among the raptor species, Elanoides forficatus are social animals, and often nest in close proximity or roost for the night in large communal groups. During migration, they may travel in groups numbering in the hundreds or thousands. Many of the animals on O’Reilly covers are endangered; all of them are important to the world. To learn more about how you can help, go to animals.oreilly.com. The cover image is from Lydekker’s The Royal Natural History. The cover fonts are URW Typewriter and Guardian Sans. The text font is Adobe Minion Pro; the heading font is Adobe Myriad Condensed; and the code font is Dalton Maag’s Ubuntu Mono.