Microsoft STP ML On AWS Technical Tech Participant Guide
User Manual:
Open the PDF directly: View PDF 
.
Page Count: 70
A W S S o l u t i o n s T r a i n i n g f o r P a r t n e r s
Machine Learning on AWS - Technical
Vijay
AWS Partner Trainer
© 2018 Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon confidential.
Machine Learning
Prediction is the process of filling in missing information; it uses data
you have to generate data you don’t have.
Learning Language Perception Problem
Solving Insight
© 2018 Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon confidential.
The Artificial Intelligence Landscape
Five Tribes / Two Breakthroughs
Tribe Origins Algorithm
Bayesians Statistics Probabilistic
Analogizers Psychology Kernel
Symbolists Logic Inverse Deduction
Evolutionaries Biology Genetic
Connectionists Neuroscience Back Propagation
•Computer Vision : CNNs
•Static / Unstructured
•Natural Language Processing : RNNs / LSTM
•Sequential / Structured
AWS Mission
Put machine learning in the hands of every developer, data scientist
and architect
© 2018 Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon confidential.
© 2018 Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon confidential.
Customers Running ML on AWS
The AWS Machine Learning Stack
Vision Amazon Rekognition
Image
Amazon Rekognition
Video
Language Lex
Translate
Comprehend
Speech Amazon Polly
Transcribe
Amazon
SageMaker AWS DeepLens
Amazon
Machine
Learning
Amazon EMR
Spark
Amazon
Mechanical
Turk
AWS Deep Learning AMI
TensorFlow Apache
MXNet Gluon Cognitive
Toolkit Caffe Keras PyTorch Chainer
Compute GPU - P3 AWS Greengrass Mobile
Platform
Services
Application
Services
Frameworks
& Infrastructure
The AWS Machine Learning Stack
Application
Services
Platform
Services
Frameworks
& Infrastructure
Vision Amazon Rekognition
Image
Amazon Rekognition
Video
Language Lex
Translate
Comprehend
Speech Amazon Polly
Transcribe
Amazon
SageMaker AWS DeepLens
Amazon
Machine
Learning
Amazon EMR
Spark
Amazon
Mechanical
Turk
AWS Deep Learning AMI
TensorFlo
w
Apache
MXNet Gluon Cognitive
Toolkit Caffe Keras PyTorch Chainer
Compute GPU - P3 AWS Greengrass Mobile
Demo 1: Vision Services
© 2018 Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon confidential.
The AWS Machine Learning Stack
Application
Services
Platform
Services
Frameworks
& Infrastructure
Vision Amazon Rekognition
Image
Amazon Rekognition
Video
Language Lex
Translate
Comprehend
Speech Amazon Polly
Transcribe
Amazon
SageMaker AWS DeepLens
Amazon
Machine
Learning
Amazon EMR
Spark
Amazon
Mechanical
Turk
AWS Deep Learning AMI
TensorFlo
w
Apache
MXNet Gluon Cognitive
Toolkit Caffe Keras PyTorch Chainer
Compute GPU - P3 AWS Greengrass Mobile
Demo 2: Language Services
© 2018 Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon confidential.
The AWS Machine Learning Stack
Application
Services
Platform
Services
Frameworks
& Infrastructure
Vision Amazon Rekognition
Image
Amazon Rekognition
Video
Language Lex
Translate
Comprehend
Speech Polly
Transcribe
Amazon
SageMaker AWS DeepLens
Amazon
Machine
Learning
Amazon EMR
Spark
Amazon
Mechanical
Turk
AWS Deep Learning AMI
TensorFlo
w
Apache
MXNet Gluon Cognitive
Toolkit Caffe Keras PyTorch Chainer
Compute GPU - P3 AWS Greengrass Mobile
The AWS Machine Learning Stack
Application
Services
Platform
Services
Frameworks
& Infrastructure
Vision Amazon Rekognition
Image
Amazon Rekognition
Video
Language Lex
Translate
Comprehend
Speech Polly
Transcribe
Amazon
SageMaker AWS DeepLens
Amazon
Machine
Learning
Amazon EMR
Spark
Amazon
Mechanical
Turk
AWS Deep Learning AMI
TensorFlow Apache
MXNet Gluon Cognitive
Toolkit Caffe Keras PyTorch Chainer
Compute GPU - P3 AWS Greengrass Mobile
Demo 3: Deep Learning AMI
© 2018 Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon confidential.
The AWS Machine Learning Stack
Application
Services
Platform
Services
Frameworks
& Infrastructure
Vision Amazon Rekognition
Image
Amazon Rekognition
Video
Language Lex
Translate
Comprehend
Speech Amazon Polly
Transcribe
Amazon
SageMaker AWS DeepLens
Amazon
Machine
Learning
Amazon EMR
Spark
Amazon
Mechanical
Turk
AWS Deep Learning AMI
TensorFlow Apache
MXNet Gluon Cognitive
Toolkit Caffe Keras PyTorch Chainer
Compute GPU - P3 AWS Greengrass Mobile
© 2018 Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon confidential.
Amazon EC2 P3 Instances
The fastest, most powerful GPU instances in
the cloud
•Up to 8 NVIDIA Tesla V100 GPUs
•16GB GPU memory with 900 GB/sec peak
bandwidth
•1 PetaFLOPs of computational
performance
•14x better than P2
•300 GB/s GPU-to-GPU communication
(NVLink)
•9X better than P2
Airbnb
Toyota Research
Institute
OpenAI
© 2018 Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon confidential.
Amazon EC2 P3 Instances
© 2018 Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon confidential.
Amazon EC2 P3 Instances
© 2018 Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon confidential.
AWS Deep Learning
Amazon
Machine Image (AMI)
•Get started quickly with easy-to-launch
tutorials
•Hassle-free setup and configuration
•Pay only for what you use – no additional
charge for the AMI
•Accelerate your model training and
deployment
•Support for popular deep learning
frameworks
TensorFlow, MXNet, Gluon, Keras, Caffe2, PyTorch, Zendesk, Matric Analytics, SCDM, etc.
© 2018 Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon confidential.
Amazon ML Solutions Lab
Lots of companies doing
Machine Learning
Unable to unlock business
potential
Lack ML
expertise
© 2018 Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon confidential.
Amazon ML Solutions Lab
Lots of companies doing
Machine Learning
Unable to unlock business
potential
Lack ML
expertise
Brainstorming Modeling Education
Amazon ML Solutions
Lab provides the
missing ML expertise
© 2018 Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon confidential.
Amazon ML Lab Customers
Johnson & Johnson
Toyota Research Institute
Washington Post
The Machine Learning Process
© 2018 Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon confidential.
The Machine Learning Process
Monitoring
& Debugging
Predictions
Yes
Model Deployment
Data
Augmentation No
Model Evaluation
Model Training
& Parameter Tuning
Feature
Engineering
Data Visualization
& Analysis
Data Preparation
Data Integration
Data Collection
ML Problem
Framing
Business Problem
Feature
Augmentation
Are business
goals met?
The ML Process
Integration: The Data Architecture
Build the data
platform:
•Amazon Simple
Storage Service
(Amazon S3)
•Amazon Athena
•Amazon EMR
•Amazon Redshift
•AWS Glue
Monitoring
& Debugging
Predictions
Yes
Model Deployment
Data
Augmentation No
Model Evaluation
Model Training
& Parameter Tuning
Feature
Engineering
Data Visualization
& Analysis
Data Preparation
Data Integration
Data Collection
ML Problem
Framing
Business Problem
Feature
Augmentation
Are business
goals met?
The ML Process
The Model Training: Undifferentiated Heavy Lifting
•Setup and
Manage
•Notebook
Environments
•Training
Clusters
•Write Data
Connectors
•Scale ML
algorithms to
large datasets
•Distribute ML
training
algorithm to
multiple
machines
•Secure model
artifacts
Monitoring
& Debugging
Predictions
Yes
Model Deployment
Data
Augmentation No
Model Evaluation
Model Training
& Parameter Tuning
Feature
Engineering
Data Visualization
& Analysis
Data Preparation
Data Integration
Data Collection
ML Problem
Framing
Business Problem
Feature
Augmentation
Are business
goals met?
The ML Process
DevOps: Undifferentiated Heavy Lifting
•Setup and Manage
Inference Clusters
•Manage and Scale
Model Inference
APIs
•Monitor and
Debug Model
Predictions
•Models versioning
and performance
tracking
•Automate New
Model version
promotion to
production (A/B
testing)
Monitoring
& Debugging
Predictions
Yes
Model Deployment
Data
Augmentation No
Model Evaluation
Model Training
& Parameter Tuning
Feature
Engineering
Data Visualization
& Analysis
Data Preparation
Data Integration
Data Collection
ML Problem
Framing
Business Problem
Feature
Augmentation
Are business
goals met?
Why Amazon SageMaker?
You Only Have to
Write Business Logic
Monitoring
& Debugging
Predictions
Yes
Model Deployment
Data
Augmentation No
Model Evaluation
Model Training
& Parameter Tuning
Feature
Engineering
Data Visualization
& Analysis
Data Preparation
Data Integration
Data Collection
ML Problem
Framing
Business Problem
Feature
Augmentation
Are business
goals met?
Amazon SageMaker
© 2018 Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon confidential.
A Fully-Dockerized Lifecycle
From Discovery to Development and Deployment
Data Scientists
Training Algorithm
Training DataAmazon S3 Amazon Elastic
Container
Registry
Amazon SageMaker
Training Algorithm Inference Engine
Training Data
Model Artifacts
A Fully-Dockerized Lifecycle
From discovery to development and deployment
Developers and Operations
Amazon S3
Amazon SageMaker
Training Data Model Artifacts
EndPoint
Model Artifacts Amazon Elastic
Container Registry
Training Algorithm Inference Engine
Inference Engine
Identification
Authorization
Logging
Analytics
A Fully-Dockerized Lifecycle
From discovery to development and deployment
Delighted Customers
API Gateway Predictive Model
© 2018 Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon confidential.
Amazon SageMaker
Launch Customers
Intuit
Digital Globe
ZipRecruiter
Hotels.com
Thomson Reuters
© 2018 Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon confidential.
Customer Example: Intuit
“With Amazon SageMaker, we can
accelerate our Artificial Intelligence
initiatives at scale by building and
deploying our algorithms on the
platform. We will create novel large-
scale machine learning and AI
algorithms and deploy them on this
platform to solve complex problems
that can power prosperity for our
customers.”
Ashok Srivastava, Chief Data
Officer, Intuit
INTUIT
Easy data exploration
in SageMaker notebooks
Building around virtualization for flexibility
Auto-scalable model hosting environment
Key Benefits of Amazon
SageMaker
at
Intuit
From To
Ad-hoc setup and management of notebook
environments
Limited choices for model deployment
Competing for compute resources across
teams
Fraud Detection using SageMaker
Data Collection Calculate Features Feature Store Model Training Model Hosting Client Service
Reader
Cleansing
Processor
Lookup
Training Amazon SageMaker
Amazon EMR
INTUIT
APACHE KAFKA
&
SPARK
STREAMING
Customer Example: DigitalGlobe
“As the world’s leading provider of high-resolution Earth
imagery, data and analysis, DigitalGlobe works with enormous
amounts of data every day. DigitalGlobe is making it easier for
people to find, access, and run compute against our entire
100PB image library, which is stored in AWS’s cloud, to apply
deep learning to satellite imagery. We plan to use Amazon
SageMaker to train models against petabytes of Earth
observation imagery datasets using hosted Jupyter notebooks,
so DigitalGlobe's Geospatial Big Data Platform (GBDX) users
can just push a button, create a model, and deploy it all within
one scalable distributed environment at scale.”
Dr. Walter Scott, CTO of Maxar Technologies and
founder of DigitalGlobe
DigitalGlobe
Customer Example: ZipRecruiter
“We’re focused on making it faster and easier than ever
to hire and get hired, training our machine learning
algorithms against hundreds of millions of historical
transactional activities in order to deliver highly relevant
job matches as quickly as possible. Amazon SageMaker
provided us with an answer to problems we had with ML
workflow management, allowing us to train, evaluate
and deploy models in a flexible way. In addition,
Amazon SageMaker's modularity provides the ability to
build and create models independently, which is a
compelling feature for ZipRecruiter.”
Avi Golan, VP of Engineering, ZipRecruiter
ZipRecruiter
Amazon SageMaker
1
I
Notebook Instances
2
I
Algorithms
3
I
ML Training Service
4
I
ML Hosting Service
Amazon SageMaker’s Components
SageMaker Notebook Instances
Zero Setup for Exploratory Data Analysis
Just add data!
•Recommendations/Personalization
•Fraud Detection
•Forecasting
•Image Classification
•Churn Prediction
•Marketing Email/Campaign
Targeting
•Log processing and anomaly
detection
•Speech to Text
•More…
Authoring &
Notebooks
ETL Access to AWS
Database services
Access to S3
Data Lake
Demo 4: A simple Jupyter Notebook
Pythagorean Theorem
© 2018 Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon confidential.
Demo 5: Predicting AWS Spot Pricing
© 2018 Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon confidential.
SageMaker Built-in Algorithms
10x Faster
Streaming datasets, for
cheaper training
Train faster, in a single
pass
Greater reliability on
extremely large
datasets
Choice of several ML
algorithms
SageMaker Built-in Algorithms
Time vs. Money
$$$$
$$$
$$
$
Minutes Hours Days Weeks Months
Single
Machine
Distributed, with
Strong Machines
SageMaker Built-in Algorithms
Streaming
Data Size
Memory
Data Size
Time/Cost
SageMaker Built-in Algorithms
Streaming
GPU State
SageMaker Built-In Algorithms
Distributed Shared State
Shared
State
GPU
GPU
GPU Local
State
Local
State
Local
State
Local
State
Local
State
Local
State
SageMaker Built-in Algorithms
Cost vs. Time
$$$$
$$$
$$
$
Minutes Hours Days Weeks Months
Best Alternative
Amazon SageMaker
Infinitely Scalable ML Algorithms
Regression (mean squared error)
SageMaker Other
1.02 1.06
1.09 1.02
0.332 0.183
0.086 0.129
83.3 84.5
Classification (F1 Score)
SageMaker Other
0.980 0.981
0.870 0.930
0.997 0.997
0.978 0.964
0.914 0.859
0.470 0.472
0.903 0.908
0.508 0.508
30 GB datasets for web-spam and web-url classification
0.
0.275
0.55
0.825
1.1
1.375
0. 7.5 15. 22.5 30.
Cost in Dollars
Billable time in Minutes
sagemaker-url sagemaker-spam other-url other-spam
Linear Learner
Log_loss F1 Score Seconds
SageMaker 0.494 0.277 820
Other (10 Iter) 0.516 0.190 650
Other (20 Iter) 0.507 0.254 1300
Other (50 Iter) 0.481 0.313 3250
Click Prediction 1 TB advertising dataset,
m4.4xlarge machines, perfect scaling.
$-
$50.00
$100.00
$150.00
$200.00
1. 2.75 4.5 6.25 8.
Cost in Dollars
Billable Time in Hours
10
machines
20
machines
30
machines
4050
Factorization Machines
0
2
4
6
8
10 100 500
Billable Time in Minutes
Number of Clusters
sagemaker other
k SageMaker Other
Text
1.2GB
10 1.18E3 1.18E3
100 1.00E3 9.77E2
500 9.18.E2 9.03E2
Images
9GB
10 3.29E2 3.28E2
100 2.72E2 2.71E2
500 2.17E2 Failed
Videos
27GB
10 2.19E2 2.18E2
100 2.03E2 2.02E2
500 1.86E2 1.85E2
Advertising
127GB
10 1.72E7 Failed
100 1.30E7 Failed
500 1.03E7 Failed
Synthetic
1100GB
10 3.81E7 Failed
100 3.51E7 Failed
500 2.81E7 Failed
Running Time vs. Number of Clusters
~10x Faster!
K-Means Clustering
More than 10x faster
at a fraction the cost!
0.00
27.50
55.00
82.50
110.00
8 10 20
Mb/Sec/Machine
Number of Machines
other sagemaker-deterministic sagemaker-randomized
Cost vs. Time Throughput and Scalability
0.
1.25
2.5
3.75
5.
0. 12.5 25. 37.5 50.
Cost in Dollars
Billable time in Minutes
other sagemaker-deterministic sagemaker-randomized
Principal Component Analysis (PCA)
Perplexity vs. Number of Topic
Input term counts vector
Document
Posterior
Sampled Document
Representation
Decoder:
Softmax
Output term counts vector
0.
2750.
5500.
8250.
11000.
13750.
0 50 100 150 200
Perplexity
Number of Topics
NTM Other
Neural Topic Modeling
Encoder: feedforward net
Mean absolute
percentage error P90 Loss
DeepAR R DeepAR R
Traffic
Hourly occupancy rate of
963 bay area freeways
0.14 0.27 0.13 0.24
Electricity
Electricity use of 370
homes over time
0.07 0.11 0.08 0.09
Page views
Page view hits
of websites
10k 0.32 0.32 0.44 0.31
180k 0.32 0.34 0.29 NA
One hour on p2.xlarge, $1
Input
Network
DeepAR
More Great ML Algorithms
Training Time vs. Number of Topics
0
50
100
150
200
250
0 25 50 75 100
Training Time in Minutes
Number of Topics
lda-data-a lda-data-b other-data-a other-data-b
Spectral LDA
Throughput vs. Number of Machines
XGBoost is one of the most
commonly used
implementations of boosted
decision trees in the world.
It is now available in Amazon
SageMaker!
0.
325.
650.
975.
1300.
1625.
0 18 35 53 70
Throughput in MB/Sec
Number of Machines (C4.8xLarge)
Boosted Decision Trees
XGBoost
English-German Translation
0.
7.5
15.
22.5
30.
0. 7.5 15. 22.5
BLEU Score
Billable Time in Hours
P2.16x P2.8x P2.x
Best known result!
Based on Sockeye and Apache
incubated MxNet, Multi-GPU,
and can be used for Neural
Machine Translation.
Supports both RNN/CNN
as encoder/decoder
Sequence to Sequence
Implementation in MxNet of
ResNet.
Other networks such as
DenseNet and Inception will
be added in the future.
Transfer learning: begin with
a model already trained on
ImageNet!
0.
0.8
1.5
2.3
3.
3.8
0 1 2 3 4 5
Speedup
Number of Machine (P2)
Speedup with Horizontal Scaling
Image Classification
Amazon
SageMaker
Built
-
In
Algorithms 10x Better
Training code
•Matrix Factorization
•Regression
•Principal Component Analysis
•K-Means Clustering
•Gradient Boosted Trees
•And More!
Amazon-Provided Algorithms
Bring Your Own Script
(Amazon SageMaker builds the Container)
Amazon SageMaker
Estimators in Apache Spark
Bring Your Own Algorithm
(You build the Container)
Apache
Spark
MxNet
TensorFlow
•Matrix Factorization
•Regression
•Principal Component Analysis
•K-Means Clustering
•Gradient Boosted Trees
•And More!
Amazon-Provided Algorithms Bring Your Own Algorithm
(You build the Container)
Amazon SageMaker Built-In Algorithms
Managed Distributed Training with Flexibility
Fetch Training
data
Save Model
Artifacts
Fully
managed
Secured
Amazon ECR
Save Inference
Image
Apache
Spark
MxNet
TensorFlow
Bring Your Own Script
(Amazon SageMaker builds the Container)
Amazon SageMaker
Estimators in Apache Spark
Training code
Demo 6: Using Amazon SageMaker Built-in Algorithms
© 2018 Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon confidential.
Amazon SageMaker Hosting Service
Easy Model Deployment to Amazon SageMaker
•InstanceType: c3.4xlarge
•InitialInstanceCount: 3
•ModelName: prod
•VariantName: primary
•InitialVariantWeight: 100
100
%
20%
Endpoint Configuration
Inference
EndPoint
Model Versions
80%
Amazon ECR
Amazon S3
Ground
Truth
SageMaker Hosting Service
Easy Model Deployment to Amazon SageMaker
•Auto-Scaling Inference APIs
•A/B Testing (more to come)
•Low Latency & High Throughput
•Bring Your Own Model
•Python SDK
Demo 7: Analyzing Breast Cancer Datasets
© 2018 Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon confidential.
Demo 8: Using Containers with Amazon SageMaker
© 2018 Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon confidential.
Amazon SageMaker
Reference Architecture
Amazon
SageMaker
Notebooks Training
Algorithm Amazon SageMaker
Training
Amazon ECR
Code Commit
Code Pipeline
Amazon SageMaker
Hosting
Coco dataset
AWS
Lambda
API
Gateway
Build
Train
Deploy
static website hosted on S3
Inference requests
Amazon S3
Amazon
Cloudfront
Web assets on
Cloudfront
Amazon SageMaker
Technology Competency Partners
Data
Services Platform Solutions SaaS and API Solutions
Alteryx Bonsai DataRobot Anodot SigOpt
CrowdFlower C3 IoT DOMINO
DATA LAB Luminoso Veritone
Paxata Databricks H2O.ai Narrative
Science x.ai
TRIFACTA Data Iku
Call To Action
•Getting started with Amazon SageMaker:
https://aws.amazon.com/sagemaker/
•Use the Amazon SageMaker SDK:
•For Python: https://github.com/aws/sagemaker-python-sdk
•For Spark: https://github.com/aws/sagemaker-spark
•SageMaker Examples: https://github.com/awslabs/amazon-
sagemaker-examples
© 2018 Amazon Web Services, Inc. or its affiliates. All rights reserved. This work may not be reproduced or redistributed, in whole or in part, without prior written permission
from Amazon Web Services, Inc. Commercial copying, lending, or selling is prohibited. Corrections or feedback on the course, please email us at: aws-course-
feedback@amazon.com. For all other questions, contact us at: https://aws.amazon.com/contact-us/aws-training/. All trademarks are the property of their owners.
Thank You
