AWS ADMINISTRATION THE DEFINITIVE GUIDE
AWS_ADMINISTRATION_THE_DEFINITIVE_GUIDE%20
User Manual: Pdf
Open the PDF directly: View PDF 
.
Page Count: 326
| Download | |
| Open PDF In Browser | View PDF |
AWS Administration – The
Definitive Guide
Learn to design, build, and manage your infrastructure
on the most popular of all the Cloud platforms—Amazon
Web Services
Yohan Wadia
professional expertise distilled
P U B L I S H I N G
BIRMINGHAM - MUMBAI
AWS Administration – The Definitive Guide
Copyright © 2016 Packt Publishing
All rights reserved. No part of this book may be reproduced, stored in a retrieval
system, or transmitted in any form or by any means, without the prior written
permission of the publisher, except in the case of brief quotations embedded in
critical articles or reviews.
Every effort has been made in the preparation of this book to ensure the accuracy
of the information presented. However, the information contained in this book
is sold without warranty, either express or implied. Neither the author nor Packt
Publishing, and its dealers and distributors will be held liable for any damages
caused or alleged to be caused directly or indirectly by this book.
Packt Publishing has endeavored to provide trademark information about all of the
companies and products mentioned in this book by the appropriate use of capitals.
However, Packt Publishing cannot guarantee the accuracy of this information.
First published: February 2016
Production reference: 1080216
Published by Packt Publishing Ltd.
Livery Place
35 Livery Street
Birmingham B3 2PB, UK.
ISBN 978-1-78217-375-5
www.packtpub.com
Credits
Author
Yohan Wadia
Reviewer
Paul Deng
Commissioning Editor
Kunal Parikh
Acquisition Editor
Rahul Nair
Content Development Editor
Anish Dhurat
Technical Editor
Pranjali Mistry
Copy Editor
Charlotte Carneiro
Project Coordinator
Bijal Patel
Proofreader
Safis Editing
Indexer
Monica Ajmera Mehta
Production Coordinator
Nilesh Mohite
Cover Work
Nilesh Mohite
About the Author
Yohan Wadia is a client-focused virtualization and cloud expert with 6 years of
experience in the IT industry.
He has been involved in conceptualizing, designing, and implementing large-scale
solutions for a variety of enterprise customers based on VMware vCloud, Amazon
Web Services, and Eucalyptus Private Cloud.
His community-focused involvement also enables him to share his passion for
virtualization and cloud technologies with peers through social media engagements,
public speaking at industry events, and through his personal blog—yoyoclouds.com
He is currently working with an IT services and consultancy company as a Cloud
Solutions Lead and is involved in designing and building enterprise-level cloud
solutions for internal as well as external customers. He is also a VMware Certified
Professional and a vExpert (2012 and 2013).
I wish to dedicate this book to both my loving parents, Ma and Paa.
Thank you for all your love, support, encouragement, and patience. I
would also like to thank the entire Packt Publishing team, especially
Ruchita Bhansali, Athira Laji, and Gaurav Sharma, for their excellent
guidance and support.
And finally, a special thanks to one of my favorite bunch of people:
the amazing team of developers, support staff, and engineers who
work at AWS for such an "AWSome" cloud platform!
Not all those who wander are lost.
- J. R. R. Tolkien
About the Reviewer
Paul Deng is a senior software engineer with over 8 years of experience in
end-to-end IoT app design and development, including embedded devices,
large-scale machine learning, cloud, and web apps.
Paul holds software algorithm patents and was a finalist of Shell Australian
Innovation Challenge 2011. He has authored several publications on IoT and cloud.
He lives in Melbourne, Australia, with his wife Cindy and son Leon. Visit his
website at http://dengpeng.de to see what he is currently exploring and to learn
more about him.
www.PacktPub.com
Support files, eBooks, discount offers, and more
For support files and downloads related to your book, please visit www.PacktPub.com.
Did you know that Packt offers eBook versions of every book published, with PDF and ePub
files available? You can upgrade to the eBook version at www.PacktPub.com and as a print
book customer, you are entitled to a discount on the eBook copy. Get in touch with us at
service@packtpub.com for more details.
At www.PacktPub.com, you can also read a collection of free technical articles, sign up for a
range of free newsletters and receive exclusive discounts and offers on Packt books and eBooks.
TM
https://www2.packtpub.com/books/subscription/packtlib
Do you need instant solutions to your IT questions? PacktLib is Packt's online digital book
library. Here, you can search, access, and read Packt's entire library of books.
Why subscribe?
•
Fully searchable across every book published by Packt
•
Copy and paste, print, and bookmark content
•
On demand and accessible via a web browser
Free access for Packt account holders
If you have an account with Packt at www.PacktPub.com, you can use this to access PacktLib
today and view 9 entirely free books. Simply use your login credentials for immediate access.
Instant updates on new Packt books
Get notified! Find out when new books are published by following @PacktEnterprise on
Twitter or the Packt Enterprise Facebook page.
Table of Contents
Preface
Chapter 1: Introducing Amazon Web Services
What is cloud computing?
Cloud computing features and benefits
Cloud computing use cases
Introducing Amazon Web Services
AWS architecture and components
Regions and availability zones
AWS platform overview
Getting started with AWS
Introducing the AWS Management Console
Getting started with AWS CLI
Plan of attack!
Summary
Chapter 2: Security and Access Management
Security and clouds
Is AWS really secure
Shared responsibility model
Identity and Access Management
Business use case scenario
Getting started with the IAM Console
Creating users and groups
Understanding permissions and policies
Creating and assigning policies
Managing access and security using the AWS CLI
Planning your next steps
Recommendations and best practices
Summary
[i]
vii
1
2
3
3
4
5
5
7
11
15
18
20
22
23
23
24
24
25
27
27
30
35
39
41
46
48
49
Table of Contents
Chapter 3: Images and Instances
51
Introducing EC2!
EC2 use cases
Introducing images and instances
Understanding images
51
52
53
53
Amazon Linux AMI
56
Understanding instances
EC2 instance pricing options
57
58
Working with instances
60
Connecting to your instance
Configuring your instances
Launching instances using the AWS CLI
69
75
77
On-demand instances
Reserved instances
Spot instances
58
59
60
Stage 1 – choose AMI
Stage 2 – choose an instance type
Stage 3 – configure instance details
Stage 4 – add storage
Stage 5 – tag instances
Stage 6 – configure security groups
Stage 7 – review instance launch
Stage 1 – create a key pair
Stage 2 – create a security group
Stage 3 – add rules to your security group
Stage 4 – launch the instance
Cleaning up!
Planning your next steps
Recommendations and best practices
Summary
Chapter 4: Security, Storage, Networking, and Lots More!
An overview of security groups
Understanding EC2 networking
Determining your instances IP addresses
Working with Elastic IP addresses
Create an Elastic IP address
Allocating Elastic IP addresses
Disassociating and releasing an Elastic IP address
Understanding EBS volumes
EBS volume types
Getting started with EBS Volumes
Creating EBS volumes
Attaching EBS volumes
Accessing volumes from an instance
62
63
64
65
66
67
68
77
78
79
79
80
81
82
83
85
85
89
92
93
95
95
97
98
99
99
100
102
103
[ ii ]
Table of Contents
Detaching EBS volumes
Managing EBS volumes using the AWS CLI
Backing up volumes using EBS snapshots
Planning your next steps
Recommendations and best practices
Summary
Chapter 5: Building Your Own Private Clouds Using
Amazon VPC
An overview of Amazon VPC
VPC concepts and terminologies
Subnets
Security groups and network ACLs
Routing tables
VPC endpoints
Internet Gateways
NAT instances
DNS and DHCP Option Sets
104
105
107
112
113
114
115
115
117
117
119
120
120
122
123
124
VPC limits and costs
Working with VPCs
VPC deployment scenarios
Getting started with the VPC wizard
125
126
126
127
Launching instances in your VPC
142
Viewing VPCs
Listing out subnets
Working with route tables
Listing Internet Gateways
Working with security groups and Network ACLs
Creating the web servers
Creating the database servers
Planning next steps
Best practices and recommendations
Summary
Chapter 6: Monitoring Your AWS Infrastructure
An overview of Amazon CloudWatch
Concepts and terminologies
Metrics
Namespaces
Dimensions
Time stamps and periods
Units and statistics
Alarms
CloudWatch limits and costs
Getting started with CloudWatch
Monitoring your account's estimate charges using CloudWatch
[ iii ]
133
135
136
137
138
142
144
144
146
147
149
149
150
150
151
151
151
152
153
154
155
155
Table of Contents
Monitoring your instance's CPU Utilization using CloudWatch
Monitoring your instance's memory and disk utilization using
CloudWatch Scripts
159
Monitoring logs using CloudWatch Logs
172
Creating CloudWatch access roles
Installing the CloudWatch monitoring scripts
Viewing the custom metrics from CloudWatch
CloudWatch Log concepts and terminologies
Getting Started with CloudWatch Logs
Viewing the logs
Creating metric filters and alarms
Planning your next steps
Recommendations and best practices
Summary
Chapter 7: Manage Your Applications with Auto Scaling
and Elastic Load Balancing
An overview of Auto Scaling
Auto scaling components
Auto scaling groups
Launch configurations
Scaling plans
166
166
168
171
172
173
177
177
180
180
181
183
184
186
186
186
187
Introducing the Elastic Load Balancer
Creating your first Elastic Load Balancer
187
189
Getting started with Auto Scaling
Creating the Launch Configuration
196
197
Creating the Auto Scaling Group
200
Verifying and testing Auto Scaling
208
Step 1 – Defining the Load Balancer
Step 2 – Assign security groups
Step 3 – configure security settings
Step 4 – Configure Health Check
Step 5 – Add EC2 instances
Step 6 – Add tags
Step 7 – Review and Create
Step 1 – Choose AMI
Step 2 – Choose Instance type
Step 3 – Configure details
Step 4 – Add storage
Step 5 – Configure Security Group
Step 6 – Review
190
192
192
193
195
195
195
198
198
198
199
199
199
Step 1 – Configure Auto Scaling group details
Step 2 – Configure scaling policies
Step 3 – Configure notifications
Step 4 – Configure tags
Step 5 – Review
[ iv ]
201
203
206
207
207
Table of Contents
Suspend, resume and delete Auto Scaling
Planning your next steps
Recommendations and best practices
Summary
Chapter 8: Database-as-a-Service Using Amazon RDS
An overview of Amazon RDS
RDS instance types
Multi-AZ deployments and Read Replicas
Working with Amazon RDS
Getting started with MySQL on Amazon RDS
Creating a MySQL DB instance
Connecting remotely to your DB instance
Testing your database
Modifying your DB instances
Backing up DB instances
Creating Read Replicas and promoting them
Logging and monitoring your DB instance
Cleaning up your DB instances
Planning your next steps
Recommendations and best practices
Summary
210
212
213
214
215
215
217
219
222
223
224
232
233
234
236
237
241
242
243
244
245
Chapter 9: Working with Simple Storage Service
247
Chapter 10: Extended AWS Services for Your Applications
273
Introducing Amazon S3
Getting started with S3
Creating buckets
Uploading your first object to a bucket
Viewing uploaded objects
Accessing buckets and objects using S3CMD
Managing an object's and bucket's permissions
Using buckets to host your websites
S3 events and notification
Bucket versioning and lifecycle management
Cross-Region Replication
Planning your next steps
Recommendations and best practices
Summary
Introducing Amazon Route53
Working with Route53
Creating hosted zones
Getting started with traffic flow
[v]
247
250
250
252
254
254
257
261
263
265
269
270
271
272
273
274
275
279
Table of Contents
Configuring health checks
Content delivery using Amazon CloudFront
Getting started with distributions
CloudFront recommendations and best practices
What's new in AWS?
Elastic Container Service
Elastic File System
Database migration made easy with Database Migration Service
Go serverless with AWS Lambda
Resources, recommendations, and best practices
Summary
Index
[ vi ]
281
284
285
289
289
289
291
293
293
294
295
297
Preface
Cloud computing has definitely matured and evolved a lot ever since its conception.
Practically all major industries and top fortune 500 companies today run their
application workloads on clouds to reap all sorts of benefits, ranging from reduced
costs, better availability of their applications, and easier manageability to on-demand
scalability, and much more! At the forefront of this cloud innovation is a market
leader like no other: Amazon Web Services (AWS).
AWS provides a ton of easy-to-use products and services that you can leverage to
build, host, deploy, and manage your applications on the cloud. It also provides a
variety of ways to interact with these services, such as SDKs, APIs, CLIs, and even a
web-based management console.
This book is a one stop shop where you can find all there is to getting started with
the core AWS services, which include EC2, S3, RDS, VPCs, and a whole lot more! If
you are a sysadmin or an architect or someone who just wants to learn and explore
various aspects of administering AWS services, then this book is the right choice for
you! Each chapter of this book is designed to help you understand the individual
services' concepts as well as gain hands-on experience by practicing simple and
easy to follow steps. The chapters also highlight some key best practices and
recommendations that you ought to keep in mind when working with AWS.
What this book covers
Chapter 1, Introducing Amazon Web Services, covers the introductory concepts and
general benefits of cloud computing along with an overview of Amazon Web
Services and its overall platform. The chapter also walks you through your first AWS
signup process, and finally ends with the configuration of the AWS CLI.
[ vii ]
Preface
Chapter 2, Security and Access Management, discusses the overall importance of
security and how you can achieve it using an AWS core service known as Identity
and Access Management (IAM). The chapter walks you through the steps required
to create and administer AWS users, groups, as well as how to create and assign
permissions and policies to them.
Chapter 3, Images and Instances, provides hands-on knowledge about EC2 instances
and images, and how you can create and manage them using both the AWS
Management Console as well as the AWS CLI.
Chapter 4, Security, Storage, Networking and Lots More!, discusses some of the key aspects
that you can leverage to provide added security for your applications and instances.
The chapter also provides an in-depth overview of EC2 instance storage as well as
networking options followed by some recommendations and best practices.
Chapter 5, Building Your Own Private Clouds Using Amazon VPC, introduces you to
the concept and benefits provided by AWS Virtual Private Cloud (VPC) service. The
chapter also provides an in-depth look at various VPC deployment strategies and
how you can best leverage them for your own environments.
Chapter 6, Monitoring Your AWS Infrastructure, covers AWS's primary monitoring
service, called as Amazon CloudWatch. In this chapter, you will learn how to
effectively create and manage alerts, loggings, and notifications for your EC2
instances, as well as your AWS environment.
Chapter 7, Manage Your Applications with Auto Scaling and Elastic Load Balancing,
discusses some of the key AWS services that you should leverage to create a
dynamically scalable and highly available web application.
Chapter 8, Database-as-a-Service Using Amazon RDS, provides an in-depth look at how
you can effectively design, create, manage, and monitor your RDS instances on AWS.
Chapter 9, Working with Simple Storage Service, provides practical knowledge and
design considerations that you should keep in mind when working with Amazon's
infinitely scalable and durable object storage known as Amazon S3.
Chapter 10, Extended AWS Services for Your Application, provides a brief overview
of add-on AWS services that you can leverage for enhancing your applications'
performance and availability.
[ viii ]
Preface
What you need for this book
To start using this book, you will need the following set of software installed on your
local desktop:
•
An SSH client such as Putty, a key generator such as PuttyGen, and a file
transferring tool such as WinSCP
•
Any modern web browser, preferably Mozilla Firefox
Who this book is for
This book is intended for any and all IT professionals who wish to learn and
implement AWS for their own environments and application hosting. Although
no prior experience or knowledge is required, it will be beneficial for you to have
basic Linux knowledge as well as some understanding of networking concepts and
server virtualization.
Conventions
In this book, you will find a number of text styles that distinguish between different
kinds of information. Here are some examples of these styles and an explanation of
their meaning.
Code words in text, database table names, folder names, filenames, file extensions,
pathnames, dummy URLs, user input, and Twitter handles are shown as follows:
"We can include other contexts through the use of the include directive."
A block of code is set as follows:
{
"Id": "Policy1448937262025",
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Stmt1448937260611",
"Effect": "Allow",
"Principal": "*",
}
}
[ ix ]
Preface
Any command-line input or output is written as follows:
CREATE TABLE doge
(
idint(11) NOT NULL auto_increment,
namevarchar(255),
description text,
New terms and important words are shown in bold. Words that you see on the
screen, for example, in menus or dialog boxes, appear in the text like this: "Next,
select the Launch DB Instance button to bring up the DB Launch Wizard:"
Warnings or important notes appear in a box like this.
Tips and tricks appear like this.
Reader feedback
Feedback from our readers is always welcome. Let us know what you think about
this book—what you liked or disliked. Reader feedback is important for us as it helps
us develop titles that you will really get the most out of.
To send us general feedback, simply e-mail feedback@packtpub.com, and mention
the book's title in the subject of your message.
If there is a topic that you have expertise in and you are interested in either writing
or contributing to a book, see our author guide at www.packtpub.com/authors.
Customer support
Now that you are the proud owner of a Packt book, we have a number of things to
help you to get the most from your purchase.
Downloading the example code
You can download the example code files from your account at http://www.
packtpub.com for all the Packt Publishing books you have purchased. If you
purchased this book elsewhere, you can visit http://www.packtpub.com/support
and register to have the files e-mailed directly to you.
[x]
Preface
Errata
Although we have taken every care to ensure the accuracy of our content, mistakes
do happen. If you find a mistake in one of our books—maybe a mistake in the text or
the code—we would be grateful if you could report this to us. By doing so, you can
save other readers from frustration and help us improve subsequent versions of this
book. If you find any errata, please report them by visiting http://www.packtpub.
com/submit-errata, selecting your book, clicking on the Errata Submission Form
link, and entering the details of your errata. Once your errata are verified, your
submission will be accepted and the errata will be uploaded to our website or added
to any list of existing errata under the Errata section of that title.
To view the previously submitted errata, go to https://www.packtpub.com/books/
content/support and enter the name of the book in the search field. The required
information will appear under the Errata section.
Piracy
Piracy of copyrighted material on the Internet is an ongoing problem across all
media. At Packt, we take the protection of our copyright and licenses very seriously.
If you come across any illegal copies of our works in any form on the Internet, please
provide us with the location address or website name immediately so that we can
pursue a remedy.
Please contact us at copyright@packtpub.com with a link to the suspected pirated
material.
We appreciate your help in protecting our authors and our ability to bring you
valuable content.
Questions
If you have a problem with any aspect of this book, you can contact us at
questions@packtpub.com, and we will do our best to address the problem.
[ xi ]
Introducing Amazon
Web Services
Being in the IT industry, we all have gone through the long and tedious process
of procuring new infrastructure for our data centers at one time or another. Let's
be honest, it is not a task for the faint-hearted At a minimum, a brand new server
can take weeks if not months to get delivered from its date of ordering, and this
is too heavily dependent on a lot of external factors which, most of the time, are
not in our control. Even if the new server comes in on time, there is the additional
burden of prepping the rack, clearing space for the new resources, cabling, cooling,
mounting, installation of software, configuration, and the list just keeps on going on
for another mile. Putting the server and storage on one side, the same can also apply
for networking, applications, software, and a whole lot of other things. But what if all
this was to change? What if tomorrow, you could simply jump start your business or
scale your application to thousands of servers, all with the simple click of a button?
Seems farfetched, doesn't it, but in reality, this is possible today with a little help
from something called as cloud computing.
Cloud computing has definitely evolved a lot over the years, and today it has become
almost a mainstream part of our lives. Everything from storing large amounts of
data, to having burst compute capacity at your fingertips and having enterprise
software applications available on demand any time anywhere are just some of the
key benefits that clouds provide today. At the helm of this new way of computing is
Amazon Web Services (AWS).
In this chapter, we are going to look at some of the key features and benefits
provided by cloud computing in general along with few interesting enterprise use
cases. Later on, you will learn a bit more about Amazon Web Services and its core
components, and finally have a look at how to sign up and get started with AWS.
[1]
Introducing Amazon Web Services
What is cloud computing?
Cloud computing has become one of the most discussed topics over the last few
years, but what does it actually mean? Why is it important for you and your
business? Let's take a quick look at what cloud computing actually is and how you as
an end user can benefit from it.
NIST defines cloud computing as a model for enabling ubiquitous, convenient,
on-demand network access to a shared pool of configurable computing resources
(for example, networks, servers, storage, applications, and services) that can be
rapidly provisioned and released with minimal management effort or service
provider interaction. Was that a bit confusing? Let's break it down to understand
it a little bit better:
•
On demand: Contrary to traditional IT data centers where requesting and
obtaining resources for your applications used to take weeks, the same
resources can be made available on an on-demand basis in the cloud, without
requiring any human interventions at all.
•
Network access: One of the key features of any cloud is that all of its resources
can be accessed over the Internet or a network in general. This makes it really
easy for the end user to access and leverage cloud from heterogeneous devices
such as laptops, workstations, mobile phones, and so on.
•
Shared pool: We must have all experienced the silo-based infrastructure
setup in our traditional IT data centers. Silos of resources are often created
by individual departments where each application gets overcommitted
resources, which are more than often wasted. In case of the cloud, IT
resources such as compute, storage, and network are pooled and abstracted
together from the end user. This pooling enables the cloud to dynamically
provide resources to its tenants as and when required without the end user
having any knowledge of where their application will actually reside in a
cluster or a rack.
•
Rapid provisioning: Rapid provisioning or elasticity as it is referred to
nowadays, is the ability of a cloud to scale its resources either horizontally
(scale out) or vertically (scale up) on an automated basis. This elasticity
provides end users with a lot of flexibility and control over their resources
and how they get consumed.
[2]
Chapter 1
Cloud computing features and benefits
Okay, so now we know what cloud computing really means, but why should you as
an end user use it? How are you and your organization going to benefit from it? Let's
look at some of the key benefits and features cloud computing has to offer:
•
Lower costs: Cloud computing does not have any upfront costs. Capital
expenditure is virtually zero as the entire necessary infrastructure and
resources are already made available and ready for use by the Cloud
provider. All the end user has to do is consume these resources and pay
only for what they use.
This also reduces the overall maintenance costs as well. Since the
organization now has less hardware to manage, it doesn't necessarily have to
bother about its server, network, and storage capacity or have a full time staff
to manage them.
•
Self-service: Cloud computing provides a simple, centralized, self-servicebased model that end users can use to interact with the cloud platform. The
interaction can be in terms of performing simple tasks, such as spinning up
new compute resources, adding more storage resources at runtime, or more
complex ones, such as scheduling resources and so on. This ensures that the
end user can leverage and consume any of the cloud's resources on an
on-demand basis.
•
Faster time to markets: Unlike traditional IT, compute resources can be
brought up in a cloud in a matter of minutes as compared to the weeks it
used to take. This provides the end users with the ability to deploy new
applications much faster, thus decreasing overall time to markets and
reducing management overheads and costs.
•
Scale as required: The best part of using the cloud is that your backend
resources can grow as your application grows. This means that you are never
lagging behind with your application's needs and demands, you are always
scaling with your application's needs.
Cloud computing use cases
With such features and benefits it is easy to see why there has been such a boom
in the overall adoption and utilization of the cloud. Let's take a quick look at some
interesting real-world use cases where your organization can leverage clouds:
•
Website hosting: Perhaps the most common of the use cases, you as an end
user can leverage cloud to build and host your websites with relative ease.
The cloud enables your website to scale up and down dynamically as per
its demands.
[3]
Introducing Amazon Web Services
•
Storage and sharing: The cloud offers virtually unlimited storage capacity
that can used to store and share anything, from documents, media, files, and
so on. Dropbox and Google Drive are classic examples of cloud being used as
a storage and sharing medium.
•
Disaster recovery (DR): This is a more upcoming use case with clouds as
more and more companies are now realizing that it is way easier and cost
efficient to host a disaster recovery environment on the cloud, rather than
hosting and managing a DR site of its own. Organizations can spin up
failover environments on the cloud in a matter of minutes, test the failover
and then shut down the entire stack. This helps save on the costs and also
reduces overall management overheads and failover time.
•
Dev/test: Dev and test are way easier to set up and run on clouds as the
entire development and test environments can be built up, tested, and torn
back down quickly as per requirements.
•
Short term projects/ advertising: Similar to the Dev/test scenario, the cloud
can also be leveraged to perform a variety of short-term projects/proof of
concepts. A classic example is the advertising campaigns hosted on the cloud
that are created for a very short duration of time, however, they need a global
presence to reach out to a wider set of audiences.
•
Big data analytics: Organizations leverage the cloud's scalability and ondemand infrastructure to capture and perform real-time analytics and data
mining on extremely large datasets (big data).
Introducing Amazon Web Services
Now, that you clearly understand what cloud computing is all about and what it can
do for you, let's get to know the main topic of this book—Amazon Web Services—a
little better.
Amazon Web Services or AWS is a comprehensive public cloud computing platform
that offers a variety of web-based products and services on an on-demand and
pay-per-use basis. AWS was earlier a part of the e-commerce giant Amazon.com,
and it wasn't until 2006 that AWS became a separate entity of its own. Today, AWS
operates globally with data centers located in USA, Europe, Brazil, Singapore, Japan,
China, and Australia. AWS provides a variety of mechanisms, using which the end
users can connect to and leverage its services, the most common form of interaction
being the web-based dashboard also called as AWS Management Console.
[4]
Chapter 1
So how does the whole thing work? Well, it is very easy to understand when you
compare the way AWS works with a power and utilities company. AWS offers its
customers certain services just as a power company would to its consumers. You
as an end user simply consume the electricity without having to worry about the
underlying necessities such as generator costs, cabling, and so on. At the end of the
month, all you get is a bill based on your electricity consumption and that's it! In a
similar way, AWS provides its products such as compute, storage, and networking
all as a service, and you only have to pay for the amount of service that you use. No
upfront costs or heavy investments whatsoever!
The other important thing worth mentioning here is that AWS allows organizations
to use their own operating systems, databases, and programming/architecting
models as well, without requiring any major re-engineering. This provides a lot of
flexibility and cost optimization to organizations as they get to operate and work
with platforms that they are familiar with. This, accompanied with AWS's massively
scalable and highly available infrastructure, ensures that your applications and data
remain secure and available for use no matter what.
AWS architecture and components
Before we begin with the actual signup process, it is important to take a look at some
of the key architecture and core components of services offered by AWS.
Regions and availability zones
We do know that AWS is spread out globally and has its presence across USA,
Europe, Asia, Australia, and so on. Each of these areas is termed as a region.
AWS currently has about 10 regions, each containing multiple data centers within
themselves. So what's with all these regions and why do they matter? In simple terms,
the resources that are geographically close to your organization are served much
faster! For example, an organization running predominantly from USA can leverage
the USA's regions to host their resources and gain access to them must faster.
For most of the AWS services that you use, you will be prompted to select a region
in which you want to deploy the service. Each region is completely isolated from the
other and runs independently as well.
AWS does not replicate resources across regions automatically.
It is up to the end user to set up the replication process.
[5]
Introducing Amazon Web Services
A list of regions and their corresponding codes is provided here for your reference.
The code is basically how AWS refers to its multiple regions:
Region
North America
South America
Europe
Asia
Name
Code
US East (N. Virginia)
us-east-1
US West (N. California)
us-west-1
US West (Oregon)
us-west-2
Sao Paulo
sa-east-1
EU (Frankfurt)
eu-central-1
EU (Ireland)
eu-west-1
Asia Pacific (Tokyo)
ap-northeast-1
Asia Pacific (Singapore)
ap-southeast-1
Asia Pacific (Sydney)
ap-southeast-2
Asia Pacific (Beijing)
cn-north-1
Each region is split up into one or more Availability Zones (AZs) and pronounced
as A-Zees. An A Z is an isolated location inside a region. AZs within a particular
region connect to other AZs via low-latency links. What do these AZs contain? Well,
ideally they are made up of one or more physical data centers that host AWS services
on them. Just as with regions, even AZs have corresponding codes to identify them,
generally they are regional names followed by a numerical value. For example, if you
select and use us-east-1, which is the North Virginia region, then it would have AZs
listed as us-east-1b, us-east-1c, us-east-1d, and so on:
[6]
Chapter 1
AZs are very important from a design and deployment point of view. Being data
centers, they are more than capable of failure and downtime, so it is always good
practice to distribute your resources across multiple AZs and design your applications
such that they can remain available even if one AZ goes completely offline.
An important point to note here is that AWS will always provide the services and
products to you as a customer; however, it is your duty to design and distribute your
applications so that they do not suffer any potential outages or failures.
RULE OF THUMB: Design for failure and nothing will fail! This is what we will
be sticking with for the remainder of this book as we go along the different AWS
services and products; so keep this in mind, always!
AWS provides a health dashboard of all its services running
across each of the regions. You can view the current status and
availability of each AWS service by visiting the following link:
http://status.aws.amazon.com/.
AWS platform overview
The AWS platform consists of a variety of services that you can use either in isolation
or in combination based on your organization's needs. This section will introduce
you to some of the most commonly used services as well as some newly launched
ones. To begin with, let's divide the services into three major classes:
•
Foundation services: This is generally the pillars on which the entire AWS
infrastructure commonly runs on, including the compute, storage, network,
and databases.
•
Application services: This class of services is usually more specific
and generally used in conjunction with the foundation services to add
functionality to your applications. For example, services such as distributed
computing, messaging and Media Transcoding, and other services fall under
this class.
•
Administration services: This class deals with all aspects of your AWS
environment, primarily with identity and access management tools,
monitoring your AWS services and resources, application deployments, and
automation.
[7]
Introducing Amazon Web Services
Let's take a quick look at some of the key services provided by AWS. However, do
note that this is not an exhaustive list:
We will discuss each of the foundation services.
Compute
This includes the following services:
•
Elastic Compute Cloud (EC2): When it comes to brute computation power
and scalability, there must be very few cloud providers out there in the
market that can match AWS's EC2 service. EC2 or Elastic Compute Cloud is
a web service that provides flexible, resizable, and secure compute capacity
on an on-demand basis. AWS started off with EC2 as one of its core services
way back in 2006 and has not stopped bringing changes and expanding
the platform ever since. The compute infrastructure runs on a virtualized
platform that predominantly consists of the open sourced Xen virtualization
engine. We will be exploring EC2 and its subsequent services in detail in the
coming chapters.
•
EC2 Container Service: A recently launched service, the EC2 Container
Service, allows you to easily run and manage docker containers across a
cluster of specially created EC2 instances.
•
Amazon Virtual Private Cloud (VPC): VPC enables you to create secure,
fully customizable, and isolated private clouds within AWS's premises. They
provide additional security and control than your standard EC2 along with
connectivity options to on premise data centers.
[8]
Chapter 1
Storage
This includes the following services:
•
Simple Storage Service (S3): S3 is a highly reliable, fault tolerant, and fully
redundant data storage infrastructure provided by AWS. It was one of
the first services offered by AWS way back in 2006, and it has not stopped
growing since. As of April 2013, an approximate 2 trillion objects have been
uploaded to S3, and these numbers are growing exponentially each year.
•
Elastic Block Storage (EBS): EBS is a raw block device that can be attached
to your compute EC2 instances to provide them with persistent storage
capabilities.
•
Amazon Glacier: It is a similar service offering to S3. Amazon Glacier offers
long-term data storage, archival, and backup services to its customers.
•
Amazon Elastic File System: Yet another very recent service offering
introduced by AWS, Elastic File System (EFS) provides scalable and
high-performance storage to EC2 compute instances in the form of an
NFS filesystem.
Databases
This includes the following services:
•
Amazon Relational Database Service (RDS): RDS provides a scalable,
high-performance relational database system such as MySQL, SQL Server,
PostgreSQL, and Oracle in the cloud. RDS is a completely managed solution
provided by AWS where all the database heavy lifting work is taken care of
by AWS.
•
Amazon DynamoDB: DynamoDB is a highly scalable NoSQL database as a
service offering provided by AWS.
•
Amazon Redshift: Amazon Redshift is a data warehouse service that is
designed to handle and scale to petabytes of data. It is primarily used by
organizations to perform real-time analytics and data mining.
Networking
This includes the following services:
•
Elastic Load Balancer (ELB): ELB is a dynamic load balancing service
provided by AWS used to distribute traffic among EC2 instances. You
will be learning about ELB a bit more in detail in subsequent chapters.
[9]
Introducing Amazon Web Services
•
Amazon Route 53: Route 53 is a highly scalable and available DNS web
service provided by AWS. Rather than configuring DNS names and settings
for your domain provider, you can leverage Route 53 to do the heavy lifting
work for you.
These are just few of the most commonly used AWS foundational services that we
listed out here. There are a lot more services and products that you can leverage
to add functionality or use to manage your applications. A few of these important
services are briefly described in the next section.
Distributed computing and analytics
This includes the following services:
•
Amazon Elastic MapReduce (EMR): As the name suggests, this service
provides users with a highly scalable and easy way to distribute and
process large amounts of data using Apache's Hadoop. You can integrate
the functionalities of EMR with Amazon S3 to store your large data or with
Amazon DynamoDB as well.
•
Amazon Redshift: This is a massive data warehouse that users can use to
store, analyze, and query petabytes of data.
Content distribution and delivery
Amazon CloudFront is basically a content delivery web service that can be used to
distribute various types of content, such as media, files, and so on, with high data
transfer speeds to end users globally. You can use CloudFront in conjunction with
other AWS services such as EC2 and ELB as well.
Workflow and messaging
This includes the following services:
•
Amazon Simple Notification Service (SNS): SNS is a simple, fully managed
push messaging service provided by AWS. You can use it to push your
messages to mobile devices (SMS service) and even to other AWS services as
API calls to trigger or notify certain activities.
•
Amazon Simple Email Service (SES): As the name suggests, SES is used
to send bulk e-mails to various recipients. These e-mails can be anything,
from simple notifications to transactions messages, and so on. Think of it
as a really large mail server that can scale as per your requirements and is
completely managed by AWS! Awesome, isn't it!
[ 10 ]
Chapter 1
Monitoring
Amazon CloudWatch is a monitoring tool provided by AWS that you can use to
monitor any and all aspects of your AWS environment, from EC2 instances to your
RDS services to the load on your ELBs, and so on. You can even create your own
metrics, set thresholds, create alarms, and a whole lot of other activities as well.
Identity and access management
AWS provides a rich set of tools and services to secure and control your
infrastructure on the cloud. The most important and commonly used service for this
is identity and access management (IAM). Using IAM, you can, as an organizational
administrator, create and manage users, assign them specific roles and permissions,
and manage active directory federations as well. We will be using a lot of IAM in the
next chapter, which covers this topic in greater depth.
Getting started with AWS
So far, you have learned a lot about AWS, its architecture, and core components.
Now, let's get started with the fun part—the signup process.
For all first time users, signing up for AWS is a very simple and straightforward
process. We will go through this shortly, but first let's take a quick look at something
called as a Free Tier! Yes, you heard it right… FREE!
So, AWS basically offers usage of certain of its products at no charge for a period of
12 months from the date of the actual signup. A brief list of a few products along with
their description is listed here for your reference. Note that some of the description
text may not make much sense now, but that's ok as this is just for your reference, and
we will be bringing this up from time to time as we progress through the book.
AWS Product
Amazon EC2
What's free?
750 hours per month of Linux micro instance usage
750 hours per month of Windows micro instance usage
5 GB of standard storage
Amazon S3
20,000 get requests
2,000 put requests
750 Hours of Amazon RDS Single-AZ micro instance usage
Amazon RDS
20 GB of DB Storage: any combination of general purpose (SSD)
or magnetic
20 GB for backups
10,000,000 I/Os
[ 11 ]
Introducing Amazon Web Services
AWS Product
Amazon ELB
What's free?
750 hours per month
15 GB of data processing
For a complete insight into the free tier usage, check
http://aws.amazon.com/free/.
Awesome! So when we have free stuff for us right from the word go, why wait? Let's
sign up for AWS. To begin with, launch your favorite web browser and type in the
following URL in the address bar: http://aws.amazon.com/.
You should see the AWS landing page similar to one shown here. Here, select
either the Create an AWS Account option or the Create a Free Account option
to get started:
The next screen will help you with the initial signup process. Provide a suitable e-mail
address or your contact number in the E-mail or mobile number field. Select the I am
a new user option and select the Sign in using our secure server button to proceed:
[ 12 ]
Chapter 1
You can alternatively sign in using your Amazon.com
credentials as well; however, its best to use separate
credentials for working with AWS.
The next couple of screens will be used to provide your basic details along with the
billing information. In the Login Credentials page, enter your Name, your E-mail
address along with a suitable Password. This password will be used by you to login
to the AWS Management Console, so ideally provide a strong password here. Click
on Create account when done.
The next screen is the Contact Information page. Provide your Full Name, Company
Name, Country, Address, City, Postal Code, and Phone Number as requested.
Check the Amazon Internet Service Pvt. Ltd. Customer Agreement checkbox and
select the Create Account and continue options.
[ 13 ]
Introducing Amazon Web Services
Enter a suitable Cardholder's Name and your Credit/Debit Card Number in the
Payment Information page as shown:
The last part of the signup process is the Identity Verification process where you
will receive an automated call from AWS as a part of the verification process. You
will have to enter the displayed four digit PIN code on your telephone's keypad
during the call. Once the verification is completed, you can click on the Continue to
select your Support Plan tab:
[ 14 ]
Chapter 1
The final step in the signup process involves the selection of the Support Plan. AWS
provides four support options to customers, each having their own SLAs and costs
associated with it. Here is a quick look at the support plans provided by AWS:
•
Basic Support: As the name suggests, this is the most basic level of support
provided by AWS. This support level provides you with access to the AWS
community forums. You can additionally contact customer services for any
queries related to your account and bill generation.
The Basic Support plan is free of charge and all
customers are entitled to it.
•
Developer Support: This is a paid support service ($49 per month). You can
create and raise tickets for your support case, which is generally answered
within 12 working hours.
•
Business Support: This is a paid support service as well and is generally
meant for enterprise-level customers running production workloads
on AWS. The SLAs for this support are much higher as a case has to be
answered within an hour from its creation. Additional support is provided
24/7, 365 days a year via phone and chat.
•
Enterprise Support: A paid support service with the highest SLA available
(15 minutes); these cases are generally handled by a separate team at AWS
called the Technical Account Manager (TAM) who are subject matter
experts in their own fields.
In our case, we opted to go with Basic Support for the time being. You can change
the support levels later on as well according to your needs. Click on Continue
to complete the signup process. You should receive a couple of e-mails on your
supplied e-mail address as well. These are introductory e-mails that will provide
you with important links such as how to get started with AWS, billing page, account
information, and so on.
With these steps completed, you are now ready to sign in to the AWS
Management Console!
Introducing the AWS Management Console
So here we are, all ready to get started with the AWS Management Console! This is
the most commonly used method to access, manage, and work with AWS services.
We shall be looking more closely at the different AWS access mechanisms in the next
chapter; however for now, let's quickly look at what the AWS Management Console
is all about.
[ 15 ]
Introducing Amazon Web Services
First off, sign in to the Management Console by launching your favorite browser and
typing in http://aws.amazon.com. Click on the Sign in to the Console option and
provide your Email Address and password as set during our signup process. Once
you sign in, you will be welcomed to the AWS Management Console main landing
page as shown here. Wow! That's a lot of products and services, right? The products
are classified into their main classes such as compute, storage and content delivery,
administration and security, and so on so forth. Take a moment and just browse
through the dashboard. Get a good feel for it.
Navigating through the dashboard is also pretty straightforward. Let's look at the
top navigation bar first. To the right-hand side you should be able to see three dropdown menus. The first should display your name as an end user. This option consists
of three submenus that will help you with configuring your account details, security
credentials, and billing management. The next tab lists the Region from where
you will currently be operating. In our case, we have been placed in the US West
(Oregon) region. Remember you can change these regions as and when you require,
so feel free to change as per your current global presence:
[ 16 ]
Chapter 1
The US East (North Virginia) region is the cheapest region in
AWS as it was one of the first regions to get set up and started.
The final tab in the list is the Support tab, and you can use it to login to the Support
Center, AWS Forums, and view the latest set of AWS Documentation as well.
Moving to the left-hand side of the tool bar, you will see four main icons listed there.
Among these is a Home Screen icon, which when clicked on will bring you back to
the AWS dashboard screen irrespective of where you currently are.
The next drop-down option in the list is named as AWS, but what it really contains
is called as Resource Groups. These are a collection of AWS resources that can be
organized and viewed as per your requirements. Think of these resource groups as a
customized console where you as an end user can view all your required information
about various AWS services in a single pane. How do resource groups work? Don't
worry. We will be looking into this in more detail in the upcoming chapter.
Adjoining the Resource Groups is the Services tab, which lists the AWS services
according to their class. It also has a history option that can be used to list and view
your recently used AWS services.
The final tab is the Edit tab. This tab is used to customize your toolbar by filling it
with those AWS services that you use frequently, kind of like a quick access bar. To
add a particular AWS service to the toolbar, simply select the AWS service, drag and
place it on the toolbar:
You can add multiple services as you see fit, and arrange them according to your
needs as well. To save the changes to the toolbar, simply select the Edit option once
again. You should see your AWS services listed out on the toolbar now. Feel free to
dig around and check out the various options under each menu.
[ 17 ]
Introducing Amazon Web Services
Getting started with AWS CLI
Now that you have a hang of the AWS Management Console, this would be a good
time to take a quick look at the AWS CLI as well. Yes, you heard it right, apart from
the standard web user interface, AWS provides a host of other mechanisms as well
to help you gain access and use the various AWS resources. But why use a CLI in the
first place? Isn't the AWS Management Console more than enough? Well, no. CLIs are
more than just simple access and management tools. Using CLIs, you can automate
the deployment and management of your AWS services using simple code and script,
much like how you would use bash and shell scripting. This provides you with a lot
of flexibility and customizability that a standard GUI simply won't provide!
The AWS CLI can be either installed on a Windows or a Linux machine. In case of
Windows, AWS provides an easy-to-use installer, which can be downloaded directly
from the AWS site. Once downloaded, all you need to do is run the installer, and
voila, your Windows server should have the CLI installed and ready for use. But I'm
not a Windows guy, so we will be walking you through the installation procedure on
a standard Linux system.
The 64-bit AWS CLI installer for Windows can be downloaded
from https://s3.amazonaws.com/aws-cli/AWSCLI64.
msi. The 32-bit installer can be downloaded from https://
s3.amazonaws.com/aws-cli/AWSCLI32.msi.
In this case, we will be installing the AWS CLI on a CentOS 6.5 64-bit OS. The Linux
distribution can be anything, from a Debian such as Ubuntu to a RedHat system; so
long as it has Python installed and running off the latest version.
Python versions supported are Python 2 version 2.6.5 and
above or Python 3 version 3.3 and above.
You will also need sudo or root privileges to install and execute the commands, so
make sure you have an appropriate user already created on your Linux system.
The installation of the CLI involves two major steps; the first involves the installation
of Python setuptools, which is a prerequisite of installing Python's pip.
Run the following commands from your Linux terminal:
1. Download the setuptools tar file from the Python source repo:
wget https://pypi.python.org/packages/source/s/setuptools/
setuptools-7.0.tar.gz
[ 18 ]
Chapter 1
2. Next, untar the setuptools installer using the tar command:
tar xvf setuptools-7.0.tar.gz
3. Once the contents of the tar file are extracted, change the directory to the
setuptools directory:
cd setuptools-7.0
4. Finally, run the setup.py script to install the setuptools package:
python setup.py install
The following is the screenshot of preceding commands of the install process:
This completes the first part of the install process. The next process is very simple as
well. We now install the Python pip package. Python pip is generally recommended
when installing Python packages.
Run the following commands from your Linux terminal to install the Python pip
package:
1. Download the Python pip installer script from Python's repo:
wget https://bootstrap.pypa.io/get-pip.py
2. Install the pip package:
python get-pip.py
3. Once pip is installed, you can now easily install the AWS CLI by executing
the following command:
pip install awscli
[ 19 ]
Introducing Amazon Web Services
Refer to the following screenshot showing the output of the
installation process:
4. Simple, wasn't it! You can test your AWS CLI by executing few simple
commands, for example, check the AWS CLI version using the following
command:
aws –version
That's just for starters! There is a whole lot more that you can achieve with the AWS
CLI, and we will make sure to utilize it in each of our chapters, just to get a good
feel for it.
Plan of attack!
For the purpose of this book, let's assume a simple use case in which a hypothetical
company called as All About Dogs (not the best of names I could find) wants to host
and manage their e-commerce website on the cloud. As a part of the hosting, the
company would like to have the following feature set provided to them by the
cloud provider:
•
High availability and fault tolerance
•
On-demand scalability
•
Security
•
Reduced management overheads and costs
Here is a simple, traditional architecture of the proposed website, which basically is
a two-tier application primarily consisting of web servers and a backend database,
something most IT admins will be familiar with, right? Let's look at the following
figure of tradition web service architecture:
[ 20 ]
Chapter 1
This traditional architecture has obvious drawbacks, such as poor scalability,
little or no fault tolerance, more management overheads, and so on. Our goal is to
leverage AWS's core services and make this obsolete architecture better! Each of the
subsequent chapters will show you how to work with and administer these core
AWS services keeping our use case in mind. By the end of this book, you should
have a fully scalable, resilient, and secure website hosted on the AWS cloud with a
design similar to this! Here is the AWS architecture:
[ 21 ]
Introducing Amazon Web Services
Awesome, isn't it! Believe it or not, this is a bare minimum website hosting
architecture on AWS. There are still many enhancements and AWS services that you
could incorporate here, but I like to keep things nice and clean, so this is what we
will stick with for the remainder of this book.
Here's a list of AWS products that we will be incorporating and primarily learning
about throughout the book, apart from few other services:
•
Identity and Access Management (IAM)
•
Elastic Compute Cloud (EC2)
•
Elastic Container Service (ECS)
•
Elastic Block Storage (EBS)
•
Amazon Virtual Private Cloud (VPC)
•
Amazon Cloudwatch
•
Autoscaling and Elastic Load Balancing (ELB)
•
Amazon Relational Database Service (RDS)
•
Amazon Simple Storage Service (S3)
Summary
Let's quickly recap what all we accomplished so far in this chapter. To begin with,
you learned a bit about what cloud computing is all about and saw a few features and
benefits that it has to offer. Next, we drove straight into our core topic, that is, Amazon
Web Services. You learned what Amazon Web Services is all about by understanding
its architecture and core service offering. We then saw how easy and straightforward
it is to sign up for AWS along with a brief walkthrough of the AWS Management
Console. Towards the end, you also learned the importance of a command line
interface and saw how to install the AWS CLI on a simple Linux server.
In the next chapter, you will learn a bit more about the CLI and see how to leverage
it to manage and work with the AWS services. We will also be looking at few easy
to use access management tools and techniques to safeguard and secure your AWS
environment, so stick around! We are just getting started!
[ 22 ]
Security and Access
Management
In the previous chapter, we accomplished a lot of things. To begin with, we got a
better understanding of what cloud computing actually is all about and how you
as an end user can benefit by leveraging it. Later on in the chapter, you had a brief
overview of AWS, its architecture, and its core service offerings and also learned
how to sign up for it.
In this chapter, you are going to learn a bit more about how to secure and provide
users access to your AWS infrastructure and services. The chapter will first talk
about security in general, and how AWS provides some of the best security there
is. Later on, we will look at an AWS core service called as Identity and Access
Management (IAM) and find out how to create, manage, and administer users
using it.
Security and clouds
Security is a core requirement for any application whether it is hosted on an
on-premise data center or a cloud such as AWS. It is a fundamental service that
protects your applications and data from a variety of cyber-attacks, security
breaches, accidental or deliberate data deletions, theft, and much more.
Most modern cloud providers offer security in a very similar way to traditional
on-premise data centers with the same amount of control and compliance. The
only difference is that in a traditional data center, you would have to deal with the
complexities and costs of securing the hardware, whereas on the cloud, this task is
performed by the cloud provider itself. This difference not only saves on overhead
costs that every organization has to bear, but also reduces the time and effort it takes
to monitor and protect all those resources.
[ 23 ]
Security and Access Management
Is AWS really secure
So the obvious question lingering in your mind right now must be, ok, we signed up
for AWS and now are going to run our applications and store all our data on it, but is
all that really secure? Is it safe to use AWS? The answer is a big yes!
Let's take a quick look at some of the different layers of security that AWS uses to
safeguard and protect its resources:
•
Physical data center security: The AWS infrastructure, which includes the
data centers, the physical hardware, and networks, is designed and managed
according to security best practices and compliance guides. The data centers
themselves are housed at non-disclosed locations and entry to them is strictly
controlled, managed, logged, and audited on a regular basis.
•
Virtualization and OS security: AWS regularly patches and updates
virtualization and operating systems against a variety of attacks such as
DDoS, and so on.
•
Regulatory compliances: The AWS infrastructure is certified against security
and data protection in accordance with various industry and government
requirements. Here are a few compliances that AWS is certified against:
°°
SOC 1 (formerly SAS 70 Type II), SOC 2, and SOC 3
°°
FISMA, DIACAP, and FedRAMP
°°
ISO 27001
°°
HIPAA
To read the complete list, visit the AWSrisk and compliance whitepaper at
http://aws.amazon.com/security/.
Shared responsibility model
As you must have noticed by now, AWS provides a lot of security and protection for
its hardware and its virtualization layers by providing patches, updates, performing
regular audits and so on, but what about your applications and data? Who protects
that? That's where AWS introduced the shared responsibility model.
According to this model, AWS provides secure infrastructure, services, and building
blocks required while you, as an end user, are responsible for securing your
operating system's data and applications. Think of it as a joint operation where you
and AWS together ensure the security objectives are met.
[ 24 ]
Chapter 2
Here is a simple depiction showing the shared responsibility model for AWS's
infrastructure services:
Image Source: AWS security best practices whitepaper.
Remember, that this is a basic shared responsibility model, which is only valid for
AWS's core infrastructure services such as EC2 and Amazon VPC. The model tends
to change as you start using more abstracted services such as Amazon S3, Amazon
DynamoDB, Amazon SES, and so on. Why? Well that's simple! The more abstracted
services you use, the less control you have over them. For example, if you are using
SES as a bulk e-mail-sending tool, you don't have to set up the infrastructure, the
operating systems, and the platforms on which the SES service works. It's already
done for you. So as an end user, all you need to worry about from the security point
of view is how is your data going to be protected at rest or in transit, whether you
are going to use encryption/decryption techniques, and so on; this is your part of the
responsibility now.
AWS provides a few services and products that are specifically designed to help
you secure your infrastructure on the cloud, such as IAM, AWS Multi-Factor
Authentication (AWS MFA), AWSCloudTrail, and much more. In the next section,
we will look into IAM and see how we can leverage it for ourselves.
Identity and Access Management
AWS Identity and Access Management or IAM is a web service that provides secured
access control mechanisms for all AWS services. You can use IAM to create users and
groups, assigning users specific permissions and policies, and a lot more. The best part
of all this is that IAM is completely FREE. Yup! Not a penny is required to use it.
[ 25 ]
Security and Access Management
Let's quickly look at some interesting IAM features in order to understand it a
bit better:
•
Shared access to a single account: With the sign in process completed, you
currently are the sole owner and user of your AWS account. But what if
you wanted to give access to few other users from within your organization
to this account? You cannot just provide them with your username and
password, right? Neither will you go and create a separate account for each
user, as it is too tedious and not good practice. However, with IAM, you can
create and provide users with shared access to your single account with real
ease. It is something we will be looking into shortly.
•
Multi-factor authentication: IAM allows you to provide two-factor
authentications to users for added security. This means that now, along with
your password, you will also have to provide a secret key/pin from a special
hardware device, such as a hard token, or even from software apps such as
Google Authenticator.
•
Integration with other AWS products: IAM integrates with almost all AWS
products and services and can be used to provide granular access rights and
permissions to each service as required.
•
Identity federation: Do you have an on-premise active directory already
that has users and groups created? Not a problem, as IAM can be integrated
with an on-premise AD to provide access to your AWS account using a few
simple steps.
•
Global reach: Remember regions and availability zones from Chapter 1,
Introducing Amazon Web Services? Well, IAM is one of the few AWS core
services that spans globally. This means that users that are created using IAM
can access and consume any AWS service from any geographic region! Neat,
right?
•
Access mechanisms: IAM can be accessed using a variety of different tools,
the most common and frequently used being the AWS Management Console.
Apart from this, IAM can also be accessed via the AWS CLI, via SDKs that
support different platforms and programming languages such as Java,
.NET, Python, Ruby, and so on, and programmatically via a secured HTTPS
API as well.
[ 26 ]
Chapter 2
Business use case scenario
Awesome! We have seen what IAM is along with its impressive features list, so
now, let's put it to some good use! In Chapter 1, Introducing Amazon Web Services, we
briefly discussed our use case scenario about hosting a website for an organization
called All-About-Dogs. In this section, let's go ahead and define some users for this
organization along with their potential roles:
In this example, Jason is the manager of All-About-Dogs, and he is responsible for
overseeing the entire operations of the organization. Jason goes ahead and hires
Dave and Chen who will act as leads for the development and testing departments,
respectively. Dave and Chen can then have multiple developers and testers within
their teams as they see fit, a typical and simple hierarchy that most of us are familiar
with and can relate to.
Getting started with the IAM Console
AWS IAM can be accessed using the AWS Management Console as well as a host
of other CLIs, tools, and SDKs. In this section, you are going to learn how to use
the AWS Management Console to create users and assign those users to individual
groups and policies:
1. To begin with, sign in to the AWS Management Console using
https://console.aws.amazon.com/.
[ 27 ]
Security and Access Management
2. Now, there are a lot of different ways to access the AWS IAM service; the
easiest is to locate the service under the Administration & Security section
as shown in the following screenshot. Selecting the Identity & Access
Management option will launch the IAM console.
Welcome to your first AWS core service! Take some time to visually inspect each
of the elements of the IAM dashboard. The dashboard can be basically split up into
two sections, the navigation pane to the left, which contains all the individual links
that will help you create your users and groups, and the main dashboard to the right
where you can view your IAM Resources and various other security statuses:
The first thing that you will notice here under the Security Status field is an option to
Delete your root access keys. Now why would you want to do something like that
even before you start creating users? And what are root access keys?
[ 28 ]
Chapter 2
Well, to begin with, in Chapter 1, Introducing Amazon Web Services, you signed in to
AWS using your e-mail ID and password, right? Well, that is the root account that
you just created. The root account, as the name suggests, has root-level access to all
AWS services, including your billing account. So, as a good practice, AWS highly
recommends that you do not use the root account unless you absolutely need to,
and more importantly, you do not create and root keys as well. Root keys simply
consist of an access ID and a secret key that can be used to programmatically access
any AWS service. Each user that you create gets its own set of keys, out of which, the
secret key has to be protected and kept under lock and key at all costs.
Coming back to the IAM console, let's take a quick look at some of the tasks that you
can perform using it. The first thing you will notice is a big, clunky-looking URL that
consists of some long numbers. Well, this is the URL that your new IAM users will
be using once they are created to log in to the AWS Management Console.
The URL basically links to a sign-in page that is created automatically when you
sign into the IAM service. But let's face it, it's not a simple URL and anyone would
have a tough time remembering it as well. You can choose to customize the URL by
providing an alias to it.
The IAM URL contains the following format: https://.signin.aws.amazon.com/console/.
Select the Customize option adjoining the IAM sign in link to get started. You
should get a Create Account Alias dialog box. Provide a suitable alias name for your
account and click on Yes, Create when done:
[ 29 ]
Security and Access Management
Voila! Your IAM user's sign-in link is now ready, but before you go ahead and use
it, first you have to create some users and groups who will access it:
Creating users and groups
With the basics out of the way, let's get to the main part of this chapter, that is,
creating and working with users and groups.
Users, as the name suggests, are your everyday typical end users who will be
interacting with the AWS products and services. Each user is provided with a unique
password and a username so they can log in to the AWS Management Console.
Along with the basic set of credentials, the users can also enhance their security by
leveraging MFA. As discussed earlier, MFA provides a uniquely generated pin or
code that is generated on a special hardware device called as a hard token. You can
use this MFA pin or code along with your secure credentials to log in to the AWS
Management Console.
Users are also provided with a set of access keys. These keys consist of an access key
ID and a secret key, both of which can be used to log in to AWS programmatically.
When the users are first created in IAM, they do not have a password or any access
keys generated for them. This is your task as an AWS administrator, and you must
make sure that each user has their own set of keys and passwords generated.
There are a lot of ways that you can start creating users. From the IAM dashboard,
select the Manage Users option listed under the Create individual IAM users dialog
box, as shown:
[ 30 ]
Chapter 2
This will bring up the users console, using which we will create our very first IAM
users. Select the Create New Users option to get started:
In the next page, type in the IAM usernames as required. You can enter up to five
names at a time. You can optionally choose to create and generate access keys for
each of the users that you create. Select the Generate an access key for each user
option as shown and click on Create to proceed:
[ 31 ]
Security and Access Management
Select the Download Credentials option to save the user's access IDs and secret keys.
This will download a CSV file on to your desktop, which has to be saved in a very
secure location. It is very important that you save the keys, as this is the last time
you will have access to it. Select Close after you have downloaded your credential
keys successfully:
Access keys are unique to each user and should not be
shared with anyone under any circumstances. Save them in
a secure place.
But wait! You are not done yet! You still need to assign your users their passwords.
To do this, from the users console page, select the individual user's checkbox, click
on the User Actions drop-down menu and select the Manage Password option as
shown. You can use this drop-down menu to manage the user's access keys, signing
certificates, MFA devices, and so on:
In the Manage Password page, you can either choose to Assign an auto-generated
password for your users or provide a custom, temporary password, which the user
can change at the first sign in attempt. In our case, we provided our user with a
strong password that does not need to be changed at the first login attempt. Once
the password is entered, click on Apply to save the changes:
[ 32 ]
Chapter 2
Follow the same process for the rest of your users as well. Make sure you provide them
with strong passwords that contain at least one upper case letter, one special character,
and some numerical values as well. You can additionally set password policies on
your entire account by selecting the Account Settings option from the IAM console's
navigation pane. Using the Password Policy page, select the security options that you
wish to enable for your account's IAM user passwords. Remember to select Apply
password policy to save and enforce the new password policy settings:
With your users created, the next logical entity to create is the group. A group is a
collection of IAM users that has a particular set of permissions assigned to it. For
example, a set of users who perform administrative tasks can be clubbed under a
common group called as administrators, and so on and so forth. In this section, we
will create an administrative group for our use case and later assign a user to it. So,
let's get started!
[ 33 ]
Security and Access Management
First up, from the IAM console, select the Groups option from the navigation pane.
This will bring up the groups console using which you can create and administer
groups for your AWS account.
Select the Create New Group option to get started. Provide a suitable name for your
administrative group; in this case, we provided the name Admin-All-About-Dogs.
The Group Name can be anything, but it's advised to keep it meaningful. Click on
Next Step to continue:
Next up, we assign permissions to the group using one or two policies. A policy is
a document that lists one or more permissions. You can attach policies to virtually
anything in AWS, from users and groups to individual AWS resources as well.
To attach a policy to a group, from the Attach Policy page, use the Filter menu
and the search box to find suitable policies. In this case, we want this group to have
full administrative privileges, and hence we are searching for an Administrator
Access policy. In the list of policies, select the appropriate policy and click on Next
Step to continue:
[ 34 ]
Chapter 2
You can use and attach two policies per group.
In the Review page, review Group Name and the policies that are attached to the
group. You can optionally choose to Edit Group Name or Edit Polices as per your
requirements here. Once done, click on Create Group to proceed with the group's
creation. Similarly, you can create groups for various other departments within your
own organization, such as Developers group, Testers group, and so on:
To add users to any particular group, simply select the group's checkbox and from
the Group Actions drop-down menu select the Add Users to Group option. This
will open up a separate page where you can select your previously created users.
You can add one or more users to a group as you see fit. A particular user can also be
a part of two or more groups at the same time; however, this is not a recommended
practice and should be avoided unless absolutely required. Once the users are
selected, click on the Add Users option to complete the process. With this stage
completed, you have now successfully created and set up users and groups within
your own organization. Now, wasn't that easy! Just remember to use the IAM users
sign in link to sign in to the AWS Management Console from now on; the rest of the
login process remains the same.
Understanding permissions and policies
To begin with, let's talk a bit about permissions first. We already used them during
the creation of our users, but what actually are permissions and what are their uses?
[ 35 ]
Security and Access Management
Permissions provide you with access to and control of various AWS resources. They
are also responsible for controlling actions that you can perform on the resources.
By default, when you create an IAM user, the user starts as a blank slate, no keys,
and no permissions at all. It is your responsibility to assign the users keys and the
necessary permissions, which can range from simple novice tasks such as listing
resources to creating, updating, and deleting resources, and so on.
Permissions can be classified into two main classes, each briefly explained here:
•
User-based permissions: As the name implies, these permissions are attached
to IAM users and allow them to perform some action over an AWS resource.
User-based permissions can be applied to groups as well. User-based
permissions branch out into two further categories called as inline policies
and managed policies, both of which we will be discussing shortly. Basically,
an inline policy is a policy that is created and managed completely by you,
whereas a managed policy is created and managed more by AWS itself.
•
Resource-based permissions: These are a special class of permissions that
allow you to specify which user has what specific level of access to a particular
AWS resource along with what actions they can perform on it. There are
a handful of AWS services and resources that support such permissions,
including S3 buckets, SNS topics, Amazon Glacier vaults, and so on. Unlike
user-based permissions, these categories of permissions are only inline-based.
This means that they are completely managed and created by you.
Not clear, eh? Not a problem. Let's walk through this simple example to get a better
feel for it. In our use case, we have users created called Jason, Dave, and Chen. Each
of these users can be specified a set of user permissions, for example, Dave can have
the ability to list, read, and write on Amazon EC2 service, whereas Chen can only have
read permissions on the EC2 instances. Jason, on the other hand, being a manager, can
have all admin rights and can perform all actions on any of the AWS services.
Resource-based permissions, on the other hand, are allocated directly to resources, so
in this case assume that an S3 bucket (a bucket is a like a storage folder where you dump
objects) has been allocated permissions to allow both Dave and Chen read-write
access, whereas Jason can read, write, and list objects stored in the bucket:
[ 36 ]
Chapter 2
With permissions covered, let's look at policies in a bit more detail. We briefly used
policies during the IAM group creation process, and there you learned that a policy is
nothing but a collection of permissions put together in a JSON-formatted document.
Policies can contain user-based or resource-based permissions. A single permission
forms a statement in a policy, and a single policy can contain multiple statements.
Let's look at a simple policy for our reference:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ec2:DescribeInstances",
"ec2:DescribeImages"
],
"Resource": "arn:aws:iam::012345678910:user/Chen"
}
]
}
[ 37 ]
Security and Access Management
As you can see, this policy will basically allow the user Chen to only list the EC2
images and instances, in short, simple read-only access. Let's take a closer look at
each of this policy's elements:
•
Version: The version specifies the policy's language. As of date, the current
•
Statement: The statement is the main starting point for your policy.
Unlike the version field, the statement field is mandatory. The statement
element is always enclosed within square brackets [] and can contain other
version of the policy language is 2012-10-17. Although not required, it is a
good practice to include the version field in your policy statements.
individual statements within itself. Each individual statement should be
enclosed by a set of curly brackets {} as shown.
•
Effect: Another mandatory statement, the Effect element specifies whether
the following Action statement should result in Allow or Deny. By default,
the effect is always set to deny access to AWS resources. This ensures that
you set explicit permissions for your IAM users when declaring policies.
•
Action: The Action element describes what specific actions are required
•
Resource: The final element required for our policy is the Resource
element. The Resource element is used to specify the object or service that
to be either allowed or denied. Each action statement consists of two main
parts, a value that identifies the particular AWS service such as EC2, S3, IAM,
and so on, followed by the action value, such as DescribeInstances and
DescribeImages.
the particular set of statements will cover. Resource names are specified
by something called as an Amazon Resource Name (ARN). ARNs are a
crucial part of IAM and are used to uniquely identify AWS resources. In our
reference policy, the ARN uniquely identifies the user Chen from our demo
AWS account ID (012345678910) as the resource, which will obtain the
necessary permissions based on the actions element.
These are just the most commonly used set of
elements that you can use to get started with your
sets of policies. There are a lot more additional sets
of elements that comprise a policy. Read more about
them at http://docs.aws.amazon.com/IAM/
latest/UserGuide/AccessPolicyLanguage_
ElementDescriptions.html.
Now that we have a basic understanding of what a policy is and what it comprises,
let's take a quick look at how you can create and assign them to your IAM users and
groups using the AWS Management Console.
[ 38 ]
Chapter 2
Creating and assigning policies
To create and assign policies, from the AWS Management Console, select the
Identity and Access Management option as done before. Next up, select the
Policies option from the IAM console's navigation plane. You should see the policy
page as shown.
Using this page, you can filter and list existing policies (both inline and manage policies)
using the Filter and Search options. You can even create, update, and delete existing
policies; attach and detach policies from users and groups using this page. For
starters, let's go ahead and create a simple policy for our IAM users. Select the Create
Policy option as shown:
This will pop up the Create Policy wizard. Here, you will be provided with three
options, briefly explained as follows:
•
Copy an AWS Managed Policy: This option will list all the policies that are
designed, created, and managed by AWS itself. This is by far the simplest
way to get started with policies.
•
Policy Generator: This is a neat tool that will help you build your very own
customized policy. The tool includes drop-down options using which you
can select various AWS services and their associated actions and effects. The
tool even has built in policy validation that verifies whether your policy is
syntactically correct or not before deploying it.
•
Create your Own Policy: Using this feature, you can actually write your own
policy or copy and paste an existing policy here. This feature too comes with
a policy validator that verifies the syntax and validity of your custom policy
before deployment.
[ 39 ]
Security and Access Management
For this scenario, let's go ahead and select the Copy an AWS Managed Policy
option. In the Set Permissions page, you can use the Filter and Search bars to search
for and select a policy of your choice. In this case, we are selecting Administrator
Access Policy created by AWS itself. This policy will ensure that the IAM user
attached to it is granted all administrative rights on all the AWS resources. Do note
that this is a very crude and high-level permission and is not recommended for use
in a production scenario.
In a production scenario, you would have to create individual administrator roles
for each of the AWS service that you plan to use and then assign individual users
to it as per your requirements. For now; click on the Select option adjoining the
AdministratorAccess policy as shown:
This will bring you to the Review Policy page where you can edit and fine tune the
policy as per your needs. Note that Policy Document has already been created for
your convenience:
[ 40 ]
Chapter 2
In the Review Policy page, you can provide a suitable Policy Name and an optional
Description for your new policy. You can even edit the Policy Document if you feel
the need to, but in our case, we will leave it as it is. In case you end up editing the
Policy Document, then make sure you select the Validate Policy option before you
go ahead and deploy the policy.
The * specified in the Action and Resource element is a
wildcard and indicates any and all objects.
Once you have completed the changes, select the Create Policy option. With this step,
your new custom policy is now ready to be attached to any group or user as you see fit.
To attach a particular policy to a set of users or groups, simply use the Filter and
Search bar to find your newly created policy. Once displayed, select the policy you
wish to apply by highlighting the checkbox adjoining it, select the Policy Actions
drop-down list, and select the Attach option as shown:
This will bring up an Attach Policy page. Here, you can select multiple users and
groups at the same time and apply your selected policy to them all in a single go! In
our case, we selected the user Jason and the Admin-All-About-Dogs group that we
created in our earlier steps. Once you have selected the users and groups, complete
the process by selecting the Attach Policy option.
Managing access and security using the
AWS CLI
In the previous chapter, we briefly talked about the merits of working with a
command line interface versus a GUI. We also got the AWS CLI installed and
running on a simple CentOS box. In this section, we are going to go a bit further with
that installation and actually configure the AWS CLI for use by an IAM user. Later
on, we will see how to use the AWS CLI to perform some common IAM tasks as
well. So without further ado, let's get started!
[ 41 ]
Security and Access Management
Configuring the AWS CLI is a very simple and straightforward process. All you need
are the access ID and the secret keys from any one of your IAM users that we created
during the earlier parts of this chapter. Next up, open up a terminal of your Linux
box, which has the AWS CLI installed on it, and type in the following command:
# aws configure
Once entered, you will be prompted to enter the user's Access Key ID and the Secret
Access Key, along with the default region name and the default output format to
use. The default region name is a mandatory field and can be any of the regions from
which your users will be operating, for example, us-east-1, us-west-2, and so on:
AWS Access Key ID [None]:TH1$is$0MUC#fuN
AWS Secret Access Key [None]:iH@vEN01De@W#@T1@mD01ng#ERe
Default region name [None]: us-west-2
Default output format [None]: table
The output format accepts any of these three values as the preferred method to
display the output of the commands: table, text, or json.
Any of these values can be changed at any time by rerunning
the aws configure command.
But what if I have multiple users and each of these users need to access the same
Linux box to run the commands? Do I need to share the keys with all the users? A
valid question with a simple answer, NO! You never share your keys with anyone!
As an alternative, you can set up named profiles for each of your users using their
own set of keys using this simple command:
# aws configure --profile jason
Here, we are creating a named profile for our user named Jason. Similarly, you can
create multiple named profiles of individual IMA users using this same syntax:
[ 42 ]
Chapter 2
AWS will store these credentials and configuration details
in two separate files named ~/.aws/credentials and
~/.aws/config, respectively.
Okay, now that we have the basic configurations done, let's try out the CLI by
executing some commands. To start off, let's try listing the users present in our
account. Type in the following command:
# aws iam list-users --profile jason
You should get a list of IAM users displayed on your terminal. Notice the output
format. Here, you may be viewing the output in a tabular format as our default
output format is currently set to table. Also, note that we ran the CLI command
using the named profile that we created a short while back. Awesome, isn't it?
Let's try running a few more commands now! Create an IAM user, assign it to an
existing group in our AWS account, and attach a policy to it! To begin with, create a
new user using this simple command:
# aws iam create-user --user-name YoYo --profile jason
This command will only create a user for you. This user still does not have any
passwords or access keys generated for it, so let's go ahead and create some! Type in
the following command to create a password for your user:
# aws iam create-login-profile --user-name YoYo --password P@$$w0rD
--profile jason
[ 43 ]
Security and Access Management
Here, we passed two mandatory arguments with the commands --user-name
and –password:
Besides these, you can additionally pass an optional argument called --passwordreset-required. This field will ensure that the IAM user has to reset his/her
password upon first login from the AWS Management Console. Only then will this
new user be authorized to work with the CLI. In our case, we have not provided
this argument, thus resulting in a fixed password which the user does not need
to change.
Once the passwords are created, we go ahead and create the user's all important
access key and Secret Key. To do so, type in the following command as shown:
# aws iam create-access-key --user-name YoYo --profile jason
The create-access-key command requires only one mandatory argument, which
is the username itself. Once executed, it will display the user's access and Secret
Keys respectively in the output. Make sure you save the Secret Key as this is the last
time it will be shown to you for obvious security reasons. With this step, your new
IAM user is all ready to be added to groups! Previously in this chapter we created
a few groups for our own reference using the AWS Management Console. You can
either attach your new users to existing groups or can even go ahead and create new
groups as per your requirements. In this case, we will be creating a new group and
attaching our user to it. Type in the following command to create a new group:
# aws iam create-group --group-name SuperUsersGroup --profile jason
[ 44 ]
Chapter 2
The output should display the new group's ARN as well as the Group ID as shown:
With the group created, it's now time to attach our new user to it. Simply type in the
following command as shown:
# aws iam add-user-to-group --user-name YoYo --group-name SuperUsersGroup
--profile jason
This command accepts two mandatory arguments, which include the username as
well as the group name to which the user has to be attached to. You should get no
output from the execution of this command if it was run successfully.
With this step, we completed adding our new user to a new group. But wait, our task
is not yet done. We still have to enforce some access permissions on this group; so
let's quickly add a simple policy to it.
First up, create a simple JSON-based file on your Linux box. This JSON file will
contain your new group's or user's set of permissions. For simplicity, I created a very
basic policy that will grant its users complete access to all of AWS's products and
resources. Run the following command to first create your policy:
# vi /tmp/MyPolicy.json
Add the following contents to your policy file as shown:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "*",
"Resource": "*"
}
]
}
[ 45 ]
Security and Access Management
The commands will look as follows:
Next, run the following command to attach this policy document to your newly
created group or user:
# awsiam put-group-policy --user-name YoYo \
--policy-name Admin-Access-All-About-Dogs \
--policy-document file:///tmp/MyPolicy.json \
--profile jason
You can replace the --user-name attribute with the
--group-name attribute in case you want to assign the
policy to a group.
With this step completed, you now should have a fully configured user and group
created for your organization using the AWS CLI. Simple, wasn't it!
Planning your next steps
Working with users, groups, and policies is just the start. There are a lot more
awesome features provided by AWS IAM that can help you with managing the
access and security for your organization that we haven't covered in this chapter.
Let's browse through some of these interesting features and services quickly.
For starters, let's talk about roles and identity providers. Roles are nothing but a
group of permissions that grant users access to some particular AWS resources and
services. But wait, doesn't a policy do the same thing? You're absolutely right! Both
are, in a sense, a set of permissions, but the difference lies in where and how you
apply them.
[ 46 ]
Chapter 2
Policies are applied to users and groups that belong to a particular AWS account,
whereas roles are applied to users who are generally not a part of your AWS account.
In a sense, you use roles to delegate access to users, applications, and services that
do not have access to your AWS resources. You can also use roles to create federated
identities where a user from your organization's corporate directory gets access to
your AWS resources on a temporary basis.
To learn more about roles and how you can leverage them in
your organization, use http://docs.aws.amazon.com/
IAM/latest/UserGuide/roles-toplevel.html.
This temporary access to AWS resources can be provided using an identity provider
as well. Ever used your Facebook or Google credentials to log in to a website? If yes,
then this is a classic example of using an identity provider to provide external users
access to some resources. In your case, your organization's active directory can be used
as an identity provider to authenticate and grant your corporate users access to AWS
resources. As of today, you can use either SAML 2.0 or OpenID Connect to establish
trust between your AWS account and your external source of identity provider.
Besides these, AWS has also introduced a couple of new services as well that help
with your account's easy administration and management. Here are a few of those
services briefly explained:
•
AWSCloudTrail: CloudTrail enables you, as an administrator, to log and
record each and every API call that is made from within your account.
These logs can contain information such as the API's request and response
parameters, who made the API call, the time of the API call, and so on. These
details are vital and can be used during security audits, compliance tracking,
and so on. To know more about AWSCloudTrail, check http://aws.
amazon.com/cloudtrail/getting-started/.
•
AWSConfig: AWSConfig is a fully automated service that enables you
to take a complete snapshot of all your AWS resource's configurations
for compliance and auditing purposes. It can also be used as a change
management tool to find out when your AWS resources were created,
updated, and destroyed. To know more about AWSConfig, check
http://aws.amazon.com/config/details/.
•
AWS Key Management Service: As the name suggest, this new service
enables you to manage your account's keys more effectively and efficiently.
It also provides add-on functionality such as centralized key management,
one click encryption of your data, automatic key rotations, and so on so forth.
To know more about AWS Key Management Service, check http://aws.
amazon.com/kms/getting-started/.
[ 47 ]
Security and Access Management
These are just some of the tools and services that you can leverage to make your
AWS environment more efficient and secure. Feel free to have a look at each of these
new services, and don't be afraid to take them out for a spin as well!
Recommendations and best practices
Here are a few key takeaways from this chapter:
•
Get rid of the Root Account, use IAM wherever necessary. Hide away the
Root key and avoid using it unless it's the end of the world!
•
Create a separate IAM users for your organization, each with their own sets
of access and Secret Keys. DO NOT SHARE YOUR KEYS OR PASSWORDS!
Sharing such things is never a good idea and can cause serious implications
and problems.
•
Create separate administrators for each of the AWS services that you use.
•
Use roles and groups to assign individual IAM users permissions. Always
employ the least privilege approach wherein a particular group or role has
the least amount of privileges assigned to it. Provide only the required level
of access and permissions that the task demands.
•
Leverage multi-factor authentication (MFA) wherever possible. Although
passwords are good, they are still not the best option when it comes to
authenticating users at times.
•
Rotate your passwords and keys on a periodic basis. Create keys only if there
is a requirement for it. If there are unused keys and/or users, then make sure
you delete them on a regular basis.
•
Maintain a logs and history of your AWS account and its services. Use
AWSCloudTrail for security and compliance auditing.
•
Use temporary credentials (IAM Roles) rather than sharing your account
details with other users and applications.
•
Leverage AWS Key Management Service to encrypt data and your keys
wherever necessary.
[ 48 ]
Chapter 2
Summary
Let's quickly recap all the things we covered so far in this chapter. First up, we took
a look at security and clouds in general, followed by a walkthrough of the shared
security model, followed by AWS. Later, we learned a bit about IAM and how you
as an end user can leverage it to provide secure access to individual users. We also
looked at the steps required to create users, groups, and policies using both the AWS
Management Console as well as the AWS CLI. Toward the end of the chapter, we
looked at a few important and newly introduced AWS administration and security
services as well. Finally, we topped it all off with some essential recommendations
and best practices!
The next chapter is even more amazing: we will dive into and explore the true power
of AWS provided by one of its core service offerings—the Elastic Compute Cloud, or
EC2. So stick around, we are just getting started!
[ 49 ]
Images and Instances
In the previous chapter, we learnt a lot about how AWS provides top of the line
security and access management capabilities to its users in the form of IAM and
various other tools.
In this chapter, we will explore one of the most popular and widely used AWS's
core services, that is, Elastic Compute Cloud (EC2). This chapter will cover
many important aspects about EC2, such as its use cases, its various terms and
terminologies, and cost-effective pricing strategies to name a few. It will also show
you how to get started with the service using both the AWS Management Console
and the AWS CLI; so buckle up and get ready for an awesome time!
Introducing EC2!
Remember the never ending hassles of a long and tedious procurement process? All
that time you spent waiting for a brand new server to show up at your doorstep so that
you could get started on it? Something we all as sysadmins have gone through. Well,
that all changed on August 25, 2006 when Amazon released the first beta version of
one of its flagship service offerings called the Elastic Compute Cloud or EC2.
EC2 is a service that basically provides scalable compute capacity on an on-demand,
pay-per-use basis to its end users. Let's break it up a bit to understand the terms
a bit better. To start with, EC2 is all about server virtualization! And with server
virtualization, we get a virtually unlimited capacity of virtual machines or, as
AWS calls it, instances. Users can dynamically spin up these instances, as and when
required, perform their activity on them, and then shut down the same while getting
billed only for the resources they consume.
[ 51 ]
Images and Instances
EC2 is also a highly scalable service, which means that you can scale up from just
a couple of virtual servers to thousands in a matter of minutes, and vice versa—all
achieved using a few simple clicks of a mouse button! This scalability accompanied
by dynamicity creates an elastic platform that can be used for performing virtually any
task you can think of! Hence, the term Elastic Compute Cloud! Now that's awesome!
But the buck doesn't just stop there! With virtually unlimited compute capacity,
you also get added functionality that helps you to configure your virtual server's
network, storage, as well as security. You can also integrate your EC2 environment
with other AWS services such as IAM, S3, SNS, RDS, and so on. To provide your
applications with add-on services and tools such as security, scalable storage and
databases, notification services, and so on and so forth.
EC2 use cases
Let's have a quick look at some interesting and commonly employed use cases for
AWS EC2:
•
Hosting environments: EC2 can be used for hosting a variety of applications
and software, websites, and even games on the cloud. The dynamic and
scalable environment allows the compute capacity to grow along with the
application's needs, thus ensuring better quality of service to end users at all
times. Companies such as Netflix, Reddit, Ubisoft, and many more leverage
EC2 as their application hosting environments.
•
Dev/Test environments: With the help of EC2, organizations can now create
and deploy large scale development and testing environments with utmost
ease. The best part of this is that they can easily turn on and off the service as
per their requirements as there is no need for any heavy upfront investments
for hardware.
•
Backup and disaster recovery: EC2 can be also leveraged as a medium for
performing disaster recovery by providing active or passive environments
that can be turned up quickly in case of an emergency, thus resulting in faster
failover with minimal downtime to applications.
•
Marketing and advertisements: EC2 can be used to host marketing
and advertising environments on the fly due to its low costs and rapid
provisioning capabilities.
•
High Performance Computing (HPC): EC2 provides specialized virtualized
servers that provide high performance networking and compute power that
can be used to perform CPU-intensive tasks such as Big Data analytics and
processing. NASA's JPL and Pfizer are some of the companies that employ
the use of HPC using EC2 instances.
[ 52 ]
Chapter 3
Introducing images and instances
To understand images and instances a bit better, we first need to travel a little back
in time; don't worry, a couple of years back is quite enough! This was the time
when there was a boom in the implementation and utilization of the virtualization
technology!
Almost all IT companies today run their workloads off virtualized platforms such
as VMware vSphere or Citrix XenServer to even Microsoft's Hyper-V. AWS, too, got
into the act but decided to use and modify a more off the shelf, open sourced Xen as
its virtualization engine. And like any other virtualization technology, this platform
was also used to spin up virtual machines using either some type of configuration
files or some predefined templates. In AWS's vocabulary, these virtual machines
came to be known as instances and their master templates came to be known as
images.
By now you must have realized that instances and images are nothing new! They
are just fancy nomenclature that differentiates AWS from the rest of the plain old
virtualization technologies, right? Well, no. Apart from just the naming convention,
there are a lot more differences to AWS images and instances as compared to your
everyday virtual machines and templates. AWS has put in a lot of time and effort
from time to time in designing and structuring these images and instances, such
that they remain lightweighted, spin up more quickly, and can even be ported easily
from one place to another. These factors make a lot of difference when it comes to
designing scalable and fault tolerant application environments in the cloud.
We shall be learning a lot about these concepts and terminologies in the coming
sections of this, as well as in the next chapter, but for now, let's start off by
understanding more about these images!
Understanding images
As discussed earlier, images are nothing more than preconfigured templates that you
can use to launch one or more instances from. In AWS, we call these images Amazon
Machine Images (AMIs). Each AMI contains an operating system which can range
from any modern Linux distro to even Windows Servers, plus some optional
software application, such as a web server, or some application server installed on it.
[ 53 ]
Images and Instances
It is important, however, to understand a couple of important things about AMIs.
Just like any other template, AMIs are static in nature, which basically means that
once they are created, their state remains unchanged. You can spin up or launch
multiple instances using a single AMI and then perform any sort of modifications and
alterations within the instance itself. There is also no restriction on the size of instances
that you can launch based on your AMI. You can select anything from the smallest
instance (also called as a micro instance) to the largest ones that are generally meant for
high performance computing. Take a look at the following image of EC2 AMI:
Secondly, an AMI can contain certain launch permissions as well. These permissions
dictate whether the AMI can be launched by anyone (public) or by someone or some
account which I specify (explicit) or I can even keep the AMI all to myself and not allow
anyone to launch instances from it but me (implicit). Why have launch permissions?
Well, there are cases where some AMIs can contain some form of propriety software or
licensed application, which you do not want to share freely among the general public.
In that case, these permissions come in really handy! You can alternatively even
create something called as a paid AMI. This feature allows you to share your AMI to
the general public, however, with some support costs associated with it.
AMIs can be bought and sold using something called as the AWS Marketplace as
well—a one stop shop for all your AMI needs! Here, AMIs are categorized according
to their contents and you as an end user can choose and launch instances off any one
of them. Categories include software infrastructure, development tools, business and
collaboration tools, and much more! These AMIs are mostly created by third parties
or commercial companies who wish to either sell or provide their products on the
AWS platform.
[ 54 ]
Chapter 3
Click on and browse through the AWS Marketplace using
https://aws.amazon.com/marketplace.
AMIs can be broadly classified into two main categories depending on the way they
store their root volume or hard drive:
•
EBS-backed AMI: An EBS-backed AMI simply stores its entire root device
on an Elastic Block Store (EBS) volume. EBS functions like a network shared
drive and provides some really cool add on functionalities like snapshotting
capabilities, data persistence, and so on. Even more, EBS volumes are not tied
to any particular hardware as well. This enables them to be moved anywhere
within a particular availability zone, kind of like a Network Attached
Storage (NAS) drive. We shall be learning more about EBS-backed AMIs and
instances in the coming chapter.
•
Instance store-backed AMI: An instance store-backed AMI, on the other hand,
stores its images on the AWS S3 service. Unlike its counterpart, instance store
AMIs are not portable and do not provide data persistence capabilities as the
root device data is directly stored on the instance's hard drive itself. During
deployment, the entire AMI has to be loaded from an S3 bucket into the
instance store, thus making this type of deployment a slightly slow process.
The following image depicts the deployments of both the instance store-backed and
EBS-backed AMIs. As you can see, the root and data volumes of the instance storebacked AMI are stored locally on the HOST SERVER itself, whereas the second
instance uses EBS volumes to store its root device and data.
[ 55 ]
Images and Instances
The following is a quick differentiator to help you understand some of the key
differences between EB-backed and Instance store-backed AMIs:
EBS backed
Instance store backed
Root device
Present on an EBS volume.
Present on the instance itself.
Disk size limit
Up to 16 TB supported.
Up to 10 GB supported.
Data persistence
Data is persistent even after the
instance is terminated.
Data only persists during the
lifecycle of the instance.
Boot time
Less than a minute. Only the parts
of the AMI that are required for the
boot process are retrieved for the
instance to be made ready.
Up to 5 minutes. The entire AMI
has to be retrieved from S3 before
the instance is made ready.
Costs
You are charged for the running
instance plus the EBS volume's
usage.
You are charged for the running
instance plus the storage costs
incurred by S3.
Amazon Linux AMI
Amazon Linux AMI is a specially created, lightweight Linux-based image that
is supported and maintained by AWS itself. The image is based off a RedHat
Enterprise Linux (RHEL) distro, which basically means that you can execute almost
any and all RHEL-based commands, such as yum and system-config, on it.
The image also comes pre-packaged with a lot of essential AWS tools and libraries
that allow for easy integration of the AMI with other AWS services. All in all,
everything from the yum repos to the AMIs security and patching is taken care of by
AWS itself!
The Amazon Linux AMI comes at no additional costs. You only
have to pay for the running instances that are created from it.
You can read more about the Amazon Linux AMI at http://
aws.amazon.com/amazon-linux-ami/.
Later on, we will be using this Amazon Linux AMI itself and launching our very
first, but not the last, instance into the cloud, so stick around!
[ 56 ]
Chapter 3
Understanding instances
So far we have only being talking about images; so now let's shift the attention over to
instances! As discussed briefly earlier, instances are nothing but virtual machines or
virtual servers that are spawned off from a single image or AMI. Each instance comes
with its own set of resources, namely CPU, memory, storage, and network, which are
differentiated by something called as instance families or instance types. When you
first launch an instance, you need to specify its instance type. This will determine the
amount of resources that your instance will obtain throughout its lifecycle.
AWS currently supports five instance types or families, which are briefly explained
as follows:
•
General purpose: This group of instances is your average, day-to-day,
balanced instances. Why balanced? Well, because they provide a good mix of
CPU, memory, and disk space that most applications can suffice with while
not compromising on performance. The general purpose group comprises
the commonly used instance types such as t2.micro, t2.small, t2.medium, and
the m3 and m4 series which comprises m4.large, m4.xlarge, and so on and
so forth. On average, this family contains instance types that range from 1
VCPU and 1 GB RAM (t2.micro) all the way to 40 VCPUs and 160 GB RAM
(m4.10xlarge).
•
Compute optimized: As the name suggests, these are specialized group of
instances that are commonly used for CPU-intensive applications. The group
comprises two main instances types, that is, C3 and C4. On an average, this
family contains instances that can range from 2 VCPUs and 2.75 GB RAM (c4.
large) to 36 VCPUs and 60 GB RAM (c4.8xlarge).
•
Memory optimized: Similar to the compute optimized, this family comprises
instances that require or consume more RAM than CPU. Ideally, databases
and analytical applications fall into this category. This group consists of
a single instance type called R3 instances, and they can range anywhere
from 2 VCPUs and 15.25 GB RAM (r3.large) to 32 VCPUs and 244 GB RAM
(r3.8xlarge).
•
Storage optimized: This family of instances comprises specialized instances
that provide fast storage access and writes using SSD drives. These
instances are also used for high I/O performance and high disk throughput
applications. The group also comprises two main instance types, namely the
I2 and D2 (no, this doesn't have anything to do with R2D2!). These instances
can provide SSD enabled storage ranging from 800 GB (i2.large) all the way
up to 48 TB (d2.8xlarge)—now that's impressive!
[ 57 ]
Images and Instances
•
GPU instances: Similar to the compute optimized family, the GPU instances
are specially designed for handling high CPU-intensive tasks but by using
specialized NVIDIA GPU cards. This instance family is generally used for
applications that require video encoding, machine learning or 3D rendering,
and so on. This group consists of a single instance type called G2, and it can
range between 1 GPU (g2.2xlarge) and 4 GPU (g2.8xlarge).
To know more about the various instance types and their use cases,
refer to http://aws.amazon.com/ec2/instance-types/.
As of late, AWS EC2 supports close to 38 instance types, each with their own set
of pros and cons and use cases. In such times, it actually becomes really difficult
for an end user to decide which instance type is right for his/her application. The
easiest and most common approach taken is to pick out the closet instance type that
matches your application's set of requirements - for example, it would be ideal to
install a simple MongoDB database on a memory optimized instance rather than a
compute or GPU optimized instance. Not that compute optimized instances are a
wrong choice or anything, but it makes more sense to go for memory in such cases
rather than just brute CPU. From my perspective, I have always fancied the general
purpose set of instances simply because most of my application needs seem to get
balanced out correctly with it, but feel free to try out other instance types as well.
EC2 instance pricing options
Apart from the various instance types, EC2 also provides three convenient instance
pricing options to choose from, namely on-demand, reserved, and spot instances.
You can use either or all of these pricing options at the same time to suit your
application's needs. Let's have a quick look at all three options to get a better
understanding of them.
On-demand instances
Pretty much the most commonly used instance deployment method, the on-demand
instances are created only when you require them, hence the term on-demand. Ondemand instances are priced by the hour with no upfront payments or commitments
required. This, in essence, is the true pay-as-you-go payment method that we always
end up mentioning when talking about clouds. These are standard computational
resources that are ready whenever you request them and can be shut down anytime
during its tenure.
[ 58 ]
Chapter 3
By default, you can have a max of 20 such on-demand instances launched within
a single AWS account at a time. If you wish to have more such instances, then you
simply have to raise a support request with AWS using the AWS Management
Console's Support tab. A good use case for such instances can be an application
running unpredictable workloads, such as a gaming website or social website. In this
case, you can leverage the flexibility of on-demand instances accompanied with their
low costs to only pay for the compute capacity you need and use and not a dime more!
On-demand instance costs vary based on whether the
underlying OS is a Linux or Windows, as well as in the
regions that they are deployed in.
Consider this simple example: A t2.micro instance costs $0.013 per hour to run in
the US East (N. Virginia) region. So, if I was to run this instance for an entire day, I
would only have to pay $0.312! Now that's cloud power!
Reserved instances
Deploying instances using the on-demand model has but one slight drawback,
which is that AWS does not guarantee the deployment of your instance. Why, you
ask? Well to put it simply, using on-demand model, you can create and terminate
instances on the go without having to make any commitments whatsoever. It is up
to AWS to match this dynamic requirement and make sure that adequate capacity is
present in its datacenters at all times. However, in very few and rare cases, this does
not happen, and that's when AWS will fail to power on your on-demand instance.
In such cases, you are better off by using something called as reserved instances,
where AWS actually guarantees your instances with resource capacity reservations
and significantly lower costs as compared to the on-demand model. You can choose
between three payment options when you purchase reserved instances: all upfront,
partial upfront, and no upfront. As the name suggests, you can choose to pay some
upfront costs or the full payment itself for reserving your instances for a minimum
period of a year and maximum up to three years.
Consider our earlier example of the t2.micro instance costing $0.0013 per hour. The
following table summarizes the upfront costs you will need to pay for a period of
one year for a single t2.micro instance using the reserved instance pricing model:
Payment method
Upfront cost
Monthly cost
Hourly cost
Savings over
on-demand
No upfront
$0
$6.57
$0.009
31%
Partial upfront
$51
$2.19
$0.0088
32%
All upfront
$75
$0
$0.0086
34%
[ 59 ]
Images and Instances
Reserved instances are the best option when the application loads are steady and
consistent. In such cases, where you don't have to worry about unpredictable
workloads and spikes, you can reserve a bunch of instances in EC2 and end up
saving on additional costs.
Spot instances
Spot instances allow you to bid for unused EC2 compute capacity. These instances
were specially created to address a simple problem of excess EC2 capacity in AWS.
How does it all work? Well, it's just like any other bidding system. AWS sets the
hourly price for a particular spot instance that can change as the demand for the spot
instances either grows or shrinks. You as an end user have to place a bid on these spot
instances, and when your bid exceeds that of the current spot price, your instances
are then made to run! It is important to also note that these instances will stop the
moment someone else out bids you, so host your application accordingly. Ideally,
applications that are non-critical in nature and do not require large processing times,
such as image resizing operations, are ideally run on spot instances.
Let's look at our trusty t2.micro instance example here as well. The on-demand cost
for a t2.micro instance is $0.013 per hour; however, I place a bid of $0.0003 per hour
to run my application. So, if the current bid cost for the t2.micro instance falls below
my bid, then EC2 will spin up the requested t2.micro instances for me until either I
choose to terminate them or someone else out bids me on the same—simple, isn't it?
Spot instances compliment the reserved and on-demand instances; hence, ideally,
you should use a mixture of spot instances working on-demand or reserved
instances just to be sure that your application has some compute capacity on standby
in case it needs it.
Working with instances
Okay, so we have seen the basics of images and instances along with various
instance types and some interesting instance pricing strategies as well. Now comes
the fun part! Actually deploying your very own instance on the cloud!
In this section, we will be using the AWS Management Console and launching our
very first t2.micro instance on the AWS cloud. Along the way, we shall also look at
some instance lifecycle operations such as start, stop, reboot, and terminate along
with steps, using which you can configure your instances as well. So, what are we
waiting for? Let's get busy!
[ 60 ]
Chapter 3
To begin with, I have already logged in to my AWS Management Console using
the IAM credentials that we created in our previous chapter. If you are still using
your root credentials to access your AWS account, then you might want to revisit
Chapter 2, Security and Access Management, and get that sorted out! Remember, using
root credentials to access your account is a strict no no!
Although you can use any web browser to access your AWS
Management Console, I would highly recommend using
Firefox as your choice of browser for this section.
Once you have logged into the AWS Management Console, finding the EC2 option
isn't that hard. Select the EC2 option from under the Compute category, as shown in
the following screenshot:
This will bring up the EC2 dashboard on your browser. Feel free to have a look
around the dashboard and familiarize yourself with it. To the left, you have the
Navigation pane that will help you navigate to various sections and services
provided by EC2, such as Instances, Images, Network and Security, Load
Balancers, and even Auto Scaling. The centre dashboard provides a real-time
view of your EC2 resources, which includes important details such as how many
instances are currently running in your environment, how many volumes, key pairs,
snapshots, or elastic IPs have been created, so on and so forth.
The dashboard also displays the current health of the overall region as well as its
subsequent availability zones. In our case, we are operating from the US West (Oregon)
region that contains additional AZs called as us-west-2a, us-west-2b, and us-west-2c.
These names and values will vary based on your preferred region of operation.
[ 61 ]
Images and Instances
Next up, we launch our very first instance from this same dashboard by selecting the
Launch Instance option, as shown in the following screenshot:
On selecting the Launch Instance option, you will be directed to a wizard driven
page that will help you create and customize your very first instance. This wizard
divides the entire instance creation operation into seven individual stages, each stage
having its own set of configurable items. Let's go through these stages one at a time.
Stage 1 – choose AMI
Naturally, our first instance has to spawn from an AMI, so that's the first step!
Here, AWS provides us with a whole lot of options to choose from, which includes
a Quick Start guide, which lists out the most frequently used and popular AMIs,
and includes the famous Amazon Linux AMI as well, as shown in the following
screenshot:
There are also a host of other operating systems provided here as well which
includes Ubuntu, SUSE Linux, Red Hat, and Windows Servers.
[ 62 ]
Chapter 3
Each of these AMIs has a uniquely referenced AMI ID, which looks something like
this: ami-e75272d7. We can use this AMI ID to spin up instances using the AWS CLI,
something which we will perform in the coming sections of this chapter. They also
contain additional information such as whether the root device of the AMI is based
on an EBS volume or not, whether the particular AMI is eligible under the Free tier
or not, and so on and so forth.
Besides the Quick Start guide, you can also spin up your instances using the AWS
Marketplace and the Community AMIs section as well. Both these options contain
an exhaustive list of customized AMIs that have been created by either third-party
companies or by developers and can be used for a variety of purposes. But for this
exercise, we are going to go ahead and select Amazon Linux AMI itself from the
Quick Start menu.
Stage 2 – choose an instance type
With the AMI selected, the next step is to select the particular instance type or size as
per your requirements. You can use the Filter by option to group and view instances
according to their families and generations as well. In this case, we are going ahead
with the general purpose t2.micro instance type, which is covered under the free
tier eligibility and will provide us with 1 VCPU and 1 GB of RAM to work with! The
following screenshot shows the configurations of the instance:
Ideally, now you can launch your instance right away, but this will not allow you to
perform any additional configurations on your instance, which just isn't nice! So, go
ahead and click on the Next: Configure instance Details button to move on to the
third stage.
[ 63 ]
Images and Instances
Stage 3 – configure instance details
Now here it gets a little tricky for first timers. This page will basically allow you
to configure a few important aspects about your instance, including its network
settings, monitoring, and lots more. Let's have a look at each of these options in
detail:
•
Number of instances: You can specify how many instances the wizard
should launch using this field. By default, the value is always set to one
single instance.
•
Purchasing option: Remember the spot instances we talked about earlier?
Well here you can basically request for spot instance pricing. For now, let's
leave this option all together:
•
Network: Select the default Virtual Private Cloud (VPC) network that is
displayed in the dropdown list. You can even go ahead and create a new
VPC network for your instance, but we will leave all that for later chapters
where we will actually set up a VPC environment.
In our case, the VPC has a default network of 172.31.0.0/16, which means we
can assign up to 65,536 IP addresses using it.
•
Subnet: Next up, select the Subnet in which you wish to deploy your new
instance. You can either choose to have AWS select and deploy your instance
in a particular subnet from an available list or you can select a particular
choice of subnet on your own. By default, each subnet's Netmask defaults to
/20, which means you can have up to 4,096 IP addresses assigned in it.
[ 64 ]
Chapter 3
•
Auto-assign Public IP: Each instance that you launch will be assigned a
Public IP. The Public IP allows your instance to communicate with the
outside world, a.k.a. the Internet! For now, select the use Subnet setting
(Enable) option as shown.
•
IAM role: You can additionally select a particular IAM role to be associated
with your instance. In this case, we do not have any roles particularly created.
•
Shutdown behaviour: This option allows you to select whether the instance
should stop or be terminated when issued a shutdown command. In
this case, we have opted for the instance to stop when it is issued a
shutdown command.
•
Enable termination protection: Select this option in case you wish to protect
your instance against accidental deletions.
•
Monitoring: By default, AWS will monitor few basic parameters about
your instance for free, but if you wish to have an in-depth insight into
your instance's performance, then select the Enable CloudWatch detailed
monitoring option.
•
Tenancy: AWS also offers you to power on your instances on a single-tenant,
dedicated hardware in case your application's compliance requirements
are too strict. For such cases, select the Dedicated option from the Tenancy
dropdown list, else leave it to the default Shared option. Do note, however,
that there is a slight increase in the overall cost of an instance if it is made to
run on a dedicated hardware.
Once you have selected your values, move on to the fourth stage of the instance
deployment process by selecting the Next: Add Storage option.
Stage 4 – add storage
Using this page, you can add additional EBS volumes to your instances. To add new
volumes, simply click on the Add New Volume button. This will provide you with
options to provide the size of the new volume along with its mount points. In our
case, there is an 8 GB volume already attached to our instance. This is the t2.micro
instance's root volume, as shown in the following screenshot:
[ 65 ]
Images and Instances
Try and keep the volume's size under 30 GB to avail the free
tier eligibility.
You can optionally increase the size of the volume and enable add-on features such
as Delete on Termination as per your requirement. Once done, proceed to the next
stage of the instance deployment process by selecting the Next: Tag instance option.
Stage 5 – tag instances
The tag instances page will allow you to specify tags for your EC2 instance. Tags are
nothing more than normal key-value pairs of text that allow you to manage your
AWS resources a lot easily. You can start, stop, and terminate a group of instances
or any other AWS resources using tags. Each AWS resource can have a maximum
of 10 tags assigned to it. For example, in our case, we have provided a tag for our
instance as ServerType:WebServer. Here, ServerType is the key and WebServer its
corresponding value. You can have other group of instances in your environment
tagged as ServerType:DatabaseServer or ServerType:AppServer based on their
application. The important thing to keep in mind here is that AWS will not assign
a tag to any of your resources automatically. These are optional attributes that you
assign to your resources in order to facilitate in easier management:
Once your tags are set, click on the Next: Configure Security Group option to
proceed.
[ 66 ]
Chapter 3
Stage 6 – configure security groups
Security groups are an essential tool used to safeguard access to your instances
from the outside world. Security groups are nothing but a set of firewall rules that
allow specific traffic to reach your instance. By default, the security groups allow
for all outbound traffic to pass while blocking all inbound traffic. By default, AWS
will auto-create a security group for you when you first start using the EC2 service.
This security group is called as default and contains only a single rule that allows all
inbound traffic on port 22.
In the Configure Security Groups page, you can either choose to Create a new
security group or Select an existing security group. Let's go ahead and create one
for starters. Select the Create a new security group option and fill out a suitable
Security group name and Description. By default, AWS would have already
enabled inbound SSH access by enabling port 22:
You can add additional rules to your security group based on your requirements as
well. For example, in our instance's case, we want the users to receive all inbound
HTTP traffic as well. So, select the Add Rule option to add a firewall rule. This will
populate an additional rule line, as shown in the preceding screenshot. Next, from
the Type dropdown, select HTTP and leave the rest of the fields to their default
values. With our security group created and populated, we can now go ahead with
the final step in the instance launch stage.
[ 67 ]
Images and Instances
Stage 7 – review instance launch
Yup! Finally, we are here! The last step toward launching your very first
instance! Here, you will be provided with a complete summary of your instance's
configuration details, including the AMI details, instance type selected, instance
details, and so on. If all the details are correct, then simply go ahead and click on the
Launch option. Since this is your first instance launch, you will be provided with an
additional popup page that will basically help you create a key pair.
A key pair is basically a combination of a public and a private key, which is used to
encrypt and decrypt your instance's login info. AWS generates the key pair for you
which you need to download and save locally to your workstation. Remember that
once a particular key pair is created and associated with an instance, you will need
to use that key pair itself to access the instance. You will not be able to download
this key pair again; hence, save it in a secure location. Take a look at the following
screenshot to get an idea of selecting the key pair:
In EC2, the Linux instances have no login passwords by
default; hence, we use key pairs to log in using SSH. In case of a
Windows instance, we use a key pair to obtain the administrator
password and then log in using an RDP connection.
Select the Create a new key pair option from the dropdown list and provide a
suitable name for your key pair as well. Click on the Download Key Pair option to
download the .PEM file. Once completed, select the Launch Instance option. The
instance will take a couple of minutes to get started. Meanwhile, make a note of the
new instance's ID (in this case, i-53fc559a) and feel free to view the instance's launch
logs as well:
[ 68 ]
Chapter 3
Phew! With this step completed, your instance is now ready for use! Your instance
will show up in the EC2 dashboard, as shown in the following screenshot:
The dashboard contains and provides a lot of information about your instance. You
can view your instance's ID, instance type, power state, and a whole lot more info
from the dashboard. You can also obtain your instance's health information using
the Status Checks tab and the Monitoring tab. Additionally, you can perform power
operations on your instance such as start, stop, reboot, and terminate using the
Actions tab located in the preceding instance table.
Before we proceed to the next section, make a note of your instance's Public DNS
and the Public IP. We will be using these values to connect to the instances from our
local workstations.
Connecting to your instance
Once your instance has launched successfully, you can connect to it using three
different methods that are briefly explained as follows:
•
Using your web browser: AWS provides a convenient Java-based web
browser plugin called as MindTerm, which you can use to connect to your
instances. Follow the next steps to do so:
1. From the EC2 dashboard, select the instance which you want to
connect to and then click on the Connect option.
2. In the Connect To Your Instance dialog box, select the option A Java
SSH Client directly from my browser (Java required) option. AWS
will autofill the Public IP field with your instance's public IP address.
[ 69 ]
Images and Instances
3. You will be required, however, to enter the User name and the
Private key path, as shown in the following screenshot:
4. The User Name for an Amazon Linux AMIs is ec2-user by default.
You can optionally choose to store the location of your private key
in the browser's cache; however, it is not at all required. Once all the
required fields are filled in, select the Launch SSH Client option.
For most RHEL-based AMIs, the user name is either root
or the ec2-user, and for Ubuntu-based AMIs, the user
name is generally Ubuntu itself.
5. Since this is going to be your first SSH attempt using the MindTerm
plugin, you will be prompted to accept an end user license agreement.
6. Select the Accept option to continue with the process. You will be
prompted to accept few additional prompts along the way, which
include the setting up of your home directory and known hosts
directory on your local PC.
[ 70 ]
Chapter 3
7. Confirm all these settings and you should now see the MindTerm
console displaying your instance's terminal, as shown in the
following screenshot:
•
Using Putty: The second option is by far the most commonly used and one
of my favorites as well! Putty, or PuTTY, is basically an SSH and telnet client
that can be used to connect to your remote Linux instances. But before you
get working on Putty, you will need a tool called PuttyGen to help you
create your private key (*.ppk).
You can download Putty, PuttyGen, and various other SSH
and FTP tools from http://www.chiark.greenend.
org.uk/~sgtatham/putty/download.html.
After creating your private key, follow the next steps to use Putty and PuttyGen:
1. First up, download and install the latest copy of Putty and PuttyGen
on your local desktops.
2. Next, launch PuttyGen from the start menu. You should see the
PuttyGen dialog as shown in the following screenshot.
[ 71 ]
Images and Instances
3. Click on the Load option to load your PEM file. Remember, this is
the same file that we downloaded during stage 7 of the instance
launch phase.
4. Once loaded, go ahead and save this key by selecting the Save
private key option.
PuttyGen will probably prompt you with a warning message stating that you
are saving this key without a passphrase and would you like to continue.
5. Select Yes to continue with the process. Provide a meaningful name
and save the new file (*.PPK) at a secure and accessible location. You
can now use this PPK file to connect to your instance using Putty.
[ 72 ]
Chapter 3
Now comes the fun part! Launch a Putty session from the Start menu. You
should see the Putty dialog box as shown in the following screenshot. Here,
provide your instance's Public DNS or Public IP in the Host Name (or IP
address) field as shown. Also make sure that the Port value is set to 22 and
the Connection type is selected as SSH.
6. Next, using Putty's Navigation | Category pane, expand the SSH
option and then select Auth, as shown in the following screenshot.
All you need to do here is browse and upload the recently saved PPK
file in the Private key file for authentication field. Once uploaded,
click on Open to establish a connection to your instance.
[ 73 ]
Images and Instances
7. You will be prompted by a security warning since this is the first
time you are trying to connect your instance. The security dialog box
simply asks whether you trust the instance that you are connecting to
or not. Click on the Yes tab when prompted.
8. In the Putty terminal window, provide the user name for your
Amazon Linux instance (ec2-user) and hit the Enter key. Voila!
Your first instance is now ready for use, as shown in the following
screenshot. Isn't that awesome!
•
Using SSH: The third and final method is probably the most simple and
straightforward. You can connect to your EC2 instances using a simple
SSH client as well. This SSH client can be installed on a standalone Linux
workstation or even on a Mac. Here, we will be using our CentOS 6.5
machine that has the AWS CLI installed and configured in it and following
the next steps, we will be able to look into our EC2 dashboard:
1. First up, transfer your private key (*.PEM) file over to the Linux
server using and SCP tool. In my case, I always use WinSCP to
achieve this. It's a simple tool and pretty straightforward to use. Once
the key is transferred, run the following command to change the
key's permissions:
# chmod 400 .pem
2. Next up, simply connect to the remote EC2 instance by using the
following SSH command. You will need to provide your EC2
instance's public DNS or its public IP address, which can be found
listed on the EC2 dashboard:
# ssh -I .pem ec2-user@
[ 74 ]
Chapter 3
And following is the output of the preceding command:
Configuring your instances
Once your instances are launched, you can configure virtually anything in it, from
packages, to users, to some specialized software or application, anything and
everything goes!
Let's begin by running some simple commands first. Go ahead and type the
following command to check your instance's disk size:
# df –h
Here is the output showing the configuration of the instance:
[ 75 ]
Images and Instances
You should see an 8 GB disk mounted on the root (/) partition, as shown in the
preceding screenshot. Not bad, eh! Let's try something else, like updating the
operating system. AWS Linux AMIs are regularly patched and provided with
necessary package updates, so it is a good idea to patch them from time to time.
Run the following command to update the Amazon Linux OS:
# sudo yum update -y
Why sudo? Well, as discussed earlier, you are not provided with root privileges
when you log in to your instance. You can change that by simple changing the
current user to root after you login; however, we are going to stick with the ec2-user
itself for now.
What else can we do over here? Well, let's go ahead and install some specific
software for our instance. Since this instance is going to act as a web server, we will
need to install and configure a basic Apache HTTP web server package on it.
Type in the following set of commands that will help you install the Apache HTTP
web server on your instance:
# sudo yum install httpd
Once the necessary packages are installed, simply start the Apache HTTP server
using the following simple commands:
# sudo service httpd start
# sudo chkconfig httpd on
You can see the server running after running the preceding commands, as shown in
the following screenshot:
[ 76 ]
Chapter 3
You can verify whether your instance is actually running a web server or not by
launching a web browser on your workstation and typing either in the instance's
public IP or public DNS. You should see the Amazon Linux AMI test page, as shown
in the following screenshot:
There you have it! A fully functional and ready-to-use web server using just a few
simple steps! Now wasn't that easy!
Launching instances using the AWS CLI
So far, we have seen how to launch and manage instances in EC2 using the EC2
dashboard. In this section, we are going to see how to leverage the AWS CLI to
launch your instance in the cloud! For this exercise, I'll be using my trusty old
CentOS 6.5 machine, which has been configured from Chapter 2, Security and Access
Management, to work with the AWS CLI. So, without further ado, let's get busy!
Stage 1 – create a key pair
First up, let's create a new key pair for our instance. Note that you can use existing
key pairs to connect to new instances; however, we will still go ahead and create a
new one for this exercise. Type in the following command in your terminal:
# aws ec2 create-key-pair --key-name \
> --output text > .pem
Once the key pair has been created, remember to change its permissions using the
following command:
# chmod 400 .pem
[ 77 ]
Images and Instances
And you can see the created key:
Stage 2 – create a security group
Once again, you can very well reuse an existing security group from EC2 for your
new instances, but we will go ahead and create one here. Type in the following
command to create a new security group:
# aws ec2 create-security-group --group-name \
> --description ""
For creating security groups, you are only required to provide a security group name
and an optional description field along with it. Make sure that you provide a simple
yet meaningful name here:
Once executed, you will be provided with the new security group's ID as the output.
Make a note of this ID as it will be required in the next few steps.
[ 78 ]
Chapter 3
Stage 3 – add rules to your security group
With your new security group created, the next thing to do is to add a few firewall
rules to it. We will be discussing a lot more on this topic in the next chapter, so to
keep things simple, let's add one rule to allow inbound SSH traffic to our instance.
Type in the following command to add the new rule:
# aws ec2 authorize-security-group-ingress --group-name \
> --protocol tcp --port 22 --cidr 0.0.0.0/0
To add a firewall rule, you will be required to provide the security group's name to
which the rule has to be applied. You will also need to provide the protocol, port
number, and network CIDR values as per your requirements:
Stage 4 – launch the instance
With the key pair and security group created and populated, the final thing to do
is to launch your new instance. For this step, you will need a particular AMI ID
along with a few other key essentials such as your security group name, the key
pair, and the instance launch type, along with the number of instances you actually
wish to launch.
Type in the following command to launch your instance:
# aws ec2 run-instances --image-id ami-e7527ed7 \
> --count 1 --instance-type t2.micro \
> --security-groups \
> --key-name
[ 79 ]
Images and Instances
And here is the output of the preceding commands:
In this case, we are using the same Amazon Linux AMI
(ami-e7527ed7) that we used during the launch of our
first instance using the EC2 dashboard.
The instance will take a good two or three minutes to spin up, so be patient! Make a
note of the instance's ID from the output of the ec2 run-instance command. We will
be using this instance ID to find out the instance's public IP address using the EC2
describe-instance command as shown:
# aws ec2 describe-instances --instance-ids
Make a note of the instance's public DNS or the public IP address. Next, use the key
pair created and connect to your instance using any of the methods discussed earlier.
Cleaning up!
Spinning up instances is one thing; you should also know how to stop and terminate
them! To perform any power operations on your instance from the EC2 dashboard,
all you need to do is select the particular instance and click on the Actions tab as
shown. Next, from the Instance State submenu, select whether you want to Stop,
Reboot, or Terminate your instance, as shown in the following screenshot:
[ 80 ]
Chapter 3
It is important to remember that you only have instance stopping capabilities
when working with EBS-backed instances. Each time an EBS-backed instance is
stopped, the hourly instance billing stops too; however, you are still charged for the
EBS volume that your instance is using. Similarly, if your EBS-backed instance is
terminated or destroyed, then by default the EBS root volume attached to it is also
destroyed, unless specified otherwise, during the instance launch phase.
Planning your next steps
So far, all we have worked with are Linux instances, so the next step that I
recommend is that you go ahead and deploy your very first Windows server
instance as well. Just a few pointers worth remembering are to make sure you enable
the firewall rule for RDP protocol (TCP Port 3389) in the security group and to
generate the administrator password using the key pair that you create. For more
in-depth steps, check out this simple tutorial at http://docs.aws.amazon.com/
AWSEC2/latest/WindowsGuide/EC2Win_GetStarted.html.
The second thing worth trying out are spot instances. Now, you may be wondering
that spot instances seem kind of hard to grasp, but in reality they are a lot easier and
cost efficient to work with. Try and spin up a simple t2.micro Linux instance using
spot pricing and compare the difference with a traditional on-demand instance.
To know more about spot instances, check out http://aws.amazon.com/ec2/
purchasing-options/spot-instances/.
Another really cool thing worth the time and effort is the AWS Management Portal
for vCenter! Yes! You heard it right! You can actually manage your AWS resources
using your standard VMware vCenter Server! All you need to do is install a simple
plugin and, voila, your entire AWS infrastructure can be managed using the familiar
vCenter dashboard. But the fun doesn't just stop there. You can also export your
on premise virtual machines hosted on the vSphere platform over to AWS using
a tool called as VM Import/ Export. Once installed within your VMware vSphere
environment, you can easily migrate any Linux and Windows Server based virtual
machine to your AWS account using a few simple steps! Now that's really amazing!
To know more about the AWS Management Portal for vCenter, refer to http://aws.
amazon.com/ec2/vcenter-portal/.
Both the AWS Management portal for vCenter as well as the
VM Import/ Export tool are absolutely free of cost! You only
have to pay for the AWS resources that you consume and not
a penny more!
[ 81 ]
Images and Instances
And last but not least, have some fun with configuring your instances! Don't stop
just at a simple Web Server; go ahead and set up a full fledge WordPress application
on your instances or launch multiple instances and set up JBoss Clustering among
them and so on. The more you configure and use the instances, the more you will
get acquainted with the terms and terminologies and find out how easy it is working
with AWS! Just remember to clean up your work after it is done.
Recommendations and best practices
Here are a few key takeaways from this chapter:
•
First and foremost, create and use separate IAM users for working with EC2.
DO NOT USE your standard root account credentials!
•
Use IAM roles if you need to delegate access to your EC2 account to other
people for some temporary period of time. Do not share your user passwords
and keys with anyone.
•
Use a standard and frequently deployed set of AMIs as they are tried and
tested by AWS thoroughly.
•
Make sure that you understand the difference between instance store-backed
and EBS-backed AMIs. Use the instance store with caution and remember
that you are responsible for your data, so take adequate backups of it.
•
Don't create too many firewall rules on a single security group. Make sure
that you apply the least permissive rules for your security groups.
•
Stop your instances when not in use. This will help you save up on costs
as well.
•
Use tags to identify your EC2 instances. Tagging your resources is a good
practice and should be followed at all times.
•
Save your key pairs in a safe and accessible location. Use passphrases as an
added layer of security if you deem it necessary.
•
Monitor your instances at all times. We will be looking at instance
monitoring in depth in the coming chapters; however, you don't have to wait
until then! Use the EC2 Status and Health Check tabs whenever required.
[ 82 ]
Chapter 3
Summary
So, let's wrap up what we have learnt so far! First up, we looked at what exactly
the AWS EC2 service is and how we can leverage it to perform our daily tasks.
Next, we understood a bit about images and instances by looking at the various
instance types and pricing options provided. Finally, we also managed to
launch a couple of instances in EC2 using both the EC2 dashboard as well as the
AWS CLI. We topped it all off with some interesting next steps and a bunch of
recommendations and best practices!
In the next chapter, we will continue with the EC2 service and explore some of
the advanced network, security, and storage options that come along with it, so
stay tuned!
[ 83 ]
Security, Storage,
Networking, and Lots More!
In the previous chapter, you learned a lot about EC2 and its images and instances.
We were able to launch our first instance in AWS, connect to it, and even configure it
as per our requirements.
In this chapter, we will be continuing where we left off and will cover some of
the remaining EC2 concepts, such as security groups, networking, and a bit about
volumes as well. We will also be looking at a few easy steps using which you can
create and publish your very own AMIs. So stick around, we are just getting started!
An overview of security groups
We talked briefly about security groups in the previous chapter, but in this section,
we will be looking at them in a bit more in detail. Security Groups are simple, yet
powerful ways using which you can secure your entire EC2 environment. You can
use Security Groups to restrict and filter out both the ingress and egress traffic of an
instance using a set of firewall rules. Each rule can allow traffic based on a particular
protocol—TCP or UDP, based on a particular port—such as 22 for SSH, or even based
on individual source and destination IP addresses. This provides you with a lot of
control and flexibility in terms of designing a secure environment for your instances to
run from.
Let's look at how you can edit an existing Security Group using the EC2 dashboard.
[ 85 ]
Security, Storage, Networking, and Lots More!
From the EC2 dashboard, select the Security Groups option located under the
Network & Security section as shown here:
This will display a list of currently created and in use Security Groups present in
your EC2 environment. Each Security Group is provided with a unique identifier
called the Group ID and a Group Name. You will also notice the presence of the
default Security Group, as shown in the following screenshot. This default Security
Group is created by AWS when you first start and sign up for the EC2 service. If you
do not specify a Security Group during the instance launch phase, then by default,
AWS assigns this default Security Group to it.
The default Security Group has no ingress (inbound) traffic rules set; there is only
one egress (outbound) rule, which allows your instances to connect to the outside
world using any port and any protocol. You can add, delete, and modify any rules
from this group; however, you cannot delete the default Security Group. As a
good practice, avoid using the default Security Group. Instead, create separate and
customized Security Groups based on your application's needs and always keep the
rules as minimalistic as possible. Here is an option of creating a new Security Group:
Let's go ahead and see how you can edit Security Groups and modify an already
configured firewall rule.
You can modify the firewall rules of your Security
Groups any time, even when your instance is running.
[ 86 ]
Chapter 4
From the dashboard, select a particular Security Group you wish to modify. Next,
from the Actions drop-down list, select the option Edit inbound rules, as shown:
As discussed earlier, each firewall rule comprises four fields. The first field is
the Type field, which specifies the type of application for which you need to
allow access. By default, AWS already has provided a list of common application
types to choose from, which includes SSH, RDP, HTTP, HTTPS, POP3, IMAP,
MySQL, SMTP, and so on so forth. You can additionally create custom TCP/ UDP
application types using this same drop-down list as well. For now, we will use the
SSH and HTTP types, as shown here:
An important thing to note here is that selecting these preconfigured application
types will autofill the next two fields as well. Thus, if you wish to specify a different
port for say SSH or HTTP, then you are better off selecting Custom TCP Rule from
the application type as discussed earlier. Next up is the Source field where you can
basically specify any of these three options:
•
Anywhere: Using this option as the source, your particular application port
will be accessible from any and all networks out there (0.0.0.0/0). This is not
a recommended configuration for any production environment and should
be avoided at all times.
[ 87 ]
Security, Storage, Networking, and Lots More!
•
My IP: As the name suggest, AWS will try and autofill the IP address of your
local computer here. The only thing that you need to be aware of here is that
your computer's IP address should not be based on a DHCP network as you
may not be able to connect to your application if your local computer's IP
address keeps on changing.
•
Custom IP: Perhaps the most preferable out of the three options, the
Custom IP option allows you to specify your own custom source IP address
or IP range as per your requirements. For example, allow the particular
application to access only via traffic coming from the network 203.20.31.0/24
CIDR. You can even add other Security Group IDs here as a reference.
Additionally, you can even add new rules to an existing Security Group by selecting
the Add Rule button, and delete existing rules by selecting the Delete Rule icon
(X). Just remember to save your Security Group settings by selecting the Save
option before you close the pop-up box. Feel free to take a look at a Security Group's
outbound rules as well. All you have to do is select a particular Security Group from
the EC2 dashboard, and from the Actions tab, select the Edit outbound rules option.
You should see the default allow all access outbound rule, as shown here:
You can even create new Security Groups using the Create Security Group option
provided on the EC2 dashboard. Selecting this option will provide you with a simple
interface using which you can create and populate a Security Group with both
inbound and outbound rules.
Provide a suitable Security group name and Description for your new Security
Group. Ideally, as a good practice, always name your Security Groups using some
meaningful conventions that can help you identify their purpose. Next up, select the
default VPC subnet from the VPC drop-down list. You can create up to 100 Security
Groups in a VPC, with each Security Group having up to fifty firewall rules.
[ 88 ]
Chapter 4
Fill in your inbound and outbound rules, and click on Create once done:
You can then assign this new Security Group to your instances either during the
launch phase or by selecting an existing instance from the EC2 dashboard and
changing its Security Group under the Change Security Groups option.
Feel free to create more such Security Groups in your EC2 account. Make sure you
follow the least privilege approach and allow traffic only for the required set of ports
and application services and nothing more.
Understanding EC2 networking
Before we understand how EC2 networking actually works, it is essential to
understand the difference between networks provided by your traditional data
centres and public clouds such as AWS. A traditional data centre network generally
comprises a number of physical switches and routers that are connected to physical
hardware and are responsible for transmitting and forwarding data or packets from
one place to another.
[ 89 ]
Security, Storage, Networking, and Lots More!
The same also applies in the case of cloud computing; however, in place of the
hardware, you now have virtual devices such as virtual servers, virtual network
cards, virtual switches, and routers. However, the main differentiator between
traditional and cloud based networks is that a cloud-based network is heavily
filtered. Most public cloud providers, including AWS itself, allow only unicast
datagrams over their networks, restricting all broadcast datagrams. Why, you ask?
Well, mostly for security purposes and to avoid DDoS attacks, besides other reasons
as well. This is an important point to remember, however, as often your applications
may require broadcast capabilities over a network to discover some services and
in such cases these applications may not necessarily fit on a public cloud. There are
ways to get past this limitation; however, that is a different topic altogether. For now,
let's take a look at how our EC2 instances are provided with their networks and IP
addresses.
To begin with, each instance that you launch in your EC2 environment is provided
with two unique IP addresses, called a private and public IP address, respectively.
This is the default behavior of an instance and is not under your control by default,
unless you are working with a VPC, which we will be discussing in the next chapter.
When you first launch an instance, AWS will provide it with a unique private IP
address using its own internal DHCP service. You can use this private IP address
to communicate with the instances present in the same network; however being a
private IP address, you cannot use this network for any communication with the
outside (Internet) world. Along with the private IP address, you also get an internal
DNS hostname for your instance. The internal or private DNS resembles something
like this string, ip-172-31-46-172.us-west-2.compute.internal, and as you can see,
it tells us a lot about our instance as well. For example, this particular private DNS
hostname resolves a private IP of 172-31-46-172 and also this particular instance
is currently deployed in the us-west-2 region. Neat, right! Let's take a look at the
following screenshot, which shows example of a private and public IP address:
[ 90 ]
Chapter 4
The same applies to the public IP address as well with the exception that this
particular IP address is reachable from the Internet and can be used to communicate
with the outside world. AWS maps the public IP address of an instance to its
corresponding private IP address using simple NAT and, just like its counterpart,
provides it with a public DNS value as well. The public DNS resembles something
like this string, ec2-54-149-173-165.us-west-2.compute.amazonaws.com, and as you
can see, this also provides us with similar information about the instance's public IP
address as well as where the instance has been launched from.
Keeping these basics in mind, there are also a few additional pieces of information
that you need to know about your instance's networking. You can control your
instance's IP address to a big extent depending on whether they are launched from a
standard EC2 environment or in a VPC.
In standard EC2 environment or as AWS calls it, EC2-Classic, you really don't have
much control over your instance's networking. Each instance is provided with a
single unique private as well as a public IP address and DNS, respectively. These
values are released to the general IP pool when your instances are either stopped
or terminated. You cannot reuse these IP addresses and DNS values once they are
released to the general pool.
AWS no longer provides the EC2-classic account since
April 12 2013.
On the other hand, a VPC provides much control and flexibility when it comes to
your instance's IP addressing. Using a VPC, you can define and run instances from
specially created subnets, which can either be isolated (private subnets) or connected
to the Internet (public subnets) depending on your requirements. You can additionally
provide your instances with more than one private and public IP address as well
using a VPC, something we will be looking at with great detail in the coming
chapter. Instances in a VPC, however, do not release their private IP addresses back
to the general pool when they are stopped.
[ 91 ]
Security, Storage, Networking, and Lots More!
Determining your instances IP addresses
AWS provides a few easy ways to determine your instance's IP addresses. The
simplest by far is using the Description tab from the
EC2 dashboard as shown here:
Select any particular running instance from the EC2 dashboard and view the
instance's Private DNS, Private IPs, Public DNS, and Public IP. Since my instances
are deployed in a VPC by default, you should see an additional row called
Secondary private IPs as well. These are the additional private IPs that you can
allocate to your instance as per your needs. If you don't see these additional rows,
then don't worry! You are probably running your instances from an EC2-Classic
account and that's fine for now.
Another way of listing your instance's network information is by using something
called instance metadata. Instance metadata is simply data about your instance.
Information such as your instance's AMI ID, instance's hostname, block device
mapping, network details, and a lot more can be obtained by querying against the
instance's metadata.
To know more about the various instance metadata
categories and how to use them, go to http://docs.
aws.amazon.com/AWSEC2/latest/UserGuide/ec2instance-metadata.html.
To determine your instance's IP addresses using instance metadata, simply connect
to your running instance and run the following command:
# curl http://169.254.169.254/latest/meta-data/local-ipv4
[ 92 ]
Chapter 4
You should receive your instance's private IP address, as shown in the following
screenshot. In case you are wondering what's up with the 169.254.169.254 IP
address, we'll try to keep it simple; it is a special-use IP address (also called a linklocal address) used by EC2 to distribute metadata to your instances.
Similarly, you can list your instance's public IP address by typing in the following
command in your instance:
# curl http://169.254.169.254/latest/meta-data/public-ipv4
You should receive your instance's Public IP address, as shown here:
Running a Windows instance? You can still query its instance
metadata by substituting curl with wget and running the
command in your Windows command prompt.
Feel free to dig around with instance metadata and list down your instance's
hostname, instance ID, security groups, and much more.
Working with Elastic IP addresses
Okay, so each of your instances receives a public and private IP address and in
standard normal circumstances these IP addresses do not persist with the instance
when it is powered off. But what if you want to assign a static IP address to your
instance? A static IP address that remains associated with your instance even if it
is powered off? In that case, you will need to use something called an Elastic IP
Address (EIP).
[ 93 ]
Security, Storage, Networking, and Lots More!
EIPs are nothing but a bunch of static public IP addresses that AWS allocates to your
account, not to your instances. Each AWS account can be associated with up to five
EIPs; however, you can always request AWS to provide additional ones as per your
requirements and needs by filling out a simple request form. Your EIPs will remain
associated with your AWS account until you choose to release them explicitly.
The really cool part of an EIP is that it can be reassigned to a different running
instance dynamically as and when needed. For example, let's consider our initial use
case, hosting a customer's website on AWS. As with all websites, this design calls
for a web server and a database server to begin with. Assume that we created and
allocated an EIP to the web server instance, as shown in the image here. This EIP can
then be mapped to a proper website name, such as all-about-dogs.com, using any
DNS service, such as AWS Route 53 and so on.
Now, if the web server instance undergoes any upgrades or maintenance activities,
you can simply create a new, similar web server instance and point your EIP to it.
Once the scheduled maintenance activity is over, simply swap the EIP back to the
previous web server instance. Simple, isn't it! When you add an EIP to your instance,
AWS automatically releases that instance's public IP address to the general IP pool.
On disassociating the EIP from your instance, AWS will once again provide your
instance with a new public IP address from the general IP pool. All this happens
really quickly, just a matter of minutes!
How is an EIP charged? Well, for the first EIP that you attach to a running instance,
you don't have to pay anything. However, you will need to shell out a minimum of
$0.005 per additional EIP for each instance on a per hourly basis.
[ 94 ]
Chapter 4
AWS imposes a small hourly charge (approx. $0.005) on
EIPs if they are attached to instances in a stopped state or
not associated with running instances. This is just to make
sure that the EIPs are used efficiently and not wasted.
Let's look at few simple steps using which you can create, associate, and disassociate
EIPs using the AWS Management Console!
Create an Elastic IP address
To create an Elastic IP address using the AWS Management dashboard, first login to
the dashboard using your IAM credentials and select the EC2 service option as EIPs
are a part of the EC2 services. Next, from the navigation pane, select the Elastic IPs
option. This will bring up the Elastic IP management dashboard as shown here. Since
this is going to be our first EIP, simply go ahead and select the Allocate New Address
option. In the confirmation dialog box, select Yes, Allocate to complete the process.
Your new Elastic IP is now ready to use! Remember, once again, that these Elastic IP
addresses are associated with your account and bear additional costs with them, so
use them wisely.
Allocating Elastic IP addresses
Once your EIP has been created, you can go ahead and allocate it to any running
instance from your current EC2 scope. Scope here can mean either EC2-Classic or
a VPC environment, depending on where your instances are currently deployed.
In my case, the instances are all running out of a VPC, so this particular EIP can be
associated with any instance currently running within my VPC. How do you tell
the scope of an EIP? Well, that's simple! Select the particular EIP and view its details
on the EIP management dashboard. You should see a column called Scope stating
whether you can deploy this EIP in a VPC or an EC2-Classic environment.
[ 95 ]
Security, Storage, Networking, and Lots More!
To allocate the EIP, select the EIP, and from the Actions tab, select the option
Associate Address, as shown:
You should see the Allocate New Address pop-up dialog box as shown. There are
two ways in which you can allocate your EIPs to your instances, either by providing
their Instance ID or by providing the instance's Network Interface information.
Provide the Instance's ID for now and leave the Network Interface option blank.
Optionally, you can even select the Reassociation checkbox if you wish to re-allocate
an EIP from one attached instance to a new instance.
You will receive a warning message informing you that associating an EIP to your
instance will release the current public IP attached to it. Accept the warning and
select the Associate tab to complete the EIP allocation process:
Verify whether the EIP was successfully attached to your instance or not by viewing
the status on the Elastic IP management dashboard.
[ 96 ]
Chapter 4
Disassociating and releasing an Elastic IP address
Disassociating an EIP from an instance is an equally important task and can be
performed quite easily using the EIP management dashboard. Select the particular
EIP from the dashboard and from the Actions tab. Then select the Dissociate
Address option. This will pop up a confirmation box detailing the EIP and its
associated instance ID information, as shown here. Select Yes, Disassociate to
complete the process:
On disassociation, AWS will automatically allocate a free and available public IP
address to the instance from its general IP pool. There is no guarantee that your
instance will receive the same public IP address as the instance had before the EIP
was added as these public IPs are always circulated and assigned on a random basis.
To release the EIP back to the pool, select the EIP from the dashboard. From the
Actions tab, select the Release Addresses option. You will be provided with a
confirmation box describing the current EIP address. Select Yes, Release to complete
the process, as shown:
[ 97 ]
Security, Storage, Networking, and Lots More!
Understanding EBS volumes
We briefly touched base on EBS volumes back in the previous chapter where we
were comparing EBS-backed and instance store-backed images. In this section, you
are going to learn a bit more about EBS volumes, their features, benefits, different
types, along with steps on how to create, attach, and delete them as well. So, what
are we waiting for? Let's get started!
First up, let's understand EBS volumes a bit better. EBS volumes are nothing more
than block-level storage devices that you can attach to your EC2 instances. They are
highly durable and can provide a host of additional functionalities to your instances,
such as data persistence, encryption, snapshotting capabilities, and so on. Majority
of the time, these EBS volumes are used for storing data for a variety of applications
and databases, however you can use it just as a normal hard drive as well. The best
part of EBS volumes is that they can persist independently from your instances. So
powering down an instance or even terminating it will not affect the state of your
EBS volumes. Your data will stay on it unless and until you explicitly delete it.
Let's look at some of the key features and benefits that EBS volumes have to offer:
•
High availability: Unlike your instance store-backed drives, EBS volumes
are automatically replicated by AWS within the availability zone in which
they are created. You can create an EBS volume and attach it to any instance
present in the same availability zone; however, one EBS volume cannot be
attached to multiple instances at the same time. A single instance, however,
can have multiple EBS volumes attached to it at any given time.
•
Encryption capabilities: EBS volumes provide an add-on feature using
which you can encrypt your volumes using standard encryption algorithms,
such as AES-256, and keys as well. These keys are autogenerated the first
time you employ encryption on a volume using the AWS Key Management
Service (KMS). You can additionally even use IAM to provide fine-grained
access control and permissions to your EBS volumes.
•
Snapshot capabilities: The state of an EBS volume can be saved using
point-in-time snapshots. These snapshots are all stored incrementally on
your Amazon S3 account and can be used for a variety of purposes, such as
creating new volumes based on an existing one, resizing volumes, backup
and data recovery, and so on.
EBS volumes cannot be copied from one AWS region to
another. In such cases, you can take a snapshot of the volume
and copy the snapshot over to a different region using the steps
mentioned at http://docs.aws.amazon.com/AWSEC2/
latest/UserGuide/ebs-copy-snapshot.html.
[ 98 ]
Chapter 4
EBS volume types
There are three different types of EBS volumes available today, each with their own
sets of performance characteristics and associated costs. Let's briefly look into each
one of them and their potential uses:
•
General purpose volumes (SSD): These are by far the most commonly used
EBS volume types as they provide a good balance between cost and overall
performance. By default, this volume provides a standard 3 IOPS per GB of
storage, so a 10 GB general purpose volume will get approximately 30 IOPS
and so on so forth, with a max value of 10,000 IOPS. You can create general
purpose volumes that can range in size from 1 GB to a maximum of 16 TB.
Such volumes can be used for a variety of purposes, such as instance root
volumes, data disks for dev and test environments, database storage, and so
on.
•
Provisioned IOPS volumes (SSD): These are a specialized set of SSDs that
can consistently provide a minimum of 100 IOPS burstable up to 20,000
IOPS. You can create Provisioned IOPS Volumes that range in size from a
minimum of 4 GB all the way up to 16 TB. Such volumes are ideally suited
for applications that are IO intensive, such as databases, parallel computing
workloads such as Hadoop, and so on.
•
Magnetic volumes: Very similar to traditional tape drives and magnetic
disks, these volumes are a good match for workloads where data is accessed
infrequently, such as log storage, data backup and recovery, and so on. On
an average, these volumes provide up to a 100 IOPS with an ability to burst
up to 1,000 IOPS. You can create Magnetic volumes that range in size from a
minimum of 1 GB all the way up to 1 TB.
Getting started with EBS Volumes
Now that we have a fair idea of what an EBS Volume is, let's look at some simple
ways that you can create, attach, and manage these volumes.
To view and access your account's EBS Volumes using AWS Management Console,
simply select the Volumes option from the EC2 dashboard's navigation pane, as
shown here:
[ 99 ]
Security, Storage, Networking, and Lots More!
This will bring up the Volume Management dashboard as shown here. In my case, I
already have a volume present here that is shown as in use. This is our first instance's
root device volume that we launched in Chapter 3, Images and Instances. Each EBSbacked instance's volume will appear here in the Volume Management dashboard.
You can use this same dashboard to perform a host of activities on your volumes,
such as create, attach, detach, and monitor performance, to name a few.
You can view any particular EBS Volume's details by simply selecting it and viewing
its related information in the Description tab, as shown. Here, you can view the
volume's ID, Size, Created date, the volume's current State as well as its Attachment
information, which displays the volume's mount point on a particular instance.
Additionally, you can also view the volume's health and status by selecting the
Monitoring and Status Checks tab, respectively. For now, let's go ahead and create a
new volume using the volume management dashboard.
Creating EBS volumes
From the Volume Management dashboard, select the Create Volume option. This
will pop up the Create Volume dialog box as shown here:
[ 100 ]
Chapter 4
Fill in the details as required in the Create Volume dialog box. For this tutorial, I
went ahead and created a simple 10-GB general purpose volume:
•
Type: From the Type drop-down list, select either General Purpose (SSD),
Provisioned IOPS (SSD), or Magnetic as per your requirements.
•
Size (GiB): Provide the size of your volume in GB. Here, I provided 10 GB.
•
IOPS: This field will only be editable if you have selected Provisioned
IOPS (SSD) as the volume's type. Enter the max IOPS value as per your
requirements.
•
Availability Zone: Select the appropriate availability zone in which you
wish to create the volume. Remember, an EBS volume can span availability
zones, but not regions.
•
Snapshot ID: This is an optional field. You can choose to populate your EBS
volume based on a third party's snapshot ID. In this case, we have left this
field blank.
•
Encryption: As mentioned earlier, you can choose whether or not you wish
to encrypt your EBS Volume. Select Encrypt this volume checkbox if you
wish to do so.
•
Master Key: On selecting the Encryption option, AWS will automatically
create a default key pair for the AWS's KMS. You can make a note of the
KMS Key ID as well as the KMS Key ARN as these values will be required
during the volume's decryption process as well.
Once your configuration settings are filled in, select Create to complete the volume's
creation process. The new volume will take a few minutes to be available for use.
Once the volume is created, you can now go ahead and attach this volume to your
running instance.
[ 101 ]
Security, Storage, Networking, and Lots More!
Attaching EBS volumes
Once the EBS volume is created, make sure it is in the available state before you go
ahead and attach it to an instance. You can attach multiple volumes to a single instance
at a time, with each volume having a unique device name. Some of these device names
are reserved, for example, /dev/sda1 is reserved for the root device volume. You can
find the complete list of potential and recommended device names at http://docs.
aws.amazon.com/AWSEC2/latest/UserGuide/device_naming.html.
The following screenshot shows the option of attaching a volume:
To attach a volume, select the volume from the Volume Management dashboard.
Then select the Actions tab and click on the Attach Volume option. This will pop up
the Attach Volume dialog box, as shown:
Type in your instance's ID in the Instance field and provide a suitable name in the
Device field as shown. In this case, I provided the recommended device name of /
dev/sdf to this volume. Click on Attach once done. The volume attachment process
takes a few minutes to complete. Once done, you are now ready to make the volume
accessible from your instance.
[ 102 ]
Chapter 4
Accessing volumes from an instance
Once the volume is attached to an instance, you can basically format it and use it like
any other block device. In this case, I'll be using the same Amazon Linux instance
that we created back in Chapter 3, Images and Instances. You can attach and mount
volumes to Windows-based instances as well.
To get started, first up connect to your running instance using putty or any other
SSH client of your choice. Next, type in the following command to check the current
disk partitioning of your instance:
# sudo df -h
You should see a /dev/xvda1 like filesystem mounted on the root (/) partition along
with few other temp filesystems, as shown here. This is more or less the standard
disk partitioning scheme that your instances will have.
Next, run the following command to list out partitions on your current instance.
You should see a default /dev/xvda partition along with its partition table and an
unformatted disk partition with the name /dev/xvdf as shown in the following
screenshot. The /dev/xvdf command is the newly added EBS volume that we will
need to format in the upcoming steps:
# sudo fdisk -l
[ 103 ]
Security, Storage, Networking, and Lots More!
Once you have verified the name of your newly added disk, you can go ahead and
format with a filesystem of your choice. In this case, I have gone ahead and used the
ext4 filesystem for my new volume:
# sudo mkfs -t ext4 /dev/xvdf
Now that your volume is formatted, you can create a new directory on your Linux
instance and mount the volume to it using your standard Linux commands:
# sudo mkdir /my-new-dir
# sudo mount /dev/xvdf /my-new-dir
Here is the screenshot of creating new directory using the preceding commands:
Here's a useful tip! Once you have mounted your new volume, you can optionally
edit the Linux instance's fstab file and add the volume's mount information there.
This will make sure that the volume is mounted and available even if the instance is
rebooted. Make sure you take a backup copy of the fstab file before you edit it, just
as a precautionary measure.
Detaching EBS volumes
Detaching EBS volumes is a fairly simple and straightforward process. You will first
need to unmount the volume from your instance and then simply detach it using
Volume Management dashboard.
Run the following command to unmount the EBS volume from the instance:
# sudo umount /dev/sdf
[ 104 ]
Chapter 4
Make sure you are unmounting the correct volume from
the instance. Do not try and unmount the /dev/sda or any
other root partitions.
Once the volume is successfully unmounted from the instance, detach the volume by
selecting the Detach Volume option from the Actions tab, as shown here:
Managing EBS volumes using the AWS CLI
You can create, attach, and manage EBS volumes using the AWS CLI as well. Let's
go ahead and create a new EBS volume using the AWS CLI. Type in the following
command:
# aws ec2 create-volume \
--size 5 --region us-west-2 --availability-zone us-west-2a \
--volume-type gp2
The --volume-type attribute accepts any one of these three values:
•
•
•
gp2: General Purpose instances (SSD)
io1: Provisioned IOPS volumes (SSD)
standard: Magnetic volumes
The following code will create a 5 GB General Purpose volume in the us-west-2a
availability zone.
[ 105 ]
Security, Storage, Networking, and Lots More!
The new volume will take a couple of minutes to be created. You should see a similar
output as the following screenshot. Make a note of the new volume's Volume ID
before proceeding to the next steps.
Now that the new volume is created, we can go ahead and attach it to our instance.
Type in the following command:
# aws ec2 attach-volume \
--volume-id vol-40993355 \
--instance-id i-53fc559a \
--device /dev/sdg
The following command will attach the volume with the volume ID vol-40993355
to our instance (i-53fc559a), and the device name will be /dev/sdg. Once again,
you can supply any meaningful device name here, but make sure that it abides by
AWS's naming conventions and best practices as mentioned in http://docs.aws.
amazon.com/AWSEC2/latest/UserGuide/device_naming.html.
Once the volume is attached to your instance, the next steps are pretty easy and
straightforward. First format the new volume with a suitable filesystem of your
choice. Next up, create a new directory inside your instance and mount the newly
formatted volume on it. Voila! Your volume is now ready for use.
You can detach and delete the volume as well using the AWS CLI. First up, we will
need to unmount the volume from the instance. To do so, type in the following
command in your instance:
# unmount /dev/sdg
[ 106 ]
Chapter 4
Make sure you are unmounting the correct volume and not the root partition. Once
the volume is unmounted, simply detach it from the instance using the following
AWS CLI code:
# aws ec2 detach-volume \
--volume-id vol-40993355
The output of the preceding command is as follows:
Finally, go ahead and delete the volume using the following AWS CLI code:
# aws ec2 delete-volume \
--volume-id vol-40993355
Remember that you cannot delete volumes if they are attached or in use by an
instance, so make sure that you follow the detachment process before deleting it.
Backing up volumes using EBS snapshots
We do know for a fact that AWS automatically replicates EBS volumes so that your
data is preserved even in case the complete drive fails. But this replication is limited
only to the availability zone in which the drive or EBS volume was created, which
means if that particular availability zone was to go down for some reason, then there
is no way for you to back up your data. Fortunately for us, AWS provides a very
simple yet highly efficient method of backing EBS volumes, called as EBS snapshots.
[ 107 ]
Security, Storage, Networking, and Lots More!
An EBS snapshot in simple terms is a state of your volume at a particular point in time.
You can take a snapshot of a volume anytime you want. Each snapshot that you take is
stored incrementally in Amazon S3, but, you will not be able to see these snapshots in
your S3 buckets; they are kind of hidden away and stored separately.
You can achieve a wide variety of tasks using snapshots. A few are listed as follows:
•
Create new volumes based on existing ones: Snapshots are a great and easy
way to spin up new volumes. A new volume spawned from a snapshot is an
exact replica of the original volume, down to the last detail.
•
Expand existing volumes: Snapshots can also be used to expand an existing
EBS Volume's size as well. It is a multistep process, which involves you
taking a snapshot of your existing EBS volume and creating a larger new
volume from the snapshot.
•
Share your volumes: Snapshots can be shared within your own account
(private) as well publicly.
•
Backup and disaster recovery: Snapshots are a handy tool when it comes to
backing up your volumes. You can create multiple replicates of an existing
volume within an AZ, across AZs that belong to a particular region, as well
as across regions, using something called an EBS Snapshot copy mechanism.
To create a snapshot of your volumes, all you need to do is select the particular
volume from the Volume Management dashboard. Click on the Actions tab and
select the Create Snapshot option, as shown here:
It is really a good practice to stop your instance before taking
a snapshot if you are taking a snapshot of its root volume.
This ensures a consistent and complete snapshot of your
volume at all times.
[ 108 ]
Chapter 4
You should see the Create Snapshot dialog box as shown in the following
screenshot. Provide a suitable Name and Description for your new snapshot.
An important thing to note here is that this particular snapshot is not supporting
encryption, but why? Well, that's simple! Because the original volume was not
encrypted, neither will the snapshot be encrypted. Snapshots of encrypted volumes
are automatically encrypted. Even new volumes created from an encrypted snapshot
are encrypted automatically. Once you have finished providing the details, click on
Create to complete the snapshot process:
You will be shown a confirmation box, which will display this particular snapshot's
ID. Make a note of this ID for future reference.
The new snapshot will take a good 3–4 minutes to go from Pending to Completed.
You can check the status of your snapshot by viewing the Status as well as the
Progress fields in the Description tab, as shown here:
[ 109 ]
Security, Storage, Networking, and Lots More!
Once the snapshot process is completed, you can use this particular snapshot
and Create Volume, Copy this snapshot from one region to another, and Modify
Snapshot Permissions to private or public as you see fit. These options are all
present in the Actions tab of your Snapshot Management dashboard:
But for now, let's go ahead and use this snapshot to create our very first AMI. Yes,
you can use snapshots to create AMIs as well. From the Actions tab, select the Create
Image option. You should see the Create Image from EBS Snapshot wizard as shown
here. Fill in the required details and click on Create to create your very first AMI:
The details contain the following options:
•
Name: Provide a suitable and meaningful name for your AMI.
•
Description: Provide a suitable description for your new AMI.
•
Architecture: You can either choose between i386 (32 bit) or x86_64 (64 bit).
[ 110 ]
Chapter 4
•
Root device name: Enter a suitable name for your root device volume.
Ideally, a root device volume should be labelled as /dev/sda1 as per EC2's
device naming best practices.
•
Virtualization type: You can choose whether the instances launched from
this particular AMI will support Paravirtualization (PV) or Hardware
Virtual Machine (HVM) virtualization.
You can read more about the various Virtualization
types supported by EC2 at http://docs.aws.
amazon.com/AWSEC2/latest/UserGuide/
virtualization_types.html.
•
RAM disk ID, Kernel ID: You can select and provide your AMI with its own
RAM disk ID (ARI) and Kernel ID (AKI); however, in this case I have opted
to keep the default ones.
•
Block Device Mappings: You can use this dialog to either expand your root
volume's size or add additional volumes to it. You can change the Volume
Type from General Purpose (SSD) to Provisioned IOPS (SSD) or Magnetic
as per your AMI's requirements. For now, I have left these to their default
values.
Click on Create to complete the AMI creation process. The new AMI will take a few
minutes to spin up. In the meantime, you can make a note of the new AMI ID from
the Create Image from EBS Snapshot confirmation box, as shown:
[ 111 ]
Security, Storage, Networking, and Lots More!
You can view your newly created AMI under the AMIs option from the EC2
dashboard's navigation pane:
So, here we have it! You very own AMI, created and ready to use.
An important point to note here is that you will not be able to delete this particular
EBS Snapshot now as it is in use by your AMI. You will have to deregister your AMI
first from the AMI Management dashboard and then try and delete the snapshot.
Planning your next steps
There are still a few important pieces that I would really recommend you try after
you have created your AMI. First up, try and launch a new instance from it. Once
the instance is launched, go ahead and check whether your instance has the correct
root partition name and size as allocated or not. Next up, try and copy your AMI
to a different region. You can refer to http://docs.aws.amazon.com/AWSEC2/
latest/UserGuide/CopyingAMIs.html for the required steps. Copying an AMI
from one region to another is just a simple way to build scalable and highly available
applications. You can try the same with your EBS volume as well. Go ahead and take
a snapshot of any volume of your choice and copy it over to some other AWS region
and attach it to a running instance.
Besides these steps, there is some additional EBS Volume related information
that I would really recommend you guys read. First up is something called
EBS-optimized instances. These are specially created instances that provided
dedicated throughput and IOPS for performance-intensive applications. This is an
add-on feature provided by AWS and is charged separately on an hourly basis. To
know more about EBS-Optimized Instances, go to http://docs.aws.amazon.com/
AWSEC2/latest/UserGuide/EBSOptimized.html.
[ 112 ]
Chapter 4
Secondly, I would recommend reading the EBS Performance tips that are provided
by AWS. These tips will help you analyze and benchmark your volumes for I/O
performance-intensive applications, configure RAID on your Linux instances, and
help you learn how to prewarm your EBS Volumes. These are all additional tips and
practices that you can choose to leverage in case you are working with production
environments and high-performance-intensive applications.
Another very interesting thing worth mentioning is public datasets. These are
basically really large repositories of publically available datasets such as the US
census data, transportation statistics, human genomic data, and so on. The whole
idea here is that AWS hosts these and provides these datasets for use completely free
of charge, so you don't spend hours of your time locating and downloading them.
Simply create a volume from any one of these public datasets which are in the form
of public snapshots and start analyzing them. Awesome, isn't it! You can read more
about public datasets at http://aws.amazon.com/public-data-sets/.
Recommendations and best practices
Here are a few handy recommendations and best practices to keep in mind when
working with volumes:
•
Create and use IAM policies and allow only a particular set of users from
accessing your EBS volumes.
•
Create and take periodic snapshots of your volumes. Always remember to
provide suitable names and descriptions for your snapshots so that they can
be easily identified and re-used.
•
Always take snapshots during the nonbusiness hours of your application.
•
Clean up unused or older snapshots to save on unnecessary costs.
•
Encrypt your EBS volumes if you have some sensitive data stored on them.
•
Select and use the correct type of EBS volume as per your application's needs.
Use performance-optimized volumes for your high-performance applications
and magnetic volumes for applications that do not need a lot of data read
and write.
[ 113 ]
Security, Storage, Networking, and Lots More!
Summary
We learned plenty of things in this chapter, so let's take a quick recap of the things
covered so far. First up, we learned how to edit and create Security Groups. Next, we
saw how our instances are provided with their networking, and we also saw how
to attach an Elastic IP address to our instance. Finally, we dived into the world of
EBS volumes and created, attached, detached, and deleted volumes using both the
AWS Management Console and the AWS CLI. Toward the end, we even created our
very first AMI and finally finished off with a set of key recommendations and best
practices.
[ 114 ]
Building Your Own Private
Clouds Using Amazon VPC
So, in the previous chapter, we covered a lot of different things! We started off with
some introductions and working examples of security groups and later on continued
with understanding how EC2 networking really works, with a brief look at Elastic IP
addresses. To top this off, we also learnt a lot about EBS volumes and their types and
how we can create, attach, and manage them.
This chapter, however, is going to be a lot different and interesting as in this chapter,
we will explore and learn about an awesome service provided by AWS called
Virtual Private Cloud (VPC)! We will learn about the different VPC concepts and
terminologies, deployment strategies, and a whole lot more, so stick around; we are
just getting started!
An overview of Amazon VPC
So far we have learnt a lot about EC2, its features, and uses, and how we can deploy
scalable and fault tolerant applications using it, but EC2 does come with its own
sets of minor drawbacks. For starters, you do not control the IP addressing of your
instances, apart from adding an Elastic IP address to your instance. By design, each
of your instances will get a single private and public IP address, which is routable on
the Internet—again, something you cannot control. Also, EC2 security groups have
the capability to add rules for inbound traffic only; there is no support for providing
any outbound traffic rules. So, although EC2 is good for hosting your applications, it
is still not that secure. The answer to all your problems is Amazon VPC!
[ 115 ]
Building Your Own Private Clouds Using Amazon VPC
Amazon VPC is a logically isolated part of the AWS cloud that enables you to build
and use your own logical subnets and networks. In a simpler sense, you get to build
your own network topology and spin up instances within it. But what actually
separates VPC from your classic EC2 environment is the ability to isolate and secure
your environment. Using VPCs, you can choose which instances are accessible over
the Internet and which are not. You can create a bunch of different subnets within
a single VPC, each with their own security policies and routing rules. VPCs also
provided an added layer of protection by enforcing something called as Network
Access Control Lists (ACLs) besides your standard use of security groups. This
ensures that you have total control over what traffic is routed in and out of your
subnets and the VPC as well.
VPCs also provide an added functionality using which you can connect and extend
your on-premise datacenters to the AWS cloud. This is achieved using an IPsec VPN
tunnel that connects from your on premise datacenter's gateway device to the VPC's
Virtual Private Gateway, as shown in the following image:
An important point to note here is that a VPC is still a part of the AWS Cloud. It is
not physically separate hosting provided by AWS, it simply is a logically isolated
part of the EC2 infrastructure. This isolation is done at the network layer and is very
similar to a traditional datacenter's network isolation; it's just that we as end users
are shielded from the complexities of it.
To know more about VPN and virtual private gateways, refer
to http://docs.aws.amazon.com/AmazonVPC/latest/
UserGuide/VPC_VPN.html.
[ 116 ]
Chapter 5
With this brief overview in mind, let's look at some of VPC's key concepts and
terminologies to get a better understanding of how things work.
VPC concepts and terminologies
By now you must have understood that VPC is nothing more than a network service
provided by AWS using which you can create logically isolated environments for
your EC2 instances. And just like any other network service, VPC too works on some
key concepts, explained as follows.
Subnets
Perhaps the most important part of the VPC, the subnets are nothing more than a
range of valid IP addresses that you specify. VPC provides you with two different
subnet creation options: a publically or Internet routed subnet called as a public
subnet and an isolated subnet called as a private subnet. You can launch your
instances in either of these subnets depending on whether you wish your instances
to be routed on the Internet or not.
How does it all work? Pretty simple! When you first create a VPC, you provide it
with a set of IP addresses in the form of a CIDR, for example, 10.0.0.0/16. The
/16 here indicates that this particular VPC can support up to 65,536 IP addresses
(2^(32-16) = 65,536, IP address range 10.0.0.0 - 10.0.255.255)! Now that's a lot! Once
the VPC's CIDR block is created, you can go ahead and carve out individual subnets
from it. For example, a subnet with the CIDR block 10.0.1.0/24 for hosting your
web servers and another CIDR block 10.0.5.0/24 for your database servers and so
on and so forth.
The idea here is that from the 65,536 IP address block, we carved out two subnets,
each supporting 256 IPs (/24 CIDR includes 256 IP addresses in it). Now you can
specify the subnet for the web servers to be public, as they will need to be routed to
the Internet and the subnet for the database servers to be private as they need to be
isolated from the outside world.
To know more about CIDRs and how they work,
refer to https://en.wikipedia.org/wiki/
Classless_Inter-Domain_Routing.
[ 117 ]
Building Your Own Private Clouds Using Amazon VPC
There is one additional thing worth mentioning here. By default, AWS will create
a VPC for you in your particular region the first time you sign up for the service.
This is called as the default VPC. The default VPC comes preconfigured with the
following set of configurations:
•
The default VPC is always created with a CIDR block of /16, which means it
supports 65,536 IP addresses in it.
•
A default subnet is created in each AZ of your selected region. Instances
launched in these default subnets have both a public and a private IP address
by default as well.
•
An Internet Gateway is provided to the default VPC for instances to have
Internet connectivity.
•
A few necessary route tables, security groups, and ACLs are also created by
default that enable the instance traffic to pass through to the Internet. Refer
to the following figure:
You can use this default VPC just as any other VPC by creating additional subnets in
it, provisioning route tables, security groups, and so on. In fact, the instances that we
launched back in Chapter 3, Images and Instances, were based out of the default VPC!
Any other VPC that you create besides the default VPC is called
as the non-default VPC. Each non-default VPC in turn contains
non-default subnets, and so on and so forth.
[ 118 ]
Chapter 5
Security groups and network ACLs
We have talked a lot about security groups in the past two chapters already. We
do know that security groups are nothing but simple firewall rules that you can
configure to safeguard your instances. You can create a maximum of 100 security
groups for a single VPC, with each Security Group containing up to 50 firewall
rules in them. Also, it is very important to remember that a Security Group does not
permit inbound traffic by default. You have to explicitly set inbound traffic rules to
allow traffic to flow to your instance. However, all outbound traffic from the instance
is allowed by default.
Network ACLs are something new. These provide an added security measure over
security groups as they are instance specific, whereas Network ACLs are subnet
specific. Unlike your security groups, however, you can both allow and restrict
inbound and outbound traffic using ACL rules. Speaking of ACL rules, they are very
much similar to your Security Group rules, however, with one small exception. Each
ACL rule is evaluated by AWS based on a number. The number can be anything
from 100 all the way up to 32,766. The ACL rules are evaluated in sequence starting
from the smallest number and going all the way up to the maximum value. The
following is a small example of how ACL rules look:
Inbound ACL rules
Rule
No.
Source IP
Protocol
Port
Allow/Deny
100
0.0.0.0/0
All
All
ALLOW
*
0.0.0.0/0
All
All
DENY
Outbound ACL rules
Rule
No.
Dest IP
Protocol
Port
Allow/Deny
100
0.0.0.0/0
all
all
ALLOW
*
0.0.0.0/0
all
all
DENY
These are the ACL rules created by AWS for your default VPC; as a result, this
particular ACL is called as the default Network ACL as well. What do these rules
mean? For starters, the rule number 100 for both the inbound and outbound ACL
specifies the traffic to flow from any protocol running on any port in and out of
the subnet. The * is also considered as a rule number and is a must in all ACLs. It
basically means that you drop any packets that do not match the ACL's rules. We
will be checking out ACLs and security groups in action later on in this chapter when
we create our very own VPC for the first time.
[ 119 ]
Building Your Own Private Clouds Using Amazon VPC
Routing tables
Route tables are pretty straightforward and easy to implement in a VPC. They are
nothing but simple rules or routes that are used to direct network traffic from a
subnet. Each subnet in a VPC has to be associated with a single route table at any
given time; however, you can attach multiple subnets to a single route table as well.
Remember the default VPC and the default subnets? Well, a similar default route
table is also created when you first start using your VPC. This default route table is
called as the main route table and it generally contains only one route information
that enables traffic to flow within the VPC itself. Subnets that are not assigned to any
route tables are automatically allocated to the main route table. You can edit and add
multiple routes in the main route table as you see fit; however, you cannot modify
the local route rule. The following an example of a main route table viewed from the
VPC Management dashboard:
As you can see, there are a couple of entries made in this table. The first is the local
route rule that allows traffic to flow within this particular subnet (10.0.0.0/16). The
second route is something called as a route for VPC endpoints. This is a private
connection made between your VPC and some other AWS service; in this case, the
service is S3. Let's look VPC endpoints a little closely.
VPC endpoints
VPC endpoints basically allow you to securely connect your VPC with other AWS
services. These are virtual devices that are highly available and fault tolerant by
design. They are scaled and managed by AWS itself, so you don't have to worry
about the intricacies of maintaining them. All you need to do is create a VPC
endpoint connection between your VPC and an AWS service of your choice, and
voila! Your instances can now communicate securely with other AWS services.
The instances in the VPC communicate with other services using their private IP
addresses itself, so there's no need to route the traffic over the Internet.
[ 120 ]
Chapter 5
AWS currently only supports VPC endpoint connections for
Amazon S3. More services are planned to be added shortly.
When an endpoint is created, you first need to select either of your VPC's route
tables. The traffic between your VPC instances and the AWS service will be routed
using this particular route table. Similar to any other route information, a VPC
endpoint route also contains a Destination field and a Target field. The Destination
filed contains the AWS service's prefix list ID, which is generally represented by
the following ID: pl-xxxxxxxx. The Target field contains the endpoint ID, which is
represented in the following format: vpce-xxxxxxxx.
In the following route table example, the prefix list ID (pl-68a54001) represents the S3
service whereas the target vpce-80cd2be9 represents the endpoint ID:
VPC Endpoint Route
Destination
Target
10.0.0.0/16
Local
pl-68a54001
vpce80cd2be9
Endpoints also provided an additional feature using which you can control and
govern access to the remote AWS service. This is achieved using something called as
endpoint policies.
Endpoint policies are nothing more than simple IAM-based resource policies that
are provided to you when an endpoint is first created. AWS creates a simple policy
document that allows full access to the AWS service by default. The following is a
sample endpoint policy that is created by AWS for the S3 service:
{
"Statement": [
{
"Action": "*",
"Effect": "Allow",
"Resource": "*",
"Principal": "*"
}
]
}
To know more about the endpoint policies and features, refer to http://docs.aws.
amazon.com/AmazonVPC/latest/UserGuide/vpc-endpoints.html.
[ 121 ]
Building Your Own Private Clouds Using Amazon VPC
Internet Gateways
Internet Gateways, as the name suggest, are primarily used to provide Internet
connectivity to your VPC instances. All you have to do is create and attach an
Internet Gateway device to your VPC and add a route entry in your public subnet's
route table to point to the Internet Gateway! That's it! The default VPC comes with
an Internet gateway already deployed in it. So, any instance that you launch from
the default subnet obtains Internet connectivity automatically. This does not apply
for non-default VPCs, however, as an instance launched in a non-default subnet does
not receive a public IP address by default. You would have to either assign one to
the instance during the launch phase or modify your non-default subnet's public IP
address attributes.
Once you have created and attached an Internet Gateway to your VPC, you will also
have to make sure that the public subnet's route table has an entry for this gateway.
Plus, you will also have to create the correct set of security groups and network ACL
rules to allow your subnet's traffic to flow through the Internet. The following is an
example of a VPC's route table showing the route for a subnet's traffic to the Internet
Gateway (igw-8c3066e9):
Besides the Internet connectivity, Internet Gateways also perform NAT on the
instance's private IPs. The instances in a subnet are only aware of their private IP
addresses that they use to communicate internally. The Internet Gateway maps the
instance's private IP with an associated public or Elastic IP and then routes traffic
outside the subnet to the Internet. Conversely, the Internet Gateway also maps
inbound traffic from the Internet to a public or Elastic IP and then translates it to the
instance's private IP address. This is how your instances receive Internet from within
a VPC, which brings us to yet another interesting topic called as NAT instances.
[ 122 ]
Chapter 5
NAT instances
So, we have just learnt that the Internet Gateway NATs the IP addresses of instances
placed out in the public subnet so that they can communicate with the Internet, but
what about instances in the private subnets? How do they communicate with the
Internet without having direct Internet connectivity via the gateway?
That's where a NAT instance comes into play. A NAT Instance is a special instance
created inside your public subnet that NATs outbound traffic from instances based
in your private subnet to the Internet. It is important to note here that the NAT
instance will only forward the outbound traffic and not allow any traffic from the
Internet to reach the private subnets, similar to a one way street.
You can create a NAT Instance out of any AMI you wish; however, AWS provides
few standard Linux-based AMIs that are well suited for such purposes. These special
AMIs are listed out in the community AMIs page and all you need to do is filter out
the amzn-ami-vpc-nat AMI from the list and spin up an instance from it.
The following example depicts the traffic flow from a private subnet (10.0.1.0/24) to
the NAT instance inside a public subnet (10.0.0.0/24) via a route table:
[ 123 ]
Building Your Own Private Clouds Using Amazon VPC
In the preceding example, outbound traffic from the public subnet's route table is
routed to the Internet Gateway (igw-8c3066e9) while the outbound traffic from the
private subnet's route table is routed to the NAT instance. Along with the route
tables, it is also essential that you correctly populate the Security Group for your
NAT instance. The following is a simple NAT instance Security Group example for
your reference:
NAT instance - inbound security Rules
Source
Protocol
Port
Remarks
10.0.1.0/24
TCP
80
Permit inbound HTTP traffic from private subnet
10.0.1.0/24
TCP
443
Permit inbound HTTPS traffic from private subnet
*
TCP
22
Permit SSH login to NAT instance from remote N/W
The * replace the field with the IP address of your
local desktop machine.
The following are the outbound security rules:
NAT Instance - outbound security rules
Source
Protocol
Port
Remarks
0.0.0.0/0
TCP
80
Permit HTTP access to Internet for the NAT instance
0.0.0.0/0
TCP
443
Permit HTTPS access to Internet for the NAT instance
DNS and DHCP Option Sets
VPCs provide an additional feature called as DHCP Option Sets using which you
can set and customize the DNS and DHCP for your instances. The default VPC
comes with a default DHCP Options Set that is used to provide the instances with
a dynamic private IP address and a resolvable hostname. Using the DHCP Options
Set, you can configure the following attributes for your VPC:
•
Domain Name Servers (DNS): You can list down up to four DNS servers
here of your own choice or even provide the Amazon DNS server details.
The Amazon DNS server is provided in your VPC and runs on a reserved
IP address. For example, if your VPC has the subnet of 10.0.0.0/16,
then the Amazon DNS Server will probably run on the IP 10.0.0.2.
You can additionally provide the Amazon DNS Server's IP address,
169.254.169.253, or the value AmazonProvidedDNS as required. Values
entered here are automatically added to your Linux instances /etc/resolv.
conf file for name resolution.
[ 124 ]
Chapter 5
Domain name: You can either provide your own domain name value or choose
to use the default AWS domain name values using this option. The default AWS
domain names can be provided only if you have selected AmazonProvidedDNS
as your DNS server. For example, instances launched in the US West region with
the Amazon DNS server value will get a resolvable private DNS hostname as uswest-2.compute.internal.
•
NTP servers: You can list up to four NTP server IP addresses using the
DHCP Options Set wizard. Note, however, that this will only accept IP
address values and not FQDNs such as pool.ntp.org.
•
NetBIOS name server: You can list down up to four NetBIOS name servers
as well; however, this field is optional.
•
NetBIOS node type: You can specify the NetBIOS node value, which can
either be 1, 2, 4, or 8. AWS recommends that you specify 2 as broadcast, and
multicasts are not currently supported.
You can create and attach only one DHCP option set with a VPC at a time. AWS uses
the default DHCP option if you do not specify one explicitly for your VPC. Instances
either running or newly launched will automatically pick up these DNS and DHCP
settings, so there is no need for you to restart or relaunch your existing instances.
VPC limits and costs
Okay, so far we have understood a lot about how the VPC works and what its
components are, but what is the cost of all this? Very simple, it's nothing! VPC is a
completely free of cost service provided by AWS; however, you do have to pay for
the EC2 resources that you use, for example, the instances, the Elastic IP addresses,
EBS volumes, and so on. Also, if you are using your VPC to connect to your on
premise datacenter using the VPN option, then you need to pay for the data transfers
over the VPN connection as well as for the VPN connection itself. AWS charges $0.05
per VPN connection hour.
Besides this, VPCs also have a few limits set on them by default. For example, you
can have a maximum of five VPCs per region. Each VPC can have a max of one
Internet gateway as well as one virtual private gateway. Also, each VPC has a limit
of hosting a maximum of up to 200 subnets per VPC. You can increase these limit
by simply requesting AWS to do so. To view the complete list of VC limits, refer to
http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Appendix_
Limits.html.
[ 125 ]
Building Your Own Private Clouds Using Amazon VPC
Working with VPCs
Enough talk! It's time to get into some action! In this section, we are going to look
at how AWS facilitates the easy deployment of VPCs using something called as the
VPC wizard as well as how to create your very first, fully fledged and operational
VPC! So, what are we waiting for? Let's get started!
VPC deployment scenarios
VPC provides a simple, easy-to-use wizard that can spin up a fully functional VPC
within a couple of minutes. All you need to do is select a particular deployment
scenario out of the four scenarios provided and configure a few basic parameters
such as subnet information, availability zones in which you want to launch your
subnets, and so on, and the rest is all taken care of by AWS itself.
Let's have a quick look at the four VPC deployment scenarios:
•
VPC with a single public subnet: This is by far the simplest of the four
deployment scenarios. Using this scenario, VPC will provision a single public
subnet with a default Internet Gateway attached to it. The subnet will also
have a few simple and basic route tables, security groups, and network ACLs
created. This type of deployment is ideal for small-scaled web applications or
simple websites that don't require any separate application or subnet tiers.
•
VPC with public and private subnets (NAT): Perhaps the most commonly
used deployment scenario, this option will provide you with a public subnet
and a private subnet as well. The public subnet will be connected to an
Internet gateway and allow instances launched within it to have Internet
connectivity, whereas the private subnet will not have any access to the
outside world. This scenario will also provision a single NAT instance inside
the public subnet using which your private subnet instances can connect with
the outside world but not vice versa. Besides this, the wizard will also create
and assign a route table to both the public and private subnets, each with the
necessary routing information prefilled in them. This type of deployment
is ideal for large-scale web applications and websites that leverage a mix of
public facing (web servers) and non-public facing (database servers).
[ 126 ]
Chapter 5
VPC with public and private subnets and hardware VPN access: This deployment
scenario is very much similar to the VPC with public and private subnets, however,
with one component added additionally, which is the Virtual Private Gateway. This
Virtual Private Gateway connects to your on premise network's gateway using a
standard VPN connection. This type of deployment is well suited for organizations
that wish to extend their on premise datacenters and networks in to the public clouds
while allowing their instances to communicate with the Internet.
•
VPC with a private subnet only and hardware VPN access: Unlike the
previous deployment scenario, this scenario only provides you with a private
subnet that can connect to your on premise datacenters using standard VPN
connections. There is no Internet Gateway provided and thus your instances
remain isolated from the Internet. This deployment scenario is ideal for cases
where you wish to extend your on premise datacenters into the public cloud but
do not wish your instances to have any communication with the outside world.
With this understanding, let's go ahead and deploy our very first, fully functional
VPC using the VPC wizard!
Getting started with the VPC wizard
Before we go ahead and deploy our VPC, let's first have a quick look at our use case.
We need to create a secure website hosting environment for our friends at AllAbout-Dogs.com, complete with the following requirements:
•
Create a VPC (US-WEST-PROD-1 - 192.168.0.0/16) with separate secure
environments for hosting the web servers and database servers.
•
Only the web servers environment (US-WEST-PROD-WEB - 192.168.1.0/24)
should have direct Internet access.
•
The database servers environment (US-WEST-PROD-DB - 192.168.5.0/24)
should be isolated from any direct access from the outside world.
•
The database servers can have restricted Internet access only through a jump
server (NAT Instance). The jump server needs to be a part of the web server
environment.
•
The web servers environment should full have access to Amazon S3.
[ 127 ]
Building Your Own Private Clouds Using Amazon VPC
The following is what the proposed environment should look like:
To get started with VPC, we first have to log in to the AWS Account using your IAM
credentials. Next, from the AWS Management Console, select the VPC option from
under the Networking group, as shown in the following screenshot:
This will bring up the VPC Dashboard using which you can create, manage, and
delete VPCs as per your requirements. The VPC dashboard lists the currently deployed
VPCs, Subnets, Network ACLs, and much more under the Resources section.
You can additionally view and monitor the health of your VPC service by viewing
the status provided by the Service Health dashboard, as shown in the following
screenshot. In my case, I'm operating my VPC out of the US West (Oregon) region.
[ 128 ]
Chapter 5
The VPC Dashboard also lists any existing VPN connections that you might have set
up earlier. You can view the VPN Connections, Customer Gateways information
as well as the Current Status of the VPN connection by using this dashboard.
Remember that a VPN connection has a cost associated with it when it is provisioned
and in the available state. You can additionally use this dashboard and even launch
your instances directly into a VPC using the Launch EC2 Instances option. These
instances will most probably be launched in your default VPC in case you haven't
already created another one.
With all this said and done, let's go ahead and create our VPC using the VPC Wizard.
Select the Start VPC Wizard option. The wizard is a simple two-step process that
will guide you with the required configuration settings for your VPC. You will be
prompted to select any one out of the four VPC scenarios, so with our use case in
mind, go ahead and select the VPC with Public and Private Subnets option. Do
note that this will create a /16 network with two /24 subnets by default, one public
subnet and the other a private subnet. You can always create more subnets as
required once the VPC is created.
Also worth mentioning here is that this VPC scenario will create and launch a NAT
instance as well in the public subnet. This instance will be powered on automatically
once your VPC is created, so be aware about its existence and power it down unless
you want to get charged for it.
This NAT instance launched by the wizard does not support
the t2.micro instance type (Free Tier eligibility) during launch;
however, you can always change this once your instance is
launched from the EC2 Management dashboard.
[ 129 ]
Building Your Own Private Clouds Using Amazon VPC
The second step of the wizard is where you get to configure your VPC network and
subnets. Fill in the following details as required:
•
IP CIDR block: Provide the IP CIDR block address for your VPC's network.
Ideally, provide a /16 subnet that will provide you with a good 65,531 IP
addresses to use.
•
VPC name: Provide a suitable name for your VPC. In this case, I have
standardized and used the following naming convention: --; so in our case, this translates to USWEST-PROD-1.
•
Public subnet: Now, since we are going with a public and private subnet
combination scenario, we have to fill in our public subnet details here.
Provide a suitable subnet block for your instances to use. In this case, I have
provided a /24 subnet which provides a good 251 usable IP addresses:
•
Availability Zone: Here's the fun part! You can deploy your subnets in any
availability zone available in that particular region. Now, US-WEST (Oregon)
has three AZs and you can use any of those there. In my case, I have gone
ahead and selected us-west-2a as the default option.
•
Public subnet name: Provide a suitable public subnet reference name. Here,
too, I have gone ahead and used the standard naming convention, so this
particular subnet gets called as US-WEST-PROD-WEB, signifying the web
server instances that will get deployed here.
[ 130 ]
Chapter 5
•
Private subnet, Availability zone, Private subnet name: Go ahead and fill
out the private subnet's details using the similar IP addressing and naming
conventions. Remember that although you can set up your private subnet
in a different AZ, as compared to the public subnet, ideally doing that is
not recommended. If you really want to set up a failover-like scenario, then
create a separate public and private subnet environment in a different AZ
altogether, for example, us-west-2c. So, even in case us-west-2a suffers
an outage, which by the way can happen, your failover subnets will still be
functioning out of the us-west-2c AZ. Refer to the following screenshot:
Next up, we specify the details of our NAT instance:
•
Instance type: Select your NAT instance type from the available dropdown
menu. In my case, I have gone ahead and selected the t2.micro instance
type as that is the smallest type available. Do remember that selecting any
other option will incur additional costs as only the t2.micro instance type is
covered under the free tier eligibility.
•
Key pair name: Select an already existing key pair from the dropdown
list. Make sure you have this particular key pair stored safely on your local
computer, as without it you will not be able to SSH into the NAT instance.
You can alternatively create a new key pair here as well using the same EC2
Management Console.
[ 131 ]
Building Your Own Private Clouds Using Amazon VPC
Moving on, we now add the S3 endpoints to our particular subnet. You can add
the endpoint to either your private or public subnets, or both of them, depending on
your requirements.
•
Subnet: As per our VPC's requirements, the S3 endpoint is only made
available to the public subnet, so go ahead and select the Public subnet
option from the dropdown list.
•
Policy: You can either choose to allow any user or service within the newly
created VPC to access your S3 or specify a custom IAM access policy as you
see fit. In our case, let's go ahead and select Full Access for now.
•
Enable DNS hostnames: Enabling this option will provide your instances
with the ability to resolve their DNS hostnames on the Internet. Select the Yes
option and continue.
•
Hardware tenancy: Although a VPC runs off a completely isolated network
environment, the underlying server hardware is still shared by default. You
can change this tenancy option by selecting either the default or dedicated
option from the dropdown list provided.
Pricing for a dedicated instance is slightly different than your
traditional EC2 instances. Check out the complete pricing details
for a dedicated EC2 instance at http://aws.amazon.com/
ec2/purchasing-options/dedicated-instances/.
Once all the required information is filled out, go ahead and click on the Create
VPC option. The VPC creation takes a few seconds to complete. You can even view
your new VPC's default routes, security groups, and Network ACLs being created.
Toward the end you will notice your NAT instance powering on, and after a few
seconds of deployment your VPC is now ready for use!
Here's a handy tip for all first timers! As soon as your NAT instance is created,
you can go ahead and change its default instance type to t1.micro from the EC2
Management Dashboard.
[ 132 ]
Chapter 5
To do so, first open up the EC2 Management Dashboard in a new tab on your
browser. You should see an instance in the running state, as shown in the following
screenshot. First up, stop your instance using the Actions tab. Select the Instance
State option and click on Stop. Wait for the instance state to change to Stop before
you proceed any further. Next, select the instance, and from the Actions tab, select
Instance Settings and then Change Instance Type.
From the Change Instance Type dialog box, select the t1.micro option and click on
Apply. Voila! Your NAT instance is now officially a part of your free tier as well!
Simple, isn't it!
Let's have a quick look at what actually happens behind the scenes when you create
a VPC using the VPC Wizard. First up, let's look at the VPC itself.
Viewing VPCs
Once the VPC is created and ready, you can view it from the VPC Management
Dashboard as well. Simply select the Your VPCs option from the Navigation pane
provided on the left-hand side of the dashboard.
[ 133 ]
Building Your Own Private Clouds Using Amazon VPC
You should see your newly created VPC, as shown in the following screenshot. You
can use the search bar to filter out results as well. Select the particular VPC to view
its details.
Use the Summary tab to view the description of your selected VPC. Here, you
can view the VPC's default DHCP options set as well as the Tenancy and DNS
hostnames and DNS resolution options. You can optionally change these values by
selecting the particular VPC and from the Actions tab, selecting either Edit DHCP
Options Set or Edit DNS Hostnames.
You can view your default VPC using the VPC Management
Dashboard as well. Simply check the default VPC column
against your listed VPCs. If the value in that column is Yes,
then that particular VPC is your account's default VPC!
You can create additional VPCs as well using the Create VPC option; however, as a
good practice, always keep things simple and minimal. Don't go over creating VPCs.
Rather use and create as many subnets as you require. Speaking of subnets, let's have
a look at newly created VPC's two subnets!
[ 134 ]
Chapter 5
Listing out subnets
You can view, add, and modify existing subnets using the VPC Management
Dashboard as well. Simply select the Subnets option from the Navigation Pane.
This will list out all the subnets present in your account, so use the search bar to filter
out the new ones that we just created. Type in the name of the subnet in the Search
bar until the particular subnet gets listed out, as shown in the following screenshot:
You can view additional information associated with your subnet by simply selecting
it and viewing the Summary tab. The Summary tab will list out the particular
subnet's associated Route table, Network ACL, CIDR, and State as well. Besides
these values, you can also configure your subnet's ability to auto assign public IPs to
its instances. By default, this feature is disabled in your subnet, but you can always
enable it as per your requirements. To do so, simply select your Public Subnet, and
from the Subnet Actions tab, select the option, as shown in the following screenshot:
In the Modify Auto-Assign Public IP dialog box, simply select the Enable auto-assign
public IP option and click on Save to complete the change setting. Do note that you
can always override this behavior for each individual instance at its launch time.
[ 135 ]
Building Your Own Private Clouds Using Amazon VPC
Besides the Summary tab, the Subnet Dashboard option also provides additional
tabs such as Route Table, Network ACL, and so on. You can use these tabs to view
the subnet's associated route table as well the network ACL; however, you cannot
add or modify the individual rules from here. To add or modify rules, you need
to go to the Network ACL option or the Route Tables option from the navigation
pane. Let's have a quick look at the route tables created for our VPC.
Working with route tables
As discussed earlier in this chapter, VPCs come with a default route table (Main
Route Table) associated with a subnet. So, since we have two subnets created in this
VPC, we get two route tables as well, out of which one is the main route table. How
do you tell whether a route table is the main one? Quite simple, actually! Just look
for the Main Column in the Route Table Dashboard, as shown in the following
screenshot. If the value in that column is Yes, then that particular route table is your
VPC's main route table. Now, here's a catch! If you do not explicitly associate a
route table with a subnet, then the subnet ends up using the main route table. In our
case, both the route tables created do not have any subnets associated with them by
default, so let's first get that done.
To associate a route table with a subnet explicitly, first you need to select the
particular route table from the dashboard. In this case, I have selected the public
subnet's router (US-WEST-PROD-WEB-RT). Next, from the Subnet Associations
tab, click on the Edit button, as shown in the following screenshot. From here, you
can select either of the two subnets that are listed down; however, since this is a
public subnet's route table, let's go ahead and select the listed public subnet
(US-WEST-PROD-WEB) as shown. Click on Save to save the configuration changes.
[ 136 ]
Chapter 5
With this step completed, your subnet is now explicitly attached with a particular
route table. But what about the individual route rules? How do we list and modify
them? That's simple as well. Simply select the Routes tab to list your route table's
existing rule set. Here, you should see at least three route rules, as shown in the
following screenshot. The first rule is created by VPC for each and every route table,
and it basically allows communication within the VPC itself. You cannot delete this
rule, so don't even try it!
The next rule is basically a VPC endpoint route rule. Remember the S3 endpoint
that we configured earlier with the public subnet? Well this rule will basically allow
communication to occur between the instances belonging to this subnet and Amazon
S3, and the best part is that this rule is auto-populated when you create a VPC endpoint!
The final rule in the list basically allows for the instances to communicate over the
Internet using the Internet Gateway as the target. You can optionally choose to
edit these rules by selecting the Edit option. Once you have made your required
changes, be sure to Save the configuration changes before you proceed with the next
steps. Don't forget to associate the private subnet (US-PROD-WEST-DB) with the
remaining route table (US-WEST-PROD-DB-RT) as well.
Listing Internet Gateways
As discussed earlier in this chapter, Internet Gateways are scalable and redundant
virtual devices that provide Internet connectivity for your instances present in
the VPC. You can list currently available Internet Gateways within your VPC by
selecting the Internet Gateways option from the VPC's navigation pane.
[ 137 ]
Building Your Own Private Clouds Using Amazon VPC
The VPC wizard will create and attach one Internet Gateway to your VPC
automatically; however, you can create and attach an Internet Gateway to a VPC at
any time using the Internet Gateway. Simply click on the Create Internet Gateway
option and provide the VPC to which this Internet Gateway has to be attached, that's
it! You can list down available Internet Gateways and filter the results using the
search bar provided as well. To view your Internet Gateway's details, simply select
the particular Internet Gateway and click on the Summary tab. You should see your
Internet Gateway's ID, State (attached or detached), as well as the Attachment state
(available or not available), as shown in the following screenshot:
Also remember that to really use the Internet Gateway, your public subnet's route
table must contain a route rule that directs all Internet-bound traffic from the subnet
to the Internet Gateway.
The Internet-bound route rule is auto-populated by the VPC
Wizard when we first configured the VPC and can be viewed
in the Public Subnet's route table.
Working with security groups and Network ACLs
The VPC is all about providing your applications a much more secure environment
than what your traditional EC2 service can offer. This security is provided in two
layers in the form of security groups and Network ACLs. The security groups can
be used to set rules that can control both inbound and outbound traffic flow from
the instance and hence work more at the instance level. The Network ACLs on the
other hand operate at the subnet level, either allowing or disallowing certain type of
traffic to flow in and out of your subnet. Important thing to remember here is that the
Network ACLs are actually optional and can be avoided altogether if your security
requirements are at a minimal. However, I would strongly recommend that you use
both the security groups and Network ACLs for your VPC environments. As the
saying goes - better safe, than sorry!
[ 138 ]
Chapter 5
Coming back to your newly created VPC, the VPC wizard creates and populates a
default Security Group and a default Network ACL option for you to use in an as-is
condition. The default Security Group has a single inbound and outbound rule, as
explained in the following:
Default inbound security rule
Source
Protocol
Port Range
Remarks
Security_Group_ID
All
All
Permits inbound traffic from instances
belonging to the same security group
Default Outbound Security Rule
Destination
Protocol
Port Range
Remarks
0.0.0.0/0
All
All
Permits all outbound traffic from the instances
You can add, edit, and modify the rules in the default Security Group; however,
you cannot delete it. As a good practice, it is always recommended that you do not
use this default Security Group but rather create your own. So, let's go ahead and
create three security groups: one for the web servers in the public subnet, one for the
database servers in the private subnet, and one for the specially created NAT Instance.
To create a new Security Group using the VPC Dashboard, select the Security
Groups option from the navigation pane. Next, from the Security Groups
dashboard, select Create Security Group, as shown in the following screenshot:
Using the Create Security Group wizard, fill in the required information as
described in the following:
•
Name tag: A unique tag name for your Security Group.
•
Group name: A suitable name for your Security Group. In this case, I have
provided it as US-WEST-PROD-WEB-SG.
•
Description: An optional description for your security group.
[ 139 ]
Building Your Own Private Clouds Using Amazon VPC
•
VPC: Select the newly created VPC from the dropdown list, as shown in the
following screenshot. Click on Yes, Create once done.
Once your Security Group has been created, select it from the Security Groups
dashboard and click on the Inbound Rules tab. Click on the Edit option to add the
following rule sets:
Web server inbound security rule
Source
Protocol
Port Range
Remarks
0.0.0.0/0
TCP
22
Permit inbound SSH access to web server instance
0.0.0.0/0
TCP
80
Permit inbound HTTP access to web server instance
0.0.0.0/0
TCP
443
Permit inbound HTTPS access to web server instance
Similarly, click on the Outbound Rules tab and fill out the Security Group's
outbound rules as described in the following:
Web server outbound security rule
Destination
Protocol
Port Range
Remarks
DB_SECURITY_GROUP
TCP
1433
Permits outbound Microsoft SQL
Server traffic to the database servers
DB_SECURITY_GROUP
TCP
3306
Permits outbound MySQL traffic to
the database servers
[ 140 ]
Chapter 5
Replace DB_SECURITY_GROUP with the Security Group ID of your database
server's Security Group. Remember to save the rules by selecting the Save option,
as shown in the following screenshot:
Similarly, let's go ahead and create a Security Group for our database servers as well.
Populate the inbound rules as described in the following:
Database server inbound security rule
Source
Protocol
Port Range
Remarks
WEB_SECURITY_GROUP
TCP
1433
Permits Web Server instances to
access the Microsoft SQL Server
WEB_SECURITY_GROUP
TCP
3306
Permits Web Server instances to
access the MySQL Server
Replace WEB_SECURITY_GROUP with the Security Group ID of your web server's
Security Group ID and save the rules before you continue with the outbound rules
additions:
Database server outbound security rule
Source
Protocol
Port Range
Remarks
0.0.0.0/0
TCP
80
Permit outbound HTTP access to database server
instance
0.0.0.0/0
TCP
443
Permit outbound HTTPS access to database
server instance
Note that here we are permitting only the outbound Internet access to the database
servers so that they can receive important patches and updates from the net. In
reality, the Internet bound traffic from these servers will be routed through the NAT
instance, which will forward the traffic to the Internet via your Internet Gateway.
[ 141 ]
Building Your Own Private Clouds Using Amazon VPC
Finally, go ahead and create the NAT instance's Security Group. Populate the
inbound security rules as mentioned in the following:
NAT instance inbound security rule
Source
Protocol
Port Range
Remarks
0.0.0.0/0
TCP
22
Permits inbound SSH access to the NAT Instance
192.168.1.0/24
TCP
80
Permit inbound HTTP access to the NAT instance
192.168.1.0/24
TCP
443
Permit inbound HTTPS access to NAT instance
NAT instance outbound security rule
Source
Protocol
Port Range
Remarks
0.0.0.0/0
TCP
80
Permit outbound HTTP access to NAT instance
0.0.0.0/0
TCP
443
Permit outbound HTTPS access to NAT instance
With the security groups created, you are now ready to launch your instances into
the VPC. Let's have a quick look at the steps required to do so!
Launching instances in your VPC
Once your VPC is ready and the security groups and Network ACLs have been
modified as per requirement, you are now ready to launch instances within
your VPC. You can either launch instances directly from the VPC Management
Dashboard or from the EC2 Management Console as well. In this case, let's go
ahead and use the EC2 Management Console.
Creating the web servers
From the EC2 Management Console, select the Launch Instance option. This will
bring up the Instance Wizard, using which you can create and launch your web
server instances. In my case, I'm using the AMI (US-P-WebServer-Image-v1.0) that
we created earlier in Chapter 4, Security, Storage, Networking, and Lots More! Click on
My AMIs and you should see the custom-created AMI listed there, as shown in the
following screenshot. Select the AMI and continue with the instance creation process.
[ 142 ]
Chapter 5
From the next page, select any instance type for the new web server instances. In my
case, I went ahead and used the default t1.micro instance type.
Next, from the Configure Instance Details page, select the newly created VPC (USWEST-PROD-1) from the Network dropdown list and provide the web server's
public subnet (US-WEST-PROD-WEB), as shown in the following screenshot. You
can optionally choose to change the Auto-assign Public IP setting; however, in this
case, make sure that this setting is set to Enable otherwise your web server instances
will not receive their public IPs.
Add the required Storage, Tag the instance, and provide it with the web server
Security Group that we created a while back. Once you have completed the
formalities, review your instance's settings and finally launch it in your VPC!
[ 143 ]
Building Your Own Private Clouds Using Amazon VPC
Once the instance starts, verify whether it received both the private and the public IP
or not. Log in to your web server instance and check whether it can reach the Internet
or not by simply pinging to one of Google's DNS servers like 8.8.8.8. If all goes well,
then your web server instances are all ready for production use!
Creating the database servers
The same process applies for the database servers as well. Simply remember to select
the correct subnet (US-WEST-PROD-DB) for the database servers, as shown in the
following screenshot:
Also note the Auto-assign Public IP setting for the database server's private subnet.
By default, this should be disabled for the private subnet as we don't want our
database instances to communicate with the Internet directly. All Internet-bound
traffic from the database servers will pass via the NAT instance only. But how do
you test whether your database servers are working correctly? By design, you cannot
SSH into the database servers directly from your local desktops as the private subnet
is isolated from the Internet. So, an alternative would be to set up something called
as a Bastion Host. A Bastion Host is a special instance that acts as a proxy using
which you can SSH into your database instances. This Bastion Host will be deployed
in your public subnet and will basically only route SSH traffic from your local
network over to the database server instances. But remember, this feature comes
with its own set of security risks! Running a weak or poorly configured Bastion Host
can prove to be harmful in production environments, so use them with care!
Planning next steps
Well we have covered a lot in this chapter, but there are a few things still that you
can try out on your own with regards to VPCs. First up, is cleaning up a VPC!
Creating a VPC is easy enough and so is its deletion. You can delete an unused VPC
from the VPC Management dashboard by simply selecting the VPC, clicking on the
Actions tab, and selecting the Delete VPC option. This will bring up the Delete VPC
dialog as shown in the following screenshot:
[ 144 ]
Chapter 5
As you can see, the delete VPC option will delete all aspects of your VPC, including
subnets, Network ACLs, Internet Gateways, and so on. You can optionally even
delete any VPN connections as well by selecting the Delete VPN Connections
when deleting the VPC checkbox. Remember that once you delete a VPC, you can't
recover it back, so make sure that you don't have any active instances running on
it before you go ahead and delete it. Also remember to clean up on the instances as
well, especially the NAT Instance and the Bastion Host if you have created them.
The second thing that I would recommend trying out is called as VPC peering. VPC
peering is nothing more than network connections between two different VPCs.
Instances in one VPC communicate with instances present in another VPC using
their private IP addresses alone, so there is no need to route the traffic over the
Internet as well. You can connect your VPC with a different VPC that is either owned
by you or by someone else's Bastion Host. All it needs is a request to be generated
from the source VPC and sent to the destination VPC, along with a few route rules
that will allow the traffic to flow from one point to the other. The following is the
image describing the VPC peering:
[ 145 ]
Building Your Own Private Clouds Using Amazon VPC
You can read more about VPC peering at http://docs.aws.amazon.com/
AmazonVPC/latest/UserGuide/vpc-peering.html.
The third thing that really is worth testing out is the hardware VPN connectivity
with your VPC. I know you are probably thinking that since it's a hardware VPN
connectivity, it means that I need some special hardware equipment like a router
and so on. Well that's not quite true! You can set up an easy VPN connection
using software as well, for example, OpenVPN. OpenVPN basically allows you to
create a secure network connection from your local network to Amazon VPC using a
VPN connection.
All you need to do is deploy an OpenVPN server in your VPC and configure that
to accept incoming traffic from your private network. Then, install an OpenVPN
client on your remote desktop and try connecting to the OpenVPN server placed in
the VPC. If all goes well, you should have access to your VPC instances from your
local desktop! Do note that you will have to open up additional security rules and
network ACLs to allow this type of traffic to flow through your VPC subnet.
Last but not least, I would also recommend for you to have a look at VPC's Flow
Logs. This is a simple logging feature provided in VPC to capture traffic information
and store it using Amazon CloudWatch Logs. Flow Logs can help you analyze your
network traffic flow for bottlenecks, observe certain traffic trends, as well as monitor
traffic that reaches your instances. You can read more about Flow Logs at http://
docs.aws.amazon.com/AmazonVPC/latest/UserGuide/flow-logs.html.
Best practices and recommendations
The following are some key best practices and recommendations to keep in mind
when using VPCs:
•
Plan and design your VPC before actually implementing one. Determine the
right choice of subnet that your application will need and build your VPC
around it.
•
Choose your VPC's network block allocation wisely. A /16 subnet can
provide you with a potential 65,534 IP addresses that rarely will get utilized.
So ideally, go for a /18 (16,382 IP addresses) or a /20 (4094 IP addresses) as
your VPC network choice.
•
Always plan and have a set of spare IP address capacity for your VPC. For
example, consider the network block for my VPC as 192.168.0.0/18.
•
In this case, we design the subnet IP addressing as follows:
°°
192.168.32.0/19 Public Subnet
°°
192.168.64.0/19 Public Subnet spares
[ 146 ]
Chapter 5
°°
192.168.128.0/20 Private Subnet
°°
192.168.192.0/20 Private Subnet spares
•
Remember that you cannot edit a network block's size once it is created for a
VPC. The only way to change the network block is by deleting this VPC and
creating a new one in its place.
•
Use different security groups to secure and manage traffic flows from your
instances. For example, a separate Security Group for web servers and a
different one for your database servers. Avoid using the default security
groups at all times.
•
Leverage multiple AZs to distribute your subnets across geographies. For
example, the US-WEST region has three AZs, namely us-west-2a, us-west2b, and us-west-2c. So an ideal situation would have you divide your VPC's
network block and create subnets in each of these AZs evenly. The more AZs,
the better the fault tolerance for your VPC.
•
Leverage IAM to secure your VPC at the user level as well. Create dedicated
users with restricted access to your VPC and its resources.
•
Create and stick with a standard naming convention so that your VPC's
resources can be easily identified and tagged. For example, in our scenarios,
we named the VPC as US-WEST-PROD-1, which clearly identifies
this particular VPC to be hosted in the US-WEST region and to be a
PRODUCTION environment.
Summary
So it's been a pretty long and intense chapter so far with lots to learn and try out!
Let's have a quick recap of the things covered!
Well we started off with a brief overview of VPCs and its components and
terminologies such as subnets, route tables, Network ACLs, Internet Gateways, and
much more. Next we had an in-depth look at some of the VPC deployment scenarios
and even went ahead and created our first VPC using the VPC wizard. Toward
the end, we looked at some key considerations to keep in mind when dealing with
security groups and NAT instances and finally topped it all off with some handy best
practices and recommendations!
[ 147 ]
Monitoring Your AWS
Infrastructure
In the previous chapter, we covered a whole lot about Amazon VPC: its features,
components, and architecture. We also looked at how you can create and deploy
your own fully functional VPC using just a few simple steps!
In this chapter, we will focus primarily on how to monitor your cloud infrastructure,
especially your EC2 instances using AWS's monitoring service called as Amazon
CloudWatch. CloudWatch is a cheap and easy-to-use centralized monitoring service
that provides a variety of features such as alerts, logging, notifications, custom
metrics, and much more! So, what are we waiting for? Let's get started right away!
An overview of Amazon CloudWatch
Before we move on to Amazon CloudWatch, it is important to understand the
difference in a traditional monitoring solution and a monitoring solution based on
the clouds. Unlike your traditional environments, infrastructure in the cloud can scale
up and down dynamically in a matter of minutes. Most traditional server monitoring
tools cannot match up to this elastic requirement in real time and thus often end up
either providing the wrong information or triggering a delayed response. There is
also the problem of sheer numbers! A standalone monitoring tool can find it difficult
to handle the monitoring of thousands of virtual machines at a single go. Plus, you as
sysadmins also need to manage the monitoring tool, which adds an extra overhead
as well. That's where a cloud-based monitoring solution is so different. A standard
cloud-based monitoring tool provides the following feature sets:
•
Ease of use and management: Most of the cloud-based monitoring tools come
with easy integration and management facilities, using which you can start
monitoring your cloud infrastructure in minutes.
[ 149 ]
Monitoring Your AWS Infrastructure
•
Dynamically track instances as they are created, add them to the monitoring
inventory, and remove them from the inventory when they are deleted.
•
Trigger real-time events and notifications based on preset alarms.
•
Monitor the instance's operating system, networking, CPU, and disk
utilizations, as well as its applications which can be web servers, databases,
application servers, and so on.
•
Perform/trigger actions based on certain thresholds getting crossed.
These are just some of the key features that are provided by Amazon CloudWatch
as well, and you don't have to install or configure it. It's available as a ready-to-use
service and you only pay for the amount of service that you use! Awesome, isn't
it! Let's have a quick look at Amazon Cloud Watch's overall architecture as well as
some of its key components and concepts.
Concepts and terminologies
Before we go ahead and start using CloudWatch, it is essential to understand some
of its key concepts and terminologies.
Metrics
Metrics form the core of Amazon CloudWatch's functionality. Essentially, these are
nothing more than certain values to be monitored. Each metric has some data points
associated with it which tend to change as time progresses. For example, the CPU
usage of any one of your instances is a metric and the values of the CPU usage over
a period of time are its associated data points! Each data point has an associated
timestamp provided with it along with a unit of measure.
There are a ton of metrics that AWS provides that can be used in as-is scenarios;
however, you can additionally create custom metrics as well, as per your
requirements. An important point to remember here is that a metric is region
specific, which means that a metric is only going to be available in the region in
which it was created. A metric is uniquely identified by a name, a namespace, or a
set of dimension.
[ 150 ]
Chapter 6
Namespaces
Namespaces are a standard string of characters that you define when you first
create a metric. These namespaces act as containers for your metrics and help
in keeping them isolated from one another. There is no default namespace
provided as such, so you will have to create one for each element that gets added
to CloudWatch. By default, all AWS namespaces follow the following naming
convention: AWS/ where can be EC2, RDS, S3, and
so on. For a full list of AWS namespaces, refer to http://docs.aws.amazon.com/
AmazonCloudWatch/latest/DeveloperGuide/aws-namespaces.html.
Dimensions
Dimensions are simple key-value pairs that help you identify your metrics.
These come in real handy when you need to filter out certain result sets which a
CloudWatch query returns. You can assign up to 10 dimensions to a single metric.
For example, consider the following combination of dimensions:
•
Server=WEB, Domain=US-WEST
•
Server=DB, Domain=US-WEST
•
Server=NAT, Domain=US-WEST
You can easily retrieve statistics based on these dimension combinations; however,
it is important to note that you will not be able to retrieve any statistics for the
combinations that you did not create. For example, just querying Server=NAT or
Domain=US-WEST or even Server=NAT, Domain=Null will not give you any
results as the corresponding metrics were never created.
Time stamps and periods
Time stamps are assigned to each of your metric's data points. These are simple date
and time values that are generally provided using the UTC or GMT time zones. The
time stamp 2015-09-12T20:45:30Z translates to the September 9, 2015 at 8:45:30 PM
as per the UTC time zone. If no time stamp value is provided, then CloudWatch will
automatically assign that data point one based on its time of arrival or generation.
[ 151 ]
Monitoring Your AWS Infrastructure
Periods are the length of time associated with a particular statistic. To put it in simple
words, a period is the time between a start time and the end time. You can specify a
period as short as 60 seconds and all the way up to 86,400 seconds, which accounts
for a complete day. Periods play a crucial role in the creation of alarms. An alarm
is generally meant to be triggered when a certain threshold value is crossed, right?
Now in many cases, you will receive false alarms even if these threshold values are
crossed for a few seconds. That's just going to flood your mailbox with unwanted
notifications! So ideally, we specify the alarms with a threshold and a time period,
say 20 seconds. So, if the threshold is breached for more than 20 seconds, only then
will CloudWatch raise the alarm. This way you have a more granular control over
when your alarms get triggered.
Units and statistics
Units help you get conceptual meaning of your metric data. Specifically, these are
very similar to units of measure, for example, the metric NetworkIn that is used to
track the number of bytes an instance receives will have a measuring unit of bytes
(for example, 300 Bytes). Similarly, the metric CPUUtilization which is used to track
your instance's CPU utilization will have a measuring unit of Percent (for example,
20% CPU utilization) and so on so forth. Here are some of the units that you will
commonly come across when working with CloudWatch:
•
Count
•
Bytes, Kilobytes, Megabytes, Gigabytes, Terabytes
•
Bytes/Second, Kilobytes/Second, Megabytes/Second, Gigabytes/Second,
Terabytes/Second
•
Percent
•
Milliseconds, Microseconds, Seconds
If you do not specify a unit for a metric, then CloudWatch
will auto-assign it with the None unit.
Statistics are metric data that has been aggregated over a period of time. There are
five statistics provided by CloudWatch, as described in the following:
Statistic Name
Statistic Description
Minimum
This specifies the lowest data value observed during a specific period of
time. This statistic is useful in determining the lowest points of activity
for your application.
[ 152 ]
Chapter 6
Statistic Name
Statistic Description
Maximum
This specifies the highest data value observed during a specific period of
time. This statistic is useful in determining the highest points of activity
for your application.
Sum
This statistic adds each of the metric data points together for the
supplied period of time and is helpful in determining the total volume
of a metric.
Average
The average is indicated by Sum divided by the sample count. The
average statistic comes in handy when you want to scale your resources
especially your EC2 instances.
SampleCount
This statistic provides the actual number of data points present in
the sample during the supplied period of time. It is useful in cases of
statistical calculations.
For a better understanding, the following is simple example depicting a few raw
metric data points collected over a period of time and their statistics:
Raw Metric Data Points
Sum
Minimum
Maximum
Average
SampleCount
120,130,50,160,185
645
50
185
129
5
15,25,100,210,15,235
600
15
235
100
6
Alarms
An alarm basically watches over a particular metric for a stipulated period of time
and performs some actions based on its trigger. These actions can be anything from
sending a notification to the concerned user using the Simple Notification Service
(SNS) or something a bit more complicated such as triggering and auto-scaling an
event. However, do remember that you can create and associate alarms to any AWS
resource provided that they reside in the same region.
Tying it all together, the following is what a CloudWatch alarm basically looks like:
•
Namespace=AWS/EC2
•
Metric name=CPUUtilization (Percent)
•
Period=5minutes
•
Statistics: Average
•
Threshold: 70
•
Dimensions (Name=Web Server, Value="ï-dd42dd1b")
•
Alarm action:
[ 153 ]
Monitoring Your AWS Infrastructure
So, we get to know a lot about this alarm just by looking at it. First off, we can
tell that this alarm is going to monitor the CPU utilization of one of our specified
instances using its instance ID. Secondly, we can also tell that the alarm will
monitor the average CPU utilization of the instance. If the CPU utilization breaches
the threshold value of 70 percent for a period of 5 minutes, then the alarm will
automatically trigger an e-mail notification based on the e-mail ID that you specify.
Simple, isn't it! The following is what the preceding example looks like schematically:
We will be learning a lot more about alarms later on in this chapter and will even go
ahead and create a few for our environment, but for now, let us have a quick look at
CloudWatch's limits and associated costs.
CloudWatch limits and costs
CloudWatch, by default, monitors all of your instances, volumes, and Elastic Load
Balancers (ELB) at a regular five-minute interval for absolutely no charge at all. This
is CloudWatch's default behavior; however, you can always change the interval to
as low as a minute if you need it. Changing the interval to a minute will cost you
approximately $3.50 per instance per month. Besides this, CloudWatch also provides
10 metrics, 10 alarms, a thousand e-mail notifications using SNS, and up to a million
API requests each month for no charge at all! Additional metrics and alarms are
charged approximately $0.50 and $0.10, respectively, on a monthly basis. CloudWatch
also provides you with free 5 GB of incoming data and 5 GB of data archiving.
[ 154 ]
Chapter 6
From a limits point of view, here are a few important limits that you need to keep in
mind when working with CloudWatch:
•
CloudWatch preserves metric data for up to 2 weeks, after which it is deleted
•
The maximum period value that you can specify is 1 day or 86,400 seconds
•
You can create up to 5,000 alarms per AWS account, with each alarm
supporting up to five actions
Keeping these things in mind, let's go ahead and create our very first alarm with
CloudWatch!
Getting started with CloudWatch
In this section, we are going to carry out two tasks. First up, we will check out some
simple steps, using which you will be able to create your very first billing alarm,
followed by creating a few simple alarms for an instance using both the AWS
Management Console as well as the AWS CLI. So, without further ado let's get
started on some CloudWatch!
Monitoring your account's estimate charges
using CloudWatch
CloudWatch provides a really simple alarm setup using which you as an end user
can monitor your account's estimated costs and usage. To work with this, you need
to log in to your AWS account as the root user and not as an IAM user, even if you
are the administrator. I know I'm not following my own rules here by using the
root user, but hey, that's what AWS says! Log in to your AWS account using your
root credentials. Once logged in, select the Billing & Cost Management option
highlighted under your account's name, as shown in the following screenshot:
[ 155 ]
Monitoring Your AWS Infrastructure
This will pop up your account's management dashboard, using which you can view
your account's Bills, set new Payment Methods, view past Payment History, and so
on so forth. For now, select the Preferences option from the navigation page to bring
up the Preferences dashboard, as shown in the following screenshot:
Select the Receive Billing Alerts checkbox to enable monitoring of your account's
usage. It's important to, however, note that once you enable this checkbox, there is no
going back! You will not be able to uncheck this option afterward!
Click on the Save Preferences option to save your new settings and then select the
Manage Billing Alerts link to bring up CloudWatch's Create Alarm wizard, as
shown in the following screenshot. This option is available from the CloudWatch
dashboard to billing in the N. Virginia region.
[ 156 ]
Chapter 6
The wizard will walk you through some simple steps to configure your first billing
alarm. Select the checkbox adjoining to the EstimatedCharges option and click
on Next to continue with the process. You can optionally change the statistic and
period; however, I have gone ahead with the default values which is Maximum and
6 Hours respectively.
AWS does not allow the billing alarm's period to be set
less than 6 hours.
Moving on to the final step of your Create Alarm wizard, provide a suitable Name
and Description for your billing alarm, as shown in the following screenshot. Next,
configure the threshold for your alarm by selecting the >= (greater than or equal to)
option and providing a threshold monetary amount such as $2 or $200, whichever is
applicable to you. You can even set the threshold to $0.01, which will notify you the
moment you start going out of the free tier eligibility. In either case, the alarm will
only trigger when the actual cost of usage exceeds the monetary threshold that you
have set. You can verify the setting by looking at the Alarm Preview graph as well.
The red line indicates the threshold value set by you, whereas the blue highlighted
portion is your account's current estimate bill:
With the Alarm's threshold set, the final thing that you need to do is define what
action the alarm must take when it is triggered. From the Notification section, fill out
the required details, as mentioned in the following:
•
Whenever this alarm: This option will allow you to determine when the
alarm will actually perform an action. There are three states of an alarm out
of which you can select any one at a single time:
°°
State is ALARM: Triggered when the metric data breaches the
threshold value set by you
[ 157 ]
Monitoring Your AWS Infrastructure
°°
State is OK: Triggered when the metric data is well within the
supplied threshold value
°°
State is INSUFFICIENT: Triggered when the alarm generally doesn't
have enough data with itself to accurately determine the alarm's state.
For this scenario, I have selected the State is ALARM option as I want to get
notified as soon as my threshold limit is breached.
•
Send notification to: As discussed earlier, CloudWatch leverages Amazon
SNS to send notifications to a particular set of users and e-mail IDs. Since this
is our first SNS topic, go ahead and select the New List option, as shown in
the following screenshot. Provide a suitable SNS topic name against the send
notification to option and a valid e-mail address in the Email list field, as
shown in the following screenshot:
You can add multiple e-mail IDs in the Email list field by separating them with
commas. Once done, click on Save Changes to complete the alarm's creation process.
[ 158 ]
Chapter 6
The alarm will take a few seconds to change from the INSUFFICIENT state to OK,
as shown in the following screenshot. This is normal behavior as the alarm generally
takes a few seconds to gather the metric data and verify it against the set threshold
value.
You can view additional details about your newly created alarm by simply selecting
it and checking out the Details and History tab provided in the following:
Oh! And one very important thing I almost forgot to mention! There is a catch
to creating alarms, specifically billing ones, using CloudWatch. Don't worry, it's
nothing serious! It's just that by design, the billing metric data of your entire account,
which includes all your regions and AWS services, is collected and stored only in the
US East (N. Virginia) region. So if you want to create or update this billing alarm at
a later stage, you will have to change your default operating region to US East (N.
Virginia) and then view and edit the billing alarm as required. You can, however,
create EC2, ELB, RDS, and other AWS services related alarms from any particular
region that you are operating from.
Monitoring your instance's CPU Utilization
using CloudWatch
With the billing alarm created, let's try out something even more exciting! In this
section, we will be creating a simple alarm to monitor an instance's CPU utilization.
If the CPU utilization breaches a certain threshold, say 75 percent, then the alarm
will trigger an e-mail notification as well as perform an additional task such as stop
the instance.
[ 159 ]
Monitoring Your AWS Infrastructure
To begin with, AWS makes creating alarms a really simple and straightforward
process. The easiest way to do this is by selecting your individual instances from
the EC2 Management Dashboard and selecting the Monitoring tab, as shown in
the following screenshot. Each instance is monitored on a five-minute interval by
default. You can modify this behavior and set the time interval as low as one minute
by selecting the Enable Detailed Monitoring option.
Enabling detailed monitoring for you instance will incur
additional costs.
Each instance, by default, gets its own set of performance graphs as well, which
can be viewed in the Monitoring tab. These graphs generally include and display
important metric information such as CPU utilization, disk Read/Writes, bytes
transferred in terms of network IO, and so on. You can expand on each of the graphs
by simply selecting them. This gives you a much better and detailed view of your
instance's performance, as shown in the following image:
[ 160 ]
Chapter 6
This is an example of an enhanced graph view of the CPU utilization metric. The
x axis displays the CPU utilization in percent whereas the y axis display the time
as per the current period's settings. You can view the individual data points and
their associated values by simply hovering over them on the graph. Alternatively,
you can also switch between the Statistics, Time Range, and Period as per your
requirements. Once you have viewed your instance's performances, you can create a
simple alarm by selecting the Create Alarm option provided in the Monitoring tab.
This method is great if you want to set alarms for your instances on an individual
basis, alternatively you can use the CloudWatch dashboard as well.
To view the CloudWatch Dashboard, from the AWS Management Console's home
page, select the CloudWatch option, as shown in the following screenshot:
This will bring up the CloudWatch dashboard for the particular region in which you
are currently operating. The dashboard is divided into two sections, a navigation
pane to the left that groups and lists out your alarms based on their current state,
for example, ALARM, OK, or INSUFFICIENT. It also provides access to the
CloudWatch Logs and Metrics, as shown in the following screenshot:
Let's go ahead and check out the steps required to create our very first instance-based
alarm. To get started, select the Create Alarm option. This will bring up the Create
Alarm wizard, as shown in the following screenshot. The wizard is a simple two-step
process that will help you with the necessary steps required to create your alarm.
[ 161 ]
Monitoring Your AWS Infrastructure
First up, we need to select the correct metric that needs to be monitored. You can use
the Browse Metrics or the search bar to filter out the particular metric, which in this
case is CPU Utilization.
Next, select the particular instance for which you want to set this alarm. You can select
multiple instances here as well. Selecting the instance will view its CPU utilization
graph which you can modify using the statistics (Average) as well as the period
(5 Minutes) dropdown lists. For now, click on Next to continue with the wizard.
The second step of the wizard is where you actually define the alarm, including its
threshold value, as well as what actions have to be performed in case the alarm is
triggered. For starters, provide a suitable Name and Description for your alarm.
In this case, I provided the alarm with a name US-WEST-PROD-WEBSERVER-CPU
—now that's pretty self-explanatory!
Moving on, the next part of your alarm's configuration is the threshold setting. As
per our scenario, this alarm has to be triggered when the CPU utilization of the
instance breaches 75 percent. Select the >= (greater than equal to) option from the
is dropdown list and provide the value 75 in its adjoining textbox, as shown in the
following screenshot. You can check your alarm's threshold settings in the Alarm
Preview box.
[ 162 ]
Chapter 6
With the threshold value set, the final thing to do is create the actions that will get
triggered when the alarm is raised. There are three basic action items that you can
create for each of your EC2 alarms described as follows:
•
Notification: This option will generate a simple e-mail-based notification
using the Amazon SNS service.
•
AutoScaling Action: This option is useful when we want to trigger an
auto-scaling event. We will be looking at AutoScaling a bit more in detail
in the coming chapter.
•
EC2 Action: This option allows you perform a set of EC2 related actions
on your instance. These actions can stop, terminate, reboot, or even recover
an instance.
For this particular scenario, we need to generate an e-mail-based notification when
the alarm is raised and perform an EC2 action on the instance as well. Let's first
create the notification action.
In the Notification section, select the option State is ALARM from the Whenever
this alarm dropdown list. Next, click on the new list option to create a new SNS
topic. Provide a suitable SNS Topic name is the Send notification to text field along
with a list of comma separated e-mail addresses in the Email list, as shown in the
following screenshot:
[ 163 ]
Monitoring Your AWS Infrastructure
When the alarm is triggered, you will start receiving e-mails on the supplied e-mail
list stating the nature of the alarm as well as the instance's metric data points at
that particular time. You will receive these mails as long as the threshold value is
breached or until the alarm state changes back to OK.
To add an additional EC2 action to this alarm, simply click on the +EC2 Action
option. Follow the same process of setting the alarm's trigger state by selecting
the State is ALARM option, as shown in the following screenshot. Next, select the
particular EC2 action that you want to perform on this alarm's breach. In this case, I
have opted for the instance to be stopped by selecting the Stop this instance option.
Do remember that stopping an instance is only possible when your instances are
backed by EBS volumes!
Now, here's something new for you. You will need to assign an IAM Role that will
basically allow AWS to perform the EC2 actions on your instance. The alarm will
auto-create an EC2ActionsAllow IAM Role for your convenience. The following is
the code sample of the IAM Role for your reference. You can optionally create and
assign your own IAM Roles as well use the IAM Management Dashboard (Chapter 2,
Security and Access Management); however, this basic role should suffice for the time:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"cloudwatch:Describe*",
"ec2:Describe*",
"ec2:RebootInstances",
[ 164 ]
Chapter 6
"ec2:StopInstances",
"ec2:TerminateInstances"
],
"Resource": "*"
}
]
}
Select the Create IAM Role checkbox and verify the newly created IAM role. You
can optionally even create additional action items that can get triggered when the
alarm's threshold value changes to OK. Simply select the +Notification option and
provide the details, as shown in the following screenshot. Remember, you can only
create up to five actions for each alarm, so use them wisely!
Click on Create Alarm to complete the alarm's creation process. You can test your
alarm's functionality by generating CPU load on your instance using a variety of
tools such as Stress (http://people.seas.harvard.edu/~apw/stress/), Lookbusy
(https://www.devin.com/lookbusy/), an so on. I personally use Lookbusy to
generate artificial loads on my instances as it's pretty straightforward and easy to
use. Do remember that these tools should only be used for testing and in no way
are these tools recommended to be deployed on production workloads or instances.
With this basic alarm created and tested, go ahead and create similar alarms for
monitoring your instances disk as well as network utilization and performance!
[ 165 ]
Monitoring Your AWS Infrastructure
Monitoring your instance's memory and disk
utilization using CloudWatch Scripts
Although CloudWatch does an excellent job at monitoring your instance's
performance and status, it still has a few short comings to it. For starters,
CloudWatch monitors your instance's CPU utilization, but cannot measure its load.
Similarly, it can monitor the instance's memory size or a disk's IO performance, but
cannot tell you the exact memory usage or the disk usage of a particular partition
or layout. Why not? Well, simply because CloudWatch gets its monitoring metrics
directly from the Xen hypervisors which host your instances. As a result, you don't
get to see the performance and utilization of your instances at a very granular level.
Luckily, CloudWatch is designed to accept metric values from other sources as
well as from the hypervisor. These metrics are called as Custom Metrics and can
be pushed into CloudWatch using a variety of ways. In this section, we are going
to send custom metrics to CloudWatch using a set of simple Perl scripts provided
by CloudWatch itself. These scripts have to be installed in your instance and are
designed to send metric data periodically to CloudWatch. But before you begin, let's
go through a few necessary prerequisite steps as follows.
Creating CloudWatch access roles
Just as with the alarm actions, the instances need to be provided with a special set of
permissions to write to CloudWatch. There are two ways to go about this. The first
method is to copy your secret and access keys to your instance, which let's face it is
not the best of options! The second method is to create a role using IAM and assign
your instances that role during their launch. The role will provide the instance with
the necessary access rights to CloudWatch without having to expose any of your keys.
So, let's go ahead and create a simple access role for our instance using the IAM
Management Dashboard.
From the IAM Management Dashboard, select Roles from the navigation pane.
Next, select the option Create New Role. Provide a suitable Role Name for your new
role and select Next Step to continue with the process:
[ 166 ]
Chapter 6
Next up, from the Select Role Type page, select the Amazon EC2 option. This will
bring up the Attach Policy page, as shown in the following screenshot. Using the
Filter, search and select the CloudWatchFulllAccess policy as shown. You can
alternatively create your very own custom CloudWatch access policy and attach
that to your role if you want or use this default policy, which is the easier of the two.
Click on Next Step to proceed with the wizard.
Review your role's information and finally select the Create Role option to complete
the process.
Once your role is defined, go ahead and launch a new instance using the EC2
Management dashboard. Remember to assign this new role to your instance using
the IAM role dropdown list, as shown in the following screenshot :
A role is only assigned to an instance during its launch phase. You
cannot assign roles to instances that are already running. To know
more about IAM roles, refer to http://docs.aws.amazon.com/
IAM/latest/UserGuide/id_roles.html.
[ 167 ]
Monitoring Your AWS Infrastructure
With your instance launched, SSH into it using any of the options discussed in
Chapter 3, Images and Instances. You are now ready to go ahead and install the
necessary CloudWatch scripts!
Installing the CloudWatch monitoring scripts
Installing the CloudWatch scripts is a fairly straightforward process. The Perl scripts
report an instance's memory, swap, and disk utilization metrics to CloudWatch. You
can run these scripts off any Linux operating system, including the Amazon Linux
AMI as well.
Run the following command in your instance's terminal to install and configure
certain pre-requisite software:
# sudo yum install perl-DateTime perl-Sys-Syslog perl-LWP-Protocol-https
Once completed, download the latest copy of the CloudWatch monitoring scripts
using the following command:
# wget http://aws-cloudwatch.s3.amazonaws.com/downloads/
CloudWatchMonitoringScripts-1.2.1.zip
The current version of the CloudWatch monitoring scripts is 1.2.1.
Next, unzip the contents of the downloaded Zip file using the following command:
# unzip CloudWatchMonitoringScripts-1.2.1.zip
# cd aws-scripts-mon
The output of the preceding commands is as follows:
[ 168 ]
Chapter 6
The following are some of the important files that the CloudWatch monitoring script
ZIP contains:
•
CloudWatchClient.pm: This is a shared Perl module file that is used to make
•
mon-put-instance-data.pl: This Perl script is responsible for collecting
•
mon-get-instance-stats.pl: This Perl script is used to query CloudWatch
•
awscreds.template: This file is used to store your AWS Secret and Access
remote procedure calls to Amazon CloudWatch from other scripts.
your instance's metrics (memory, swap, disk space utilization) and sending
them to Amazon CloudWatch for processing.
and display the most recent utilization statistics for the EC2 instance on
which this script is executed.
Keys. We will not be requiring this file as we have opted to use an IAM
Role instead.
With this basic understanding in mind, let's use the mon-put-instance-data.
pl script to view the instance's memory utilization (mem-util). Run the following
command as shown here:
# ./mon-put-instance-data.pl --mem-util --verify --verbose
Note that this command will not publish any metrics to CloudWatch because of the
--verify attribute. Instead, it will only output the instance's memory utilization on
the terminal, as shown in the following:
The script will initially search for the presence of the secret and access keys in the
awscreds.template file. Since we have not provided the keys explicitly in the
instance, the script then resorts to using the IAM role which we created earlier. Make
sure that you receive confirmation from the script of its successful verification before
you move on to the next steps.
[ 169 ]
Monitoring Your AWS Infrastructure
You can additionally use the --mem-used (memory used)
and --mem-avail (memory available) metrics to query
your instance's memory performance as well.
Next, run the following command to collect all your instance's memory related
metrics and send them to CloudWatch:
# ./mon-put-instance-data.pl --mem-util --mem-used --mem-avail
You can even create a cron job and schedule the mon-put-instance-data.pl script
to collect and send metric data over a period of time using the following set of
commands. First, create a new file and save the following cron task in it:
# vi /etc/cron.d/Monitor_MEM
Add the following lines to your cron file:
*/5 * * * * ~/aws-scripts-mon/mon-put-instance-data.pl --mem-util --memused --mem-avail --from-cron
The cron will execute every five minutes and send the instance's memory details over
to CloudWatch:
You can create additional cron files to monitor the instance's disk utilization as well.
For example, if you want to be notified when the instance's root (/) or /var partition
starts to fill up. In that case, create a simple cron task with the following information
in it:
*/5 * * * * ~/aws-scripts-mon/mon-put-instance-data.pl --disk-space-avail
--disk-path=/ --disk-path=/var --from-cron
[ 170 ]
Chapter 6
You can optionally even use the --disk-space-util (disk utilization) and the
--disk-space-used (disk space used) metrics to query your instance's disk
performance as well.
Viewing the custom metrics from CloudWatch
You can view your custom metrics from the CloudWatch management dashboard
as well. Simply select the Metrics option from the CloudWatch navigation pane.
Next, browse the listed metrics for a Linux System metrics, as shown in the following
screenshot. You can optionally even use the Browse Metrics search bar to filter out
the required metrics.
Select the Linux System Metrics option to view your instance's memory and disk
utilizations, as shown in the following screenshot. You can use these metrics to list
and create your very own custom alarms, as well use the Create Alarm option.
[ 171 ]
Monitoring Your AWS Infrastructure
For example, raise an alarm when Memory Utilization of the instance crosses a
threshold of 75 percent, or send a notification alert to the concerned sysadmins
when the root (/) partition's available disk space is below 10 percent, and so on.
With this, we have now successfully started monitoring our instance's memory and
disk utilizations as well! Next up, let's look at how you can leverage CloudWatch to
monitor your instance's or your application's log files using CloudWatch Logs!
Monitoring logs using CloudWatch Logs
Imagine that you have a bunch of web server instances with some web applications
running on top of them. Now, what if you wanted to collect the log files off these
instances and the web app and store it in a central repository such that you can
troubleshoot errors and faults more effectively? That's precisely what CloudWatch
Logs is all about!
CloudWatch Logs basically allows you to monitor custom application log files
as well as log files generated by your EC2 instances in real time. You can even
create and associate CloudWatch alarms which can send you notifications in case a
particular log file displays errors. For example, you can monitor your application's
logs for NullPointerExceptions or even your classical 404 status codes provided
by your Apache web servers log files. You can additionally even store your log files
to S3 for further analysis or to be loaded into some other log processing system.
In this section, we will learn how to monitor our application's web server (Apache
HTTP) logs using CloudWatch Logs; however, before we proceed with that, let's first
have a look at some of CloudWatch Log's concepts and terminologies.
CloudWatch Log concepts and terminologies
The following are some important concepts and terms that you will come across
while using CloudWatch Logs:
•
Log events: A log event is an activity that is recorded by either your OS
or your application. It consists of two main parts: a timestamp entry that
signifies when the log event was generated and a raw message that describes
the logging event. For example, a simple log event for an HTTP web server
would look like: [17/Sept/2015:14:44:54 +0000] "GET /index.html
HTTP/1.1 404".
•
Log stream: A sequence of log events generated from the same source is
called as a log stream.
[ 172 ]
Chapter 6
•
Log groups: A log group is a collection of log streams that share some
common set of properties together. For example, an HTTP log group can
contain log streams for Apache's HTTP as well as Nginx web servers.
•
Metric filters: Metric filters are responsible for extracting certain key pieces
of information from your log files and then converting them into
CloudWatch metrics.
•
Retention policies: Retention policies dictate how long a particular log event
has to be retained. Retention policies are applied to log groups and thus are
inherited by log streams as well.
•
Log agent: Log agents are small agent based software that you need to install
on your individual EC2 instances. Each log agent is responsible for storing
and pushing the log events to CloudWatch.
So, how does all this work? Well for starters, we will need to allow our instances to
communicate with CloudWatch Logs just as we did with the custom metrics. You
can go ahead and create a different role or use the CloudWatchFullAccess role as we
did before.
Once a role is associated to your instances, the next steps require us to configure
CloudWatch Logs and install the log agent on the instance itself. Let's go ahead and
get started with the CloudWatch Logs dashboard first.
Getting Started with CloudWatch Logs
CloudWatch Logs can be accessed from the CloudWatch management dashboard
itself. Select the Logs option from the navigation pane to bring up the CloudWatch
Logs dashboard. From the main page, select the option Create Log Group, as shown
in the following screenshot:
[ 173 ]
Monitoring Your AWS Infrastructure
In this scenario, we are going to monitor the HTTP logs of our web server instance,
so go ahead and provide a suitable name for the log group, as shown in the
following. Click on Create Log Group when done. In this case, I have named my log
group as HTTP_LOG_GROUP.
With the log group now created, the next step is to create a log stream associated
with it. From the log groups dashboard, select the name of your newly created
log group. Here, you can create different log streams for your applications or OSs
based on your requirements. Next, click on the Create Log Stream option. Provide
a suitable name for your log stream and select the Create Log Stream button to
complete the process. In this case, I have named my log stream as HTTP_LOG_
STREAM, as shown in the following screenshot:
With these basic steps out of the way, now comes the fun part where we actually
get to install and configure the log agent on the instance. To do so, launch a new
Linux-based instance (I would recommend the Amazon Linux AMI or the Private
Web Server AMI that we created in Chapter 4, Security, Storage, Networking, and Lots
More!) and associate the CloudWatch access role with it. SSH into the instance and
type in the following command to install the log agent:
# sudo yum install awslogs
[ 174 ]
Chapter 6
With the log agent now installed, there are just a couple of files that you need to edit
in order for the agent to work. The first file is the awscli.conf file. Open the file
using any text editor of your choice and in the [default] section and specify the region
where you want to view the log data. Since I am operating my instance out of the
US-WEST (Oregon) region, I have provided us-west-2 as the region of my choice.
You can provide either of these values as per your requirements: us-east-1, us-west-1,
us-west-2, eu-west-1, eu-central-1, ap-southeast-1, ap-southeast-2, or ap-northeast-1.
Now, run the following command:
# vi /etc/awslogs/awscli.conf
Once done, save the file and exit the editor:
You can optionally provide your AWS secret key and access key
information in the awscli.conf file; however, we have not done
that as our instance is already associated with an IAM role.
The next file that you will need to edit is the awslogs.conf file. This is the primary
log agent configuration file, using which you can define one or more log streams as
well as define the logs that you want to track. Open the file using any text editor of
your choice and paste the following lines toward the end of the file:
# vi /etc/awslogs/awslogs.conf
[/etc/httpd/logs/access_log]
file = /etc/httpd/logs/access_log
datetime_format = %b %d %H:%M:%S
initial_position = start_of_file
log_group_name = HTTP_LOG_GROUP
log_stream_name = HTTP_LOG_STREAM
[ 175 ]
Monitoring Your AWS Infrastructure
What do all these lines mean? Here's a quick look at the awslogs.conf file's
parameters:
•
file: The file parameter specifies the log file whose contents you want to
•
datetime_format: This parameter specifies how the timestamp value is
push on to the CloudWatch Logs. In my case, I want to specify the HTTP web
server's access log file, hence the path of the Apache web server's access log
file (/etc/httpd/logs/access_log).
extracted from supplied log file.
°°
%b specifies month as in Jan, Feb, and so on.
°°
%d specifies day of month in numbers as in 1,2,3,…31.
°°
%H specifies hour in a 24 hour clock format.
°°
%M specifies minutes.
°°
%S specifies seconds.
The access_log file's log entries have the timestamp of %b %d %H:%M:%S
which translates to Sep 9 18:45:59.
•
Initial_position: This parameter specifies where to start to read the
log data from the log file. It supports two values: start_of_file and
end_of_file.
•
log_group_name: As the name implies, this parameter refers to the log group
•
log_stream_name: This parameter refers to the destination log stream name.
name that you created in CloudWatch Logs earlier.
Besides these standard parameters, the awslogs.conf file supports a few additional
parameters as well. A complete list can be found at http://docs.aws.amazon.com/
AmazonCloudWatch/latest/DeveloperGuide/AgentReference.html.
With the necessary entry made in the awslogs.conf file, we are now ready to start
the log agent. Type in the following command in your instance's terminal screen to
start the log agent service:
# sudo service awslogs start
# sudo chkconfig awslogs on
[ 176 ]
Chapter 6
Viewing the logs
With the log agent up and running, you can now go ahead and view the HTTP logs
from the CloudWatch Logs UI. Select the particular log group for which you want to
view the log data. Next, select the appropriate log stream for the same. In this case, I
had to select HTTP_LOG_GROUP as my log group and HTTP_LOG_STREAM as
its associated log stream. You should see a bunch of log statements, as shown in the
following screenshot:
You can use the Filter option to search for particular errors or status events from
your log data. Alternatively, you can also use the Date/Time adjustor to view log
data from a particular time period. With this step, we are now ready to go ahead and
create a few metric filters for our log data.
Creating metric filters and alarms
CloudWatch Logs provide a really awesome method, using which you can filter
and search out patterns, phrases, and even values from your log data. For example,
you can create and set a filter that will raise an alarm when it encounters the words
FATAL or ERROR from your application logs or even create a filter that searches
for any 4XX-based errors from your HTTP logs such as 400 - Bad Request, 401 Unauthorized, 403 - Forbidden, 404 - Not Found, and so on. Each time any of
these values are found, CloudWatch registers them as a metric value, which can then
be compared with the rest of the log data.
[ 177 ]
Monitoring Your AWS Infrastructure
To create a metric filter, select your log group's name from the CloudWatch Log
dashboard and select the Create Metric Filter option. This will bring up the Metric
Filter wizard, as shown in the following screenshot:
Provide a suitable pattern to filter your log stream in the Filter Pattern field. In my
case, I have created a simple filter that will extract the host, logName, user, request,
size, and the status_code option if the status code has any 4XX values in it, which
includes 401,403, 404, and so on.
You can read more about filter patterns and how to use them
at http://docs.aws.amazon.com/AmazonCloudWatch/
latest/DeveloperGuide/FilterAndPatternSyntax.html.
Next, select the correct log stream on which you wish to test this filter pattern. In
my case, I selected the HTTP_LOG_STREAM option. Next, select the Test Pattern
option to test your filter pattern. You should see a few results show up in the Results
section if your filter pattern is accurate. This validates that the filter patter is correct,
so move on to the next phase of the wizard where we assign a metric to this filter.
Click on Assign Metric to continue. You should see the Create Metric Filter and
Assign a Metric page, as shown in the following screenshot:
[ 178 ]
Chapter 6
Using this page, you can assign a metric to your filter that can be used to graph
as well as set alarms. Provide a suitable Filter Name for your newly created filter
pattern. In my case, I have used the default values itself. Next, in the Metric Details
section, provide an appropriate Metric Namespace, Metric Name, as well as a
Metric Value for your filter. Click on Create Filter to complete the metric filter
creation process. You should receive a confirmation box, as shown in the following
screenshot. Click on Create Alarm to create and assign an alarm to this newly
created metric filter.
That's all there is to it! You can refer to some interesting and easy to follow metric
filter examples by following http://docs.aws.amazon.com/AmazonCloudWatch/
latest/DeveloperGuide/MonitoringPolicyExamples html.
[ 179 ]
Monitoring Your AWS Infrastructure
Planning your next steps
Well, we have covered a lot about CloudWatch in this chapter; however, there are a
few things that I would really recommend you to try out next. First off is exporting
your log data to S3. Although an optional step, exporting your logs over to S3 can be
really beneficial in terms of analyzing and monitoring your application's as well as
your instance's performance and trends. How do you get started with this? Well, it's
very simple!
You will need to first create an S3 bucket in the same region as that of your log
data. Next, provide a set of permissions to your S3 bucket so that it and its contents
are writeable by CloudWatch Logs. You can use an IAM policy or even use S3's
access polices for the same. Finally, create a CloudWatch Logs Export task that
includes your log group's name as the input and the S3 bucket's destination as the
output. That's it! You can read more about the detailed steps required for exporting
logs to S3 at http://docs.aws.amazon.com/AmazonCloudWatch/latest/
DeveloperGuide/S3ExportTasks.html.
The second thing worth trying out is log data processing using Amazon Kinesis.
Why is this so important? Well to be honest, CloudWatch Logs is a good tool, but
it is not designed to process and handle large log files that too close to real time.
That's where Amazon Kinesis comes into play! Kinesis is a managed service used
for the rapid processing of large amounts of data, particularly logs, application
usage statistics, and so on. Working with Kinesis is pretty straightforward. To begin
with, you will first need to create a Kinesis stream. This is where your log events
will be delivered to for processing. Next, you will need to create something called
as a subscription filter using CloudWatch. A subscription filter basically will filter
out the required log events from your log data using a defined filter pattern. These
log events are then sent to the Amazon Kinesis stream for further analysis and
processing. You can read more about subscription filters and Amazon Kinesis at
http://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/
Subscriptions.html.
Recommendations and best practices
The following are some key recommendations and best practices to keep in mind
when working with CloudWatch:
•
Create a monitoring plan for your infrastructure and abide by it. Note down
all the metrics that you need to collect along with the method of its collection
before actually deploying your infrastructure on the cloud.
•
Monitor each and every aspect of your infrastructure, including EC2
instances, EBS volumes, Elastic Load Balancers, and so on. Create specific
[ 180 ]
Chapter 6
alarms for monitoring each AWS resource independently
•
Avoid storing secret and access keys in your instances, and instead create
and use specific IAM roles for permitting the instances to communicate with
CloudWatch.
•
Create and check log files periodically for application or instance related
faults
and alerts.
•
Perform stress tests on your application and instances and create alarms that
respond accordingly.
Summary
Phew! This has been a long but interesting and worthwhile chapter indeed! Let's
quickly recap the things we learnt so far!
First up, we started off with a quick introduction to Amazon CloudWatch, its features,
concepts, and terminologies. Next, we went ahead and created our very first alarm
in the form of an estimate billing alarm. We then even saw how to create alarms for
monitoring the performance of our EC2 instances, as well as how to perform certain
actions when the alarms are triggered. Toward the end of this chapter, we looked at
CloudWatch Logs and how you can leverage it to monitor your web server's logs. And,
finally, we finished the chapter with a brief look at custom metrics and metric filters
and how we can use them effectively to monitor our instances and applications.
In the next chapter, we will be taking CloudWatch and monitoring to the next level
by exploring the awesome concept of auto scaling, so stay tuned!
[ 181 ]
Manage Your Applications
with Auto Scaling and
Elastic Load Balancing
In the previous chapter, you learnt a lot about monitoring our AWS infrastructure,
especially the EC2 instances using Amazon CloudWatch. We also created our very
first alarms using CloudWatch and monitored our instance's CPU, memory, and disk
utilization and performance using the same.
In this chapter, we are going continue where we last dropped off and introduce an
amazing and awesome concept called Auto Scaling! AWS has been one of the first
public cloud providers to provide this feature and really it is something that you
must try out and use in your environments! This chapter will teach you the basics of
Auto Scaling, its concepts and terminologies, and even how to create an auto scaled
environment using AWS. It will also cover Amazon Elastic Load Balancers and how
you can use them in conjunction with Auto Scaling to manage your applications
more effectively! So without wasting any more time, let's first get started by
understanding what Auto Scaling is and how it actually works!
[ 183 ]
Manage Your Applications with Auto Scaling and Elastic Load Balancing
An overview of Auto Scaling
We have been talking about AWS and the concept of dynamic scalability, also known
as Elasticity in general throughout this book; well now is the best time to look at it in
depth with the help of Auto Scaling!
Auto Scaling basically enables you to scale your compute capacity (EC2 instances)
either up or down, depending on the conditions you specify. These conditions could
be as simple as a number that maintains the count of your EC2 instances at any
given time, or even complex conditions that measure the load and performance of
your instances such as CPU utilization, memory utilization, and so on. But a simple
question that may arise here is why do I even need Auto Scaling? Is it really that
important? Let's look at a dummy application's load and performance graph to get a
better understanding of things; let's take a look at the following screenshot:
[ 184 ]
Chapter 7
The graph to the left depicts the traditional approach that is usually taken to map an
application's performance requirements with a fixed infrastructure capacity. Now,
to meet this application's unpredictable performance requirements, you would have
to plan and procure additional hardware upfront, as depicted by the red line. And
since there is no guaranteed way to plan for unpredictable workloads, you generally
end up procuring more than you need. This is a standard approach employed by
many businesses and it doesn't come without its own set of problems. For example,
the region highlighted in red is when most of the procured hardware capacity is idle
and wasted as the application simply does not have that high a requirement. Whereas
there can be cases as well where the procured hardware simply did not match the
application's high performance requirements, as shown by the green region. All
these issues, in turn, have an impact on your business, which frankly can prove to be
quite expensive. That's where the elasticity of a cloud comes into play. Rather than
procuring at the nth hour and ending up with wasted resources, you grow and shrink
your resources dynamically as per your application's requirements, as depicted in
the graph on the right. This not only helps you in saving overall costs but also makes
your application's management a lot more easy and efficient. And don't worry if your
application does not have an unpredictable load pattern! Auto Scaling is designed
to work with both predictable and unpredictable workloads so that no matter what
application you may have, you can rest assured that the required compute capacity is
always going to be made available for use when required. Keeping that in mind, let us
summarize some of the benefits that AWS Auto Scaling provides:
•
Cost savings: By far the biggest advantage provided by Auto Scaling, you
can actually gain a lot of control over the deployment of your instances
as well as costs by launching instances only when they are needed and
terminating them when they aren't required.
•
Ease of use: AWS provides a variety of tools using which you can create
and manage your Auto Scaling, such as the AWS CLI and even the EC2
Management Dashboard. Auto Scaling can be programmatically created and
managed via a simple and easy to use web service API as well.
•
Scheduled scaling actions: Apart from scaling instances as per a given
policy, you can additionally even schedule scaling actions that can be
executed in the future. This type of scaling comes in handy when your
application's workload patterns are predictable and well known in advance.
•
Geographic redundancy and scalability: AWS Auto Scaling enables you
to scale, distribute, and load balance your application automatically across
multiple availability zones within a given region.
•
Easier maintenance and fault tolerance: AWS Auto Scaling replaces unhealthy
instances automatically based on predefined alarms and thresholds.
[ 185 ]
Manage Your Applications with Auto Scaling and Elastic Load Balancing
With these basics in mind, let us understand how Auto Scaling actually works out
in AWS.
Auto scaling components
To get started with Auto Scaling on AWS, you will be required to work with three
primary components, each described briefly as follows.
Auto scaling groups
An Auto Scaling group is a core component of the Auto Scaling service. It is basically
a logical grouping of instances that share some common scaling characteristics
between them. For example, a web application can contain a set of web server
instances that can form one Auto Scaling group and another set of application
server instances that become a part of another Auto Scaling group and so on. Each
group has its own set of criteria specified that includes the minimum and maximum
number of instances that the group should have, along with the desired number of
instances that the group must have at all times.
The desired number of instances is an optional field in an Auto
Scaling group. If the desired capacity value is not specified, then
the Auto Scaling Group will consider the minimum number of
instance values as the desired value instead.
Auto Scaling Groups are also responsible for performing periodic health checks
on the instances contained within them. An instance with degraded health is then
immediately swapped out and replaced by a new one by the Auto Scaling Group,
thus ensuring that each of the instances within the Group works at optimum levels.
Launch configurations
A launch configuration is a set of blueprint statements that the Auto Scaling Group
uses to launch instances. You can create a single launch configuration and use it
with multiple Auto Scaling Groups; however, you can only associate one Launch
Configuration with a single Auto Scaling Group at a time. What does a Launch
Configuration contain? Well to start off with, it contains the AMI ID using which Auto
Scaling launches the instances in the Auto Scaling Group. It also contains additional
information about your instances such as instance type, the security group it has to be
associated with, block device mappings, key pairs, and so on. An important thing to
note here is that once you create a Launch Configuration, there is no way you can edit
it again. The only way to make changes to a Launch Configuration is by creating a
new one in its place and associating that with the Auto Scaling Group.
[ 186 ]
Chapter 7
Scaling plans
With your Launch Configuration created, the final step left is to create one or more
scaling plans. Scaling Plans describe how the Auto Scaling Group should actually
scale. There are three scaling mechanisms you can use with your Auto Scaling
Groups, each described as follows:
•
Manual scaling: Manual scaling by far is the simplest way of scaling your
resources. All you need to do here is specify a new desired number of
instances value or change the minimum or maximum number of instances
in an Auto Scaling Group and the rest is taken care of by the Auto Scaling
service itself.
•
Scheduled scaling: Scheduled scaling is really helpful when it comes to
scaling resources based on a particular time and date. This method of scaling
is useful when the application's load patterns are highly predictable, and thus
you know exactly when to scale up or down. For example, an application that
process a company's payroll cycle is usually load intensive during the end of
each month, so you can schedule the scaling requirements accordingly.
•
Dynamic scaling: Dynamic scaling, or scaling on demand is used when the
predictability of your application's performance is unknown. With dynamic
scaling, you generally provide a set of scaling policies using some criteria;
for example, scaling the instances in my Auto Scaling Group by 10 when
the average CPU utilization exceeds 75 percent for a period of 5 minutes.
Sounds familiar, right? Well that's because these dynamic scaling policies
rely on Amazon CloudWatch to trigger scaling events. CloudWatch monitors
the policy conditions and triggers the auto scaling events when certain
thresholds are breached. In either case, you will require a minimum of two
such scaling policies: one for scaling in (terminating instances) and one for
scaling out (launching instances).
Before we go ahead and create our first Auto Scaling activity, we need to understand
one additional AWS service that will help us balance and distribute the incoming
traffic across our auto scaled EC2 instances. Enter the Elastic Load Balancer!
Introducing the Elastic Load Balancer
The Elastic Load Balancer or ELB is a web service that allows you to automatically
distribute incoming traffic across a fleet of EC2 instances. In simpler terms, an ELB
acts as a single point of contact between your clients and the EC2 instances that are
servicing them. The clients query your application via the ELB; thus, you can easily
add and remove the underlying EC2 instances without having to worry about any of
the traffic routing or load distributions. It is all taken care of by the ELB itself!
[ 187 ]
Manage Your Applications with Auto Scaling and Elastic Load Balancing
Coupled with Auto Scaling, ELB provides you with a highly resilient and fault
tolerant environment to host your applications. While the Auto Scaling service
automatically removes any unhealthy EC2 instances from its group, the ELB
automatically reroutes the traffic to some other healthy instance. Once a new healthy
instance is launched by the Auto Scaling service, ELB will once again re-route the
traffic through it and balance out the application load as well. But the work of the
ELB doesn't stop there! An ELB can also be used to safeguard and secure your
instances by enforcing encryption and by utilizing only HTTPS and SSL connections.
Keeping these points in mind, let us look at how an ELB actually works.
Well to begin with, when you create an ELB in a particular AZ, you are actually
spinning up one or more ELB nodes. Don't worry, you cannot physically see these
nodes nor perform any actions on them. They are completely managed and looked
after by AWS itself. This node is responsible for forwarding the incoming traffic to
the healthy instances present in that particular AZ. Now here's the fun part! If you
configure the ELB to work across multiple AZs and assume that one entire AZ goes
down or the instances in that particular AZ become unhealthy for some reason, then the
ELB will automatically route traffic to the healthy instances present in the second AZ:
[ 188 ]
Chapter 7
How does it do the routing? The ELB, by default, is provided with a public DNS
name, something similar to MyELB-123456789.region.elb.amazonaws.com.
The clients send all their requests to this particular Public DNS name. The AWS
DNS servers then resolve this public DNS name to the public IP addresses of the
ELB nodes. Each of the nodes has one or more listeners configured on them which
constantly check for any incoming connections. Listeners are nothing but processes
that are configured with a combination of protocols; for example, HTTP and a port,
for example, 80. The ELB node that receives the particular request from the client
then routes the traffic to a healthy instance using a particular routing algorithm. If
the listener was configured with an HTTP or HTTPS protocol, then the preferred
choice of routing algorithm is the least outstanding requests routing algorithm.
If you have configured your ELB with a TCP listener, then
the preferred routing algorithm is Round Robin.
Confused? Well don't be, as most of these things are handled internally by the ELB
itself. You don't have to configure the ELB nodes nor the routing tables. All you need
to do is set up the listeners in your ELB and point all client requests to the ELB's
Public DNS name, and that's it! Keeping these basics in mind, let us go ahead and
create our very first ELB!
Creating your first Elastic Load Balancer
Creating and setting up an ELB is a fairly easy and straightforward process provided
you have planned and defined your Elastic Load Balancer's role from the start. The
current version of ELB supports HTTP, HTTPS, and TCP, as well as SSL connection
protocols; however, for the sake of simplicity, we will be creating a simple ELB for
balancing HTTP traffic only. I'll be using the same VPC environment that we have
been developing since Chapter 5, Building Your Own Private Clouds Using Amazon VPC;
however, you can easily substitute your own infrastructure in this place as well.
[ 189 ]
Manage Your Applications with Auto Scaling and Elastic Load Balancing
To access the ELB Dashboard, you will have to first access the
EC2ManagementConsole. Next, from the navigation pane, select the LoadBalancers
option, as shown in the following screenshot. This will bring up the ELB Dashboard
as well, using which you can create and associate your ELBs. An important point
to note here is that although ELBs are created using this particular portal, you can,
however, use them for both your EC2 and VPC environments. There is no separate
portal for creating ELBs in a VPC environment:
Since this is our first ELB, let us go ahead and select the Create Load Balancer
option. This will bring up a seven-step wizard using which you can create and
customize your ELBs.
Step 1 – Defining the Load Balancer
To begin with, provide a suitable name for your ELB in the Load Balancer name
field. In this case, I have opted to stick to my naming convention and name the
ELB US-WEST-PROD-LB-01. Next up, select the VPC option in which you wish
to deploy your ELB. Again, I have gone ahead and selected the US-WEST-PROD-1
(192.168.0.0/16) VPC that we created in Chapter 5, Building Your Own Private
Clouds Using Amazon VPC. You can alternatively select your own VPC environment
or even select a standalone EC2 environment if it is available. Do not check the
Create an internal load balancer option as in this scenario, we are creating an
Internet-facing ELB for our Web Server instances.
There are two types of ELB that you can create and use based on your requirements.
The first is an Internet-facing Load Balancer, which is used to balance out client
requests that are inbound from the Internet. Ideally, such Internet-facing load
balancers connect to the public subnets of a VPC. Similarly, you also have something
called as Internal Load Balancers that connect and route traffic to your private
subnets. You can use a combination of these depending on your application's
requirements and architecture; for example, you can have one Internet-facing ELB as
your application's main entry point and an internal ELB to route traffic between your
public and private subnets; however, for simplicity, let us create an Internet-facing
ELB for now.
[ 190 ]
Chapter 7
With these basic settings done, we now provide our ELB's Listeners. A Listener is
made up of two parts: a protocol and port number for your frontend connection
(between your client and the ELB), and a protocol and a port number for a backend
connection (between the ELB and the EC2 instances).
In the Listener Configuration section, select HTTP from the Load Balancer Protocol
drop-down list and provide the port number 80 in the Load Balancer Port field, as
shown in the following screenshot. Provide the same protocol and port number for
the Instance Protocol and Instance Port fields as well:
What does this mean? Well, this listener is now configured to listen on the ELB's
external port (Load Balancer Port) 80 for any client's requests. Once it receives
the requests, it will then forward it out to the underlying EC2 instances using the
Instance Port, which in this case is port 80 as well. There is no rule of thumb as such
that both the port values must match; in fact, it is actually good practice to keep them
different. Although your ELB can listen on port 80 for any client's requests, it can use
any ports within the range of 1-65,535 for forwarding the request to the instances.
You can optionally add additional listeners to your ELB such as a listener for the
HTTPS protocol running on port 443 as well; however, that is something that I will
leave you do to later.
The final configuration item left in step 1 is where you get to select the Subnets
option to be associated with your new Load Balancer. In my case, I have gone
ahead and created a set of subnets each in two different AZs so as to mimic a
high-availability scenario:
[ 191 ]
Manage Your Applications with Auto Scaling and Elastic Load Balancing
Select any particular subnets and add them to your ELB by selecting the adjoining
+ sign. In my case, I have selected two subnets, both belonging to the web server
instances; however, both present in two different AZs.
You can select a single subnet as well; however, it is
highly recommended that you go for a high available
architecture, as described earlier.
Once your subnets are added, click on Next: Assign Security Groups to continue
over to step 2.
Step 2 – Assign security groups
Step 2 is where we get to assign our ELB with a security group. Now, here's
the catch: You will not be prompted for a Security Group if you are using an
EC2-Classic environment for your ELB. This Security Group is only necessary for
VPC environments and will basically allow the port you designated for inbound
traffic to pass through:
In this case, I have created a new dedicated Security Group for the ELB. Provide a
suitable Security group name as well as a Description, as shown in the preceding
screenshot. The new security group already contains a rule that allows traffic to the
port that you configured your Load Balancer to use; in my case it's port 80. Leave the
rule at its default value and click on Next: Configure Security Settings to continue.
Step 3 – configure security settings
This is an optional page that basically allows you to secure your ELB by using either
the HTTPS or the SSL protocol for your frontend connection. But since we have
opted for a simple HTTP-based ELB, we can ignore this page for now. Click on Next:
Configure Health Check to proceed to the next step.
[ 192 ]
Chapter 7
Step 4 – Configure Health Check
Health checks are a very important part of an ELB's configuration and hence you
have to be extra cautious when setting them up. What are health checks? To put
it in simple terms, these are basic tests that the ELB conducts to ensure that your
underlying EC2 instances are healthy and running optimally. These tests include
simple pings, attempted connections, or even some send requests. If the ELB senses
either of the EC2 instances in an unhealthy state, it immediately changes its Health
Check Status to OutOfService. Once the instance is marked as OutOfService, the
ELB no longer routes any traffic to it. The ELB will only start sending traffic back to
the instance only if its Health Check State changes to InService again.
To configure the health checks for your ELB, fill in the following information as
described here:
•
Ping protocol: This field indicates which protocol the ELB should use to
connect to your EC2 instances. You can use the TCP, HTTP, HTTPS, or the
SSL options; however, for simplicity, I have selected the HTTP protocol here.
•
Ping port: This field is used to indicate the port which the ELB should use
to connect to the instance. You can supply any port value from the range 1
to 65,535; however, since we are using the HTTP protocol, I have opted to
stick with the default value of port 80. This port value is really essential as
the ELB will periodically ping the EC2 instances on this port number. If any
instance does not reply in a timely fashion, then that instance will be deemed
unhealthy by the ELB.
•
Ping path: This value is usually used for the HTTP and HTTPS protocols.
The ELB sends a simple GET request to the EC2 instances based on the Ping
Port and Ping Path. If the ELB receives a response other than an OK, then
that particular instance is deemed to be unhealthy by the ELB and it will
no longer route any traffic to it. Ping paths generally are set with a forward
slash, /, which indicates the default home page of a web server. However,
you can also use a /index.html or a /default.html value as you see fit.
In my case, I have provided the /index.php value as my dummy web
application is actually a PHP app.
[ 193 ]
Manage Your Applications with Auto Scaling and Elastic Load Balancing
Besides the ping checks, there are also a few advanced configuration details that you
can configure based on your application's health check needs:
•
Response time: The Response Time is the time the ELB has to wait in order
to receive a response. The default value is 5 seconds with a maximum value
up to 60 seconds. Let's take a look at the following screenshot:
•
Health Check Interval: This field indicates the amount of time (in seconds)
the ELB waits between health checks of an individual EC2 instance. The
default value is 30 seconds; however, you can specify a maximum value of
300 seconds as well.
•
Unhealthy Threshold: This field indicates the number of consecutive failed
health checks an ELB must wait before declaring an instance unhealthy. The
default value is 2 with a maximum threshold value of 10.
•
Healthy Threshold: This field indicates the number of consecutive successful
health checks an ELB must wait before declaring an instance healthy. The
default value is 2 with a maximum threshold value of 10.
Once you have provided your values, go ahead and select the Next: Add EC2
Instances option.
[ 194 ]
Chapter 7
Step 5 – Add EC2 instances
In this section of the wizard, you can select any running instance from your Subnets
to be added and registered with the ELB. But since we are setting this particular ELB
for use with Auto Scaling, we will leave this section for now. Click on Next: Add
Tags to proceed with the wizard.
Step 6 – Add tags
We already know the importance of tagging our AWS resources, so go ahead and
provide a suitable tag for categorizing and identifying your ELB. Note that you can
always add/edit and remove tags at a later time as well using the ELB Dashboard.
With the tags all set up, click on Review and Create.
Step 7 – Review and Create
The final step of our ELB creation wizard is where we simply review our ELB's
settings, including the Health Checks, EC2 instances, tags, and so on. Once reviewed,
click on Create to begin your ELB's creation and configuration.
The ELB takes a few seconds to get created, but once it's ready, you can view and
manage it just like any other AWS resource using the ELB dashboard, as shown in
the following screenshot:
Select the newly created ELB and view its details in the Description tab. Make a note
of the ELB's public DNS Name as well. You can optionally even view the Status as
well as the ELBScheme (whether Internet-facing or internal) using the Description tab.
You can also view the ELB's Health Checks as well as the Listeners configured with
your ELB.
[ 195 ]
Manage Your Applications with Auto Scaling and Elastic Load Balancing
Before we proceed with the next section of this chapter, here are a few important
pointers to keep in mind when working with ELB. Firstly, the configurations that we
performed on our ELB are all very basic and will help you to get through the basics;
however, ELB also provides us with additional advanced configuration options such
as Cross-Zone Load Balancing, Proxy Protocols, Sticky Sessions, and so on, which
can all be configured using the ELB dashboard. To know more about these advanced
settings, refer to http://docs.aws.amazon.com/ElasticLoadBalancing/latest/
DeveloperGuide/elb-configure-load-balancer.html. Second important thing
worth mentioning is the ELB's costs. Although it is free (Terms and Conditions apply)
to use under the Free Tier eligibility, ELBs are charged approximately $0.025 per
hour used. There is a nominal charge on the data transferring as well, which is
approximately $0.008 per GB of data processed.
With these points in mind and our ELB all prepped, let us go ahead and get started
with the fun part of Auto Scaling!
Getting started with Auto Scaling
With your ELB all set up, you are now ready to go ahead and start configuring the
Auto Scaling service. As discussed earlier, there are basically three parts to Auto
Scaling: The Launch Configurations, the Auto Scaling Group, and, finally, the Scaling
Triggers. In this section, we are going to check out some simple steps using which
you will be able to create and configure your own auto scaled environment; but
before that, here are a few tips and tricks worth mentioning!
First up, prepare your Machine Image or AMI. By prepare I mean make sure you have
already installed and configured your web server (in my case, I'm using a simple Apache
HTTP web server) to start on instance boot up as well as place your application's
code or website files in the correct directories. Additionally, you can even install and
configure the CloudWatch Log agent in your AMI such that it captures the essential
web server logs and sends them to CloudWatch for further processing.
The second most important part of any Auto Scaling activity is planning and
understanding instance quantity, that is, what is the desired capacity of your Auto
Scaling Group and what are the minimum and maximum number of instances
you want your application to scale to. Since we are just starting off, I would really
recommend doing the basics. Have a desired capacity set to 1 instance, which means
that there will be a single instance hosting your web server at all times. Next, set up
a minimum instance value of 1 and a maximum instance value of, say, 5. Remember
that you can always change the Auto Scaling Group values but not the Launch
Configuration details.
[ 196 ]
Chapter 7
The third and final thing that I would like to mention is the famous moto: Plan for
failure, and nothing will fail. Although you can set up Auto Scaling using a single AZ,
it is highly recommended that you distribute your workloads across AZs as much as
possible. The following is a look at my simple Auto Scaling example:
In this case, we have already deployed and configured our ELB, and the only thing
remains is the Auto Scaling configuration. So without further ado, let's get started!
Creating the Launch Configuration
The first step to setting up an Auto Scaling activity is to create and configure a Launch
Configuration. To do so, from the EC2 Management Dashboard option, select the
AutoScaling Groups option from the navigation pane as shown in the following
screenshot. This will bring up the Auto Scaling Groups dashboard. Next, select the
Create Auto Scaling group option to bring up the Auto Scaling setup wizard.
The wizard is a simple six-step process that will first enable you to create and
configure a Launch Configuration, followed by a five-step process to associate an
Auto Scaling Group with it. Let us go through each of the steps in detail.
[ 197 ]
Manage Your Applications with Auto Scaling and Elastic Load Balancing
Step 1 – Choose AMI
From the Choose AMI page, select your choice of AMI for your Launch Configuration.
You can either select the Amazon Linux AMI to get started with or even select the
custom AMI that we created back in Chapter 4, Security, Storage, Networking, and Lots
More!, by selecting the My AMIs option and then selecting your particular custom AMI.
Step 2 – Choose Instance type
Select the appropriate instance type required for your Launch Configuration. In
my case, I have selected the General purpose t2.micro instance type for my demo
purposes; however, feel free to select an appropriate instance type as per your
requirements. Click on Next: Configure details to continue with the process.
Step 3 – Configure details
Provide a suitable Name for your Launch Configuration. In my case, I have named
it using the same naming convention that I'm following throughout this book, that
is, US-WEST-PROD-WEB-LC-1. You can Request Spot Instances instead of the
default On-Demand Instances as well. Spot instances are a great way to save costs
compared to your on-demand instances; however, use them with caution. Spot
instances are spun up the moment your bid price rises above the instance's market
value and are terminated when the market value exceeds your spot price.
You can optionally even assign a particular IAM Role for your Auto Scaled instances
by selecting an appropriate Role name from the IAM role drop-down list. In this
case, I have not provided any roles to my Launch Configuration. Select the Enable
CloudWatch detailed monitoring checkbox if you wish to have your instances
monitored for a duration of 60 seconds. By default your instances will be monitored
by CloudWatch for a minimum period of 300 seconds (five minutes) for no charge at
all. Selecting detailed monitoring will incur additional charges, so use it with caution.
Note: Enabling the CloudWatch detailed monitoring
option is highly recommended in case the instances belong
to a production environment.
[ 198 ]
Chapter 7
Once your basic details are filled out, you can even set the instance's IP addressing
scheme by selecting the Assign a public IP address to every instance option from the
Advanced Details section. This option comes in handy when you wish to connect
and log into to your VPC instances from your home network.
Step 4 – Add storage
With your Launch Configuration created, you can now continue to adding and
configuring the remaining elements of your instances, which includes the Storage
and Security Groups. You can add an optional Volume to your instances by selecting
the Add NewVolume button on the Add Storage page. The rest of the fields are
pretty self-explanatory, so I'm really not going to talk about them here. In my case,
I have not provided any additional volumes to my instances and opted for only a
single EBS root volume (/dev/xvda). Click on Next: Configure Security Group to
either create or select an existing security group for your auto scaled instances.
Step 5 – Configure Security Group
From the Configure Security Group page, select an appropriate Security Group for
your Auto Scaled instances. Since we are working with web server instances, I have
selected my US-WEST-PROD-WEB-SG (Web Server Security Group). The group has
the following set of inbound rules:
Web Server Inbound Security Rules
Source
Protocol
Port Range
Remarks
0.0.0.0/0
TCP
22
Permit inbound SSH access to web server instance
0.0.0.0/0
TCP
80
Permit inbound HTTP access to web server instance
0.0.0.0/0
TCP
443
Permit inbound HTTPS access to web server instance
You can optionally even create a new Security Group as per your requirements;
however, make sure to have the SSH as well as the HTTP ports opened up before
you proceed with the next steps.
Step 6 – Review
Phew! After all those intense configurations, we are now officially ready to review
and create our Launch Configuration. Make sure the AMI details, Instance Type,
and your Launch Configuration settings are correct. Once verified, click on Create
Launch Configuration to complete the process.
[ 199 ]
Manage Your Applications with Auto Scaling and Elastic Load Balancing
The wizard will now automatically set and create a new Launch Configuration based
on your specifications. You can create as many Launch Configurations as you need;
however, you will be able to specify only a single LC for an Auto Scaling Group at
a time. Also, once a LC is created, there is no way you can edit its configurations.
The only way to do so is by creating a new LC and associating that with your
Auto Scaling Group. Instances that were launched as a part of the old LC remain
unaffected by this change; however, any new instances that are created will use the
new LC as their blueprint for the Auto Scaling activity.
Creating the Auto Scaling Group
With your Launch Configuration ready, the next and final stage of creating your first
Auto Scaling task involves the setting up of an Auto Scaling Group. We have already
talked about Auto Scaling Groups in the beginning of this chapter, so let's have a
quick recap of the same.
As discussed previously, an Auto Scaling Group is nothing more than a logical
grouping of instances that share some common scaling characteristics between them.
Each group has its own set of criteria specified which includes the minimum and
maximum number of instances that the group should have along with the desired
number of instances which the group must have at all times. Besides these, an
Auto Scaling group also helps us to create and define scaling triggers which, when
triggered, result in either instances getting added or removed from the group. These
scaling triggers rely on CloudWatch Metrics and periodic health checks to determine
whether a particular instance is unhealthy or unresponsive. If such an instance is
found, then the Auto Scaling service will automatically terminate the unhealthy
instance and replace it with a brand new one! Awesome, isn't it!
With these basics in mind, let us go ahead and continue where we left off from
the Launch Configurations stage. Log in to your AWS account using your IAM
credentials and select the EC2 option from the AWS Management Console. Next,
from the navigation pane provided, select the Auto Scaling Groups option. This will
bring up the Auto Scaling Group dashboard, as shown in the following screenshot.
Select the Create Auto Scaling Group option to get started:
[ 200 ]
Chapter 7
Here, you will be provided with two options: either create a new launch
configuration or create an Auto Scaling Group out of an existing one. Since we have
already created our LC, select the Create an Auto Scaling group from an existing
launch configuration option. Select the newly created LC and click on Next Step
to proceed. Now comes the fun part where we actually get to configure the Auto
Scaling Groups. Follow the next steps carefully and fill out the required fields as per
your requirements:
Step 1 – Configure Auto Scaling group details
The first step in creating your Auto Scaling Group requires you to provide a suitable
name for your Auto Scaling Group as well as its Network and Load Balancing details.
Fill in the required fields as per your requirements:
•
Group name: Provide a suitable name for your Auto Scaling Group. In this
case, I have used the name US-WEST-PROD-WEB-ASG-1.
•
Group size: Here, enter the desired capacity for your Auto Scaling Group.
Remember that the value entered here represents the number of instances
Auto Scaling must have at all times, so choose a smaller number to start with.
In my case, I have chosen 1 as this is just a demo:
•
Network: From the Network drop-down list, select your appropriate VPC in
which you wish to enable Auto Scaling.
•
Subnet: Once your Network is selected, you can now select your
corresponding subnets. Auto Scaling will then launch the instances based
on the subnet that you select here. In my case, I have selected two subnets,
each created in a different AZ. This setup maximizes the availability of your
application while minimizing any unwanted downtime.
[ 201 ]
Manage Your Applications with Auto Scaling and Elastic Load Balancing
Each instance in this Auto Scaling Group will be provided
with a public IP address.
With these basic settings filled out, we now move on to the Advanced Details section
of our Auto Scaling Group:
•
Load Balancing: These are optional settings that you can configure to work
with your Auto Scaling Group. Since we have already created and configured
our ELB, we will be using that itself to balance out incoming traffic for our
instances. Select the Receive traffic from Elastic Load Balancer(s) option, as
shown in the following screenshot. Next, type in the name of your ELB
(US-WEST-PROD-LB-01) in the Load Balancing text field:
•
Health Check Type: You can use either your EC2 instances or even your
ELB as a health check mechanism to make sure that your instances are in
a healthy state and performing optimally. By default, Auto Scaling will
check your EC2 instances periodically for their health status. If an unhealthy
instance is found, Auto Scaling will immediately replace that with a healthy
one. Here, I have selected ELB as my health check type, so all the instance
health checks are now going to be performed by the ELB itself.
•
Health Check Grace Period: Enter the health check's grace period in seconds.
By default, this value is set to 300 seconds.
[ 202 ]
Chapter 7
Once your Auto Scaling Group's basic configuration is complete, the next step is
where you actually get to create and define the scaling policies. Click on Next:
Configure scaling policies to continue with the process.
Step 2 – Configure scaling policies
The second most important part of creating any Auto Scaling Group is defining its
scaling policies. A scaling policy is a set of instructions used by the Auto Scaling
service to make adjustments in your Auto Scaling group's size (number of instances).
Each Scaling Policy is attached with a CloudWatch alarm and a notification action.
When the alarm is breached, the appropriate scaling policy is triggered, which will
either add or remove instances from your Auto Scaling Group depending on its
definition. Let us go ahead and create a few such scaling policies for our own use.
First up, we need to define whether this particular scaling policy will be used to
maintain the Auto Scaling Group's initial size (desired capacity) only or whether you
wish to adjust the Group's size by adding or removing instances. In this case, I have
selected the Use scaling policies to adjust the capacity of this group option, as
shown in the following screenshot. I have also provided the minimum (1 instance)
and maximum number (5 instances) of instances I want to the group to scale
between. You can provide your own set of values here as per your needs; however,
stick to the basics if this is your first time:
Next, we define and create our scaling policies. There are two policies that are used
by an Auto Scaling Group: one to increase the instance count based on certain alarms
and the other to decrease the instance count. To begin with, let us first go ahead and
populate the Increase Group Size policy, as shown in the following screenshot:
•
Name: Provide a suitable name for your scale-out policy.
[ 203 ]
Manage Your Applications with Auto Scaling and Elastic Load Balancing
•
Execute policy when: In this field, you have to select a pre-configured
alarm using which the policy will get triggered. Since this is our first time
configuring, select the Add new alarm option. This will pop up the Create
Alarm dialog, as shown in the following screenshot:
Filling out and creating the alarm is a very simple process; for example,
we want our Auto Scaling Group to be monitored based on the CPU
Utilization metric for an interval of 5 minutes. If the average CPU
Utilization is greater than or equal to 50 percent for at least one consecutive
period, then send a notification mail to the specified SNS Topic (in this case,
All-About-Dogs-Admins). You can verify your alarm's configuration by
comparing it to the Alarm Graph as well. Once you are satisfied with your
settings, click on Create Alarm.
•
Take the action: With your basic alarm now set, you can further tell your
policy what action it has to take if the particular threshold is breached. Select
Add from the dropdown list and provide a suitable number of instances
that you wish to add when a certain condition matches. For example, I
have created a four-step scaling policy that first adds one instance to the
group when the average CPU utilization is within a particular threshold
range, such as 50-55 percent. Next, another instance is added when the CPU
utilization increases even further to 55-65 percent, and so on so forth. You
can add multiple such steps by selecting the Add step option, as shown in
the following screenshot. Once the steps are added, your Increase Group Size
policy should look something like the following:
[ 204 ]
Chapter 7
Adding steps in a policy is an optional setting and is only
meant to provide you with finer grained control over
when exactly your instances are to be launched.
•
Instances need: With the steps added, the final field left is the Cooldown
period. By default, this value is set to 300 seconds and can be changed as per
your requirements. What is this cooldown period and why is it important?
Well, a Cooldown period is kind of like a grace period that we assign to the
Auto Scaling Group to ensure that we don't launch or terminate any more
resources before the effects of previous scaling activities are completed. It is
just a way of telling the Auto Scaling Group to wait for a short period of time
before initiating another scaling event. Ideally, this field is very important
and should not be taken for granted. There have been cases where the Auto
Scaling activity goes into a loop-like condition where an instance is launched
and terminated repeatedly, only because the cooling period and the ELB
health check timeout did not match, so use this value with utmost care!
Once the Increase Group Size policy is created, you can conversely create and
configure the Decrease Group Size policy as well. Follow the same steps by first
creating and assigning an alarm that now triggers when the average CPU Utilization
is less than or equal to 75 percent. Next, add the scaling steps that will remove one
instance from the Auto Scaling Group at a time depending on the alarm's threshold
ranges, such as removing one instance from the group when the average CPU
utilization is between 65 percent and 75 percent and so on so forth. Once configured,
click on Next: Configure Notifications to proceed with the next steps.
[ 205 ]
Manage Your Applications with Auto Scaling and Elastic Load Balancing
Step 3 – Configure notifications
Notifications play a very important role in an Auto Scaling activity. You can basically
configure your Auto Scaling Group to send notifications out to any particular
endpoint, such as an e-mail address, whenever a specified event gets triggered, such
as the successful launch of an instance, or a failure to launch an instance, and so on.
To configure notifications, all you need to do is create an SNS Topic and subscribe
when it has to notify you in case a particular event is triggered. To create a new SNS
Topic, simply click on the create topic option, as shown in the following screenshot.
Fill in the SNS Topic Name as well as the required Email Addresses that you wish
to subscribe to. In this case, I have already created an SNS Topic that will send
notifications to the administrators whenever the instances successfully launch and
terminate as well as when they fail to launch or terminate correctly:
You can optionally add more notifications to your Auto Scaling Group by selecting the
Add notification option as well. Once done, click on Next: Configure Tags to proceed.
[ 206 ]
Chapter 7
Step 4 – Configure tags
We do know the importance of tags and tagging from our previous chapters.
Tagging helps us organize, manage, and identify our resources more effectively
and efficiently by specifying one or more metadata in the form of a key and a value
pair. Auto Scaling Groups too can be assigned tags using the Configure Tags page.
Provide a suitable Key and Value for your new Auto Scaling Group. You can
optionally even tag your instances that will be launched by your Auto scaling Group
by selecting the Tag New Instances checkbox, as shown in the following screenshot:
Tag keys and values are case sensitive.
Remember, you can add up to 10 tags for each Auto Scaling Group that you create as
well as remove them at any time. Once you have tagged your Auto Scaling Group,
move on to the final step of the process by clicking on the Review option.
Step 5 – Review
Congratulations if you made it this far! You are almost done with your first fully
functional Auto Scaling Group, but before you finish, review the settings once more
and make sure that all the configuration settings and auto scaling values are correct.
Once satisfied with the checks, click on Create Auto Scaling Group. The group will
first check the number of desired instances that you have specified. In our scenario,
we specified one as the desired value, so Auto Scaling will automatically launch one
instance in either of the subnets that we specified during the group's configuration
stage. You can view and verify the instance launch from the EC2 Management
Dashboard as well. With the instance successfully launched, we now move on to an
important part of verifying and actually testing the Auto Scaling configurations.
[ 207 ]
Manage Your Applications with Auto Scaling and Elastic Load Balancing
Verifying and testing Auto Scaling
Once your Auto Scaling configuration and deployment is completed, you are now
ready to go ahead and verify its validity. The first step to do so is by checking the
instance deployment itself using the ELB dashboard. Select the Elastic Load Balancer
option from the EC2 dashboard and select your ELB, as shown in the following
screenshot. Next, select the Instances tab and make a note of the instance's Status
column. It should display the status as InService as shown in the following screenshot.
This basically means that the instance is associated with the ELB and that its health
status is being continuously monitored by the ELB as well. In some cases, your Status
column may show the status as Pending, so don't worry, give it a bit more time and
the status should change back to InService again. You can optionally even view the
number of instances launched as well as the overall health of your AZ by viewing the
Instance Count as well as the Health columns highlighted in the following:
The next way to verify the Auto Scaling configuration as well as the ELB is by
actually viewing the ELB's DNS name in a local web browser. If all goes well, you
should see your application's landing page, or in this case the index.php page. Make
a note of the ELB's public DNS name and copy it over to any web browser of your
choice. Don't forget to append the /index.php landing path to the public DNS as
well, as shown in the following screenshot:
[ 208 ]
Chapter 7
Now in my case, the index.php file is nothing more than a simple PHP page that
displays the IP address of the running underlying instance along with a simple
welcome message.
The code for the index.php file is as follows:
This IP address is actually a part of the Web Server Subnet (us-west-2a) that we
created back in Chapter 5, Building Your Own Private Clouds Using Amazon VPC. If
you got it till this far, then you are on the right track! However, in case you don't see
your application's landing page here, then I might suggest changing your instance's
cooldown period as well as the ELB's Health Check period to suit your needs.
Once the verification is completed, you can go ahead and test your Auto Scaling
configuration. To do so, simply SSH into your launched instance and increase its
load using any load synthesizer tool you can find. I personally like Stress as well
as Lookbusy as they are really simple and easy to use. For this scenario, I used
Lookbusy to increase the instance's CPU Utilization to 60 percent. After a good 5
minutes of continuously loading the instance, the Increase Capacity scaling policy
was triggered and a new instance was automatically created to balance out the
application's load. You can verify this newly added instance by once again checking
your Instances tab from the ELB dashboard. You should see two instances now, as
shown in the following screenshot:
Similarly, gradually increase the CPU utilization of your instances from 60 percent to
70 percent and, finally, to 85 percent. If all goes well, you should see instances launch
up as per your increase scaling policy. Once the instances are all added, gradually
reduce the CPU utilization and watch the instances terminate automatically as well.
This is one of the real reasons why I love AWS so much! It's simple and so easy to
use! Once configured properly, your Auto Scaling service as well as ELB can run
completely independently from all manual interventions, so you can concentrate on
your business while the heavy lifting is taken care of by AWS itself!
[ 209 ]
Manage Your Applications with Auto Scaling and Elastic Load Balancing
Suspend, resume and delete Auto Scaling
Yup! You heard it right! You can additionally even suspend and resume an Auto
Scaling activity in your Auto Scaling Group. Why would someone want to do that?
Well, at times, you may want to run some minor configuration changes in your
instances but don't want that to trigger an auto scaling event; or there may be a
configuration issue in your Auto Scaling Group, or some problem with your application
and you want to investigate it but without starting up the Auto Scaling process, and so
on. In such cases, suspending an Auto Scaling process comes in really handy!
There is a small catch though! You cannot suspend or resume an Auto Scaling
activity using the EC2 Management Console; you have to use the AWS CLI to get
this done. Let us go through few commands and see how easy it is to put an Auto
Scaling Group in a suspended state.
First up, let us describe our Auto Scaling Group using the AWS CLI. Type in the
command as shown in the following and substitute the Auto Scaling Group's name
with your own:
# aws autoscaling describe-auto-scaling-groups --auto-scaling-groupnames US-WEST-PROD-WEB-ASG-1
Now go ahead and suspend the Auto Scaling Group using the following command:
# aws autoscaling suspend-processes --auto-scaling-group-name USWEST-PROD-WEB-ASG-1
Verify the status of your Auto Scaling Group by running the describe-autoscaling-groups command once again. You should see its status shown as
SUSPENDEDPROCESSES:
[ 210 ]
Chapter 7
To resume an Auto Scaling Group, type in the following command as shown in
the following:
# awsautoscaling resume-processes --auto-scaling-group-name US-WESTPROD-WEB-ASG-1
Once again, check the status of your Auto Scaling Group to make sure that the process
has indeed been initiated. Both the suspend-processes and resume-processes
commands can be used to suspend and resume, respectively, the entire scaling activity
in one go; however, if you wish to suspend or resume only a particular process from
the entire Auto Scaling activity, then you will have to use the --scaling-processes
attribute along with your suspend-processes and resume-processes commands.
For example, consider the following example that suspends an Auto Scaling's Health
Check process:
# awsautoscaling suspend-processes --auto-scaling-group-name US-WESTPROD-WEB-ASG-1 --scaling-processes HealthCheck
You can use the following set of processes with the suspend/resume commands:
Launch, Terminate, HealthCheck, ReplaceUnhealthy, AZRebalance,
AlarmNotification, ScheduledActions, and AddToLoadBalancer.
Tip: To know more about each of these individual processes,
refer to http://docs.aws.amazon.com/AutoScaling/
latest/DeveloperGuide/US_SuspendResume.html.
With your process suspend and resume operations done, the final thing left to do
is clean up your Auto Scaling Group. To delete your Auto Scaling Group, you first
need to make sure that it has no running instances in it. To do so, simply set the
minimum and maximum number of instances to zero! You can edit the group's
size by using either the Auto Scaling Group Dashboard or the AWS CLI using the
following set of commands:
First, set the minimum and maximum size to zero using the following command:
# awsautoscaling update-auto-scaling-group --auto-scaling-group-name
US-WEST-PROD-WEB-ASG-1 --max-size 0 --min-size 0
[ 211 ]
Manage Your Applications with Auto Scaling and Elastic Load Balancing
Before you proceed with the deletion of the group, check the status of the group
using the describe-auto-scaling-groups command. Make sure there are no
instances running at all. Next, type in the following command to delete your Auto
Scaling Group:
# awsautoscaling delete-auto-scaling-group --auto-scaling-group-name
US-WEST-PROD-WEB-ASG-1
Once the Auto scaling Group is deleted, you can optionally go ahead and delete the
Launch Configuration as well as the ELB using the following set of commands:
# awsautoscaling delete-launch-configuration --launch-configurationname US-WEST-PROD-WEB-LC-1
Similarly, delete the ELB as well using the following command:
# awselb delete-load-balancer US-WEST-PROD-LB-01
To know more about the various additional Auto Scaling CLI commands and
their usage, refer to http://docs.aws.amazon.com/cli/latest/reference/
autoscaling/index.html.
Planning your next steps
Well, we covered a lot about Auto Scaling and ELB in this chapter; however, there
are a few things that I would really recommend you try out next. First up, let's have
a look at ELB! In this chapter, we have looked only at how to set up and configure
a very basic HTTP-based ELB. However, in a real production scenario, this just
doesn't cut it. That's where you need to deploy your ELB using HTTPS and SSLlike secure protocols. To know more about how you can create and leverage ELBs
securely, refer to http://docs.aws.amazon.com/ElasticLoadBalancing/latest/
DeveloperGuide/elb-https-load-balancers.html.
With your ELB securely configured, there's one additional step that you can
configure to enable easier access to your applications hosted on AWS. Route 53 is a
highly scalable and available DNS service provided by AWS that can be leveraged to
replace the long and complicated public DNS name of an ELB with something a bit
more convenient and easier to remember, such as all-about-dogs.com. Amazon
Route 53 effectively connects your user requests to infrastructure running in AWS,
whether it is your EC2 instances, or ELB, or even your S3 bucket. It can also be used
as a health check mechanism to periodically monitor the health of your application
and its endpoints. To know more about Route 53 and how you can leverage it with
your applications, refer to http://docs.aws.amazon.com/Route53/latest/
DeveloperGuide/routing-to-elb-load-balancer.html.
[ 212 ]
Chapter 7
On a similar note, there are a couple of things that you can try out for Auto Scaling
as well. First up is Scheduled Auto Scaling. We have already talked about it in
the beginning of this chapter, so I'll not dwell on it for long. Just a few pointers
that you should keep in mind when working with Scheduled Auto Scaling: Each
scheduled scaling action has to have a unique date and time provided to it in the
UTC format that is generally represented as YYYY-MM-DDThh:mm:ssZ. You can
create a recurring scheduled scaling activity as well; however, note that this will not
work side by side with a onetime scheduled activity. Last but not the least, AWS
currently does not support scheduled scaling using the EC2 Management Console,
which means that you will have to use the AWS CLI for it. To know more on how to
leverage Scheduled Auto Scaling for your environments, refer to http://docs.aws.
amazon.com/AutoScaling/latest/DeveloperGuide/schedule_time.html.
Note: Auto scaling can also be applied to AWS SQS. I think it is
nice to just mention it and point to further reading at:
http://docs.aws.amazon.com/AutoScaling/latest/
DeveloperGuide/as-using-sqs-queue.html.
The final recommendation with regard to Auto Scaling is something a bit new and
is called as Lifecycle Hooks. Hooks basically allow you to add a custom event to
your instances before they are actually terminated or added to the Auto Scaling
Group by the Auto Scaling service. These events can be anything from retrieving
logs from your instances to installing and configuring software, and so on. The main
idea behind hooks is very similar to the concept of suspending and resuming Auto
Scaling processes; however, here we don't suspend the Auto Scaling activity but just
put the instance into a definite wait state. It is during this wait state that you get to
perform your selected action on your instance. Do note, however, that the default
wait period is only an hour. So if you do not perform any action over your instances
during this period, Auto Scaling will automatically terminate the instances once the
time has passed. And very similar to Schedule Auto Scaling, you cannot perform
Lifecycle Hooks using the EC2 Management Console. You can use the AWS CLI or
the AWS API to perform the same.
Recommendations and best practices
Now that we have come to the end of this chapter, let us look at some key
recommendations and best practices that you need to keep in mind when working
with ELBs and Auto Scaling:
•
Plan and provide the ELB with enough of a grace period (by default its 300
seconds) so that it does not put an instance in the unhealthy state even before
the application has had time to initialize completely.
[ 213 ]
Manage Your Applications with Auto Scaling and Elastic Load Balancing
•
Use Amazon Route 53 and provide a suitable domain name for your
applications. Additionally, leverage Route 53 to balance your application's
load across multiple regions as well.
•
Although ELBs can handle large loads (up to 20k/sec), they can only do
so if the load increases gradually, say over a period of several hours. If
your application spikes in load in minutes rather than hours, then you
are better off by using pre-warmed ELBs. To know more about prewarmed ELBs and how to get them, refer to http://aws.amazon.com/
articles/1636185810492479#pre-warming.
•
Configure HTTPS and SSL listeners for your ELB whenever possible.
•
Plan your Auto Scaling well in advance. This includes deciding on the
number of instances that will be required for your application as well as the
type of instance family.
•
Plan on which monitoring metrics (CPUUtilization, MemoryUtilization, Disk Space
used, and so on) you tend to use and set up the scaling policies accordingly.
•
Deploy your Auto Scaling Groups across multiple AZs. This provides an
additional layer of high availability in case an entire AZ should fail.
•
Prepare, test, and bootstrap your application on an AMI before adding it to the
Auto Scaling activity. Try and keep your application as decoupled as possible.
•
Always monitor and set up notifications for your Auto Scaling activity.
This will help you track and maintain your application's as well as
instances' performance.
Summary
So it's been a really long but interesting chapter and I really hope that you have
got to learn about Amazon ELB as well as Auto Scaling as much as possible. Let's
quickly recap all the things covered so far in this chapter.
To begin with, we talked about the importance of Auto Scaling and how it proves to
be super beneficial when compared to the traditional mode of scaling infrastructure.
You then learnt a bit about AWS Auto Scaling and its core components. Next, you
learnt about a new service offering called Elastic Load Balancers and saw how easy it
is to deploy one for your own use. Toward the end of this chapter, we also deployed
our first Launch Configuration and an Auto Scaling Group and, finally, topped it all
off with some simple steps to help verify and test the entire setup.
In the next chapter we are going to dive into the amazing world of databases and
learn how AWS provides some simple and easy to use database services, so stay
tuned for lots more coming your way!
[ 214 ]
Database-as-a-Service
Using Amazon RDS
In the previous chapter, you learnt a lot about the concepts of Auto Scaling and
Elastic Load Balancing, and how you can leverage them to host highly scalable and
fault tolerant applications.
In this chapter, we are going to shift our attention from all those web servers and
EC2 instances and talk about more on the database offerings provided by AWS, with
some special emphasis on Amazon RDS. This chapter will help you understand the
overall concept of RDS and even demonstrate how you can leverage RDS in your
own application's hosting environment. We will also be studying some of AWS's
other popular database-as-a-service options along the way; so let's get started
without any further ado!
An overview of Amazon RDS
Before we go ahead and dive into the amazing world of RDS, it is essential to
understand what exactly AWS provides you when it comes to database-as-a-service
offerings and how can you effectively use them. To start off, AWS provides a bunch
of awesome and really simple-to-use database services that are broadly divided
into two classes: the relational databases, which consist of your MySQL and Oracle
databases, and the non-relational databases, which consist of a propriety NoSQL
database similar to MongoDB. Each of these database services is designed by AWS to
provide you with the utmost ease and flexibility of use along with built-in robustness
and fault tolerance. This means that all you need to do as an end user or a developer
is simply configure the databases service once, run it just as you would run any
standard database without worrying about the internal complexities of clustering,
sharding, and so on, and only pay for the amount of resources that you use! Now
that's awesome, isn't it!
[ 215 ]
Database-as-a-Service Using Amazon RDS
However, there is a small catch to this! Since the service is provided and maintained
by AWS, you as a user or developer are not provided with all the fine tuning and
configuration settings that you would generally find if you were to install and
configure a database on your own. If you really want to have complete control over
your databases and their configurations, then you might as well install them on EC2
instances directly. Then you can fine-tune them just as you would on any traditional
OS, but remember that in doing so, you will have to take care of the database and all
its inner complexities.
With these basic concepts in mind, let us go ahead and learn a thing or two about
Amazon Relational Database Service (RDS). Amazon RDS is a database service that
basically allows you to configure and scale your popular relational databases such
as MySQL and Oracle based on your requirements. Besides the database, RDS also
provides additional features such as automated backup mechanisms, point-in-time
recovery, replication options such as multi-AZ deployments and Read Replicas, and
much more! Using these services you can get up and running with a completely
scalable and fault tolerant database in a matter of minutes, all with just a few clicks
of a button! And the best part of all this is that you don't need to make any major
changes to your existing applications or code. You can run your apps with RDS just
as you would run them with any other traditional hosted database, with one major
advantage: you don't bother about the underlying infrastructure or the database
management. It is all taken care of by AWS itself!
RDS currently supports five popular relational database engines, namely MySQL,
Oracle, Microsoft's SQL Server, PostgreSQL, and MariaDB as well. Besides these,
AWS also provides a MySQL-like propriety database called Amazon Aurora.
Aurora is a drop-in replacement for MySQL that provides up to five times the
performance that a standard MySQL database provides. It is specifically designed
to scale with ease without having any major consequences for your application or
code. How does it achieve that? Well, it uses a combination of something called as
an Aurora Cluster Volume, as well as one Primary Instance and one or more Aurora
Replicas. The Cluster Volume is nothing more than virtual database storage that
spans across multiple AZs. Each AZ is provided with a copy of the cluster data so
that the database is available even if an entire AZ goes offline. Each cluster gets one
Primary Instance that's responsible for performing all the read/write operations,
data modifications, and so on. With the Primary Instance, you also get a few Aurora
Replicas (also like Primary Instances). A Replica can only perform read operations
and is generally used to distribute the database's workload across the Cluster. You
can have up to 15 Replicas present in a Cluster besides the Primary Instance, as
shown in the following image:
[ 216 ]
Chapter 8
You can also read more on Amazon Aurora at
http://docs.aws.amazon.com/AmazonRDS/
latest/UserGuide/CHAP_Aurora.html.
With this basic information in mind, let us now understand some of RDS's core
components and take a look at how RDS actually works.
RDS instance types
To begin with, RDS does operate in a very similar way as EC2. Just as you have EC2
instances configured with a certain amount of CPU and storage resources, RDS too
has instances that are spun up each time you configure a database service. The major
difference between these instances and your traditional EC2 ones is that they cannot
be accessed remotely via SSH even if you want to. Why? Well, since it's a managed
service and everything is provided and maintained by AWS itself, there is no need
for you to SSH into them! Each instance already has a particular database engine
preinstalled and configured in it. All you need to do is select your particular instance
type and assign it some storage, and voila! There you have it! A running database
service of your choice in under 5 minutes! Let's have a quick look at some of the RDS
instance types and their common uses:
•
Micro instances (db.t1.micro): Just as we have micro instances in our EC2
environments, the same is also provided for RDS as well. Each database
micro instance is provided with just 1 CPU and approximately 600 MB of
RAM, which is good enough if you just want to test RDS or play around
with it. This instance type, however, is strictly not recommended for any
production-based workloads at all. Along with this particular micro instance,
RDS also provides a slightly better instance type in the form of a db.m1.
small, which provides 1 CPU with a slightly better 1.7 GB RAM.
[ 217 ]
Database-as-a-Service Using Amazon RDS
•
Standard instances (db.m3): Besides your micro instances, RDS provides a
standard set of instance types that can be used on a daily basis for moderate
production workloads. This class of instance provides up to 8 CPUs and
about 30 GB of RAM as well, but more importantly, these instances are
specially created for better network performance as well.
•
Memory optimized (db.r3): As the name suggests, this instance class provides
really high-end, memory optimized instances that are capable of faster
performance and more computing capacity as compared to your standard
instance classes. This instance class provides a maximum of 32 CPUs with
a RAM capacity of up to 244 GB along with a network throughput of
10 GB/second.
The db.r3 DB instance classes are not presently
available in the South America (Sao Paulo) and AWS
GovCloud (US) regions.
•
Burst capable (db.t2): This instance class provides a baseline performance
level with the ability to burst to full CPU usage if required. This particular
class of database instance, however, can only be launched in a VPC
environment. The maximum CPU offered in this category is up to 2 CPUs
with approximately 8 GB of RAM.
Along with an instance type, each RDS instance is also backed by an EBS
volume. You can use this EBS volume for storing your database files, logs,
and lots more! More importantly, you can also select the type of storage to
go with your instances as per your requirements. Here's a quick look at the
different storage types provided with your RDS instances:
•
Magnetic (standard): Magnetic storage is an ideal choice for applications that
have a light to moderate I/O requirement. A magnetic volume can provide
up to 100 IOPS approximately on average with burst capability of up to
hundreds of IOPS. The disk sizes can range anywhere between 5 GB to 3 TB.
An important point to note here, however, is that since magnetic storage is
kind of shared, your overall performance can vary depending on the overall
resource usage by other customers as well.
[ 218 ]
Chapter 8
•
General purpose (SSD): These are the most commonly used storage types
from the lot and are generally a good choice of storage if you are running a
small to medium-sized database. General purpose or SSD-backed storage
can provided better performance as compared to your magnetic storage at
much lower latencies and higher IOPs. General purpose storage volumes
can provide a base performance of three IOPS/GB and have the ability to
burst up to 3,000 IOPS as per the requirements. These volumes can range in
size from 5 GB to 6 TB for MySQL, MariaDB, PostgreSQL, and Oracle DB
instances, and from 20 GB to 4 TB for SQL server DB instances.
•
Provisioned IOPs: Although general purpose volumes are good for
moderate database workloads, they are not a good option when it comes
to dedicated performance requirements and higher IOPs. In such cases,
provisioned IOPs are the best choice of storage type for your instances. You
can specify IOPs anywhere between the values 1,000 all the way up to 30,000
depending on the database engine you select as well as the amount of disk
size that you specify. A MySQL, MariaDB, PostgreSQL, or Oracle database
instance with approximately 6 TB of storage can get up to 30,000 IOPs.
Similarly, an SQL server DB instance with approximately 4 TB of disk size
can get up to 20,000 IOPs.
You cannot decrease the storage of your RDS instance once
it is allocated to it.
With the RDS instance types in mind, let's now look at some of the key services as
well as processes provided by Amazon RDS.
Multi-AZ deployments and Read Replicas
We all know the importance and the hard work needed to keep a database,
especially the one running a production workload up and running at all times.
This is no easy feat, especially when you have to manage the intricacies and all the
tedious configuration parameters. But thankfully, Amazon RDS provides us with a
very simple and easy-to-use framework, using which tasks such as providing high
availability to your databases, clustering, mirroring, and so on are all performed
using just a click of a button!
[ 219 ]
Database-as-a-Service Using Amazon RDS
Let's take high availability for example. RDS leverages your region's availability
zones and mirrors your entire primary database over to some other AZ present in the
same region. This is called as a Multi-AZ deployment and it can easily be enforced
using the RDS database deployment wizard. How does it work? Well it's quite
simple actually. It all starts when you first select the Multi-AZ deployment option
while deploying your database. At that moment, RDS will automatically create and
maintain another database as a standby replica in some different AZ. Now if you use a
MySQL, MariaDB, Oracle, or PostgreSQL as your database engine, then the mirroring
technology used by RDS is AWS propriety. Whereas, if you go for an SQL server
deployment, then the mirroring technology used is SQL server mirroring by default.
Once the standby replica database instance is created, it continuously syncs up with
the primary database instance from time to time, and in the event of a database failure
or even a planned maintenance activity, RDS will automatically failover from the
primary to the standby replica database instance within a couple of minutes:
Amazon RDS guarantees an SLA of 99.95 percent! To know
more about the RDS SLA agreement, refer to http://aws.
amazon.com/rds/sla/.
However remarkable and easy multi-AZ deployment may be, it still has some minor
drawbacks of its own. Firstly, you can't use a multi-AZ deployment for scaling out
your databases, and, secondly, there is no failover provided if your entire region
goes down. With these issues in mind, RDS provides an additional feature for our
database instances called as Read Replicas.
[ 220 ]
Chapter 8
Read Replicas are database instances that enable you to offload your primary
database instance's workloads by having all the read queries routed through
them. The data from your primary instance is copied asynchronously to the read
replica instance using the database engine's built-in replication engine. How does
it all work? Well it's very similar to the steps required for creating an AMI from a
running EC2 instance! First up, RDS will create a snapshot based on your primary
database instance. Next, this snapshot is used to span a read replica instance. Once
the instance is up and running, the database engine will then start the asynchronous
replication process such that whenever a change is made to the data in the primary,
it gets automatically replicated over to the read replica instance as well. You can then
connect your application to the new read replica and offload all your read queries
to it! As of date, RDS supports only MySQL, MariaDB, and PostgreSQL database
engines for read replicas:
You can create up to five Read Replicas for a given
database instance.
You can additionally use these Read Replicas as a failover mechanism as well by
deploying read replicas in a different region altogether. The only downside to this is
that you will have to manually promote the replica as a primary when the latter fails.
We will be creating and promoting a Read Replica later on in this chapter, but for now
let's look at how you can create and get started with your first database using RDS.
[ 221 ]
Database-as-a-Service Using Amazon RDS
Working with Amazon RDS
In this section, we are going to create our very first scalable database using the
Amazon RDS service. For simplicity, I will be deploying a simple MySQL database
using the RDS Management Console; however, you can use any of the database
engines provided by RDS for your testing purposes, including Oracle, MariaDB,
PostgreSQL, as well as SQL Server. Let's first examine our use case up to now:
For starters, we have already set up Auto Scaling and Load Balancing for our
application's web server instances (see Chapter 7, Manage Your Applications with Auto
Scaling and Elastic Load Balancing), as shown in the preceding image. We have also
created a separate private subnet in each AZ for hosting our database instances. These
subnets are named US-WEST-PROD-DB-1 (192.168.5.0/24) and US-WEST-PRODDB-2 (192.168.6.0/24), respectively. Another extremely important point here is that
the communication between the public subnets and the private subnets is also set up
using a combination of network ACLs as well as security groups that can be found in
Chapter 5, Building Your Own Private Clouds Using Amazon VPC. Now, if you haven't
been following this book from the very beginning, you might find all these things a bit
vague to set up all over again, but don't worry! You can replicate the next steps even
with a standalone VPC subnet as well.
[ 222 ]
Chapter 8
With our subnets in place, the next thing to do is jot down the database's essential
configuration parameters as well as plan whether you want to leverage a MultiAZ deployment and Read Replicas for your deployment or not. The configuration
parameters include the database name, the database engine's version to use, the
backup and maintenance window details, and so on. For this deployment, I will be
deploying my database using the Multi-AZ deployment option as well. Do note,
however, that the Multi-AZ deployment scheme is not included in the AWS Free
Tier eligibility and, hence, you will be charged for the same. To know more about the
costs associated with your RDS services, refer to https://aws.amazon.com/rds/
pricing/. Once you have thoroughly planned out these details, you can go ahead
and start off with the actual deployment of the database.
Getting started with MySQL on Amazon RDS
You can access and manage your RDS deployments by using the AWS CLI, the AWS
SDK, as well as the AWS Management Console. For this activity, we will be using the
AWS Management Console. Log on to your AWS account using the IAM credentials,
and from the AWS Management Console, select the RDS option from the Database
group, as shown in the following screenshot:
Next, from the RDS Management Dashboard, select the option Subnet Groups from
the navigation pane. A Subnet Group is an essential step toward setting up the
security of your database. For starters, a subnet group is a logical grouping or cluster
of one or more subnets that belong to a VPC; in this case, the cluster is of our two
database subnets (US-WEST-PROD-DB-1 and 2).When we first launch a DB Instance
in a VPC, the subnet group is responsible for providing the database instance with an
IP address from a preferred subnet present in a particular availability zone.
[ 223 ]
Database-as-a-Service Using Amazon RDS
To get started, provide a suitable Name and Description for your DB Subnet Group
as shown in the following screenshot. Next, from the VPC ID drop-down list, select
a VPC of your choice. In my case, I have selected the US-WEST-PROD-1 VPC
(192.168.0.0/16). Once your VPC is selected, you can now add the required set of
subnets to your DB Subnet Group. To do so, first select the preferred Availability
Zone and its corresponding Subnet ID. Click on Add to add your subnet to the DB
Subnet Group:
Now as a good practice, provide at least two subnets that are present in different
AZs for your DB Subnet Group. For example, in my case, I have provided two
private subnets that are present in us-west-2a (US-WEST-PROD-DB-1) and
us-west-2c (US-WEST-PROD-DB-2), respectively. Click on Create when done.
With this step complete, you can now go ahead and create your first RDS database
instance in your VPC!
Creating a MySQL DB instance
Creating an RDS database instance involves a simple four-step process. To
begin with, select the Instances option from the navigation pane, as shown in the
following screenshot. Next, select the Launch DB Instance button to bring up the
DB Launch Wizard:
[ 224 ]
Chapter 8
Step 1 – Select Engine
To get started, select the appropriate database engine of your choice. For our
scenario, I have selected the MySQL database; however, feel free to select any of the
database engines as per your requirements.
Step 2 – Production?
Now here comes the fun part! RDS basically allows you to create a database based
on your requirements; for example, a production database with multi-AZ support
and provisioned IOPS storage or a simpler database that has none of these addon features. With Multi-AZ deployments, your DB instance is guaranteed with a
monthly uptime SLA of 99.95 percent! However, because of such high SLAs, MultiAZ deployments are not covered under the AWS Free Tier usage scheme. Click on
Next Step to continue, as shown in the following screenshot:
Step 3: Specify DB Details
The next page of the wizard will help you configure some important settings for
your DB instance:
•
License Model: Select the appropriate database License Model as per your
database engine's selection. MySQL databases have only one license model;
that is, general-public-license. Other propriety databases such as Oracle
and SQL servers offer two license modes: Licenses Included and the BYOL
(Bring Your Own Licenses) model. With licenses included, AWS provides the
required license keys for your databases, so you don't have to separately
purchase one. Alternatively, you can even use the BYOL model to provide
your own licenses or obtain new ones from the database provider itself.
[ 225 ]
Database-as-a-Service Using Amazon RDS
•
DB Engine Version: Select the appropriate DB Engine Version as per your
requirements. RDS provides and supports a variety of database engine
versions that you can choose from. In this case, I have selected the MySQL
database engine version 5.6.23 as shown:
•
DB Instance Class: From this dropdown list, select the appropriate class of
DB instance you wish to provide for your DB instance. For a complete list
of supported instance class types, refer to http://docs.aws.amazon.com/
AmazonRDS/latest/UserGuide/Concepts.DBInstanceClass.html.
•
Multi-AZ Deployment: Select Yes from the dropdown list to ensure a multiAZ deployment for your database. Selecting No will create your DB instance
only in a single availability zone.
•
Storage type: Select an appropriate storage option from the dropdown list.
In this case, I have opted for General Purpose (SSD); however, you can also
select between Magnetic and Provisioned IOPS as well.
•
Allocate storage: Allocate some storage for your database instance. You can
provide anywhere between 5 GB to 6 TB.
[ 226 ]
Chapter 8
With these basic configurations out of the way, configure your database's settings as
shown in the following screenshot:
Here are the parameters you need to provide in Settings panel:
•
DB Instance Identifier: Provide a suitable name for your DB instance. This name
will be a unique representation of your DB instance in the region it is getting
deployed in. In my case, I have provided the name US-WEST-PROD-DB.
•
Master Username: Provide a suitable username for your MySQL database.
You will use this username to log in to your DB instance once it is deployed.
•
Master Password: Provide a strong password for your DB instance. You will
use this password to log in to your DB instance once it is deployed. You can
provide a password that's up to 41 characters long; however, do not provide
the following characters in it: (@, " , /).
With the settings configured, click on Next Step to proceed with your database's
configuration.
Step 4: Configure Advanced Settings
The final step of configuring your database instance can be split up into three parts.
The first part involves the setting up of the DB instance's Network & Security, that
includes selecting the VPC along with the Subnet Group that we created a while
back. The second part involves configuring various database options such as the
database name, the database port number on which the application can connect
to it, and so on. The final part consists of the database's Backup and Maintenance
window details. Let's have a quick look at each part a bit more in detail:
•
VPC: Select the name of the VPC that will host your MySQL DB instance.
You can optionally select the option Not in VPC as well if you wish to
deploy your DB instance in a standard EC2 Classic environment.
[ 227 ]
Database-as-a-Service Using Amazon RDS
•
Subnet Group: Select the newly created Subnet Group from the dropdown
list, as shown in the following screenshot:
•
Publicly Accessible: You can optionally set your DB instance to have public
connectivity by selecting Yes from the Publicly Accessible dropdown list;
however, as best practice, avoid making your DB instances public at all times.
•
Security Group(s): There are two levels of security group that you can use
here. The first is a DB security group that is basically used to control access
to your DB instances that are outside a VPC. When working with DB security
groups, you only need to specify the subnet CIDR associated with your DB
instance, and no DB port or protocol details are required. The second is your
traditional VPC security group that can be used to control access to your DB
instances that are present in a VPC. Here, however, you need to specify both
inbound and outbound firewall rules, each with associated port numbers and
supported protocols.
You can select one or more security groups here for your DB instance; in my case,
I have selected a VPC security group as shown in the previous screenshot. Just
remember to open up only the required ports whenever you work with VPC security
groups. In this case, I have opened up ports 3306 (MySQL) and 1433 (SQL Server).
Moving on to the second part of the Advanced Settings, we will now set up the
Database Options as shown in the following:
•
Database Name: Provide a suitable database name here. RDS will not create
and initialize any database unless you specify a name here.
•
Database Port: Provide the port number using which you wish to access
your database. MySQL's default port number is 3306:
[ 228 ]
Chapter 8
You will not be able to change the database port number
once the DB instance is created.
•
DB Parameter Group: DB parameter groups are logical groupings of
database engine configurations that you can apply to one or more DB
instances at the same time. RDS creates a default DB parameter group that
contains mostly AWS specific configuration settings and default values. You
cannot edit the default DB parameter group, so in order to make changes,
you will have to create a DB parameter group of your own. In this case, I
have left it as the default value.
•
Option Group: This option is similar to DB parameter groups in that they
too provide and support few additional configuration parameters that make
it easy to manage databases; for example, MySQL DB Instances support for
Memcached and so on. RDS currently supports option groups for Oracle,
MySQL, and SQL Server database engines. To know more about option
groups, refer to http://docs.aws.amazon.com/AmazonRDS/latest/
UserGuide/USER_WorkingWithOptionGroups.html.
•
Enable Encryption: RDS provides standard AES-256 encryption algorithms
for encrypting data at rest. This includes your DB instance, its associated
Read Replicas, DB Snapshots, as well as the automated backups. An
important point to note here is that encryption is not supported on the
t2.micro DB instances.
For encryption to work, you will need your DB instance to be one of the following
instance classes:
Instance Type
Supported Instance Class
db.m3.medium
General purpose (M3) current generation
db.m3.large
db.m3.xlarge
db.m3.2xlarge
[ 229 ]
Database-as-a-Service Using Amazon RDS
Instance Type
Supported Instance Class
db.r3.large
db.r3.xlarge
Memory optimized (R3) current generation
db.r3.2xlarge
db.r3.4xlarge
db.r3.8xlarge
Burst capable (T2) current generation
db.t2.large
In this case, we are not going to encrypt our DB instance, so select No from the
Enable Encryption field as shown in the previous screenshot.
The final part of the Advance Settings page is the Backup and Maintenance window
selection. Using this section, you can configure automated backups for your database
as well as provide designated maintenance windows for the same. You can set the
Backup Retention Period as well as the Backup window's Start Time and Duration,
as shown in the following screenshot. In my case, I have opted for the backups to be
taken at 12:00AM UTC. If you do not supply a backup window time, then RDS will
automatically assign a 30-minute backup window based on your region. For example,
the default backup time block for the US West (Oregon) region is 06:00 to 14:00 UTC.
RDS will select a 30-minute backup window from this block on a random basis:
[ 230 ]
Chapter 8
The same can be set for your Maintenance window as well. An additional feature
provided here is that you can choose whether or not the database should receive
automated minor version updates from AWS or not. These minor updates for the
database engine will be automatically installed on the database based on their
availability as well as the maintenance window's time frame. You can make changes
in these settings even after your DB instance is created; however, remember that the
backup window should not overlap the weekly maintenance window for your DB
instance. Once you have configured the settings, click on Launch DB Instance to
complete the launch process.
The DB instance will take a good 2 to 3 minutes to spin up depending on whether
you have opted for the Multi-AZ deployment or not. You can check the status of
your newly created DB instance using the RDS management dashboard, as shown in
the following screenshot. Simply check the Status column for all the stat us changes
that occur while your DB instance is created:
Let's take a quick look at some of the states that a DB instance goes through during
its lifecycle:
•
Creating: This is the first stage of any DB instance's lifecycle where the
instance is actually created by RDS. During this time, your database will
remain inaccessible.
•
Modifying: This state occurs whenever the DB instance enters any
modifications either set by you or by RDS itself.
•
Backing-up: RDS will automatically take a backup of your DB instance when
it is first created. You can view all your DB instance snapshots using the
Snapshots option on the navigation pane.
•
Available: This status indicates that your DB instance is available and
ready for use. You can now access your database remotely by copying the
database's endpoint.
[ 231 ]
Database-as-a-Service Using Amazon RDS
To read the complete list of DB instance status messages, refer
to http://docs.aws.amazon.com/AmazonRDS/latest/
UserGuide/Overview.DBInstance.Status.html.
Connecting remotely to your DB instance
Once your DB Instance is in the Available state, you can now access your database
remotely from any other EC2 instance or even remotely from your desktop if you
have set the security groups right. In my case, I have launched a new EC2 instance
in my VPC, as shown in the following screenshot. This instance is a part of the Web
ServerSubnet (US-WEST-PROD-WEB-1) we used in the previous chapter:
The first thing to do is make sure you have the required MySQL client packages
installed on your web server EC2 instance. To do so, simply type in the following
commands as shown:
# sudo yum install mysql
With the client installed, you can now access your remote RDS database using the
following command:
# mysql -u -h -p
Substitute the values with your master username and password that you set for
the database during the Create DB Instance phase. Here, is
the Endpoint (.xxxxxxxxxxxx.us-west-2.rds.amazonaws.
com:3306) that is provided by each DB instance when it is created. If all goes well,
you should see the MYSQL command prompt. Go ahead and run a few MYSQL
commands and check whether your database was created or not:
> show databases;
[ 232 ]
Chapter 8
You can additionally connect your database with tools such as the MySQL
workbench as well. Just remember to provide your database's endpoint in the
hostname field followed by the master username and password. With the database
connected successfully, you can now run a few simple tests just to make sure that the
DB instance is accessible and working as expected.
Testing your database
In this section, I'm going to show you a simple exercise, using which you can test the
configurations and the working of your database, as well as your DB instance. First
up, log in to your database using the following command as done earlier:
# mysql -u -h -p
Next, let's go ahead and create a simple dummy table called doge. Type the
following command in your MySQL prompt:
CREATE TABLE doge
(
idint(11) NOT NULL auto_increment,
namevarchar(255),
description text,
sizeenum('small','medium','large'),
date timestamp(6),
PRIMARY KEY (id)
);
Fill in some data in your newly created table using the following INSERT commands:
INSERT INTO doge (name,description ,size,date) VALUES('Xena','Black
Labrador Retreiver','medium',NOW());
INSERT INTO doge (name,description ,size,date) VALUES('Betsy','Browndachs
hund','medium',NOW());
INSERT INTO doge (name,description ,size,date) VALUES('Shark','Mix breadHalf dachshund','small',NOW());
[ 233 ]
Database-as-a-Service Using Amazon RDS
With your basic table and data created, you can now access the same using your
Web Server Instances. In my case, I'm using a simple PHP script (index.php) that is
installed on the web server instance itself to print the database name as well as the
table's data. Remember that as per our use case scenario, the web server instances are
isolated from the database instances by different subnets as well as security groups
and network ACLs, so make sure your subnets can communicate with each other
correctly before testing. If all goes well, you should see your database, as well as the
newly created table and its data, as shown in the following screenshot:
Modifying your DB instances
Once your DB Instances are created and launched, you can further modify them
using two methods. The first method is by using the AWS CLI, where you can use the
modify-db-instance command along with a bunch of options to specify and assign
new parameters and values to your DB instances. For example, we need to expand the
storage as well as the instance class of our DB instance so that it can accommodate the
growing database's needs. To do so, type in the following command:
# aws
modify-db-instance --db-instance-identifier us-west-prod-db \
--allocate-storage 100 \
--db-instance-class db.m1.large
[ 234 ]
Chapter 8
The preceding command will update the DB instance with the identifier us-westprod-db with 100 GB of disk space and change its instance class to db.m1.large as
well. The CLI provides a host of additional parameters as well which you can use to
configure almost any aspect of your DB Instance, such as the master user's password,
the preferred backup and maintenance window, the database engine versions, and
so on. You can find the complete list of parameters and their descriptions at http://
docs.aws.amazon.com/cli/latest/reference/rds/modify-db-instance.html.
Changing the instance class of a DB instance will result in an
outage, so plan the changes in advance and perform them
during the maintenance window only.
The second method of modifying the DB instances is by using the RDS Management
dashboard itself. Select your DB instance, and from the Instance Actions dropdown
list, select the Modify option, as shown in the following screenshot:
Using the Modify page, you can change almost all configuration parameters of
your DB instance just as you would by using the CLI. You can optionally set the
changes to take effect immediately as well by selecting the Apply Immediately
checkbox. Note, however, that by doing so, your DB instance will try to accept the
made changes instantly, which can cause outages and even performance degradation
at certain times. So as good practice, avoid setting this checkbox unless absolutely
necessary. Changes made otherwise are reflected in your DB instance during its next
scheduled maintenance window.
[ 235 ]
Database-as-a-Service Using Amazon RDS
Backing up DB instances
RDS provides two mechanisms using which you can perform backups of your
database instances as per your requirements. The first is an automated backup job
that can be scheduled to run at a particular backup job interval, preferably when
the database is at its least utilization point. This is something that we configured
sometime back while creating our DB Instance. The second is a manual database
instance snapshot that you can perform at any point in time. Here's a look at both
the techniques in a bit more detail:
•
Automated backup: Automated backups are conducted periodically by RDS
on a daily user configured backup window. These backups are kept stored by
RDS until the backup's retention period doesn't expire. By default, your first
new database instance will have these backups enabled for ease of use. You
can use these backups to restore your database to any point in time, down
to the last second. The only thing that you need to be aware of is the slight
freeze in storage IO operations that occurs when RDS actually performs
the backups.
•
DB snapshots: DB snapshots are point in-time snapshots that are created by
you as and when required. To create a DB Instance snapshot, select the Take
Snapshot option from the Instance Actions dropdown list, as shown in the
following screenshot:
This will bring up the Take Snapshot page where all you need to do is provide a
suitable name for your snapshot and click on Take Snapshot to complete the process.
Alternatively, you can also use the AWS CLI for performing a manual DB instance
snapshot. Type in the following command:
# aws rds-create-db-snapshot -i -s
[ 236 ]
Chapter 8
Once you have taken your DB instance snapshot, you can view them on the RDS
Management dashboard under the Snapshots page, as shown in the following
screenshot:
The snapshot dashboard allows you to perform various operations on your DB
snapshots including copying DB snapshots from one region to another for high
availability, restoring the state of a DB instance based on a particular snapshot,
as well as options to migrate your MySQL database completely over to the
Amazon Aurora database engine!
Creating Read Replicas and promoting them
We have already discussed the concept of read replicas in some depth, and how they
can be useful for offloading the read operations from your primary DB instance as
well as providing a mechanism using which you can create and set up Read Replicas
across AWS regions. In this section, we are going to check out a few simple steps
using which you can create and set up read replicas for your own environment using
the RDS Management dashboard.
To get started, first select your newly created database from the RDS dashboard, as
shown in the following screenshot. Next, using the Instance Actions tab, select the
Create Read Replica option:
[ 237 ]
Database-as-a-Service Using Amazon RDS
This will bring up the Create Read Replica DB Instance page, as shown in the
following screenshot. The page is pretty self-explanatory and easy to configure. Start
off by selecting an appropriate DB Instance Class from the dropdown list. You can
alternatively select a high-end DB instance class here as compared to the primary DB
instance. Next, select a Storage Type for your Read Replica DB instance. In this case,
I have opted to go for the General Purpose (SSD) volumes:
Next, select your primary DB instance as the source for your Read Replica using the
Read Replica Source dropdown list, and provide a suitable and unique name for your
Read Replica in the DB Instance Identifier field, as shown in the preceding screenshot.
Now comes the fun part where you actually get to specify where you wish to deploy
your Read Replica DB instance. Remember that you can have a maximum of five read
replicas for a single primary DB instance, so ideally have your replicas spread out
across the AZs that are present in your operating region or even have them residing
in a different region altogether. Select an appropriate Destination Region and its
corresponding Availability Zone. In this case, I have opted for the same region (US
West (Oregon)) as well as same AZ (us-west-2a) as my primary DB instance.
Besides the placement of your replica instance, you will also need to make it a part of
your existing DB Subnet Group. Select the same subnet group as provided for your
primary DB instance from the Destination DB Subnet Group field, as shown in the
following screenshot. Leave the rest of the fields to their default values and click on
the Create Read Replica option:
[ 238 ]
Chapter 8
Here's what will happen next. First up, RDS will start off by taking a snapshot of
your primary DB instance. During this process, the DB instance will face a brief
moment of IO freeze which is an expected behavior. Here's a handy tip that you
can use to avoid the IO freeze! Deploy your DB instances using the multi-AZ
deployment. Why? Well, because when it comes to taking the snapshot, RDS will
perform the snapshot on the primary DB instance's standby copy, thereby not
affecting any performance on your primary DB instance. Once the snapshot is taken,
RDS will start to spin up a new Read Replica based on your specified configurations,
as shown in the following screenshot.
During the replica's creation phase, your primary DB instance will change from
a backing up state to modifying, and ultimately back to available status once the
Replica is launched. Each Replica will behave as a DB instance on its own; hence,
each of them will be provided with a unique DB endpoint as well. Refer to the
following screenshot as an example of multiple Replicas:
In case you create multiple Replicas at the same time using the same primary DB
instance, then RDS will only perform the snapshot activity once, and that too at the
start of the first replica's creation process. You can even perform the same process
using the AWS CLI's rds-create-db-instance-read-replica command, as
shown in the following:
# rds-create-db-instance-read-replica -s
In this case, RDS will create a new Replica DB instance based on your supplied
database identifier value while keeping all the configurations same as that of
the primary DB instance. To know more about the various options and related
operations that you can perform using this command, refer to http://docs.aws.
amazon.com/AmazonRDS/latest/CommandLineReference//CLIReference-cmdCreateDBInstanceReadReplica.html.
[ 239 ]
Database-as-a-Service Using Amazon RDS
Once your Read Replica instance is created and functioning, you can promote it as
a primary DB instance as well. This feature comes in real handy when you have to
perform a DB failure recovery, where your primary DB instance fails and you need
to direct all traffic to the newly promoted Read Replica, and so on. To promote any
Read Replica instance, all you need to do is select it from the RDS Management
dashboard and select the Promote Read Replica option from the Instance Action
drop-down list. This will bring up the Promote Read Replica page, as shown in the
following screenshot:
Enable the automatic backups as well as fill out the Backup Retention Period and
Backup Window details as per your requirements. Click on Continue to proceed
to the next page. Acknowledge the Replica instance's promotion and click on Yes,
Promote to complete the process.
As a good practice, always enable the automatic backups
option for your DB Instances.
During this process, your Read Replica instance will reboot itself once, post which
it will become available as a standalone DB instance. You can then perform all sorts
of activities on this DB instance, such as taking manual snapshots and creating new
Read Replica instances from it as well.
You can promote a Read Replica using the AWS CLI as well. Type in the following
command while replacing the value with your own Replica
instance's name:
# rds-promote-read-replica \
--backup-retention-period 7 \
--preferred-backup-window 00:00-00:30
The preceding command will promote the to a standalone
primary DB instance. It will also set the automated backup retention period to 7 and
configure the backup window for half an hour between 00:00 UTC and 00:30 UTC.
[ 240 ]
Chapter 8
Logging and monitoring your DB instance
AWS provides a variety of tools and services to track and monitor the performance
of your DB instances—the most popular and commonly used being Amazon
CloudWatch itself. Besides this, RDS, too, comes with a list of simple tools that you
can use to keep an eye on your DB instances. For example, you can list and view the
DB instance's alarms and events by simply selecting the DB instance from the RDS
Management dashboard, as shown in the following screenshot:
You can additionally view the DB instance's security group and snapshot events
using this page as well. RDS will store the events for a period of 14 days, after
which they are deleted. The DB instance quick view page also displays the DB
instance's memory as well as storage utilization in near real time. Each of these fields
has a custom threshold that RDS sets. If the threshold value is crossed, RDS will
automatically trigger notifications and alarms to inform you about the same. You can
also view the database's Read/Write IOPS value using this page.
RDS also provides a page using which you can view the DB Instance's real time
performance graphs. To do so, simply select Launch DB Instance and the Show
Monitoring option, as shown in the following screenshot:
[ 241 ]
Database-as-a-Service Using Amazon RDS
Each graph can be further expanded by selecting it. You can optionally view graphs
for the past hour or a later duration by selecting the appropriate time from the Time
Range dropdown list.
Furthermore, RDS also allows you to view your database's essential logs using the
RDS Management dashboard. Select your DB instance, and from the dashboard,
select the Logs option. This will bring up the Logs page, as shown in the following
screenshot:
You can use this page to view as well as download the appropriate logs as per your
requirements. RDS obtains logs from the database at short, regular intervals (mostly
5 minutes) and stores them in files that rotate as well. Selecting the Watch option
adjoining a log file will display the log file in real time within your browser. You can
view up to 1,000 lines of your logs at a time using this feature.
Cleaning up your DB instances
Once you have completed work with your DB instances, it is equally important to
clean up your environment as well. You can delete a DB instance at any time you
want using both the RDS Management dashboard and the AWS CLI.
To delete a DB instance using the RDS Management dashboard, select the Delete
option from the Instance Actions dropdown list. You will be prompted to Create
a final Snapshot? for your DB instance before you proceed, as shown in the
following screenshot:
[ 242 ]
Chapter 8
It is strongly recommended that you create a snapshot of your DB instance before
you go ahead and delete it. Once you select the Delete option, RDS will delete the DB
instance along with all the automated backups that were taken earlier. The manual
snapshots, however, are not deleted and thus can be used to restore your DB instance
to its original state if you want to revert to your original settings.
To delete a DB instance using the AWS CLI, simply type in the following command
and replace with the name of your DB instance:
# aws rds-delete-db-instance \
--final-db-snapshot-identifier MyFinalDBSnapshot
The command will delete your DB Instance but will additionally first create a
snapshot for it by the name of MyFinalDBSnapshot.
Planning your next steps
There are a ton of amazing things that you can do besides the steps that we have
covered in this chapter. To begin with, try out encrypting your databases using
the Encryption facility provided by RDS itself. RDS encrypts as well as decrypts
data without any major impact on your DB instance's performance. The encryption
process can only be set up during the DB instance's initial deployment phase, so plan
and use the facility before you actually deploy your database. An important point to
remember here, though, is that encryption is not supported on the t1.micro instance
class. You will have to use the memory optimized (R3) or burst capable (T2) for the
same. To know more about how you can use and set up the encryption on your DB
instances, refer to http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/
Overview.Encryption.html.
Besides RDS, AWS also provides a host of other database-related services that
you can try out depending on your application's needs. For example, if you are
looking for a high performance and low cost non-relational database, then Amazon
DynamoDB is your obvious choice! With DynamoDB you can actually set up
and start using a NoSQL database in a matter of minutes! Plus, you don't have to
worry about any of the management overheads such as clustering or scaling. RDS
automatically replicates and synchronizes your data across multiple AZs present in a
region, thereby providing high availability and durability.
[ 243 ]
Database-as-a-Service Using Amazon RDS
AWS also provides a highly scalable caching service in the form of Amazon
ElastiCache. You can use this service as an in-memory caching service for your web
applications and improve their overall performance by retrieving information much
faster than a disk-based database. Along with this, AWS also provides a petabyte
scalable data warehousing service called as RedShift that you can use to query
extremely large datasets. It's easy to set up and the best part is that it scales as per
your needs! The more data you feed it, the larger it grows, and all this without any
upfront costs or commitments! To know more about these services and how to use
them, refer to https://aws.amazon.com/running_databases.
Recommendations and best practices
Here are some key recommendations and good practices to keep in mind when
working with RDS:
•
To begin with, always monitor your DB instances for overall performance
and usage. Leverage CloudWatch and its metrics to set thresholds and
customized alarms that notify you in case of any issues.
•
Additionally, you can also enable event notifications for your DB instance
that will inform you of all the events that occur with your instance.
•
Leverage Multi-AZ deployments in conjunction with Read Replicas to
increase the overall availability of your database along with its performance.
•
Always enable automatic snapshots for your DB instances. Also take manual
snapshots of your DB instances before performing any maintenance activities
on them.
•
For a small to medium-sized database, set the storage type of your DB
instance to General Purpose (SSD).
•
If your database has a high performance requirement, then do use the DB
instances with Provisioned IOPS.
•
Tune your options group as well as your parameters group to improve your
database's overall performance.
•
Secure your DB instances with a proper security group and encryption
facilities. Also remember to assign and use IAM users with specific rights
and privileges.
[ 244 ]
Chapter 8
Summary
With this we come to the end of yet another chapter and yet another awesome AWS
service. Let's quickly recap all the things covered so far. First up, we started off by
understanding and learning what RDS is all about, followed by an in-depth look
at DB instances and how RDS actually works. Next, you learnt about how you can
leverage high availability for your databases by using something called as MultiAZ deployments and Read Replicas. You also learnt a lot of basic actions that you
can perform on a database using RDS, such as creating a DB instance, connecting to
one, testing it, and so on. Toward the end, we topped it all off with some easy-toremember recommendations and best practices that you should keep in mind when
working with RDS!
In the next chapter, we are going to look at yet another AWS core service that
provides us with virtually unlimited storage for all our needs! So stick around,
there's more to learn just around the corner!
[ 245 ]
Working with Simple
Storage Service
In the previous chapter, we covered a lot about Amazon RDS and how you can
leverage it to host highly scalable and fault-tolerant databases.
In this chapter, we will be exploring yet another popular and widely used AWS
core service, that is, the Simple Storage Service (S3). This chapter will cover many
important aspects of S3, such as its use cases, its various terms and terminologies,
along with a few steps on how to use S3 to store and retrieve objects. It will also go
through few simple steps using which you can archive your data using both the
AWS Management Console and the AWS CLI. So, buckle up and get ready for an
awesome time.
Introducing Amazon S3
Ever used Dropbox to store and back up your important data and files? Or how
about Netflix to watch your favorite TV shows online? Both Dropbox and Netflix
have one very interesting thing in common, which you may have guessed already!
They are both using Amazon S3 to store and retrieve data. How much data are we
talking about here? Well, way back in 2008, S3 was storing approximately 30 billion
objects or unique data elements in it. This number has grown exponentially ever
since with approximately 2 trillion objects reportedly stored in S3 as of April 2013, so
no prizes for guessing what this number has gone up to today! But enough numbers,
let's learn a bit more about what Amazon S3 actually is.
[ 247 ]
Working with Simple Storage Service
To begin with, Amazon S3 is a highly scalable, durable, and low cost storage as a
service option provided by AWS for everyone to use. Using S3, you can upload
virtually any file, folder, or data from anywhere on the web and retrieve it just as
easily all the while paying only for the storage that you use! Now that's amazing,
isn't it!
How much of data can you upload to S3? Well, its virtually unlimited, so you can
feel free to upload your songs, movies, high-resolution pictures, anything and
everything goes! S3 will treat each of the files that you upload as individual objects
and store them redundantly across the underlying secure hardware. You don't have
to worry about the replication process or even for the hardware's scalability, it is all
taken care of by AWS itself.
You can leverage S3 for a variety of purposes; a few listed as follows:
•
S3 serves as an ideal place to store and back up all your data, including
pictures, videos, documents, and so on
•
Since each object in S3 is provided and accessed by a web URL, you can
actually host a website on it as well, provided your website is completely
static by design
You can upload objects as small as 1 KB or as large as 5 TB at a
time to Amazon S3, with virtually unlimited storage capacity.
How does it all work? Well, to begin with, you first need to create something called
as a Bucket. A Bucket is a top level entity in S3 and acts as a logical container that
will hold all your objects. You can create multiple buckets and store various objects
in them as you please; however, there are a few pointers that you must always keep
in mind when working with them:
•
Bucket names have to be unique across your entire AWS account.
•
Bucket names always start in lowercase. Although you can specify uppercase
letters in your names, it is advised that you avoid doing so.
•
Buckets can be accessed globally; however, they are still created and located
within a particular region.
[ 248 ]
Chapter 9
It is equally important to note that S3 is not some hierarchical organization of objects,
although you can create folders and store objects in them. Folders are just a logical
representation that AWS provides you with for easier object storing and arrangement,
but underneath all this, S3 really does not use any hierarchy at all as it is a flat storage
system. This enables S3 to add new storage and scale virtually without any limits,
without having to worry about the objects that already reside in it.
Buckets also provide us with some simple access control mechanisms, using which
you can restrict users to operations such as create, delete, or list all the objects
present in the bucket. You can even assign the bucket permissions that govern who
can upload or even download data from it.
S3 also provides different storage classes for the objects that you store on it. Each
storage class has its own performance and cost associated with it, as described here:
•
Standard: This is the default storage class used to store all your objects unless
you specify a different value. This storage class comes in really handy for
common S3 workloads where data is accessed on a frequent basis. For the
first 1 TB that you use per month, the Standard storage class will cost about
$0.0300 per GB of data stored on S3.
•
Standard_IA: This is a special storage class used to store objects that are less
frequently accessed. You can transition an object to move from the Standard
to the Standard_IA storage class after say a period of 30 days. This helps
you save on the costs as Standard_IA will cost you about $0.0125 per GB.
However, note that there is a separate minimal retrieval fee in case you use
Standard_IA as your storage class.
•
Glacier: Glacier is yet another less frequently accessed storage class, with
a retrieval time of nearly 2 to 3 hours. You cannot assign an object with the
Glacier storage class directly. The object has to be transitioned from Standard
or Standard_IA to Glacier and vice versa when it comes to retrieving as well.
Glacier storage is by far the cheapest, costing about $0.007 per GB of data
stored in S3.
•
Reduced Redundancy Storage (RRS): Each of the previous storage classes
are designed to sustain data losses by replicating data across multiple data
centers. RRS, however, is designed for non-critical data and also maintains
fewer redundant copies of data compared to its counterparts. This enables
you to reduce costs, however, with less durability (only 99.99%).
With this basic understanding in mind, let's see how we can use the AWS
Management Console to create and upload a few objects to a bucket of our choice.
[ 249 ]
Working with Simple Storage Service
Getting started with S3
Getting started with S3 is by far the simplest and most straightforward thing you
will ever do! Simply log in to your AWS account using your IAM credentials and
select the S3 option:
This will bring up the S3 Management Dashboard as shown in the following
screenshot. You can use this dashboard to create, list, upload, and delete objects from
buckets as well as provide fine-grained permissions and access control rights as well.
Let's start off by creating a simple bucket for our demo website all-about-dogs.com.
Creating buckets
To get started with your first bucket, simply select the Create Bucket option from the
S3 dashboard. Provide a suitable name for your new bucket. Remember, your bucket
name will have to be unique and will have to start with a lowercase character. Next,
select a particular Region where you would like your bucket to be created. Although
buckets are global entities in AWS, you still need to provide it with a Region option.
This comes in handy, especially when you wish to create a bucket close to your
location to optimize latency or meet certain regulatory compliances. Also keep in
mind that you are not allowed to change the bucket's name after it has been created,
so make sure you provide it a correct and meaningful name before you proceed. In
this case, I opted to create my bucket in the Oregon region, as shown:
[ 250 ]
Chapter 9
You are not charged for creating a bucket; you are charged
only for storing objects in the bucket and for transferring
objects in and out of the bucket.
You can optionally enable logging for your bucket as well, by selecting the Set Up
Logging option. This will store detailed access logs of your bucket to a different
bucket of your choice. By default, logging of a bucket is disabled; however, you
can always re-enable it even after your bucket is created. AWS will not charge you
for any of the logging that it will perform; however, it will still charge you for the
storage capacity that your logs will consume on S3.
Log records are delivered by S3 on a best effort basis. This means
that most records will be delivered to your bucket within a few
hours of their creation. However, not all logs may be delivered on
time, depending on the overall traffic your bucket handles.
Once your details are in place, select the Create option to create your new bucket.
The bucket is created within a few seconds, and you should see the following
landing page for your bucket as shown here:
You can even create one or more folders in your bucket by selecting the Create
Folder option. Folders are just a nice way to represent and categorize your objects
more effectively. You can even perform additional operations on your bucket using
this dashboard, such as assign permissions, enable logging, versioning, cross-region
replication, and so on. We shall be exploring each of these operations in detail
throughout this chapter, but for now, let's upload some objects to our newly
created bucket.
[ 251 ]
Working with Simple Storage Service
Uploading your first object to a bucket
With your bucket now created, you can easily upload any object to it. But first, let's
take a closer look at what an S3 object actually comprises:
•
Key: This is nothing but the unique name using which you upload objects
into S3. Each object has its own key, which can be used to identify and
retrieve the object when necessary.
•
Value: This can be defined as a sequence of bytes used to store the object's
content. As discussed previously, an object's value can range anywhere
between zero bytes to 5 TB.
•
Version ID: This is yet another entity that in conjunction with a key can be
used to uniquely identify an object by S3. Version ID is equally important
for maintaining an object's version count. Using S3, you can keep multiple
versions of an object in a single bucket. Versioning helps protect your objects
against accidental overwrites as well as deletions by maintaining a separate
version number for each new object that you upload into the bucket. By default,
versioning is disabled on your bucket and thus your objects get the version ID
Null. It is your responsibility to enable versioning on your buckets in case you
wish to protect them against accidental deletions and overwrites.
•
Metadata: These are nothing but simple name-value pairs that define some
information regarding a given object. There are two types of metadata
provided in S3: the first is system-defined metadata, which is generated by S3
itself when an object is first uploaded and it generally contains information
such as the object's creation date, version ID, storage class, and so on. The
second is user-defined metadata, which, as the name suggests, requires you
as a user to provide some additional name-value information to your objects
when they are uploaded.
•
Sub resources: Sub resources are a set of resources that can be associated
with either objects or buckets. S3 currently supports two sub resources with
objects. The first is an Access Control List (ACL), which consists of a list of
users and permissions that are granted access over the object. The second sub
resource is called torrent and is used to return the torrent file associated with
any particular object.
Apart from the traditional client-server model, S3 also
supports the BitTorrent protocol that you can use to distribute
your objects over a large number of users. To know more
about BitTorrent protocol and how you can leverage it using
S3, go to http://docs.aws.amazon.com/AmazonS3/
latest/dev/S3Torrent.html.
[ 252 ]
Chapter 9
•
Access control: This provides the access information of a particular
object. You can control access to your objects that are stored in S3 using a
combination of access control mechanisms that are discussed briefly in the
later parts of this chapter.
With this basic understanding in mind, we are now ready to upload our first object
into S3. You can upload objects directly into your buckets or within sub folders that
you may have created. To get started, simply select the Upload option from your
bucket or folder. In the Upload–Select Files and Folders dialog box, shown in the
following screenshot, you can browse and select any video, document, media file,
and so on, of your choice and upload it to S3. The wizard also provides you with an
advanced enhanced uploader that is basically a Java applet that can help you upload
entire folders into S3 with ease.
The enhanced uploader will be installed on your local machine's browser only once
per console session. In my case, I opted to upload a single video file to my S3 bucket,
and hence opted to select the Add Files option rather than Enhanced Uploader.
Once your required files are loaded into the wizard, start the upload process
by selecting the Start Upload button. You can view the transfer process of your
individual files by selecting the transfer panel shown here:
Keep in mind though that you can upload files up to 5 TB in size at any given point
in time using the Upload—Select the Files and Folders dialog box. Once the file is
uploaded to S3, you are ready to view it!
[ 253 ]
Working with Simple Storage Service
Viewing uploaded objects
Each uploaded object in S3 is provided with a URL that you can use to view
your object using a browser of your choice. The URL is in the following format:
https://s3.amazonaws.com//. You can view the
URL of your object by simply selecting your object from the dashboard and the
Properties option, as shown in the following screenshot. Copy the URL presented
against the Link attribute and paste it into a web browser of your choice:
But wait! That's really not going to help you much. In fact if you did try and access
your object's URL from a browser, you make have ended up with an XML-based
access denied warning! Well, that's because all buckets and objects in S3 are set to
private by default. You can change this default behavior by simply selecting the
Make Public option from the Actions tab. This will modify your object's permissions
and enable everyone to view as well as download your object. You can even perform
the same action by selecting the Permissions option from the object's Properties tab.
Once the object is made public, you can view it using the URL copied earlier.
Accessing buckets and objects using S3CMD
Now here's the fun part! S3 provides a wide variety of CLI tools using which you can
manipulate your buckets and objects; one of the popular ones being S3CMD. In this
section, we will walk through some simple steps to install S3CMD on a local Linux
box and then check out some cool commands using which you can work with S3.
[ 254 ]
Chapter 9
So what is S3CMD? In simple words it's a Python-based open source tool used to
query any cloud storage service that supports and uses the S3 protocol, including
Amazon S3 and even Google's cloud storage. S3CMD can be installed and configured
on any major Linux OS, such as CentOS, RHEL, Ubuntu, and so on, and even comes
with Windows OS support in the form of a commercial tool called S3Express. The
main reason I'm talking about S3CMD here is because of its high versatility and use.
If you are capable of writing bash scripts and cron jobs, then you can easily perform
automated backups of your files and folders in S3 using S3CMD in a few easy steps.
To know more about S3CMD, check
http://s3tools.org/s3cmd.
First off, let's get started by installing S3CMD on our trusty Linux box. To do so,
simply type in the following command. However, S3CMD requires Python Version
2.6 and above, so make sure this prerequisite is met before you proceed further:
# wget http://sourceforge.net/projects/s3tools/files/s3cmd/1.6.0/s3cmd1.6.0.tar.gz
Here as a screenshot of the preceding command:
Once the tar is downloaded, extract its contents using the following command:
# tar -xvfs3cmd-1.6.0.tar.gz
Next, install S3CMD on your Linux box by executing the following command:
# cd s3cmd-1.6.0
# python setup.py install
[ 255 ]
Working with Simple Storage Service
With this, S3CMD is now successfully installed. The next step is to configure it to
work with your Amazon S3. To do so, type in the following command and follow the
on screen instructions provided:
# s3cmd --configure
The configuration utility will request an AWSaccess and Secret Key. It will also
prompt you to set your operating region, which is set to US by default. You can even
enable S3CMD to communicate with Amazon S3 using the https protocol; however,
do note that this setting can have a slight impact on S3CMD's overall performance.
The entire configuration is saved locally in the Linux user's home directory in a file
called .s3cfg. Once done, you are now ready to test your S3CMD! Here are some
basic commands that you can use to query your Amazon S3. First up, let's list our
Buckets using the following S3CMD command:
# s3cmdlss3://
You should see the contents of your bucket listed using this command. If you do not
specify the bucket name, then by default the s3cmdls command will print out all
the buckets present in that particular region. Next, let's try to upload some data to
our bucket. Uploading is performed using the s3cmd put command and conversely,
downloads are performed using the s3cmd get command. Type in the following
command in your Linux terminal:
# s3cmd put -r /opt s3://
This command will recursively upload all the contents present in the /opt directory
to the bucket name that you specify. Remember one important thing though! Trailing
slashes after the /opt directory would have copied only the directory's content over
to the bucket, but not the directory itself.
[ 256 ]
Chapter 9
To know more about the usage of various s3cmd commands
visit http://s3tools.org/usage.
On a similar note, you can perform a wide variety of operations using the s3cmd
tool. For example, you can upload your instance log files to an S3 bucket on a
periodic basis. Here is a base snippet of the /etc/logrotate.d/httpd file where we
use the s3cmd command with the sync attribute. The sync attribute is a really useful
tool for transferring files over to Amazon S3. sync performs conditional transfers,
which means that only files that don't exist at the destination in the same version
are transferred.
In this snippet, we are assuming that a bucket with the name httpd-logger has
already been created in S3. The code will sync the instance's httpd error log file
(/etc/httpd/logs/error_log) and the httpd access log file (/etc/httpd/logs/
access_log) and transfer them to their respective folders:
BUCKET=httpd-logger
INSTANCE_ID=`curl --silent http://169.254.169.254/latest/meta-data/
instance-id`
/usr/bin/s3cmd -m text/plain sync /etc/httpd/logs/access_log*
s3://${BUCKET}/httpd/access_log/instance=${INSTANCE_ID}/
/usr/bin/s3cmd -m text/plain sync /etc/httpd/logs/error_log*
s3://${BUCKET}/httppd/error_log/instance=${INSTANCE_ID}/
Managing an object's and bucket's
permissions
Just like we talked about IAM permissions and policies back in Chapter 2, Security and
Access Management, security and access management, S3 too provides permissions
and policies using which you can control access to both your buckets and the objects
they contain. In this section, we will have a quick look at two such methods provided
by S3, namely resource-based policies and user-based policies, as follows:
•
Resource-based policies: Resource-based policies are simple Json-based
policies that are generally created and enforced on S3 resources by the
bucket or the resource owner themselves. These S3 resources include the
object lifecycle management configuration information, the versioning
configuration, the website config details, and a few other parameters.
Resource-based policies can be further sub-classified into two types: Bucket
Policies and Access Control Lists (ACLs).
[ 257 ]
Working with Simple Storage Service
•
Bucket policies: These are enforced on the bucket level or on the individual
objects contained within it. Here is a simple example of a Bucket Policy
that basically will allow any user to perform any operation on the specified
bucket name, provided the request source is generated from the IP address
specified in the condition (23.226.51.110):
{
"Id": "Policy1448937262025",
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Stmt1448937260611",
"Effect": "Allow",
"Principal": "*",
"Action": "s3:*",
"Resource": "arn:aws:s3:::/*",
"Condition": {
"IpAddress": {"aws:SourceIp": "23.226.51.110"}
}
}
]
}
You will notice that a lot of the syntax actually matches up with what we have
already seen in Chapter 2, Security and Access Management, while discussing the
building blocks of an IAM policy. Well, here, most of the things remain the same.
The only notable difference will be the inclusion of the Principal element, which
lists the principals or owners that bucket policy controls access for. The Principal
element is not mandatory when creating an IAM policy as it is by default the entity
to which the IAM policy is going to be attached. The best part of all this is that AWS
provides a really easy to use policy generator tool that you can use to interactively
set and create your S3 policies with. You can try out and create your own policies at
http://awspolicygen.s3.amazonaws.com/policygen.html.
How do you apply bucket policies? Well that's really simple! Select your bucket from
the S3 dashboard and from the Properties panel, select the Permissions drop-down
menu, as shown in the following screenshot. Here, select the Add bucket policy
option. This will bring up a Bucket Policy Editor dialog box using which you can
type in your policy or even use the AWS policy generator to create one for yourself
interactively.
[ 258 ]
Chapter 9
Remember to save your policy in the Bucket Policy Editor before closing the
dialog box.
ACLs are very similar to bucket policies. An ACL is basically a list of grant operations
comprising a grantee and a set of permissions granted. Every bucket that you create
in S3 along with each object that you upload gets a set of ACLs with them. ACLs are
a great way to control which users get access to your buckets and resources, whether
they are AWS users or even some random normal user. To view your bucket's or
object's ACLs, simply select the bucket or object from the S3 dashboard and select the
Properties option. There, select the Permissions drop-down to view the associated
ACLs, as shown here:
[ 259 ]
Working with Simple Storage Service
A default ACL is provided with each object that you upload
into S3. This default ACL has one grant attribute set for the
owner of the bucket.
ACLs have a predefined set of user groups created using which you can configure
access control for your buckets. These user groups include Everyone, Any
Authenticated AWSUser, Log Delivery, along with your bucket's creator. ACLs can
enforce the following set of permissions (read/write) over an object as well as bucket.
On the basis of these permissions, a user can perform operations such as uploading
new files or delete existing objects. Here's a quick look at the ACL permissions
provided by S3 and how they are associated with both buckets and objects:
Permissions
Associated with Buckets
Associated with Objects
READ
Users can list the object names, their size,
and last modified date from a Bucket.
Users can download the object.
WRITE
Users can upload new objects in your
Bucket. They can also delete files on
which they don't have permission.
Users can replace the object or
delete it.
READ_ACP
Users can read the ACL associated with a
Bucket, but cannot make any writes to it.
Users can read the ACL
associated with that object but
cannot make writes to it.
WRITE_
ACP
Users can modify the ACL associated
with the bucket.
Users can modify the ACL
associated with the object.
FULL_
CONTROL
Users get READ, WRITE, READ_ACP,
and WRITE_ACP permissions on the
associated bucket.
Users get READ, READ_ACP,
and WRITE_ACP permissions
on the associated object.
Bucket ACLs are completely independent from Object ACLs.
This means that bucket ACLs can be different from ACLs set
on any Object contained in a bucket.
Now the obvious question running through your mind right now is what do I use for
my S3 buckets and objects? ACLs or bucket policies? The answer for this is two folds.
First off, understand that the main difference between an ACL and a bucket policy
is that an ACL grants access permissions to buckets or objects individually, whereas
a bucket policy will help you write a policy that will either grant or deny access to a
bucket or its objects. Ideally, you can use ACLs when each object in the bucket needs
to be provided with some explicit grant permissions. You will also need to use ACLs
instead of bucket policies when your policy's size reaches 20 KB. Bucket policies have
a size limit of 20KB, so if you have a very large number of objects and users to grant
access to, you might want to consider the use of ACLs.
[ 260 ]
Chapter 9
User-based policies, perhaps the most simple and easy to use, the user-based policies
are nothing more than simple IAM policies that you can create and use to manage
access to your Amazon S3 resources. Using these policies, you can create users,
groups, and roles in your account and attach specific access permissions to them.
Consider this simple example IAM policy in which we grant the user-specific rights
to put objects into S3, to get objects, list them, as well as delete them. Notice the
syntax remains quite the same as we have seen throughout this book. You can create
multiple such IAM policies and attach them to your users and groups as described
here in the AWSS3 documentation page: http://docs.aws.amazon.com/AmazonS3/
latest/dev/example-policies-s3.html.
The following is an example of the IAM policy:
{
"Statement": [
{
"Effect":"Allow",
"Action": [
"s3:PutObject",
"s3:GetObject",
"s3:DeleteObject",
"s3:ListBucket"
],
"Resource":"arn:aws:s3:::/*"
}
]
}
Using buckets to host your websites
Yes! Believe it or not, you can actually use Amazon S3 to host your websites,
provided that they are static in nature. How does it work? Well, its quite simple and
easy, actually! All you need to do is create a bucket and enable the website hosting
attribute on it. Once done, you can easily upload the website's index.html page
along with the other static web pages and voila! You got yourself a really simple
website up and running in a matter of minutes.
[ 261 ]
Working with Simple Storage Service
Here's a simple example in which I used my previously created bucket as a website
host. All you need to do is select your bucket and from the Properties panel select
the Static Website Hosting option as shown here:
Here, you will see an Endpoint (http://.s3-website-.
amazonaws.com) provided to you. This is your website's end URL, which you can
copy and paste in a web browser to view your website; however, before you do that,
don't forget to make your website public! To do so, copy and paste the following
bucket policy in your bucket's Policy Editor dialog box:
{
"Version":"2012-10-17",
"Statement":[
{
"Sid":"BucketWebsiteHostingPolicy",
"Effect":"Allow",
"Principal": "*",
"Action":["s3:GetObject"],
"Resource":["arn:aws:s3:::/*"
]
}
]
}
Save the policy in the bucket policy editor, and then upload your index.html file as
well as an optional error.html file to your bucket. Type your endpoint URL in a web
browser of your choice, and you should see your website's landing (index.html) page.
[ 262 ]
Chapter 9
Using static website hosting, you can also redirect all requests to an alternate DNS
hostname or even to an alternate bucket. Simply select the Redirect all requests to
another host name option and provide an alternative bucket's name in the Redirect
all requests to field, as shown in the following screenshot. Alternatively, you can
even setup Amazon Route53 to act as your DNS provider by providing a few DNS
records and a valid new domain name for your website, such as all-about-dogs.com:
S3 events and notification
Amazon S3 provides an easy-to-use notification feature that can send notifications
in case a certain event gets triggered in your bucket. Currently, S3 supports
notifications on the following set of events:
•
Object created: This event includes PUT, POST, and COPY operations and
even something called as Complete Multi-part Uploads. Multi-part uploads
is a feature leveraged by S3 where a large object is broken down into smaller,
more manageable chunks (approx 10 MB), and then each chunk is uploaded
to an S3 bucket in a parallel fashion, thereby cutting down on the overall
upload time as well as costs.
•
Objects removed: This event includes any delete operations that are
performed either on the bucket or on the objects contained within it.
•
Object lost: Don't worry! We are not talking about S3 misplacing any of your
objects here! This event is raised only when an object of the RRS storage class
has been lost.
[ 263 ]
Working with Simple Storage Service
To enable the notification service, select your bucket and from the Properties panel,
select the Events drop-down menu, as shown. Fill in the required details and click on
Save once done to enable the notification service:
The parameters are as follows:
•
•
•
•
•
•
Name: Provide a suitable name for your notification alert. In this case, I
named my alert -PUT-ALERT for notifying me against any
PUT operations that are performed on the bucket.
Events: Type in the event for which you wish to get notified. Here I have
specified the PUT event, but you can specify anything from POST, COPY,
CompleteMultiPartUpload, to DELETE, based on your requirements.
Prefix: This is an optional attribute and is used to limit the notifications of an
object based on its matching characters. For example, notify me in case any
PUT operation is performed in the Images directory.
Suffix: Once again this is an optional attribute that is used to limit the
notifications of an object based on its suffix. For example, notify me in case
any PUT operation is performed and the key contains a .png as a suffix.
Send To: Currently, S3 supports sending notifications to three AWS services,
namely SNS, SQS, and Amazon Lambda.
SNS topic: Select a SNS topic using which S3 will send notifications to. You
can optionally create a new SNS topic here as well.
Once all the fields are filled in, click on Save to enable the notification service.
Remember to alter the Bucket's access policy to allow S3 to publish statements to SNS
using the sample policy snippet provided here. Replace with an
actual SNS Topic ARN value (arn:aws:sns:aws-region:account-id:topic-name)
and with your bucket name:
{
"Version": "2008-10-17",
[ 264 ]
Chapter 9
"Id": "Policy1448937262025",
"Statement": [
{
"Sid": "Stmt1448937260611",
"Effect": "Allow",
"Principal": {
"Service": "s3.amazonaws.com"
},
"Action": [
"SNS:Publish"
],
"Resource": "",
"Condition": {
"ArnLike": {
"aws:SourceArn": "arn:aws:s3:*:*:"
}
}
}
]
}
Bucket versioning and lifecycle management
Versioning is perhaps the most important and useful feature provided by S3. In a
way, it is a means to create and maintain multiple copies of a single object present
in a bucket. Each copy of the same object that you upload into the bucket receives a
unique version ID, which can later be used to retrieve and restore the object in case
of an accidental deletion or failure. To enable versioning on a bucket, all you need
to do is select the particular bucket from the S3 dashboard, and from the Properties
panel, select the Versioning drop-down menu, as shown here:
[ 265 ]
Working with Simple Storage Service
By default, versioning is disabled on all buckets, so you will have to explicitly enable
versioning on the ones that require it. Once enabled, all newly uploaded objects
will receive a unique version ID. Older objects stored in the bucket prior to enabling
versioning also contain a version ID parameter, but the value of that is set to null.
However, once versioning is enabled on the bucket, it applies to all the objects
contained in the bucket, so all objects uploaded henceforth, new or old, will obtain
version IDs. An important point to remember here is that once you have enabled
versioning on a bucket, there is no way you can disable it. You can, however,
suspend it by selecting the Suspend Versioning option.
Each object version that you upload is an entire object in itself
and hence each version upload you do will be charged the
normal S3 rates for storage as well as data transfers.
So how do you upload versions of your objects to a bucket? Well, the process
remains absolutely the same for any object that you upload into S3. The only
difference now is that with each new version of the object that you upload, S3 will
assign it a unique version ID and store it in the same bucket as its originator. To list
the different versions of an object, all you need to do is toggle between the Hide and
Show versions buttons as shown in the following screenshot.
Selecting the Show versions will display all the uploaded versions of that object
including the object's creation date, version ID, and size. You can then download
any of the versioned objects that you want by simply selecting it from the dashboard
and, from the Actions drop-down menu, select the Download option. In case the
download request is issued on the main object, then S3 will simply fetch the latest
uploaded version of that object and download it for you:
[ 266 ]
Chapter 9
Versioning also comes in real handy when you want to protect your data from any
accidental deletions or overwrites. How? Well, to put in simple words, when you try
to DELETE an object that is versioned in S3, you actually don't wipe it from existence
immediately. In fact, S3 will insert something called as a delete marker in the bucket
and that marker becomes the current version of the object with a new version ID.
When you try to GET an object whose current version is a delete marker, Amazon S3
behaves as though the object has been deleted and returns a 404 error even though
that object is not physically deleted from the bucket. To permanently wipe out the
object, you will need to use the DELETE object along with its version ID. Want to
try it out? Then go ahead and delete a main object from your versioned S3 bucket.
You will notice that although the main object is successfully deleted and not visible
from the S3 dashboard, its versions are still pretty much intact. So even if this was an
accidental DELETE operation, you can still retrieve the main image from the version
ID! Amazing, isn't it!
Another sweet part of enabling versioning on a bucket is that you can specify an
object's transition or lifecycle as well. This feature comes in real handy when you
want the objects stored in your bucket to get auto-archived to, let's say, Amazon
Glacier after a long period of storage in S3, or wish to transition the storage class
of an object from Standard to Standard_IA for infrequent access. Logs are a classic
example of where this feature comes in really handy. Your bucket can store the logs
for up to a week's duration using standard storage and then post that you transition
the logs to either Glacier for archiving or even delete them permanently. To enable
lifecycle management, you will need to select your bucket from the S3 Dashboard
and from the Properties panel, select the Lifecycle drop-down menu, as shown:
Next, select the Add rule option to create a lifecycle rule for your bucket. This will
pop up the Lifecycle Rules wizard using which you can configure as well as choose
the target objects for which you wish to enable the lifecycle management. The first
page of the wizard is Choose Rule Target, where you can specify either a prefix or
the entire bucket to which the lifecycle rules shall apply. A prefix is more or less like
a folder name that you can specify, for example, in my case I provided the prefix
Videos/, which is an actual directory inside my bucket. Select Configure Rule to
move on to the next page of the wizard.
[ 267 ]
Working with Simple Storage Service
You can use lifecycle rules to manage all versions of your
objects; both current as well as previously created.
In the Action on Current Version page, you get to choose and specify the type of
action you wish to perform over your selected objects. For example, you can enable
an object's transition to the Standard - Infrequent Access Storage class 30 days after
the object's creation date, which is the ideal time to set for an object, or you can even
enable Archiving to the Glacier Storage Class post 60 days your object's creation,
and so on. There are a few rules that you have to keep in mind, however, when
performing transition actions:
•
You cannot transition from Standard-IA to Standard or reduced redundancy
•
You cannot transition from Glacier to any other storage class
•
You cannot transition from any storage class to reduced redundancy
The following screenshot shows the lifecycle configuration:
If you are happy with your lifecycle configuration, click on Review to complete the
process. In the Review and Name pages, you can specify an optional name for this
particular transition rule as well. Make sure you review your rules correctly before
selecting the Create and Activate Rule options. Post your rules creation. You can
further edit or even delete them from the same Lifecycle drop-down menu found in
the Properties panel of the S3 dashboard. To know more about lifecycle management
and how you can leverage it for your objects and buckets, check http://docs.aws.
amazon.com/AmazonS3/latest/UG/LifecycleConfiguration.html.
[ 268 ]
Chapter 9
Cross-Region Replication
Versioning in S3 also provides us with yet another easy to use feature, which is
called Cross-Region Replication. As the name suggests, this feature enables you to
copy the contents of your bucket or the bucket itself asynchronously to a bucket
present in some different AWS region. The copy process will basically copy all
aspects of your source bucket and objects, including their creation date and time,
version IDs, metadata, and so on, over to the destination bucket.
To enable Cross-Region Replication on a bucket, all you need to do is first make sure
that the bucket has versioning enabled. Next, from the Properties panel, select the
option Cross-Region Replication, as shown here:
Click on Enable Cross-Region Replication to get things started. You can either select
the entire bucket as the Source or even specify a prefix in the bucket, for example,
Images/. Next, select an appropriate Destination Region for your bucket replication.
In my case, I opted for the Northern California region. Select the Destination Bucket
of your choice as well. This destination bucket is not auto created by S3, so it's your
responsibility to go ahead and create one in the destination region of your choice:
[ 269 ]
Working with Simple Storage Service
You can optionally change the Destination Storage Class of your replicated
objects as well. For example, if you wish to minimize your costs of replication, you
can instruct S3 to store all replicated objects in the destination bucket using the
Standard-IA storage class. Finally, you will also need to setup an IAM Role to grant
S3 permission to replicate objects on your behalf. Click on Create/ Select IAM Role
to bring up the IAM Dashboard. There, select the Roles option from the navigation
pane and attach the following role policy:
{
"Version":"2012-10-17",
"Statement":[
{
"Effect":"Allow",
"Principal":{
"Service":"s3.amazonaws.com"
},
"Action":"sts:AssumeRole"
}
]
}
Once done, S3 will asynchronously copy the contents from your source bucket
over to the destination bucket. This includes any new objects that you add to the
source bucket as well. An important point to note here is that using Cross-Region
Replication, S3 will only replicate your actions over to the destination bucket, such
as adding, copying, and deleting objects. Lifecycle configuration actions, such as
transitioning objects from Standard to Standard-IA or to Glacier are not replicated.
You can however, configure the same lifecycle configurations as your source bucket
over to your destination buckets manually.
Planning your next steps
There are plenty of amazing things that you can configure and try on S3 besides the
steps we have covered in this chapter. For example, you can leverage the encryption
functionality provided by S3 to encrypt data in transit as well as at rest. Amazon S3
supports two types of encryption technique especially for this purpose: client-side
encryption and server-side encryption.
[ 270 ]
Chapter 9
Client-side encryption comes in really handy when you as an end user want to
manage the encryption process, the encryption keys, tools, and so on. Generally, this
encryption process is performed on the object before it gets uploaded to S3. You can
also protect your data in transit using client-side encryption facilities such as SSL.
Server-side encryption is where Amazon S3 encrypts and decrypts your data for you
before it is stored within its data centers. Server-side encryption can be leveraged
along with AWS Key Management Service (KMS) as well as with Amazon S3
managed keys. You can read about both in depth using this link http://docs.aws.
amazon.com/AmazonS3/latest/dev/UsingEncryption.html.
Another feature worth trying out in S3 is the presigned URLs. These URLs are used
to provide temporary access for downloading any particular object from S3. Each
URL comes with its own expiry date and time, which denies access to the object once
it expires. S3 provides SDKs in Java and .NET using which you can create your own
pre-signed URLs. To read more about presigned URLs and how to generate them for
your own objects, go to http://docs.aws.amazon.com/AmazonS3/latest/dev/
ShareObjectPreSignedURL.html.
Recommendations and best practices
Here are some key best practices and recommendations that you ought to keep in
mind when working with Amazon S3:
•
Before creating your buckets, plan and choose a region that has closer
proximity to your users. You may also want to consider any legal or
regulatory compliance before selecting a particular region.
•
Leverage S3's versioning and lifecycle management for automatically
archiving or cleaning up of your buckets and objects. This will help you save
a lot on storage costs as well.
•
Employ server-side encryption for encrypting your data at rest. Although
all objects and buckets in S3 are private by default, you can still enforce
additional security by encrypting them using either AWSKMS or using S3
Managed Keys.
•
Design and use bucket policies for restricting delete operations on buckets.
You can even enable Multi-factor authentication (MFA) for certain users
who will be required to provide additional authentication to perform a
change in an object's version ID or even delete it.
•
Leverage multi-part upload when it comes to uploading large objects into
Amazon S3. Using multi-part uploads, you can even resume your upload
process in case it was abruptly stopped or failed.
[ 271 ]
Working with Simple Storage Service
•
You can optionally enable Amazon CloudFront to speed up your static
website's performance as well by caching your HTML code, photos, and
videos. To know more about how you can leverage CloudFront for your S3
buckets, go to http://docs.aws.amazon.com/gettingstarted/latest/
swh/getting-started-create-cfdist.html.
Summary
So yet another chapter and yet another awesome AWS Service walkthrough comes to
an end! Let's take a quick flashback into the things we've learned so far.
First off, we started by learning what exactly Amazon S3 is, along with the various
storage class options provided with it. Next, we saw how easy and effortless it is to
create buckets, upload objects into them, and view them. We even tried out a simple
CLI tool called as S3CMD for syncing log files from your instances over to an S3
bucket. Toward the end, you even learned how to host static websites on S3 and
enable lifecycle management on objects as well.
In the next and final chapter, you will be learning a bit more about a few key AWS
services, such as Route53 and CloudFront, along with a quick look at some of the
newer AWS service offerings, such as Amazon EFS and ECS, so stay tuned!
[ 272 ]
Extended AWS Services for
Your Applications
In the previous chapter, you learned a lot about how you can leverage S3 to store
your objects and even perform some pretty interesting and useful lifecycle operations
on them.
In this final chapter, we will be exploring a few additional AWS services that
you can leverage to enhance your application's overall performance as well as
availability. The two services that I'm going to cover are Amazon Route53 and
Amazon CloudFront. After this, we will also take a quick look at some of the recently
launched AWS services and products and how you can leverage them for your own
environments, along with a final word on how to get going with AWS; so without
further ado, let's get busy!
Introducing Amazon Route53
Amazon Route53 is a highly available and scalable authoritative Domain Name
Service (DNS), which is responsible for routing users to internet-based applications.
How does it do that? Well, Route53 works like any other DNS but on a much larger
scale. It translates names such as www.all-about-dogs.com to either an instance's IP
address, such as 192.168.0.15, or even to an elastic load balancer's or Amazon S3's
endpoints. But that's not all! The real power of Route53 comes with its ability to route
traffic intelligently, which is achieved with the use of health checks and route-based
traffic flows that route traffic based on latencies and geographies, and in a weighted
round robin fashion. Recently, Route53 has also launched its very own domain
registration service, using which you can register your very own custom domain
names with AWS at absolutely nominal rates.
[ 273 ]
Extended AWS Services for Your Applications
In this section, we will be exploring a few of these features along with some simple
to follow examples and use cases that you can use to extend the functionality and
availability of your applications.
Working with Route53
Getting started with Route53 is a very simple and straightforward process. From the
AWS Management Console, select the Route53 option from the Networking group,
as shown here:
This will bring up the Route53 Management Console for the first time. Using this
console, you can create and configure your very own custom domain names as well
as configure health checks and traffic routing policies as per your requirements. First
up, let's go ahead and create a simple hosted zone for our demo application allabout-dogs.com.
A hosted zone is nothing more than a logical container that holds information on
how you wish to route traffic for your application. You create a hosted zone for
your custom domain (all-about-dogs.com) and then create one or more resource
record sets to tell the DNS service how you want traffic to be routed for that domain.
If your Hosted Zone routes traffic over the Internet, then it is called a public hosted
zone, and conversely if you are routing within an Amazon VPC, it is called a private
hosted zone. Once your hosted zone is created, Route 53 will automatically create a
Name Server (NS) record and a Start Of Authority (SOA) record for that zone. The
NS record identifies the four name servers that Route53 creates for you. You can then
provide these four NS records to a registrar or your DNS service provider, such as
https://in.godaddy.com/, so that your application's DNS queries are routed to
Amazon Route53's name servers.
You can create more than one hosted zone with the same
name and add different resources to it.
[ 274 ]
Chapter 10
Here is a pictorial representation of how things actually work out. In this case, our
demo application's domain name is already registered with http://www.godaddy.
com/; however, you can alternatively use any other domain provider of your choice,
or Route53's newly launched domain registrar service as well:
Creating hosted zones
To create your very own hosted zone, select the Create Hosted Zone option from
the Route53 dashboard. This will bring up the Create Hosted Zone panel, as shown
in the following screenshot. Provide a suitable Domain Name (in this case, I have
provided our demo application's name all-about-dogs.com) and an optional
Comment as well. Next, from the Type drop-down menu, select the option Public
Hosted Zone, as shown. Click on Create once all the required fields are filled in:
[ 275 ]
Extended AWS Services for Your Applications
You should get two record sets created as shown. The first record set is your hosted
zone's NS record set that contains the four name servers, and the second record set
is the SOA record set. Each new hosted zone that you create will contain its own
unique NS and SOA record sets:
Once your hosted zone is created, the next step is where you create a new resource
record set. A resource record set basically tells the DNS how the traffic should be
routed for that particular domain. For example, you may want to route all Internet
traffic for your domain name all-about-dogs.com to a specific IP address of an
instance or an ELB, or you may even want all your e-mails to be routed to a specific
mail server with the domain name mail.all-about-dogs.com and so on.
Each resource record set that you create will include the name of your domain, a
record type, and other miscellaneous information applicable to that particular record
type, such as Time To Live (TTL), routing policies, and health checks. Here are some
of the commonly used record set types provided by Route53:
•
A record: A simple IP address in the form 192.168.0.15. You can provide
the IP address of an EC2 instance in this set.
•
AAAA record: A simple IPv6 IP address in the form 2001:0db8:85a3:0000:
0000:8a2e:0370:7334.
•
CNAME records: A Canonical Name (CNAME) is basically a record set that
acts as an alias, pointing one domain name to another. For example, you can
route all internet traffic from your domain name all-about-dogs.com to an
ELBal's DNS name.
[ 276 ]
Chapter 10
•
MX records: MX records are used to specify the priority and domain name of
a mail server, for example, 10 mail.all-about-dogs.com; here, 10 specifies
the priority of the MX record.
To create your own record set, simply select the Create Record Set option from the
hosted zone dashboard. This will bring up the Create Record Set panel as shown
in the following screenshot. Now in my case, I want to route all of my incoming
application traffic on the ELB, which is hosted in us-west-2, via a domain name of
www.all-about-dogs.com. In this case, I provided the value www in the record set
Name field. Alternatively, you can provide any meaningful value there as per your
requirements. Next, from the Type drop-down menu, select the CNAME - Canonical
Name option. We will be using CNAME because the ELB is provided by a default
DNS name, which you can obtain by selecting your ELB from the EC2 Management
Dashboard and noting down its DNS name, which is generally of the following
format: US-WEST-PROD-LB-01-1582564436.us-west-2.elb.amazonaws.com. Once
done, paste the ELB's DNS name in the Value field. You can optionally edit the TTL
(seconds) value; however, I have chosen to stick with the default of 300 seconds.
Next, select an appropriate routing policy based on your requirements. There are five
routing policies provided, using which you can decide the best mechanism to route
your queries to Route53. They are as follows:
•
Simple routing: This is used when you only have a single source where you
want your queries to be routed to, such as a single web server or a standalone
server connected to Route53.
•
Weighted routing: As the name suggests, here each record set is provided
with some definite numeric value or weight that Route53 uses to
proportionately divide traffic into. As a result, you can now associate more
than one resource with a single DNS name using this approach. For example,
a single DNS name, such as all-about-dogs.com, routes traffic to five web
server instances, out of which three are assigned the weight 1, whereas
the rest of the two are assigned the weight 2. Then, on average, Route53
will select each of the first three resource record sets 1/7th of the time
(1+1+1+2+2=7) and the rest of the two record sets 2/7th of the time.
[ 277 ]
Extended AWS Services for Your Applications
•
Latency-based routing: Perhaps one of the most commonly used modes of
routing, latency-based routing comes in really handy when you have your
application's instances spread across multiple EC2 regions. In this case,
Route53 determines the origin of the request made to your application and
routes the traffic to the instances where the latency is at its minimum. For
example, consider your application is spread across the us-west (Oregon) and
a European region (Frankfurt) and a user request originates from, let's say,
Texas; then, Route53 will route that particular request to the us-west region
as the latency between Texas and Frankfurt is far greater than the latency
between Texas and Oregon.
Latency-based routing is based on latency measurements
performed over a period of time and can change as a result
of improved network connectivity and routing.
•
Geolocation routing: Geolocation works on a similar principle as latencybased routing with the difference that here you can choose the resources
that serve your application's traffic, based on the geographical location of
your users. For example, you can route all traffic that originates from India
and send it to a particular instance or an ELB. This method of routing has
numerous applications, such as restricting distribution of content to only
a few geographies, routing users to the same endpoint, and so on. You can
specify geographical locations by either continent or by country or even by
state in the United States.
•
Failover routing: Failover routing policy is yet another simple routing
technique that you can use to route traffic from one downed region to an
active region. Failover routing only works if your hosted zone is Public and
can only be configured for an active-passive failover scenario.
Coming back to our record set, for this particular scenario I opted to go for the
latency-based routing for my application. Select the Latency option from the Routing
Policy drop-down list. This will provide you with an option to select an associated
region to route your traffic to. In this case, I selected the us-west-2 region where my
ELB is currently present. The final step is to provide a meaningful and unique Set
ID or description for this particular record set. Once done, complete the record set
creation process by selecting the Create option, as shown:
[ 278 ]
Chapter 10
Once your record set is created, it will be displayed along the NS and the SOA records
that were created by Route53 when the hosted zone was first deployed. You can create
additional record sets for your hosted zone using the same steps as mentioned.
Getting started with traffic flow
It's all ok to create individual record sets using the previously described method, but
what if you had multiple resources present across multiple locations and each resource
required a different routing policy and configuration? This can become a real pain
point when you look at it from a management point of view and that is precisely the
reason why AWS recently launched an interactive visual tool called traffic flow.
Traffic flow basically provides you with an easy to use interface using which you can
create and manage complex traffic policies, all within a fraction of the time. Using the
tool is a fairly simple and straightforward process. You start off by creating one or
more traffic policies, each containing multiple routing and configuration options in the
form of policy records. You can even create multiple versions of the same traffic policy
and use different versions to roll out or roll back configuration changes as you see fit.
[ 279 ]
Extended AWS Services for Your Applications
To create your first traffic policy, select the Traffic policies option from the Route53
Management dashboard. Next, select the Create traffic policy option to get started.
Provide a suitable Policy name and Version description as required. Click on Next
to continue. This will bring up the Create traffic policy page where you can use the
Start Point and the Connect to options to create child rules and endpoints. To delete
any child rule, simply select close (marked by x) in the upper-right corner of each
rule box.
The Start Point actually is where you choose the DNS type that you want Route 53 to
assign to all of the resource record sets. Use the following DNS types if you wish to
route traffic to the following AWS resources:
•
ELB: You can provide either an A record (IP Address) or an AAAA Record
(IPv6 Address) here.
•
Amazon S3 bucket: Provide an A record of your S3 bucket. Note that this is
only going to work if your bucket is configured as a static website container.
•
CloudFront distribution: Provide an A record for your CloudFront
distribution here.
Once you have defined the Start Point value, you can use the Connect to option to
select an applicable rule or endpoint based on the design for your configuration.
The rules can be anything from Weighted rules to Failover, Geolocation, and even
Latency-based rules. Once done, click on the Create traffic policy option to complete
the traffic policy's creation:
[ 280 ]
Chapter 10
Do note that once your traffic policy is created, each edit of it will end up creating a
new version of it. The previous versions of the policy are retained unless you choose
to delete them explicitly. You can even create a new Traffic policy by importing a
JSON-based document that describes all of the endpoints and rules that you want
to include in the policy. To know more about traffic policy document format and
how you can use them for your own scenarios, go to http://docs.aws.amazon.
com/Route53/latest/APIReference/api-policies-traffic-policy-documentformat.html.
Configuring health checks
Route53 also provides a mechanism using which you can effectively monitor the
health and performance of your web application as well as other resources using
the health check facility provided. Health checks can be configured to periodically
monitor the health of your application in a very similar way that ELB does. All
you need to do is provide your application's URL or endpoint and configure the
notification alarm. That's it! The rest is completely taken care of by Route53.
To create your very own health check, select the Health checks option from the
Route53 dashboard. Next, click on the Create health check option to get started.
This will bring up the Configure health check page. The first thing you need to do is
provide a suitable name for your health check using the Name field. Next, select for
what you wish to configure this health check. There are two options available: you can
either monitor an Endpoint, such as an instance or an ELB or even your application's
endpoint, or you can select to monitor the Status of other health checks as well:
For this example, I opted to go for Endpoint monitoring. Next, configure the details
to monitor your endpoint in the Monitor an endpoint section. Here, too, you have
the flexibility to select between having an IP address or a Domain name as an
endpoint.
You cannot change the Specify endpoint by option once the
health check is created.
[ 281 ]
Extended AWS Services for Your Applications
From the Protocol drop-down list, select the appropriate protocol you want Route53
to use to check the health of your endpoint. Currently, the valid values provided are
HTTP, HTTPS and TCP. For this particular scenario, I opted for the HTTP protocol
as shown. In this case, Route 53 will try to establish a connection with the application
using the HTTP protocol. If the connection is successful, Route 53 will submit an
HTTP request and wait for an HTTP response in the form of a status code of 200 or
greater but less than 400.
In the Domain name field, provide the domain name of the endpoint that you wish
Route53 to monitor and also provide the Port on which Route53 will communicate.
By default, port 80 is selected for your HTTP protocol and so port 443 is selected,
in case you have selected HTTPS as the protocol. The final field left now is the Path
field, which specifies the path that you wish Route53 to request when performing
health checks. Do note that this field is only valid for HTTP and HTTPS protocols.
If no value is provided here, then Route53 will automatically start requesting the
Domain name itself:
Once done, you can optionally edit the Request Interval and Failure threshold levels
from the Advanced configuration section, as shown in the following screenshot. Do
remember that changing the request interval from Standard (30 seconds) to Fast
(10 seconds) results in an additional payment. To know more about how Route53 is
priced, check http://aws.amazon.com/route53/pricing/#Health_Checks:
[ 282 ]
Chapter 10
With these settings configured, you can now proceed to the final step of the
configuration where you can create or reuse an SNS notification to notify you in case
a health check fails. Click on Next to continue with the health check creation process.
You can specify health checks to provide you with SNS notifications using the final
configuration page as shown here. Select Yes to create an alarm and choose either
Existing SNS topic or New SNS topic to send notifications to. Once you are happy
with your settings, click on Create health check to complete the process:
Health checks also help you to design and configure DNS failover scenarios. For
example, if your web application is running ten EC2 instances in the backend,
five present in one region and the other five in another, then you can configure
Route53 to check the health of those instances and respond only to the servers that
are healthy. Using this mechanism, you can configure active-active or even activepassive failover scenarios and maintain the uptime of your applications. For a
complete overview and guidance on how you can leverage DNS failover for your
applications, refer to Route53's developer's guide: http://docs.aws.amazon.com/
Route53/latest/DeveloperGuide/dns-failover-configuring.html.
[ 283 ]
Extended AWS Services for Your Applications
Content delivery using Amazon
CloudFront
Moving on, Route53 is yet another awesome service provided by AWS that is
specially designed for distributing and delivering content across the globe and it
is called Amazon CloudFront. For some reason, CloudFront is not one of the most
commonly used services in the AWS service family; nevertheless, it is still a good
alternative to S3 when it comes to distributing content geographically. How does it
all work? Well, it's quite simple actually! To begin with, the first thing that you need
to do is configure an Origin Server. An Origin Server is nothing more than a place
from where CloudFront retrieves the files or content for distribution. Origin Servers
can be anything from an S3 bucket to even an EC2 instance running in a VPC. Once
an Origin is defined, the next step involves the upload of objects to your Origin
Server. Objects can be anything from images, media files, to even web pages! Yes,
you heard it right! Web pages as well! Anything and everything that can be served
over the HTTP protocol or a supported version of the Real Time Messaging protocol
(RTMP). Refer to the following diagram as an example of Origin Server:
The next step is perhaps the most important one and involves the creation of a
CloudFront distribution. The distribution describes which Origin Server to use when
a user initiates a request to an image or a media object from your application or
website. This step will provide your objects with a new CloudFront URL, which you
need to substitute in your application. For example, a standard image URL may look
something like this once it is referenced by CloudFront: http://112233.cloudfront.
net/myimage.jpg. CloudFront then sends this distribution configuration to all of its
specialized edge locations that are spread out across the world.
[ 284 ]
Chapter 10
As of date, Amazon CloudFront has forty-plus edge locations
spread across five continents, including North and South
America, Europe, Asia, and Australia.
These edge locations are nothing more than small data centers where CloudFront
caches copies of your objects and keeps them ready for distribution. When a user
accesses or requests the object from your application or website, the DNS will route
it to the nearest edge location. CloudFront will then check its cache for the requested
file in the edge location. If the file is present in the cache, then it is returned to the
user. If not, then CloudFront will request the file from the Origin Server and cache
it in its edge location. This delivery method comes in really handy when you have
a lot of data that gets requested or accessed by users on high frequency basis. It
also improves the overall performance of your application and website as well as
increasing reliability and availability.
Getting started with distributions
Now that the basic concepts are out of the way, let's look at some simple steps using
which you can get your own CloudFront distribution up and running in a matter of
minutes! First up, access the CloudFront option from the AWS Management Console,
as shown here:
This will bring up the CloudFront Management dashboard where you will be able
to create your distributions as well as perform a wide variety of monitoring and
analysis tasks, such as view Cache Statistics, Popular Objects based on frequency
of requests, Usage, alarms, and so on.
[ 285 ]
Extended AWS Services for Your Applications
To create your first distribution, select the Create Distribution button. This
will bring up a page where you need to select the distribution's delivery
method. Currently, CloudFront supports two modes of delivering content;
they are as follows:
•
Web: Create a web distribution if you wish to distribute static as well as
dynamic content in the form of HTML pages, or even CSS, PHP pages, and
static images. Do remember that web distributions serve the following content
over the HTTP or HTTPS protocols only. You can use either an S3 bucket or
even an EC2 instance such as a web server for your web distribution.
•
RTMP: The RTMP distribution is only meant for live streaming data and
media files such as videos. This distribution only supports an S3 bucket as
the origin server. The following variants of the RTMP protocol are supported
by CloudFront:
°°
RTMP: Adobe's Real-Time Message Protocol.
°°
RTMPT: Adobe streaming tunnel over HTTP.
°°
RTMPE: Adobe encrypted.
°°
RTMPTE: Adobe encrypted tunnel over HTTP.
For this little demonstration, I'll be using one of the S3 buckets that we created in the
previous chapter (Chapter 9, Working with Simple Storage Service). The bucket contains
a simple video that I wish to distribute using CloudFront's edge locations, so the
obvious choice for the distribution selection in this case will be RTMP. In case you do
not have a bucket by now, follow the simple steps outlined in Chapter 9, Working with
Simple Storage Service, and create a bucket, upload a video to it, and make sure that
the bucket has public permissions provided to it.
Next, fill out the following details to create your very own RTMP distribution:
•
Origin Domain Name: Provide the DNS name of the S3 bucket from which
you want CloudFront to get objects for this origin. In my case, the value
provided here is master-doggy-12-01-2015.s3.amazonaws.com.
•
Restrict Bucket Access: CloudFront provides an added layer of security
using which you can restrict end users from accessing objects using only
CloudFront URLs and not using Amazon S3 URLs. This particular feature
is called as Origin Access Identity (OAI). For now, let's go ahead and use
this feature to safeguard our bucket. Select the Yes option, as shown in the
following image.
[ 286 ]
Chapter 10
•
Origin Access Identity: Once you have opted to restrict bucket access, the
next step involves the creation of an OAI. Select the Create a New Identity
option as shown and provide a meaningful name for the new identity in the
Comment section as well:
•
Grant Read Permissions on Bucket: Select the Yes, Update Bucket Policy
option to allow CloudFront to automatically grant the OAI the permission to
read objects stored in your Amazon S3 bucket. You can choose to update the
permissions manually as well.
•
Restrict Viewer Access: Select the Yes option if you want to use a
CloudFront-signed URL or signed cookies as a medium to provide access
to objects in your Amazon S3 bucket. This is yet another advanced security
feature using which you can restrict user access to your buckets. For now, I
have selected No as the option here.
•
Price Class: CloudFront provides three basic pricing classes that can basically
help you pay lower prices based on the content you deliver out of CloudFront.
The Price Classes field come in three variants: All, which includes all the edge
locations present in AWS; Price Class 200, which includes US, Europe, a bit
of Southeast Asia, and India; and finally Price Class 300, which only includes
edge locations present in US and Europe. Depending on your application's
reach, you can select the Price Class as per your requirements. For a complete
overview of how CloudFront charges you as well as its Price Class, check
http://aws.amazon.com/cloudfront/pricing/.
[ 287 ]
Extended AWS Services for Your Applications
•
Alternate Domain Names (CNAMES): This is an optional field you can use
if you want to replace the CloudFront URL's domain name with something a
bit more customized and meaningful, such as your own domain name.
•
Logging: You can enable logging at any time to log information about each
request made to an object. Simply select the On option, as shown in the
following screenshot, and fill out the Bucket for Logs option as well as
Log Prefix values. You can optionally provide a Comment as
well if required:
•
Distribution State: Before you complete the distribution's creation, you
need to specify whether you want the distribution to be enabled or disabled
after its creation. By selecting the Enabled option, your users can access the
CloudFront distribution immediately after its creation, whereas Disabled
means that even though the distribution is ready for use, the end users will
not be able to use it.
Once your configurations are completed, select the Create Distribution option. The
distribution will take a couple of minutes to change from pending state to enabled.
Once it's ready, you can use the CloudFront Domain Name to retrieve your objects
from the Origin Server:
[ 288 ]
Chapter 10
You can edit your distribution's settings anytime by simply selecting the distribution
entry and clicking on Distribution Settings as shown. You can also Disable or
Delete the state of your distribution using this dashboard. Follow the same steps and
you can also configure a web distribution for your application, so give that a try as
well.
CloudFront recommendations and
best practices
Here are some key takeaways and best practices to keep in mind when working
with CloudFront:
•
Cache at every layer of your application, wherever possible.
•
Use a combination of Amazon S3 and CloudFront to distribute static data.
Remember that data transfer between S3 and CloudFront is free!
•
Control access of data on CloudFront and S3. Make use of OAIs to ensure
that there are no unwanted content leaks.
•
Don't forward any headers, cookies, or query strings. Use Signed cookies
instead of signed URLs.
•
Use Route53 to check the health of your origin servers. In this way, you
can configure Route53 to divert all traffic from the failed Origin to the
healthy one.
•
Use Price Classes to optimize content delivery costs.
•
Make use of the alarms and notification services using Amazon CloudWatch.
What's new in AWS?
With the basic services now covered, here's a quick look at some of the newer
AWS services and how you can potentially leverage them to build and host your
applications and infrastructure. First up on the list is Elastic Container Service (ECS).
Elastic Container Service
Before I talk about Elastic Container Service, it is essential to understand what a
container is all about and why is it getting so much of importance lately.
[ 289 ]
Extended AWS Services for Your Applications
A container is a logical entity that consists of one entire runtime environment. This
environment can include an application, its dependencies, all of its libraries, and
configuration files needed to run it, all packed into one small package. But wait a
minute! Doesn't this all seem a bit familiar? Well to be honest, containers are nothing
like virtualization, in fact I see them replacing virtualization very soon. If you see a
virtual machine today, it basically comprises an entire OS plus the application hosted
on top of it. You can have one or more such VMs running on top of a virtualization
layer in the form of a hypervisor, which again has its own set of memory and CPU
requirements. In contrast to VMs, a server running three containers runs on top
of a single OS, and each container shares the OS kernel with the other containers.
This means that the containers are much leaner and lightweight and use far fewer
resources than conventional virtual machines, as depicted in the image here:
Containers are a potent solution to the problem of porting one application from one
environment to the other. These environments can be anything from a simple laptop
to virtualized environments to public clouds. That's where ECS comes into picture.
ECS is a highly scalable container management service using which you can create,
run, manage, and scale Docker-based containers. Here are a few key benefits and
features of using ECS:
•
Automated Clustering: One of the key advantages of ECS is that it can
automatically manage clusters of containers. ECS can maintain the state of
your clusters as well as help you to scale past thousands of containers, all
with relative ease.
[ 290 ]
Chapter 10
•
Container Scheduling: ECS also provides you with an in-built scheduling
functionality using which you can maintain the availability as well as the
scalability of your containers across your clusters.
•
Portability and extensibility: Containers that run on your ECS environment
can be made to run on any other Docker-based environment without any
changes made at all. This feature can come in really handy when you don't
want to get tied down with a particular cloud provider and wish to have the
flexibility to move your workloads anytime anywhere you want.
•
AWS integration: You can leverage ECS containers to work with other AWS
services as well, such as CloudWatch, VPC, S3, Elastic File System (EFS),
ELBs, and so on.
ECS was made generally available for use in the mid of 2015, and ever since then,
AWS has continued to make further improvements and enhancements to it. I would
really recommend trying out ECS and containers just because they are so cool
to work with! There are plenty of starter guides available out there, so go ahead
and try to deploy a simple web application on containers using ECS. Here is the
Getting Started with ECS guide provided by AWS: http://docs.aws.amazon.com/
AmazonECS/latest/developerguide/ECS_GetStarted.html.
Elastic File System
Elastic File System or EFS is also newly launched, still in preview service provided
by AWS that provides scalable file storage services for your EC2 instances. This of
EFS as a highly scalable and available NFS server that multiple EC2 instances can use
at the same time as a common data source for your applications, a central repository,
and so on. The best part of working with EFS is that it is designed to scale up and
down automatically, so the more data that you put in, the larger it grows, and vice
versa. It is also based on the pay-per-use model, which means that you only have to
pay for the storage used by your filesystem and not a penny more!
[ 291 ]
Extended AWS Services for Your Applications
How does it work? Well, it's just like any other NFS server that you would create in
an on-premises environment. You can start off by simply creating an EFS, mounting
it to your EC2 instances and using it just as any NFS mount point would. You can
even use EFS along with your VPC environments as well, but with a minor change.
In the case of VPC, you end up creating one or more special mount points called
as mount targets, and you mount your filesystem to the EC2 instances using these
mount targets. A pictorial representation of mount targets is shown here:
Courtesy: Amazon Web Services.
[ 292 ]
Chapter 10
Database migration made easy with Database
Migration Service
Yet another newly launched service by AWS, Database Migration Service provides
customers with an easy to use and secure way to migrate their on-premises databases
to AWS. The databases can be anything from propriety ones, such as Microsoft SQL
and Oracle, to open source ones such as MySQL as well. Once a database migration is
initiated using the Database Migration Service, AWS manages all the complexities of the
migration process. It also makes sure that any data changes made in the source database
are automatically replicated to the target database during the migration process.
The Database Migration Service also comes equipped with a Schema Conversion
Tool that makes heterogeneous database migrations a real ease. This is made possible
by converting the entire schema of the source database to a format compatible with
that of the target database. If any schema code is not converted during the process
automatically, it is clearly marked by the Schema Conversion Tool so that it can be
converted manually at a later time.
To know more about the Database Migration Service and how you can sign up for
the preview, go to https://aws.amazon.com/dms/.
Go serverless with AWS Lambda
Imagine if you had the power of running and scaling your code dynamically based
on certain events getting triggered! That's precisely what AWS Lambda is all about.
AWS Lambda is basically a compute service very similar to EC2 or Elastic Beanstalk
where you upload your code to Lambda and the service runs the code on your behalf
using the underlying AWS infrastructure. Once your code is uploaded, you need to
create some custom functions called as Lambda functions. These functions take care
of provisioning and managing the underlying instances that you use to run the code.
But how is this so different from your EC2 service? Well in the case of Lambda, all
the heavy lifting and complex tasks such as server management and provisioning, OS
patching, code monitoring, logging, and so on, are managed by Lambda itself. All you
need to do is upload your code and voila! The rest is all taken care of by Lambda itself.
As of date, AWS Lambda supports Node.js, Java, and
Python as the languages for your application's code.
[ 293 ]
Extended AWS Services for Your Applications
So when and where is Lambda useful? Well if you have any real-time log processing
or analyzing a stream of data for pattern analysis, social media analysis, or even
if you want to build scalable backend services for your mobile or web applications,
then Lambda is the right choice for you. To know more about Lambda and how
it works, check http://docs.aws.amazon.com/lambda/latest/dg/lambdaintroduction.html.
Resources, recommendations, and best
practices
There are a ton of resources present on the web and on AWS's website itself where
you can find good content, guides, how-to tutorials, and much more:
•
For anyone just starting off with AWS, I would really recommend reading
the Getting Started with AWS guides. These are some well written and to
the point guides covering topics such as hosting static websites, deploying
web apps, analyzing Big Data, and so on. Go to https://aws.amazon.com/
documentation/gettingstarted/ to know more.
•
Next, the holy grail of all AWS services and a must to read if you are
planning to work with AWS—the AWS documentations page. This page is a
one stop shop for all your AWS service user guides, CLI and API references
as well. Refer to https://aws.amazon.com/documentation/ for more
information.
•
Although not mandatory, the AWS case studies page is yet another
important place where you can read about how and what customers are
using AWS for. You can filter case studies based on their use cases, all
provided at https://aws.amazon.com/solutions/case-studies/.
•
Make sure you also have a look at the pricing for each of the AWS services
that you use by following this link: https://aws.amazon.com/pricing/
services/.
•
Here are some best practice guides provided by AWS as well, which are a
must read if you are planning to use AWS as a production environment:
°°
General AWS Cloud Best Practices to follow: http://media.
°°
How to build fault tolerant application on AWS: http://media.
amazonwebservices.com/AWS_Cloud_Best_Practices.pdf.
amazonwebservices.com/AWS_Building_Fault_Tolerant_
Applications.pdf.
[ 294 ]
Chapter 10
°°
A few design considerations and best practices to keep in
mind when designing DR solutions on AWS: http://media.
amazonwebservices.com/AWS_Disaster_Recovery.pdf.
°°
AWS Security Best Practices guide: http://media.
amazonwebservices.com/AWS_Security_Best_Practices.pdf.
Summary
Well it has really been a wonderful journey writing this book! You started off with
learning the basics of Cloud Computing and slowly, but gradually, covered so
much. From compute (EC2) to networks (VPC) to storage (S3), identity and access
management (IAM), databases (RDS), DNS (Route53), and content delivery services
(CloudFront).
Although this book may seem a lot to read and grasp, trust me, this is all just a drop
in the ocean. AWS is a rapidly expanding and highly innovative public cloud that,
if used correctly, can bring your business and organization a lot of benefits such as
scalability, flexibility, and cost savings. The principle, however, remains the same—
plan out your way before you start, make sure you have designed for failure, and
continuously monitor and automate your infrastructure. Remember these and you
should be just fine!
[ 295 ]
Index
Symbols
32-bit AWS CLI installer
URL 18
64-bit AWS CLI installer
URL 18
A
Access Control Lists (ACLs) 252, 257
administration services
about 7
identity and access management 11
monitoring 11
administrators 33
alarms 153, 154
Amazon Aurora
reference 217
Amazon CloudFront
about 10
distributions 285
URL 272, 289
used, for content delivery 284, 285
Amazon CloudWatch. See CloudWatch
Amazon DynamoDB 9
Amazon Elastic File System 9
Amazon Elastic Load Balancer. See Elastic
Load Balancer
Amazon Elastic MapReduce (EMR) 10
Amazon Glacier 9
Amazon Kinesis
URL 180
Amazon Linux AMI
about 56
URL 56
Amazon Machine Images (AMIs) 53, 54
Amazon Redshift 9, 10
Amazon Relational Database
Service (Amazon RDS)
about 9
best practices 244
instance types 217
MySQL, working with 223, 224
overview 215-217
recommendations 244
steps, planning 243, 244
working with 222
Amazon Resource Name (ARN) 38
Amazon Route53
about 10, 273
failover routing 278
geolocation routing 278
health checks, configuring 281-283
hosted zones, creating 275-279
latency-based routing 278
simple routing 277
traffic flow 279-281
URL 282, 283
weighted routing 277
working with 274, 275
Amazon Simple Email Service (SES) 10
Amazon Simple Notification
Service (SNS) 10
Amazon Simple Storage Service (S3)
about 9, 247-249
best practices and
recommendations 271, 272
bucket permissions, managing 257-260
bucket policies 258
buckets accessing, S3CMD used 254-257
buckets, creating 250, 251
buckets, used for hosting websites 261-263
[ 297 ]
bucket, versioning 265-267
Cross-Region Replication 269, 270
events 263, 264
first object, uploading to bucket 252, 253
getting started 250
Glacier, storage class 249
lifecycle management 265-268
next steps, planning 270, 271
notification 263
object permissions, managing 257-260
objects accessing, S3CMD used 254-257
Reduced Redundancy Storage (RRS),
storage class 249
resource-based policies 257
Standard_IA, storage class 249
Standard, storage class 249
storage class 249
uploaded Objects, viewing 254
Amazon Virtual Private Cloud (VPC).See
Virtual Private Cloud (VPC)
Amazon Web Services (AWS)
about 1, 4, 5
architecture 5
availability zones 5, 7
components 5
platform overview 7
regions 5, 7
signup process 11-15
support plans 15
URL 7, 12
URL, for free tier usage 12
AMI
URL 112
application services
about 7
content distribution and delivery 10
distributed computing and analytics 10
messaging 10
workflow 10
automated backup, DB instances 236
Auto Scaling
about 183-185
AMI, selecting 198
CLI commands, URL 212
components 186
deleting 210, 211
details, configuring 198
getting started 196, 197
group 186
Instance type, selecting 198
Launch Configuration, creating 197
launch configurations 186
next steps, planning 212, 213
resuming 210-212
Review 199
scaling plans 187
Security Group, configuring 199
storage, adding 199
suspending 210, 211
testing 208, 209
URL 213
verifying 208, 209
Auto Scaling Group
creating 200
details, configuring 201, 202
notifications, configuring 206
Review 207
scaling policies, configuring 203-205
tags, configuring 207
Availability Zones (AZs) 6
AWS
DR solutions, URL 295
fault tolerant application, URL 294
Security Best Practices, URL 295
security layers 24
shared responsibility model 24, 25
URL 294
AWS CLI
about 18-20
used, for launching instances 77
used, for managing access 41-46
used, for managing EBS volumes 105-107
used, for managing security 41-46
AWS Cloud Best Practices
URL 294
AWSCloudTrail
about 47
URL 47
AWSConfig
about 47
URL 47
AWSIAM
features 46
[ 298 ]
AWS Key Management Service
about 47
URL 47
AWS Lambda
about 293
URL 294
awslogs.conf file
URL 176
AWS Management Console
about 15-17
URL 16, 27
AWS Marketplace
URL 55
AWS Multi-Factor Authentication
(AWS MFA) 25
AWS namespaces
URL 151
AWS naming conventions
URL 106
AWSrisk
URL 24
AWSS3 documentation page
URL 261
AWS SQS
URL 213
B
Bastion Host 144
best practices 48
BitTorrent protocol
URL 252
C
CIDRs
URL 117
cloud computing
about 2
benefits 3
features 3
use cases 3, 4
CloudFront. See Amazon CloudFront
clouds
security 23
CloudWatch
about 11, 149-155
access roles, creating 166-168
alarms 153, 154
best practices 180, 181
concept 150
custom metrics, viewing 171, 172
dimensions 151
limits and costs 154, 155
metrics 150
monitoring scripts, installing 168-170
namespaces 151
recommendations 180, 181
scripts, used, for monitoring memory and
disk utilization 166
time stamps and periods 151
units and statistics 152
used, for monitoring accounts estimate
charges 155-159
used, for monitoring instance's CPU
Utilization 159-165
CloudWatch logs
about 173-176
alarms, creating 177, 179
events 172
groups 173
log agents 173
metric filters 173
metric filters, creating 177
monitoring 172
retention policies 173
stream 172
used, for monitoring logs 172
viewing 177
Cross-Region Replication
enabling, on bucket 269, 270
Custom Metrics 166
D
Database migration
Database Migration Service with 293
DB Snapshots 236
DHCP Option 124
dimensions 151
DNS service provider
URL 274
DNS types
Amazon S3 bucket 280
CloudFront distribution 280
[ 299 ]
ELB 280
Domain name 125
Domain Name Servers (DNS) 124
E
EBS-backed AMI
about 55
and Instance store backed AMI,
differences 56
EBS-optimized instances 112
EBS snapshots
used, for backing up volumes 107-109
EBS volumes
about 98
attaching 102
attaching, from instance 103-105
benefits 98
creating 100, 101
detaching 104
getting started 99, 100
managing, AWS CLI used 105-107
recommendations and best practices 113
URL 102
EBS volumes, types
General purpose volumes (SSD) 99
Magnetic volumes 99
Provisioned IOPS volumes (SSD) 99
EC2-Classic 91
EC2 Container Service (EC2) 8
EC2 dashboard
used, for editing security dashboards 85-89
EC2 networking
about 89-91
instances IP addresses, determining 92, 93
e-commerce website
hosting, on cloud 20-22
Elastic Block Storage (EBS) 9
Elastic Block Store (EBS) 55
Elastic Compute Cloud (EC2)
about 8, 51, 52
URL 132
use cases 52
Elastic Compute Cloud (EC2), use cases
Backup and disaster recovery 52
Dev/Test environment 52
High Performance Computing (HPC) 52
Hosting environment 52
Marketing and advertisements 52
Elastic Container Service (ECS)
about 289, 291
benefits 290
URL 291
Elastic File System (EFS) 9, 291, 292
Elastic IP Address (EIP)
about 93, 94
allocating 95, 96
creating 95
disassociating 97
releasing 97
Elasticity 184
Elastic Load Balancer (ELB)
about 9, 154, 183, 187, 188
creating 189
Dashboard, URL 196
defining 190-192
EC2 Instances, adding 195
Health Check, configuring 193, 194
Review and Create 195, 196
Security Groups, assigning 192
Security Settings, configuring 192
Tags, adding 195
URL 212, 214
endpoint policies
URL 121
F
filter patterns
URL 178
Flow Logs
URL 146
foundation services
about 7
compute 8
databases 9
networking 9
storage 9
G
G2 58
General purpose volumes (SSD) 99
[ 300 ]
H
hosted zones
creating 275, 276
I
IAM Console
groups, creating 30-35
permissions 35-38
policies 35-38
users, creating 30-35
using 27-29
IAM roles
URL 47, 167
Identity and Access Management (IAM)
about 23, 25
business use case scenario 27
features 26
Identity and Access Management
(IAM), features
access mechanisms 26
global reach 26
Identity federation 26
integration, with other AWS products 26
multi-factor authentication 26
shared access to single account 26
images 53-56
individual processes
URL 211
instance metadata 92
instances
about 53, 57
cleaning up 80, 81
compute optimized 57
configuring 75-77
connecting to 69-75
general purpose 57
GPU instances 58
launching, AWS CLI used 77
memory optimized 57
Putty, using 71
storage optimized 57
types, URL 58
web browser, using 69
instances launching, AWS CLI used
about 79, 80
key pair, creating 77
rules, adding to security group 79
security group, creating 78
instances launching, in Virtual Private
Cloud (VPC)
about 142
database servers, creating 144
web servers, creating 142-144
instances, pricing options
about 58
on-demand instances 58, 59
reserved instances 59
spot instances 60
instance store-backed AMI 55
instances, working with
about 60-62
AMI, selecting 62
instance details, configuring 64, 65
instance launch, reviewing 68, 69
instance type, selecting 63
security groups, configuring 67
storage, adding 65, 66
tag instances 66
Internet Gateways
about 122
listing 137, 138
K
Key Management Service (KMS) 98
URL 271
L
Lambda functions 293
lifecycle, DB instance
available 231
backing-up 231
creating 231
modifying 231
lifecycle management
URL 268
link-local address 93
Lookbusy
URL 165
M
main route table 120
[ 301 ]
metric filter examples
URL 179
metrics 150
MindTerm 69
Multi-AZ deployment
about 220
Multi-factor authentication (MFA) 271
MySQL DB instance
advanced settings, configuring 227-231
creating 224
Database engine, selecting 225
DB details, specifying 225-227
production database 225
MySQL, on Amazon RDS
about 223, 224
database, testing 233
DB instance, cleaning up 242, 243
DB Instance, connecting remotely 232
DB instance, logging 241, 242
DB instance, monitoring 241, 242
DB instances, backing up 236, 237
DB instances, modifying 234, 235
MySQL DB instance, creating 224
Read Replicas, creating 237-240
N
Name Server (NS) record 274
namespaces 151
NAT instances 123, 124
NetBIOS name server 125
NetBIOS node type 125
Network Access Control Lists (ACLs) 116
Network Attached Storage (NAS) drive 55
NTP servers 125
O
Origin Access Identity (OAI) 286
P
permissions, IAM Console
about 35, 37
resource-based permissions class 36
user-based permissions class 36
policies, IAM Console
about 35-37
Action 38
assigning 39, 41
creating 39, 41
Effect 38
Resource 38
statement 38
URL 38, 258
version 38
presigned URLs
URL 271
Price Class
URL 287
private hosted zone 274
private subnet 117
Provisioned IOPS volumes (SSD) 99
public datasets
URL 113
public DNS
URL 208
public hosted zone 274
public subnet 117
PuttyGen
URL 71
R
RDS instance types
about 217
burst capable 218
memory optimized 218
micro instances 217
standard instance 218
Read Replicas 221
Real Time Messaging protocol (RTMP)
protocol 284
record set types
AAAA record 276
about 276
Canonical Name (CNAME) 276
MX records 277
record 276
RedHat Enterprise Linux (RHEL) 56
RedShift 244
roles 46
Route 53. See Amazon Route53
route tables
working with 136, 137
[ 302 ]
S
S3 bucket
about 36
URL 180
S3CMD
about 272
URL 255
security 23
security groups
about 85-89
editing, EC2 dashboard used 85, 86
security layers, AWS
physical data center security 24
regulatory compliances 24
virtualization and OS security 24
services, Amazon Web Services (AWS)
administration services 7
application services 7
AWSCloudTrail 47
AWSConfig 47
foundation services 7
Key Management Service 47
Simple Notification Service (SNS). See
Amazon Simple Notification
Service (SNS)
Simple Storage Service (S3). See Amazon
Simple Storage Service (S3)
snapshots 108-111
spot instances
URL 81
Start Of Authority (SOA) record 274
storage types, RDS instances
general purpose (SSD) 219
magnetic (standard) 218
provisioned IOPs 219
Stress
URL 165
subnets
private subnet 117
public subnet 117, 118
T
Technical Account Manager (TAM) 15
time stamps and periods 151
Time To Live (TTL) 276
traffic flow
about 279-281
U
units and statistics 152, 153
user-based permissions
inline policies 36
managed policies 36
V
vCenter
URL 81
virtualization types
URL 111
Virtual Private Cloud (VPC)
about 8, 64, 115-117
best practices 146, 147
concepts 117
costs 125
deleting 144
deployment scenarios 126, 127
DNS and DHCP Option Sets 124, 125
endpoints 120, 121
instances, launching 142
Internet Gateways 122
limits 125
limits, URL 125
NAT instances 123, 124
network ACLs 119
peering, 145
recommendations 146, 147
routing tables 120
Security Groups 119
subnets 117
URL 116, 146
wizard 127-133
working with 126
Virtual Private Cloud (VPC) wizard
about 127-133
Internet Gateways, listing 137, 138
Network ACLs 138-142
route tables, working with 136, 137
Security Groups 138-142
subnets, listing 135, 136
viewing 133, 134
Virtual Private Gateway 116
[ 303 ]
Thank you for buying
AWS Administration – The Definitive Guide
About Packt Publishing
Packt, pronounced 'packed', published its first book, Mastering phpMyAdmin for Effective MySQL
Management, in April 2004, and subsequently continued to specialize in publishing highly
focused books on specific technologies and solutions.
Our books and publications share the experiences of your fellow IT professionals in adapting
and customizing today's systems, applications, and frameworks. Our solution-based books give
you the knowledge and power to customize the software and technologies you're using to get
the job done. Packt books are more specific and less general than the IT books you have seen in
the past. Our unique business model allows us to bring you more focused information, giving
you more of what you need to know, and less of what you don't.
Packt is a modern yet unique publishing company that focuses on producing quality,
cutting-edge books for communities of developers, administrators, and newbies alike.
For more information, please visit our website at www.packtpub.com.
About Packt Enterprise
In 2010, Packt launched two new brands, Packt Enterprise and Packt Open Source, in order
to continue its focus on specialization. This book is part of the Packt Enterprise brand, home
to books published on enterprise software – software created by major vendors, including
(but not limited to) IBM, Microsoft, and Oracle, often for use in other corporations. Its titles
will offer information relevant to a range of users of this software, including administrators,
developers, architects, and end users.
Writing for Packt
We welcome all inquiries from people who are interested in authoring. Book proposals should
be sent to author@packtpub.com. If your book idea is still at an early stage and you would
like to discuss it first before writing a formal book proposal, then please contact us; one of our
commissioning editors will get in touch with you.
We're not just looking for published authors; if you have strong technical skills but no writing
experience, our experienced editors can help you develop a writing career, or simply get some
additional reward for your expertise.
Learning AWS
ISBN: 978-1-78439-463-9
Paperback: 236 pages
Design, build, and deploy responsive applications
using AWS cloud components
1.
Build scalable and highly available real-time
applications.
2.
Make cost-effective architectural decisions by
implementing your product's functional and
non-functional requirements.
3.
Develop your skills with hands-on exercises
using a three-tiered service oriented application
as an example.
Amazon Web Services: Migrating
your .NET Enterprise Application
ISBN: 978-1-84968-194-0
Paperback: 336 pages
Evaluate your Cloud requirements and successfully
migrate your .NET Enterprise Application to the
Amazon Web Services Platform
1.
Get to grips with Amazon Web Services from a
Microsoft Enterprise .NET viewpoint.
2.
Fully understand all of the AWS products
including EC2, EBS, and S3.
3.
Quickly set up your account and manage
application security.
4.
Learn through an easy-to-follow sample
application with step-by-step instructions.
Please check www.PacktPub.com for information on our titles
Implementing Cloud Design
Patterns for AWS
ISBN: 978-1-78217-734-0
Paperback: 228 pages
Create highly efficient design patterns for scalability,
redundancy, and high availability in the AWS Cloud
1.
Create highly robust systems using cloud
infrastructure.
2.
Make web applications resilient against
scheduled and accidental down-time.
3.
Explore and apply Amazon-provided services
in unique ways to solve common problems.
AWS Development Essentials
ISBN: 978-1-78217-361-8
Paperback: 226 pages
Design and build flexible, highly scalable, and
cost-effective applications using Amazon Web Services
1.
Integrate and use AWS services in an
application.
2.
Reduce the development time and billing cost
using the AWS billing and management console.
3.
This is a fast-paced tutorial that will cover
application deployment using various tools
along with best practices for working with
AWS services.
Please check www.PacktPub.com for information on our titles
Source Exif Data:
File Type : PDF File Type Extension : pdf MIME Type : application/pdf PDF Version : 1.6 Linearized : No Tagged PDF : Yes XMP Toolkit : Adobe XMP Core 4.0-c316 44.253921, Sun Oct 01 2006 17:14:39 Instance ID : uuid:b69be51a-3266-40c3-ae6a-02098c358b67 Original Document ID : adobe:docid:indd:4992da54-27df-11de-a18e-f5498a2a904f Document ID : xmp.id:14CA62D4CBD0E511BA519E71B05AB638 Rendition Class : proof:pdf Derived From Instance ID : xmp.iid:13CA62D4CBD0E511BA519E71B05AB638 Derived From Document ID : xmp.did:4CA8ED60EACBE511B117FE932790DDD3 Derived From Original Document ID: adobe:docid:indd:4992da54-27df-11de-a18e-f5498a2a904f Derived From Rendition Class : default History Action : converted History Parameters : from application/x-indesign to application/pdf History Software Agent : Adobe InDesign CS6 (Windows) History Changed : / History When : 2016:02:11 20:01:19+05:30 Create Date : 2016:02:11 20:01:19+05:30 Modify Date : 2016:02:12 17:28:20+05:30 Metadata Date : 2016:02:12 17:28:20+05:30 Creator Tool : Adobe InDesign CS6 (Windows) Format : application/pdf Producer : Adobe PDF Library 10.0.1 Trapped : False Has XFA : No Page Count : 326 Creator : Adobe InDesign CS6 (Windows)EXIF Metadata provided by EXIF.tools