Painless Docker Basic Edition: A Practical Guide To Master And Its Ecosystem Based On Real World Examples Edition Real%

User Manual: Pdf

Open the PDF directly: View PDF .
Page Count: 375

Download
Open PDF In Browser	View PDF

Table of Contents
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.

Introduction 1.1
Preface 1.2
Chapter I - Introduction To Docker & Containers 1.3
Chapter II - Installation & Configuration 1.4
Chapter III - Basic Concepts 1.5
Chapter IV - Advanced Concepts 1.6
Chapter V - Working With Docker Images 1.7
Chapter VI - Working With Docker Containers 1.8
Chapter VII - Working With Docker Machine 1.9
Chapter VIII - Docker Networking 1.10
Chapter IX - Composing Services Using Compose 1.11
Chapter X - Docker Logging 1.12
Chapter XI - Docker Debugging And Troubleshooting 1.13
Chapter XII - Orchestration - Docker Swarm 1.14
Chapter XIII - Orchestration - Kubernetes 1.15
Chapter XIV - Orchestration - Rancher/Cattle 1.16
Chapter XV - Docker API 1.17
Chapter XVI - Docker Security 1.18
Chapter XVII - Docker, Containerd & Standalone Runtimes Architecture 1.19
Final Words 1.20

Painless Docker
About The Author
Aymen is a Cloud & Software Architect, Entrepreneur, Author, CEO of Eralabs, A DevOps & Cloud Consulting
Company and Founder of DevOpsLinks Community.
He has been using Docker since the word Docker was just a buzz. He worked on web development, system
engineering, infrastructure & architecture for companies and startups. He is interested in Docker, Cloud Computing,
the DevOps philosophy, the lean programming and the tools/methodologies.
You can find Aymen on Twitter.
Don't forget to join DevOpsLinks & Shipped newsletters and the community Job Board JobsForDevOps. You can
also follow this course Twitter account for future updates.
Wishing you a pleasant reading.

Disclaimer
Docker and the Docker logo are trademarks or registered trademarks of Docker, Inc. in the United States and/or
other countries.

Preface
Docker is an amazing tool, may be you have tried using or testing it or may be you started using it in some or all of
your production servers but managing and optimizing it can be complex very quickly, if you don't understand some
basic and advanced concepts that I am trying to explain in this book.
The fact that the ecosystem of containers is rapidly changing is also a constraint to stability and a source of
confusion for many operation engineers and developers.
Most of the examples that can be found in some blog posts and tutorials are -in many cases- promoting Docker or
giving tiny examples, managing and orchestrating Docker is more complicated, especially with high-availability
constraints.
This containerization technology is changing the way system engineering, development and release management are
working since years, so it requires all of your attention because it will be one of the pillars of future IT technologies
if it is not actually the case.
At Google, everything runs in a container. According to The Register, two billion containers are launched every
week. Google has been running containers since years, when containerization technologies were not yet
democratized and this is one of the secrets of the performance and ops smoothness of Google search engine and all
of its other services.
Some years ago, I was in doubt about Docker usage, I played with Docker in testing machines and I decided later to
use it in production. I have never regretted my choice, some months ago I created a self-service in my startup for
developers : an internal scalable PaaS - that was awesome ! I gained more than 14x on some production metrics and
I realized my goal of having a service with SLA and Appdex score of 99%.

Appdex (Application Performance Index) is an open standard that defines a standardized method to report,
benchmark, and track application performance.

SLA (Service Level Agreement) is a contract between a service provider (either internal or external) and the
end user that defines the level of service expected from the service provider.

It was not just the usage of Docker, this would be too easy, it was a list of todo things, like moving to micro-services
and service-oriented architectures, changing the application and the infrastructure architecture, continuous
integration ..etc But Docker was one of the most important things on my checklist, because it smoothed the whole
stack's operations and transormation, helped me out in the continuous integration and the automation of routine task
and it was a good platform to create our own internal PaaS.
Some years ago, computers had a central processing unit and a main memory hosted in a main machine, then come
mainframes whose were inspired from the latter technology. Just after that, IT had a new born called virtual
machines. The revolution was in the fact that a computer hardware using a hypervisor, allows a single machine to
act as if it where many machines. Virtual machines were almost run in on-premise servers, but since the emergence

of cloud technologies, VMs have been moved to the cloud, so instead of having to invest heavily in data centers and
physical servers, one can use the same virtual machine in the infrastructure of servers providers and benefit from the
'pay-as-you-go' cloud advantage.
Over the years, requirements change and new problems appears, that's why solutions also tends to change and new
technologies emerge.
Nowadays, with the fast democratization of software development and cloud infrastructures, new problems appears
and containers are being largely adopted since they offer suitable solutions.
A good example of the arising problems is supporting software environment in an identical environment to the
production when developing.Weird things happen when your development and testing environments are not the
same, same thing for the production environments. In this particular case, you should provide and distribute this
environment to your R&D and QA teams.
But running a Node.js application that has 1 MB of dependencies plus the 20MB Node.js runtime in a Ubuntu 14.04
VM will take you up to 1.75 GB. It's better to distribute a small container image than 1G of unused libraries..
Containers contains only the OS libraries and Node.js dependencies, so rather than starting with everything
included, you can start with minimum and then add dependencies so that the same Node.js application will be 22
times smaller! When using optimized containers, you could run more applications per host.
Containers are a problem solver and one of the most sophisticated and adopted containers solutions is Docker.

To Whom Is This Book Addressed ?
To developers, system administrators, QA engineers, operation engineers, architects and anyone faced to work in
one of these environments in collaboration with the other or simply in an environment that requires knowledge in
development, integration and system administration.
The most common idea is that developers think they are here to serve the machines by writing code and
applications, systems administrators think that machines should works for them simply by making them happy
(maintenance, optimization ..etc ).
Moreover, within the same company there is generally some tension between the two teams:
System administrators accuse developers to write code that consumes memory, does not meet system
security standards or not adapted to available machines configuration.
Developers accuse system administrators to be lazy, to lack innovation and to be seriously uncool!
No more mutual accusations, now with the evolution of software development, infrastructure and Agile engineering,
the concept of DevOps was born.
DevOps is more a philosophy and a culture than a job (even if some of the positions I occupied were called
"DevOps"). By admitting this, this job seeks closer collaboration and a combination of different roles involved in
software development such as the role of developer, responsible for operations and responsible of quality assurance.
The software must be produced at a frenetic pace while at the same time the developing in cascade seems to have
reached its limits.
If you are a fan of service-oriented architectures, automation and the collaboration culture
if you are a system engineer, a release manager or an IT administrator working on DevOps, SysOps
or WebOps
If you are a developer seeking to join the new movement
This book is addressed to you. Docker one the most used tools in DevOps environments.
And if you are new to Docker ecosystem and no matter what your Docker level is, through this book, you will firstly
learn the basics of Docker (installation, configuration, Docker CLI ..etc) and then move easily to more complicated
things like using Docker in your development, testing and live environments.
You will also see how to write your own Docker API wrapper and then master Docker ecosystem, form
orchestration, continuous integration to configuration management and much more.
I believe in learning led by practical real-world examples and you ill be guided through all of this book by tested
examples.

How To Properly Enjoy This Book
This book contains technical explanations and shows in each case an example of a command or a configuration to
follow. The only explanation gives you a general idea and the code that follows gives you convenience and help you
to practice what you are reading. Preferably, you should always look both parts for a maximum of understanding.
Like any new tool or programming language you learned, it is normal to find difficulties and confusions in the
beginning, perhaps even after. If you are not used to learn new technologies, you can even have a modest
understanding while being in an advanced stage of this book. Do not worry, everyone has passed at least once by
this kind of situations.
At the beginning you could try to make a diagonal reading while focusing on the basic concepts, then you could try
the first practical manipulation on your server or using your laptop and occasionally come back to this book for
further reading on a about a specific subject or concept.
This book is not an encyclopedia but sets out the most important parts to learn and even to master Docker and its
fast-growing ecosystem. If you find words or concepts that you are not comfortable with, just try to take your time
and do your own on-line research.
Learning can be serial so understanding a topic require the understanding of an other one, do not lose patience : You
will go through chapters with good examples of explained and practical use cases.
Through the examples, try to showcase your acquired understanding, and, no, it will not hurt to go back to previous
chapters if you are unsure or in doubt.
Finally, try to be pragmatic and have an open mind if you encounter a problem. The resolution begins by asking the
right questions.

Conventions Used In This Book
Basically, this is a technical book where you will find commands (Docker commands) and code (YAML, Python
..etc).
Commands and code are written in a different format.
Example :
docker run hello-world

This book uses italic font for technical words such as libraries, modules, languages names. The goal
is to get your attention when you are reading and help you identify them.
You will find two icons, I have tried to be as simple as possible so I have chosen not to use too many
symbols, you will only find:

To highlight useful and important information.
To highlight a warning or a cautionary advice.
Some containers/services/networks identifiers are long to be well formatted, you will find for example some ids in
this format 4..d , e..3 : instead of writing all of the string : 4lrmlrazrlm4213lkrnalknra125009hla94l1419u14N14d , I only use
the first and the last character, which gives 4..d .

How To Contribute And Support This Book ?
This work will be always a work in progress but it does not mean that it is not a complete learning resource - writing
a perfect book is impossible on the contrary of iterative and continuous improvement.
I am an adopter of the lean philosophy so the book will be continuously improved in function of many criteria but
the most important one is your feedback.
I imagine that some readers do not know how "Lean publishing" works. I'll try to explain briefly:
Say, the book is 25% complete, if you pay for it at this stage, you will pay the price of the 25% but get all of the
updates until 100%.
Another point, lean publishing for me is not about money, I refused several interesting offers from known publishers
because I want to be free from restrictions and DRM ..etc
If you have any suggestion or if you encountered a problem, it would be better to use a tracking system for issues
and recommendations about this book, I recommend using this github repository.
You can find me on Twitter or you can use my blog contact page if you would like to get in touch.
This book is not perfect, so you can find typo, punctuation errors or missing words.
Contrariwise every line of the used code, configurations and commands was tested before.
If you enjoyed reading Painless Docker and would like to support it, your testimonials will be more than welcome,
send me an email, if you need a development/testing server to manipulate Docker, I recommend using Digital
Ocean, you can also show your support by using this link to sign up.
If you wan to join more than 1000 developers, SRE engineers, sysadmins and IT experts, you can subscribe to a
DevOpsLinks community and you will be invited to join our newsletter and join out team chat.

Chapter I - Introduction To Docker & Containers
o
o

^__^
(oo)\_______
(__)\
)\/\
||----w |
||
||

What Are Containers
"Containers are the new virtualization." : This is what pushed me to adopt containers, I have been always curious
about the virtualization techniques and types and containers are - for me - a technology that brings together two of
my favorite fields: system design and software engineering.
Container like Docker is the technology that allows you to isolate, build, package, ship and run an application.
A container makes it easy to move an application between development, testing, staging and production
environments.
Containers exist since years now, it is not a new revolutionary technology: The real value it gives is not the
technology itself but getting people to agree on something. In the other hand, it is experiencing a rebirth with easyto-manage containerization tools like Docker.

Containers Types
The popularity of Docker made some people think that it is the only container technology but there are many others.
Let's enumerate most of them.
System administrators will be more knowledgeable about the following technologies but this book is not just for
Linux specialists, operation engineers and system architects, it is also addressed to developers and software
architects.
The following list is ordered from the least to the most recent technology.

Chroot Jail
Historically, the first container was the chroot.
Chroot is a system call for nix OSs that changes the root directory of the current running process and their children.
The process running in a chroot jail will not know about the real filesystem root directory.
A program that is run in such environment cannot access files and commands outside that environmental directory
tree. This modified environment is called a chroot jail.

FreeBSD Jails
The FreeBSD jail mechanism is an implementation of OS-level virtualization.
A FreeBSD-based operating system could be partitioned into several independent jails. While chroot jail restricts
processes to a particular filesystem view, the FreeBSD is an OS-level virtualization: A jail restricting the activities of
a process with respect to the rest of the system. Jailed processes are "sandboxed".

Linux-VServer
Linux-VServer is a virtual private server using OS-level virtualization capabilities that was added to the Linux
Kernel.
Linux-VServer technology has many advantages but its networking is based on isolation, not virtualization which
prevents each virtual server from creating its own internal routing policy.

Solaris Containers
Solaris Containers are an OS-level virtualization technology for x86 and SPARC systems. A Solaris Container is a
combination of system resource controls and the boundary separation provided by zones.
Zones are the equivalent of completely isolated virtual servers within a single OS instance. System administrators
place multiple sets of application services onto one system and place each into isolated Sloaris Container.

OpenVZ
Open Virtuozzo or OpenVZ is also OS-level virtualization technology for Linux. OpenVZ allows system
administrators to run multiple isolated OS instances (containers), virtual private servers or virtual environments.

Process Containers
Engineers at Google (primarily Paul Menage and Rohit Seth) started the work on this feature in 2006 under the
name "Process Containers". It was then called cgroups (Control Groups). We will see more details about cgroups
later in this book.

LXC
Linux Containers or LXC is an OS-level virtualization technology that allows running multiple isolated Linux
systems (containers) on a control host using a single Linux kernel. LXC provides a virtual environment that has its
own process and network space. It relies on cgroups (Process Containers).
The difference between Docker and LXC is detailed later in this book.

Warden
Warden used LXC at its initial stage and was later on replaced with a CloudFoundry implementation. It provides
isolation to any other system than Linux that support isolation.

LMCTFY
Let Me Contain That For You or LMCTFY is the Open Source version of Google’s container stack, which provides
Linux application containers.
Google engineers have been collaborating with Docker over libcontainer and porting the core lmctfy concepts and
abstractions to libcontainer.
The project is not actively being developed, in future the core of lmctfy will be replaced by libcontainer.

Docker
This is what we are going to discover in this book.

RKT
CoreOs started building a container called rkt (pronounce Rocket).
CoreOs is designing rkt following the original premise of containers that Docker introduced but with more focus on:
Composable ecosystem
Security
A different image distribution philosophy
Openness
rkt is Pod-native which means that its basic unit of execution is a pod, linking together resources and user
applications in a self-contained environment.

Introduction To Docker
Docker is a containerization tool with a rich ecosystem that was conceived to help you develop, deploy and run any
application, anywhere.
Unlike a traditional virtual machine, Docker container share the resources of the host machine without needing an
intermediary (a hypervisor) and therefore you don't need to install an operating system. It contains the application
and its dependencies, but works in an isolated and autonomous way.

In other words, instead of a hypervisor with a guest operating system on top, Docker uses its engine and containers
on top.
Most of us used to use virtual machines, so why containers and Docker are taking an important part of today
infrastructures ?
This table explains briefly the difference and the advantages of using Docker over a VM:

Size
Startup Time
Integration
Dependency Hell
Versionning

VM
Small CoreOs = 1.2GB
Measured in minutes
Difficult
Frustration
No

Docker
A Busybox container = 2.5 MB
An optimized Docker container, will run in less that a second
More open to be integrated to other tools
Docker fixes this
Yes

Docker is a process isolation tool that used LXC (an operating-system-level virtualization method for running
multiple isolated Linux systems (containers) on a control host using a single Linux Kernel) until the version 0.9.
The basic difference between LXC and VMs is that with LXC there is only one instance of Linux Kernel running.
For curious readers, LXC was replaced by Docker own libcontainer library written in the Go programming
language.
So, a Docker container isolate your application running in a host OS, the latter can run many other containers. Using
Docker and its ecosystem, you can easily manage a cluster of containers, stop, start and pause multiple applications,
scale them, take snapshots of running containers, link multiple services running Docker, manage containers and
clusters using APIs on top of them, automate tasks, create applications' watchdogs and many other features that are
complicated without containers.
Using this book, you will learn how to use all of these features and more.

What Is The Relation Between The Host OS And Docker
In a simple phrase, the host OS and the container share the same Kernel.
If you are running Ubuntu as a host, container's Kernel is going to use the same Kernel as Ubuntu system, but you
can use CentOs or any other OS image inside your container. That is why the main difference between a virtual
machine and a Docker container is the absence of an intermediary part between the Kernel and the guest, Docker
takes place directly within your host's Kernel.
You are probably saying "If Docker's using the host Kernel, why should I install an OS within my container ?".
You are right, in some cases you can use Docker's scratch image, which is an explicitly empty image, especially for
building images from scratch. This is useful for containers that contain only a single binary and whatever it requires,
such as the "hello-world" container that we are going to use in the next section.
So Docker is a process isolation environment and not an OS isolation environment (like virtual machines), you can,
as said, use a container without an OS. But imagine you want to run an Nginx or an Apache container, you can run
the server's binary, but you will need to access the file system in order to configure nginx.conf, apache.conf,
httpd.conf ..etc or the available/enabled sites configurations.
In this case, if you run a containers without an OS, you will need to map folders from the container to the host like
the /etc directory (since configuration files are under /etc).
You can actually do it but you will lose the change management feature that docker containers offers: So every
change within the container file system will be also mapped to the host file system and even if you map them on
different folders, things could become complex with advanced production/development scenarios and environments.
Therefore, amongst other reasons, Docker containers running an OS are used for portability and change
management.
In the examples explained in this book, we often rely on official images that can be found in the official Docker hub,
we will also create some custom images.

What Does Docker Add To LXC Tools ?
LXC owes its origin to the development of cgroups and namespaces in the Linux kernel. One of the most asked
questions on the net about Docker is the difference between Docker and VMs but also the difference between
Docker and LXC.
This question was asked in Stackoverflow and I am sharing the response of Solomon Hykes (the creator of Docker)
pusblished under CC BY-SA 3.0 license:
Docker is not a replacement for LXC. LXC refers to capabilities of the Linux Kernel (specifically namespaces and
control groups) which allow sandboxing processes from one another, and controlling their resource allocations.
On top of this low-level foundation of Kernel features, Docker offers a high-level tool with several powerful
functionalities:
Portable deployment across machines
Docker defines a format for bundling an application and all its dependencies into a single object which can be
transferred to any docker-enabled machine, and executed there with the guarantee that the execution environment
exposed to the application will be the same. LXC implements process sandboxing, which is an important prerequisite for portable deployment, but that alone is not enough for portable deployment. If you sent me a copy of
your application installed in a custom LXC configuration, it would almost certainly not run on my machine the way
it does on yours, because it is tied to your machine's specific configuration: networking, storage, logging, distro, etc.
Docker defines an abstraction for these machine-specific settings, so that the exact same Docker container can run unchanged - on many different machines, with many different configurations.
Application-centric
Docker is optimized for the deployment of applications, as opposed to machines. This is reflected in its API, user
interface, design philosophy and documentation. By contrast, the LXC helper scripts focus on containers as
lightweight machines - basically servers that boot faster and need less ram. We think there's more to containers than
just that.
Automatic build
Docker includes a tool for developers to automatically assemble a container from their source code, with full control
over application dependencies, build tools, packaging etc. They are free to use make, Maven, Chef, Puppet,
SaltStack, Debian packages, RPMS, source tarballs, or any combination of the above, regardless of the
configuration of the machines.
Versioning
Docker includes git-like capabilities for tracking successive versions of a container, inspecting the diff between
versions, committing new versions, rolling back etc. The history also includes how a container was assembled and
by whom, so you get full traceability from the production server all the way back to the upstream developer. Docker
also implements incremental uploads and downloads, similar to git pull , so new versions of a container can be
transferred by only sending diffs.
Component re-use
Any container can be used as an "base image" to create more specialized components. This can be done manually or
as part of an automated build. For example you can prepare the ideal Python environment, and use it as a base for 10
different applications. Your ideal PostgreSQL setup can be re-used for all your future projects. And so on.

Sharing
Docker has access to a public registry (https://registry.hub.docker.com/) where thousands of people have uploaded
useful containers: anything from Redis, Couchdb, PostgreSQL to IRC bouncers to Rails app servers to Hadoop to
base images for various distros. The registry also includes an official "standard library" of useful containers
maintained by the Docker team. The registry itself is open-source, so anyone can deploy their own registry to store
and transfer private containers, for internal server deployments for example.
Tool ecosystem
Docker defines an API for automating and customizing the creation and deployment of containers. There are a huge
number of tools integrating with Docker to extend its capabilities. PaaS-like deployment (Dokku, Deis, Flynn),
multi-node orchestration (Maestro, Salt, Mesos, OpenStack Nova), management dashboards (Docker-UI, OpenStack
horizon, Shipyard), configuration management (chef, puppet), continuous integration (jenkins, strider, travis), etc.
Docker is rapidly establishing itself as the standard for container-based tooling.

Docker Use Cases
Docker has many use cases and advantages:

Versionning & Fast Deployment
Docker registry (or Docker Hub) could be considered as a version control system for a given application. Rollbacks
and updates are easier this way.
Just like Github, BitBucket or any other Git system, you can use tags to tag your images versions. Imagine you can
tag differently a container with each application release, it will be easier to deploy and rollback to the n-1 release.

As you may already know, Git-like systems gives you commit identifiers like 2.1-3-xxxxxx , these are not tags, you
can also use your Git system to tag you code, but for deployment you will need to download these tags or their
artifacts.
If your fellow developers are working on an application with thousands of packages dependencies like JavaScript
apps (using the package.json), you may be facing a deployment of an application with very small files to download
or update with probably some new configurations. A single Docker image with your new code, already builded and
tested and configurations will be easier and faster to ship and deploy.
Tagging is done with docker tag command, these tags are the base for the commit. Docker versionning and tagging
system is working also in this way.

Distribution & Collaboration
If you would like to share images and containers, Docker allows this social feature so that anyone can contribute to a
public (or private) image.
Individuals and communities can collaborate and share images. Users can also vote for images. In Docker Hub, you
can find trusted (official) and community images.
Some images have a continuous build and security scan feature to keep them up-to-date.

Multi Tenancy & High Availability
Using the right tools from the ecosystem, it is easier to run many instances of the same application in the same
server with Docker than the "main stream" way.
Using a proxy, a service discovery and a scheduling tool, you can start a second server (or more) and load-balance
your traffic between the cluster nodes where your containers are "living".

CI/CD
Docker is used in production systems but it is considered as a tool to run the same application in developer's
laptop/server. Docker may move from development to QA to production without being changed. If you would like
to be as close as possible to production, then Docker is a good solution.
Since it solves the problem of "it works on my machine", it is important to highlight this use case. Most problems in
software development and operations are due to the differences between development and production environments.
If your R&D team use the same image that QA team will test against and the same environment will be pushed to
live servers, it is sure that a great part of the problems (dev vs ops) will disappear.
There are many DevOps topologies in the software industry now and "container-centric" (or "container-based")
topology is one of them.
This topology makes both Ops and Dev teams share more responsabilities in common, which is a DevOps approach
to blur the boundaries between teams and encourage the co-creation.

Isolation & The Dependency Hell
Dockerizing an application is also isolating it into a separate environment.
Imagine you have to run two APIs with two different languages or running them with the same language but with
different versions.
You may need two incompatible versions of the same language, each API is running one of them, for example

Python 2 and Python 3.
If the two apps are dockerized, you don't need to install nothing on your host machine, just Docker, every version
will run in an isolated environment.
Since I start running Docker in production, most of my apps were dockerized, I stopped using the host system
package manager since that time, every new application or middleware were installed inside the container.
Docker simplifies the system packages management and eliminates the "dependency hell" by its isolation feature.

Using The Ecosystem
You can use Docker with multiple external tools like configuration management tools, orchestration tools, file
storage technologies, filesystem types, logging softwares, monitoring tools, self-healing tools ..etc
On the other hand, even with all the benefits of Docker, it is not always the best solution to use, there are always
exceptions.

Chapter II - Installation & Configuration
o
o

^__^
(oo)\_______
(__)\
)\/\
||----w |
||
||

In Painless Docker book, we are going to use a Docker version superior or equal to the 1.12.
I used to use previous stable version like 1.11, but a new important feature which is the Swarm Mode was introduced
in version 1.12. Swarm orchestration technology is directly integrated into Docker and just before it was an add-on.
I am a GNU/Linux user, but for Windows and Mac users, Docker unveiled with the same version, the first full
desktop editions of the software for development on Mac and Windows machines.
There are many other interesting features, enhancements and simplifications in the version 1.12 of Docker, you can
find the whole list in Docker github repository.
If you are completely new to Docker, you will not get all of the new following features, but you will be able to
understand them as you go along with this book.
The most important new features in Docker 1.12 are about the Swarm Mode:
Built-in Virtual-IP based internal and ingress load-balancing using IPVS
Routing Mesh using ingress overlay network
New swarm command to manage swarms with these subcommands:
init
join

,
,

join-token
leave

,

,

update

New service command to manage swarm-wide services with
subcommands

create

,

inspect

New node command to manage nodes with accept , promote , demote , inspect ,
subcommands
New stack and deploy commands to manage and deploy multi-service applications
Add support for local and global volume scopes (analogous to network scopes)

update

,

update

,

ps

,

,
ls

,

ps

and

rm

rm

Other important features in Docker were introduced in preceding versions like the multi-build support in the version
v17.05.0-ce.
When writing this book, I used Ubuntu 16.04 server edition with a 64 bit architecture as my main operating system,
but you will see how to install Docker in other OSs like Windows and MacOS.
For other Linux distributions users, things are not really different, except the package manager (apt/aptitude) that
you should replace by your own one - we are going to explain the installation for other distributions like CentOs.

Requirements & Compatibility
Docker itself does not need many resources so a little RAM could help you install and run Docker engine. But
running containers depends on what are you running exactly, in the case you are running a Mysql or a busy
MongoDB inside a container, you will need more memory than running a small Nodejs or Python application .
Docker requires a 64 bit Kernel.
For developers using Windows or Mac, you have the choice to use Docker Toolbox or native Docker. Native Docker
is for sure faster but you still have the choice.
If you will use Docker Toolbox:
Mac users: Your Mac must be running OS X 10.8 "Mountain Lion" or newer to run Docker.
Windows users: Your machine must have a 64-bit operating system running Windows 7 or higher. You should
have an enabled virtualization.
If you prefer Docker for Mac as it is mentioned in the official Docker website:
Your Mac must be a 2010 or newer model, with Intel’s hardware support for memory management unit (MMU)
virtualization; i.e., Extended Page Tables (EPT) OS X 10.10.3 Yosemite or newer
You must have at least 4GB of RAM
You must have VirtualBox prior to version 4.3.30 must NOT be installed (it is incompatible with Docker for
Mac : uninstall the older version of VirtualBox and re-try the install if you already missed this).
And if you prefer Docker for Windows:
Your machine should have a 64bit Windows 10 Pro, Enterprise and Education (1511 November update, Build
10586 or later).
The Hyper-V package must be enabled and if it will be installed by Docker for Windows installer, it will enable
it for you

Installing Docker On Linux
Docker is supported by all Linux distributions satisfying the requirements, but not with all of the versions and this is
due to compatibility of Docker with old Kernel versions.
Kernels older than 3.10 will not support Docker and can cause data loss or any other bugs.
Check your Kernel by typing:
uname -r

Docker recommends making an upgrade, a dist upgrade and having the latest Kernel version for your servers before
using it in production.

Ubuntu
For Ubuntu, only those versions are supported to run and manage containers:
Ubuntu Xenial 16.04 (LTS)
Ubuntu Wily 15.10
Ubuntu Trusty 14.04 (LTS)
Ubuntu Precise 12.04 (LTS)
Update your package manager, add the apt key & Docker list then type the update command.
sudo
sudo
sudo
echo
sudo

apt-get update
apt-get install apt-transport-https ca-certificates
apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys 58118E89F3A912897C070ADBF76221572C52609D
"deb https://apt.dockerproject.org/repo ubuntu-trusty main"|tee -a /etc/apt/sources.list.d/docker.list
apt-get update

Purge the old lxc-docker if you were using it before and install the new Docker Engine:
sudo apt-get purge lxc-docker
sudo apt-get install docker-engine

If you need to run Docker without root rights (with your actual user), run the following commands:
sudo groupadd docker
sudo usermod -aG docker $USER

If everything was ok, then running this command will create a container that will print a Hello World message than
exits without errors:
docker run hello-world

There is a good explanation about how Docker works in the output, if you have not noticed it, here it is:
Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
1. The Docker client contacted the Docker daemon.
2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
3. The Docker daemon created a new container from that image which runs the
executable that produces the output you are currently reading.
4. The Docker daemon streamed that output to the Docker client, which sent it
to your terminal.

CentOS
Docker runs only on CentOS 7.X. Same installation may apply to other EL7 distributions (but they are not supported
by Docker)
Add the yum repo.
sudo tee /etc/yum.repos.d/docker.repo <<-'EOF'
[dockerrepo]
name=Docker Repository
baseurl=https://yum.dockerproject.org/repo/main/centos/7/
enabled=1
gpgcheck=1
gpgkey=https://yum.dockerproject.org/gpg
EOF

Install Docker:
sudo yum install docker-engine

Start its service:
sudo service docker start

Set the daemon to run at system boot:
sudo chkconfig docker on

Test the Hello World image:
docker run hello-world

If you see a similar output to the following one, than your installation is fine:
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
c04b14da8d14: Pull complete
Digest: sha256:0256e8a36e2070f7bf2d0b0763dbabdd67798512411de4cdcf9431a1feb60fd9
Status: Downloaded newer image for hello-world:latest
Hello from Docker!
This message shows that your installation appears to be working correctly.
To generate this message, Docker took the following steps:
1. The Docker client contacted the Docker daemon.
2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
3. The Docker daemon created a new container from that image which runs the
executable that produces the output you are currently reading.
4. The Docker daemon streamed that output to the Docker client, which sent it
to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
$ docker run -it ubuntu bash
Share images, automate workflows, and more with a free *Docker Hub* account:
https://hub.docker.com
For more examples and ideas, visit:
https://docs.docker.com/engine/userguide/

Now if you would like to create a Docker group and add your current user to it in order to avoid running command
with sudo privileges :
sudo groupadd docker
sudo usermod -aG docker $USER

Verify your work by running the hello-world container without sudo.

Debian
Only:
Debian testing stretch (64-bit)
Debian 8.0 Jessie (64-bit)
Debian 7.7 Wheezy (64-bit) (backports required)
are supported.
We are going to use the installation for Weezy. In order to install Docker on Jessie (8.0), change the entry for
backports and source.list entry to Jessie.
First of all, enable backports:
sudo su
echo "deb http://http.debian.net/debian wheezy-backports main"|tee -a /etc/apt/sources.list.d/backports.list
apt-get update
Purge other Docker versions if you have already used them:
``` bash
apt-get purge "lxc-docker*"
apt-get purge "docker.io*"

and update you package manager:
apt-get update

Install apt-transport-https and ca-certificates
apt-get install apt-transport-https ca-certificates

Add the GPG key.
apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys 58118E89F3A912897C070ADBF76221572C52609D

Add the repository:

echo "deb https://apt.dockerproject.org/repo debian-wheezy main"|tee -a /etc/apt/sources.list.d/docker.list
apt-get update

And install Docker:
apt-get install docker-engine

Start the service
service docker start

Run the Hello World container in order to check if everything is good:
sudo docker run hello-world

You will have a similar output to the following, if Docker is installed without problems:
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
c04b14da8d14: Pull complete
Digest: sha256:0256e8a36e2070f7bf2d0b0763dbabdd67798512411de4cdcf9431a1feb60fd9
Status: Downloaded newer image for hello-world:latest
Hello from Docker!
This message shows that your installation appears to be working correctly.
To generate this message, Docker took the following steps:
1. The Docker client contacted the Docker daemon.
2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
3. The Docker daemon created a new container from that image which runs the
executable that produces the output you are currently reading.
4. The Docker daemon streamed that output to the Docker client, which sent it
to your terminal.
To try something more ambitious, you can run an Ubuntu container with:
$ docker run -it ubuntu bash
Share images, automate workflows, and more with a free *Docker Hub* account:
https://hub.docker.com
For more examples and ideas, visit:
https://docs.docker.com/engine/userguide/

Now, in order to use your current user (not root user) to manage and run Docker, add the docker group if it does not
already exist.
exit # Exit from sudo user
sudo groupadd docker

Add your preferred user to this group:
sudo gpasswd -a ${USER} docker

Restart the Docker daemon.
sudo service docker restart

Test the Hello World container to check if your current user have right to execute Docker commands.

Docker Toolbox
Few months ago, installing Docker for my developers using MacOs and Windows was a pain. Now the new Docker
Toolbox have made things easier. Docker Toolbox is quick and easy installer that will setup a full Docker
environment. The installation includes Docker, Machine, Compose, Kitematic, and VirtualBox.

Docker Toolbox could be downloaded from Docker's wesbite.
Using this tool you will be able to work with:

docker-machine commands
docker commands
docker-compose commands
The Docker GUI (Kitematic)
a shell preconfigured for a Docker command-line environment
and Oracle VirtualBox
The installation is quite easy:

This is a screenshot for Docker

If you would like a default installation press "Next" to accept all and then click on Install. If you are running
Windows, make sure you allow the installer to make the necessary changes.

Now that you finished the installation, on the application folder, click on "Docker Quickstart Terminal".

Mac users, type the following command in order to :
Create a machine dev...
Create a VirtualBox VM...
Create SSH key...
Start the VirtualBox VM...
Start the VM...
Start the machine dev
Set the environment variables for machine dev
Windows users you can also follow the following instructions, since there are common commands between the two
OSs.
bash '/Applications/Docker Quickstart Terminal.app/Contents/Resources/Scripts/start.sh'

Running the following command will show you how to connect Docker to this machine:

docker-machine env dev

Now for testing, use the Hello World container:
docker run hello-world

You will probably see this message:
Unable to find image 'hello-world:latest' locally

This is not an error but Docker is saying that the image Hello World will not be used from your local disk but it will
be pulled from Docker Hub.
latest: Pulling from library/hello-world
535020c3e8ad: Pull complete
af340544ed62: Pull complete
Digest: sha256:a68868bfe696c00866942e8f5ca39e3e31b79c1e50feaee4ce5e28df2f051d5c
Status: Downloaded newer image for hello-world:latest
Hello from Docker.
This message shows that your installation appears to be working correctly.

If you are using Windows, it is actually not very different.
Click on the "Docker Quickstart Terminal" icon, if your operating system displays a prompt to allow VirtualBox,
choose yes and a terminal will show on your screen.
To test if Docker is working, type:
docker run hello-world

You will see the following message:
Hello from Docker.

You may also notice the explanation of how Docker is working on your local machine.
To generate this message ("Hello World" message), Docker took the following steps:

- The *Docker Engine CLI* client contacted the *Docker Engine daemon*.
The
*Docker
Engine
daemon*
pulled
the
*helloworld* image from the *Docker Hub*. (Assuming it was not already locally available.)
- The *Docker Engine daemon* created a new container from that image which runs the executable that produces the output you are currently reading
- The *Docker Engine daemon* streamed that output to the *Docker Engine CLI* client, which sent it to your terminal.

After the installation, you can also start using the GUI or the command line, click on the create button to create a
Hello World container just to make sure if everything is OK.

Docker Toolbox is a very good tool for every developer but you may need more performance with larger projects in
your local development. Docker for Mac and Docker for Windows are native for each OS.

Docker For Mac
Use the following link to download the .dmg file and install the native Docker.
https://download.docker.com/mac/stable/Docker.dmg

To use native Docker, get back to the requirements section and make sure of your system configuration.
After the installation, drag and drop Docker.app to your Applications folder and start Docker from your applications
list.

You will a whale icon on your status bar and when you click on it, you can see a list of choices and you can also
click on About Docker to verify if you are using the right version.
If you prefer using the CLI, open your terminal and type:
docker --version

or
docker -v

If you installed Docker 1.12, you will see:
Docker version 1.12.0, build 8eab29e

If you go to Docker.app preferences, you can find some configurations, but one of the most important ones are
sharing drivers.

In many cases, your containers running in your local machines can use a file system mounted to a folder in your
host, we will not need this for the moment but you should remember later in this book that if you mount a container
to a local folder, you should get back to this step and share the concerned files, directories, users or volumes on your
local system with your containers.

Docker For Windows
Use the following link to download the .msi file and install the native Docker.
https://download.docker.com/win/stable/InstallDocker.msi

Same thing for Windows: To use native Docker, get back to the requirements section and make sure of your
system configuration.
Double-click InstallDocker.msi and run the installer
Follow the installation wizard
Authorize Docker if you were asked for that by your system
Click Finish to start Docker

If everything was OK, you will get a popup with a success message.
Now open cmd.exe (or PowerShell) and type
docker --version

or
docker version

If your containers running on your local development environment may need in many cases (that we will see in this
book) to access to your file system, folders, files or drives. This is the case when you mount a folder inside the
Docker container to your host file system. We will see many examples of this type so you should remember to get
back here and make the right configurations if mounting a directory or a file will be needed later in this book.

Docker Experimental Features
Even if Docker has a stable version that can be safely used in production environments, many features are still in
development and you may need to plan for your future projects using Docker, so in this case, you will need to test
some of these features.
I have been testing Docker Swarm Mode since it was experimental and I needed to evaluate this feature in order to
prepare the adequate architecture, servers and adopt development and integration work flows to the coming changes.

You may find some instability and bugs using the experimental installation packages which is normal.

Docker Experimental Features For Mac And Windows
For native Docker running on both systems, to evaluate the experimental features, you need to download the beta
channel installation packages.
For Mac:
https://download.docker.com/mac/beta/Docker.dmg

For Windows
https://download.docker.com/win/beta/InstallDocker.msi

Docker Experimental Features For Linux
Running the following command will install the experimental version of Docker:
You should have curl installed
curl -sSL https://experimental.docker.com/ | sh

Genrally curl | bash is not a good security practice even if the transport is over HTTPS. Content can be
modified on the server.
You can download the script, read it and execute it:
wget https://experimental.docker.com/

Or you can get one of the following binaries in function of your system architecture:
https://experimental.docker.com/builds/Linux/i386/docker-latest.tgz
https://experimental.docker.com/builds/Linux/x86_64/docker-latest.tgz

For the remainder of the installation :
tar -xvzf docker-latest.tgz
mv docker/* /usr/bin/
sudo dockerd &

Removing Docker
Let's take Ubuntu as an example.
Purge the Docker Engine:
sudo apt-get purge docker-engine
sudo apt-get autoremove --purge docker-engine
sudo apt-get autoclean

This is enough in most cases, but to remove all of Docker's files, follow the next steps.
If you wish to remove all the images, containers and volumes:
sudo rm -rf /var/lib/docker

Then remove docker from apparmor.d:
sudo rm /etc/apparmor.d/docker

Then remove docker group:
sudo groupdel docker

You have successfully deleted completely docker.

Docker Hub
Docker Hub is a cloud registry service for Docker.
Docker allows to package artifacts/code and configurations into a single image. These images can be reusable by
you, your colleague or even your customer. If you would like to share your code you will generally use a git
repository like Github or Bitbucket.
You can also run your own Gitlab that will allows you to have your own private on-premise Git repositories.
Things are very similar with Docker, you can use a cloud-based solution to share your images like Docker Hub or
use your own Hub (a private Docker registry).
Docker Hub is a public Docker repository, but if you want to use a cloud-based solution while keeping your
images private, the paid version of Docker Hub allows you to have privates repositories.
Docker Hub allows you to
Access to community, official, and private image libraries
Have public or paid private image repositories to where you can push your images and from where your could
pull them to your servers
Create and build new images with different tags when the source code inside your container changes
Create and configure webhooks and trigger actions after a successful push to a repository
Create workgroups and manage access to your private images
Integrate with GitHub and Bitbucket

Basically Docker Hub could be a component of your dev-test pipeline automation.
In order to use Docker Hub, go to the following link and create an account:
https://hub.docker.com/

If you would like to test if your account is enabled, type
docker login

Login with your Docker ID to push and pull images from Docker Hub. If you don't have a Docker ID, head
over to https://hub.docker.com to create one.
Now, go to Docker Hub website and create a public repository. We will see how to send a running container as an
image to Docker Hub and for the same reason we are going to use a sample app genrally used by Docker for demos,
called vote (that you can also find on Docker official Github repository).
Vote application is a Python webapp that lets you vote between two options, it uses a Redis queue to collects new
votes, .NET worker which consumes votes and stores them in a Postgres database backed by a Docker volume and a
Node.Js webapp which shows the results of the voting in real time.

I consider that you created a working account on Docker Hub, typed the login command and entered the right
password.
If have a starting level with Docker, you may not understand all of the next commands but the goal of this section is
just to demonstrate how a Docker Registry works (In this case, the used Docker Registry is a cloud-based one built
by Docker, and as said, it is called Docker Hub).
When you type the following command, Docker will search if it has the image locally, otherwise it will check if it is
on Docker Hub:
docker run -d -it -p 80:80 instavote/vote

You can find the image here:
https://hub.docker.com/r/instavote/vote/

Now type this command to show the running container. This is the equivalent of ps command in Linux systems for
Docker:
docker ps

You can see here that the nauseous_albattani container (a name given automatically by Docker), is running the vote
application pulled from instavote/vote repository.
CONTAINER ID
IMAGE
COMMAND
136422f45b02
instavote/vote
"gunicorn
>80/tcp
nauseous_albattani

CREATED
app:app -b "

8

STATUS
minutes

ago

PORTS
Up 8

The container id is : 136422f45b02 and the application is reachable via http://0.0.0.0:80

minutes

NAMES
0.0.0.0:80-

Just like using git, we are going to commit and push the image to our Docker Hub repository. No need to create a
new repository, the commit/push can be used in a lazy mode, it will create it for you.
Commit:
docker commit -m "Painless Docker first commit" -a "Aymen El Amri" 136422f45b02 eon01/painlessdocker.com_voteapp:v1
sha256:bf2a7905742d85cca806eefa8618a6f09a00c3802b6f918cb965b22a94e7578a

And push:
docker push eon01/painlessdocker.com_voteapp:v1
The push refers to a repository [docker.io/eon01/painlessdocker.com_voteapp]
1f31ef805ed1: Mounted from eon01/painless_docker_vote_app
3c58cbbfa0a8: Mounted from eon01/painless_docker_vote_app
02e23fb0be8d: Mounted from eon01/painless_docker_vote_app
f485a8fdd8bd: Mounted from eon01/painless_docker_vote_app
1f1dc3de0e7d: Mounted from eon01/painless_docker_vote_app
797c28e44049: Mounted from eon01/painless_docker_vote_app
77f08abee8bf: Mounted from eon01/painless_docker_vote_app
v1: digest: sha256:658750e57d51df53b24bf0f5a7bc6d52e3b03ce710a312362b99b530442a089f size: 1781

Change eon01 by your username.
Notice that a new repository is added automatically to my Docker Hub dashboard:

Now you can pull the same image with the latest tag:
docker pull eon01/painlessdocker.com_voteapp

Or with a specific tag:
docker pull eon01/painlessdocker.com_voteapp:v1

In our case, the v1 is the latest version, so the result of the two above commands will be the same image pulled to
your local image.

Docker Registry
Docker Registry is a scalable server side application conceived to be an on-premise Docker Hub. Just like Docker
Hub, it helps you push, pull and distribute your images.
The software powering Docker Registry is open-source under Apache license. Docker Registry could be also a
cloud-based solution, because Docker's commercial offer called Docker Trusted Registry is cloud-based.
Docker Registry could be run using Docker. A Docker image for the Docker Registry is available here:
https://hub.docker.com/_/registry/

It is easy to create a registry, just pull and run the image like this:
docker run -d -p 5000:5000 --name registry registry:2.5.0

Let's test it : We will pull an image from Docker Hub, tag and push it to our own registry .
docker
docker
docker
docker

pull ubuntu
tag ubuntu localhost:5000/myfirstimage
push localhost:5000/myfirstimage
pull localhost:5000/myfirstimage

Deploying Docker Registry On Amazon Web Services
You need to have:
An Amazon Web Services account
The good IAM privileges
Your aws CLI configured
Type:
aws configure

Type your credentials, choose your region and your preferred output format:
AWS Access Key ID [None]: ******************
AWS Secret Access Key [None]: ***********************
Default region name [None]: eu-west-1
Default output format [None]: json

Create an EBS disk (Elastic Block Store), specify the region you are using and the availability zone.
aws ec2 create-volume --size 80 --region eu-west-1 --availability-zone eu-west-1a --volume-type standard

You should have an similar output to the following one:

{
"AvailabilityZone": "eu-west-1a",
"Encrypted": false,
"VolumeType": "standard",
"VolumeId": "vol-xxxxxx",
"State": "creating",
"SnapshotId": "",
"CreateTime": "2016-10-14T15:29:35.400Z",
"Size": 80
}

Keep the output, because we are going to use the volume id later.
Our choice for the volume type was 'standard' and you must choose your preferred volume type and it is mainly
about the iops.
The following table could help:

Magnetic
GP
PIOPS

IOPS
Up to 100 IOPS/volume
Up to 3000 IOPS/volume
Up to 4000 IOPS/volume

Use Case
Little access
Larger access needs, suitable for the majority of classic cases
High speed access

Start an EC2 instance:
aws ec2 run-instances --image-id ami-xxxxxxxx --count 1 --instance-type t1.medium --key-name MyKeyPair --security-groupids sg-xxxxxxxx --subnet-id subnet-xxxxxxxx

Replace your image id, instance type, key name, security group ids and subnet id with your proper values.
On the output look for the instance id because we are going to use it.
{
"OwnerId": "xxxxxxxx",
"ReservationId": "r-xxxxxxx",
"Groups": [
{
[..]
}
],
"Instances": [
{
"InstanceId": "i-5203422c",
[..]
}

aws ec2 attach-volume --volume-id vol-xxxxxxxxxxx --instance-id i-xxxxxxxx --device /dev/sdf

Now that the volume is attached, you should check your volumes in the EC2 instance with a
your new attached instance.
In this example, let's say the attached EBS has the following device name
system upon the volume:

/dev/xvdf

sudo mkfs -t ext4 /dev/xvdf
mkdir /data

Make sure you get the right device name for the new attached volume.
Now go to the fstab configuration file:

df -kh

and you will see

. Create a folder and a new file

/etc/fstab
and add :
``` bash
/dev/xvdf

/data

ext4

defaults

1 1

Now mount the volume by typing :
mount -a

You should have Docker installed in order to run a private Docker registry.
The next step is running the registry:
docker run -d -p 80:5000 --restart=always -v /data:/var/lib/registry registry:2

If you type

docker ps

CONTAINER ID
bb6201f63cc5

, you should see the registry running:

IMAGE
registry:2

COMMAND
"/entrypoint.sh /etc/"

CREATED
21 hours

STATUS
Up About 1h

PORTS
0.0.0.0:80->5000/tcp

Now you should create an ELB but first create its Security Group (expose port 443).
Create the ELB using AWS CLI or AWS Console and redirect traffic to the EC2 port number 80. You can get the ELB
DNS since we are going to use it to push and pull images.
Opening port 443 is needed since the Docker Registry needs it to send and receive data, that's why we used ELB
since the latter has integrated certificate management and SSL decryption.
It is also used to build a highly available system.
Now let's test it by pushing an image:
docker
docker
docker
docker

pull hello-world
tag hello-world load_balancer_dns/hello-world:1
push load_balancer_dns/hello-world
pull load_balancer_dns/hello-world

If you don't want to use ELB, you should bring your own certificates and run:
docker run -d -p 5000:5000 --restart=always --name registry \
-v `pwd`/certs:/certs \
-v /data:/var/lib/registry \
-e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt \
-e REGISTRY_HTTP_TLS_KEY=/certs/domain.key \
registry:2

Another option to use is to run storage on AWS S3:
docker run \
-e SETTINGS_FLAVOR=s3 \
-e AWS_BUCKET=my_bucket \
-e STORAGE_PATH=/data \
-e AWS_REGION="eu-west-1"
-e AWS_KEY=*********** \
-e AWS_SECRET=*********** \
-e SEARCH_BACKEND=sqlalchemy \
-p 80:5000 \

registry

In this case, you should not forget to add a policy for S3 that allows the Docker Registry to read and write your
images to S3.

Deploying Docker Registry On Azure
Using Azure, we are going to deploy the same Docker Registry using Azure Storage service.
You will also need a configured Azure CLI.
We need to create a storage account using the Azure CLI:
azure storage account create -l "North Europe" 

Change



by your proper value.

Now we need to list the storage account keys to use one of them later:
azure storage account keys list 

Run:
docker run -d -p 80:5000 \
-e REGISTRY_STORAGE=azure \
-e REGISTRY_STORAGE_AZURE_ACCOUNTNAME="" \
-e REGISTRY_STORAGE_AZURE_ACCOUNTKEY="" \
-e REGISTRY_STORAGE_AZURE_CONTAINER="registry" \
--name=registry \
registry:2

If the port 80 is closed on your Azure virtual machine, you should open it:
azure vm endpoint create  80 80

Configuring security for the Docker Registry is not covered in this part.

Docker Store
Docker Store is a Docker inc product and it is designed to provide a scalable self-service system for ISVs to publish
and distribute trusted and enterprise-ready content
It provides a publishing process that includes:
security scanning
component inventory
The open-source license usage
Image construction guidelines

In other words, it is an official marketplace with workflows to create and distribute content were you can find free
and commercial images.

Chapter III - Basic Concepts
o
o

^__^
(oo)\_______
(__)\
)\/\
||----w |
||
||

Hello World, Hello Docker
Through the following chapters, you will learn how to manipulate Docker containers (create, run, delete, update,
scale ..etc).
But to ensure a better understanding of some concepts, we need to learn at least how to run a Hello World container.
We have already seen this, but as a reminder, and to be sure that your installation was good, we will run the Hello
World container again:
docker run --rm -it hello-world

You can see a brief explanation of how Docker has created the container on the command output:
To generate this message, Docker took the following steps:
1. The Docker client contacted the Docker daemon.
2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
3. The Docker daemon created a new container from that image which runs the
executable that produces the output you are currently reading.
4. The Docker daemon streamed that output to the Docker client, which sent it
to your terminal.

This gives a good explanation of how Docker created a container and it will be a waste of energy to re-explain this
in my own words, albeit Docker uses more energy to do the same task :-)

General Information About Docker
Docker is a very active project and its code is frequently changing, to understand many concepts about this
technology, I have been following the project on Github and browsing its issues when I had problems and especially
when I found bugs.
Therefore, it is important to know the version you are using in your production servers.
version and the build number you are using.

docker -v

will give you the

Example:
Docker version 1.12.2, build bb80604

You can get other general information about the server/client version, the architecture, the Go version ..etc. Use
docker version to get the latter information. Example:
Client:
Version:
API version:
Go version:
Git commit:
Built:
OS/Arch:

1.12.2
1.24
go1.6.3
bb80604
Tue Oct 11 18:19:35 2016
linux/amd64

Server:
Version:
API version:
Go version:
Git commit:
Built:
OS/Arch:

1.12.2
1.24
go1.6.3
bb80604
Tue Oct 11 18:19:35 2016
linux/amd64

Docker Help
You can view the Docker help using the command line:
docker --help

You will get a list of
options like:
--config=~/.docker
-D, --debug
-H, --host=[]
-h, --help
-l, --log-level=info
--tls
--tlscacert=~/.docker/ca.pem
--tlscert=~/.docker/cert.pem
--tlskey=~/.docker/key.pem
--tlsverify
-v, --version

Location of client config files
Enable debug mode
Daemon socket(s) to connect to
Print usage
Set the logging level
Use TLS; implied by --tlsverify
Trust certs signed only by this CA
Path to TLS certificate file
Path to TLS key file
Use TLS and verify the remote
Print version information and quit

commands like:
attach
build
commit
cp
create
diff

Attach to a running container
Build an image from a Dockerfile
Create a new image from a container's changes
Copy files/folders between a container and the local filesystem
Create a new container
Inspect changes on a container's filesystem

If you need more help about a specific command like cp or rmi, you need to type:
docker cp --help
docker rmi --help

In some cases, you may get "the command third level" of help like:
docker swarm init --help

Docker Events
To start this section, let's run a MariaDB container and list Docker Events. For this manipulation, you can use
terminator to split your screen into two and notice in the same time the events output while typing the following
command:
docker run --name mariadb -e MYSQL_ROOT_PASSWORD=password -v /data/db:/var/lib/mysql -d mariadb
Unable to find image 'mariadb:latest' locally
latest: Pulling from library/mariadb
386a066cd84a: Already exists
827c8d62b332: Pull complete
de135f87677c: Pull complete
05822f26ca6e: Pull complete
ad65f56a251e: Pull complete
d71752ae05f3: Pull complete
87cb39e409d0: Pull complete
8e300615ba09: Pull complete
411bb8b40c58: Pull complete
f38e00663fa6: Pull complete
fb7471e9a58d: Pull complete
2d1b7d9d1b69: Pull complete
Digest: sha256:6..c
Status: Downloaded newer image for mariadb:latest
0..7

The events launched by the last command are:
docker events
2016-12-09T00:54:51.827303500+01:00
2016-12-09T00:54:52.000498892+01:00
2016-12-09T00:54:52.137792817+01:00
2016-12-09T00:54:52.550648364+01:00

image pull mariadb:latest (name=mariadb)
container create 0..7 (image=mariadb, name=mariadb)
network connect 8..d (container=0..7, name=bridge, type=bridge)
container start 0..7 (image=mariadb, name=mariadb)

Explicitly:
image pull : The image is pulled from the public repository by it identifier Mariadb.
container create : The container is created from the pulled image and it was given a Docker identifier
0..7 . At this stage the container is not running yet
network connect : The container is attached to a network called
At this stage the container is not running yet

bridge

having the identifier

8..d

.

container start : The container at this stage is running on your host system with the same identifier
0..7

This is the full list of Docker events (44) that can be reported by images, containers, networks, plugins, volumes and
Docker daemon:
-

attach
commit
copy
create
destroy
detach
die
exec_create
exec_detach
exec_start
export
health_status
kill
oom
pause
rename
resize
restart
start
stop
top
unpause
update
delete
import
load
pull
push
save
tag
untag
install
enable
disable
remove
create
mount
unmount
destroy
create
connect
disconnect
destroy
reload

Using Docker API To List Events

There is a chapter about Docker API that will help you discover it in more details, but there is no harm to start using
it now.
You can, in fact, use it to see Docker Events while manipulating containers, images, networks .. etc.
Install curl and type curl --unix-socket /var/run/docker.sock http:/events then open another terminal window (or a new
tab) and type docker pull mariadb , docker rmi -f mariadb to remove the pulled image then docker pull mariadb to pull it
again.
These are the 3 events reported by the 3 commands typed above:
Action = Pulling and image that already exists in the host:
{
"status":"pull",
"id":"mariadb:latest",
"Type":"image",
"Action":"pull",
"Actor": {
"ID":"mariadb:latest",
"Attributes": {
"name":"mariadb"
}
},
"time":1481381043,
"timeNano":1481381043157632493
}

Action = Removing it:
{
"status":"untag",
"id":"sha256:0..c",
"Type":"image",
"Action":"untag",
"Actor" {
"ID":"sha256:0..c",
"Attributes" {
"name":"sha256:0..c"
}
},
"time":1481381060,
"timeNano":1481381060026443422
}

Action = Pulling the same image again from a distant repository:
{
"status":"pull",
"id":"mariadb:latest",
"Type":"image",
"Action":"pull",
"Actor":{
"ID":"mariadb:latest",
"Attributes":{
"name":"mariadb"
}
},
"time":1481381069,
"timeNano":1481381069629194420
}

Each event has a status (pull, untag for removing the image ..etc), a resource identifier (id) with its Type (image,
container, network ..) and other information like:
Actor,

time
and timeNano
You can use DoMonit (a Docker API wrapper) that I created for this book to discover Docker API.

Docker API is detailed in a separate chapter.
To test DoMonit, you can create a folder and install a Python virtual environment:
virtualenv DoMonit/
New python executable in /home/eon01/DoMonit/bin/python
Installing setuptools, pip, wheel...done.

cd DoMonit

Clone the repository:
git clone https://github.com/eon01/DoMonit.git
Cloning into 'DoMonit'...
remote: Counting objects: 237, done.
remote: Compressing objects: 100% (19/19), done.
remote: Total 237 (delta 9), reused 0 (delta 0), pack-reused 218
Receiving objects: 100% (237/237), 106.14 KiB | 113.00 KiB/s, done.
Resolving deltas: 100% (128/128), done.
Checking connectivity... done.

List the files inside the created folder:
ls -l
total 24
drwxr-xr-x
drwxr-xr-x
drwxr-xr-x
drwxr-xr-x
drwxr-xr-x
-rw-r--r--

2
6
2
3
2
1

eon01
eon01
eon01
eon01
eon01
eon01

sudo
sudo
sudo
sudo
sudo
sudo

4096
4096
4096
4096
4096
60

Dec
Dec
Dec
Dec
Dec
Dec

11
11
11
11
11
11

14:46
14:46
14:45
14:45
14:45
14:46

bin
DoMonit
include
lib
local
pip-selfcheck.json

Activate the execution environment:
. bin/activate

Install the requirements:
pip install -r DoMonit/requirements.txt

Collecting pathlib==1.0.1 (from -r DoMonit/requirements.txt (line 1))
/home/eon01/DoMonit/local/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:318:
SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform.
This may cause the server to present an incorrect TLS certificate, which can cause validation failures.
You can upgrade to a newer version of Python to solve this.
For more information, see https://urllib3.readthedocs.io/en/latest/security.html#snimissingwarning.
SNIMissingWarning
/home/eon01/DoMonit/local/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:122:
InsecurePlatformWarning:
A true SSLContext object is not available.
This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail.
You can upgrade to a newer version of Python to solve this.
For more information, see https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning.
InsecurePlatformWarning
Collecting PyYAML==3.11 (from -r DoMonit/requirements.txt (line 2))

Collecting requests==2.10.0 (from -r DoMonit/requirements.txt (line 3))
Using cached requests-2.10.0-py2.py3-none-any.whl
Collecting requests-unixsocket==0.1.5 (from -r DoMonit/requirements.txt (line 4))
Collecting simplejson==3.8.2 (from -r DoMonit/requirements.txt (line 5))
Collecting urllib3==1.16 (from -r DoMonit/requirements.txt (line 6))
Using cached urllib3-1.16-py2.py3-none-any.whl
Installing collected packages: pathlib, PyYAML, requests, urllib3, requests-unixsocket, simplejson
Successfully installed PyYAML-3.11 pathlib-1.0.1 requests-2.10.0 requests-unixsocket-0.1.5 simplejson-3.8.2 urllib3-1.16

and start streaming the events using:
python DoMonit/events_test.py

On another terminal, create a network and notice the output of the executed program:
docker network create test1
6d517f8251736446874e14bafeb08c76cdc6d8219537b1ed1af64984bb3590b2

The output should be something like:
{
"Type": "network",
"Action": "create",
"Actor": {
"ID": "6d517f8251736446874e14bafeb08c76cdc6d8219537b1ed1af64984bb3590b2",
"Attributes": {
"name": "test1",
"type": "bridge"
}
},
"time": 1481464270,
"timeNano": 1481464270716590446
}

This program called the Docker Events API:
curl --unix-socket /var/run/docker.sock http:/events

Let's get back to Docker Events command since we haven't seen yet the options used to stream events.
Docker Events supports the following filters (using in most cases the name or the id of the resource):
container=
event=
image=
plugin=
label= / label==
type=
volume=
network=
daemon=

Examples:
docker events --filter 'event=start'
docker events --filter 'image=mariadb'

docker events --filter 'container=781567cd25632'
docker events --filter 'container=781567cd25632' --filter 'event=stop'
docker events --filter 'type=volume'

In the next sections we are going to see what each Docker component (volumes, containers ..etc) reports as an event.

Monitoring A Container Using Docker Events
Note that we are always using the container of MariaDB database that we can create like this:
docker run --name mariadb -e MYSQL_ROOT_PASSWORD=password -v /data/db:/var/lib/mysql -d mariadb

If you haven't created the container yet, do it.
Based on the examples of Docker Events given above, we are going to create a small monitoring script to send us an
email if MariaDB containers reports a stop event.
We are going to listen to a file in

/tmp

and send an email as soon as we read a line.

tail -f /tmp/events | while read line; do

echo "$line" | mail -s subject "amri.aymen@gmail.com"; done

Yes, this is my real email address if you would like to stay in touch !
Using the Docker Events command, we can redirect all the logs to a file (in this case we will use
docker events --filter 'event=stop' --filter 'container=mariadb' > /tmp/events

If you have already created the container, you can test this simple monitoring system using :
docker stop mariadb
docker start mariadb

This is what you should get by email:
{
"Type":"network",
"Action":"connect",
"Actor":{
"ID":"7b64910a4b7b6b2b90245db1c7a1d8330fb1d0db9237da8c64378c0ad7745e9e",
"Attributes":{
"container":"9..d",
"name":"bridge",
"type":"bridge"
}
},
"time":1481466943,
"timeNano":1481466943153467120
}{
"Type":"network",
"Action":"connect",
"Actor":{
"ID":"7b64910a4b7b6b2b90245db1c7a1d8330fb1d0db9237da8c64378c0ad7745e9e",
"Attributes":{
"container":"9..d",
"name":"bridge",
"type":"bridge"
}
},

/tmp/events

).

"time":1481466943,
"timeNano":1481466943153467120
}{
"status":"start",
"id":"9..d",
"from":"mariadb",
"Type":"container",
"Action":"start",
"Actor":{
"ID":"9..d",
"Attributes":{
"image":"mariadb",
"name":"mariadb"
}
},
"time":1481466943,
"timeNano":1481466943266967807
}{
"s
TAG
hello-world
tatus":"start",
"id":"9..d",
"from":"mariadb",
"Type":"container",
"Action":"start",
"Actor":{
"ID":"9..d",
"Attributes":{
"image":"mariadb",
"name":"mariadb"
}
},
"time":1481466943,
"timeNano":1481466943266967807
}{
"status":"kill",
"id":"9..d",
"from":"mariadb",
"Type":"container",
"Action":"kill",
"Actor":{
"ID":"9..d",
"Attributes":{
"image":"mariadb",
"name":"mariadb",
"signal":"15"
}
},
"time":1481466945,
"timeNano":1481466945523301735
}{
"status":"kill",
"id":"9..d",
"from":"mariadb",
"Type":"container",
"Action":"kill",
"Actor":{
"ID":"9..d",
"Attributes":{
"image":"mariadb",
"name":"mariadb",
"signal":"15"
}
},
"time":1481466945,
"timeNano":1481466945523301735
}{
"status":"die",
"id":"9..d",
"from":"mariadb",
"Type":"container",
"Action":"die",
"Actor":{
"ID":"9..d",
"Attributes":{
"exitCode":"0",
"image":"mariadb",
"name":"mariadb"
}
},
"time":1481466948,
"timeNano":1481466948241703985
}{
"status":"die",
"id":"9..d",
"from":"mariadb",
"Type":"container",
"Action":"die",
"Actor":{
"ID":"9..d",
"Attributes":{
"exitCode":"0",
"image":"mariadb",

IMAGE ID

CREATED

SIZE

"name":"mariadb"
}
},
"time":1481466948,
"timeNano":1481466948241703985
}{
"Type":"network",
"Action":"disconnect",
"Actor":{
"ID":"7b64910a4b7b6b2b90245db1c7a1d8330fb1d0db9237da8c64378c0ad7745e9e",
"Attributes":{
"container":"9..d",
"name":"bridge",
"type":"bridge"
}
},
"time":1481466948,
"timeNano":1481466948357530397
}{
"Type":"network",
"Action":"disconnect",
"Actor":{
"ID":"7b64910a4b7b6b2b90245db1c7a1d8330fb1d0db9237da8c64378c0ad7745e9e",
"Attributes":{
"container":"9..d",
"name":"bridge",
"type":"bridge"
}
},
"time":1481466948,
"timeNano":1481466948357530397
}{
"status":"stop",
"id":"9..d",
"from":"mariadb",
"Type":"container",
"Action":"stop",
"Actor":{
"ID":"9..d",
"Attributes":{
"image":"mariadb",
"name":"mariadb"
}
},
"time":1481466948,
"timeNano":1481466948398153788
}2016-12-11T15:35:48.398153788+01:00{
"status":"stop",
"id":"9..d",
"from":"mariadb",
"Type":"container",
"Action":"stop",
"Actor":{
"ID":"9..d",
"Attributes":{
"image":"mariadb",
"name":"mariadb"
}
},
"time":1481466948,
"timeNano":1481466948398153788
}

This is a simple example for sure ! But it could give you ideas to probably create a home-made event
alerting/monitoring system using simple Bash commands.

Docker Images
If you type

docker images

REPOSITORY
hello-world

TAG
latest

, you will see that you have the hello-world image, even if its container is not running:
IMAGE ID
c54a2cc56cbb

CREATED
6 months ago

SIZE
1.848 kB

An image has a tag (latest), an id (c54a2cc56cbb), a creation date (3 months ago) and a size (1.848 kB).
Images are generally stored in a registry unless it is an image from scratch that you created using the tar method and
didn't push to any of the public or private registires.

Registries are to Docker images what git is to code.

Docker images responds to the following Docker Events:
delete,
import,
load,
pull,
push,
save,
tag,
untag

Docker Containers
Docker is a containerization software that solves many problems in the modern IT like it is said in the introduction
of this book.
A container is basically an image that is running in your host OS. The important thing to remember is that there is a
big difference between containers and VMs and that's why Docker is so darn popular:
Containers use shared operating systems while VMs emulate in a different way the hardware.
The command to create a container is
container creation and its arguments.

docker create

followed by options, image name, command to execute at the

Other commands are used to start, pause, stop, run and delete containers and we are going to see these commands
and more in another chapter.
Keep in mind that a container have a life cycle with different phases that responds to the defined events :
attach
commit
copy
create
destroy
detach
die
exec_create
exec_detach
exec_start
export
health_status
kill
oom
pause
rename
resize
restart
start
stop
top
unpause
update

Docker Volumes
We still using the example of MariaDB:
docker run --name mariadb -e MYSQL_ROOT_PASSWORD=password -v /data/db:/var/lib/mysql -d mariadb

In the docker run command we are using to create a dockerized database, we are using the -v flag to mount a
volume. The mounted volume inside the container is pointing on /var/lib/mysql which is the Mysql data directory
where we can find databases data and other Mysql data.
Without logging inside the container, you can see the data directory mapped to the host in the
ls -l /data/db/
total 110632
-rw-rw---- 1 999
-rw-rw---- 1 999
-rw-rw---- 1 999
-rw-rw---- 1 999
-rw-rw---- 1 999
drwx------ 2 999
drwx------ 2 999

/data/db

directory:

999
16384 Dec 11 21:27 aria_log.00000001
999
52 Dec 11 21:27 aria_log_control
999 12582912 Dec 11 21:27 ibdata1
999 50331648 Dec 11 21:27 ib_logfile0
999 50331648 Dec 11 21:27 ib_logfile1
999
4096 Dec 11 21:27 mysql
999
4096 Dec 11 21:27 performance_schema

You can create the same volume without mapping it to the host filesystem:
docker run --name mariadb -e MYSQL_ROOT_PASSWORD=password -v /var/lib/mysql -d mariadb

In this way you can't access the files of your MariaDB databases from the host machine.
Nope ! In reality, you can. Docker volumes can be found in:
/var/lib/docker/volumes/

You can see a list of volumes identifiers:
ls -lrth /var/lib/docker/volumes/
3..e
3..7
5..0
b..2
b..7
metadata.db

One of the above identifiers is the volume that was created using the command:
docker run --name mariadb -e MYSQL_ROOT_PASSWORD=password -v /var/lib/mysql -d mariadb

Docker has a helpful command to inspect a container called

docker inspect

.

We are going to see it in details later but we will use it in order to identify the Docker volumes attached to our
running container.
docker inspect -f '{{ (index .Mounts 0).Source }}' mariadb

It should output one of the ids above:
/var/lib/docker/volumes/b..2/_data

The Docker inspect command lets you have several information about a container (or an image). It returns generally
a json containing all of the information but using the -f or the --format flag, you can filter an attribute. docker inspect
-f '{{ (index .Mounts 0).Source }}' mariadb will shows the first element of a json dictionary referenced by its name
Mounts.
This is the part of the json output containing the information about the mounts:
"Mounts": [
{
"Name": "b..2",
"Source": "/var/lib/docker/volumes/b..2/_data",
"Destination": "/var/lib/mysql",
"Driver": "local",
"Mode": "",
"RW": true,
"Propagation": ""
}
],

Now, you can understand why the used format
/var/lib/docker/volumes/b..2/_data .

'{{

(index

.Mounts

0).Source

}}'

will shows

Keep in mind that the host directory /var/lib/docker/volumes/b..2/_data depends on the host that's why creating
a volume using Dockerfile does not allow mounting a volume on the host filesystem.

Data Volumes
Data volumes are different from the previous volumes created to host MariaDB databases. They are separate
containers, dedicated to storing data, they are persistent, they can be shared between containers, they can be used as
a backup, a restore point or to used in data migration.
For the same container (MariaDB), we are going to create a volumes in a separate container:
docker run --name mariadb_storage -v /var/lib/mysql -it -d busybox

Now let's run the MariaDB container without creating a new volumes but using the previously created data
container:
docker run --name mariadb -e MYSQL_ROOT_PASSWORD=password --volumes-from mariadb_storage -d mariadb

Notice the usage of
container.

--volumes-from mariadb_storage

that will attach the data container as a volume to the MariaDB

In the same way, we can inspect the data container:
docker inspect -f '{{ (index .Mounts 0).Source }}' mariadb_storage

Say we have this as an output:

/var/lib/docker/volumes/f..9/_data

This means that the latter directory will contains all of our databases files:
You can check it by typing:
ls -l /var/lib/docker/volumes/f..9/_data
total 110656
-rw-rw---- 1
-rw-rw---- 1
-rw-rw---- 1
-rw-rw---- 1
-rw-rw---- 1
-rw-rw---- 1
drwx------ 2
drwx------ 2
-rw-rw---- 1

999
999
999
999
999
999
999
999
999

999
16384 Dec 11 22:23 aria_log.00000001
999
52 Dec 11 22:23 aria_log_control
999 12582912 Dec 11 22:23 ibdata1
999 50331648 Dec 11 22:23 ib_logfile0
999 50331648 Dec 11 22:23 ib_logfile1
999
0 Dec 11 22:23 multi-master.info
999
4096 Dec 11 22:23 mysql
999
4096 Dec 11 22:23 performance_schema
999
24576 Dec 11 22:23 tc.log

Now that we have a separate volume container, we can create 3 or more MariaDB containers using the same
volume:
docker run \
--name mariadb1 \
-e MYSQL_ROOT_PASSWORD=password \
--volumes-from mariadb_storage \
-d mariadb \
&& \
docker run \
--name mariadb2 \
-e MYSQL_ROOT_PASSWORD=password \
--volumes-from mariadb_storage \
-d mariadb \
&& \
docker run \
--name mariadb3 \
-e MYSQL_ROOT_PASSWORD=password \
--volumes-from mariadb_storage \
-d mariadb
f0ff0622ea755c266e069a42a2a627380042c8f1a3464521dd0fb16ef3257534
0c10ea5ec23a3057de30dc4f97e485349ac7b0a335a98e1c3deece56f3594964
128e66ffa823e3156fb28b48f45e3234d235949508780f1c1cef8c38ed5bfb0b

Now, you can see all of our 4 containers:
docker ps
CONTAINER ID
049824509c18
128e66ffa823
0c10ea5ec23a
03808f86ba9b

IMAGE
mariadb
mariadb
mariadb
busybox

COMMAND
PORTS
NAMES
"docker-entrypoint.sh" 3306/tcp mariadb1
"docker-entrypoint.sh" 3306/tcp mariadb3
"docker-entrypoint.sh" 3306/tcp mariadb2
"sh"
mariadb_storage

Sharing a data volume between three MariaDB Docker instances in this example is just for testing and
learning purpose, it is not production-ready.
We can in fact manage to have multiple containers sharing same volumes. However, the fact that it is possible to do,
does not simply means that it is safe to do it. Multiple containers writing to a single shared volume may cause data
corruption or a high level application problem like consumer/producer problems.
Now you can type again

docker ps

and I am pretty sure that only one MariaDB will be running, the others will stop.

If you want more explanation about this, you can find MariaDB logs of the stopped container and you will notice
that mysqld could not get an exclusive lock and that the latter could be used by another process and that's what
actually happening.

[ERROR] mysqld: Got error 'Could not get an exclusive lock; file is probably in use by another process' when trying to use aria control file '/va

Other logs are showing the same problem:
2016-12-11 22:02:25 140419388524480 [ERROR] InnoDB: Can't open './ibdata1'
2016-12-11 22:02:25 140419388524480 [ERROR] InnoDB: Could not open or create
the system tablespace. If you tried to add new data files to the system tablespace,
and it failed here, you should now edit innodb_data_file_path in my.cnf back to what it was,
and remove the new ibdata files InnoDB created in this failed attempt.
InnoDB only wrote those files full of zeros, but did not yet use them in any way.
But be careful: do not remove old data files which contain your precious data!
2016-12-11 22:02:25 140419388524480 [ERROR] Plugin 'InnoDB' init function returned error.
2016-12-11 22:02:25 140419388524480 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2016-12-11 22:02:25 140419388524480 [ERROR] mysqld: Can't lock aria control file
'/var/lib/mysql/aria_log_control' for exclusive use, error: 11. Will retry for 30 seconds
2016-12-11 22:02:56 140419388524480 [ERROR] mysqld: Got error 'Could not get an exclusive lock;
file is probably in use by another process' when trying to use aria control file '/var/lib/mysql/aria_log_control'
2016-12-11 22:02:56 140419388524480 [ERROR] Plugin 'Aria' init function returned error.
2016-12-11 22:02:56 140419388524480 [ERROR] Plugin 'Aria' registration as a STORAGE ENGINE failed.
2016-12-11 22:02:56 140419388524480 [Note] Plugin 'FEEDBACK' is disabled.
2016-12-11 22:02:56 140419388524480 [ERROR] Unknown/unsupported storage engine: InnoDB
2016-12-11 22:02:56 140419388524480 [ERROR] Aborting

Make sure your applications are designed to write to shared data containers.
Executing

will remove the container from the host but if you type
ls -l
you can still find your data on the host machine and you can check that the Docker
volume container is always running independently from removed container.
docker

rm

-f

mariadb

/var/lib/docker/volumes/f..9/_data

Cleaning Docker Dangling Containers
You can find yourself in the situation where some volumes are not used (or referenced) by any container, we call
this type of volumes "dangling volumes" or "orphaned". To see your orphaned volumes, type:
docker volume ls -qf dangling=true
3..e
3..7
5..0
b..2
b..7

The fastest way to cleanup your host from these volumes is to run:
docker volume ls -qf dangling=true | xargs -r docker volume rm

Docker Volumes Events
Docker volumes report the following events: create, mount, unmount, destroy.

Docker Networks
Like a VM, a container can be part of one or many networks.
You may have noticed - just after the installation of Docker - that you have a new network interface on your host
machine:
ifconfig

docker0

Link encap:Ethernet HWaddr 02:42:ef:e0:98:84
inet addr:172.17.0.1 Bcast:0.0.0.0 Mask:255.255.0.0
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:5 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:356 (356.0 B) TX bytes:0 (0.0 B)

And if you type
NETWORK ID
2aaea08714ba
608a539ce1e3
ede46dbb22d7

docker network ls

NAME
bridge
host
none

you can see the list of the default 3 networks.

DRIVER
SCOPE
bridge
local
host
local
null
local

Docker Networks Types
When you typed
NETWORK ID
2aaea08714ba
608a539ce1e3
ede46dbb22d7

docker network ls

NAME
bridge
host
none

you had 3 default networks implemented in Docker:

DRIVER
SCOPE
bridge
local
host
local
null
local

bridge (driver: bridge)
none (driver: null)
host (driver: host)

Bridge Networks
The bridge network is one of the host network interfaces (docker0). It is what you see when you type
All Docker containers are connected to this network unless you add --network flag to the container.
Let's verify this by creating a Docker container and inspecting the default network (bridge):
docker run --name mariadb_storage -v /var/lib/mysql -it -d busybox

Let's inspect the bridge network:

ifconfig

.

docker network inspect bridge

[
{
"Name": "bridge",
"Id": "2aaea08714ba2e3927334e8d0044afeb85f68ec0a0624ae7ab1f81d976b49292",
"Scope": "local",
"Driver": "bridge",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "172.17.0.0/16",
"Gateway": "172.17.0.1"
}
]
},
"Internal": false,
"Containers": {
"3c9afd0c72eb4738750ad1b83d38587a1c47636429e5f67c5deec5ec5dfa3210": {
"Name": "mariadb_storage",
"EndpointID": "ed47ef978ed78c5677c81a11d439d0af19a54f0620387794db1061807e39c2b7",
"MacAddress": "02:42:ac:11:00:02",
"IPv4Address": "172.17.0.2/16",
"IPv6Address": ""
}
},
"Options": {
"com.docker.network.bridge.default_bridge": "true",
"com.docker.network.bridge.enable_icc": "true",
"com.docker.network.bridge.enable_ip_masquerade": "true",
"com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
"com.docker.network.bridge.name": "docker0",
"com.docker.network.driver.mtu": "1500"
},
"Labels": {}
}
]

We can see that the mariadb_storage data container is living inside the bridge network.
Now, if you create a new network db_network:
docker network create db_network

Then create a Docker container attached to the same network (db_network):
docker run --name mariadb -e MYSQL_ROOT_PASSWORD=password --network db_network -d mariadb

If you inspect the default bridge network, you will notice that there is no containers attached to it or at least the
newly created container is not attached to it:
docker network inspect bridge
[
{
"Name": "bridge",
"Id": "2aaea08714ba2e3927334e8d0044afeb85f68ec0a0624ae7ab1f81d976b49292",
"Scope": "local",
"Driver": "bridge",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "172.17.0.0/16",
"Gateway": "172.17.0.1"
}
]
},
"Internal": false,
"Containers": {},

"Options": {
"com.docker.network.bridge.default_bridge": "true",
"com.docker.network.bridge.enable_icc": "true",
"com.docker.network.bridge.enable_ip_masquerade": "true",
"com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
"com.docker.network.bridge.name": "docker0",
"com.docker.network.driver.mtu": "1500"
},
"Labels": {}
}
]

And if you inspect the new network, you can see that mariadb container is running in this network (db_network).
docker network inspect db_network
[
{
"Name": "db_network",
"Id": "ec809d635d4ad22f852a2419032812cb7c9361e8bb0111815dc79a34fee30668",
"Scope": "local",
"Driver": "bridge",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": {},
"Config": [
{
"Subnet": "172.18.0.0/16",
"Gateway": "172.18.0.1/16"
}
]
},
"Internal": false,
"Containers": {
"3ddfbc0f03f4a574480d842330add4169f834cfcde96e52956ee34164629dd52": {
"Name": "mariadb",
"EndpointID": "0efca766ec11f7442399a4c81b6e79db825eeb81736db1760820fe61f24b9805",
"MacAddress": "02:42:ac:12:00:02",
"IPv4Address": "172.18.0.2/16",
"IPv6Address": ""
}
},
"Options": {},
"Labels": {}
}
]

This network has 172.18.0.0/16 as a subnet and the container address is 172.18.0.2. In all cases, if you create a
network without specifying its type, it will have bridge as a default type.

Overlay Networks
Generally, overlay network is a network that is built on top of another network. For your information, VoIP, VPN,
CDNs are overlay networks.
Docker uses also overlay networks, precisely in Swarm mode where you can create this type of networks on a
manager node. While you can do this to managed containers (Swarm mode or using an external key-value store),
unmanaged containers can not be part of an overlay network.
Docker overlay networks allows creating a multi-host network.
To create a network called database_network
docker network create --driver overlay --subnet 10.0.9.0/32 database_network

This will shows an error message, telling you that there is an error response from daemon that failed to allocate
gateway.
Since overlay networks will only work in Swarm mode, you should run your Docker engine in this mode before
creating the network:

docker swarm init --advertise-addr 127.0.0.1

Swarm mode is probably the easiest way to create and manage overlay networks, but you can use external
components like Consul, Etcd or ZooKeeper because overlay networks require a valid key-value store service.

Using Swarm Mode
Docker Swarm is the mode where you are using the built-in container orchestrator implemented in Docker engine
since its version 1.12.
We are going to see in detail Docker Swarm mode and Docker orchestration but for the networking part, it is
important to define briefly the orchestration.
Container Orchestration allows users to coordinate containers in the cloud running a multi-container environment.
Say you are deploying a web application, with an orchestrator you can define a service for your web app, deploy its
container and scale it to 10 or 20 instance, create a network and attach your different containers to it and consider all
of the web application containers as single entity from an deploying, availability, scaling and networking point of
view.
To start working with Swarm mode you need at least the version 1.12. Initialize the Swarm cluster and don't forget
to put your machine IP instead of 192.168.0.47 :
docker swarm init --advertise-addr 192.168.0.47

You will notice an information message similar to the following one:
Swarm initialized: current node (0kgtmbz5flwv4te3cxewdi1u5) is now a manager.
To add a worker to this swarm, run the following command:
docker swarm join \
--token SWMTKN-1-1re2072eii1walbueysk8yk14cqnmgltnf4vyuhmjli2od2quk-8qc8iewqzmlvp1igp63xyftua \
192.168.0.47:2377
To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.

In this part of the book, since we are focusing on networking rather than the Swarm mode, we are going to use a
single node.
Now that the Swarm mode is activated, we can create a new network that we call webapps_network:
docker network create --driver overlay --subnet 10.0.9.0/16 --opt encrypted webapps_network

Let's check if our network using
NETWORK ID
1wdbz5gldym1

docker network ls

NAME
webapp_network

DRIVER
overlay

:
SCOPE
swarm

If you inspect the network by typing docker inspect webapp_network you will be able to see different information about
webapp_network like its name, its id, its IPv6 status, the containers using it, some configurations like the subnet, the
gateway and the scope which is swarm.
[
{
"Name": "webapp_network",
"Id": "1wdbz5gldym121beq4ftdttgg",
"Scope": "swarm",

"Driver": "overlay",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "10.0.9.0/16",
"Gateway": "10.0.0.1"
}
]
},
"Internal": false,
"Containers": null,
"Options": {
"com.docker.network.driver.overlay.vxlanid_list": "257",
"encrypted": ""
},
"Labels": null
}
]

After learning how to use and manage Docker Swarm mode you will be able to create services and containers and
attach them to a network but for the moment let's just continue exploring briefly the networking part.
In our example, we are considering that the service webapp_service was created and that we are running 3 instances
of webapp_container.
If we type the same command to inspect we will notice that the 3 Docker instances are included now:
[
{
"Name": "webapp_network",
"Id": "2tb6djzmq4x4u5ey3h2e74y9e",
"Scope": "swarm",
"Driver": "overlay",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "10.0.9.0/16",
"Gateway": "10.0.0.2"
}
]
},
"Internal": false,
"Containers": {
"ab364a18eb272533e6bda88a0c705004df4b7974e64ad63c36fd3061a716f613": {
"Name": "webapp.3.3fmrpa3cd7y6b4ol33x7m413v",
"EndpointID": "e6980c952db5aa7b6995bc853d5ab82668a4692af98540feef1aa58a4e8fd282",
"MacAddress": "02:42:0a:00:00:06",
"IPv4Address": "10.0.0.6/16",
"IPv6Address": ""
},
"c5628fcfcf6a5f0e5a49ddfe20b141591c95f01c45a37262ee93e8ad3e3cff7e": {
"Name": "webapp.2.dckgxiffiay2kk7t2n149rpnj",
"EndpointID": "ce05566b0110f448018ccdaed4aa21b402ae659efa2adff3102492ee6c04fce8",
"MacAddress": "02:42:0a:00:00:05",
"IPv4Address": "10.0.0.5/16",
"IPv6Address": ""
},
"f0a7b5201e84c010616e6033fee7d245af7063922c61941cfc01f9edd0be7226": {
"Name": "webapp.1.73m1914qlcj8agxf5lkhzb7se",
"EndpointID": "1dce8dd253b818f8b0ba129a59328fed427da91d86126a1267ac7855666ec434",
"MacAddress": "02:42:0a:00:00:04",
"IPv4Address": "10.0.0.4/16",
"IPv6Address": ""
}
},
"Options": {
"com.docker.network.driver.overlay.vxlanid_list": "258",
"encrypted": ""
},
"Labels": {}
}
]

Let's make some experiments to better discover networking in Swarm mode.
If we execute the Linux command

route

to see the default

If we execute

traceroute google.com

from inside one of the containers, this is a sample output:

traceroute to google.com (216.58.209.238), 30 hops max, 46 byte packets
1 172.19.0.1 (172.19.0.1) 0.003 ms 0.012 ms 0.004 ms
2 192.168.3.1 (192.168.3.1) 2.535 ms 0.481 ms 0.609 ms
3 192.168.1.1 (192.168.1.1) 2.228 ms 1.442 ms 1.826 ms
4 80.10.234.169 (80.10.234.169) 10.377 ms 5.951 ms 17.026 ms
[...]
11 par10s29-in-f14.1e100.net (216.58.209.238) 4.723 ms 3.428 ms 4.191 ms

To go outside and reach the Internet, the first hop was 172.19.0.1 . If you type ifconfig in your host machine, you
can find that this address is allocated to docker_gwbridge interface that we are going to see later in this section.
docker_gwbridge Link encap:Ethernet HWaddr 02:42:d2:fd:fd:dc
inet addr:172.19.0.1 Bcast:0.0.0.0 Mask:255.255.0.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:7334 errors:0 dropped:0 overruns:0 frame:0
TX packets:3843 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:595520 (595.5 KB) TX bytes:358648 (358.6 KB)

The containers provisioned in docker swarm mode can be accessed in service discovery in two ways:
Via Virtual IP (VIP) and routed through the docker swarm ingress overlay network.
Or via a DNS round robbin (DNSRR)
Because VIP is not on the ingress overlay network, you need to create a user-defined overlay network in order to use
VIP or DNSRR.
Natively, Docker Engine in its Swarm mode supports overlay networks so you don't need an external key-value store
and container-to-container networking is enabled.
The eth0 interface represents the container interface that is connected to the overlay network. So if you create an
overlay network, you will have those VIP associated to it. eth1 interface represents the container interface that is
connected to the docker_gwbridge network, for external connectivity outside of container cluster.

Using External Key/Value Store
Our choice for this part will be Consul, a service discovery and configuration manager for distributed systems
implementing a key/value storage.
To use Consul 0.7.1, you should download it:
cd /tmp
wget https://releases.hashicorp.com/consul/0.7.1/consul_0.7.1_linux_amd64.zip

unzip consul_0.7.1_linux_amd64.zip

chmod +x consul
mv consul /usr/bin/consul

Create the configuration directory for Consul:
mkdir -p /etc/consul.d/

If you want to install the the web UI download it and unzip:
cd /tmp
wget https://releases.hashicorp.com/consul/0.7.1/consul_0.7.1_web_ui.zip
mkdir -p /opt/consul
cd /opt/consul
unzip /tmp/consul_0.6.0_web_ui.zip

Now you can run a Consul agent:
consul agent \
-server \
-bootstrap-expect 1 \
-data-dir /tmp/consul \
-node=agent-one \
-bind=192.168.0.47 \
-client=0.0.0.0 \
-config-dir /etc/consul.d \
-ui-dir /opt/consul-ui

Change the options values by your own ones, where:
-bootstrap-expect=0
-bind=0.0.0.0
-client=127.0.0.1
-node=hostname
-config-dir=foo

-ui-dir=path

Sets server to expect bootstrap mode.
Sets the bind address for cluster communication
Sets the address to bind for client access.
This includes RPC, DNS, HTTP and HTTPS (if configured)
Name of this node. Must be unique in the cluster
Path to a directory to read configuration files
from. This will read every file ending in ".json"
as configuration in this directory in alphabetical
order. This can be specified multiple times.
Path to directory containing the Web UI resources

You can use other options like:
-bootstrap
-advertise=addr
-atlas=org/name
-atlas-join
-atlas-token=token
-atlas-endpoint=1.2.3.4
-http-port=8500
-config-file=foo
-data-dir=path
-recursor=1.2.3.4
-dc=east-aws
-encrypt=key
-join=1.2.3.4
-join-wan=1.2.3.4
-retry-join=1.2.3.4
-retry-interval=30s
-retry-max=0
-retry-join-wan=1.2.3.4
-retry-interval-wan=30s
-retry-max-wan=0
-log-level=info
-protocol=N
-rejoin
-server
-syslog
-pid-file=path

Sets server to bootstrap mode
Sets the advertise address to use
Sets the Atlas infrastructure name, enables SCADA.
Enables auto-joining the Atlas cluster
Provides the Atlas API token
The address of the endpoint for Atlas integration.
Sets the HTTP API port to listen on
Path to a JSON file to read configuration from.
This can be specified multiple times.
Path to a data directory to store agent state
Address of an upstream DNS server.
Can be specified multiple times.
Datacenter of the agent
Provides the gossip encryption key
Address of an agent to join at start time.
Can be specified multiple times.
Address of an agent to join -wan at start time.
Can be specified multiple times.
Address of an agent to join at start time with
retries enabled. Can be specified multiple times.
Time to wait between join attempts.
Maximum number of join attempts. Defaults to 0, which
will retry indefinitely.
Address of an agent to join -wan at start time with
retries enabled. Can be specified multiple times.
Time to wait between join -wan attempts.
Maximum number of join -wan attempts. Defaults to 0, which
will retry indefinitely.
Log level of the agent.
Sets the protocol version. Defaults to latest.
Ignores a previous leave and attempts to rejoin the cluster.
Switches agent to server mode.
Enables logging to syslog
Path to file to store agent PID

You can see the execution logs on your terminal:

==>
==>
==>
==>
==>

WARNING: BootstrapExpect Mode is specified as 1; this is the same as Bootstrap mode.
WARNING: Bootstrap mode enabled! Do not enable unless necessary
Starting Consul agent...
Starting Consul agent RPC...
Consul agent running!
Node name: 'agent-one'
Datacenter: 'dc1'
Server: true (bootstrap: true)
Client Addr: 0.0.0.0 (HTTP: 8500, HTTPS: -1, DNS: 8600, RPC: 8400)
Cluster Addr: 192.168.0.47 (LAN: 8301, WAN: 8302)
Gossip encrypt: false, RPC-TLS: false, TLS-Incoming: false
Atlas: 

==> Log data will now stream in as it occurs:
2016/12/17
2016/12/17
2016/12/17
2016/12/17
2016/12/17
2016/12/17
2016/12/17
2016/12/17
2016/12/17
2016/12/17
2016/12/17
2016/12/17
2016/12/17
2016/12/17
2016/12/17

18:50:55
18:50:55
18:50:55
18:50:55
18:50:55
18:50:55
18:50:56
18:50:56
18:50:56
18:50:56
18:50:56
18:50:56
18:50:56
18:50:56
18:50:56

[INFO] raft: Node at 192.168.0.47:8300 [Follower] entering Follower state
[INFO] serf: EventMemberJoin: agent-one 192.168.0.47
[INFO] consul: adding LAN server agent-one (Addr: 192.168.0.47:8300) (DC: dc1)
[INFO] serf: EventMemberJoin: agent-one.dc1 192.168.0.47
[ERR] agent: failed to sync remote state: No cluster leader
[INFO] consul: adding WAN server agent-one.dc1 (Addr: 192.168.0.47:8300) (DC: dc1)
[WARN] raft: Heartbeat timeout reached, starting election
[INFO] raft: Node at 192.168.0.47:8300 [Candidate] entering Candidate state
[INFO] raft: Election won. Tally: 1
[INFO] raft: Node at 192.168.0.47:8300 [Leader] entering Leader state
[INFO] consul: cluster leadership acquired
[INFO] consul: New leader elected: agent-one
[INFO] raft: Disabling EnableSingleNode (bootstrap)
[INFO] consul: member 'agent-one' joined, marking health alive
[INFO] agent: Synced service 'consul'

You can check Consul nodes by typing
Node
agent-one

Address
192.168.0.47:8301

Status
alive

consul members

Type
server

Build
0.6.0

Now go to your Docker configuration file:
configurations:

:

Protocol
2

DC
dc1

/etc/default/docker

and add the following line with the good

DOCKER_OPTS="--cluster-store=consul://192.168.0.47:8500 --cluster-advertise=192.168.0.47:4000"

Restart Docker:
service docker restart

And create an overlay network:
docker network create --driver overlay --subnet=10.0.1.0/24 databases_network

Check if the network was created:
docker network ls

NETWORK ID
6d6484687002

NAME
databases_network

DRIVER
overlay

SCOPE
global

If you inspect the recently created network using
similar to the following output:
[
{

docker network inspect databases_network

, you will have something

"Name": "databases_network",
"Id": "6d64846870029d114bb8638f492af7fc9b5c87c2facb67890b0f40187b73ac87",
"Scope": "global",
"Driver": "overlay",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": {},
"Config": [
{
"Subnet": "10.0.1.0/24"
}
]
},
"Internal": false,
"Containers": {},
"Options": {},
"Labels": {}
}
]

Notice that the network Scope is now global :
"Scope": "global",

If you compare it to the other network created using the Swarm mode, you will notice that the scope changed from
swarm to global.

Docker Networks Events
Docker networks system reports the following events: create, connect, disconnect, destroy.

Docker Daemon & Architecture
Docker Daemon
Like the init daemon, cron daemon crond, dhcp daemon dhcpd, Docker has its own daemon dockerd.
To list Docker daemons, list all Linux daemons:
ps -U0 -o 'tty,pid,comm' | grep ^?

And grep Docker on the output:
ps -U0 -o 'tty,pid,comm' | grep ^?|grep -i dockerd
?
2779 dockerd

Note that you may see

docker-containe

or any other short version of

If you are already running Docker, when you type

dockerd

docker-containerd-shim

.

you will have a similar error message to this :

FATA[0000] Error starting daemon: pid file found, ensure docker is not running or delete /var/run/docker.pid

Now let's stop Docker service docker stop and run is daemon directly using dockerd command. Running the Docker
daemon command using dockerd is a good debugging tool, as you may see, you will have the running traces right on
your terminal screen:
INFO[0000] libcontainerd: new containerd process, pid: 19717
WARN[0000] containerd: low RLIMIT_NOFILE changing to max current=1024 max=4096
INFO[0001] [graphdriver] using prior storage driver "aufs"
INFO[0003] Graph migration to content-addressability took 0.63 seconds
WARN[0003] Your kernel does not support swap memory limit.
WARN[0003] mountpoint for pids not found
INFO[0003] Loading containers: start.
INFO[0003] Firewalld running: false
INFO[0004] Removing stale sandbox ingress_sbox (ingress-sbox)
INFO[0004] Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. \
Daemon option --bip can be used to set a preferred IP address
INFO[0004] Loading containers: done.
INFO[0004] Listening for local connections addr=/var/lib/docker/swarm/control.sock proto=unix
INFO[0004] Listening for connections
addr=[::]:2377 proto=tcp
INFO[0004] 61c88d41fce85c57 became follower at term 12
INFO[0004] newRaft 61c88d41fce85c57 [peers: [], term: 12, commit: 290, applied: 0, lastindex: 290, lastterm: 12]
INFO[0004] 61c88d41fce85c57 is starting a new election at term 12
INFO[0004] 61c88d41fce85c57 became candidate at term 13
INFO[0004] 61c88d41fce85c57 received vote from 61c88d41fce85c57 at term 13
INFO[0004] 61c88d41fce85c57 became leader at term 13
INFO[0004] raft.node: 61c88d41fce85c57 elected leader 61c88d41fce85c57 at term 13
INFO[0004] Initializing Libnetwork Agent Listen-Addr=0.0.0.0 Local-addr=192.168.0.47 Adv-addr=192.168.0.47 Remote-addr =
INFO[0004] Daemon has completed initialization
INFO[0004] Initializing Libnetwork Agent Listen-Addr=0.0.0.0 Local-addr=192.168.0.47 Adv-addr=192.168.0.47 Remote-addr =
INFO[0004] Docker daemon
commit=7392c3b graphdriver=aufs version=1.12.5
INFO[0004] Gossip cluster hostname eonSpider-3e64aecb2dd5
INFO[0004] API listen on /var/run/docker.sock
INFO[0004] No non-localhost DNS nameservers are left in resolv.conf. Using default external servers :
[nameserver 8.8.8.8 nameserver 8.8.4.4]
INFO[0004] IPv6 enabled; Adding default IPv6 external servers :
[nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844]
INFO[0000] Firewalld running: false

Now if you create or remove containers for example, you will see that Docker daemon connects you (Docker client)

to Docker containers.
This is how Docker daemon communicate with the rest of Docker modules:

Containerd
Containerd is one of the recent projects in the Docker ecosystem and its purpose is breaking up more modularity to
Docker architecture and more neutrality visà-vis the other industry actors (Cloud providers and other orchestrator
services).
According to Solomon Hykes, containerd is already deployed on millions of machines since April 2016 when it was
included in Docker 1.11. The announced roadmap to extend containerd get its input from the cloud providers and
actors like Alibaba Cloud, AWS, Google, IBM, Microsoft, and other active members of the container ecosystem.
More Docker engine functionality will be added to containerd so that containerd 1.0 will provide all the core
primitives you need to manage containers with parity on Linux and Windows hosts:
Container execution and supervision
Image distribution
Network Interfaces Management
Local storage
Native plumbing level API
Full OCI support, including the extended OCI image specification
To build, ship and run containerized applications, you may continue to use Docker but if you are looking for
specialized components you could consider containerd.
Docker Engine 1.11 was the first release built on runC (a runtime based on Open Container Intiative technology)
and containerd.

Formed in June 2015, the Open Container Initiative (OCI) aims to establish common standards for software
containers in order to avoid a potential fragmentation and divisions inside the container ecosystem.
It contains two specifications:
runtime-spec: The runtime specification
image-spec: The image specification
The runtime specification outlines how to run a filesystem bundle that is unpacked on disk:
A standardized container bundle should contain the needed information and configurations to load and run a
container in a config.json file residing in the root of the bundle directory.
A standardized container bundle should contain a directory representing the root filesystem of the container.
Generally this directory has a conventional name like rootfs.
You can see the json file if you export and extract an image. In the following example, we are going to use busybox
image.
mkdir my_container
cd my_container
mkdir rootfs
docker export $(docker create busybox) | tar -C rootfs -xvf -

Now we have an extracted busybox image inside of rootfs directory.
tree -d my_container/
my_container/
└── rootfs
├── bin
├── dev
│
├── pts
│
└── shm
├── etc
├── home
├── proc
├── root
├── sys
├── tmp
├── usr
│
└── sbin
└── var
├── spool
│
└── mail
└── www

We can generate the config.json file:
docker-runc spec

You could also use runC to generate your json file:
runc spec

This is the generated configuration file (config.json):
{
"ociVersion": "1.0.0-rc2-dev",
"platform": {
"os": "linux",
"arch": "amd64"
},

"process": {
"terminal": true,
"consoleSize": {
"height": 0,
"width": 0
},
"user": {
"uid": 0,
"gid": 0
},
"args": [
"sh"
],
"env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"TERM=xterm"
],
"cwd": "/",
"capabilities": [
"CAP_AUDIT_WRITE",
"CAP_KILL",
"CAP_NET_BIND_SERVICE"
],
"rlimits": [
{
"type": "RLIMIT_NOFILE",
"hard": 1024,
"soft": 1024
}
],
"noNewPrivileges": true
},
"root": {
"path": "rootfs",
"readonly": true
},
"hostname": "runc",
"mounts": [
{
"destination": "/proc",
"type": "proc",
"source": "proc"
},
{
"destination": "/dev",
"type": "tmpfs",
"source": "tmpfs",
"options": [
"nosuid",
"strictatime",
"mode=755",
"size=65536k"
]
},
{
"destination": "/dev/pts",
"type": "devpts",
"source": "devpts",
"options": [
"nosuid",
"noexec",
"newinstance",
"ptmxmode=0666",
"mode=0620",
"gid=5"
]
},
{
"destination": "/dev/shm",
"type": "tmpfs",
"source": "shm",
"options": [
"nosuid",
"noexec",
"nodev",
"mode=1777",
"size=65536k"
]
},
{
"destination": "/dev/mqueue",
"type": "mqueue",
"source": "mqueue",
"options": [
"nosuid",
"noexec",
"nodev"
]
},
{
"destination": "/sys",
"type": "sysfs",

"source": "sysfs",
"options": [
"nosuid",
"noexec",
"nodev",
"ro"
]
},
{
"destination": "/sys/fs/cgroup",
"type": "cgroup",
"source": "cgroup",
"options": [
"nosuid",
"noexec",
"nodev",
"relatime",
"ro"
]
}
],
"hooks": {},
"linux": {
"resources": {
"devices": [
{
"allow": false,
"access": "rwm"
}
]
},
"namespaces": [
{
"type": "pid"
},
{
"type": "network"
},
{
"type": "ipc"
},
{
"type": "uts"
},
{
"type": "mount"
}
],
"maskedPaths": [
"/proc/kcore",
"/proc/latency_stats",
"/proc/timer_list",
"/proc/timer_stats",
"/proc/sched_debug",
"/sys/firmware"
],
"readonlyPaths": [
"/proc/asound",
"/proc/bus",
"/proc/fs",
"/proc/irq",
"/proc/sys",
"/proc/sysrq-trigger"
]
}
}

Now you can edit any of the configurations listed above and run again a container without even using Docker, just
runC:
runc run container-name

Note that you should install runC first in order to use it.
also install it from sources:
mkdir -p ~/golang/src/github.com/opencontainers/
cd ~/golang/src/github.com/opencontainers/
git clone https://github.com/opencontainers/runc
cd ./runc
make

sudo apt install runc

for Ubuntu 16.04. You could

sudo make install

runC, a standalone containers runtime, is at its full spec, it allows you to spin containers, interact with them, and
manage their lifecycle and that's why containers built with one engine (like Docker) can run on another engine.
Containers are started as a child process of runC and can be embedded into various other systems without having to
run a daemon (Docker Daemon).
runC is built on libcontainer which is the same container library powering a Docker engine installation. Prior to the
version 1.11, Docker engine was used to manage volumes, networks, containers, images etc.. Now, the Docker
architecture is broken into four components: Docker engine, containerd, containerd-shm and runC. The binaries are
respectively called docker, docker-containerd, docker-containerd-shim, and docker-runc.
To run a container, Docker engine creates the image, pass it to containerd. containerd calls containerd-shim that
uses runC to run the container. Then, containerd-shim allows the runtimes (runC in this case) to exit after it starts
the container : This way we can run daemon-less containers because we are not having to have the long running
runtime processes for containers.

Currently, the creation of a container is handled by runc (via containerd) but it is possible to use another
binary (instead of runC) that expose the same command line interface of Docker and accepting an OCI bundle.
You can see the different runtimes that you have on your host by typing:
docker info|grep -i runtime

Since I am using the default runtime, this is what I should get as an output:
Runtimes: runc
Default Runtime: runc

To add another runtime, you should follow this command:
docker daemon --add-runtime "="

Example:
docker daemon --add-runtime "oci=/usr/local/sbin/runc"

There is only one containerd-shim by process and it manages the STDIO FIFO and keeps it open for the container in
case containerd or Docker dies.
It is also in charge of reporting the container's exit status to a higher level like Docker.

Container runtime, lifecycle support and the execution (create, start, stop, pause, resume, exec, signal & delete) are
some features implemented in Containerd. Some others are managed by other components of Docker (volumes,
logging ..etc). Here is a table from the Containerd Github repository that lists the different features and tell if they
are in or out of scope.
Name

Description
Provide an extensible
execution execution layer for executing a
container
Built in functionality for
cow
overlay, aufs, and other copy
filesystem
on write filesystems for
containers
Having the ability to push and
pull images as well as
distribution
operations on images as a first
class api object
Providing network
low-level
functionality to containers
networking
along with configuring their
drivers
network namespaces

In/Out
in

Reason
Create,start, stop pause, resume exec, signal, delete

in

in

containerd will fully support the management and
retrieval of images

in

Network support will be added via interface and network
namespace operations, not service discovery and service
abstractions.

build

Building images as a first
class API

out

volumes

Volume management for
external data

out

logging

Persisting container logs

out

Build is a higher level tooling feature and can be
implemented in many different ways on top of
containerd
The api supports mounts, binds, etc where all volumes
type systems can be built on top of.
Logging can be build on top of containerd because the
container’s STDIO will be provided to the clients and they
can persist any way they see fit. There is no io copying of
container STDIO in containerd.

If we run a container:
docker run --name mariadb -e MYSQL_ROOT_PASSWORD=password -v /data/lists:/var/lib/mysql -d mariadb
Unable to find image 'mariadb:latest' locally
latest: Pulling from library/mariadb
75a822cd7888:
b8d5846e536a:
b75e9152a170:
832e6b030496:
034e06b5514d:
374292b6cca5:
d2a2cf5c3400:
f75e0958527b:
1826247c7258:
68b5724d9fdd:
d56c5e7c652e:

Pull
Pull
Pull
Pull
Pull
Pull
Pull
Pull
Pull
Pull
Pull

complete
complete
complete
complete
complete
complete
complete
complete
complete
complete
complete

b5d709749ac4: Pull complete
Digest: sha256:0ce9f13b5c5d235397252570acd0286a0a03472a22b7f0384fce09e65c680d13
Status: Downloaded newer image for mariadb:latest
db5218c494190c11a2fcc9627ea1371935d7021e86b5f652221bdac1cf182843

If you type

ps aux

you can notice the docker-containerd-shim process relative to this container running with

db5218c494190c11a2fcc9627ea1371935d7021e86b5f652221bdac1cf182843
/var/run/docker/libcontainerd/db5218c494190c11a2fcc9627ea1371935d7021e86b5f652221bdac1cf182843

and runC binary (

docker-runc

) as parameters:

docker-containerd-shim \
db5218c494190c11a2fcc9627ea1371935d7021e86b5f652221bdac1cf182843 \
/var/run/docker/libcontainerd/db5218c494190c11a2fcc9627ea1371935d7021e86b5f652221bdac1cf182843 \
docker-runc

db5218c494190c11a2fcc9627ea1371935d7021e86b5f652221bdac1cf182843

is the id of the container that you can see at the

end of the container creation.
ls -l /var/run/docker/libcontainerd/db5218c494190c11a2fcc9627ea1371935d7021e86b5f652221bdac1cf182843
total 4
-rw-r--r-- 1 root root 3653 Dec 27 22:21 config.json
prwx------ 1 root root
0 Dec 27 22:21 init-stderr
prwx------ 1 root root
0 Dec 27 22:21 init-stdin
prwx------ 1 root root
0 Dec 27 22:21 init-stdout

Docker Daemon Events
Docker daemon reports the following event: reload.

Docker Plugins
Even if they are not widely used, but Docker plugins are an interesting feature because it extends Docker by thirdparty plugins like network or storage plugins. Docker plugins run out of Docker processes and expose a webhooklike functionality which the Docker daemon uses to send HTTP POST requests in order the plugins acts as Docker
Events.
Let's take Flocker as an example.
Flocker is an open-source container data volume orchestrator for dockerized applications. Flocker gives Ops teams
the tools they need to run containerized stateful services like databases in production, since it provides a tool that
migrates data along with containers as they change hosts.
The standard way to run a volume container is :
docker run --name my_volume_container_1 -v /data -it -d busybox

But if you want to use Flocker plugin, you should run it like this:
docker run --name my_volume_container_2 -v /data -it --volume-driver=flocker -d busybox

Docker community is one of the most active communities during 2016 and the value of plugins is to allow these
Docker developers to contribute in a growing ecosystem.
Another known plugin is developed by Weave Net which integrates with Weave Scope so you can see how
containers are connected. Using Weave Flux works you can automate request routing in order to turn containers into
microservices.
We have seen how to use a storage plugin and it is something that looks like:
docker run -v : --volume-driver= <..> 

Network plugins has a different usage, you create the network first of all:
docker network create -d  

Then you use it:
docker run --net= <..> 

You can use some commands to manage Docker plugins, but they are all experimental and may change before it
becomes generally available.
To use the following commands, you should install the experimental version of Docker.
docker
docker
docker
docker
docker
docker

plugin
plugin
plugin
plugin
plugin
plugin

ls
enable
disable
inspect
install
rm

Overview Of Available Plugins
This is an inexhaustive overview of available plugins, that you can also find in the official documentation:
Contiv Networking An open source network plugin to provide infrastructure and security policies for
a multi-tenant micro services deployment, while providing an integration to physical network for
non-container workload. Contiv Networking implements the remote driver and IPAM APIs available
in Docker 1.9 onwards.
Kuryr Network Plugin A network plugin is developed as part of the OpenStack Kuryr project and
implements the Docker networking (libnetwork) remote driver API by utilizing Neutron, the
OpenStack networking service. It includes an IPAM driver as well.
Weave Network Plugin A network plugin that creates a virtual network that connects your Docker
containers - across multiple hosts or clouds and enables automatic discovery of applications. Weave
networks are resilient, partition tolerant, secure and work in partially connected networks, and other
adverse environments - all configured with delightful simplicity.
Azure File Storage plugin Lets you mount Microsoft Azure File Storage shares to Docker containers
as volumes using the SMB 3.0 protocol. Learn more.
Blockbridge plugin A volume plugin that provides access to an extensible set of container-based
persistent storage options. It supports single and multi-host Docker environments with features that
include tenant isolation, automated provisioning, encryption, secure deletion, snapshots and QoS.
Contiv Volume Plugin An open source volume plugin that provides multi-tenant, persistent,
distributed storage with intent based consumption. It has support for Ceph and NFS.
Convoy plugin A volume plugin for a variety of storage back-ends including device mapper and
NFS. It’s a simple standalone executable written in Go and provides the framework to support
vendor-specific extensions such as snapshots, backups and restore.
DRBD plugin A volume plugin that provides highly available storage replicated by
DRBD. Data written to the docker volume is replicated in a cluster of DRBD nodes.
Flocker plugin A volume plugin that provides multi-host portable volumes for Docker, enabling you
to run databases and other stateful containers and move them around across a cluster of machines.
gce-docker plugin A volume plugin able to attach, format and mount Google Compute persistentdisks.
GlusterFS plugin A volume plugin that provides multi-host volumes management for Docker using
GlusterFS.
Horcrux Volume Plugin A volume plugin that allows on-demand, version controlled access to your
data. Horcrux is an open-source plugin, written in Go, and supports SCP, Minio and Amazon S3.
HPE 3Par Volume Plugin A volume plugin that supports HPE 3Par and StoreVirtual iSCSI storage
arrays.
IPFS Volume Plugin An open source volume plugin that allows using an ipfs filesystem as a volume.
Keywhiz plugin A plugin that provides credentials and secret management using Keywhiz as a central
repository.
Local Persist Plugin A volume plugin that extends the default local driver’s functionality by
allowing you specify a mountpoint anywhere on the host, which enables the files to always persist,
even if the volume is removed via docker volume rm .
NetApp Plugin (nDVP) A volume plugin that provides direct integration with the Docker ecosystem
for the NetApp storage portfolio. The nDVP package supports the provisioning and management of

storage resources from the storage platform to Docker hosts, with a robust framework for adding
additional platforms in the future.
Netshare plugin A volume plugin that provides volume management for NFS 3/4, AWS EFS and
CIFS file systems.
OpenStorage Plugin A cluster-aware volume plugin that provides volume management for file and
block storage solutions. It implements a vendor neutral specification for implementing extensions
such as CoS, encryption, and snapshots. It has example drivers based on FUSE, NFS, NBD and EBS
to name a few.
Portworx Volume Plugin A volume plugin that turns any server into a scale-out converged
compute/storage node, providing container granular storage and highly available volumes across any
node, using a shared-nothing storage backend that works with any docker scheduler.
Quobyte Volume Plugin A volume plugin that connects Docker to Quobyte’s data center file system,
a general-purpose scalable and fault-tolerant storage platform.
REX-Ray plugin A volume plugin which is written in Go and provides advanced storage
functionality for many platforms including VirtualBox, EC2, Google Compute Engine, OpenStack,
and EMC.
Virtuozzo Storage and Ploop plugin A volume plugin with support for Virtuozzo Storage distributed
cloud file system as well as ploop devices.
VMware vSphere Storage Plugin Docker Volume Driver for vSphere enables customers to address
persistent storage requirements for Docker containers in vSphere environments.
Twistlock AuthZ Broker A basic extendable authorization plugin that runs directly on the host or
inside a container. This plugin allows you to define user policies that it evaluates during
authorization. Basic authorization is provided if Docker daemon is started with the –tlsverify flag
(username is extracted from the certificate common name).

Docker Plugins Events
Docker plugins report the following events: install, enable, disable, remove.

Docker Philosophy
Docker is a new way to build, ship & run differently but easier than "traditional" ways. This isn't just related to the
technology but also to the philosophy.

Build Ship & Run
Visualization is a useful technology to run different OSs on a single bare-metal server but containers go further, they
decouple applications and software from the underlying OS and could be used to create a self-service of an
application or a PaaS. Docker is development-oriented and user friendly it could be used in DevOps pipelines in
order to develop, test and run applications.
On the same way Docker allows DevOps teams to have a local development environment similar to production
easily, just build and ship.
After finishing his tests, the developer could publish the container using Docker machine.
In the developer-driven era, cloud infrastructures, IoT, mobile devices and other technologies are innovating each
day, it is really important from a DevOps view to reduce the complexity of software production, quality, stability
and scalability by integrating containers as wrappers to standardize runtime systems.

Docker Is Single Process
A container can be single or multiprocess. While other containerization allows containers to run multiple processes
(lxc), Docker restricts containers to run as a single process and if you would like to run n processes for your
application , you should run n Docker containers.
In reality, you can run multiple processes, since Docker has an instruction called ENTRYPOINT that starts a
program, you can set the ENTRYPOINT to execute a script and run as many programs as you want, but it is not
completely aligned with Docker philosophy.
Building an application based on a single-process containers is efficient to create microservices architectures,
develop PaaS (platform as a service), create FaaS (function as a service) platforms, serverless computing or eventbased programming..

We a going to learn about building microservices applications using Docker later in this book. Briefly, it is an
approach to implement distributed architectures where services are independent processes that communicate with
each other over a network.

Docker Is Stateless
A Docker container consists of a series of a series of layers created previously in an image, once the image becomes
a container, these layers become read-only layers.
Once a modification happens in the container by a process, a diff is made and Docker notices the difference
between a container's layer and the matched images's layer but only when you type
docker commit 

In this case a new image will be created with the last modifications but if you delete the container, the
disappear.

diff

will

Docker does not support a persistent storage but if you want to keep your storage persistent, you should use Docker
volumes and mount your files in the host filesystem, and in this case your code will be a part of a volume not a
container.
Docker is stateless, if any change happens in a running container, and new and different image could be created and
this is one of its strengths because you can do a fast and an easy rollback anytime.

Docker Is Portable
I created some public images that I pushed to my public Docker Hub, like eon01/vote, eon01/nginx-static - these
images were pulled more than 5k times. I created both of them using my laptop and all of the people who pulled
them used them as containers in different enviroments: bare-metal servers, desktops, laptops, virtual machines .. etc
all of these containers was built in a machine and run in the same way in thousands of other machines.

Chapter IV - Advanced Concepts
o
o

^__^
(oo)\_______
(__)\
)\/\
||----w |
||
||

Namespaces
According to Linux man pages, namespace wraps a global system resource in an abstraction that makes it appear to
the processes within the namespace that they have their own isolated instance of the global resource. Changes to the
global resource are visible to other processes that are members of the namespace, but are invisible to other
processes. One use of namespaces is to implement containers.
| Namespace | Constant
| Isolates
|
|------------|------------------|-------------------------------------|
| Cgroup
| CLONE_NEWCGROUP | Cgroup root directory
|
| IPC
| CLONE_NEWIPC
| System V IPC, POSIX message queues |
| Network
| CLONE_NEWNET
| Network devices, stacks, ports, etc.|
| Mount
| CLONE_NEWNS
| Mount points
|
| PID
| CLONE_NEWPID
| Process IDs
|
| User
| CLONE_NEWUSER
| User and group IDs
|
| UTS
| CLONE_NEWUTS
| Hostname and NIS domain name
|

Namespaces are the first form of isolation where a process running in a container A can not see or affect another
process running in a container B even if both processes are running in the same host machine.
Kernel namespaces allows a single host to have multiple:
Network devices
IP addresses
Routing rules
Netfilter rules
Timewait buckets
Bind buckets
Routing cache
etc ..
Every Docker container will have its own IP address and will see other docker containers as hosts connected to the
same "switch".

Control Groups (cgroups)
According to Wikipedia, cgroups or control groups is a Linux kernel feature that limits, accounts for, and isolates the
resource usage (CPU, memory, disk I/O, network, etc.) of a collection of processes.
This feature was introduced by engineers at Google in 2006 under the name "process containers". In late 2007, the
nomenclature changed to "control groups" to avoid confusion caused by multiple meanings of the term "container"
in the Linux kernel context, and the control groups functionality was merged into the Linux kernel mainline in kernel
version 2.6.24, which was released in January 2008.
Control Groups are also used by Docker not only to implement resource accounting and isolate its usage but
because they provide many useful metrics that help ensure that each container gets enough memory, CPU, disk I/O,
memory cache, CPU cache ..etc.
Control Groups provides also a level of security because using the feature ensure that a single container will not
bring down the host OS by exhausting its resources.
Public and private PaaS are also making use of cgroups in order to guarantee a stable service with no downtimes
when an application exhaust CPU, memory or any other critical resource. Other software and containers
technologies use cgroups like LXC, libvirt, systemd, Open Grid Scheduler and lmctfy.

Linux Capabilities
Unix was designed to run two categories of processes :
privileged processes, with the user id 0 which is the root user.
unprivileged processes where the user id is different from zero.
If you are familiar with Linux, you can find all of users ids when you type:
ps -eo uid

Every process in a UNIX has an owner (designed by its UID) and a group owner (identified by a GID). When Unix
starts a process, its GID is set to the GID of its parent process. The main purpose of this schema is performing
permission checks.
While privileged processes bypass all kernel permission checks, all the other processes are subject to a permission
check ( the Unix-like system Kernel make the permission check based on the UID and the GID of the non root
process.
Starting with Kernel 2.2, Linux divides the privileges into distinct units that we call capabilities. Capabilities can be
independently enabled and disabled, which is a great security and permission management feature. A process can
have CAP_NET_RAW capability while it is denied to use the CAP_NET_ADMIN capability. This is just an example
explaining how capabilities works. You can find the complete list of the other capabilities in Linux manpage:
man capabilities

By default, Docker starts containers with a limited list of capabilities so that containers will not use a real root
privileges and the root user within the container will not access all of the privileges of a real root user.
This is the list of the capabilities used by a default Docker installation :
s.Process.Capabilities = []string{
"CAP_CHOWN",
"CAP_DAC_OVERRIDE",
"CAP_FSETID",
"CAP_FOWNER",
"CAP_MKNOD",
"CAP_NET_RAW",
"CAP_SETGID",
"CAP_SETUID",
"CAP_SETFCAP",
"CAP_SETPCAP",
"CAP_NET_BIND_SERVICE",
"CAP_SYS_CHROOT",
"CAP_KILL",
"CAP_AUDIT_WRITE",
}

You can find the same list in Docker source code on github

Secure Computing Mode (seccomp)
Since Kernel version 2.6.12, Linux allows a process to make a one-way transition into a "secure" state where it
cannot make any system calls to an open file descriptors (fd).
Only exit(), sigreturn(), read() and write() system calls are allowed, any other call made by a process with a "secure"
state will beget its termination : the Kernel will send him a SIGKILL. This is the sandboxing mechanism in Linux
Kernel based on seccomp Kernel's feature.
Docker allows using seccomp to restrict the actions available within a container and restrict your application's
access.
To use the seccomp feature, Kernel should be configured with CONFIG_SECCOMP enabled and Docker should be
built with seccomp .
If you would like to check if your Kernel has seccomp activated, type:
cat /boot/config-`uname -r` | grep CONFIG_SECCOMP=

If it is activated, yous should see this:
CONFIG_SECCOMP=y

Docker uses seccomp profiles, which are configurations that disable around 44 system calls out of 300+. You can
find the default profile in the official repository.
You can choose a specific seccomp profile when running a container by adding the following options to the Docker
run command:
--security-opt seccomp=/path/to/seccomp/profile.json

The following table can be found in Docker official repository and it represents some of the blocked syscalls.
Syscall
acct
add_key
adjtimex
bpf
clock_adjtime
clock_settime
clone
create_module
delete_module
finit_module
get_kernel_syms
get_mempolicy
init_module

Description
Accounting syscall which could let containers disable their own resource limits or process
accounting. Also gated by CAP_SYS_PACCT .
Prevent containers from using the kernel keyring, which is not namespaced.
Similar to clock_settime and settimeofday , time/date is not namespaced.
Deny loading potentially persistent bpf programs into kernel, already gated by
CAP_SYS_ADMIN
.
Time/date is not namespaced.
Time/date is not namespaced.
Deny cloning new namespaces. Also gated by CAP_SYS_ADMIN for CLONE_* flags, except
CLONE_USERNS
.
Deny manipulation and functions on kernel modules.
Deny manipulation and functions on kernel modules. Also gated by CAP_SYS_MODULE .
Deny manipulation and functions on kernel modules. Also gated by CAP_SYS_MODULE .
Deny retrieval of exported kernel and module symbols.
Syscall that modifies kernel memory and NUMA settings. Already gated by CAP_SYS_NICE .
Deny manipulation and functions on kernel modules. Also gated by CAP_SYS_MODULE .

ioperm

Prevent containers from modifying kernel I/O privilege levels. Already gated by
.
Prevent containers from modifying kernel I/O privilege levels. Already gated by
CAP_SYS_RAWIO
.
Restrict process inspection capabilities, already blocked by dropping CAP_PTRACE .
Sister syscall of kexec_load that does the same thing, slightly different arguments.
Deny loading a new kernel for later execution.
Prevent containers from using the kernel keyring, which is not namespaced.
Tracing/profiling syscall, which could leak a lot of information on the host.
Syscall that modifies kernel memory and NUMA settings. Already gated by CAP_SYS_NICE .
Deny mounting, already gated by CAP_SYS_ADMIN .
Syscall that modifies kernel memory and NUMA settings.
Sister syscall to open_by_handle_at . Already gated by CAP_SYS_NICE .
Deny interaction with the kernel nfs daemon.
Cause of an old container breakout. Also gated by CAP_DAC_READ_SEARCH .
Tracing/profiling syscall, which could leak a lot of information on the host.
Prevent container from enabling BSD emulation. Not inherently dangerous, but poorly tested,
potential for a lot of kernel vulns.
Deny pivot_root , should be privileged operation.
Restrict process inspection capabilities, already blocked by dropping CAP_PTRACE .
Restrict process inspection capabilities, already blocked by dropping CAP_PTRACE .
Tracing/profiling syscall, which could leak a lot of information on the host. Already blocked
by dropping CAP_PTRACE .
Deny manipulation and functions on kernel modules.
Quota syscall which could let containers disable their own resource limits or process
accounting. Also gated by CAP_SYS_ADMIN .
Don't let containers reboot the host. Also gated by CAP_SYS_BOOT .
Prevent containers from using the kernel keyring, which is not namespaced.
Syscall that modifies kernel memory and NUMA settings. Already gated by CAP_SYS_NICE .
Deny associating a thread with a namespace. Also gated by CAP_SYS_ADMIN .
Time/date is not namespaced. Also gated by CAP_SYS_TIME .
Time/date is not namespaced. Also gated by CAP_SYS_TIME .
Deny start/stop swapping to file/device. Also gated by CAP_SYS_ADMIN .
Deny start/stop swapping to file/device. Also gated by CAP_SYS_ADMIN .
Obsolete syscall.
Obsolete, replaced by /proc/sys.
Should be a privileged operation. Also gated by CAP_SYS_ADMIN .
Should be a privileged operation.
Deny cloning new namespaces for processes. Also gated by CAP_SYS_ADMIN , with the exception
of unshare --user .
Older syscall related to shared libraries, unused for a long time.
Userspace page fault handling, largely needed for process migration.
Obsolete syscall.
In kernel x86 real mode virtual machine. Also gated by CAP_SYS_ADMIN .
In kernel x86 real mode virtual machine. Also gated by CAP_SYS_ADMIN .
CAP_SYS_RAWIO

iopl
kcmp
kexec_file_load
kexec_load
keyctl
lookup_dcookie
mbind
mount
move_pages
name_to_handle_at
nfsservctl
open_by_handle_at
perf_event_open
personality
pivot_root
process_vm_readv
process_vm_writev
ptrace
query_module
quotactl
reboot
request_key
set_mempolicy
setns
settimeofday
stime
swapon
swapoff
sysfs
_sysctl
umount
umount2
unshare
uselib
userfaultfd
ustat
vm86
vm86old

If you would like to run a container without the default seccomp profile defined below, you can run:
docker run --rm -it --security-opt seccomp=unconfined hello-world

If you are using a distribution that does not support seccomp profiles, like Ubuntu 14.04, you will have the
following error : docker: Error response from daemon: Linux seccomp: seccomp profiles are not supported on this daemon, you
cannot specify a custom seccomp profile.

Application Armor (Apparmor)
Application Armor is a Linux Kernel security module. It allows the Linux administrator to restrict programs'
capabilities with per-program profiles.
In other words, this is a tool to lock applications by limiting their access to the only resources they are supposed to
use without disturbing neither their execution nor their performance.
Profiles can allow capabilities like raw socket, network access, read, write or execute files. This functionality was
added to Linux to complete the Discretionary Access Control (DAC) model. The Mandatory Access Control (MAC)
was then introduced.
You can check your Apparmor status by typing:
sudo apparmor_status

You will see a similar output to the following one, if it is activated:
apparmor module is loaded.
19 profiles are loaded.
18 profiles are in enforce mode.
/sbin/dhclient
/usr/bin/evince
/usr/bin/evince-previewer
/usr/bin/evince-previewer//sanitized_helper
/usr/bin/evince-thumbnailer
/usr/bin/evince-thumbnailer//sanitized_helper
/usr/bin/evince//sanitized_helper
/usr/bin/lxc-start
/usr/lib/NetworkManager/nm-dhcp-client.action
/usr/lib/connman/scripts/dhclient-script
/usr/sbin/cups-browsed
/usr/sbin/mysqld
/usr/sbin/ntpd
/usr/sbin/tcpdump
docker-default
lxc-container-default
lxc-container-default-with-mounting
lxc-container-default-with-nesting
1 profiles are in complain mode.
/usr/sbin/sssd
8 processes have profiles defined.
8 processes are in enforce mode.
/sbin/dhclient (1815)
/usr/sbin/cups-browsed (1697)
/usr/sbin/ntpd (3363)
docker-default (3214)
docker-default (3380)
docker-default (3381)
docker-default (3382)
docker-default (3390)
0 processes are in complain mode.
0 processes are unconfined but have a profile defined.

You can see in the list above that Docker profile is active and it is called
when you run a container, the following command is explicitly executed :
docker run --rm -it --security-opt apparmor=docker-default hello-world

The default profile is the following:
#include 

profile docker-default flags=(attach_disconnected,mediate_deleted) {

docker-default

. It is the default profile so

#include 

network,
capability,
file,
umount,
deny
deny
deny
deny
deny

@{PROC}/{*,**^[0-9*],sys/kernel/shm*} wkx,
@{PROC}/sysrq-trigger rwklx,
@{PROC}/mem rwklx,
@{PROC}/kmem rwklx,
@{PROC}/kcore rwklx,

deny mount,
deny
deny
deny
deny
deny
deny
deny

/sys/[^f]*/** wklx,
/sys/f[^s]*/** wklx,
/sys/fs/[^c]*/** wklx,
/sys/fs/c[^g]*/** wklx,
/sys/fs/cg[^r]*/** wklx,
/sys/firmware/efi/efivars/** rwklx,
/sys/kernel/security/** rwklx,

}

You can confine containers using Apparmor but not the programs running inside a container. If you are
running Nginx under a container profile to protect the host, you will not be able to confine the Nginx with a different
profile to protect the container.
Apparmor is used since Linux Kernel since version 2.6.36 release.

Docker Union Filesystem
A Docker image is built of layers. UnionFS is the technology that allows that.
Union Filesystem or UnionFS is a filesystem service used in Linux, FreeBSD and NetBSD. It allows a set of files
and directories to form a single filesystem by grouping them in a single branch. Any Docker image is in reality a set
of layered filesystems superposed, one over the other.
UnionFS layers are transparently overlaid and form a coherent file system that we call branches. They may be either
read-only or read-write file systems.
Docker uses UnionFS in order to avoid duplicating a branch. The boot filesystem (bootfs) for example is one of the
branches that are used in many containers and it resembles the typical Unix boot filesystem. It is used in Ubuntu,
Debian, CentOs and many other containers.
When a container is started, Docker mounts a filesystem on top of any layers below and make it writable. So any
change is applied only to this layer. If you want to modify a file, it will be moved from the read-only layer below
into the read-write layer at the top.
Let's start a container and see its filesystem layers.
docker create -it redis

You can notice that Docker is downloading different layers in the output:
latest: Pulling from library/redis
6a5a5368e0c2: Pull complete
2f1103ce5ca9: Pull complete
086a40c85e01: Pull complete
9a5e9d112ec4: Pull complete
dadc4b601bb4: Pull complete
b8066982e7a1: Pull complete
2bcdfa1b63bf: Pull complete
Digest: sha256:38e873a0db859d0aa8ab6bae7bcb03c1bb65d2ad120346a09613084b49185912
Status: Downloaded newer image for redis:latest
6ab94a06bc0263110b973174d65cbc6ebd6d9fc637526b2c9dd3eac3c3bcf032

`docker create creates a writable container layer over the specified image hello-world in the last case) and
prepares it for running a command. It is similar to docker run -d except the container will not start running.

When running the last command, you will have an id that will be printed on your terminal. It is the id of the prepared
container.
6ab94a06bc0263110b973174d65cbc6ebd6d9fc637526b2c9dd3eac3c3bcf032

The container is ready, let's start it:
docker start 0eeab42751ff0172f845b9cb737c966471be9f10282cf1684519bc7f5da80170

The command

docker start

creates a process space around the UnionFS block.

We can't have more than one process space per container.

A

docker ps

will show that the container is running:

CONTAINER ID
6ab94a06bc02

IMAGE
redis

COMMAND
"docker-entrypoint.sh"

PORTS
6379/tcp

NAMES
trusting_carson

If you would like to see the layers, then type:
ls -l /var/lib/docker/aufs/layers

Layers of the same images will have a name that starts with the container id

6ab94a06bc02

.

We haven't seen yet the concept of Dockerfile, but don't worry if it is the first time you saw it. For the moment, let's
just limit our understanding to the explanation figuring in the official documentation:
Docker can build images automatically by reading the instructions from a Dockerfile. A Dockerfile is a text
document that contains all the commands a user could call on the command line to assemble an image. Using
Docker build users can create an automated build that executes several command-line instructions in
succession.
In general, when writing a Dockerfile, every line in this file will create an additional layer.
FROM busybox
MAINTAINER Aymen El Amri
RUN ls

# This is a layer
# This is a layer
# This is a layer

Let's build this image:
docker build .

The id of the build, on the output for this example was
7f49abaf7a69

In other words, a layer is a change in an image.
So the second advantage of using UnionFS and the layered filesystem is the isolation of modifications : Any change
that happens to the running container will be initialized once the containers is restarted. The bootfs layer is one of
the layers that users will not interact with.
To sum up, when Docker mounts the rootfs, it starts as a read-only system, as in a traditional Linux boot, but then,
instead of changing the file system to the read-write mode, Docker takes advantage of a union mount to add a readwrite filesystem over the first read-only filesystem. There may be multiple read-only file systems layered on top of
each other. These file systems are called layers.
Docker supports a number of different union file systems. The next chapter about images will reinforce your
understanding about this.

Storage Drivers
OverlayFS, AUFS, VFS or Device Mapper are some storage technologies that are compatible with Docker. They are
easily "pluggable" to Docker and we call them storage drivers. Every technology has its own specificities and
choosing one depends on your technical requirements and criteria (like stability, maintenance, backup, speed,
hardware ..etc)
When we started running an image, Docker used a mechanism called copy-on-write (CoW).
CoW is a standard Unix pattern that provides a single shared copy of some data until it is modified. In general, CoW
is an implicit sharing of resources used to implement a copy operation on modifiable resources.
// define x
std::string x("Hello");
// x and y use the same buffer
std::string y = x;
// now y uses a different buffer while x still uses the same old buffer
y += ", World!";

In storage, CoW is used as an underlying mechanism to create snapshots like it is the case for for logical volume
management done by ZFS, AUFS .. etc
In a CoW algorithm, the original storage is never modified and when a write operation is requested it is
automatically redirected to a new storage (the original one is not modified). This redirection is called Redirect-onwrite or ROW
When a write request is made, all the data is copied into a new storage resource. We call this second step Copy-onwrite" or COW.
You have probably seen blog posts about one storage drivers being better than others or some post mortems. Keep in
mind that there is no technology better than the other, every technology has its pro and it cons.

OverlayFS
OverlayFS is a filesystem service for Linux that implements a union mount for other file systems. In the commit
453552c8384929d8ae04dcf1c6954435c0111da0 of Docker, OverlayFS was added to Docker by @alexlarsson from
Redhat. In the signature, the commit was described as:
Each container/image can have a "root" subdirectory which is a plain filesystem hierarchy, or they can use
overlayfs.
If they use overlayfs there is a "upper" directory and a "lower-id" file, as well as "merged" and "work"
directories. The "upper" directory has the upper layer of the overlay, and "lower-id" contains the id of the
parent whose "root" directory shall be used as the lower layer in the overlay. The overlay itself is mounted in
the "merged" directory, and the "work" dir is needed for overlayfs to work.

When a overlay layer is created there are two cases, either the parent has a "root" dir, then we start out with a
empty "upper" directory overlaid on the parents root. This is typically the case with the init layer of a container
which is based on an image. If there is no "root" in the parent, we inherit the lower-id from the parent and start
by making a copy if the parents "upper" dir. This is typically the case for a container layer which copies its
parent -init upper layer.
Additionally we also have a custom implementation of ApplyLayer which makes a recursive copy of the parent
"root" layer using hardlinks to share file data, and then applies the layer on top of that. This means all chile
images share file (but not directory) data with the parent.
Overlay2 was added to Docker in the pull #22126

Pro
OverlayFS is similar to aufs but it is supported and was merged into the mainline Linux kernel since version 3.18.
Like aufs it enables shared memory between containers using the same shared libraries (in the same disk). The
advantage of using OverlayFS is the fact that it is on a continued development. It is simpler than aufs and in most
cases faster.
Since its version 1.12, Docker provides overlay2 storage driver which is more efficient than overlay. The version 2
of OverlayFS is compatible with Kernel 4.0 and later.
Many users had issues with OverlayFS because they run out of inode and this problem was solved by Overlay2.

Testing shows that there is a significant reduction in the number of used inode used, this result is available
especially for images having multiple layers. This is clearly solving the problem of inode usage.

Cons
Mainly, the inode exhaustion problems. It is a serious problem that was fixed by OverlayFS 2.
Some other issues but the majority of these problems were fixed in the version 2. If you are planning to use
OverlaFS use OverlayFS 2 instead.
On the other hand, Overlay2 is a young codebase and Docker 1.12 is the first release offering Overlay2 and logically

some other bugs will be discovered in the future so it is a system to use with vigilance.

AUFS
aufs (short for advanced multi-layered unification filesystem) is Union Filesystem that existed in Docker since the
beginning. It was developed to improve reliability and performance. It introduced some new concepts, like writable
branch balancing. In its manpage, aufs is described as a stackable unification filesystem such as Unionfs, which
unifies several directories and provides a merged single directory.
In the early days, aufs was entirely re-designed and re-implemented Unionfs Version 1.x series.
After many original ideas, approaches and improvements, it becomes totally different from Unionfs while keeping
the basic features.

Pro
aufs is one of the most popular drivers for Docker. It is stable and used by many distributions like Knoppix, Ubuntu
10.04, Gentoo Linux 12.0 and Puppy Linux live CDs distributions. aufs help different containers to share memory
pages if they are loading the same shared libraries from the same layer.
The longest existing and possibly the most tested graphdriver backend for Docker. Reasonably performant and
stable for wide range of use cases, even though it is only available on Ubuntu and Debian Kernels (as noted below),
there has been significant use of these two distributions with Docker allowing for lots of airtime for the aufs driver
to be tested in a broad set of environments. Enables shared memory pages for different containers loading the same
shared libraries from the same layer (because they are the same inode on disk).

Cons
aufs was rejected for merging into mainline Linux. Its code was criticized for being "dense, unreadable, and
uncommented".
Aufs consists of about 20,000 lines of dense, unreadable, uncommented code, as opposed to around 10,000 for
Unionfs and 3,000 for union mounts and 60,000 for all of the VFS. The aufs code is generally something that
one does not want to look at. source
Instead, OverlayFS was merged in the Linux kernel.

Btrfs
Btrfs is a modern CoW filesystem for Linux.
The philosophy behind Btrfs is to implement advanced features while focusing on fault tolerance and easy
administration. Btrfs is developed at multiple companies and licensed under the GPL. The name stands for "B TRee
File System" so it is easier and probably acceptable to pronounce "B-Tree FS" instead of b-t-r-fs, but the real
pronunciation is "Butter FS".
Btrfs has native features named subvolumes and snapshots providing together CoW-like features.
It actually consists of three types of on-disk structures:
block headers,
keys,
and items
currently defined as follows:

struct btrfs_header {
u8 csum[32];
u8 fsid[16];
__le64 blocknr;
__le64 flags;
u8 chunk_tree_uid[16];
__le64 generation;
__le64 owner;
__le32 nritems;
u8 level;
}
struct btrfs_disk_key {
__le64 objectid;
u8 type;
__le64 offset;
}
struct btrfs_item {
struct btrfs_disk_key key;
__le32 offset;
__le32 size;
}

Btrfs was added to Docker in this commit by Alex Larsson from Red Hat.

Pro
Btrfs was introduced in the mainline Linux kernel in 2007 and was used for some few years and it was used by Linus
Torvalds as his root file system on one of his laptops. When Btrfs was released it was optimized compared to oldschool filesystems. According to the official wiki, this filesystem has the following features:
Extent based file storage
2^64 byte == 16 EiB maximum file size (practical limit is 8 EiB due to Linux VFS)
Space-efficient packing of small files
Space-efficient indexed directories
Dynamic inode allocation
Writable snapshots, read-only snapshots
Subvolumes (separate internal filesystem roots)
Checksums on data and metadata (crc32c)
Compression (zlib and LZO)
Integrated multiple device support
File Striping
File Mirroring
File Striping+Mirroring
Single and Dual Parity implementations (experimental, not production-ready)
SSD (flash storage) awareness (TRIM/Discard for reporting free blocks for reuse) and optimizations (e.g.
avoiding unnecessary seek optimizations, sending writes in clusters, even if they are from unrelated files. This
results in larger write operations and faster write throughput)
Efficient incremental backup
Background scrub process for finding and repairing errors of files with redundant copies
Online filesystem defragmentation
Offline filesystem check
In-place conversion of existing ext3/4 file systems
Seed devices. Create a (readonly) filesystem that acts as a template to seed other Btrfs filesystems. The original
filesystem and devices are included as a readonly starting point for the new filesystem. - - Using copy on write,
all modifications are stored on different devices; the original is unchanged.
Subvolume-aware quota support
Send/receive of subvolume changes
Efficient incremental filesystem mirroring
Batch, or out-of-band deduplication (happens after writes, not during)

Cons
Btrfs hasn't been a real choice for many Linux distributions and this fact made of Btrf a technology without much
testing and bug hunting.

Device Mapper
The device mapper is a framework provided by the Linux kernel (Kernel-based framework) to map physical block
devices to virtual block devices which is a higer-level abstraction. LVM2, software RAIDs and dm-crypt disk
encryption are targets of this technology. It also offers other features like file system snapshots. Device Mapper
algorithm works at the block level and not the file level:
The device mapper driver stores every image and container on a separate virtual device that are provisioned CoW
snapshot devices. We call this thin provision or thip.

Pro
It was tested and use by many communities unlike other filesystems.
Many projects and Linux features are built on top of the device mapper:
LVM2 – logical volume manager for the Linux kernel
dm-crypt – mapping target that provides volume encryption
dm-cache – mapping target that allows creation of hybrid volumes
dm-log-writes – mapping target that uses two devices, passing through the first device and logging the write
operations performed to it on the second device
dm-verity – validates the data blocks contained in a file system against a list of cryptographic hash values,
developed as part of the Chromium OS project
dmraid – provides access to "fake" RAID configurations via the device mapper
DM Multipath – provides I/O failover and load-balancing of block devices within the Linux kernel
Linux version of TrueCrypt
DRBD (Distributed Replicated Block Device)
kpartx – utility called from hotplug upon device maps creation and deletion
EVMS (deprecated)
cryptsetup – utility used to conveniently setup disk encryption based on dm-crypt
And of course Docker that uses device mapper to create copy-on-write storage for software containers.

Cons
If you would like to get device mapper storage driver performance, you should consider doing some configurations
and you should not run "loopback" mode in production.
You should try 15 steps explained in Docker official documentation, in order to use devicemapper driver in the
direct-lvm mode.
I have never used it but I saw some feedback from users having problems with its usage.

ZFS
This filesystem was first merged into the Docker engine in May 2015 and was available since Docker 1.7 release.
ZFS and Docker/go-zfs wrapper requires the installation of zfs-utils or zfs (e.g Ubuntu 16.04).
ZFS or Zettabyte File System is a combined file system and logical volume manager created at Sun Microsystems. It
was used by OpenSolaris since 2005.

Pro
ZFS is a native filesystem to Solaris, OpenSolaris, OpenIndiana, illumos, Joyent SmartOS, OmniOS, FreeBSD,
Debian GNU/kFreeBSD systems, NetBSD ans OSv.
ZFS is a killer-app for Solaris, as it allows straightforward administration of disks and pool of disks while giving
performance and integrity. It protects data against "silent error" of the disk ( caused by firmware bugs or even
hardware malfunctions like bad cables..etc ).

According to Sun Microsystems official website, ZFS meets the needs of a file system for everything from desktops
to data centers and offers:
Simple administration : ZFS automates and consolidates complicated storage administration concepts, reducing
administrative overhead by 80 percent.
Provable data integrity : ZFS protects all data with 64-bit checksums that detect and correct silent data
corruption.
Unlimited scalability : As the world's first 128-bit file system, ZFS offers 16 billion billion times the capacity of
32- or 64-bit systems.
Blazing performance : ZFS is based on a transactional object model that removes most of the traditional
constraints on the order of issuing I/Os, which results in huge performance gains.

It achieves its performance through a number of techniques:
Dynamic striping across all devices to maximize throughput
Copy-on-write design makes most disk writes sequential
Multiple block sizes, automatically chosen to match workload
Explicit I/O priority with deadline scheduling
Globally optimal I/O sorting and aggregation
Multiple independent prefetch streams with automatic length and stride detection
Unlimited, instantaneous read/write snapshots
Parallel, constant-time directory operations
Creating a new zpool is needed to use zfs :
sudo zpool create -f zpool-docker /dev/xvdb

Cons
The ZFS license is incompatible with Linux license, so Linux does not have a ZFS implementation and not every OS
can use ZFS (e.g Windows).
It takes some learning to use, so if you are using it as your main filesystem you will probably need some knowledge
about it. zfs lacks inode sharing for shared libraries between containers, but in reality it is not the only driver not
implementing this.

VFS
vfs simply stand for Virtual File System which is the abstraction layer on top of a concrete physical filesystem but it
does not use Union File System and CoW and that is why it is used by developers for debugging only.

You can find Docker volumes using vfs in

/var/lib/docker/vfs/dir

.

Pro
To test Docker engine, VFS is very useful since it is simpler to validate tests using this simple FS. This filesystem is
helpful to run Docker in Docker (dind). The official Docker image uses vfs as the default filesystem.
--storage-driver=vfs

vfs is the only driver which is guaranteed to work regardless of the underlying filesystem in the case of Docker in
Docker but it could be very slow and inefficient. Running Docker in Docker will be detailed later in this book.

Cons
It is not recommended at all to run VFS on production since it is not intended to be used with Docker production
clusters.

What Storage Driver To Choose
Your choice of the storage driver could not based on only pros and cons of each filesystem but there are other choice
criteria. For example, overlayFS and overlay2 can not be used on the top of a btrfs.
In general, some of these storage drivers can operate on top of different backing filesystems (host filesystem) but not
all. These table explains the common usage of each storage driver:
Storage Driver
overlay
overlay2
aufs
btrfs
devicemapper
zfs
vfs

Commonly Used On Top Of
xfs & ext4
xfs & ext4
xfs & ext4
btrfs
direct-lvm
zfs
debugging

Docker community created a practical diagram that shows simply the strength and the weakness of some storage
drivers and their best use case.

Finally, there is no storage driver adapted to all use cases, there is no "ultimate choice" to make but it depends on
your use case. If you don't know what to choose exactly go for aufs or Overlay2.

Chapter V - Working With Docker Images
o
o

^__^
(oo)\_______
(__)\
)\/\
||----w |
||
||

Managing Docker Images
Images, Intermediate Images & Dangling Images
If you are running Docker on your laptop or your server, you can see a list of the images that Docker used or uses by
typing:
docker images

You will have a similar list:
REPOSITORY

scratch
xenial


TAG
IMAGE ID
CREATED
SIZE
none
57ce8f25dd63
2 days ago
229.7 MB
latest
f02aa3980a99
2 days ago
0 B
latest
7a409243b212
2 days ago
229.7 MB
none
149b13361203
2 days ago
12.96 MB

Images with are untagged images.
You can print a custom output where you choose to view the ID and the size of the image:
docker images --format "{{.ID}}: {{.Size}}"

Or to view the repository:
docker images --format "{{.ID}}: {{.Repository}}"

In many cases, we just need to get IDs:
docker images -q

If you want to list all of them, then type:
docker images -a

or
docker images --all

In this list you will see all of the images even the intermediate ones.
REPOSITORY
my_app


TAG
latest


IMAGE ID
7f49abaf7a69
afe4509e17bc

SIZE
1.093 MB
225.6 MB

All of the

:

:

are intermediate images.

images will grow exponentially with the numbers of images you download.

As you know each docker image is composed of layers with a parent-child hierarchical relationship .
These intermediate layers are a result of caching build operations which decrease disk usage and speed up builds.
Every build step is cached, that's why you may experienced some disk space problems after using Docker for a
while.

All docker layers are stored in

/var/lib/docker/graph

called the graph database.

Some of the intermediary images are not tagged, they are called dangling images.
docker images --filter "dangling=true"

Other filters may be used:
- label= or label==
- before=([:tag]||)
- since=([:tag]||)

Finding Images
If you type :
docker search ubuntu

you will get a list of images called Ubuntu that people shared publicly in the Docker Hub (hub.docker.com).
NAME
DESCRIPTION
STARS
ubuntu
Ubuntu is a Debian-based Linux operating s...
4958
ubuntu-upstart
Upstart is an event-based replacement for ...
67
rastasheep/ubuntu-sshd
Dockerized SSH service, built on top of of...
47
ubuntu-debootstrap
debootstrap --variant=minbase --components...
28
torusware/speedus-ubuntuAlways updated official Ubuntu docker imag...
27
consol/ubuntu-xfce-vnc
Ubuntu container with "headless" VNC sessi...
26
ioft/armhf-ubuntu
ABR] Ubuntu Docker images for the ARMv7(a...
19
nickistre/ubuntu-lamp
LAMP server on Ubuntu
10
nuagebec/ubuntu
Simple always updated Ubuntu docker images...
9
nimmis/ubuntu
maxexcloo/ubuntu
jordi/ubuntu
admiringworm/ubuntu
darksheer/ubuntu
lynxtp/ubuntu
datenbetrieb/ubuntu
teamrock/ubuntu
labengine/ubuntu
esycat/ubuntu
ustclug/ubuntu
widerplan/ubuntu
konstruktoid/ubuntu
vcatechnology/ubuntu
webhippie/ubuntu

This is a docker images different LTS vers...
5
Base image built on Ubuntu with init, Supe...
2
Ubuntu Base Image
1
Base ubuntu images based on the official u...
1
Base Ubuntu Image -- Updated hourly
1
https://github.com/lynxtp/docker-ubuntu
0
custom flavor of the official ubuntu base ...
0
TeamRock's Ubuntu image configured with AW...
0
Images base ubuntu
0
Ubuntu LTS
0
ubuntu image for docker with USTC mirror
0
Our basic Ubuntu images.
0
Ubuntu base image
0
A Ubuntu image that is updated daily
0
Docker images for ubuntu
0

OFFICIAL
[OK]
[OK]

AUTOMATED

[OK]
[OK]
[OK]
[OK]
[OK]
[OK]
[OK]
[OK]
[OK]
[OK]
[OK]
[OK]
[OK]
[OK]
[OK]
[OK]
[OK]
[OK]
[OK]
[OK]
[OK]
[OK]

As you can notice, there are images having automated builds while others don't have this feature activated. An
automated builds allow your image to be up-to-date with changes on your private and public git (Github or
Bitbucket) code.

Notice that if you type the search command you will get only 25 image and if you want more, you could use the
limit option:
docker search --limit 100 mongodb

NAME
mongo
tutum/mongodb
frodenas/mongodb
agaveapi/mongodb-sync
sameersbn/mongodb
bitnami/mongodb
waitingkuo/mongodb
tobilg/mongodb-marathon
appelgriebsch/mongodb
azukiapp/mongodb
tianon/mongodb-mms
cpuguy83/mongodb
zokeber/mongodb
triply/mongodb
hairmare/mongodb
networld/mongodb
jetlabs/mongodb
mattselph/ubuntu-mongodb
oliverwehn/mongodb
gorniv/mongodb
vaibhavtodi/mongodb
tozd/meteor-mongodb
ncarlier/mongodb
pulp/mongodb
ulboralabs/alpine-mongodb
mminke/mongodb
kardasz/mongodb
tcaxias/mongodb
whatwedo/mongodb
guttertec/mongodb
peerlibrary/mongodb
jecklgamis/mongodb
unzeroun/mongodb
falinux/mongodb
hysoftware/mongodb
birdhouse/mongodb
airdock/mongodb
bitergia/mongodb
tianon/mongodb-server
andreynpetrov/mongodb
lukaszm/mongodb
radiantwf/mongodb
luca3m/mongodb-example
derdiedasjojo/mongodb
faboulaye/mongodb
tcloud/mongodb
omallo/mongodb
romeoz/docker-mongodb
denmojo/mongodb
pl31/debian-mongodb
kievechua/mongodb
apiaryio/base-dev-mongodb
babim/mongodb
blkpark/mongodb
partlab/ubuntu-mongodb
jbanetwork/mongodb
baselibrary/mongodb
glnds/mongodb
hpess/mongodb
hope/mongodb
recteurlp/mongodb

DESCRIPTION
MongoDB document databases provide high av...
MongoDB Docker image – listens in port 2...
A Docker Image for MongoDB
Docker image that regularly backs up and/o...

STARS
2616
166
12
7
6
Bitnami MongoDB Docker Image
5
MongoDB
2.4.9
4
A Docker image to start a dynamic MongoDB ...
4
Configurable MongoDB container based on Al...
3
Docker image to run MongoDB by Azuki - htt...
3
https://mms.mongodb.com/
2
2
MongoDB Dockerfile in CentOS 7
2
Extension of official mongodb image that a...
1
MongoDB on Gentoo
1
Networld PaaS MongoDB image in default ins...
1
Build MongoDB 3 image
1
Ubuntu 14.04 LTS with Mongodb 2.6.4
1
Out-of-the-box app-ready MongoDB server wi...
1
MongoDB Docker image
1
A MongoDB Docker image on Ubuntu 14.04.3. ...
1
MongoDB server image for Meteor applications.
1
MongoDB Docker image based on debian.
1
1
Docker Alpine Mongodb
1
Mongo db image which downloads the databas...
1
MongoDB
0
Percona's MongoDB on Debian. Storage Engin...
0
0
MongoDB is a free and open-source cross-pl...
0
0
mongodb
0
Mongodb image
0
mongodb docker image.
0
Docker mongodb image for hysoftware.net
0
Docker image for MongoDB used in Birdhouse.
0
0
MongoDB Docker image (deprecated)
0
0
mongodb
0
MongoDB
0
MongoDB Enterprise Docker image
0
Sample mongodb app
0
mongodb cluster prepared
0
Mongodb container
0
mongodb
0
MongoDB image build
0
MongoDB container image which can be linke...
0
A simple mongodb container that can be lin...
0
mongodb from debian packages
0
Based on Tutum's Mongodb with official image
0
WARNING: to be replaced by apiaryio/mongodb
0
docker-mongodb
0
mongodb
0
Docker image to run an out of the box Mong...
0
mongodb
0
ThoughtWorks Docker Image: mongodb
0
CentOS 7 / MongoDB 3
0
0
MongoDB image
0
Fedora DockerFile for MongoDB
0

OFFICIAL
[OK]

AUTOMATED
[OK]
[OK]
[OK]
[OK]
[OK]
[OK]
[OK]
[OK]
[OK]
[OK]
[OK]
[OK]
[OK]
[OK]
[OK]
[OK]
[OK]
[OK]
[OK]
[OK]
[OK]
[OK]
[OK]
[OK]
[OK]
[OK]
[OK]
[OK]
[OK]
[OK]
[OK]
[OK]
[OK]
[OK]
[OK]
[OK]
[OK]
[OK]
[OK]
[OK]
[OK]
[OK]
[OK]
[OK]
[OK]
[OK]
[OK]
[OK]
[OK]
[OK]
[OK]
[OK]
[OK]
[OK]
[OK]
[OK]
[OK]
[OK]
[OK]
[OK]

docker search --limit 100 mongodb|wc -l
101

To refine your search, you can filter it using the

--filter

option.

Let's search for the best Mongodb images according to the community (images with more than 5 stars):
docker search --filter=stars=5

mongo

--

Tutum, Frodenas and Bitnami have the most popular Mongodb images:
NAME
tutum/mongodb
frodenas/mongodb
bitnami/mongodb

DESCRIPTION
MongoDB..
A Docker..
Bitnami..

STARS
166
12
5

OFFICIAL
AUTOMATED
[OK]
[OK]
[OK]

Images could be official or not, just like any Open Source project, Docker public images are made by anyone who
may have access to the Docker Hub so consider double-checking images before using them in your production
servers.
Official images could be filtered in this way:
docker search --filter=is-official=true

NAME
mongo
mongo-express

mongo

DESCRIPTION
MongoDB document databases provide high av...
Web-based MongoDB admin interface, written...

STARS
2616
89

OFFICIAL
[OK]
[OK]

AUTOMATED

Filter output based on these conditions:
stars=
is-automated=(true|false)
is-official=(true|false)
The images you can find are even public images or your own private images stored in Docker Hub. In the next
section we will use a private registry.

Finding Private Images
If you have a private registry, you can use the Docker API:
curl -X GET http://localhost:5000/v1/search?q=ubuntu

Or use Docker client:
docker search localhost:5000/ubuntu

In the last example, I am using a local private registry without any SSL encryption, change
secure remote server domain or IP address.

Pulling Images
If you want to pull the latest tag of an image, say Ubuntu, you just need to type:
docker pull ubuntu

But Ubuntu image has many tags like: devel, 16.10, 14.04, trusty ..etc

localhost

by your

You can find all of the tags here: https://hub.docker.com/r/library/ubuntu/tags/
To pull the development image, type:
docker pull ubuntu:devel

You will see the tagged image in your download:
devel: Pulling from library/ubuntu
8e21f82d32cf:
54d6ba364cfb:
451f851c9d9f:
55e763f0d444:
b112a925308f:

Pulling fs layer
Pulling fs layer
Pulling fs layer
Waiting
Waiting

Removing Images
Simply type:
docker rmi $(docker images -q)

We can also safely remove only dangling images by typing:
docker rmi $(docker images -f "dangling=true" -q)

Creating New Images Using Dockerfile
The Dockerfile is the file that Docker reads in order to build images. It is a simple text file with a specific
instructional language to assemble the different layers of an image.
You can find below a list of the different instructions that could be used to create an image and then we will see how
to build the image using Dockerfile.

FROM
In the Dockerfile, the first line should start with this instruction.
This is not widely used but when we want to build multiple images, we can have multiple FROM instruction in the
same Dockerfile.
FROM :

Example:
FROM ubuntu:14.04

If the tag is not specified then Docker will download the latest tagged image.

MAINTAINER
The maintainer is not really an instruction but it indicates the name, email or website of the image maintainer. It is
the equivalent of author in code documentations.
MAINTAINER 

Example:
MAINTAINER Aymen EL Amri - @eon01

RUN
You can run commands (like Linux CLI commands) or executables.
RUN 

or
RUN ["", "", "", ... ,""]

The first run block will run a command just like any other Linux command using
commands are executed using cmd /S /C shell.

/bin/sh -c

shell. Windows

Examples:
RUN ls -l

During the build process, you will see the output of the command:
Step 4 : RUN ls -l
---> Running in b3e87d26c09a
total 64
drwxr-xr-x
2 root root 4096
drwxr-xr-x
2 root root 4096
drwxr-xr-x
5 root root 360
drwxr-xr-x 64 root root 4096
drwxr-xr-x
2 root root 4096
drwxr-xr-x 12 root root 4096
drwxr-xr-x
2 root root 4096
drwxr-xr-x
2 root root 4096
drwxr-xr-x
2 root root 4096
drwxr-xr-x
2 root root 4096
dr-xr-xr-x 267 root root
0
drwx-----2 root root 4096
drwxr-xr-x
8 root root 4096
drwxr-xr-x
2 root root 4096
drwxr-xr-x
2 root root 4096
dr-xr-xr-x 13 root root
0
drwxrwxrwt
2 root root 4096
drwxr-xr-x 11 root root 4096
drwxr-xr-x 13 root root 4096

Oct
Apr
Nov
Nov
Apr
Oct
Oct
Oct
Apr
Oct
Nov
Oct
Oct
Oct
Oct
Nov
Oct
Oct
Oct

6
10
3
3
10
6
6
6
10
6
3
6
13
13
6
3
6
13
13

07:47
2014
21:55
21:55
2014
07:47
07:47
07:46
2014
07:46
21:55
07:47
21:13
21:13
07:46
21:54
07:47
21:13
21:13

bin
boot
dev
etc
home
lib
lib64
media
mnt
opt
proc
root
run
sbin
srv
sys
tmp
usr
var

The same command could be called like this:
RUN ["/bin/sh", "-c", "ls", "-l"]

The latter is called the exec from.

CMD
CMD command helps you to identify which executable should be run when a container is started from your image.
Like the run command, you can use the shell from:
CMD    .. 

The exec from:
CMD ["", "", "", ... ,""]

or as a default parameter to ENTRYPOINT instruction (explained later):
CMD [<"param1">,<"param2"> ..

<"paramN">]

Using CMD, the same instruction (that we used in RUN) will be run but not during the build, it will be executed
during the execution of the container.
Example:

CMD ["ls", "-l"]

LABEL
The LABEL instruction is useful in case you want to add metadata to a given image.
LABEL = = .. =

Labels are key-value pairs.
Not only Docker's images can have labels but also:
Docker containers
Docker daemons
Docker volumes
Docker networks (and Swarm networks)
Docker Swarm nodes
Docker Swarm services

EXPOSE
When running an application or a service inside a Docker container, it is obvious that this service needs to listen and
send its data to the outside. Imagine we a php/Mysql web application in a host. We created two two containers, a
Mysql container and a webserver container, say Apache.
At this stage, neither the DB server cannot communicate with the web server and the web server can not request a
database. In addition, both servers are not accessible from the outside of the host.
EXPOSE 3306

EXPOSE 80, 443

Now if we would like opening the ports 80 and 443 so that Nginx can be reached from outside the host, we can use

the same instruction with the -p or -P flag.
-p publish a range of ports or the and -P publish all of the exposed ports. You can expose one port number and
publish it externally under another number.
We are going to see this later in this book but keep in mind that exposing ports in the Dockerfile is not mapping
ports to host's network interfaces.

To expose a list of ports, say 3000, 3001, 3002, .., 3999, 4000, you can use this:
EXPOSE 3000-4000

ENV
The ENV is an instruction that sets environment variables. It is the equivalent of Linux:
export variable=value

ENV works with

/

pair. You can use ENV instruction in two manners:

ENV variable1 This is value1
ENV variable2 This is value2

or like this:
ENV variable1="this is value1" variable2="this is value2"

If you are used to Dockerfile and building your own imafes, you may have seen this:
ENV DEBIAN_FRONTEND noninteractive

This is discouraged because the environment variable persists after the build.
However, you can set it via ARG ( ARG instruction is explained later in this section).

ARG DEBIAN_FRONTEND=noninteractive

ADD
As its name may indicate, the ADD instruction will add files from the host to the guest.
ADD is part of Docker from the beginning and supports a few additional tricks (compared to COPY) beyond simply
copying files.
ADD has two forms:
ADD ... 

And if your path contains whitespaces, you may use this form:
ADD ["",... ""]

You can use some tricks like the * operator:
ADD /var/www/* /var/www/

Or the ? operator to replace a character. If we want to copy all of the following files:
-rwxrwxrwx
-rwxrwxrwx
-rwxrwxrwx
-rwxrwxrwx
-rwxr-xr-x
-rwxrwxrwx
-rwxrwxrwx
-rwxrwxrwx
-rwxrwxrwx

1
1
1
1
1
1
1
1
1

eon01
eon01
root
eon01
eon01
eon01
eon01
eon01
eon01

sudo
sudo
root
sudo
sudo
sudo
sudo
sudo
sudo

11474
35163
5233
22411
13550
3235
395
466
272

Nov
Nov
Nov
Nov
Nov
Nov
Nov
Nov
Nov

3
3
3
3
6
6
3
3
3

00:50
00:50
00:50
00:50
02:26
01:15
00:51
00:51
00:51

chapter1
chapter2
chapter3
chapter4
chapter5
chapter6
chapter7
chapter8
chapter9

We can use this:
ADD /home/eon01/painlessdocker/chapter? /var/www

Using Docker ADD, you can use download files from links.
ADD https://github.com/eon01/PainlessDocker/blob/master/README.md /var/www/index.html

This instruction will copy the html file called

README.md

to

index.html

under

/var/www

Docker's ADD will not discover the URL for files, you should not use something like

https://github.com

The ADD instruction will recognize formats like gzip, bzip2 or xz.. So if the is a local tar archive is is directly
unpacked as a directory ( tar -x ).
We haven't seen WORKDIR yet but keep in mind the following point. If we would like to copy
/var/www/painlessdocker/ we should do this:
ADD index.html /var/www/painlessdocker/

index.html

to

But, if our WORKDIR instruction referenced

/var/www/

as out work directory, we can use the following instruction:

ADD index.html painlessdocker/

This form that uses an absolute path in the directory will not work:
ADD ../index.html /var/www/

COPY
Like the ADD instruction, the COPY instruction have two forms:
COPY ... 

and
COPY ["",... ""]

The second form is used for files path with spaces.
COPY ["/home/eon01/Painless Docker.html", "/var/www/index.html"]

You can use some tricks like the * operator:
COPY /var/www/* /var/www/

Or the ? operator to replace a character. If we want to add all of the following files:
-rwxrwxrwx
-rwxrwxrwx
-rwxrwxrwx
-rwxrwxrwx
-rwxr-xr-x
-rwxrwxrwx
-rwxrwxrwx
-rwxrwxrwx
-rwxrwxrwx

1
1
1
1
1
1
1
1
1

eon01
eon01
root
eon01
eon01
eon01
eon01
eon01
eon01

sudo
sudo
root
sudo
sudo
sudo
sudo
sudo
sudo

11474
35163
5233
22411
13550
3235
395
466
272

Nov
Nov
Nov
Nov
Nov
Nov
Nov
Nov
Nov

3
3
3
3
6
6
3
3
3

00:50
00:50
00:50
00:50
02:26
01:15
00:51
00:51
00:51

chapter1
chapter2
chapter3
chapter4
chapter5
chapter6
chapter7
chapter8
chapter9

We can use this:
COPY /home/eon01/painlessdocker/chapter? /var/www

If we would like to copy

index.html

to

/var/www/painlessdocker/

we should do this:

COPY index.html /var/www/painlessdocker/

But, if our WORKDIR instruction referenced

/var/www/

as out work directory, we can use the following instruction:

COPY index.html painlessdocker/

This form that uses an absolute path in the directory will not work:
COPY ../index.html /var/www/

Unlike the ADD instruction, the COPY instruction will not work with archive files and URLs.

ENTRYPOINT
A question that you may ask is what happens when a container starts ?
Imagine we have a tiny Python server that the container should start, the ENTRYPOINT should be :
python -m SimpleHTTPServer

Or a Node.js application to run it inside the container:
node app.js

The ENTRYPOINT is what helps us to start a server in this case or to execute a command in the general case.
That's why we need the ENTRYPOINT instruction.
Whenever we start a Docker container, declaring what command should be executed is important. Otherwise the
container will shutdown.
In the general case, the ENTRYPOINT is declared in the Dockerfile.
This instruction has two forms.
The exec form is the preferred one:
ENTRYPOINT [<"executable">, <"param1">, <"param2">.... <"paramN">]

The second one is the shell from:
ENTRYPOINT    .. 

Example:

ENTRYPOINT ["node", "app.js"]

Let's take the example of a simple *Python* application.
``` python
print("Hello World")

We want the container to run the Python script as an ENTRYPOINT.
Now that we know some instructions like FROM, COPY and ENTRYPOINT, we can create a Dockerfile just using
those instructions.
echo "print('Hello World')" > app.py
touch Dockerfile

This is the content of the Dockerfile:
FROM python:2.7
COPY app.py .
ENTRYPOINT python app.py

Reading this Dockerfile, we understand that:
The image will be downloaded from Docker Hub python:2.7
The file app.py will be copied inside the container
The command python app.py will be executed when the container upstarts.
We haven't seen that yet in Painless Docker but this is the process of building and running the command ( we are
going to see the build and the run commands will be detailed later in this book.).
The build:
docker build .
Sending build context to Docker daemon 3.072 kB
Step 1 : FROM python:2.7
---> d0614bfb3c4e
Step 2 : COPY app.py .
---> Using cache
---> 6659a70e1775
Step 3 : ENTRYPOINT python app.py
---> Using cache
---> 236e1648a508
Successfully built 236e1648a508

The run:
docker run 236e1648a508
Hello World

Notice that the Python script was executed just after running the container.

VOLUME
The VOLUME instruction creates an external mount point from an internal directory. Any external volume mounted
using this instruction could be used by another Docker container.
We can use this form:
VOLUME[<"Directory">]

This form

VOLUME   .. 

Or a JSON array.
But why ?
Docker containers are ephemeral so to keep the data persistent even through container restarts, stops
or disappear.
By default, a Docker container does not share its data with the host, volumes allow the host to access
to the container data.
By default, two containers even running in the same host cannot share data so sharing files between a
container in the form of a docker volume will allow other containers to access its data.
Setting up the permission or the ownership of a volume should be done before the VOLUMES instruction in th
Dockerfile.
For example, this form of setting up the owner is wrong.
VOLUME /app
ADD app.py /app/
RUN chown -R foo:foo /app

However the following Dockerfile has a good syntax:
RUN mkdir /app
ADD app.py /app/
RUN chown -R foo:foo /app
VOLUME /app

A good scenario to use Docker volumes is databases containers, where data is mounted outside the container. This
allows making an easy backup to the database.

USER
When running a command you may need doing it using another user (not the default user which is the Root user).
This is when the USER instruction shall be used.
USER is used like this:
USER 

So, Root is the default user and Docker has a full access to the host system. You may consider USER as a security
option to consider.
Normally an image should never use the Root user but another user that you choose using the instruction USER.
Example:
USER my_user

Sometimes, you need to run a command using Root, you can simply switch between different users:

USER root
RUN 
USER my_user

The USER instruction applies to RUN, CMD and ENTRYPOINT instructions so that any command run by one of
these instructions is attributed to the chosen user.

WORKDIR
WORKDIR is use this way:
WORKDIR 

This instruction sets the a working directory so that any of the following commands: RUN, CMD, ENTRYPOINT,
COPY & ADD will be executed in this directory.
Example:
To copy

index.html

to

/var/www

, you may write this:

ADD index.html /var/www

Note that if the WORKDIR does not exist, it will be created.

ARG
If you want to assign a value to a variable but just during the build, you can use the ARG instruction.
The syntax is :
ARG [=]

or
ARG 

You can declare variables that you can use during the build with
later.

docker build

Example:
ARG time

You can also set the value of the arguments (variables) in the Dockerfile.
ARG time=3s

command that we are going to see

ONBUILD
Software should be built automatically that's why Docker has the ONBUILD instruction. It is a trigger instruction
executed when the image is used as the base image for another build.
ONBUILD 

The trigger will be executed in the context of the downstream build, as if it had been inserted immediately after the
FROM instruction in the downstream Dockerfile.
The ONBUILD instruction is available in Docker since its version 0.8.
Let's dive int details. First of all, let's define what a child image is :
If the image A contains an ONBUILD instruction and if imageB uses the imageA as its base image in order to add
other instructions (and layers) upon it then imageB is a child image of imageA.
This is the imageA Dockerfile
FROM ubuntu:16.04
ONBUILD RUN echo "I will be automatically executed only in the child image"

THis is the imageB Dockerfile
FROM imageA

If you build the imageA based on Ubuntu 16.04, the ONBUILD instruction will not run the RUN echo "<..>" . But
when you build the imageB, the same instruction will be run just after the execution of the FROM instruction.
This is an example of the imageB build output where we see the message printed automatically:
Uploading context 4.51 kB
Uploading context
Step 0 : FROM imageA
# Executing 1 build triggers
Step onbuild-0 : RUN echo "I will be automatically executed only in the child imager"
---> Running in acefe7b39c5
I will be automatically executed only in the child image

STOPSIGNAL
The STOPSIGNAL instruction let you set the system call signal that will be sent to the container to exit.
STOPSINGAL 

This signal can be a valid unsigned number like 9 or a signal name like SIGNAME, SIGKILL, SIGINT ..etc
SIGTERM is the default in Docker and it is and equivalent to running

kill 

.

In the following example, let's change it by SIGINT ( the same signal sent when pressing ctrl-C ):
FROM ubuntu/16:04
STOPSIGNAL SIGINT

HEALTHCHECK
The HEALTHCHECK instruction is one of the useful instruction I have been using since the version 1.12.
It has two forms. Either to check a container health by running a command inside the container:
HEALTHCHECK [OPTIONS] CMD 

or to disable any healthcheck ( all healthchecks inherited from the base image will be disabled ).
HEALTHCHECK NONE

There are 3 options that we can use before the CMD command.
--interval=

The healthcheck will be executed every unit of time. The default interval duration is 30s.
--timeout=

The default timeout duration is 30s. The healthcheck will be timeout after unit of time.
--retries=N

The default number of retries is 3. The healthcheck could be failed but no more than times.
An example of a Docker healthcheck that will run every 1 minute with a check that does not take longer than 3
seconds before it will be considered as failed if the same check will be repeated for more than 3 time :
HEALTHCHECK --interval=1m --timeout=3s CMD curl -f http://localhost/ || exit 1

SHELL
When using Docker, the default shell that executes all of the commands is "/bin/sh -c". This means that
will be in reality run inside the container like this:
/bin/sh -c ls -l

The default shell for Windows is:
cmd /S /C

SHELL instruction must be written in JSON form in the Dockerfile.
SHELL [<"executable">, <"parameters">]

CMD ls -l

This instruction could be interesting for Windows users to choose between cmd and powershell.
Example:
SHELL ["powershell", "-command"]
SHELL ["cmd", "/S"", "/C"]

In nix you can of course work with bash, zsh, csh, tcsh* ..etc for example:
SHELL ["/bin/bash", "-c"]

ENTRYPOINT VS CMD
Both CMD and ENTRYPOINT instructions allows us to define a command that will be executed once a container
starts.
Back to the CMD instruction: We have seen that the CMD could be a default parameter to ENTRYPOINT
instruction when we use this from :
CMD [<"param1">,<"param2"> ..

<"paramN">]

This is a simple Dockerfile:
FROM python:2.7
COPY app.py .
ENTRYPOINT python app.py

You may say that the Dockerfile could be written like this:
FROM python:2.7
COPY app.py .
ENTRYPOINT ["python"]
CMD ["app.py"]

Good guess .. But when you will run the container it will show you an error.
The CMD command in this from should be called like this:
FROM python:2.7
COPY app.py .
ENTRYPOINT ["/usr/bin/python"]
CMD ["app.py"]

As a conclusion,
ENTRYPOINT python app.py

could be called like this
ENTRYPOINT ["/usr/bin/python"]
CMD ["app.py"]

CMD and ENTRYPOINT could be used alone or together but in all cases, you should use one of them at least.
Using neither CMD not ENTRYPOINT will fail the execution of the container.
You can find almost the same table in the official Docker documentation, but this is the best way to understand all of
the possibilities:
ENTRYPOINT
[“entrypoint_exec”,
“entrypoint_param1”]
Will generate an /bin/sh -c entrypoint_exec
entrypoint_exec
error
entrypoint_param1
entrypoint_param1
/bin/sh -c entrypoint_exec
entrypoint_exec
cmd_exec
entrypoint_param1 cmd_exec
entrypoint_param1 cmd_exec
cmd_param1
cmd_param1
cmd_param1
/bin/sh -c entrypoint_exec
entrypoint_exec
cmd_param1
entrypoint_param1 cmd_param1
entrypoint_param1 cmd_param1
cmd_param2
cmd_param2
cmd_param2
/bin/sh -c
/bin/sh -c entrypoint_exec
entrypoint_exec
cmd_exec
entrypoint_param1 /bin/sh -c cmd_exec entrypoint_param1 /bin/sh -c
cmd_param1
cmd_param1
cmd_exec cmd_param1
No
ENTRYPOINT

No CMD
CMD
[“cmd_exec”,
“cmd_param1”]
CMD
[“cmd_param1”,
“cmd_param2”]
CMD cmd_exec
cmd_param1

ENTRYPOINT entrypoint_exec
entrypoint_param1

Building Images
The Base Image
Probably the smallest Dockerfile (not the smallest image) is the following one:
FROM 

Docker needs an image to run : no image, no container.
The base image is an image which you add layers on the top of it to create another image containing your
application. As seen in the last chapter, an image is a set of layers and the FROM  is the necessary layer to
create the image.
Using www.imagelayers.io we can visualize an image online. Let's take te example of
find on Docker Hub website just by concatenating:

tutum/hello-world

https://hub.docker.com/r/

and
tutum/hello-world

which mean:
https://hub.docker.com/r/tutum/hello-world/

The Dockerfile of this image is the following:
FROM alpine
MAINTAINER support@tutum.co
RUN apk --update add nginx php-fpm && \
mkdir -p /var/log/nginx && \
touch /var/log/nginx/access.log && \
mkdir -p /tmp/nginx && \
echo "clear_env = no" >> /etc/php/php-fpm.conf
ADD www /www
ADD nginx.conf /etc/nginx/
EXPOSE 80
CMD php-fpm -d variables_order="EGPCS" && (tail -F /var/log/nginx/access.log &) && exec nginx -g "daemon off;"

The base image of this simple application is Alpine.
This image of 18 MiB has:
7 unique layers
an average layer of 3 MiB
its largest layer is 13 MB

that you can

Let's take another image and use the Docker
docker

history

command to see its layers

pull nginx

This will pull the latest nginx image from the Docker Hub.
docker history nginx

will show the different layers of nginx :

Docker history command show you the history of an image and it s different layers. You can see more information
with human readable output about an image by using :
docker history --no-trunc -H nginx

with:
-H, --human
Print sizes and dates in human readable format (default true)
--no-trunc
Don't truncate output

Let's take a look at the output:

Dockerfile
The Dockerfile is a kind of a script file with different instructive commands and arguments that describe how you
image base image will be at the end of the build. It is possible to run an image directly without building it (like a
public or a private image from Docker Hub or a private repository), but if you want to have your own specific
images and if you want to organize your deployments while distributing the same image to developers and QA
teams, creating a Dockerfile for your application is a good point to start.
The first rule: A Dockerfile should start with the FROM instruction which explains that nothing could be done
without having a base image. The syntax to create a Dockerfile is quite simple and explicit since they should be
followed to the letter. If you need to execute more things than the Docker instructions permit, than you can just RUN
nix commands or use shell scripts with CMD and ENTRYPOINT*.
We wen through the different instructions and the differences between them, you should be able to create Dockerfile
just using this, but we are going to see more examples later like creating micro images for Python and Node.js or
like in the Mongodb example.

Creating An Image Build Using Dockerfile
So we have seen the different instructions that can help us create a Dockerfile. Now in order to have a complete
image, we should build it.
The command to build a Docker image is:
docker build .

The '.' is indicating that the Dockerfile is in the same directory where you are running the build command which is
the context.
The context are simply your local files.
ls -l .

Dockerfile
app/
scripts/

The context is the directory and all the subdirectories where you execute the

docker build

command.

If you are executing the build from a different directory you can use -f:
docker build -f /path/to/the/Dockerfile/TheDockerfile /path/to/the/context/

Example:
If your Dockerfile is under

/tmp

and your files are in the

/app

directory, the command should be:

docker build -f /tmp/Dockerfile /app

Optimizing Docker Images
You can find many images of the same application in the Internet but not all of them are optimized , they can be
really big, they may take time to build or to send through network and of course your deployment time could
increase.
Layers are actually what decides the size of a Docker image and optimizing your Docker clusters start from
optimizing the layers of your image.
This image has three layers:
FROM ubuntu
RUN apt-get update -y
RUN apt-get install python -y

While this one has only two layers:
FROM ubuntu
RUN apt-get update -y && apt-get install python -y

Both images are installing Python in a Ubuntu docker image.
Docker images can get really big. Many are over 1G in size. How do they get so big? Do they really need to be this
big? Can we make them smaller without sacrificing functionality?

Tagging Images
Like git, Docker has the ability to tag specific points in history as being important.
Before using private Docker registry or Docker Hub, you should first use

docker login

information.

Docker Hub:

docker login
Login with your Docker ID to push and pull images from Docker Hub. If you don't have a Docker ID, head over to https://hub.docker.com to create o
Username (eon01):
Password:
Login Succeeded

Private registry:

docker login https://localhost:5000
Username: admin
Password:
Login Succeeded

Typically Docker developers use the tagging functionality to mark a release or a version. In this section, you’ll learn
how to list images tags and how to create new tags.
When you type

docker images

REPOSITORY
mongo
alpine
docker/whalesay
hello-world

TAG
latest
latest
latest
latest

you can get a list of the images you have on your laptop/server.
IMAGE ID
c5185a594064
baa5d63471ea
fb434121fc77
91c95931e552

CREATED
SIZE
5 days ago
342.7 MB
5 weeks ago
4.803 MB
4 hours ago
247 MB
5 weeks ago
910 B

You can notice in the output of the latter command that every image has a unique id but also a tag.
docker images|awk {'print $3'}|tail -n +2

c5185a594064
baa5d63471ea
fb434121fc77
91c95931e552

If you would like to download the same images, you should pull them with the right tags:
docker
docker
docker
docker

pull
pull
pull
pull

mongo:latest
alpine:latest
docker/whalesay:latest
hello-world:latest

As you can see in the docker images output, the id of the Alpine image is
give the image a new tag using the following syntax:

baa5d63471ea

. We are going to use this to

docker tag  /:

You can also use
docker tag : /:

Example:
docker tag baa5d63471ea alpine:me

Now when you list your images, you will notice both of the last and original tags are listed:
alpine
alpine

latest
me

You can try also:

baa5d63471ea
baa5d63471ea

5 weeks ago
5 weeks ago

4.803 MB
4.803 MB

docker tag mongo:latest mongo:0.1

Then list your images and you will find the new mongo tag:
REPOSITORY
mongo
mongo
alpine
alpine

TAG
0.1
latest
latest
me

IMAGE ID
c5185a594064
c5185a594064
baa5d63471ea
baa5d63471ea

Like in a simple git flow you can pull, update, commit and push your changes to a remote repository. The commit
operation can happen on a running container, that's why we are going to run the mongo container. In this section, we
haven't seen yet how to run containers in production environments but we are going to use the run command now
and see all of its details later in another chapter.
docker run -it -d -p 27017:21017 --name mongo mongo

Verify Mongodb container is running:
docker ps

CONTAINER ID
d5faf7fd8e4d

(1):

IMAGE
mongo

COMMAND
"/entrypoint.sh mongo"

PORTS
(1)

NAMES
mongo

27017/tcp, 0.0.0.0:27017->21017/tcp

We will use the container id d5faf7fd8e4d in order to use the commit command, sure we haven't had any changes
made to the running container until now, but this is just a test to show you the basic command usage. Note that we
an change or keep the original tag when committing:
In the following example, the Docker image that the container
mongo:0.2 :

d5faf7fd8e4d

is running will be tagged with a new tag

docker commit d5faf7fd8e4d mongo:0.2

Type

docker images

for your verifications and notice the new tag

REPOSITORY
mongo
localhost:5000/user/mongo
mongo
mongo
localhost:5000/mongo
registry
alpine
alpine

TAG
0.2
0.1
0.1
latest
0.1
latest
latest
me

mongo:0.2

:

IMAGE ID
d022237fd80d
c5185a594064
c5185a594064
c5185a594064
c5185a594064
c9bd19d022f6
baa5d63471ea
baa5d63471ea

A running container could also have some changes while running. Let's take the example of the Mongodb container,
we will add an administrative account to the running Mongodb instance. Log into the container:
docker exec -it mongo bash

Inside your running container, connect to your running database using

mongo

command:

root@d5faf7fd8e4d:/# mongo
MongoDB shell version: 3.2.11
connecting to: test
Server has startup warnings:
I CONTROL [initandlisten]
I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
I CONTROL [initandlisten] **
We suggest setting it to 'never'
I CONTROL [initandlisten]
I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
I CONTROL [initandlisten] **
We suggest setting it to 'never'
I CONTROL [initandlisten]
>

Now add a new administrator identified by login root and password 958d12fc49437db0c7baac22541f9b93 with the
administrative role root:
> use admin
switched to db admin
> db.createUser(
...
{
...
user: "root",
...
pwd: "958d12fc49437db0c7baac22541f9b93",
...
roles:["root"]
...
})
Successfully added user: { "user" : "root", "roles" : [ "root" ] }
> exit
bye

Exit your container:
root@d5faf7fd8e4d:/# exit
exit

Now that we had made important changes to our running container, commit those changes. You can use a private
registry or Docker Hub, but I am going to use my public Docker Hub account in this example:
docker commit d5faf7fd8e4d eon01/mongodb
sha256:269685eeaecdea12ddd453cf98685cad1e6d3c76cccd6eebb05d3646fe496688

After the commit operation, we are sure that our changes are saved, we can push the image:
docker push eon01/mongodb

The push refers to a repository [docker.io/eon01/mongodb]
c01c6c921c0b: Layer already exists
80c558316eec: Layer already exists
031fad254fc0: Layer already exists
ddc5125adfe9: Layer already exists
31b3084f360d: Layer already exists
77e69eeb4171: Layer already exists
718248b95529: Layer already exists
8ba476dc30da: Layer already exists
07c6326a8206: Layer already exists
fe4c16cbf7a4: Layer already exists
latest: digest: sha256:ee50ec95fd490d60796d7782a9348ef824d84110beea4f86ced1ed15a1c8976c size: 2406

Notice the The push refers to a repository [docker.io/eon01/mongodb] . This telling us that you can find this public image
on: https://hub.docker.com/r/eon01/mongodb/

Using Docker Hub we can add a description, a README file, change the image to a private one (paid feature) ..etc
If you execute a pull command on the same remote image, you can notice that nothing will be downloaded because
you already have all of the image layers locally:
docker pull eon01/mongodb
Using default tag: latest
latest: Pulling from eon01/mongodb
Digest: sha256:ee50ec95fd490d60796d7782a9348ef824d84110beea4f86ced1ed15a1c8976c
Status: Image is up to date for eon01/mongodb:latest

We can run it like this:
docker run -it -d -p 27018:27017 --name mongo_container eon01/mongodb

Notice that we used the port mapping 27018:27017 because the first Mongodb container is mapped to the host port
27017 and it is impossible to map two containers to the same local port - same thing for the container name.
We have not seen this detail yet but we will go through all of these details in the next chapter.
docker ps
CONTAINER ID
ff4eccdd6bee
d5faf7fd8e4d
3118896db039

IMAGE
eon01/mongodb
mongo
registry

COMMAND
"/entrypoint.sh mongo"
"/entrypoint.sh mongo"
"/entrypoint.sh /etc/"

PORTS
(1)
(2)
(3)

NAMES
zen_dubinsky
mongo
my_registry

(1):

21017/tcp, 0.0.0.0:27018->27017/tcp

(2):

27017/tcp, 0.0.0.0:27017->21017/tcp

(3):

0.0.0.0:5000->5000/tcp

If you log into your container:
docker exec -it mongo_container bash

Start Mongodb:
root@db926ae25f1e:/# mongo
MongoDB shell version: 3.2.11
connecting to: test
Server has startup warnings:
I CONTROL [initandlisten]
I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
I CONTROL [initandlisten] **
We suggest setting it to 'never'
I CONTROL [initandlisten]
I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
I CONTROL [initandlisten] **
We suggest setting it to 'never'
I CONTROL [initandlisten]

List users:
> use admin
switched to db admin
> db.getUsers()
[
{
"_id" : "admin.root",
"user" : "root",
"db" : "admin",
"roles" : [
{
"role" : "root",
"db" : "admin"
}
]
}
]
>

You can see that the changes you made are already stored in the container.
In the same way, you can add other changes, commit, tag (if you need it) and push to your Docker Hub or private
registry.
This is what we have done until now: we had an image mongo that we created from a build (we are going to see the
Dockerfile later in this chapter), run the container, made some changes and push it to a repository.

Your Private Registry
If you are using a private Docker repository, just add your host domain/IP like this:
docker tag : /:

Example:

docker tag mongo:latest localhost:5000/mongo:0.1

Let's run a simple local private registry to test this:
docker run -d -p 5000:5000 --name my_registry registry

Unable to find image 'registry:latest' locally
latest: Pulling from library/registry
3690ec4760f9: Already exists
930045f1e8fb: Pull complete
feeaa90cbdbc: Pull complete
61f85310d350: Pull complete
b6082c239858: Pull complete
Digest: sha256:1152291c7f93a4ea2ddc95e46d142c31e743b6dd70e194af9e6ebe530f782c17
Status: Downloaded newer image for registry:latest
3118896db039c26a74127031eefd42264e310d7cdc435e126fa8630bf8ee8c60

Verify that you are really running this with
CONTAINER ID
IMAGE
3118896db039
registry
>5000/tcp
my_registry

docker ps

:

COMMAND
CREATED
"/entrypoint.sh /etc/"
2 minutes ago

STATUS
Up 2 minutes

PORTS

NAMES
0.0.0.0:5000-

Tag your image with your private URL:
docker tag mongo:latest localhost:5000/mongo:0.1

Check your new tag with the
REPOSITORY
localhost:5000/mongo
mongo
mongo
registry
alpine
alpine

docker images

TAG
0.1
0.1
latest
latest
latest
me

command:

IMAGE ID
c5185a594064
c5185a594064
c5185a594064
c9bd19d022f6
baa5d63471ea
baa5d63471ea

Note that if you are testing private Docker registry, your default username/password are admin/admin.

Optimizing Images
You are probably used to Ubuntu (or any other major distribution) so you will may be use Ubuntu on your Docker
container. Ostensibly, your image is fine, it is using a stable distribution and your are just running Ubuntu inside
your container.
FROM ubuntu
ADD app /var/www/
CMD ["start.sh"]

The problem is that you don't really need a complete OS, you installed all Ubuntu files but you are not going to use
them, your container does not need all of them.
When you build an image, generally you will configure CMD or ENTRYPOINT or both of them with something
that will be executed at the container startup. So the only processes that will be running inside the container is the
ENRYPOINT command, and all processes that it spawns, all of the other OS processes will not run. And I am not
sure you will need them.
This is a part of the output of
USER
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root

PID %CPU %MEM
1 0.0 0.0
2 0.0 0.0
3 0.0 0.0
5 0.0 0.0
7 0.0 0.0
8 0.0 0.0
9 0.0 0.0
10 0.0 0.0
11 0.0 0.0
12 0.0 0.0
13 0.0 0.0
14 0.0 0.0
15 0.0 0.0
16 0.0 0.0
17 0.0 0.0
18 0.0 0.0
19 0.0 0.0
20 0.0 0.0
21 0.0 0.0
22 0.0 0.0
23 0.0 0.0
24 0.0 0.0
25 0.0 0.0
26 0.0 0.0
27 0.0 0.0
28 0.0 0.0
29 0.0 0.0
31 0.0 0.0
32 0.0 0.0
33 0.0 0.0
34 0.0 0.0
36 0.0 0.0
37 0.0 0.0
38 0.0 0.0
39 0.0 0.0
41 0.0 0.0
42 0.0 0.0
43 0.0 0.0
44 0.0 0.0
45 0.0 0.0
46 0.0 0.0
47 0.0 0.0
48 0.0 0.0
49 0.0 0.0
50 0.0 0.0
51 0.0 0.0
52 0.0 0.0
53 0.0 0.0
54 0.0 0.0
55 0.0 0.0
56 0.0 0.0
60 0.0 0.0
61 0.0 0.0
62 0.0 0.0

ps aux

VSZ
34172
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0

of my current Ubuntu system:

RSS
3980
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0

TTY
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?

STAT
Ss
S
S
S<
S
S
S
S
S
S
S
S
S
S
S
S
S
S
S
S
S
S
S
S
S
S
S
S<
S
S
S
S<
S
S
S
S<
S<
S
S<
S
S<
SN
SN
S<
S<
S<
S<
S<
S
S<
S<
S
S
S

START
00:36
00:36
00:36
00:36
00:36
00:36
00:36
00:36
00:36
00:36
00:36
00:36
00:36
00:36
00:36
00:36
00:36
00:36
00:36
00:36
00:36
00:36
00:36
00:36
00:36
00:36
00:36
00:36
00:36
00:36
00:36
00:36
00:36
00:36
00:36
00:36
00:36
00:36
00:36
00:36
00:36
00:36
00:36
00:36
00:36
00:36
00:36
00:36
00:36
00:36
00:36
00:36
00:36
00:36

TIME
0:02
0:00
0:00
0:00
0:55
0:17
0:15
0:17
0:12
0:00
0:00
0:00
0:00
0:00
0:00
0:00
0:00
0:00
0:00
0:00
0:00
0:00
0:00
0:00
0:00
0:00
0:00
0:00
0:00
0:00
0:00
0:00
0:00
0:00
0:00
0:00
0:00
0:00
0:00
0:00
0:00
0:00
0:20
0:00
0:00
0:00
0:00
0:00
0:00
0:00
0:00
0:10
0:00
0:00

COMMAND
/sbin/init
[kthreadd]
[ksoftirqd/0]
[kworker/0:0H]
[rcu_sched]
[rcuos/0]
[rcuos/1]
[rcuos/2]
[rcuos/3]
[rcuos/4]
[rcuos/5]
[rcuos/6]
[rcuos/7]
[rcu_bh]
[rcuob/0]
[rcuob/1]
[rcuob/2]
[rcuob/3]
[rcuob/4]
[rcuob/5]
[rcuob/6]
[rcuob/7]
[migration/0]
[watchdog/0]
[watchdog/1]
[migration/1]
[ksoftirqd/1]
[kworker/1:0H]
[watchdog/2]
[migration/2]
[ksoftirqd/2]
[kworker/2:0H]
[watchdog/3]
[migration/3]
[ksoftirqd/3]
[kworker/3:0H]
[khelper]
[kdevtmpfs]
[netns]
[khungtaskd]
[writeback]
[ksmd]
[khugepaged]
[crypto]
[kintegrityd]
[bioset]
[kblockd]
[ata_sff]
[khubd]
[md]
[devfreq_wq]
[kswapd0]
[fsnotify_mark]
[ecryptfs-kthrea]

root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root

74
75
77
99
100
108
149
166
168
169
170
171
172
173
174
252
253
254
256
293
471
509
510
540
547
595
622
623

0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0

0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0

0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
28948
0
0
0
19480
52116
0
0
0

0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
2080
0
0
0
1448
2244
0
0
0

?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?

S<
S<
S<
S<
S<
S
S<
S
S<
S
S<
S
S<
S
S<
S<
S
S<
S<
S
S<
S
S<
S
Ss
S<
S<
S<

00:36
00:36
00:36
00:36
00:36
00:36
00:36
00:36
00:36
00:36
00:36
00:36
00:36
00:36
00:36
00:36
00:36
00:36
00:36
00:36
00:36
00:36
00:36
00:36
00:36
00:36
00:36
00:36

0:00
0:00
0:00
0:00
0:00
0:03
0:00
0:00
0:00
0:00
0:00
0:00
0:00
0:00
0:00
0:00
0:01
0:00
0:00
0:00
0:00
0:06
0:00
0:00
0:00
0:00
0:00
0:00

[kthrotld]
[acpi_thermal_pm]
[ipv6_addrconf]
[deferwq]
[charger_manager]
[kworker/3:1]
[kpsmoused]
[scsi_eh_0]
[scsi_tmf_0]
[scsi_eh_1]
[scsi_tmf_1]
[scsi_eh_2]
[scsi_tmf_2]
[scsi_eh_3]
[scsi_tmf_3]
[kworker/1:1H]
[jbd2/sda1-8]
[ext4-rsv-conver]
[kworker/0:1H]
mountall --daemon
[kworker/2:1H]
[jbd2/sda3-8]
[ext4-rsv-conver]
upstart-udev-bridge --daemon
/lib/systemd/systemd-udevd --daemon
[ktpacpid]
[kmemstick]
[hd-audio1]

Look at all of these system processes, why should they be running inside a container ? This is the output of a
running container running PHP FPM image :
USER
root
www-data
www-data
root
root

PID %CPU %MEM
VSZ
RSS TTY
1 0.0 0.0 113464 1084 ?
6 0.0 0.0 113464 1044 ?
7 0.0 0.0 113464 1064 ?
8 0.0 0.0 20228 3188 ?
14 0.0 0.0 17500 2076 ?

STAT
Ss
S
S
Ss
R

COMMAND
php-fpm: master process (/usr/local/etc/php-fpm.conf)
php-fpm: pool www
php-fpm: pool www
bash
ps aux

You can see that the only running processes are php-fpm processes and the other processes spawned by php.
If we type use Docker history command to see the CMD command, we can notice that the only process run is phpfpm:
docker history --human --no-trunc php:7-fpm|grep -i cmd

sha256:1..3

5 weeks ago /bin/sh -c #(nop)

CMD ["php-fpm"]

Another inconvenient of using Ubuntu in this case is the size of the image, you will:
increase your build time
increase your deployment time
increase the development time
Using a minimal image will reduce all of this.
You don't also need the init system of an operating system since inside Docker you don't have access to all of the
Kernel resources. If you would like to use a "full" operating system, you are adding problems to your problem list.
This is the case for many other OSs used inside Docker like Centos or Debian unless they are optimized to run
Docker and follow its philosophy.

From Scratch
Actually, the smallest image that we can find in the official Docker Hub is the scratch image.

Even if you can find this image Docker’s Hub, you can’t:
pull it,
run it
tag any image with the same name ("scratch")
But you can still use it as a reference to an image in the FROM instruction.
The scratch image is small, fast, secure and bugless.

Busybox
According to Wikipedia:
It runs in a variety of POSIX environments such as Linux, Android, and FreeBSD, although many of the tools
it provides are designed to work with interfaces provided by the Linux kernel. It was specifically created for
embedded operating systems with very limited resources. The authors dubbed it "The Swiss Army knife of
Embedded Linux", as the single executable replaces basic functions of more than 300 common commands. It is
released as free software under the terms of the GNU General Public License v2.
BusyBox is software that provides several stripped-down Unix tools in a single executable file so that the
command (as an example) could be run this way:

ls

/bin/busybox ls

Busybox is the winner of the smallest images (2.5 MB) that we can use with Docker. Executing a
will take almost 1 second.

docker pull busybox

With the advantage of the very minimal size comes some cons : Busybox does not have neither a package manager
nor a gcc compiler.
You can run the busybox executable from Docker to see the the list of the binaries it includes :
docker run busybox busybox

BusyBox v1.25.1 (2016-10-07 18:17:00 UTC) multi-call binary.
BusyBox is copyrighted by many authors between 1998-2015.
Licensed under GPLv2. See source distribution for detailed
copyright notices.
Usage:
or:
or:
or:

busybox [function [arguments]...]
busybox --list[-full]
busybox --install [-s] [DIR]
function [arguments]...

BusyBox is a multi-call binary that combines many common Unix
utilities into a single executable. Most people will create a
link to busybox for each function they wish to use and BusyBox
will act like whatever it was invoked as.
Currently defined functions:
[, [[, acpid, add-shell, addgroup, adduser, adjtimex, ar, arp, arping,
ash, awk, base64, basename, beep, blkdiscard, blkid, blockdev,
bootchartd, brctl, bunzip2, bzcat, bzip2, cal, cat, catv, chat, chattr,

chgrp, chmod, chown, chpasswd, chpst, chroot, chrt, chvt, cksum, clear,
cmp, comm, conspy, cp, cpio, crond, crontab, cryptpw, cttyhack, cut,
date, dc, dd, deallocvt, delgroup, deluser, depmod, devmem, df,
dhcprelay, diff, dirname, dmesg, dnsd, dnsdomainname, dos2unix, du,
dumpkmap, dumpleases, echo, ed, egrep, eject, env, envdir, envuidgid,
ether-wake, expand, expr, fakeidentd, false, fatattr, fbset, fbsplash,
fdflush, fdformat, fdisk, fgconsole, fgrep, find, findfs, flock, fold,
free, freeramdisk, fsck, fsck.minix, fstrim, fsync, ftpd, ftpget,
ftpput, fuser, getopt, getty, grep, groups, gunzip, gzip, halt, hd,
hdparm, head, hexdump, hostid, hostname, httpd, hush, hwclock,
i2cdetect, i2cdump, i2cget, i2cset, id, ifconfig, ifdown, ifenslave,
ifplugd, ifup, inetd, init, insmod, install, ionice, iostat, ip,
ipaddr, ipcalc, ipcrm, ipcs, iplink, iproute, iprule, iptunnel,
kbd_mode, kill, killall, killall5, klogd, last, less, linux32, linux64,
linuxrc, ln, loadfont, loadkmap, logger, login, logname, logread,
losetup, lpd, lpq, lpr, ls, lsattr, lsmod, lsof, lspci, lsusb, lzcat,
lzma, lzop, lzopcat, makedevs, makemime, man, md5sum, mdev, mesg,
microcom, mkdir, mkdosfs, mke2fs, mkfifo, mkfs.ext2, mkfs.minix,
mkfs.vfat, mknod, mkpasswd, mkswap, mktemp, modinfo, modprobe, more,
mount, mountpoint, mpstat, mt, mv, nameif, nanddump, nandwrite,
nbd-client, nc, netstat, nice, nmeter, nohup, nsenter, nslookup, ntpd,
od, openvt, passwd, patch, pgrep, pidof, ping, ping6, pipe_progress,
pivot_root, pkill, pmap, popmaildir, poweroff, powertop, printenv,
printf, ps, pscan, pstree, pwd, pwdx, raidautorun, rdate, rdev,
readahead, readlink, readprofile, realpath, reboot, reformime,
remove-shell, renice, reset, resize, rev, rm, rmdir, rmmod, route, rpm,
rpm2cpio, rtcwake, run-parts, runlevel, runsv, runsvdir, rx, script,
scriptreplay, sed, sendmail, seq, setarch, setconsole, setfont,
setkeycodes, setlogcons, setserial, setsid, setuidgid, sh, sha1sum,
sha256sum, sha3sum, sha512sum, showkey, shuf, slattach, sleep, smemcap,
softlimit, sort, split, start-stop-daemon, stat, strings, stty, su,
sulogin, sum, sv, svlogd, swapoff, swapon, switch_root, sync, sysctl,
syslogd, tac, tail, tar, tcpsvd, tee, telnet, telnetd, test, tftp,
tftpd, time, timeout, top, touch, tr, traceroute, traceroute6, true,
truncate, tty, ttysize, tunctl, ubiattach, ubidetach, ubimkvol,
ubirename, ubirmvol, ubirsvol, ubiupdatevol, udhcpc, udhcpd, udpsvd,
uevent, umount, uname, unexpand, uniq, unix2dos, unlink, unlzma,
unlzop, unshare, unxz, unzip, uptime, users, usleep, uudecode,
uuencode, vconfig, vi, vlock, volname, wall, watch, watchdog, wc, wget,
which, who, whoami, whois, xargs, xz, xzcat, yes, zcat, zcip

So there is no real package manager for this tiny distribution which is a pain, Alpine is a very good alternative, if
you need a package manager.

Alpine Linux
FROM alpine

Let's build it and see if it is working with just a -one-line Dockerfile.
docker build .

It is a working image, you can see the output:
Sending build context to Docker daemon 2.048 kB
Step 1 : FROM alpine
latest: Pulling from library/alpine
3690ec4760f9: Pull complete
Digest: sha256:1354db23ff5478120c980eca1611a51c9f2b88b61f24283ee8200bf9a54f2e5c
Status: Downloaded newer image for alpine:latest
---> baa5d63471ea
Successfully built baa5d63471ea

Alpine Linux is a security-oriented, lightweight Linux distribution based on musl libc and busybox.
Alpine Linux is very popular with Docker images since it is only 5.5MB and includes a package manager with many
packages.
Alpine Linux is suitable to run micro containers. The idea behind containers is also to allow the distribution of

packages easily, using images like Alpine is helpful for this case.

Phusion Baseimage
Phusion Baseimage or Baseimage-docker consumes 6 MB RAM, it is a special Docker image that is configured for
correct use within Docker containers and it is based on Ubuntu.
According to its authors it has the following feature:
Modifications for Docker-friendliness.
Administration tools that are especially useful in the context of Docker.
Mechanisms for easily running multiple processes, without violating the Docker philosophy.
You can use it as a base for your own Docker images.
Baseimage-docker is composed of :
Component
Ubuntu 16.04
LTS

Why is it included? / Remarks
The base system.

Main article: Docker and the PID 1 zombie reaping problem. According to the Unix process
model, the init process -- PID 1 -- inherits all orphaned child processes and must [reap them]
(https://en.wikipedia.org/wiki/Wait(system_call)). Most Docker containers do not have an init
process that does this correctly, and as a result their containers become filled with zombie
A correct init processes over time. Furthermore, docker stop sends SIGTERM to the init process, which is then
process
supposed to stop all services. Unfortunately most init systems don't do this correctly within
Docker since they're built for hardware shutdowns instead. This causes processes to be hard killed
with SIGKILL, which doesn't give them a chance to correctly deinitialize things. This can cause
file corruption. Baseimage-docker comes with an init process /sbin/my_init that performs both
of these tasks correctly.
Fixes APT
incompatibilities See https://github.com/dotcloud/docker/issues/1024 .
with Docker
A syslog daemon is necessary so that many services - including the kernel itself - can correctly
syslog-ng
log to /var/log/syslog. If no syslog daemon is running, a lot of important messages are silently
swallowed. Only listens locally. All syslog messages are forwarded to "docker logs".
logrotate
Rotates and compresses logs on a regular basis.
Allows you to easily login to your container to inspect or administer things. _SSH is disabled by
default and is only one of the methods provided by baseimage-docker for this purpose. The other
SSH server
method is through docker exec. SSH is also provided as an alternative because docker exec
comes with several caveats. Password and challenge-response authentication are disabled by
default. Only key authentication is allowed.
cron
The cron daemon must be running for cron jobs to work.
Replaces Ubuntu's Upstart. Used for service supervision and management. Much easier to use
runit
than SysV init and supports restarting daemons when they crash. Much easier to use and more
lightweight than Upstart.
A tool for running a command as another user. Easier to use than su , has a smaller attack vector
setuser
than sudo , and unlike chpst this tool sets $HOME correctly. Available as /sbin/setuser .
source: https://github.com/phusion/baseimage-docker/blob/master/README.md
We are going to see how to use this image, but you can find more detailed information in the official github
repository.
The same team created a base image for running Ruby, Python, Node.js and Meteor web apps called passenger-

docker.

Running The Init System
In order to use this special init system, use the CMD instruction:
FROM phusion/baseimage:
CMD ["/sbin/my_init"]

Adding Additional Daemons
To add additional daemons, write a shell script which runs it and add it to the following directory:
/etc/service/

Let's take the same example used in the official documentation: memcached.
echo "#!/bin/sh" > /etc/service/memcached
echo "exec /sbin/setuser memcache /usr/bin/memcached >>/var/log/memcached.log 2>&1" >> /etc/service/memcached
chmod +x /etc/service/memcached

And in your Dockerfile:
FROM phusion/baseimage:
CMD ["/sbin/my_init"]
RUN mkdir /etc/service/memcached
ADD memcached.sh /etc/service/memcached/run

Running Scripts At A Container Startup
You should also create a small script and add it to

/etc/my_init.d/

echo "#!/bin/sh" > /script.sh
date > /script.sh

In the Dockerfile add the script under the

/etc/my_init.d

directory:

RUN mkdir -p /etc/my_init.d
ADD script.sh /etc/my_init.d/script.sh

Creating Environment Variables
While you can use default Docker commands and instructions to work with these variables, you can use the
/etc /container_environment

folder to store the variables that could be used inside your container :
RUN echo LOGIN my_login > /etc/container_environment/username

If you log inside your container, you can verify that the username has been set as an environment variable:
echo $LOGIN
my_login

Building A MongoDB Image Using An Optimized Base Image
In this example we are going to use a known the phusion/baseimage which is based on Ubuntu. As said, the authors
of this image describe it as "A minimal Ubuntu base image modified for Docker-friendliness". This image only
consumes 6 MB RAM and is much powerful than Busybox or Alpine
A Docker image should start by the FROM instruction:
FROM phusion/baseimage

You can then add your name and/or email:
MAINTAINER Aymen El Amri - eon01.com - 

Update the system
RUN apt-get update

Add the installation prerequisites:
RUN apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 7F0CEB10
RUN echo 'deb http://downloads-distro.mongodb.org/repo/ubuntu-upstart dist 10gen' | tee /etc/apt/sources.list.d/mongodb.list

Then install Mongodb and don't forget to remove the unused files:
RUN apt-get update
RUN apt-get install -y mongodb-10gen && rm -rf /var/lib/apt/lists/*

Create the Mongodb data directory and tell Docker that the port 27017 could be used by another container:
RUN mkdir -p /data/db
EXPOSE 27017

To start Mongodb, we are going to use the command
contains the following two lines:
CMD ["--port 27017"]
ENTRYPOINT usr/bin/mongod

/usr/bin/mongod --port 27017

, that's why the Dockerfile will

The final Dockerfile is:
FROM phusion/baseimage
MAINTAINER Aymen El Amri - eon01.com - 
RUN apt-get update
RUN apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 7F0CEB10
RUN echo 'deb http://downloads-distro.mongodb.org/repo/ubuntu-upstart dist 10gen' | tee /etc/apt/sources.list.d/mongodb.list
RUN apt-get update
RUN apt-get install -y mongodb-10gen
RUN mkdir -p /data/db
EXPOSE 27017
CMD ["--port 27017"]
ENTRYPOINT usr/bin/mongod

Creating A Python Application Micro Image
As said Busybox combines tiny versions of many common UNIX utilities into a single small executable. To run a
Python application we need Python already installed but without a package manager, we are going to use StaticPython: A fork of cpython that supports building a static interpreter and true standalone executables.
We are going to get the executables from here.
wget https://github.com/pts/staticpython/raw/master/release/python3.2-static

The executable is only 5,7M:
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.16.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 5930789 (5,7M) [application/octet-stream]
Saving to: ‘python3.2-static’

As a Python application, we are going to run this small script:
cat app.py
print("This is Python !")

So we have 3 files:
ls -l code/chapter5_staticpython/example1
total 5800
-rw-r--r-- 1 eon01 sudo
26 Nov 12 18:05 app.py
-rw-r--r-- 1 eon01 sudo
159 Nov 12 18:28 Dockerfile
-rw-r--r-- 1 eon01 sudo 5930789 Nov 12 18:05 python3.2-static

We should add the Python executable and he app.py into the container. Then execute the script:
FROM busybox
ADD app.py python3.2-static /
RUN chmod +x /python3.2-static
CMD ["/python3.2-static", "/app.py"]

Another different approach to cerate an image is to integrate the wget inside the Dockerfile. The Python application
could be also downloaded (or pulled from a git repository) directly from inside the image.
FROM busybox
RUN busybox wget  && busybox wget 

RUN chmod +x /python3.2-static
ENTRYPOINT ["/python3.2-static", "/app.py"]

Creating A Node.js Application Micro Image
Let's create a simple Node.js application using one of the smallest possible images in order to run a really micro
container.
We have seen that Alpine is a good OS for a base image. Note that apk is the Alpine package manager.
This is the Dockerfile:
FROM alpine
RUN apk update && apk upgrade
RUN apk add nodejs
WORKDIR /usr/src/app
ADD app.js .
CMD [ "node", "app.js" ]

The Node.js script that we are going to run is in

app.js

console.log("********** Hello World **********");

Put the js file in the same directory as the Dockerfile and build the image using
The image (with the upgrades) has 5 MB as you can see:
Sending build context to Docker daemon 3.072 kB
Step 1 : FROM alpine
---> baa5d63471ea
Step 2 : RUN apk update && apk upgrade
---> Running in 0a8bb4d3be05
fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/community/x86_64/APKINDEX.tar.gz
v3.4.5-31-g7d74397 [http://dl-cdn.alpinelinux.org/alpine/v3.4/main]
v3.4.4-21-g75fc217 [http://dl-cdn.alpinelinux.org/alpine/v3.4/community]
OK: 5975 distinct packages available
(1/3) Upgrading musl (1.1.14-r12 -> 1.1.14-r14)
(2/3) Upgrading busybox (1.24.2-r11 -> 1.24.2-r12)
Executing busybox-1.24.2-r12.post-upgrade
(3/3) Upgrading musl-utils (1.1.14-r12 -> 1.1.14-r14)
Executing busybox-1.24.2-r12.trigger
OK: 5 MiB in 11 packages

docker build .

.

Creating Your Own Docker Base Image
Building your own Docker image is possible.
As said, a Docker Images is a read-only layer, it never changes but in order to make it writable, the Union File
System should add a read-write file system over the read-only file system.
The base image we are going to build is one of the type of images that has no parents. There are two ways to create a
base image.

Using Tar
A base image is a working Linux OS like Ubuntu, Debian .. etc Creating a base image using tar may be different
from a distribution to another. Let's see how to create a Debian based distribution.
We can use debootstrap: is used to fetch the required Debian packages to build the base system.
First of all, if you haven't installed it yet, use your package manager to download the package:
apt-get install debootstrap

This is the man description of this package:
DESCRIPTION
debootstrap

bootstraps

a

basic

Debian

system

of

SUITE

into TARGET from MIRROR by running SCRIPT.

MIRROR can be an http:// or https:// UR

The SUITE may be a release code name (eg, sid, jessie, wheezy) or a symbolic name (eg, unstable, testing, stable, oldstable)
Notice that file:/ URLs are translated to file:/// (correct scheme as described in RFC1738 for local filenames), and file://
ssh ://USER@HOST/PATH URLs are retrieved using scp; use of ssh-agent or similar is strongly recommended.
Debootstrap

can

not

w

be used to install Debian in a system without using an installation disk but can also be used to run a different Debian flavor

At this step, we will create an image based on Ubuntu 16.04 Xenial.
sudo debootstrap xenial xenial > /dev/null

Once it is finished you can check you Ubuntu image files:
ls xenial/
bin boot dev

will

etc

home

lib

lib64

media

mnt

opt

proc

root

run

sbin

srv

sys

tmp

usr

var

Let's create an archive to import it later by Docker import command:
sudo tar -C xenial/ -c . | sudo docker import - xenial

The import command creates a new filesystem image from the contents of a tarball It is used this way:
docker import [OPTIONS] file|URL|- [REPOSITORY[:TAG]]

The possible options are:

-c

and

-m

.

-c or --change
 is used to apply Dockerfile instruction to the created image.
used to set a commit message for imported image.

-m

or

--message



is

You can run the container with the base image you had created :
docker run xenial

But since the container has no command to run at its startup, you will have a similar error to this one:
docker: Error response from daemon: No command specified.
See 'docker run --help'.

You can try for example a command to run like

date

(just for the testing purpose):

docker run xenial date

Or you can verify the distribution of the created base image's OS:
docker run trusty cat /etc/lsb-release

If you are not familiar yet with the run command, we are going to see a chapter about running containers.
This is just a dirty verification of the integrity image.
You may use the local xenial bas image that you have just created in a Dockerfile:
FROM xenial
RUN ls

Sending build context to Docker daemon 252.9 MB
Step 1 : FROM xenial
---> 7a409243b212
Step 2 : RUN ls
---> Running in 5f544041222a
bin
boot
dev
etc
home
lib
lib64
media
mnt
opt
proc
root
run
sbin
srv
sys
tmp
usr
var
---> 57ce8f25dd63
Removing intermediate container 5f544041222a
Successfully built 57ce8f25dd63

Using Scratch
You don't need in reality to create the scratch image because it already exists, but let's see how to create one.
Actually, a scratch image is an image that was created using an empty tar archive.
tar cv --files-from /dev/null | docker import - scratch

You can see an example in the Docker library Github repository:
FROM scratch
COPY hello /
CMD ["/hello"]

This image uses an executable called hello that you can download here:
wget https://github.com/docker-library/hello-world/raw/master/hello-world/hello

Chapter VI - Working With Docker Containers
o
o

^__^
(oo)\_______
(__)\
)\/\
||----w |
||
||

Creating A Container
To create a container, simply run:
docker create    

A simple example to start using this command is running:
docker create hello-world

Verify that the container was created by typing:
docker ps -a

or
docker ps --all

Docker will pick a random name for your container if you did not explicitly specify a name.
CONTAINER ID
ff330cd5505c

IMAGE
hello-world

COMMAND
"/hello"

CREATED
2 minutes ago

NAMES
elegant_cori

You can set a name:
docker create --name hello-world-container hello-world
e..8

Verify the creation of the container:
docker ps -a
CONTAINER ID
e490edeef083
ff330cd5505c

IMAGE
hello-world
hello-world

COMMAND
"/hello"
"/hello"

CREATED
24 seconds ago
3 minutes ago

STATUS
Created
Created

NAMES
hello-world-container
elegant_cori

The docker create uses the specified image and add a new writable layer to creates the container and waits for the
specified command to run inside the created container. You may notice that docker create is generally used with it options where:
-i or --interactive
-t or --tty

Other options may be used like:

To keep STDIN open even if not attached
To allocate a pseudo-TTY

-p or --publish value
-v or --volume value
--dns value
-e or --env value
-l or --label value
-m or --memory string
--no-healthcheck
--volume-driver string
--volumes-from value
-w or --workdir string

To publish a container's port(s) to the host
To bind mount a volume.
To set custom DNS servers (default [])
To set environment variables (default [])
To set meta data on a container (default [])
To set a memory limit
To disable any container-specified HEALTHCHECK
To set the volume driver for the container
To mount volumes from the specified container(s) (default [])
To set the working directory inside the container

Example:
docker create --name web_server -it -v /etc/nginx/sites-enabled:/etc/nginx/sites-enabled -p 8080:80 nginx
Unable to find image 'nginx:latest' locally
latest: Pulling from library/nginx
75a822cd7888: Pull complete
0aefb9dc4a57: Pull complete
046e44ee6057: Pull complete
Digest: sha256:fab482910aae9630c93bd24fc6fcecb9f9f792c24a8974f5e46d8ad625ac2357
Status: Downloaded newer image for nginx:latest
e..4

You can see the other used options in the official Docker documentation but the most used ones are listed above.
docker create

is similar to docker run -d but the container is never started until you make it explicitly using
which is e..4 in the last example.

start 

docker

Starting And Restarting A Container
We still use the nginx container example that we created using docker
enabled:/etc/nginx/sites-enabled -p 8080:80 nginx that have e..4 as in id.

create --name web_server -it -v /etc/nginx/sites-

We can use the id with the start command to start the created container:
docker start e..4

You can check if the container is running using
CONTAINER ID
e70fbc6e867a

IMAGE
nginx

COMMAND
"nginx -g 'daemon off"

docker ps

:

CREATED
8 minutes ago

PORTS
NAMES
443/tcp, 0.0.0.0:8080->80/tcp
web_server

The start command is used like this:
docker start   .. 

Where options could be:
-a or --attach
--detach-keys string
--help
-i or --interactive

To attach STDOUT/STDERR and forward signals
To override the key sequence for detaching a container
To print usage
To attach container's STDIN

You can start multiple containers using one command:
docker start e..4 e..8

Same as start command, you can restart a container using the restart command:
Nginx container:
docker restart e..4
e..4

hello-world container:
docker restart e..8
e..8

If you want to see the different configurations of a stopped or a running container, say the nginx container that has
the id e..4, you can go to /var/lib/docker/containers/e..4 where you will find these files:
/var/lib/docker/containers/e..4/
├── config.v2.json
├── e..4-json.log
├── hostconfig.json
├── hostname
├── hosts
├── resolv.conf

├── resolv.conf.hash
└── shm

This is the

config.v2.json

configuration file (the original one is not really formatted but compressed):

{
"StreamConfig":{
},
"State":{
"Running":true,
"Paused":false,
"Restarting":false,
"OOMKilled":false,
"RemovalInProgress":false,
"Dead":false,
"Pid":18722,
"StartedAt":"2017-01-02T23:37:03.770516537Z",
"FinishedAt":"2017-01-02T23:37:02.408427125Z",
"Health":null
},
"ID":"e..4",
"Created":"2017-01-02T23:00:06.494810718Z",
"Managed":false,
"Path":"nginx",
"Args":[
"-g",
"daemon off;"
],
"Config":{
"Hostname":"e70fbc6e867a",
"Domainname":"",
"User":"",
"AttachStdin":true,
"AttachStdout":true,
"AttachStderr":true,
"ExposedPorts":{
"443/tcp":{
},
"80/tcp":{
}
},
"Tty":true,
"OpenStdin":true,
"StdinOnce":true,
"Env":[
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"NGINX_VERSION=1.11.8-1~jessie"
],
"Cmd":[
"nginx",
"-g",
"daemon off;"
],
"Image":"nginx",
"Volumes":null,
"WorkingDir":"",
"Entrypoint":null,
"OnBuild":null,
"Labels":{
}
},
"Image":"sha256:0..f",
"NetworkSettings":{
"Bridge":"",
"SandboxID":"e..c",
"HairpinMode":false,
"LinkLocalIPv6Address":"",
"LinkLocalIPv6PrefixLen":0,
"Networks":{
"bridge":{
"IPAMConfig":null,
"Links":null,
"Aliases":null,
"NetworkID":"a..b",
"EndpointID":"f..e",
"Gateway":"172.17.0.1",
"IPAddress":"172.17.0.2",
"IPPrefixLen":16,
"IPv6Gateway":"",
"GlobalIPv6Address":"",
"GlobalIPv6PrefixLen":0,
"MacAddress":"02:42:ac:11:00:02"
}
},

"Service":null,
"Ports":{
"443/tcp":null,
"80/tcp":[
{
"HostIp":"0.0.0.0",
"HostPort":"8080"
}
]
},
"SandboxKey":"/var/run/docker/netns/eeb3ec8d995d",
"SecondaryIPAddresses":null,
"SecondaryIPv6Addresses":null,
"IsAnonymousEndpoint":false
},
"LogPath":"/var/lib/docker/containers/e..4/e..4-json.log",
"Name":"/web_server",
"Driver":"aufs",
"MountLabel":"",
"ProcessLabel":"",
"RestartCount":0,
"HasBeenStartedBefore":false,
"HasBeenManuallyStopped":false,
"MountPoints":{
"/etc/nginx/sites-enabled":{
"Source":"/etc/nginx/sites-enabled",
"Destination":"/etc/nginx/sites-enabled",
"RW":true,
"Name":"",
"Driver":"",
"Relabel":"",
"Propagation":"rprivate",
"Named":false,
"ID":""
}
},
"AppArmorProfile":"",
"HostnamePath":"/var/lib/docker/containers/e..4/hostname",
"HostsPath":"/var/lib/docker/containers/e..4/hosts",
"ShmPath":"/var/lib/docker/containers/e..4/shm",
"ResolvConfPath":"/var/lib/docker/containers/e..4/resolv.conf",
"SeccompProfile":"",
"NoNewPrivileges":false
}

Information used by Containerd could be used - as you have seen in the previous chapters - are in
/var/run/docker/libcontainerd/e..4/config.json

Pausing And Unpausing A Container
To unpause the Nginx container use:
docker pause e..4
You can verify the STATUS of the container using
CONTAINER ID
e70fbc6e867a

IMAGE
nginx

COMMAND
"nginx -g 'daemon off"

docker ps

:

STATUS
Up About a minute (**Paused**)

PORTS
443/tcp, 0.0.0.0:8080->80/tcp

To get the container back, use the unpause command:
docker unpause e..4

This is the output of

ps aux|grep docker-containerd-shim

when the container was paused:

root
5620
0.0
0.0 282896
5512
shim e..4 /var/run/docker/libcontainerd/e..4 docker-runc

?

Sl

01:48

0:00

docker-containerd-

Sl

01:48

0:00

docker-containerd-

And this is the same command output when the container was unpaused:
root
5620
0.0
0.0 282896
5512
shim e..4 /var/run/docker/libcontainerd/e..4 docker-runc

?

After pausing a container, Docker uses the cgroups freezer to suspend all processes in the Nginx container, you can
check the different configurations in relation with the container freezed status in /sys/fs/cgroup/freezer/docker
├──
├──
├──
│
│
│
│
│
│
│
├──
├──
├──
├──
└──

cgroup.clone_children
cgroup.procs
e..4
├── cgroup.clone_children
├── cgroup.procs
├── freezer.parent_freezing
├── freezer.self_freezing
├── freezer.state
├── notify_on_release
└── tasks
freezer.parent_freezing
freezer.self_freezing
freezer.state
notify_on_release
tasks

The running tasks could be found here
cat

/sys/fs/cgroup/freezer/docker/e..4/tasks

.

/sys/fs/cgroup/freezer/docker/e..4/tasks

5637
5662

Now if you type,

docker top e..4

, you will see the PID of the freezed tasks:

UID
PID
PPID
C
STIME
TTY
TIME
CMD
root
5637
5620
0
01:48
pts/11
00:00:00
nginx: master process nginx -g daemon off;
dnsmasq
5662
5637
0
01:48
pts/11
00:00:00
nginx: worker process

So, cgroup freezer creates some files.
freezer.state (read-write) : When read, returns the effective state of the cgroup "THAWED", "FREEZING" or
"FROZEN"
freezer.self_freezing (read-only) : Shows the self-state. 0 if the self-state is THAWED; otherwise, 1
freezer.parent_freezing (read-only) : Shows the parent-state. 0 if none of the cgroup's ancestors is frozen;
otherwise, 1

Stopping A Container
Using Docker Stop
You can use the stop command in the following way to stop one or multiple containers:
docker stop   .. 

Example:
We create a container first using
use the following command:

docker run -it -d --name my_container busybox

and to stop it using the

--time

you can

docker stop --time 20 my_container

The last command will wait 20 seconds before killing the container. The default number of seconds is 10, so
executing docker stop my_container will wait 10 seconds.
Executing docker stop command will ask nicely top stop the container and in the case when it does not comply
within 10 seconds it will be killed but ungracefully.
Stopping a container is sending a SIGTERM signal to the root process (PID 1) in the container but if the process and
if it has not exited within the timeout period (10 second) a SIGKILL signal will be sent.

Using Docker Kill
Docker kill could be used in this way:
docker kill   .. 

Docker kill does not exit gracefully the container process does not have any sort of timeout period and send a
SIGKILL to terminate the container process by default.
Docker kill is like sending a Linux signal
SIGTERM:

kill -9

but you can choose a different signal to send like SIGINT ot

docker kill --signal=SIGINT my_container
docker kill --signal=SIGTERM my_container

Using Docker rm -f
To remove one or multiple containers you can use
docker rm   .. 

Say we are running this container:

docker rm

command.

docker run -it

-d

--name my_container busybox

When checking its status, we can see that it is up:
docker ps
CONTAINER ID
ec53b9228910

IMAGE
busybox

COMMAND
"sh"

CREATED
1 seconds ago

STATUS
Up 1 seconds

PORTS
NAMES
my_container

Docker engine can not remove a container while it is running, you can verify this by typing:
docker rm my_container

using the short id:
docker rm ec53b9228910

or the long id:
docker rm e..8

You will have a similar output to this one:
Error response from daemon: You cannot remove a running container e..8. Stop the container before attempting removal or use f

Removing a container is different from stopping it and you should add the
force removing and then stopping it.

-f

or the

--force

option, in order to

docker rm -f my_container

Adding the --force command is like sending a SIGKILL signal to the container process and it is also an alternative
to executing two commands:
docker stop my_container
docker rm my_container

The final option for stopping a running container is to use the --force or -f flag in conjunction with the docker rm
command.
Using

docker rm

command with

-f

option will not only stop a running container but erases all of its traces.

Using this option will not stop and remove the container gracefully.

Docker Signals
https://blog.confirm.ch/sending-signals-docker-container/
By default executing docker kill command will send a SIGKILL signal to the container's main process. If you want
to send another signal like SIGTERM, SIGHUP, SIGUSR1, SIGUSR2 you should use the -s kk or the --signal ```
options followed by the signal name. Note that you can send the same signal to multiple containers.
docker kill  .. 

or
docker kill --signal   .. 

I will use the SIGHUP signal that will tell the Docker container main process to reload its configuration and to do
this we have three different approaches:
The first one is sending a signal to the container's pid
The second option is to execute the kill command inside the container
The third way to do it is using docker kill command
kill -SIGHUP 
docker exec  kill -SIGHUP 1 (the main process has always 1 as pid)
docker kill --signal=HUP 

Container Life Cycle
The following diagrams is taken from the official Docker documentation and I have not found a more clear
explanation than using the same diagram of Docker states.

Through this chapter and the previous parts of this book, we have seen how to create, run, pause, unpause, stop and
kill Docker containers. The Docker CLI allows us to communicate and send events to Docker engine in order to
manage Docker containers and their state.
A container could be in one of the different statuses:
Created: The image is pulled and the container is configured to run but not running yet.
Running: Docker create a new RW layer in top of an the pulled image and starts running the container.
Paused: Docker container is paused but ready to run again without any restart, just run the unpause command.
Stopped: The container is stopped and not running. You should start another new container, if you want to run
the same application.
Deleted: A container could not be removed unless it is stopped. In this status, there is no container at all
(neither in the disk, nor in the memory of the host machine).

Running Docker In Docker
In short, it is not recommended to run Docker in Docker. If you are curious about the details:
Probably some people tried this for fun but it could be a real use case, you may have a continuous integration server
running in a Docker container and it should - at the same time - run and spin containers: You could be using Docker
to run Jenkins for example and need to run some Docker containers inside the container running Jenkins.
This scenario could create many problems but most of them could be resolved by bind-mounting the Docker
container socket into already running Jenkins container (which is different from running a Docker container inside
another one).
With Docker in Docker, you could have some problems like the incompatibility between internel and external
containers' security profiles: To resolve this kind of dysfunctions, the outer container should run with extra
privileges ( --privileged=true ) but keep in mind that in the same time, this does not cooperate well with Linux
Security Modules (LSM).
When you run a container inside a Jenkins container, you will run a CoW filesystem (like AUFS or Devicemapper)
at the top of another CoW filesystem, while what you want to run is a CoW filesystem (guest container) on the top of
a normal filesystem (like EXT3 or EXT4). This could create errors in your internal Docker filesystem.
The author of the feature that made possible for Docker to run inside a Docker container, Jérôme Petazzoni wrote a
blog post explaining why you should not run Docker in Docker. The alternative is to expose the Docker socket to
your Jenkins container, by binding /var/run/docker.sock using the -v flag but his solution is not also reliable in the
latest version since Docker Engine is no longer distributed.
Using Docker API is the good solution in this case, you can also use a wrapper like:
Dockerode: https://github.com/apocas/dockerode
Docker-py: https://github.com/docker/docker-py
DoMonit: https://github.com/eon01/DoMonit
In all cases, I don't recommend using Docker in Docker unless you really need to run it in this way or you are
knowledgeable about Docker.

Spotify's Docker Garbage Collector
This tool developed by Spotify allows you to clean your host from :
Containers that exited more than an hour ago are removed.
Images that do not belong to any remaining container after that are removed.
As it is explained in the Git repository of docker-gc:
Although docker normally prevents removal of images that are in use by containers, we take extra care to not
remove any image tags (e.g., ubuntu:14.04, busybox, etc) that are in use by containers. A naive docker rmi
$(docker images -q) will leave images stripped of all tags, forcing docker to re-pull the repositories when starting
new containers even though the images themselves are still on disk.
The easiest way to run the garbage collector is using its Docker container:
docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -v /etc:/etc spotify/docker-gc

[2016-11-04T16:26:32]
[2016-11-04T16:26:32]
[2016-11-04T16:26:32]
[2016-11-04T16:26:33]
[2016-11-04T16:26:33]
[2016-11-04T16:26:33]
[2016-11-04T16:26:33]
[2016-11-04T16:26:33]
[2016-11-04T16:26:33]
[2016-11-04T16:26:33]
[2016-11-04T16:26:33]
[2016-11-04T16:26:33]
[..etc]

[INFO]
[INFO]
[INFO]
[INFO]
[INFO]
[INFO]
[INFO]
[INFO]
[INFO]
[INFO]
[INFO]
[INFO]

:
:
:
:
:
:
:
:
:
:
:
:

Container
Container
Container
Container
Container
Container
Container
Container
Container
Container
Container
Container

running
not running
not running
not running
not running
not running
not running
not running
not running
not running
not running
not running

/dreamy_feynman
/nauseous_gates
/cranky_leakey
/small_fermat
/elated_bose
/sad_goldberg
/gigantic_banach
/small_payne
/reverent_jennings
/kickass_brahmagupta
/silly_fermat
/dreamy_jang

This is pretty useful when it is run with a cron:
* 1 * * * docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -v /etc:/etc spotify/docker-gc

Using A Volume From Another Container
tree
.
├── data
│
├── app.js
│
└── Dockerfile
└── nodejs
└── Dockerfile

#cat data/Dockerfile
FROM alpine
ADD . /code
VOLUME /code
CMD ["tail", "-f","/dev/null"]

cat data/app.js
// Load the http module to create an http server.
var http = require('http');
// Configure our HTTP server to respond with Hello World to all requests.
var server = http.createServer(function (request, response) {
response.writeHead(200, {"Content-Type": "text/plain"});
response.end("Hello World\n");
});
// Listen on port 8000, IP defaults to 127.0.0.1
server.listen(8000);
// Put a friendly message on the terminal
console.log("Server running at http://127.0.0.1:8000/");

cat nodejs/Dockerfile
FROM node
VOLUME /code
WORKDIR /code
CMD ["node", "app.js"]

docker build -t my_data . docker build -t my_nodejs .
docker run -it -d --name my_data my_data docker run -it --volumes-from my_data --name my_nodejs my_nodejs

Performing A Docker Backup
Using the --volume-from flag, it is possible to backup data from inside a Docker container. Say we want to create a
Mysql container and we want to run a some databases backup.
Normally we should run this command to start a Mysql container:
docker run --name my_db -e MYSQL_ROOT_PASSWORD=my_secret -d mysql

When we want explicitly declare a volume for our Mysql data, we can run this command instead:
docker run --name my_db -e MYSQL_ROOT_PASSWORD=my_secret -d -v /var/lib/mysql mysql

Now we can run another container:
docker run --rm --volumes-from my_db -v $(pwd):/backup ubuntu tar cvf /backup/backup.tar /var/lib/mysql mysql

This command launches a new container and mounts the volume from the my_db container. It then mounts a local
directory /backup and backs up the contents of the my_db volume to a .TAR file within the /backup directory.
In order to do the restore, you should run:
Create a new container:
docker run -v /var/lib/mysql --name my_db_2 ubuntu /bin/bash

Un-tar the backup file in the new container`s data volume:
docker run --rm --volumes-from my_db_2 -v $(pwd):/backup ubuntu bash -c "cd /var/lib/mysql && tar xvf /backup/backup.tar -strip 1"

Chapter VII - Working With Docker Machine
o
o

^__^
(oo)\_______
(__)\
)\/\
||----w |
||
||

What Is Docker Machine & When Should I Use It ?
Docker Machine is a Docker tool that let you use and install Docker Engine on different Docker hosts.
Docker Machine is useful when it comes to creating and managing Docker using a local virtual machine. We can
create as many as Docker hosts as we want, we are not limited to a single host (localhost). We can create several
Docker hosts on our local host, in a private server or a cloud providers like AWS or Azure.

If you have an old laptop or desktop system, you are more probably unable to use the Docker "normal" installation
(Docker Engine). Many Windows and MacOS users are using Docker Machine instead of Docker.
Since Docker Machine can create provisioned virtual machines in different cloud providers like EC2 instances for
AWS, it could be used to provision Docker hosts on a remote system (your laptop, or your integration server ..etc).

Installation
*nix
Installing Docker Machine can be done using these 3 commands:
sudo apt-get install virtualbox
curl -L https://github.com/docker/machine/releases/download/v0.10.0/docker-machine-`uname
machine
chmod +x /tmp/docker-machine
sudo cp /tmp/docker-machine /usr/local/bin/docker-machine

-s`-`uname

-m`

>/tmp/docker-

You can also add the auto-completion feature using:
curl
L https://raw.githubusercontent.com/docker/docker/master/contrib/completion/bash/docker > /etc/bash_completion.d/docker

-

MacOS
If you are using a MacOS, you can install Docker Machine using the following commands:
curl
-L
https://github.com/docker/machine/releases/download/v0.10.0/docker-machine-`uname
m` >/usr/local/bin/docker-machine && \
chmod +x /usr/local/bin/docker-machine

-s`-`uname

-

To install the command completion, use:
curl
-L
https://raw.githubusercontent.com/docker/docker/master/contrib/completion/bash/docker
prefix`/etc/bash_completion.d/docker

>

`brew

--

A lightweight virtualization solution for MacOS called Hyperkit is used with Docker Machine, this why you you
should have these requirements:
OS X 10.10.3 Yosemite or later
a 2010 or later Mac (i.e. a CPU that supports EPT)
Oracle Virtualbox driver will be used to create local machines because there is no
HyperKit but you can still use HyperKit and VirtualBox on the same system.

docker-machine create

driver for

Windows
Windows users should git bash:
if [[ ! -d "$HOME/bin" ]]; then mkdir -p "$HOME/bin"; fi
curl -L https://github.com/docker/machine/releases/download/v0.10.0/docker-machine-Windows-x86_64.exe > "$HOME/bin/dockermachine.exe"
chmod +x "$HOME/bin/docker-machine.exe"

Because Hyper-V is not compatible with Oracle VirtualBox and because Docker for Windows uses Microsoft HyperV for virtualization, it is not possible to run the two platforms at the same time, but you can still use Docker Machine
for your local VMs using Hyper-V driver.

Using Docker Machine Locally
Creating Docker Machines
You can use Docker Machine to start a public cloud instance like AWS or DO but you can also use it to start a local
VirtualBox virtual machine. Creating a machine is done using the docker create command:
docker-machine create -d virtualbox default

First time running this command, your local machine will
Search in cache for the image:
(default) Image cache directory does not exist, creating it at /root/.docker/machine/cache...

Download the Boot2Docker ISO
(default) No default Boot2Docker ISO found locally, downloading the latest release...
(default) Latest release for github.com/boot2docker/boot2docker is v17.04.0-ce
(default) Downloading /root/.docker/machine/cache/boot2docker.iso from https://github.com/boot2docker...
(default) 0%....10%....20%....30%....40%....50%....60%....70%....80%....90%....100%
Creating machine...
(default) Copying /root/.docker/machine/cache/boot2docker.iso to /root/.docker/machi...

Create the VirtualBox VM with its configurations (network, ssh ..etc):
(default) Creating VirtualBox VM...
(default) Creating SSH key...
(default) Starting the VM...
(default) Check network to re-create if needed...
(default) Found a new host-only adapter: "vboxnet0"
(default) Waiting for an IP...
Waiting for machine to be running, this may take a few minutes...
Detecting operating system of created instance...
Waiting for SSH to be available...
Detecting the provisioner...
Provisioning with boot2docker...

Connecting Docker Machines To Your Shell
To see how to connect your Docker Client to the Docker Engine running on this virtual machine, run:
env default ̀ :
export DOCKER_TLS_VERIFY="1"
export DOCKER_HOST="tcp://192.168.99.100:2376"
export DOCKER_CERT_PATH="/root/.docker/machine/machines/default"
export DOCKER_MACHINE_NAME="default"
# Run this command to configure your shell:
# eval $(docker-machine env default)

If your machine has a different name then you should use it instead of default.
docker machine env 

̀ docker-machine

To connect a shell to the created machine and set environment variables for the current shell that the Docker client
will read you need to type the following command and execute it each time you open a new shell or restart your
machine. Type:
eval $(docker-machine env default)

In general:
eval $(docker-machine env )

Working With Multiple Docker Machines
You can see the created machine using
NAME
default

ACTIVE
-

DRIVER
virtualbox

STATE
Running

docker-machine ls

:

URL
tcp://192.168.99.100:2376

SWARM

DOCKER
v17.04.0-ce

ERRORS

Let's create two machines host1 and host2:
docker-machine create --driver virtualbox host1 && docker-machine create --driver virtualbox host2

Let's get the IP addresses of these two machines:
docker-machine ip host1 && docker-machine ip

host2

In my case, the two machines have the following IPs:
192.168.99.100
192.168.99.101

We would like to create two Apache web servers, one in the first machine and the other in the second machine.
In order to do this, connect the first to your shell:
eval $(docker-machine env host1)

Create the first web server:
docker run -it -p 8000:80 --name httpd1 -d httpd

Connect the second machine:
eval $(docker-machine env host2)

Then create the second web server:

docker run -it -p 8000:80 --name httpd2 -d httpd

You may notice that both containers have the same host port but this is not a problem. When we connected our shell
to the first machine, the first web server was created inside the first machine and the second web server container
was created just after connecting the shell to the second machine.
In our case every container is deployed inside a different machine, that is why we can have the same host port 8000.
You can see the active machine by typing:

docker-machine active

.

Getting More Information About Docker Machines
Until now we created default, host1 and host2 machines.
We can see the status of our machines using

docker-machine status

:

docker-machine status host1
Running

docker-machine status host2
Running

If we want more information about driver name, authentication settings, engine information or swarm data, we can
use the inspect command. As an example, you can use docker-machine inspect host1 and you will get a similar json:
{
"ConfigVersion": 3,
"Driver": {
"IPAddress": "192.168.99.100",
"MachineName": "host1",
"SSHUser": "docker",
"SSHPort": 46718,
"SSHKeyPath": "/home/eon01/.docker/machine/machines/host1/id_rsa",
"StorePath": "/home/eon01/.docker/machine",
"SwarmMaster": false,
"SwarmHost": "tcp://0.0.0.0:3376",
"SwarmDiscovery": "",
"VBoxManager": {},
"HostInterfaces": {},
"CPU": 1,
"Memory": 1024,
"DiskSize": 20000,
"NatNicType": "82540EM",
"Boot2DockerURL": "",
"Boot2DockerImportVM": "",
"HostDNSResolver": false,
"HostOnlyCIDR": "192.168.99.1/24",
"HostOnlyNicType": "82540EM",
"HostOnlyPromiscMode": "deny",
"UIType": "headless",
"HostOnlyNoDHCP": false,
"NoShare": false,
"DNSProxy": true,
"NoVTXCheck": false,
"ShareFolder": ""
},
"DriverName": "virtualbox",
"HostOptions": {
"Driver": "",
"Memory": 0,
"Disk": 0,
"EngineOptions": {
"ArbitraryFlags": [],
"Dns": null,
"GraphDir": "",
"Env": [],
"Ipv6": false,
"InsecureRegistry": [],
"Labels": [],
"LogLevel": "",
"StorageDriver": "",
"SelinuxEnabled": false,

"TlsVerify": true,
"RegistryMirror": [],
"InstallURL": "https://get.docker.com"
},
"SwarmOptions": {
"IsSwarm": false,
"Address": "",
"Discovery": "",
"Agent": false,
"Master": false,
"Host": "tcp://0.0.0.0:3376",
"Image": "swarm:latest",
"Strategy": "spread",
"Heartbeat": 0,
"Overcommit": 0,
"ArbitraryFlags": [],
"ArbitraryJoinFlags": [],
"Env": null,
"IsExperimental": false
},
"AuthOptions": {
"CertDir": "/home/eon01/.docker/machine/certs",
"CaCertPath": "/home/eon01/.docker/machine/certs/ca.pem",
"CaPrivateKeyPath": "/home/eon01/.docker/machine/certs/ca-key.pem",
"CaCertRemotePath": "",
"ServerCertPath": "/home/eon01/.docker/machine/machines/host1/server.pem",
"ServerKeyPath": "/home/eon01/.docker/machine/machines/host1/server-key.pem",
"ClientKeyPath": "/home/eon01/.docker/machine/certs/key.pem",
"ServerCertRemotePath": "",
"ServerKeyRemotePath": "",
"ClientCertPath": "/home/eon01/.docker/machine/certs/cert.pem",
"ServerCertSANs": [],
"StorePath": "/home/eon01/.docker/machine/machines/host1"
}
},
"Name": "host1"
}

Starting, Stopping, Restarting & Killing Machines
Docker Machine allows users to create, stop, start, delete machines using command that are quite similar to Docker
Engine commands.
Let's do some operations to see how this work, restart host1:
docker-machine restart host1
If you execute this, you will see this message or a similar one:
Restarted machines may have new IP addresses. You may need to re-run the `docker-machine env` command.

Like it is said in the message above, Docker Machine don't necessarily give static IPs to a new machine. If it
restarts, you should docker-machine env host1 again and yous should connect it to yur shell again eval $(docker-machine
env host1) .
This is the same case when you stop and start a machine:
docker-machine stop host1
docker-machine start host1

You can abruptly force stopping host1 by typing :
docker-machine kill host1

To completely remove a machine (host1) execute this command:

docker-machine rm host1

or
docker-machine rm --force -y host1

The second command will remove the machine without prompting for confirmation.

Upgrading Docker Machines
Check your Docker Machine version by typing:
docker-machine version

Example:
docker-machine version
docker-machine version 0.10.0, build 76ed2a6

In order to see the version of a running machine (host1), type:
docker-machine version host1

You can upgrade it by typing
docker-machine upgrade host1

This operation will stop, upgrade and restart host1 with a new IP address:
Upgrading docker...
Stopping machine to do the upgrade...
Upgrading machine "host1"...
Copying /home/eon01/.docker/machine/cache/boot2docker.iso to /home/eon01/.docker/machine/machines/host1/boot2docker.iso...
Starting machine back up...
(host1) Check network to re-create if needed...
(host1) Waiting for an IP...
Restarting docker...

Using Docker Machine With External Providers
Docker Machine could be used locally using a generic or VirtualBox drivers or on some cloud providers like :
Amazon Web Services
Microsoft Azure
Digital Ocean
Exoscale
Google Compute Engine
Microsoft Hyper-V
OpenStack
Rackspace
IBM Softlayer
VMware vCloud Air
VMware Fusion
VMware vSphere

Create Machines On Amazon Web Services
First thing to have when using AWS are the credentials that are generally saved in ~/.aws/credentials . You can use
different AWS profiles by providing access-key and secret-key as arguments in the Docker Machine command.
We want to create a Docker Machine using AWS EC2 driver while choosing Ubuntu AMI, eu-west-1 as a region, euwest-1b as a zone, the security group,
docker-machine create --driver amazonec2 \
--amazonec2-access-key **************************** \
--amazonec2-secret-key ********************* \
--amazonec2-ami ami-a8d2d7e \
--amazonec2-region eu-west-1 \
--amazonec2-vpc-id vpc-f65c4212 \
--amazonec2-zone b \
--amazonec2-subnet-id subnet-a1991bcd \
--amazonec2-security-group live_eon01_ec2_sg \
--amazonec2-tags Name,aws-host-1 \
--amazonec2-instance-type t2.micro \
--amazonec2-ssh-keypath /home/eon01/.ssh/id_rsa \
aws-host-1

If you don't need a public IP address you can use this option:
--amazonec2-private-address-only

You can also add other options to the last command like --amazonec2-monitoring to enable CloudWatch monitoring or
--amazonec2-use-ebs-optimized-instance to create an EBS optimized instance.
If the machine creation fails and you want to start a new machine with the same name you should manually
remove the keypair created for aws-host-1.
What I generally do is also force removing the machine (if it is safe to do, of course, otherwise be sure to backup
your data):

docker-machine rm --force -y

aws-host-1 && aws ec2 delete-key-pair --profile me --region eu-west-1 --key-name aws-host-1

You can use your own configurations, choose more or less options, like for example the default Access/Secret Keys,
the default Instance Type, Security Group ..etc:
docker-machine create -d amazonec2 \
--amazonec2-region eu-west-1 \
--amazonec2-ssh-keypath /home/eon01/.ssh/id_rsa \
aws-host-1

We are going to use the first command create command. The latter operation will:
Run some pre-create checks
Start the EC2 instance
Detect some EC2 system configurations
Provision the machine with ubuntu(systemd)
Install Docker
Copy certs to the local machine directory
Copy certs to the remote machine
Setup Docker configuration on the remote daemon
Verify of Docker is up and running on the remote machine
Running pre-create checks...
Creating machine...
(aws-host-1) Launching instance...
Waiting for machine to be running, this may take a few minutes...
Detecting operating system of created instance...
Waiting for SSH to be available...
Detecting the provisioner...
Provisioning with ubuntu(systemd)...
Installing Docker...
Copying certs to the local machine directory...
Copying certs to the remote machine...
Setting Docker configuration on the remote daemon...
Checking connection to Docker...
Docker is up and running!
To see how to connect your Docker Client to the Docker Engine running on this virtual machine, run: docker-machine env awshost-1

Let's now connect Docker Client to the Remote Docker Engine:
eval $(docker-machine env aws-host-1)

Let's verify if the created machine is connected to the local Docker Client:
docker-machine active

We should see the name of the EC2 machine here:
aws-host-1

Creating A Docker Swarm Cluster Using Docker Machine
In this part we are going to demonstrate how to create a Swarm cluster using Docker Machine (1 manager + 1
worker). In order to do this we can follow these steps:

Create the first machine using awscloud driver. This EC2 instance will be a manager.
docker-machine create --driver amazonec2 \
--amazonec2-access-key **************************** \
--amazonec2-secret-key ********************* \
--amazonec2-ami ami-a8d2d7e \
--amazonec2-region eu-west-1 \
--amazonec2-vpc-id vpc-f65c4212 \
--amazonec2-zone b \
--amazonec2-subnet-id subnet-a1991bcd \
--amazonec2-security-group live_eon01_ec2_sg \
--amazonec2-tags Name,aws-host-1 \
--amazonec2-instance-type t2.micro \
--amazonec2-ssh-keypath /home/eon01/.ssh/id_rsa \
aws-host-1

Connect it to the local Docker Client
eval $(docker-machine env aws-host-1)

Initialize the Swarm:
docker swarm init

Copy the generated command in order to use it in the second machine
Create the second machine. This EC2 instance is the worker:
docker-machine create --driver amazonec2 \
--amazonec2-access-key **************************** \
--amazonec2-secret-key ********************* \
--amazonec2-ami ami-a8d2d7e \
--amazonec2-region eu-west-1 \
--amazonec2-vpc-id vpc-f65c4212 \
--amazonec2-zone b \
--amazonec2-subnet-id subnet-a1991bcd \
--amazonec2-security-group live_eon01_ec2_sg \
--amazonec2-tags Name,aws-host-2 \
--amazonec2-instance-type t2.micro \
--amazonec2-ssh-keypath /home/eon01/.ssh/id_rsa \
aws-host-2

Connect it to the local Docker Client
eval $(docker-machine env aws-host-2)

Make this machine join the manager:
docker swarm join \
--token SWMTKN-1-*********************** \
172.32.1.13:2377

Now if you run a Docker service, it will be deployed to the Swarm cluster. We are going the same infinite image that
I created for Docker Swarm chapter:

service create --name infinite_service

eon01/infinite

If you run this directly without connecting the manager to the Docker Client, you will get this error

Error response
from daemon: This node is not a swarm manager. Worker nodes can't be used to view or modify cluster state. Please run this command
on a manager node or promote the current node to a manager.

which is normal, because you are connected to the worker.

Now, we should get back to the manager machine:
eval $(docker-machine env aws-host-1)

Create a new service :
service create --name infinite_service

eon01/infinite

In order to verify that the cluster is working and that both of EC2 created machines are receiving containers, I scaled
the service up to 50 containers docker service scale infinite_service=50 , then I checked the containers living in each
machine using docker ps , each server was hosting 25 instance.
If you would like to automate the creation of Docker Swarm clusters using Docker Machine, you can use these
commands:
docker $(docker-machine config aws-host-1) swarm init --listen-addr $(docker-machine ip aws-host-1):2377
docker $(docker-machine config aws-host-2) swarm join $(docker-machine ip aws-host-1):2377 --listen-addr
machine ip aws-host-2):2377

$(docker-

Using docker-machine config  will avoid changing each time from aws-host-1 to aws-host-2, since the output
of this command is similar to the following:
--tlsverify
--tlscacert="/home/eon01/.docker/machine/machines/aws-host-1/ca.pem"
--tlscert="/home/eon01/.docker/machine/machines/aws-host-1/cert.pem"
--tlskey="/home/eon01/.docker/machine/machines/aws-host-1/key.pem"
-H=tcp://54.229.238.222:2376

The latter configuration will "override" the Docker Daemon default configurations and run a container suing
different TLS parameters and a different tcp socket.
The output of the other embedded command (

docker-machine ip 

) is the IP address of an EC2 machine.

Create Machines On DigitalOcean
The first step is generating a token, login to your DigitalOcean account and generate a new Token:

The second step is to grab the list of images used to provision DO virtual machines. You should execute an API call
curl
-X
GET
"https://api.digitalocean.com/v2/images"
H "Authorization: Bearer ed98d5c230b121ce7f4807a369c5dd1ae8cfcdf87da444ec3ffd50de45c60cd5"

-

You will get a JSON:
Example: If you want to use CoreOs 1353.4.0 (beta), you should look at :
{
"id": 23857817,
"name": "1353.4.0 (beta)",
"distribution": "CoreOS",
"slug": "coreos-beta",
"public": true,
"regions": [
"nyc1",
"sfo1",
"nyc2",
"ams2",
"sgp1",
"lon1",
"nyc3",
"ams3",
"fra1",
"tor1",
"sfo2",
"blr1"
],
"created_at": "2017-04-01T01:56:28Z",
"min_disk_size": 20,
"type": "snapshot",
"size_gigabytes": 0.33
}

We need also to get a list of DO regions:
curl
-X
GET
"https://api.digitalocean.com/v2/regions"
H "Authorization: Bearer ed98d5c230b121ce7f4807a369c5dd1ae8cfcdf87da444ec3ffd50de45c60cd5"

-

In order to get a formatted JSON output, you can add a Python command:
curl
-X
GET
"https://api.digitalocean.com/v2/regions"
H "Authorization: Bearer ed98d5c230b121ce7f4807a369c5dd1ae8cfcdf87da444ec3ffd50de45c60cd5" | python -m json.tool

This is the output of regions list:
{
"links": {},
"meta": {
"total": 12
},
"regions": [
{
"available": true,
"features": [
"private_networking",
"backups",
"ipv6",
"metadata",
"install_agent",
"storage"
],
"name": "New York 1",
"sizes": [
"512mb",
"1gb",
"2gb",
"4gb",
"8gb",
"16gb",
"32gb",
"48gb",
"64gb"
],
"slug": "nyc1"
},
{
"available": true,
"features": [
"private_networking",
"backups",
"ipv6",
"metadata",
"install_agent"
],
"name": "San Francisco 1",
"sizes": [
"512mb",
"1gb",
"2gb",
"4gb",
"8gb",
"16gb",
"32gb",
"48gb",
"64gb"
],
"slug": "sfo1"
},
{
"available": true,
"features": [
"private_networking",
"backups",
"ipv6",
"metadata",
"install_agent"
],
"name": "New York 2",
"sizes": [
"512mb",
"1gb",
"2gb",
"4gb",
"8gb",
"16gb",
"32gb",
"48gb",
"64gb"
],
"slug": "nyc2"
},
{
"available": true,

-

"features": [
"private_networking",
"backups",
"ipv6",
"metadata",
"install_agent"
],
"name": "Amsterdam 2",
"sizes": [
"512mb",
"1gb",
"2gb",
"4gb",
"8gb",
"16gb",
"32gb",
"48gb",
"64gb"
],
"slug": "ams2"
},
.
.
.
.
{
"available": true,
"features": [
"private_networking",
"backups",
"ipv6",
"metadata",
"install_agent"
],
"name": "Bangalore 1",
"sizes": [
"512mb",
"1gb",
"2gb",
"4gb",
"8gb",
"16gb",
"32gb",
"48gb",
"64gb"
],
"slug": "blr1"
}
]
}

We are going to use the "San Fransisco 1" region.
I am a user of DigitalOcean so I had already my SSH key registered. In order to use with Docker Machine, I should
find its fingerprint. In order to this, it depends on your ssh-keygen version, but one of these commands should work
for you:
ssh-keygen -lf /path/to/ssh/key
ssh-keygen -E md5 -lf /path/to/ssh/key

Create the machine:
docker-machine create \
--driver digitalocean \
--digitalocean-access-token=ed98d5c230b121ce7f4807a369c5dd1ae8cfcdf87da444ec3ffd50de45c60cd5 \
--digitalocean-region="sfo1" \
--digitalocean-ssh-key-fingerprint="3c:b1:dd:eg:c9:31:23:4e:13:81:6d:15:5d:e7:32:2d" \
do-host-1

This command will be
Running pre-create checks
Creating SSH key
Assuming Digital Ocean private SSH is located at ~/.ssh/id_rsa

Creating Digital Ocean Droplet
Waiting for IP address to be assigned to the Droplet
Waiting for machine to be running, this may take a few minutes
Detecting operating system of created instance
Waiting for SSH to be available
Detecting the provisioner
Provisioning with ubuntu(systemd)
Installing Docker
Copying certs to the local machine directory
Copying certs to the remote machine
Setting Docker configuration on the remote daemon
Checking connection to Docker
You can add other options like the SSH user name, the image you would like to use .. etc. Example:
docker-machine create \
--driver digitalocean \
--digitalocean-access-token= \
--digitalocean-region="" \
--digitalocean-ssh-key-fingerprint="" \
--digitalocean-ssh-user="" \
--digitalocean-image="" \


Other configurations are possible like:
--digitalocean-size
--digitalocean-ipv6
--digitalocean-private-networking
--digitalocean-backups
--digitalocean-userdata
--digitalocean-ssh-port

If you want to remove the created machine locally and from DO, you should use this command:
docker-machine rm --force -y do-host-1

Like already done, if you want to connect to connect to the remote Docker, type:
eval $(docker-machine env do-host-1)

Now you can create Docker containers in your laptop and see them running in the DO server.

Chapter VIII - Docker Networking
o
o

^__^
(oo)\_______
(__)\
)\/\
||----w |
||
||

Single Host Vs Multi Host Networking
There two different ways of doing networking in Docker:
Networking in a single host
Networking in a cluster of two ore more hosts

Single Host Networking
By default, any Docker container or host will get an IP address that will give it the possibility to communicate with
other containers in the same host or with the host machine. It is possible - as we are going to see - that a Docker
container finds another container by its name, since the IP address could be assigned dynamically at the container
start up, a name is more efficient to find a running container.
Containers in a single host, could also communicate and reach the outside world.
Create a simple container:
docker run -it -d

--name my_container

busybox

And test if you can ping Google:
docker exec -it my_container ping -w3 google.com
PING google.com (216.58.204.142): 56 data bytes
64 bytes from 216.58.204.142: seq=1 ttl=48 time=2.811 ms
--- google.com ping statistics --3 packets transmitted, 1 packets received, 66% packet loss
round-trip min/avg/max = 2.811/2.811/2.811 ms

Now if you inspect the container using
and its IP address:

docker inspect my_container

you will be able to see its network configuration

"NetworkSettings": {
"Bridge": "",
"SandboxID": "555a60eaffdb4b740f7b869bac61859ecca1e39be95ee5856ca28019509e4255",
"HairpinMode": false,
"LinkLocalIPv6Address": "",
"LinkLocalIPv6PrefixLen": 0,
"Ports": {},
"SandboxKey": "/var/run/docker/netns/555a60eaffdb",
"SecondaryIPAddresses": null,
"SecondaryIPv6Addresses": null,
"EndpointID": "20b1b218462e6771155de75788f53b731bbff12019d977aefa7094f57275887d",
"Gateway": "172.17.0.1",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"IPAddress": "172.17.0.2",
"IPPrefixLen": 16,
"IPv6Gateway": "",
"MacAddress": "02:42:ac:11:00:02",
"Networks": {
"bridge": {
"IPAMConfig": null,
"Links": null,
"Aliases": null,
"NetworkID": "2094b393faacbb1cc049f1f136437b1cce6fc41abc304cf2c1ae558a62c5ee2e",
"EndpointID": "20b1b218462e6771155de75788f53b731bbff12019d977aefa7094f57275887d",
"Gateway": "172.17.0.1",
"IPAddress": "172.17.0.2",
"IPPrefixLen": 16,

"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": "02:42:ac:11:00:02"
}
}
}

my_container has the IP address 172.17.0.2 that the host could reach:
ping -w1 172.17.0.2
PING 172.17.0.2 (172.17.0.2) 56(84) bytes of data.
64 bytes from 172.17.0.2: icmp_seq=1 ttl=64 time=0.050 ms
64 bytes from 172.17.0.2: icmp_seq=2 ttl=64 time=0.045 ms
--- 172.17.0.2 ping statistics --2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.045/0.047/0.050/0.007 ms

If you run a web server, your users must reach the port 80 (or 443) of your server, in this case an nginx container, for
example, should be reached at its port 80 (or 443) and it is done through port forwarding that connects it to the host
machine and then an external network (Internet in our case).
Let's create the web server container, forward the port host port 8080 to the container port 80 and test how it
responds:
docker run -d -p 8080:80 --name my_web_server nginx

Ngninx should reply if your port 8080 is not used by other applications:
curl http://0.0.0.0:8080







Welcome to nginx!
If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.
For online documentation and support please refer to
nginx.org.

Commercial support is available at
nginx.com.
Thank you for using nginx.



In a single host, containers are able to see each other, to see the external world (if they are not running in isolated
networks) and they can receive traffic from an external network.

Multi Host Networking
An application could be run using many containers with a load balancer, these containers could be spread across
multiple hosts. Networking in multi host environments is entirely different from single host environments.
Containers even spread across servers should be able to communicate together, this is where service discovery plays

an important role in networking. Service discovery allows you to request a container name and get its IP address
back.
Docker comes with a default network driver called overlay that we are going to see later in this chapter.
If you would like to operate in the multi host mode, you have two options:
Using Docker engine in swarm mode
Running a cluster of hosts using a key value store (like Zookeeper, etcd or Consul) that allows the service
discovery

Docker Networks
Docker Default Networks
The networking is one of the most important parts in building Docker clusters and microservices.
The first thing to know about networking in Docker is listing the different networks that Docker engine uses:
docker network ls

You should get the list of your Docker networks at least those who are pre-defined networks:
NETWORK ID
e5ff619d25f5
98d44bf13233
608a539ce1e3
8sm2anzzfa0i
ede46dbb22d7

NAME
DRIVER
SCOPE
bridge
bridge
local
docker_gwbridge
bridge
local
host
host
local
ingress
overlay
swarm
none
null
local

none, host and bridge are the names of default networks that you can find in any Docker installation running a
single host. The activation of the swarm mode creates another default (or predefined) network called ingress.
Clearly, these networks are not physical networks, they are emulated networks that abstracts hardware and of course
they are in built and managed in higher levels that the physical layer and this one of properties of software-defined
networking.

None Network
The network called none using the null driver is a predefined network that isolates a container in a way it can neither
connect to outisde not communicate with other containers in the same host.
NETWORK ID
ede46dbb22d7

NAME
none

DRIVER
SCOPE
null
local

Let's verify this by running a busybox container in this network:
docker run --net=none -it -d --name my_container busybox

If you inspect the created container docker inspect my_container , you can see the different network configuration of
this container and you will notice that is attached to null network with the id

ede46dbb22d7f3ab2dc95b11228de06e2d27e240a3f651bc2f6fd3ea0c4a2ca7. The container does not have any
gateway.
"NetworkSettings": {
"Bridge": "",
"SandboxID": "566b9a74d37c7f47e02d769b79e168df437a5b23ee030fc199d99f7d94b353b7",
"HairpinMode": false,
"LinkLocalIPv6Address": "",
"LinkLocalIPv6PrefixLen": 0,
"Ports": {},
"SandboxKey": "/var/run/docker/netns/566b9a74d37c",
"SecondaryIPAddresses": null,
"SecondaryIPv6Addresses": null,
"EndpointID": "",
"Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"IPAddress": "",
"IPPrefixLen": 0,
"IPv6Gateway": "",
"MacAddress": "",
"Networks": {
"none": {
"IPAMConfig": null,
"Links": null,
"Aliases": null,
"NetworkID": "ede46dbb22d7f3ab2dc95b11228de06e2d27e240a3f651bc2f6fd3ea0c4a2ca7",
"EndpointID": "b42debd75af122d113c202ad373d46e0b08d32e9ef6e9361e49515045ae6288d",
"Gateway": "",
"IPAddress": "",
"IPPrefixLen": 0,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": ""
}
}
}

Since my_container is attached to none network, it will not have any access to external and internal connections,
let's log into the container and check the network configuration:
docker exec -it my_container sh

Once you are logged inside the container, type
loopback interface.

ifconfig

and you will notice that there is no other interface apart the

/ # ifconfig
lo
Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

If you ping or traceroute an external IP or domain name, you will not be able to do it:
/ # traceroute painlessdocker.com
traceroute: bad address 'painlessdocker.com'
/ # ping painlessdocker.com
ping: bad address 'painlessdocker.com'

Doing the same thing with the loopback address 127.0.0.1 will work:
/ # ping -c 1 127.0.0.1
PING 127.0.0.1 (127.0.0.1): 56 data bytes
64 bytes from 127.0.0.1: seq=0 ttl=64 time=0.085 ms

--- 127.0.0.1 ping statistics --1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0.085/0.085/0.085 ms

The last tests confirm that any container attached to the null network does not know about the outside networks and
no host from outside could access to my_container.
If you want to see the different configurations of this network, type

docker network inspect none

:

[
{
"Name": "none",
"Id": "ede46dbb22d7f3ab2dc95b11228de06e2d27e240a3f651bc2f6fd3ea0c4a2ca7",
"Scope": "local",
"Driver": "null",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": []
},
"Internal": false,
"Containers": {
"69806d9de46c2959d4d20f99660e7d58d7c35c1e0b33511f0b85a395b696786f": {
"Name": "my_container",
"EndpointID": "b42debd75af122d113c202ad373d46e0b08d32e9ef6e9361e49515045ae6288d",
"MacAddress": "",
"IPv4Address": "",
"IPv6Address": ""
}
},
"Options": {},
"Labels": {}
}
]

Docker Host Network
If you want a container to run with a similar networking configuration to the host machine, then you should use the
host network.
NETWORK ID
608a539ce1e3

NAME
host

DRIVER
host

To see the configuration of this network, type

SCOPE
local

docker inspect network host

:

[
{
"Name": "host",
"Id": "608a539ce1e3e3b97964c6a2fe06eb0e0a9b539e659025fbd101b24e327d8da6",
"Scope": "local",
"Driver": "host",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": []
},
"Internal": false,
"Containers": {},
"Options": {},
"Labels": {}
}
]

Let's run a web server using nginx and use this network:

docker run -d -p 80 nginx

Verify the output of
CONTAINER ID
d0df14bf80a0
69806d9de46c

docker ps

IMAGE
nginx
busybox

:

COMMAND
"nginx -g 'daemon off"
"sh"

PORTS
443/tcp, 0.0.0.0:32769->80/tcp

NAMES
stoic_montalcini
my_container

Notice that we are using only the port 80, since we added the -p flag ( -p 80 ) that made the port 80 accessible
from the host at port 32769 (that Docker chose automatically) . We could have used an external binding to port 80
that we choose manually, say port 8080, in this case the command to run this container will be:
docker run -d -p 8080:80 nginx

In all cases, the port 80 of the container will be accessible either from the port 32769 or the port 8080 (in the second
case) but in both cases, the IP address will be the same as the host IP ( 127.0.0.1 or 0.0.0.0 ).
Let's verify it by running

curl -I http://0.0.0.0:32768

to see whether the server response will be 200 or not:

HTTP/1.1 200 OK
Connection: Keep-Alive
Keep-Alive: timeout=20
Date: Sat, 07 Jan 2017 20:49:02 GMT
Content-Type: text/html

So when running

curl -I http://0.0.0.0:23768

, nginx will reply with a 200 response. This is the content of the page:

curl http://0.0.0.0:32768
Index of /
../

modified: Sat, 07 Jan 2017 20:38:03 GMT
directory - 4.00 kbyte




Bridge Network
This is the default containers network, any network that runs without the --net flag will be attached automatically
to this network. Two Docker containers running in this network could see each other.
To create two containers, type:
docker run -it -d
docker run -it -d

--name my_container_1 busybox
--name my_container_2 busybox

These containers will be attached to the bridge network, if you type
that they are and what IP addresses the will have:

docker network inspect bridge

[
{
"Name": "bridge",
"Id": "e5ff619d25f5dfa2e9b4fe95db8136b74fa61b588fb6141b7d9678adafd155a7",
"Scope": "local",
"Driver": "bridge",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{

, you will notice

"Subnet": "172.17.0.0/16",
"Gateway": "172.17.0.1"
}
]
},
"Internal": false,
"Containers": {
"1172afcb3363f36248701aaa0ba9c1080ebc94db6a168f188f6ba98907e22102": {
"Name": "my_container_1",
"EndpointID": "b8f4741fb2008b70b60a0375446653f820fcaf6b1d8279c1b7d0abbb5775aeaf",
"MacAddress": "02:42:ac:11:00:04",
"IPv4Address": "172.17.0.4/16",
"IPv6Address": ""
},
"6895aa358faea0226ba646544056c34063a0ef5b83d10e68500936d0a397bb7b": {
"Name": "my_container_2",
"EndpointID": "2d3c6c8ca175c0ecb35459dcd941c0456fbcbf8fcce4885aa48eb06b9cff19b8",
"MacAddress": "02:42:ac:11:00:05",
"IPv4Address": "172.17.0.5/16",
"IPv6Address": ""
}
},
"Options": {
"com.docker.network.bridge.default_bridge": "true",
"com.docker.network.bridge.enable_icc": "true",
"com.docker.network.bridge.enable_ip_masquerade": "true",
"com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
"com.docker.network.bridge.name": "docker0",
"com.docker.network.driver.mtu": "1500"
},
"Labels": {}
}
]

my_container_1 has the IP address 172.17.0.4/16
my_container_2 has the IP address 172.17.0.5/16
From any of the created containers, say my_container_1 you could see the other container, just type
my_container_1 ping 172.17.0.5 to ping the my_container_2 from my_container_1:

docker exec -it

docker exec -it my_container_1 ping 172.17.0.5
PING 172.17.0.5 (172.17.0.5): 56 data bytes
64 bytes from 172.17.0.5: seq=0 ttl=64 time=0.156 ms
64 bytes from 172.17.0.5: seq=1 ttl=64 time=0.071 ms

The containers running in the bridge network could see each other by IP address, let's see if it is possible to ping
using the container name:
docker exec -it my_container_1 ping my_container_2
ping: bad address 'my_container_2'

As you see, it is not possible for Docker to associate a container name to an IP and this is not possible
because we do not run any discovery service. Creating a user-defined network could solve this problem.

docker_gwbridge Network
When you create a swarm cluster or join one, Docker will create by default a network called docker_gwbridge that
will be used to connect different containers from different hosts. These hosts are part of the swarm cluster.
In general, this network provides the containers not having an access to external networks with connectivity.
docker network ls
NETWORK ID

NAME

DRIVER

SCOPE

98d44bf13233

docker_gwbridge

bridge

local

Running overlay networks always need the docker_gwbridge.

Software Defined & Multi Host Networks
This type of networks, in opposite to the default networks, does not come with a fresh Docker installation , but
should be created by the user. The simplest way to create a new network is:
docker network create my_network

For more general use, this is the command to use:
docker network create  

Bridge Networks
You can use many network drivers like bridge or overlay, you may also need to set up a IP range or a subnet for
your network or you will probably need to setup your own gateway. Type docker network create --help for more
options and configurations:
--aux-address value
-d or --driver string
--gateway value
--help
--internal
--ip-range value
--ipam-driver string
--ipam-opt value
--ipv6
--label value
-o or --opt value
--subnet value

Auxiliary IPv4 or IPv6 addresses used by Network driver (default map[])
Driver to manage the Network (default "bridge")
IPv4 or IPv6 Gateway for the master subnet (default [])
Print usage
Restrict external access to the network
Allocate container ip from a sub-range (default [])
IP Address Management Driver (default "default")
Set IPAM driver specific options (default map[])
Enable IPv6 networking
Set metadata on a network (default [])
Set driver specific options (default map[])
Subnet in CIDR format that represents a network segment (default [])

In order to use the Docker service discovery, let's create a second bridge network:
docker network create --driver bridge my_bridge_network

To see the new network, type:
docker network ls

NETWORK ID
5555cd178f99

NAME
my_bridge_network

DRIVER
bridge

SCOPE
local

my_container_1 and my_container_2 are running in the default bridge network, we want them attached to the new
network, we should type:
docker network connect my_bridge_network my_container_1
docker network connect my_bridge_network my_container_2

Now, the service discovery is working and both a Docker container could access another one by its name.
docker exec -it my_container_1 ping my_container_2
PING my_container_2 (172.18.0.3): 56 data bytes
64 bytes from 172.18.0.3: seq=0 ttl=64 time=0.120 ms
64 bytes from 172.18.0.3: seq=1 ttl=64 time=0.081 ms

Docker containers running in a user-defined bridge network could see each other by their containers' names.
Let's create a more personalized network with a specified subnet, gateway and IP range and let's also change the
behavior of networking in this network by decreasing the size of the largest network layer protocol data unit that can
be communicated in a single network transaction, or what we call MTU (Maximum Transmission Unit):
docker network create -d bridge \
--subnet=192.168.0.0/16 \
--gateway=192.168.0.100 \
--ip-range=192.168.1.0/24 \
--opt "com.docker.network.driver.mtu"="1000"
my_personalized_network

You can change other options using the

--opt

or

-o

flag like:

: bridge name to be used when creating the Linux bridge
: Enable IP masquerading
com.docker.network.bridge.enable_icc : Enable or Disable Inter Container Connectivity
com.docker.network.bridge.host_binding_ipv4 : Default IP when binding container ports
com.docker.network.bridge.name

com.docker.network.bridge.enable_ip_masquerade

For any newly created bridge network, an interface is created in the container, just type
container to see them:

ifconfig

inside the

Let's connect my_container_1 to my_personalized_network:
docker network connect my_personalised_network my_container_1

Executing

ifconfig

inside this container (

docker exec -it my_container_1 ifconfig

) will show us two things:

This container is running inside more than a container because it was connected to my_bridge_network and
now we connected it to my_personalized_network.
The MTU changed to 1000.
eth0

Link encap:Ethernet HWaddr 02:42:AC:11:00:04
inet addr:172.17.0.4 Bcast:0.0.0.0 Mask:255.255.0.0
inet6 addr: fe80::42:acff:fe11:4/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:194 errors:0 dropped:0 overruns:0 frame:0
TX packets:21 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:34549 (33.7 KiB) TX bytes:1410 (1.3 KiB)

eth1

Link encap:Ethernet HWaddr 02:42:C0:A8:01:00
inet addr:192.168.1.0 Bcast:0.0.0.0 Mask:255.255.0.0
UP BROADCAST RUNNING MULTICAST MTU:1000 Metric:1
RX packets:1 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:87 (87.0 B) TX bytes:0 (0.0 B)

lo

Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:16 errors:0 dropped:0 overruns:0 frame:0
TX packets:16 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:0
RX bytes:1240 (1.2 KiB) TX bytes:1240 (1.2 KiB)

docker run supports only one network, bur
add it to many networks.

docker network connect

could be used after the container creation to

By default, two containers living in the same host but in different networks will not see each other. Docker
daemon runs a tiny DNS server that allows user-defined networks to make service discovery.

docker_gwbridge Network
You can create new docker_gwbridge networks using

docker network create

command.

Example:
docker network create --subnet 172.3.0.0/16 \
--opt com.docker.network.bridge.name=another_docker_gwbridge \
--opt com.docker.network.bridge.enable_icc=false \
--opt com.docker.network.bridge.enable_ip_masquerade=true \
another_docker_gwbridge

Overlay Networks
Overlay networks are used in multi host environments (like the swarm mode of Docker). You can create an overlay
network using docker network create command but after the activation of swarm mode:
docker swarm init

docker network create --driver overlay --subnet 10.0.9.0/24

my_network

This is the configuration of the latter network:
[
{
"Name": "my_network",
"Id": "2g3i0zdldo4adfqisvqjn6gpt",
"Scope": "swarm",
"Driver": "overlay",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "10.0.9.0/24",
"Gateway": "10.0.9.1"
}
]
},
"Internal": false,
"Containers": null,
"Options": {
"com.docker.network.driver.overlay.vxlanid_list": "257"
},
"Labels": null
}
]

Say we have two hosts in a Docker cluster (having 192.168.10.22 and 192.168.10.23 as IP addresses) , the different
containers attached to the latter overlay network will have dynamically allocated IP addresses in the network subnet
10.0.9.0/24.

VXLAN (Virtual Extensible LAN) - according to Wikipedia - is a network virtualization technology that attempts to
improve the scalability problems associated with large cloud computing deployments.
It uses a VLAN-like encapsulation technique to encapsulate MAC-based OSI layer 2 Ethernet frames within layer 4
UDP packets, using 4789 as the default destination UDP port number.
To better understand the context, this diagram illustrates the 7 layers of the OSI model:

VXLAN endpoints, which terminate VXLAN tunnels and may be both virtual or physical switch ports, are known as
VXLAN tunnel endpoints (VTEPs).
VXLAN is an evolution of efforts to standardize on an overlay encapsulation protocol. It increases scalability up to
16 million logical networks and allows for layer 2 adjacency across IP networks.
Open vSwitch is an example of a software-based virtual network switch that supports VXLAN overlay networks.
The network driver overlay works on the VXLAN tunnels (connecting VTEPs) and need a key/value store.

A VTEP has two logical interfaces: an uplink and a downlink where the uplink is acting like a tunnel endpoint
having an IP address to receive sent VXLAN frames.

Flannel

Kubernetes does away with port-mapping and assigns a unique IP address to each pod and this works well in Google
Compute but for some other cloud providers a host can not get an entire subnet, this is why Flannel solves this
problem and creates an overlay mesh network that provision each host with a subnet: Each pod (if you are using
Kubernetes) or container has a unique and routable IP inside the cluster.

According to CoreOs creators, Kubernetes and then Flannel works great with CoreOS to distribute a workload
across a cluster.
Flannel was designed to use with Kubernetes but it could be used as a generic overlay network driver to create
software-defined overlay networks since it supports VXLAN, AWS VPC, and the default layer 2 UDP overlay
network.
Flannel uses etcd to store the network configuration (VMs subnets,hosts' IPs etc ..) and among other back-ends it
uses UDP and a TUN device in order to encapsulate an IP fragment in a UDP packet. The latter transports some
information like the MAC, the outer IP, the inner IP and the playload.

Note that the IP fragmentation is a process that happens in the Internet Protocol (IP) in order to fragment or
breaks the sent datagrams into smaller pieces (called generally fragments). This way, the formed packets could pass
through a link with a smaller maximum transmission unit (MTU) than the original UDP datagram size. And of
course the fragments are reassembled by the receiving host.
This schema taken from the official Github repository explains well how Flannel networking works in general:

To install Fannel you need to build it from source:
sudo apt-get install linux-libc-dev golang gc
git clone https://github.com/coreos/flannel.git
cd flannel; make dist/flanneld

Weave
Weaveworks created Weave (or Weave Net) which is a virtual network that connects Docker containers deployed
across multiple hosts.

In order to install Weave:
sudo curl -L git.io/weave -o /usr/local/bin/weave
sudo chmod a+x /usr/local/bin/weave

Now just type

weave

to download

weaveworks/weaveexec

Docker image and see the help:

Usage:
weave --help | help
setup
version
weave launch

launch-router [--password ] [--trusted-subnets ,...]
[--host ]
[--name ] [--nickname ]
[--no-restart] [--resume] [--no-discovery] [--no-dns]
[--ipalloc-init ]
[--ipalloc-range  [--ipalloc-default-subnet ]]
[--log-level=debug|info|warning|error]
 ...
launch-proxy [-H ] [--without-dns] [--no-multicast-route]
[--no-rewrite-hosts] [--no-default-ipalloc] [--no-restart]
[--hostname-from-label ]
[--hostname-match ]
[--hostname-replacement ]
[--rewrite-inspect]
[--log-level=debug|info|warning|error]
launch-plugin [--no-restart] [--no-multicast-route]
[--log-level=debug|info|warning|error]
weave prime
weave env
config
dns-args

[--restore]

weave connect
forget

[--replace] [ ...]
 ...

weave run

[--without-dns] [--no-rewrite-hosts] [--no-multicast-route]
[ ...]  ...

start
attach
detach
restart

[ ...] 
[ ...] 
[ ...] 


weave expose
hide

[ ...] [-h ]
[ ...]

weave dns-add

[ ...]  [-h ] |
 ... -h 
[ ...]  [-h ] |
 ... -h 


dns-remove
dns-lookup
weave status
report
ps

[targets | connections | peers | dns | ipam]
[-f ]
[ ...]

weave stop
stop-router
stop-proxy
stop-plugin
weave reset
rmpeer

where 






[--force]
 ...

=
=
=
=
=
=

[:]
/
[ip:] | net: | net:default
[tcp://][]: | [unix://]/path/to/socket
 | 
consensus[=] | seed=,... | observer

Start Weave router:
weave launch

After typing the latter command, you will notice that some other Docker images are pulled, it's alright, Weave needs
weavedb and weaveplugin:
Unable to find image 'weaveworks/weavedb:latest' locally
latest: Pulling from weaveworks/weavedb
1266eb846caf: Pulling fs layer
1266eb846caf: Download complete
1266eb846caf: Pull complete
Digest: sha256:c43f5767a1644196e97edce6208b0c43780c81a2279e3421791b06806ca41e5f
Status: Downloaded newer image for weaveworks/weavedb:latest
Unable to find image 'weaveworks/weave:1.8.2' locally
1.8.2: Pulling from weaveworks/weave
e110a4a17941: Already exists
199ab7eb2ba4: Already exists
8c419735a809: Already exists
1888d0f92b68: Already exists
f4d1c90c86a4: Already exists
1d6a7435ac59: Already exists
7372f3ee9e8b: Already exists
17004cbabd74: Already exists
b8e5c537a426: Already exists
4e295f039ae0: Pulling fs layer
d67a003dc85f: Pulling fs layer
2a84c77046e7: Pulling fs layer
d67a003dc85f: Download complete
2a84c77046e7: Verifying Checksum
2a84c77046e7: Download complete
4e295f039ae0: Verifying Checksum
4e295f039ae0: Download complete
4e295f039ae0: Pull complete
d67a003dc85f: Pull complete
2a84c77046e7: Pull complete
Digest: sha256:7a9ec1daa3b9022843fd18986f1bd5c44911bc9f9f40ba9b4d23b1c72c51c127
Status: Downloaded newer image for weaveworks/weave:1.8.2
Unable to find image 'weaveworks/plugin:1.8.2' locally
1.8.2: Pulling from weaveworks/plugin
e110a4a17941: Already exists
199ab7eb2ba4: Already exists
8c419735a809: Already exists
1888d0f92b68: Already exists
f4d1c90c86a4: Already exists
1d6a7435ac59: Already exists
7372f3ee9e8b: Already exists
17004cbabd74: Already exists
b8e5c537a426: Already exists

4e295f039ae0: Already exists
d67a003dc85f: Already exists
2a84c77046e7: Already exists
2b57c438a07b: Pulling fs layer
2b57c438a07b: Verifying Checksum
2b57c438a07b: Download complete
2b57c438a07b: Pull complete
Digest: sha256:3a38cec968bff6ebc4b1823673378b14d52ef750dec89e3513fe78119d07fdf2
Status: Downloaded newer image for weaveworks/plugin:1.8.2

Let's create a subnet for the host:
weave expose 10.10.0.1/16

And inspect the output of
bridge name
br-0ebcbf638b08
br-5555cd178f99
br-da47f0537ffa
docker0
docker_gwbridge
lxcbr0
weave

brctl show

bridge id
8000.0242a98e8f09
8000.0242d38260e6
8000.024261465302
8000.02422492c7fe
8000.0242e5db7cff
8000.000000000000
8000.26eea4e57577

command as well as the newly created interface:
STP enabled
interfaces
no
no
no
no
no
veth6e6c262
no
no
vethwe-bridge

ifconfig weave

weave

Link encap:Ethernet HWaddr 26:ee:a4:e5:75:77
inet addr:10.10.0.1 Bcast:0.0.0.0 Mask:255.255.0.0
UP BROADCAST RUNNING MULTICAST MTU:1410 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:67 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:10738 (10.7 KB)

You can use

ifdown weave

to stop the created interface.

brctl is used to set up, maintain, and inspect the ethernet bridge configuration in the Linux Kernel.
If you want more information about Weave running in your host, type

Version: 1.8.2 (up to date; next check at 2017/01/16 01:16:44)
Service:
Protocol:
Name:
Encryption:
PeerDiscovery:
Targets:
Connections:
Peers:
TrustedSubnets:

router
weave 1..2
26:ee:a4:e5:75:77(eonSpider)
disabled
enabled
0
0
1
none

Service:
Status:
Range:
DefaultSubnet:

ipam
idle
10.32.0.0/12
10.32.0.0/12

Service:
Domain:
Upstream:
TTL:
Entries:

dns
weave.local.
127.0.1.1
1
0

Service: proxy
Address: unix:///var/run/weave/weave.sock
Service: plugin

weave status

:

DriverName: weave

Now you can start a container directly from Weave CLI:
weave run 10.2.0.2/16 -it -d busybox

When you type
CONTAINER ID
0bedd8bf148a
575aee35ec8d
9e7a5a87b137
0edc17ad4a49

docker ps

you will see the last created container as well as Weave containers:

IMAGE
busybox
weaveworks/plugin:1.8.2
weaveworks/weaveexec:1.8.2
weaveworks/weave:1.8.2

COMMAND
NAMES
"sh"
sick_raman
"/home/weave/plugin"
weaveplugin
"/home/weave/weavepro" weaveproxy
"/home/weave/weaver -" weave

To connect two containers in two distinct hosts using Weave, launch these commands in the first host ($HOST1):
weave launch
eval $(weave env)
docker run --name c1 -ti busybox

and in the second host, tell $HOST2 to peer with Weave already started on $HOST1:
weave launch $HOST1
eval $(weave env)
docker run --name c2 -ti busybox

$HOST1 is the IP address of the first host.
The following image explains how a simple communication between two containers living in two distinct hosts can
communicate:

In order to automate the discovery process in a Swarm cluster, start by having the Swarm token:
curl -X POST https://discovery-stage.hub.docker.com/v1/clusters && echo -e "\n"

This is my token, you should of course get a different one:
10fd726a3e5341f86fb90658208e564a

If you haven't already launched weave, do it:
weave launch

Now download the discovery script:
curl -O https://raw.githubusercontent.com/weaveworks/discovery/master/discovery && chmod a+x discovery

The script will be downloaded to the current directory, you should move it to a directory like
want to use as a system executable.

/usr/bin

if you

Do you remember your token ? You will use it here:
discovery join --advertise-router token://10fd726a3e5341f86fb90658208e564a

Until now, we are working in $HOST1. Go to $HOST2 and repeat the same commands:
weave launch
curl -O http://git.io/vmW3z && chmod a+x discovery
discovery join --advertise-router token://10fd726a3e5341f86fb90658208e564a

Both Weave routers should be and should stay connected.
This is how you can use the discovery command:
Weave Discovery
discovery join [--advertise=ADDR]
[--advertise-external]
[--advertise-router]
[--weave=ADDR[:PORT]]



where  = backend://path
 = host|IP

To leave a cluster, you should use the following command in the host that you want to leave the cluster:
discovery leave

If you are using a KV store like etcd, you can also consider using it:
discovery join etcd://some/path

These are the important steps to use Weave service discovery, it is quite similar to Swarm CLI. We are going to see
Swarm in details in some next parts of this book and you will be able to better understand the discovery.

Open vSwitch
Licensed under the open source Apache 2.0 license, the multilayer virtual switch Open vSwitch is designed to enable
massive network automation through programmatic extension, while still supporting standard management
interfaces and protocols (e.g. NetFlow, sFlow, IPFIX, RSPAN, CLI, LACP, 802.1ag).
Open vSwitch is also designed to support distribution across multiple physical servers similar to VMware's
vNetwork distributed vswitch or Cisco's Nexus 1000V.

In order to connect containers in multiple hosts, you need to install OpenvSwitch in all hosts:
apt-get install -y openvswitch-switch bridge-utils

You may need some dependencies:
sudo apt-get install -y build-essential fakeroot debhelper \
autoconf automake bzip2 libssl-dev \
openssl graphviz python-all procps \
python-qt4 python-zopeinterface \
python-twisted-conch libtool

Then install

ovs

utility:

cd /usr/bin
wget https://raw.githubusercontent.com/openvswitch/ovs/master/utilities/ovs-docker
chmod a+rwx ovs-docker

Single Host

We start by creating an OVS bridge called ovs-br1:
ovs-vsctl add-br ovs-br1

Then we activate it and give it an IP and a netmask:
ifconfig ovs-br1 173.17.0.1 netmask 255.255.255.0 up

Verify your new interface configuration by typing:
ifconfig ovs-br1

ovs-br1

Link encap:Ethernet HWaddr e6:58:8f:58:89:43
inet addr:173.17.0.1 Bcast:173.17.0.255 Mask:255.255.255.0
inet6 addr: fe80::e458:8fff:fe58:8943/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1
RX bytes:0 (0.0 B) TX bytes:648 (648.0 B)

Let's create two containers:
docker run -it --name mycontainer1 -d
docker run -it --name mycontainer2 -d

busybox
busybox

And connect them to the bridge:
ovs-docker add-port ovs-br1 eth1 mycontainer1 --ipaddress=173.16.0.2/24
ovs-docker add-port ovs-br1 eth1 mycontainer2 --ipaddress=173.16.0.3/24

You can now ping the second container from the first one:
PING 173.16.0.3 (173.16.0.3): 56 data bytes
64 bytes from 173.16.0.3: seq=0 ttl=64 time=0.338 ms
64 bytes from 173.16.0.3: seq=1 ttl=64 time=0.061 ms
^C

Do the same thing from the second container:
docker exec -i mycontainer2 ping 173.16.0.2
PING 173.16.0.2 (173.16.0.2): 56 data bytes
64 bytes from 173.16.0.2: seq=0 ttl=64 time=0.067 ms
64 bytes from 173.16.0.2: seq=1 ttl=64 time=0.077 ms
^C

Multi Host
One can ask oneself, why can't we use regular Linux bridges ? I will use the official FAQ of Open vSwitch to answer
this:
Q: Why would I use Open vSwitch instead of the Linux bridge? A: Open vSwitch is specially designed to make
it easier to manage VM network configuration and monitor state spread across many physical hosts in dynamic
virtualized environments. Please see [WHY-OVS.md] for a more detailed description of how Open vSwitch
relates to the Linux Bridge
Create two hosts (Host1 and Host2) accessible to each other.
For this example, we have two VMs and of course, openvswitch-switch, Docker and ovs-docker tool should be
installed in both hosts.

wget -qO- https://get.docker.com/|sh

apt-get install openvswitch-switch bridge-utils openvswitch-common

On Host1: Create a new ovs bridge and a veth pair, set them to up then create the tunnel between Host1 and Host2.
Make sure to change  by the real IP of Host2.
ovs-vsctl add-br br-int
ip link add veth0 type veth peer name veth1
ovs-vsctl add-port br-int veth1
brctl addif docker0 veth0
ip link set veth1 up
ip link set veth0 up
ovs-vsctl add-port br-int gre0 -- set interface gre0 type=gre options:remote_ip=

On Host2: Do the same thing, create a new ovs bridge and a veth pair, set them to up then create the tunnel between
Host1 and Host2. Make sure to change  by the real IP of Host1.
ovs-vsctl add-br br-int
ip link add veth0 type veth peer name veth1
ovs-vsctl add-port br-int veth1
brctl addif docker0 veth0
ip link set veth1 up
ip link set veth0 up
ovs-vsctl add-port br-int gre0 -- set interface gre0 type=gre options:remote_ip=

You can see the created bridge on each host by typing

ovs-vsctl show

.

On Host1:
0aaba889-1d8c-4db2-b783-d7a203853d44
Bridge br-int
Port "veth1"
Interface "veth1"
Port br-int
Interface br-int
type: internal
Port "gre0"
Interface "gre0"
type: gre
options: {remote_ip=""}
ovs_version: "2.5.0"

On Host2:
7e782730-8990-4786-b2b0-efef7721665b
Bridge br-int
Port "veth1"
Interface "veth1"
Port "gre0"
Interface "gre0"
type: gre
options: {remote_ip=""}
Port br-int
Interface br-int
type: internal
ovs_version: "2.5.0"

and are of course changed by their real values. You can also use
Now, create a container in Host1:
docker run -it --name container1 -d busybox

View its IP address:

brctl show

command for more information.

docker inspect --format '{{.NetworkSettings.IPAddress}}' container1

On Host2, do the same thing
docker run -it --name container1 -d busybox
docker inspect --format '{{.NetworkSettings.IPAddress}}' container1

You will notice that both containers have the same IP address 172.17.0.2 , this could create a conflict in the cluster,
so we are going to create a second container. The latter will have a different IP:
docker run -it --name container2 -d busybox
docker inspect --format '{{.NetworkSettings.IPAddress}}' container2

From the container1 in Host1, ping container2 in Host2:
docker exec -it container1 ping -c 2 172.17.0.3
PING 172.17.0.3 (172.17.0.3): 56 data bytes
64 bytes from 172.17.0.3: seq=0 ttl=64 time=0.985 ms
64 bytes from 172.17.0.3: seq=1 ttl=64 time=0.963 ms
--- 172.17.0.3 ping statistics --2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.963/0.974/0.985 ms

From container2 in Host2, ping the container1 in Host1, but before this remove the container1 in the same host
(Host2) in order to be sure that we are pinging the right container (container1 in Host1):
docker rm -f container1

docker exec -it container2 ping -c 2 172.17.0.2
PING 172.17.0.2 (172.17.0.2): 56 data bytes
64 bytes from 172.17.0.2: seq=0 ttl=64 time=1.475 ms
64 bytes from 172.17.0.2: seq=1 ttl=64 time=1.139 ms
--- 172.17.0.2 ping statistics --2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 1.139/1.307/1.475 ms

Now you have seen how can we connect two containers on different hosts using Open vSwitch.
To go beyond that, you may notice that both docker0 interface has the same IP address which is 172.17.0.1:

This similarity could create confusion within a multihost network.
This is why we are going to remove docker0 bridge interface and create a new one with different subnets. You are
free to make any choice of private IP addresses, I am going to use this:
Host1 : 192.168.10.1/16
Host2 : 192.168.11.1/16
In order to change an IP address of an interface, you can use
ifconfig
Usage:
ifconfig [-a] [-v] [-s]  [[] 
]
[add 
[/]]
[del 
[/]]
[[-]broadcast [
]] [[-]pointopoint [
]]
[netmask 
] [dstaddr 
] [tunnel 
]
[outfill ] [keepalive ]
[hw  
] [metric ] [mtu ]
[[-]trailers] [[-]arp] [[-]allmulti]
[multicast] [[-]promisc]
[mem_start ] [io_addr ] [irq ] [media ]
[txqueuelen ]
[[-]dynamic]
[up|down] ...

Usage: brctl [commands]

ifconfig

command and

brctl

command.

This is a practical example but before that make sure your firewall rules will not stop you doing the next steps, make
also sure that you stop Docker service docker stop :
Deactivate

docker0

by bringing it down:

ifconfig docker0 down

Delete the bridge and create a new one having the same interface name:
brctl delbr docker0
brctl addbr docker0

Bring it up while assigning it a new address and a new

mtu

:

sudo ifconfig docker0 192.168.10.1/16 mtu 1400 up

The last change will not be persistent unless you add it to
bip=192.168.10.1/16 to DOCKER_OPTS .

/etc/default/docker

configuration file and add

--

Example:
DOCKER_OPTS="--dns 8.8.8.8 --dns 8.8.4.4 --bip=10.11.12.1/24"

If you are using Ubuntu/Debian, you can use a script that I found in a Github gist and tested, it will do this
automatically for you,I forked it here: https://gist.github.com/eon01/b7fbfa3309ed4f514bc742045ce9b5a2 , you can use it this
way:
Example1:
Example2:

wget http://bit.ly/2kHIbVc && bash configure_docker0.sh 192.168.10.1/16
wget http://bit.ly/2kHIbVc && bash configure_docker0.sh 192.168.11.1/16

For formatting reasons, I used bit.ly to shorten the url, this is the real url :
https://gist.githubusercontent.com/eon01/b7fbfa3309ed4f514bc742045ce9b5a2/raw/7bb94c774510505196151c5d787ce865140ace9c/configure_docker0.sh

To use this script:

Make sure you choose the network you want to use instead of the networks used in the examples
Make sure you are using Debian/Ubuntu
This script must be run with your root user
Docker must be stopped
Change will happen after starting Docker

Project Calico
Calico provides a different approach since it uses the layer 3 to provide the virtual networking feature. It includes
pre-integration with Kubernetes and Mesos (as a CNI network plugin), Docker (as a libnetwork plugin) and
OpenStack (as a Neutron plugin). It supports many public and private cloud like AWS, GCE.

Almost all of the other networking solutions (like Weave and Fannel) encapsulate layer 2 traffic into a higher level
to build an overlay network while the primary operating mode of project Calico requires no encapsulation.
Based on the same scalable IP network principles as the Internet, Calico leverages the existing Linux Kernel
forwarding engine without the need for virtual switches or overlays. Each host propagates workload reachability
information (routes) to the rest of the data center – either directly in small scale deployments or via infrastructure
route reflectors to reach Internet level scales in large deployments.
Like it is described in the official documentation of the project, Calico simplifies the network topology, removing
multiple encapsulation and de-encapsulation which gives some strengths to this networking system:
Smaller packet sizes mean that there is reduction in possible packet fragmentation.
There is a reduction in CPU cycles handling the encap and de-encap.
Easier to interpret packets, and therefore easier to troubleshoot.

Project Calico is most compatible with data centers where you have control over the physical network fabric.

Pipework
Pipework is a Software-Defined Networking tools for LXC that lets you connect together containers in arbitrarily
complex scenarios. It uses cgroups and namespace, works with containers created with lxc-start (plain LXC) and
with Docker.
In order to install it, you can execute the installation script from its Github repository :
sudo bash -c "curl https://raw.githubusercontent.com/jpetazzo/pipework/master/pipework > /usr/local/bin/pipework"

Since its creation, Docker is allowing more complex scenarios, and Pipework is becoming obsolete. Given the
Docker, Inc., acquisition of SocketPlane and the introduction of the Overlay Driver, you better use Docker Swarm
built-in orchestration unless you have very specific needs.

OpenVPN
Using OpenVPN, you can create virtual private networks (VPNs), you can use a VPN network to connect different
VMs in the same data center or a multi-cloud VMs in order to connect distributed containers. This connection will be
of course secure (TLS).

Service Discovery
Etcd
etcd is a distributed, key-value store for shared configuration and service discovery, with features like:
A user-facing API (gRPC)
Automatic TLS with optional client cert authentication
Rapidity (Benchmarked 10,000 writes/sec according to CoreOS)
Properly distributed using Raft
etcd is written in Go and uses the Raft consensus algorithm to manage a highly-available replicated log. It is a
production-ready software widely used with tools like Kubernetes, fleet, locksmith, vulcand, Doorman.
In order to rapidly setup use etcd on AWS, you can use the official AMI

To test etcd, you can create a CoreOS cluster (with 3 machines) and for simplicity sake, I am going to use Digital
Ocean.
The first thing to do here before having a new CoreOS cluster is is generating a new discovery URL. You can do
this by using https://discovery.etcd.io url. This will print a new discovery url:
curl -w "\n" "https://discovery.etcd.io/new?size=3"

This is my discovery url :
https://discovery.etcd.io/d9fe2c6051e8204e2fa730ccc815e76b

We are going to use this url in the cloud-config configuration.
You should change the discovery url by your own generated url:
#cloud-config
coreos:
etcd2:
# generate a new token for each unique cluster from https://discovery.etcd.io/new:
discovery: https://discovery.etcd.io/d9fe2c6051e8204e2fa730ccc815e76b
# multi-region deployments, multi-cloud deployments, and Droplets without
# private networking need to use $public_ipv4:
advertise-client-urls: http://$private_ipv4:2379,http://$private_ipv4:4001
initial-advertise-peer-urls: http://$private_ipv4:2380
# listen on the official ports 2379, 2380 and one legacy port 4001:
listen-client-urls: http://0.0.0.0:2379,http://0.0.0.0:4001

listen-peer-urls: http://$private_ipv4:2380
fleet:
public-ip: $private_ipv4
# used for fleetctl ssh command
units:
- name: etcd2.service
command: start
- name: fleet.service
command: start

When creating your 3 CoreOS VMs make sure to activate private networking and pasting your cloud-config
configuration.

The cloud-config will not work for you if you forget to add the first line
Create your 3 machines:

#cloud-config

Go get a cup of coffee unless you created small VMs:

Log to one of the created machines, now you can type fleetctl list-machines in order to see all of the created
machines.
If you want another machine to join the same cluster, you can use the same cloud-config file again and your machine
will join the cluster automatically.
You can find the discovery url by typing

grep DISCOVERY /run/systemd/system/etcd2.service.d/20-cloudinit.conf

etcd is written in the Go language and developed by CoreOS team.

Consul
Consul is a tool for service discovery and configuration that runs on Linux, Mac OS X, FreeBSD, Solaris, and
Windows. Consul is distributed, highly available, and extremely scalable.
It provides several key features:

Service Discovery - Consul makes it easy for services to register themselves and to discover other services via
a DNS or HTTP interface. External services such as SaaS providers can also be registered.
Health Checking - Health Checking enables Consul to quickly alert operators about any issues in a cluster. The
integration with service discovery prevents routing traffic to unhealthy hosts and enables service level circuit
breakers.
Key/Value Storage - A flexible key/value store enables storing dynamic configuration, feature flagging,
coordination, leader election and more. The simple HTTP API makes it easy to use anywhere.
Multi-Datacenter - Consul is built to be datacenter aware, and can support any number of regions without
complex configuration.
For simplicity sake, I will use a Docker container to run Consul:
docker run -p 8400:8400 -p 8500:8500 \
> -p 8600:53/udp -h consul_s progrium/consul -server -bootstrap
Unable to find image 'progrium/consul:latest' locally
latest: Pulling from progrium/consul
c862d82a67a2: Pull complete
0e7f3c08384e: Pull complete
0e221e32327a: Pull complete
09a952464e47: Pull complete
60a1b927414d: Pull complete
4c9f46b5ccce: Pull complete
417d86672aa4: Pull complete
b0d47ad24447: Pull complete
fd5300bd53f0: Pull complete
a3ed95caeb02: Pull complete
d023b445076e: Pull complete
ba8851f89e33: Pull complete
5d1cefca2a28: Pull complete
Digest: sha256:8cc8023462905929df9a79ff67ee435a36848ce7a10f18d6d0faba9306b97274
Status: Downloaded newer image for progrium/consul:latest
==> WARNING: Bootstrap mode enabled! Do not enable unless necessary
==> WARNING: It is highly recommended to set GOMAXPROCS higher than 1
==> Starting raft data migration...
==> Starting Consul agent...
==> Starting Consul agent RPC...
==> Consul agent running!
Node name: 'consul_s'
Datacenter: 'dc1'
Server: true (bootstrap: true)
Client Addr: 0.0.0.0 (HTTP: 8500, HTTPS: -1, DNS: 53, RPC: 8400)
Cluster Addr: 172.17.0.3 (LAN: 8301, WAN: 8302)
Gossip encrypt: false, RPC-TLS: false, TLS-Incoming: false
Atlas: 
==> Log data will now stream in as it occurs:
2017/01/29
2017/01/29
2017/01/29
2017/01/29
2017/01/29
2017/01/29
2017/01/29
2017/01/29
2017/01/29
2017/01/29
2017/01/29
2017/01/29
2017/01/29
2017/01/29
2017/01/29

01:02:23
01:02:23
01:02:23
01:02:23
01:02:23
01:02:23
01:02:25
01:02:25
01:02:25
01:02:25
01:02:25
01:02:25
01:02:25
01:02:25
01:02:25

[INFO] serf: EventMemberJoin: consul_s 172.17.0.3
[INFO] serf: EventMemberJoin: consul_s.dc1 172.17.0.3
[INFO] raft: Node at 172.17.0.3:8300 [Follower] entering Follower state
[INFO] consul: adding server consul_s (Addr: 172.17.0.3:8300) (DC: dc1)
[INFO] consul: adding server consul_s.dc1 (Addr: 172.17.0.3:8300) (DC: dc1)
[ERR] agent: failed to sync remote state: No cluster leader
[WARN] raft: Heartbeat timeout reached, starting election
[INFO] raft: Node at 172.17.0.3:8300 [Candidate] entering Candidate state
[INFO] raft: Election won. Tally: 1
[INFO] raft: Node at 172.17.0.3:8300 [Leader] entering Leader state
[INFO] consul: cluster leadership acquired
[INFO] consul: New leader elected: consul_s
[INFO] raft: Disabling EnableSingleNode (bootstrap)
[INFO] consul: member 'consul_s' joined, marking health alive
[INFO] agent: Synced service 'consul'

You can see that
Now a Docker Consul container is running and maps the ports 8500 for the HTTP API and 8600 for the DNS
endpoint.
CONTAINER ID
40d56ae6d179

(1):

IMAGE
progrium/consul

COMMAND
"/bin/start -serve..."

PORTS
(1)

53/tcp, 0.0.0.0:8400->8400/tcp, 8300-8302/tcp, 8301-8302/udp, 0.0.0.0:8500->8500/tcp, 0.0.0.0:8600->53/udp

You can use the HTTP endpoint to show a list of connected nodes:
curl localhost:8500/v1/catalog/nodes

[{"Node":"consul_s","Address":"172.17.0.3"}]

In order to use the DNS endpoint try

dig @0.0.0.0 -p 8600 node1.node.consul

.

; <<>> DiG 9.9.5-3ubuntu0.10-Ubuntu <<>> @0.0.0.0 -p 8600 consul_s
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 22307
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;consul_s.
IN
A

Consul could be used from a GUI, you can give it a try at
you localhost).

You can use Consul with different options like:
Using a service definition with Consul
Example for 1 service:
{
"service": {
"name": "redis",
"tags": ["primary"],
"address": "",
"port": 8000,
"enableTagOverride": false,
"checks": [
{
"script": "/usr/local/bin/check_redis.py",
"interval": "10s"
}
]
}
}

http://0.0.0.0:8500/

(if you are running the container on

For more than 1 service, just use

services

instead of

service

:

{
"services": [
{
"id": "red0",
"name": "redis",
"tags": [
"primary"
],
"address": "",
"port": 6000,
"checks": [
{
"script": "/bin/check_redis -p 6000",
"interval": "5s",
"ttl": "20s"
}
]
},
{
"id": "red1",
"name": "redis",
"tags": [
"delayed",
"secondary"
],
"address": "",
"port": 7000,
"checks": [
{
"script": "/bin/check_redis -p 7000",
"interval": "30s",
"ttl": "60s"
}
]
},
...
]
}

You can use tools like traefik or fabio as a Consul backend
If you want to use fabio, you should:
Install it, you can also use Docker: docker pull magiconair/fabio
Register your service in consul
Register a health check in consul
Register one urlprefix- tag per host/path prefix it serves, e.g.:

urlprefix-/css

,

urlprefix-i.com/static

mysite.com/

An example:
{
"service": {
"name": "foobar",
"tags": ["urlprefix-/foo, urlprefix-/bar"],
"address": "",
"port": 8000,
"enableTagOverride": false,
"checks": [
{
"id": "api",
"name": "HTTP API on port 5000",
"http": "http://localhost:5000/health",
"interval": "2s",
"timeout": "1s"
}
]
}
}

Start fabio without a configuration file (a consul agent should run on
Watch fabio logs

localhost:8500

).

,

urlprefix-

Send all your HTTP traffic to fabio on port 9999
This is a good video that explains how fabio works: https://www.youtube.com/watch?v=gvxxu0PLevs
Finally, you can write your own process that registers the service through the HTTP API

ZooKeeper
ZooKeeper (or ZK) is a centralized service for configuration management with distributed synchronization
capabilities. ZK organizes its data in a hierarchy of znodes .
It exposes a simple set of primitives that distributed applications can build upon to implement higher level services
for synchronization, configuration maintenance, and groups and naming. It is designed to be easy to program to, and
uses a data model styled after the familiar directory tree structure of file systems. It runs in Java and has bindings for
both Java and C.

From the official documentation, the ZooKeeper implementation is described a putting a premium on high
performance, highly available, strictly ordered access. The performance aspects of ZooKeeper means it can be used
in large, distributed systems. The reliability aspects keep it from being a single point of failure. The strict ordering
means that sophisticated synchronization primitives can be implemented at the client.
It allows distributed processes to coordinate with each other through a shared hierarchal namespace which is
organized similarly to a standard file system. The name space consists of data registers - called znodes, in
ZooKeeper parlance - and these are similar to files and directories. Unlike a typical file system, which is designed
for storage, ZooKeeper data is kept in-memory, which means ZooKeeper can achieve high throughput and low
latency numbers.
Like the distributed processes it coordinates, ZooKeeper itself is intended to be replicated over a sets of hosts called
an ensemble.
The servers that make up the ZooKeeper service must all know about each other. They maintain an in-memory
image of state, along with a transaction logs and snapshots in a persistent store. As long as a majority of the servers
are available, the ZooKeeper service will be available.
Clients connect to a single ZooKeeper server. The client maintains a TCP connection through which it sends
requests, gets responses, gets watch events, and sends heart beats. If the TCP connection to the server breaks, the
client will connect to a different server.
ZooKeeper stamps each update with a number that reflects the order of all ZooKeeper transactions. Subsequent
operations can use the order to implement higher-level abstractions, such as synchronization primitives. It is
especially fast in "read-dominant" workloads. ZooKeeper applications run on thousands of machines, and it
performs best where reads are more common than writes, at ratios of around 10:1.
Its API, support mainly these operations:
: creates a node at a location in the tree
: deletes a node
exists : tests if a node exists at a location
get data : reads the data from a node
set data : writes data to a node
create
delete

: retrieves a list of children of a node
: waits for data to be propagated

get children
sync

Load Balancers
Nginx
Nginx can integrate with some service discovery tools like etcd/confd. Nginx is a popular web server, reverse proxy
and load balancer and the advantage of using Nginx is the community behind it and its very good performance.
A simple configuration, would be creating the right Nginx upstream that can redirect traffic to the Docker containers
is a cluster, for example a Swarm cluster. Example: We want to run an API deployed using Docker Swarm, the
service is mapped to port 8080:
docker service create --name api --replicas 20 --publish 8080:80 my/api:2.3

We now that the API service is mapped to port 8080 in the leader node. We can create a simple Nginx configuration
file:
server {
listen 80;
location / {
proxy_pass http://api;
}
}
upstream api {
server :8080;
}

This file will be used to run an Nginx load balancer:
docker service create --name my_load_balancer --mount type=bind,source=/data/,target=/etc/nginx/conf.d --publish 80:80 nginx

Nginx could be integrated with Consul, Registrator and Consul-template.
Consul will be used as the service discovery tool
Registrator will be used to automatically register the new started services to Consul
Consul-template will be used to automatically recreate the HAProxy configuration from a given template
Adding and removing a node from Nginx configuration is not a good solution. Here is a simple configuration of
Nginx that works with Consul:
upstream frontend { {{range service "app.frontend"}}
server {{.Address}};{{end}}
}

HAProxy
HAProxy is a very common, high-performance load balancing software that could be used as a load balancer set up
in front of a Docker cluster.
You can for example, integrate it with Consul, Registrator and Consul-template.

Consul will be used as the service discovery tool https://github.com/hashicorp/consul
Registrator will be used to automatically register the new started services to Consul
https://github.com/gliderlabs/registrator
Consul-template will be used to automatically recreate the HAProxy configuration from a given template
https://github.com/hashicorp/consul-template
Like Nginx proxy balancer, adding and removing nodes from HAProxy could be done using Consul Template:
backend frontend
balance roundrobin{{range "app.frontend"}}
service {{.ID}} {{.Address}}:{{.Port}}{{end}}

You may also consider using HAProxy with Docker Swarm mode. You can use the dockerfile/haproxy to run
HAProxy:
docker run -d -p 80:80 -v :/haproxy-override dockerfile/haproxy

where



is an absolute path of a directory that could contain:

: custom config file (replace
: custom error responses

haproxy.cfg
errors/

/dev/log

with

127.0.0.1

, and comment out daemon)

This is an example of HAProxy configuration:
global
debug
defaults
log global
mode
http
timeout connect 5000
timeout client 5000
timeout server 5000
listen http_proxy :8443
mode tcp
balance roundrobin
server server1 docker:8000 check
server server2 docker:8001 check

Traefik
Traefik is a HTTP reverse proxy and load balancer made to deploy microservices. It supports several backends like
Docker, Swarm, Mesos/Marathon, Consul, Etcd, Zookeeper, BoltDB, Rest API, file...etc Its configuration could be
automatically and dynamically managed.

You can run a Docker container to deploy Traefik:
docker run -d -p 8080:8080 -p 80:80 -v $PWD/traefik.toml:/etc/traefik/traefik.toml traefik

Or Docker Compose:
traefik:
image: traefik
command: --web --docker --docker.domain=docker.localhost --logLevel=DEBUG

ports:
- "80:80"
- "8080:8080"
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- /dev/null:/traefik.toml

This is the official example that you can test and follow to understand how Traefik in running:
Create docker-compose.yml file:
traefik:
image: traefik
command: --web --docker --docker.domain=docker.localhost --logLevel=DEBUG
ports:
- "80:80"
- "8080:8080"
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- /dev/null:/traefik.toml
whoami1:
image: emilevauge/whoami
labels:
- "traefik.backend=whoami"
- "traefik.frontend.rule=Host:whoami.docker.localhost"
whoami2:
image: emilevauge/whoami
labels:
- "traefik.backend=whoami"
- "traefik.frontend.rule=Host:whoami.docker.localhost"

Run it:
docker-compose up -d

Now you can test the load balancing using curl:
curl -H Host:whoami.docker.localhost http://127.0.0.1
curl -H Host:whoami.docker.localhost http://127.0.0.1

Kube-Proxy
Kube-Proxy is one of the components of Kubernetes. On each Kubernetes node, Kube-Proxy allows us to do simple
TCP,UDP stream forwarding or round robin TCP,UDP forwarding across a set of backends. Service cluster IPs and
ports are currently found through Docker-links-based service that specifies ports opened by the service proxy.

Vulcand
Vulcand is a programmatic extendable proxy for microservices and API management. It is inspired by Hystrix and
powers Mailgun microservices infrastructure.
It uses Etcd as a configuration backend. Has an API and a CLI and supports canary deploys, realtime metrics and
resiliency.

Moxy
Moxy is a reverse load balancer that could be automatically configured to discover services deployed on Apache
Mesos and Marathon.

servicerouter.py
Marathon's servicerouter.py is a replacement for the haproxy-marathon-bridge. It reads Marathon task information
and generates HAProxy configuration. It supports advanced functions like sticky sessions, HTTP to HTTPS
redirection, SSL offloading, VHost support and templating. It is implemented in Python.
You can run the official Docker image to deploy it:
docker run -d \
--name="servicerouter" \
--net="host" \
--ulimit nofile=8204 \
--volume="/dev/log:/dev/log" \
--volume="/tmp/ca-bundle.pem:/etc/ssl/mesosphere.com.pem:ro" \
uzyexe/marathon-servicerouter

Chapter IX - Composing Services Using Compose
o
o

^__^
(oo)\_______
(__)\
)\/\
||----w |
||
||

What Is Docker Compose
We have seen how to run containers but individually, say we want to run a LAMP/LEMP stack, at this case, we
should a php container then start the webserver container and make the link between them. Using Docker Compose,
it is possible to run a multi-container application using a declarative YAML file (Compose file). Using a single
command, we can start multiple containers that run all of our services (a LAMP or LEMP stack in this case).

Installing Docker Compose
Docker Compose comes in a separate binary file, even if you already installed Docker, you should install Compose.

Docker Compose For Mac And Windows
If you're a Mac or Windows user, the best way to install Compose and keep it up-to-date is Docker for Mac and
Windows. Docker for Mac and Windows will automatically install the latest version of Docker Engine for you. You
can use on of these links:
Docker Community Edition for Mac: https://store.docker.com/editions/community/docker-ce-desktop-mac
Docker Community Edition for Windows: https://store.docker.com/editions/community/docker-ce-desktopwindows

Docker For Linux
In order to install Docker for Linux, download Compose binary and move it the binaries path:
curl
-L
https://github.com/docker/compose/releases/download/1.13.0/docker-compose-`uname
m` > /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose

-s`-`uname

-

You can get a different version by changing 1.13.0 by the right version. The Compose file format has a version that
could be compatible with a Docker Engine version. For example, Compose version 3.0 – 3.2 is 1.13.0+ compatible.

Running Wordpress Using Docker Compose
One of the advantage of Docker Compose is the fact that a Compose file could be shared and distributed, creating an
application that uses several services and components could be done using a single Compose command.
This is what most users running Wordpress with Compose are using:
version: '3'
services:
db:
image: mysql:5.7
volumes:
- db_data:/var/lib/mysql
restart: always
environment:
MYSQL_ROOT_PASSWORD: mypassword
MYSQL_DATABASE: wordpress
MYSQL_USER: user
MYSQL_PASSWORD: mypassword
wordpress:
depends_on:
- db
image: wordpress:latest
ports:
- "8000:80"
restart: always
environment:
WORDPRESS_DB_HOST: db:3306
WORDPRESS_DB_USER: myuser
WORDPRESS_DB_PASSWORD: mypassword
volumes:
db_data:

The above content should goes into a file called

docker-compose.yml

. This is the file tree we are using :

Running_Wordpress_Using_Docker_Compose/
└── docker-compose.yml

Now type cd Running_Wordpress_Using_Docker_Compose and run docker-compose up . After downloading and running the
different services, you can go to http://127.0.0.1:8000 in order to see a running Wordpress.

When running the
file.

docker-compose up

command, you should absolutely be inside the folder containing the

docker-compose

What we have actually run in the

docker-compose.yml

docker run --name db \
-e MYSQL_ROOT_PASSWORD=mypassword \
-e MYSQL_DATABASE=wordpress \
-e MYSQL_USER=user \
-e MYSQL_PASSWORD=mypassword \
-d mysql:5.7
docker run --name wordpress \
--link db:mysql \
-p 8080:80 \
-e WORDPRESS_DB_USER=myuser \
-e WORDPRESS_DB_PASSWORD=mypassword \
-d wordpress:latest

file is the equivalent of running two commands:

Running LEMP Using Docker Compose
We are going to use the Compose version 3. After downloading and installing Docker Compose, create a new folder
and a new file called docker-compose.yml :
mkdir Running_LEMP_Using_Docker_Compose
cd LEMP_Using_Docker_Compose
vi docker-compose.yml

Inside the Compose file, start by mentioning the version:
version: '3'

Now add the first service:
web:

We will use Nginx with the latest image, so our service will look like this:
web:
image: nginx:latest

And to declare port mapping, the file becomes:
version: '3'
services:
web:
image: nginx:latest
ports:
- "8000:80"

We are going to declare 3 volumes:
code: where we are going to put the php files
configs: where the Nginx configuration files go
scripts: where the script to start at the service startup goes
version: '3'
services:
web:
image: nginx:latest
ports:
- "8000:80"
volumes:
- ./code:/code
- ./configs/site.conf:/etc/nginx/conf.d/default.conf
- ./scripts:/scripts

Let's create a PHP file where we will use the
mkdir code

phpinfo()

funtion:

cd code
echo " index.php

We named Nginx configuration file to

site.conf

and this is its content:

server {
index index.php;
server_name php-docker.local;
error_log /var/log/nginx/error.log;
access_log /var/log/nginx/access.log;
root /code;
location ~ \.php$ {
try_files $uri =404;
fastcgi_split_path_info ^(.+\.php)(/.+)$;
fastcgi_pass php:9000;
fastcgi_index index.php;
include fastcgi_params;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
fastcgi_param PATH_INFO $fastcgi_path_info;
}
}

You should creae the folder

configs

Now create another floder called

and the file/content above should goes inside this folder

scripts

and add a new script file, we are going to name it

start_services.sh

:

#!/usr/bin/env bash
chown -R www-data:www-data /code
nginx
for (( ; ; ))
do
sleep 1d
done

This script will be the entrypoint to the Nginx container, you can customize it depending on your needs, but don't
forget to execute chmod +x scripts/start_services.sh .
This is the structure of our folder:
.
├──
│
├──
│
├──
└──

code
└── index.php
configs
└── site.conf
docker-compose.yml
scripts
└── start_services.sh

This is our new

docker-compose

file:

version: '3'
services:
web:
image: nginx:latest
ports:
- "8000:80"
volumes:
- ./code:/code
- ./configs/site.conf:/etc/nginx/conf.d/default.conf
- ./scripts:/scripts
links:
- php
entrypoint: ./scripts/start_services.sh
restart: always

Now that web service is linked to the php service, let's add the remainder of the Compose file:

version: '3'
services:
web:
image: nginx:latest
ports:
- "8000:80"
volumes:
- ./code:/code
- ./configs/site.conf:/etc/nginx/conf.d/default.conf
- ./scripts:/scripts
links:
- php
entrypoint: ./scripts/start_services.sh
restart: always
php:
image: php:7-fpm
volumes:
- ./code:/code
restart: always
expose:
- 9000

Now we simply need to run a single command to start the webserver with the PHP backend:
docker-compose up

You should see something like this:
Creating network "runninglempusingdockercompose_default" with the default driver
Creating runninglempusingdockercompose_php_1 ...
Creating runninglempusingdockercompose_php_1 ... done
Creating runninglempusingdockercompose_web_1 ...
Creating runninglempusingdockercompose_web_1 ... done
Attaching to runninglempusingdockercompose_php_1, runninglempusingdockercompose_web_1

Now you can open your browser and visit

http://127.0.0.1:8000

.

Something helpful that we can use is sending the Nginx access logs to a remote log server/service. I am going to use
AWS cloudwatch service, but you can configure your own service/server like syslog or fluentd ..etc
version: '3'
services:
web:
image: nginx:latest
ports:
- "8000:80"
volumes:
- ./code:/code
- ./configs/site.conf:/etc/nginx/conf.d/default.conf
- ./scripts:/scripts
links:
- php
entrypoint: ./scripts/start_services.sh
restart: always
logging:
driver: "awslogs"
options:
awslogs-group: "dev"
awslogs-stream: "web_logs"
php:
image: php:7-fpm
volumes:
- ./code:/code
restart: always
expose:
- 9000
logging:
driver: "awslogs"
options:
awslogs-group: "dev"
awslogs-stream: "php_logs"

If you want to run this LEMP Docker stack as a daemon, you should use:

If you want to pause the stack use

docker-compose pause

If you want to unpause the stack use

If you want to stop the stack use

.

docker-compose unpause

docker-compose down

.

.

docker-compose up -d

.

Scaling Docker Compose
Using Docker Compose it is possible to scale a running service. Say we need 5 PHP containers that should run
behind our webserver. The command docker-compose scale php=5 will start 4 other containers running the same service
(PHP) and since the Nginx service in linked to the PHP one, all of the containers of the latter service can be seen by
Nginx container.
You may say, why we haven't scaled the Nginx bottle neck. As you can see in the Compose file, there is a mapping
between the port 8000 (external/host) and the port 80 (internal/container), the host has a single port 8000 and
creating a new container running the Nginx service will try to use the same external port and cause a conflict, so it is
not possible to scale a service using port mapping with Docker Compose.

Docker Compose Use Cases
I used Docker Compose in production but it was not a critical application. Docker Compose is mainly created for
testing and development purposes. When developing an application, having an isolated environment is crucial.
Docker Compose create this environment for a developer so adding, removing, modifying the version of a software
or a middleware is easy wand will not create any dependency or multi-version problems.
Docker Compose is a good way to share stacks between users and teams, e.g. sharing the production stack with
developers using different development configurations could be done using Docker Compose.
It is also useful to run tests by providing the environment to run them, after running them, the container could be
destroyed.

Chapter X - Docker Logging
o
o

^__^
(oo)\_______
(__)\
)\/\
||----w |
||
||

Docker Native Logging
When you start a container running a web application, a web server or any other application, sooner or later, you
will need to view your application logs. Let's see an example to view the access logs of a webserver.
Run an Nginx container:
docker run -it -p 8000:80 -d --name webserver nginx

Then visit localhost at port

8000

or execute this command:

curl http://0.0.0.0:8000

Now when you type

docker logs webserver

you can see the last lines of the access log:

172.17.0.1 - - [27/May/2017:21:33:59 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.47.0" "-"

The last command will view the log file and exits. If you want to execute the equivalent of
Docker container, add the -f flag:

tail -f

command on a

docker logs -f webserver

or
docker logs --follow webserver

In order to tail the last 10 lines and exit you can execute docker logs --tail 10 webserver . You can also use the --since
flag that takes a string and shows logs since timestamp (e.g. 2013-01-02T13:23:37) or relative (e.g. 42m for 42
minutes):
docker logs --since 42s webserver

If you want to print the timestamp use the

-t

flag:

docker logs --since 2h -t webserver

docker logs -f -t webserver

The

docker logs

command uses the output of STDOUT and STDERR

Adding New Logs
The docker logs command uses the output of STDOUT and STDERR. When we run an Nginx container, the access
logs and the remainder of Nginx logs like error logs are redirected respectively to STDOUT and STDERR. Let's
examine the official Nginx Dockerfile in order to see how this was done:
#
# Nginx Dockerfile
#
# https://github.com/dockerfile/nginx
#
# Pull base image.
FROM dockerfile/ubuntu
# Install Nginx.
RUN \
add-apt-repository -y ppa:nginx/stable && \
apt-get update && \
apt-get install -y nginx && \
rm -rf /var/lib/apt/lists/* && \
echo "\ndaemon off;" >> /etc/nginx/nginx.conf && \
chown -R www-data:www-data /var/lib/nginx
# Define mountable directories.
VOLUME ["/etc/nginx/sites-enabled", "/etc/nginx/certs", "/etc/nginx/conf.d", "/var/log/nginx", "/var/www/html"]
# Define working directory.
WORKDIR /etc/nginx
# Define default command.
CMD ["nginx"]
# Expose ports.
EXPOSE 80
EXPOSE 443

We are certainly looking fot this line echo "\ndaemon off;" >> /etc/nginx/nginx.conf && \ . When running Nginx with
daemon off; configuration, Nginx will run in foreground and everything will be redirected to the screen (STDOUT).
For Apache Dockerfile, if we examine it we will notice that the container execute a script file at its startups. The
content of this script is:
#!/bin/bash
set -e
# Apache gets grumpy about PID files pre-existing
rm -f /usr/local/apache2/logs/httpd.pid
exec httpd -DFOREGROUND

The

-DFOREGROUND

option will allow Docker to get Apache logs.

What if we have custom log files ? Say we have our application write into a file called
folder.
./logs/app.log

In this case, you need to redirect the

app.log

FROM ..
..
RUN ln -sf /dev/stdout logs/app.log
RUN ln -sf /dev/stderr logs/app-errors.log
..etc

content to the STDOUT or STDERR:

app.log

inside the

logs

Docker Logging Drivers
Docker can interface with other logging services like AWS cloudwatch, Fluentd, syslog ..etc and send all of the logs
to the remote servic. When you use a logging driver, the native docker logs  become deactivated. These
are the supported logging drivers:
syslog: Writes logging messages to the syslog facility. The syslog daemon must be running on the host
machine.
journald: Writes log messages to journald. The journald daemon must be running on the host machine.
gelf: Writes log messages to a Graylog Extended Log Format (GELF) endpoint such as Graylog or Logstash.
fluentd: Writes log messages to fluentd (forward input). The fluentd daemon must be running on the host
machine.
awslogs: Writes log messages to Amazon CloudWatch Logs.
splunk: Writes log messages to splunk using the HTTP Event Collector.
etwlogs: Writes log messages as Event Tracing for Windows (ETW) events. Only available on Windows
platforms.
gcplogs: Writes log messages to Google Cloud Platform (GCP) Logging.

Using Fluentd Log Driver
First thing that we need to do is installing Fluentd in the host that will collect the logs. If your package manager is
RPM, you can use curl -L https://toolbelt.treasuredata.com/sh/install-redhat-td-agent2.sh | sh
If you are using Ubuntu or Debian:
For Xenial,
curl -L https://toolbelt.treasuredata.com/sh/install-ubuntu-xenial-td-agent2.sh | sh

For Trusty,
curl -L https://toolbelt.treasuredata.com/sh/install-ubuntu-trusty-td-agent2.sh | sh

For Precise,
curl -L https://toolbelt.treasuredata.com/sh/install-ubuntu-precise-td-agent2.sh | sh

For Lucid,
curl -L https://toolbelt.treasuredata.com/sh/install-ubuntu-lucid-td-agent2.sh | sh

For Debian Jessie,
curl -L https://toolbelt.treasuredata.com/sh/install-debian-jessie-td-agent2.sh | sh

For Debian Wheezy,
curl -L https://toolbelt.treasuredata.com/sh/install-debian-wheezy-td-agent2.sh | sh

For Debian Squeeze,
curl -L https://toolbelt.treasuredata.com/sh/install-debian-squeeze-td-agent2.sh | sh

In order to use Fluentd, create the configuration file, we are going to name it

docker.conf


type forward
port 24224
bind 0.0.0.0


type stdout


You should now start fluentd with the configuration file after adapting it to your needs:

.

fluentd -c docker.conf
>
>
>
>
>
>
>
>
>
>
>
>
>
>
<
>
>

2015-09-01 15:07:12
2015-09-01 15:07:12
2015-09-01 15:07:12
2015-09-01 15:07:12
2015-09-01 15:07:12
2015-09-01 15:07:12
2015-09-01 15:07:12

@type forward
port 24224
bind 0.0.0.0


@type stdout


2015-09-01 15:07:12

-0600
-0600
-0600
-0600
-0600
-0600
-0600

[info]:
[info]:
[info]:
[info]:
[info]:
[info]:
[info]:

reading config file path="docker.conf"
starting fluentd-0.12.15
gem 'fluent-plugin-mongo' version '0.7.10'
gem 'fluentd' version '0.12.15'
adding match pattern="*.*" type="stdout"
adding source type="forward"
using configuration file: 

-0600 [info]: listening fluent socket on 0.0.0.0:24224

It is also possible to run Fluentd in a Docker container:
docker run -it -p 24224:24224 -v docker.conf:/fluentd/etc/docker.conf -e FLUENTD_CONF=docker.conf fluent/fluentd:latest

Now run
docker run --log-driver=fluentd ubuntu echo "Hello Fluentd!"

Fluentd could be on a different remote host and in this case you should add

--log-opt

followed by the host address:

docker run --log-driver=fluentd --log-opt fluentd-address=192.168.1.10:24225 ubuntu echo "..."

If you are using multiple log types/files, you can tag each container/service with a different tag using
docker run --log-driver=fluentd --log-opt fluentd-tag=docker.{{.ID}} ubuntu echo "..."

fluentd-tag

.

Using AWS CloudWatch Log Driver
Amazon CloudWatch is a monitoring service for AWS cloud resources and the applications you run on AWS. You
can use Amazon CloudWatch to collect and track metrics, collect and monitor log files, set alarms, and automatically
react to changes in your AWS resources. AWS CloudWatch can also be used in collecting and centralizing Docker
logs.
In order to use CloudWatch, you need to allow the used user to execute these actions:
logs:CreateLogGroup
logs:CreateLogStream
logs:PutLogEvents
logs:DescribeLogStreams
e.g:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents",
"logs:DescribeLogStreams"
],
"Resource": [
"arn:aws:logs:*:*:*"
]
}
]
}

You need now to go to your AWS console, go to CloudWatch service then create a log group, call it
on the created group and create a new stream, call it my-stream .

my-group

. Click

The awslogs logging driver sends your Docker logs to a specific region so you could use the awslogs-region log
option or the AWS_REGION environment variable to set the region. You should use the same region where you created
the log group/stream.
docker run --log-driver=awslogs --log-opt awslogs-region=us-east-1 ubuntu echo "..."

In order to add the logs group/stream, execute:
docker run --log-driver=awslogs
stream my-stream

--log-opt

awslogs-region=us-east-1

--log-opt

awslogs-group=my-group

--log-opt

You can configure the default logging driver by passing the --log-driver option to the Docker daemon:
dockerd --log-driver=awslogs

awslogs-

Chapter XI - Docker Debugging And Troubleshooting
o
o

^__^
(oo)\_______
(__)\
)\/\
||----w |
||
||

Docker Daemon Logs
When there is a problem, one of the first things for many of you is checking the Docker Daemon logs. Docker logs
are accessible in different ways and this depends on your system:
OSX - ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/log/d​​ocker.log
Debian - /var/log/daemon.log
CentOS - Run /var/log/daemon.log | grep docker
CoreOS - Run journalctl -u docker.service
Ubuntu upstart - /var/log/upstart/docker.log
Ubuntu systemd - Run journalctl -u docker.service command
Fedora - Run journalctl -u docker.service
Red Hat Enterprise Linux Server - Run /var/log/messages | grep docker
OpenSuSE - Run journalctl -u docker.service
Boot2Docker - /var/log/docker.log
Windows - AppData\Local
Another way of troubleshooting the daemon is running it in foreground:
dockerd

If you are already running Docker you should stop it and start the daemon.
sudo dockerd

> INFO[0000] libcontainerd: new containerd process, pid: 9898
> WARN[0000] containerd: low RLIMIT_NOFILE changing to max current=1024 max=65536
> WARN[0001] failed to rename /var/lib/docker/tmp for background deletion: %!s(). Deleting synchronously
> INFO[0001] [graphdriver] using prior storage driver: aufs
> INFO[0001] Graph migration to content-addressability took 0.00 seconds
> WARN[0001] Your kernel does not support swap memory limit
> WARN[0001] Your kernel does not support cgroup rt period
> WARN[0001] Your kernel does not support cgroup rt runtime
> INFO[0001] Loading containers: start.
>
INFO[0002]
Default
bridge
(docker0)
is
assigned
with
an
IP
address
172.17.0.0/16.
Daemon
option
-bip can be used to set a preferred IP address
>
INFO[0002]
No
nonlocalhost DNS nameservers are left in resolv.conf. Using default external servers: [nameserver 8.8.8.8 nameserver 8.8.4.4]
> INFO[0002] IPv6 enabled; Adding default IPv6 external servers: [nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844]
>
INFO[0002]
No
nonlocalhost DNS nameservers are left in resolv.conf. Using default external servers: [nameserver 8.8.8.8 nameserver 8.8.4.4]
> INFO[0002] IPv6 enabled; Adding default IPv6 external servers: [nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844]
> WARN[0002] Failed to allocate and map port 8000-8000: Bind for 0.0.0.0:8000 failed: port is already allocated
> WARN[0002] failed to cleanup ipc mounts:
> failed to umount /var/lib/docker/containers/d929a0878ea9282cd3eeb1ed65c1a6448e0a9da67b1dce2dba305c746ecc2371/shm: invalid argument
> ERRO[0002] Failed to start container d929a0878ea9282cd3eeb1ed65c1a6448e0a9da67b1dce2dba305c746ecc2371: driver failed programming external conne

Docker Debugging
In order to do that, set the debug key to true in the daemon.json file. Generally, you will find this file under
/etc/docker . You may need to create this file, if it does not yet exist.
{
"debug": true
}

Possible values are

debug

,

info

,

warn

,

error

,

fatal

.

Now send a HUP signal to the daemon to cause it to reload its configuration:
execute service docker stop && dockerd :
You will be able to see all of the actions that Docker is doing:
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]

Registering routers
Registering GET, /containers/{name:.*}/checkpoints
Registering POST, /containers/{name:.*}/checkpoints
Registering DELETE, /containers/{name}/checkpoints/{checkpoint}
Registering HEAD, /containers/{name:.*}/archive
Registering GET, /containers/json
Registering GET, /containers/{name:.*}/export
Registering GET, /containers/{name:.*}/changes
Registering GET, /containers/{name:.*}/json
Registering GET, /containers/{name:.*}/top
Registering GET, /containers/{name:.*}/logs
Registering GET, /containers/{name:.*}/stats
Registering GET, /containers/{name:.*}/attach/ws
Registering GET, /exec/{id:.*}/json
Registering GET, /containers/{name:.*}/archive
Registering POST, /containers/create
Registering POST, /containers/{name:.*}/kill
Registering POST, /containers/{name:.*}/pause
Registering POST, /containers/{name:.*}/unpause
Registering POST, /containers/{name:.*}/restart
Registering POST, /containers/{name:.*}/start
Registering POST, /containers/{name:.*}/stop
Registering POST, /containers/{name:.*}/wait
Registering POST, /containers/{name:.*}/resize
Registering POST, /containers/{name:.*}/attach
Registering POST, /containers/{name:.*}/copy
Registering POST, /containers/{name:.*}/exec
Registering POST, /exec/{name:.*}/start
Registering POST, /exec/{name:.*}/resize
Registering POST, /containers/{name:.*}/rename
Registering POST, /containers/{name:.*}/update
Registering POST, /containers/prune
Registering PUT, /containers/{name:.*}/archive
Registering DELETE, /containers/{name:.*}
Registering GET, /images/json
Registering GET, /images/search
Registering GET, /images/get
Registering GET, /images/{name:.*}/get
Registering GET, /images/{name:.*}/history
Registering GET, /images/{name:.*}/json
Name To resolve: php.
Registering POST, /commit
Query php.[1] from 127.0.0.1:53704, forwarding to udp:127.0.1.1
Registering POST, /images/load
Registering POST, /images/create
Registering POST, /images/{name:.*}/push
Registering POST, /images/{name:.*}/tag
Registering POST, /images/prune
Registering DELETE, /images/{name:.*}
Registering OPTIONS, /{anyroute:.*}
Registering GET, /_ping
Registering GET, /events
Registering GET, /info
Registering GET, /version
Registering GET, /system/df
Registering POST, /auth
Registering GET, /volumes
Registering GET, /volumes/{name:.*}

sudo kill -SIGHUP $(pidof dockerd)

or

DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]
DEBU[0012]

Registering
Registering
Registering
Registering
Registering
Registering
Registering
Registering
Registering
Registering
Registering
Registering
Registering
Registering
Registering
Registering
Registering
Registering
Registering
Registering
Registering
Registering
Registering
Registering
Registering
Registering
Registering
Registering
Registering
Registering
Registering
Registering
Registering
Registering
Registering
Registering
Registering
Registering
Registering
Registering
Registering
Registering
Registering
Registering
Registering
Registering
Registering
Registering

POST, /volumes/create
POST, /volumes/prune
DELETE, /volumes/{name:.*}
POST, /build
POST, /swarm/init
POST, /swarm/join
POST, /swarm/leave
GET, /swarm
GET, /swarm/unlockkey
POST, /swarm/update
POST, /swarm/unlock
GET, /services
GET, /services/{id}
POST, /services/create
POST, /services/{id}/update
DELETE, /services/{id}
GET, /services/{id}/logs
GET, /nodes
GET, /nodes/{id}
DELETE, /nodes/{id}
POST, /nodes/{id}/update
GET, /tasks
GET, /tasks/{id}
GET, /tasks/{id}/logs
GET, /secrets
POST, /secrets/create
DELETE, /secrets/{id}
GET, /secrets/{id}
POST, /secrets/{id}/update
GET, /plugins
GET, /plugins/{name:.*}/json
GET, /plugins/privileges
DELETE, /plugins/{name:.*}
POST, /plugins/{name:.*}/enable
POST, /plugins/{name:.*}/disable
POST, /plugins/pull
POST, /plugins/{name:.*}/push
POST, /plugins/{name:.*}/upgrade
POST, /plugins/{name:.*}/set
POST, /plugins/create
GET, /networks
GET, /networks/
GET, /networks/{id:.+}
POST, /networks/create
POST, /networks/{id:.*}/connect
POST, /networks/{id:.*}/disconnect
POST, /networks/prune
DELETE, /networks/{id:.*}

Checking Docker Status
Checking if Docker is running can be done using service docker status or any other alternative way like ps aux|grep
docker , ps -ef |grep docker ..etc It is also possible to use any other Docker command like docker info command in
order to see of Docker responds or not. In the negative case, your terminal will show something like
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

You can use other ways depending on your system tools like

systemctl is-active docker

.

Debugging Containers
Whether you are using standalone Docker containers or managed services, it is possible to inspect the details of a
service or a container.
docker service inspect 
docker inspect 

Let's inspect this container docker run -it -p 8000:80 -d --name webserver nginx using
These are the list of information the Docker inspect command will give us:
[
{
"Id": "3..9",
"Created": "2017-05-27T23:58:20.848318438Z",
"Path": "nginx",
"Args": [
"-g",
"daemon off;"
],
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 13730,
"ExitCode": 0,
"Error": "",
"StartedAt": "2017-05-27T23:58:21.257901373Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Image": "sha256:3..7",
"ResolvConfPath": "/var/lib/docker/containers/3..9/resolv.conf",
"HostnamePath": "/var/lib/docker/containers/3..9/hostname",
"HostsPath": "/var/lib/docker/containers/3..9/hosts",
"LogPath": "/var/lib/docker/containers/3..9/3..9-json.log",
"Name": "/webserver",
"RestartCount": 0,
"Driver": "aufs",
"MountLabel": "",
"ProcessLabel": "",
"AppArmorProfile": "docker-default",
"ExecIDs": null,
"HostConfig": {
"Binds": null,
"ContainerIDFile": "",
"LogConfig": {
"Type": "json-file",
"Config": {}
},
"NetworkMode": "default",
"PortBindings": {
"80/tcp": [
{
"HostIp": "",
"HostPort": "8000"
}
]
},
"RestartPolicy": {
"Name": "no",
"MaximumRetryCount": 0
},
"AutoRemove": false,
"VolumeDriver": "",
"VolumesFrom": null,
"CapAdd": null,
"CapDrop": null,
"Dns": [],
"DnsOptions": [],
"DnsSearch": [],
"ExtraHosts": null,
"GroupAdd": null,
"IpcMode": "",
"Cgroup": "",
"Links": null,
"OomScoreAdj": 0,

docker inspect webserver

command.

"PidMode": "",
"Privileged": false,
"PublishAllPorts": false,
"ReadonlyRootfs": false,
"SecurityOpt": null,
"UTSMode": "",
"UsernsMode": "",
"ShmSize": 67108864,
"Runtime": "runc",
"ConsoleSize": [
0,
0
],
"Isolation": "",
"CpuShares": 0,
"Memory": 0,
"NanoCpus": 0,
"CgroupParent": "",
"BlkioWeight": 0,
"BlkioWeightDevice": null,
"BlkioDeviceReadBps": null,
"BlkioDeviceWriteBps": null,
"BlkioDeviceReadIOps": null,
"BlkioDeviceWriteIOps": null,
"CpuPeriod": 0,
"CpuQuota": 0,
"CpuRealtimePeriod": 0,
"CpuRealtimeRuntime": 0,
"CpusetCpus": "",
"CpusetMems": "",
"Devices": [],
"DeviceCgroupRules": null,
"DiskQuota": 0,
"KernelMemory": 0,
"MemoryReservation": 0,
"MemorySwap": 0,
"MemorySwappiness": -1,
"OomKillDisable": false,
"PidsLimit": 0,
"Ulimits": null,
"CpuCount": 0,
"CpuPercent": 0,
"IOMaximumIOps": 0,
"IOMaximumBandwidth": 0
},
"GraphDriver": {
"Data": null,
"Name": "aufs"
},
"Mounts": [],
"Config": {
"Hostname": "37068ac691fb",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"ExposedPorts": {
"80/tcp": {}
},
"Tty": true,
"OpenStdin": true,
"StdinOnce": false,
"Env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"NGINX_VERSION=1.13.0-1~stretch",
"NJS_VERSION=1.13.0.0.1.10-1~stretch"
],
"Cmd": [
"nginx",
"-g",
"daemon off;"
],
"ArgsEscaped": true,
"Image": "nginx",
"Volumes": null,
"WorkingDir": "",
"Entrypoint": null,
"OnBuild": null,
"Labels": {},
"StopSignal": "SIGQUIT"
},
"NetworkSettings": {
"Bridge": "",
"SandboxID": "4..0",
"HairpinMode": false,
"LinkLocalIPv6Address": "",
"LinkLocalIPv6PrefixLen": 0,
"Ports": {
"80/tcp": [
{
"HostIp": "0.0.0.0",

"HostPort": "8000"
}
]
},
"SandboxKey": "/var/run/docker/netns/4a0ab4faeb4e",
"SecondaryIPAddresses": null,
"SecondaryIPv6Addresses": null,
"EndpointID": "8..b",
"Gateway": "172.17.0.1",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"IPAddress": "172.17.0.2",
"IPPrefixLen": 16,
"IPv6Gateway": "",
"MacAddress": "02:42:ac:11:00:02",
"Networks": {
"bridge": {
"IPAMConfig": null,
"Links": null,
"Aliases": null,
"NetworkID": "b..b",
"EndpointID": "8..b",
"Gateway": "172.17.0.1",
"IPAddress": "172.17.0.2",
"IPPrefixLen": 16,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": "02:42:ac:11:00:02"
}
}
}
}
]

It is possible to get a single element like for example, the IP address:
docker inspect --format='{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' webserver

Or the port binding list:
docker
inspect
--format='{{range
$p,
> {{(index $conf 0).HostPort}} {{end}}' webserver

$conf

:=

.NetworkSettings.Ports}}

Another way of debugging containers is executing debug commands inside the container like
webserver ps aux or docker exec -it webserver cat /etc/resolv.conf ..etc
Using

docker stats

and

docker events

{{$p}}

-

docker exec -it

commands could give you also some information when debugging.

Troubleshooting Docker Using Sysdig
Sysdig is a Linux system exploration and troubleshooting tool with support for containers.
To install Sysdig automatically in one step, simply run the following command. This is the recommended
installation method.
curl -s https://s3.amazonaws.com/download.draios.com/stable/install-sysdig | sudo bash

Then add your username to the same group as sysdig:
groupadd sysdig
usermod -aG sysdig $USER

Use visudo to edit the sudo-config. Add the line %sysdig ALL= /path/to/sysdig and save. The path is most likely
/usr/local/bin/sysdig , but you can make sure by running which sysdig.
Sysdig is an open source project and it can be used to get information about
Networking
Containers
Application
Disk I/O
Processes and CPU usage
Performance and Errors
Security
Tracing
Debugging containers is also debugging the host, so sysdig can be used to make a general troubleshooting. What
does interest us in this part is the container-related commands.
In order to list the running containers with their resource usage
sudo csysdig -vcontainers

Listing all of the processes with container context can be done using
sudo csysdig -pc

To view the CPU usage of the processes running inside the my_container container, use:
sudo sysdig -pc -c topprocs_cpu container.name=my_container

Bandwidth can be monitored using:

sudo sysdig -pc -c topprocs_net container.name=my_container

Processes using most network bandwidth can be checked using:
sudo sysdig -pc -c topprocs_net container.name=my_container

To view the top network connections:
sudo sysdig -pc -c topconns container.name=my_container

Top used files consuming I/O bytes could be checkeck using:
sudo sysdig -pc -c topfiles_bytes container.name=my_container

And to show all the interactive commands executed inside the my_container container, use:
sudo sysdig -pc -c spy_users container.name=my_container

Chapter XII - Orchestration - Docker Swarm
o
o

^__^
(oo)\_______
(__)\
)\/\
||----w |
||
||

Docker Swarm
Docker Swarm is the solution that Docker inc developed to create an orchestration tool like Google's Kubernetes. It
provides native clustering capabilities to turn a group of Docker engines into a single, virtual Docker Engine.
Distributed applications requires compute resources that are also distributed and that's why Docker Swarm was
introduced. You can use it to manage pooled resources in order to scale out an application as if it was running on a
single, huge computer.
Before Docker engine 1.12, Docker Swarm should be integrated with a kv store and a service discovery tool but after
this version, Docker Swarm integrated these tools and it can be used without having the need to use other tools.

Swarm Features
Docker has a continuous active development and it is changing a lot, Swarm mode introduced many new feature that
solved many problems. Like it is described in its official website, Docker Swarm serves the standard Docker API, so
any tool which already communicates with a Docker daemon can use Docker Swarm to transparently scale to
multiple hosts: Dokku, Docker Compose, Krane, Flynn, Deis, DockerUI, Shipyard, Drone, Jenkins and of course the
Docker client itself. Includes ability to pull from private repositories or Docker public Hub as well.
Swarm mode is a built-in native solution to Docker, you can use Docker Networking, Volumes and plugins through
their respective Docker commands via this mode.
Its scheduler has useful filters like node tags, affinity and strategies like spread, binpack ..etc These filters assign
containers to the underlying nodes to optimize performance and resource utilization.
Swarm is production ready and according to Docker inc it is tested to scale up to 1,000 nodes and fifty thousand
50,000 containers with no performance degradation in spinning up incremental containers onto the node cluster.
A test stress done by Docker to spin up 1,000 nodes, 30,000 containers managed by 1 Swarm manager gave these
results:
Percentile
50th
90th
99th

API Response Time
150ms
200ms
360ms

Scheduling Delay
230ms
250ms
400ms

During this test, Consul was used as a discovery backend, every node hosted 30 containers (1,000 nodes), the
manager was an EC2 m4.xlarge (4 CPUs, 16GB RAM) machine and nodes were EC2 t2.micro (1 CPU, 1 GB RAM)
machines and container images were using ubuntu 14.04.

Here is what was published in the blog post introducing the results:
We wanted to stress test a single Swarm manager, to see how capable it would be, so we used one Swarm
manager to manage all our nodes. We placed fifty containers per node. Commands were run 1,000 times
against Swarm and and we generated percentiles for 1) API Response time and 2) Scheduling delay. We found
that we were able to scale up to 1,000 nodes running 30,000 containers. 99% of the time each container took
less than half a second to launch. There was no noticeable difference in the launch time of the 1st and 30,000th
container. We used docker info to measure API response time, and then used docker run -dit ubuntu bash to
measure scheduling delay.

Another serious collaborative test was done called Swarm2k. This test was using Docker 1.12, a total of 2,384
servers was part of the Swarm cluster and there were 3 managers.

To achieve such a bug number of nodes, Docker ensure a highly available Swarm Manager. You can create multiple
Swarm masters and specify policies on leader election in case the primary master experiences a failure.
Swarm comes with a built-in scheduler, but you can easily plugin the Mesos or Kubernetes backend while still using
the Docker client. To find nodes in your cluster, Docker Swarm can use either a hosted discovery service, static file,
etcd, consul and zookeeper.

Installation
Nothing different from the default Docker installation, since the Swarm mode is a built-in feature, you need just to
install Docker:
curl -fsSL https://get.docker.com/ | sh

You can use docker -v to see the installed version, in all cases if it is greater than 1.12, you should have the Swarm
feature integrated in Docker engine.

The Raft Consensus Algorithm , Swarm Managers & Best
Practices
In a Docker cluster, you should have at least one manager. One of the things that I was missing when I started
experimenting Docker is that the number of managers should not be equal to 2. Then I understood that 1 manager or
3 managers is better. In fact, when running with two managers, you double the chance of a manager failure.

Swarm manager nodes use the Raft Consensus Algorithm to manage the swarm state.
Raft achieves consensus via an elected leader. A server in a raft cluster is either:
a leader
a candidate
or a follower
The leader is responsible for log replication to the followers. Using heartbeat messages, the leader regularly informs
the followers of its existence.
Each follower has a timeout in which it expects the heartbeat from the leader. The timeout is reset on receiving the
heartbeat (it is typically between 150 and 300ms). In the case when no heartbeat is received, the follower changes its
status to candidate and starts a new leader election. The election starts by increasing the term counter and sending a
RequestVote message to all other servers that will vote (only once) for the first candidate that sends them this
RequestVote message.

Three scenarios are possible in the last case:
If the candidate receives a message from a leader with a term number equal to or larger than the current term,
then its election is defeated and the candidate changes into a follower.
If a candidate receives a majority of votes, then it becomes the new leader.
If neither happens, a new leader election starts after a timeout.
Raft tolerates up to (N-1)/2 failures and requires a majority or quorum (also called a majority of managers) of
(N/2)+1 members to agree on values proposed to the cluster.
Raft will not tolerate the case when among 5 Managers, 3 nodes are unavailable, the system will not process any
more requests.
In all cases, you should maintain an odd number of managers in the swarm to support manager node failures, it
ensures higher chance for the quorum availability.
Number Of Nodes (N)
1
2
3
4
5
6

Quorom Majority (N/2)+1
1
2
2
3
3
4

Fault Tolerance (N-1)/2
0
0
1
1
2
2

You don't really need to know how Raft works, but you need to know that having an odd number of managers is a
must. You need also to know, that in order to maximize the availability of your nodes, you should think about
distributing manager nodes across a minimum of 3 availability-zones. You may have less managers and one
availability zone, it will work fine but you minimize the probability of teloreating data centers problems. If you will
deploy high-available fault-tolerant system, this table describes how you should make the repartition of managers
across 3 availability zones:
Number of Node Managers
3
5
7
9
11

Repartition Of Managers Across 3 Availability Zones
1-1-1
2-2-1
3-2-2
3-3-3
4-4-3

Another thing to know is that a node running as a swarm manager is not different than a non-manager node (a
worker) so it is fine to have a swarm cluster with only managers, without any worker. In fact, manager nodes by
default acts like a worker nodes. Swarm scheduler can assign tasks to a manager node. If you have a small swarm
cluster, managers could be assigned to execute tasks with lower risk.
You can also restrict the role of a manager in order to acting only as a manager and not a worker. Draining managers
nodes, make them unavailable as worker nodes:
docker node update --availability drain 

It is may be evident but you should assign a static IP to each of your Swarm managers, a worker in contrast could
have dynamic IP (since it will be discovered) but workers and managers - in all cases - should be able to
communicate together over network.
The following ports must be available:

TCP port 2377 for cluster management communications
TCP and UDP port 7946 for communication among nodes
TCP and UDP port 4789 for overlay network traffic

Creating Swarm Managers & Workers
In order to create a manger, you should have Docker installed in the node, then use the swarm initialization
command:
docker swarm init --advertise-addr [:port]

If you want to customize your security, you can use these options:
--cert-expiry duration
--external-ca external-ca

Validity period for node certificates (ns|us|ms|s|m|h) (default 2160h0m0s)
Specifications of one or more certificate signing endpoints

Other options can be used to changer the heartbeat duration, the number of log entries between Raft snapshots or the
task history retention limit.
--autolock
--dispatcher-heartbeat duration
--max-snapshots uint
--snapshot-interval uint
--task-history-limit int

Enable manager autolocking (requiring an unlock key to start a stopped manager)
Dispatcher heartbeat period (ns|us|ms|s|m|h) (default 5s)
Number of additional Raft snapshots to retain
Number of log entries between Raft snapshots (default 10000)
Task history retention limit (default 5)

When losing the quorum, you can use the following option to bring back the failed node online.
--force-new-cluster

Force create a new cluster from current state

Example:
Say our

eth0

interface has

138.197.35.0

as an IP address.

To create a cluster with a first manager, type:
docker swarm init --force-new-cluster --advertise-addr 138.197.35.0

This will show an instruction to execute a command in order to add a worker:
Swarm initialized: current node (vhvebboq9fp4j62w83dujxzjr) is now a manager.
To add a worker to this swarm, run the following command:
docker swarm join \
--token SWMTKN-1-5b54vz0sie1li0ijr0epkhyjvmbbh2pg746skh8ba5674g1p6x-cmgrvib1disaeq08x8a5ln7zo \
138.197.35.0:2377
To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.

If you want to add a worker to this cluster, create a new server and execute:
docker
swarm
join
cmgrvib1disaeq08x8a5ln7zo 138.197.35.0:2377

You should of course have the port
follwing command:
docker swarm join-token manager

2377

--token

SWMTKN-1-5b54vz0sie1li0ijr0epkhyjvmbbh2pg746skh8ba5674g1p6x-

open. But, if you want to add a new manager, you should execute the

Docker will generate a new command that you can execute in a second manager:
docker
swarm
join
354t5zlz0re5kh1jqcliofecs 138.197.35.0:2377

--token

SWMTKN-1-5b54vz0sie1li0ijr0epkhyjvmbbh2pg746skh8ba5674g1p6x-

Now you can get a list of all the available workers and managers in you cluster by typing:
docker node ls

I have a single node in my cluster and of course it is a manager, what I see is the following result:
ID
vhvebboq9fp4j62w83dujxzjr *

HOSTNAME
swarm-1

STATUS
Ready

AVAILABILITY
Active

MANAGER STATUS
Leader

Deploying Services
Creating A Container
Nothing is really different from what your learned before, you should create an image, build it, create a container,
tag it, commit it, push it ..etc then you can use your image to create the container.
For the sake of simplicity, I created a container based on Alpine Linux that will execute and infinite loop.
This is the Dockerfile:
FROM alpine
ENTRYPOINT tail -f /dev/null

I built it:
docker build -t eon01/infinite .
Sending build context to Docker daemon 11.78 kB
Step 1/2 : FROM alpine
latest: Pulling from library/alpine
0a8490d0dfd3: Pull complete
Digest: sha256:dfbd4a3a8ebca874ebd2474f044a0b33600d4523d03b0df76e5c5986cb02d7e8
Status: Downloaded newer image for alpine:latest
---> 88e169ea8f46
Step 2/2 : ENTRYPOINT tail -f /dev/null
---> Running in 986f0fd1f5f9
---> d1400705c370
Removing intermediate container 986f0fd1f5f9
Successfully built d1400705c370

Run it:
docker run -it --name infinite -d eon01/infinite
fc476e6f8312492b2fd9cb620c8eaf5115c2e18a52d95160849436087dec2b68

And it should keep running because of the infinite

tail -f /dev/null

:

docker ps
CONTAINER ID
fc476e6f8312
c 'tail ..."

IMAGE
3 seconds ago

COMMAND

CREATED
eon01/infinite

Up 2 seconds

STATUS

PORTS
"/bin/sh

NAMES
-

infinite

You can use this image directly from my public Docker Hub:
docker run -it --name infinite -d eon01/infinite

But this is not actually how we use Swarm. The common usage of Swarm is creating services first before thinking in
term of containers.

Creating & Configuring Swarm Services

In order to create a service you can use the

docker service create

command.

docker service create    

This command can be used with multiple options like:
to set custom DNS servers
--env to set a list of environment variables
--env-file to read in a file of environment variables
--health-cmd to set a health check command
--log-driver and --log-opt to set logging driver and driver options
-p or --publish to publish a port as a node port
--replicas to control the number of running tasks of a given service
--with-registry-auth to send registry authentication details to swarm agents
--mode to select if the service should be replicated or global
--network to attach a service to a network
--endpoint-mode to choose between vip or dnsrr
And other options that you may have used in the previous sections of this book like
user , -t or --tty , --dns-option or --dns-search ..etc
--dns
-e

or

-w

or

--workdir

,

-u

or

--

Here is an example:
docker service create --name infinite_service

eon01/infinite

After creating the new service, you can check using docker ps command, that you have two containers, the first one
that you created using docker run command and the other one created by default when creating the Swarm service.
CONTAINER ID
cf265656d24a
982687e9d979

IMAGE
COMMAND
STATUS
eon01/infinite
"/bin/sh -c 'tail ..." Up 10 hours (unhealthy)
eon01/infinite@sha256:e53.. "/bin/sh -c 'tail ..." Up 10 hours

We do not really need the first "unmanaged" container, remove it using

docker rm -f infinite

NAMES
infinite
infinite_service.1.eq..

.

Scaling Docker Containers
After we created infinite_service, you will notice that there is one container running. One of the Swarm features is
the creation of scalable services and easy to scale. One command will allow us to scale a running service.
This is the running container that the service infinite_service created. By default a service will start 1 container:
a2f13154e772

eon01/infinite

"/bin/sh -c 'tail ..."

In order to scale infinite_service we can use the

infinite_service.1.

docker service scale

command:

To scale up to 2 containers:
root@swarm-1:~# docker service scale infinite_service=2
infinite_service scaled to 2

To run 100 containers :
root@swarm-1:~# docker service scale infinite_service=100
infinite_service scaled to 100

Scaling to 100 container (or may be less or more - this depends really on your application and architecture) is
not always the Solution to your performance problems. I always make tests to choose what is the best scale to have.
Scaling up to 100 containers may make some performance regressions happens since you will be spending more
time on networking and/or service name resolution ..etc. So be sure to know the scale rate in production using load
and networking tests.
In order to see your containers you may use
containers per service:
docker service ps

ID
bfvu50gte1gd
mope64fq4q1x
k8fv6qk2ez8c
e7luxqzwtb66
pryz06z8prbz
kvfpi2ajbie3
e3i6q69ighth
yqzq8rfne0wh
iwrcuuz4lh7i
ytpfh7u5w69b
so5am191gsmm
.
.
.
47tt3jgx6wjt
87bmjq1p9w7a
xuket5mmp4uw
tpsy9hllzxz4
eyfzgknxem9f
m6z044ym12kt
lvr69efjkh1y
o56fj14cwkjg
gwjs8et91a0z
vh5te7refgjh
c8xwnlconcfb
j724xf8y0hut
xgmoeyz777sf

docker ps

, but I prefer using an alternative command to show the

infinite_service

NAME
infinite_service.1
infinite_service.2
infinite_service.3
infinite_service.4
infinite_service.5
infinite_service.6
infinite_service.7
infinite_service.8
infinite_service.9
infinite_service.10
infinite_service.11

IMAGE
eon01/infinite:latest
eon01/infinite:latest
eon01/infinite:latest
eon01/infinite:latest
eon01/infinite:latest
eon01/infinite:latest
eon01/infinite:latest
eon01/infinite:latest
eon01/infinite:latest
eon01/infinite:latest
eon01/infinite:latest

NODE
swarm-1
swarm-1
swarm-1
swarm-1
swarm-1
swarm-1
swarm-1
swarm-1
swarm-1
swarm-1
swarm-1

DESIRED
Running
Running
Running
Running
Running
Running
Running
Running
Running
Running
Running

infinite_service.66
infinite_service.67
infinite_service.68
infinite_service.69
infinite_service.70
infinite_service.71
infinite_service.72
infinite_service.73
infinite_service.74
infinite_service.75
infinite_service.76
infinite_service.77
infinite_service.78

eon01/infinite:latest
eon01/infinite:latest
eon01/infinite:latest
eon01/infinite:latest
eon01/infinite:latest
eon01/infinite:latest
eon01/infinite:latest
eon01/infinite:latest
eon01/infinite:latest
eon01/infinite:latest
eon01/infinite:latest
eon01/infinite:latest
eon01/infinite:latest

swarm-1
swarm-1
swarm-1
swarm-1
swarm-1
swarm-1
swarm-1
swarm-1
swarm-1
swarm-1
swarm-1
swarm-1
swarm-1

Running
Running
Running
Running
Running
Running
Running
Running
Running
Running
Running
Running
Running

CURRENT
Running
Running
Running
Running
Running
Running
Running
Running
Running
Running
Running

STATE
9 minutes
5 minutes
5 minutes
5 minutes
5 minutes
3 minutes
3 minutes
3 minutes
3 minutes
3 minutes
3 minutes

ago
ago
ago
ago
ago
ago
ago
ago
ago
ago
ago

Running 3 minutes ago
Running 3 minutes ago
Running 3 minutes ago
Running 3 minutes ago
Running 3 minutes ago
Running 3 minutes ago
Running 3 minutes ago
Running 4 minutes ago
Running 3 minutes ago
Running 3 minutes ago
Running 3 minutes ago
Running 3 minutes ago
Running 3 minutes ago

e1wcnr6uj8tc
qxm4cqk8rynd
74c9jkn4sdam
07xzshv893x9
twvbe48cx4wl
jhftjr6f14b0
qgbnv32oe9yk
b689gaghcxzd
zo7mu3zw6rw6
xt1p06nvzzi9
kw57wh4thsei
qdws30214xr2
rprasrpkksnf
4894dp78376t
4s6k9hxqwbrr
rct9syhyztuv
b391cod0r2wz
xr7z5l1x14va
rruxgcoaxmjr
tascvsbjidtp
oazgbooulfne
kc7d5xmcvxvy

infinite_service.79
infinite_service.80
infinite_service.81
infinite_service.82
infinite_service.83
infinite_service.84
infinite_service.85
infinite_service.86
infinite_service.87
infinite_service.88
infinite_service.89
infinite_service.90
infinite_service.91
infinite_service.92
infinite_service.93
infinite_service.94
infinite_service.95
infinite_service.96
infinite_service.97
infinite_service.98
infinite_service.99
infinite_service.100

eon01/infinite:latest
eon01/infinite:latest
eon01/infinite:latest
eon01/infinite:latest
eon01/infinite:latest
eon01/infinite:latest
eon01/infinite:latest
eon01/infinite:latest
eon01/infinite:latest
eon01/infinite:latest
eon01/infinite:latest
eon01/infinite:latest
eon01/infinite:latest
eon01/infinite:latest
eon01/infinite:latest
eon01/infinite:latest
eon01/infinite:latest
eon01/infinite:latest
eon01/infinite:latest
eon01/infinite:latest
eon01/infinite:latest
eon01/infinite:latest

swarm-1
swarm-1
swarm-1
swarm-1
swarm-1
swarm-1
swarm-1
swarm-1
swarm-1
swarm-1
swarm-1
swarm-1
swarm-1
swarm-1
swarm-1
swarm-1
swarm-1
swarm-1
swarm-1
swarm-1
swarm-1
swarm-1

Running
Running
Running
Running
Running
Running
Running
Running
Running
Running
Running
Running
Running
Running
Running
Running
Running
Running
Running
Running
Running
Running

Running
Running
Running
Running
Running
Running
Running
Running
Running
Running
Running
Running
Running
Running
Running
Running
Running
Running
Running
Running
Running
Running

3
3
3
3
4
4
4
3
4
3
4
3
3
4
4
4
3
3
4
3
3
3

minutes
minutes
minutes
minutes
minutes
minutes
minutes
minutes
minutes
minutes
minutes
minutes
minutes
minutes
minutes
minutes
minutes
minutes
minutes
minutes
minutes
minutes

ago
ago
ago
ago
ago
ago
ago
ago
ago
ago
ago
ago
ago
ago
ago
ago
ago
ago
ago
ago
ago
ago

Note that containers names are numbered and contains the service name : infinite_service.1, infinite_service.10,
infinite_service.100 The last command will give you more information about containers, information like the desired
state and the current state will give you helpful information about your containers.
The desired-state could take the values:
running
shutdown
accepted
A container current state could be:
running
shutdown
assigned
preparing
You have also the choice to scale you services when starting them using
docker service create --name infinite_service --replicas 10

docker service create

command:

eon01/infinite

infinite_service is a replicated service. Global is another type. In Swarm service could be replicated or global. If
you have used Docker without Swarm, you may notice that Swarm moved Docker from a transactional approach
(operations made directly on containers) to abstraction (there are services and containers and a container is the
"physical" representation of a service and it obeys to Docker distribution strategy when the cluster contains more
than 1 node). We are going to see the difference between the two modes later in this chapter.

Replicated VS Global
Replicated services are distributed by Swarm manager(s) in the form of a specific number of replica tasks among the
nodes. The scale is defined by the user and if it is not defined, it will be 1 by default. We scaled infinite_service to
10, if we have 5 nodes, it is not nece
Global services, are run as one task in every node of the Swarm cluster.
To create a global service, you should use a similar command to this one:
docker service create --name infinite_service --mode global

eon01/infinite

By default services are replicated. So using:
docker service create --name infinite_service --mode replicated

eon01/infinite

is like using:
docker service create --name infinite_service eon01/infinite

You can scale replicated services but it is not allowed to scale global services:
docker service create --name infinite_service --mode global

eon01/infinite

docker service scale infinite_service=2
infinite_service: scale can only be used with replicated mode

Creating & Configuring Swarm Networks
As seen in the networking chapter, Docker has some default built-in networks:
bridge that allow the communication between physical or virtual network interfaces in the host's system.
docker_gwbridge that allows the containers to have external connectivity outside of their cluster. By default
every container is connected to this network.
host that allows containers to have a similar networking configuration to the host.
ingress that exposes containers externally available to the swarm based on a port mapping model
none that helps to create a container-specific network
docker network ls
NETWORK ID
f97187f3cbf9
3a0e460df3be
09493ae999c4
czm3d1slqjxz
31dffc9464fd

NAME
DRIVER
SCOPE
bridge
bridge
local
docker_gwbridge
bridge
local
host
host
local
ingress
overlay
swarm
none
null
local

You can of course create your own bridge network, your own gateway ..etc but since we are using the networking
features built in Docker by default, we are not going to create more networks other than default ones, because we
may need some overlay networks. If we want to create a multi-host networking stack, we need overlay networks.
Creating an overlay network, will make it only available to the nodes attached to our Swarm cluster.
Docker overlay networks are not available to unmanaged containers even if they are living inside a Swarm
node.
By default the nodes encrypt and authenticate information they exchange via gossip using the AES algorithm in
GCM mode but you can add another encryption layer to an overlay network
When using Swarm mode built-in discovery, you don't need to expose service-specific ports to make the
service available to other services on the same overlay network. Swarm will do the job for you.
We can create an overlay network using

docker network create

command. Here is the simplest example:

docker network create -d overlay infinite_net

or
docker network create --driver overlay infinite_net

Let's update the command to create the previous service in order to attach it to this network. Remove the service and
re-create it:
docker service rm infinite_service
docker service create --name infinite_service --network infinite_net

eon01/infinite

A Docker service could be connected to two networks, let's test this:
docker network create -d overlay infinite_net_2
docker service rm infinite_service

docker service create --name infinite_service --network infinite_net --network infinite_net_2

eon01/infinite

Note that you can connect or disconnect a container to a given network using:
docker network connect  
docker network disconnect  

If you want to remove the network called infinite_net_2 , you can use docker network rm infinite_net_2 , but in reality
you will not be able to do it because you had already attached the service infinite_service to this network.

You can not remove a network being used by a service.
For this test, I was using Digital Ocean but for other tests I used some AWS EC2 machines and I had a problem with
the DNS. My solution was to add the option --dns :
docker service rm infinite_service
docker service create --name infinite_service --network infinite_net --dns 8.8.8.8 --dns 8.8.4.4

eon01/infinite

Sometimes, you create multiple networks and play with them but later you may forget to delete the unused ones.
Docker allows you to remove them, you should just type docker network prune .

Inter-Services Communication & Discovery
Let's create two new services attached to two different networks:
docker service rm infinite_service
docker network create -d overlay net_1
docker network create -d overlay net_2
docker service create --name service_1 --network net_1 eon01/infinite
docker service create --name service_2 --network net_2 eon01/infinite

Every service has 1 running container:
CONTAINER ID
dcfb1cc4a2a3
1c09cbbae151

dcfb1cc4a2a3

&

IMAGE
eon01/infinite
eon01/infinite

1c09cbbae151

COMMAND
"/bin/sh -c 'tail ..."
"/bin/sh -c 'tail ..."

NAMES
service_2.1.ilz36bzvph6wbve28wdxzgn08
service_1.1.whz2slvqigmzp2ebh0ku68tft

are the ids of our containers.

If you type docker service inspect
the endpoint is in vip mode:

service_1

, you will see the network configuration where the network is net_1 and

"Networks": [
{
"Target": "net_1"
}
],
"EndpointSpec": {
"Mode": "vip"
}

VIP is a container load balancing mode and we are going to see later the difference between VIP-based load
balancing and DNS-based load balancing.
Note that you can specify the CIDR of your user-defined networks.
Example:
docker network create -d overlay --subnet=192.168.1.0/24 net_1
docker network create -d overlay --subnet=192.168.2.0/24 net_2

Let's re-create the two same services where every one of them will be attached to a different network.
docker service rm service_1 service_2
docker service create --name service_1 --network net_1 eon01/infinite
docker service create --name service_2 --network net_2 eon01/infinite

Let's see what we have on our cluster:
docker ps
CONTAINER ID
c0a751e49756
8fda59787b76

IMAGE
eon01/infinite@sha256:
eon01/infinite@sha256:

COMMAND
"/bin/sh -c 'tail ..."
"/bin/sh -c 'tail ..."

NAMES
service_2.1.
service_1.1.

To see if service_1 and service_2 could establish a communication, we are doing here a simple test using
command:

ping

service_2 can ping service_1:
docker exec -it c0a751e49756 ping -c 2 service_2
PING service_2 (10.0.3.2): 56 data bytes
64 bytes from 10.0.3.2: seq=0 ttl=64 time=0.091 ms
64 bytes from 10.0.3.2: seq=1 ttl=64 time=0.138 ms
--- service_2 ping statistics --2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.091/0.114/0.138 ms

service_1 can ping service_2:
docker exec -it 8fda59787b76 ping -c 2 service_1
PING service_1 (10.0.2.2): 56 data bytes
64 bytes from 10.0.2.2: seq=0 ttl=64 time=0.160 ms
64 bytes from 10.0.2.2: seq=1 ttl=64 time=0.179 ms
--- service_1 ping statistics --2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.160/0.169/0.179 ms

We can see that both services can communicate with each other.
When you create a service using Swarm, you should give it a name and it is going to be resolved by Docker like we
have seen for our two services. There is a mapping between the subnet/IP of a service and its name.
For example: If in the same Docker Swarm cluster, you have php service called php_client that should call a Python
API service called python_api running on port 8000, you can connect from your php code to the API service using a
code similar to this one:
$response = file_get_contents('http://python_api/path/to/api/call?param=x');
$response = json_decode($response);

The container/service will resolve

python_api/

for php to allow it to connect to the API.

Within a swarm cluster, containers can call and request data from other containers using their service's built-in DNS
resolution that can find appropriate distant IP and port automatically. If you call a service (from outside or inside the
cluster), you are in reality requesting one of the containers of this service.
If a service is scaled to 2 or more containers, there is a load balancing algorithm that will redirect the traffic to a
given container. If a container is down (generally because its entry process is not running, it will be cut from traffic
and the load balancer will not redirect traffic to this container. Requests going to a service's containers would be
round-robin load-balanced and it will work even if you did not forward any ports when you created your docker
service.
Docker service discovery uses iptables and IPVS features of Linux Kernel.

VIP vs DNS Based Load Balancing
Since Docker 1.12 was released, two important services were added to the default installation: Service discovery and
load balancing.
Load balancing uses also iptables and IPVS features of Linux Kernel.
IPVS (IP Virtual Server) implements transport-layer load balancing inside the Linux kernel, so called Layer-4
switching. IPVS running on a host acts as a load balancer at the front of a cluster of real servers, it can direct
requests for TCP/UDP based services to the real servers, and makes services of the real servers to appear as a virtual
service on a single IP address.
Docker service discovery and load balancing uses iptables and IPVS features of Linux kernel. Iptables is a packet
filtering technology available in Linux kernel. Iptables can be used to classify, modify and take decisions based on
the packet content. IPvS is a transport level load balancer available in the Linux kernel.
docker service scale service_1=3
docker service scale service_2=2

We scaled our services: service_1 to 3 containers and service_2 to 2 containers. But when a client will request the
service_1, his request will be routed to a single container and the built-in load balancer will be responsible for
routing the request.

If we a cluster of 3 nodes: 2 worker nodes and a managed, containers will be distributed across these nodes and the
load balancing algorithm will keep working the same way and balancing traffic to other nodes in the cluster (not just
within the same machine).

Only managers will load-balance the traffic to services' containers.
This schema shows how a swam manager can distribute traffic to itself and the other worker nodes in the same
cluster.

Previously, when we created our two services, we attached service_1 to net_1 and service_2 to net_2, that's why all
of the three nodes recognize the 2 overlay networks:

DNS Load Balancing
Docker engine comes with an embedded DNS server, if service_1 has 3 running containers, when you request it, the
DNS will return the list of container's IPs without a particular order and your request will be redirected to the frist IP
in the list.

You can specify the DNS mode in a Swarm explicitly using
the same option and add the service_1 to the network net_2:

--endpoint-mode

. Let's re-create our two services using

docker service rm service_1 service_2
docker service create --name service_1 --network net_1 --network net_2 --endpoint-mode dnsrr eon01/infinite
docker service create --name service_2 --network net_2 --endpoint-mode dnsrr eon01/infinite

docker service inspect service_1

You can notice the DNS mode (dnsrr):
"EndpointSpec": {
"Mode": "dnsrr"
}

The DNS load balancing uses the Round Robin algorithm, we are going to see a example of how it works. To see
this, let's scale service_1 to two instances and try to ping it from a container running service_2:
docker service scale service_1=2

Using

docker ps

you can identify the id of the container running service_2. Let's ping service_1 from this container:

docker exec -it 8b9d83c20454 ping -c 1 service_1
PING service_1 (10.0.3.2): 56 data bytes
64 bytes from 10.0.3.2: seq=0 ttl=64 time=0.272 ms

And try it again:
docker exec -it 8b9d83c20454 ping -c 20 service_1
PING service_1 (10.0.3.3): 56 data bytes
64 bytes from 10.0.3.3: seq=0 ttl=64 time=0.266 ms

We can notice that service_1 replied in the first ping with 10.0.3.2 and in the second one with another IP address
10.0.3.3. Docker containers running service_1 have two differnet adresses and the internal load balancer will route
traffic to both of them using a Round Robin balancing method:
docker inspect 75dc68288a2e|grep -i ipv4
"IPv4Address": "10.0.2.3"
docker inspect 91ab0b66aa3a|grep -i ipv4
"IPv4Address": "10.0.2.2"

At this moment, Docker DNS load balancing may have some issues distributing the traffic to scaled services, you
may read this blog post if you are interested to know the effect of RFC3484 on DNS Round Robin balancing. If you
experienced some problems using this feature, you may use the other alternatives like weave, consul, registrator, or
the VIP mode that we are going to see together in the following part of this chapter.

VIP Load Balancing
Starting from 1.12.0 release, Docker added the VIP in-built support for service load balancing using the previously
explained IPVS. The DNS name of the service is mapped to a VIP (Virtual IP).

Using the VIP load balancing, each service will have a single static IP that maps to multiple running containers, if
one ore more containers disappear, the associated IP will not change as long as the service is up.
Let's see how this mode works. Create a new service_1 with two containers using the VIP load balancing and the
service_2:
docker service rm service_1 ; docker service create --name service_1 --network net_1 --network net_2 --replicas 2 -endpoint-mode vip eon01/infinite
docker service rm service_2 ; docker service create --name service_2 --network net_2 --endpoint-mode vip eon01/infinite

Using

docker ps

get the id of the container running service_2 and ping service_1 from that container:

docker exec -it 77a783cfde4f ping -c 1 service_1
PING service_1 (10.0.3.2): 56 data bytes
64 bytes from 10.0.3.2: seq=0 ttl=64 time=0.111 ms

You can try it again and again, but the service_1 will always have the same IP: 10.0.3.2.
As said, this is done using Linux Kernel iptables (for firewall rules and ipvs (for load balacing) and we are going to
use the nsenter command:
Using

docker ps

|grep -i sandbox

, get the id of one of the running service_1 containers, then use this command
in order to find the network namespace.

docker inspect 3877e31787f8|grep -i sandbox
"SandboxID": "f3886fa3028fc42364d7a157b467216abcf466ef8a72efbf69ff11a294addea7",
"SandboxKey": "/var/run/docker/netns/f3886fa3028f",

Now go to

/var/run/docker/netns/

and execute the

nsenter

command with the network

cd /var/run/docker/netns/
nsenter --net=f3886fa3028f sh

Now type:
iptables -nvL -t mangle

You may see something similar to this output:
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target
prot opt in
out
source

destination

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target
prot opt in
out
source

destination

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target
prot opt in
out
source

destination

Chain OUTPUT (policy
pkts bytes target
0 0 MARK all -- *
0 0 MARK all -- *

ACCEPT 0 packets, 0 bytes)
prot opt in
out
source
destination
* 0.0.0.0/0 10.0.2.2 MARK set 0x107
* 0.0.0.0/0 10.0.3.2 MARK set 0x108

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target
prot opt in
out
source
destination

The interesting line for us is :
0 0 MARK all -- * *

0.0.0.0/0 10.0.3.2 MARK set 0x108

f3886fa3028f

:

docker inspect

As you may see, any request of the IP 10.0.3.2, which is the IP of our service is marked by 0x108 (which is 264 in
decimal).
If in the same CLI, you type

ipvsadm

, you will see something like this:

IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port
Forward Weight ActiveConn InActConn
FWM 263 rr
-> 10.0.2.3:0
Masq
1
0
0
-> 10.0.2.4:0
Masq
1
0
0
FWM 264 rr
-> 10.0.3.3:0
Masq
1
0
0
-> 10.0.3.4:0
Masq
1
0
0

The 264 entry (0x108 in hexa) contains the IPs of the two containers running service_1:
10.0.3.3
10.0.3.4

Mangling or packet mangling is a modification applied to network packets in a packet-based network
interface before and/or after routing.
Now if we kill all of the containers running service_1 and ping again, we still have the same VIP address.

As a conclusion, using only 10.0.3.2, the VIP mode will load-balance traffic between 10.0.3.3 and 10.0.3.4.

Updating Services
On of the important features in Docker Swarm is the possibility to apply rolling updates on running containers. This
is done by updating the service. As we have seen, one can scale a service while it is running without having
interruptions:
docker service scale service_1=20

We can also, apply updates on the running image and this could be useful for developers and ops engineers to
upgrade an application. In the following example, we are going to upgrade service_1 from one version to another:
docker service update --image eon01/infinite:0.1 service_1

After applying these updates, you can monitor its status using the inspect command

docker service inspect service_1

"UpdateStatus": {
"State": "updating",
"StartedAt": "2017-02-19T00:41:30.326692292Z",
"CompletedAt": "1970-01-01T00:00:00Z",
"Message": "update in progress"
}

If the update is finished, the status will change:
"UpdateStatus": {
"State": "completed",
"StartedAt": "2017-02-19T00:41:30.326692292Z",
"CompletedAt": "2017-02-19T00:42:02.636303087Z",
"Message": "update completed"
}

If an error happens, you will see something like this:
"UpdateStatus": {
"State": "paused",
"StartedAt": "2017-02-19T00:41:30.326692292Z",
"Message": "update paused due to failure or early termination of task 1p7eth457h8ndf0ui9s0q951b"
}

In this case, just typing

docker service update service_1

will restart the service.

If you need to go back to the previous image, you can run:
docker service update --rollback service_1
You can update DNS:
docker service update --dns-add 8.8.8.8 service_1

Change the host name:
docker service update --hostname host_1 service_1

:

This will change containers' hostnames to host_1:
docker exec -it d86832832991 hostname
host_1
docker exec -it 5362c0d7cc91 hostname
host_1

You can update other configurations and options like:
--args string
--constraint-add list
--constraint-rm list
--container-label-add list
--container-label-rm list
--dns-option-add list
--dns-option-rm list
--dns-rm list
--dns-search-add list
--dns-search-rm list
--endpoint-mode string
--env-add list
--env-rm list
--group-add list
--group-rm list
--health-cmd string
--health-interval duration
--health-retries int
--health-timeout duration
--host-add list
--host-rm list
--label-add list
--label-rm list
--limit-cpu decimal
--limit-memory bytes
--log-driver string
--log-opt list
--mount-add mount
--mount-rm list
--no-healthcheck
--publish-add port
--publish-rm port
--reserve-cpu decimal
--reserve-memory bytes
--restart-condition string
--restart-delay duration
--restart-max-attempts uint
--restart-window duration
--secret-add secret
--secret-rm list
--stop-grace-period duration
--tty, -t
--update-delay duration
--update-failure-action string
--update-max-failure-ratio float
--update-monitor duration
--update-parallelism uint
--user, -u string
--with-registry-auth
--workdir, -w string

Service command args
Add or update a placement constraint (default [])
Remove a constraint (default [])
Add or update a container label (default [])
Remove a container label by its key (default [])
Add or update a DNS option (default [])
Remove a DNS option (default [])
Remove a custom DNS server (default [])
Add or update a custom DNS search domain (default [])
Remove a DNS search domain (default [])
Endpoint mode (vip or dnsrr)
Add or update an environment variable (default [])
Remove an environment variable (default [])
Add an additional supplementary user group to the container (default [])
Remove a previously added supplementary user group from the container (default [])
Command to run to check health
Time between running the check (ns|us|ms|s|m|h)
Consecutive failures needed to report unhealthy
Maximum time to allow one check to run (ns|us|ms|s|m|h)
Add or update a custom host-to-IP mapping (host:ip) (default [])
Remove a custom host-to-IP mapping (host:ip) (default [])
Add or update a service label (default [])
Remove a label by its key (default [])
Limit CPUs (default 0.000)
Limit Memory (default 0 B)
Logging driver for service
Logging driver options (default [])
Add or update a mount on a service
Remove a mount by its target path (default [])
Disable any container-specified HEALTHCHECK
Add or update a published port
Remove a published port by its target port
Reserve CPUs (default 0.000)
Reserve Memory (default 0 B)
Restart when condition is met (none, on-failure, or any)
Delay between restart attempts (ns|us|ms|s|m|h)
Maximum number of restarts before giving up
Window used to evaluate the restart policy (ns|us|ms|s|m|h)
Add or update a secret on a service
Remove a secret (default [])
Time to wait before force killing a container (ns|us|ms|s|m|h)
Allocate a pseudo-TTY
Delay between updates (ns|us|ms|s|m|h) (default 0s)
Action on update failure (pause|continue) (default "pause")
Failure rate to tolerate during an update
Duration after each task update to monitor for failure (ns|us|ms|s|m|h) (default 0s)
Maximum number of tasks updated simultaneously (0 to update all at once) (default 1)
Username or UID (format: [:])
Send registry authentication details to swarm agents
Working directory inside the container

If you need to force an update even if no changes require it, you can use

--force

.

Locking & Unlocking Swarm
This feature is compatible with Docker 1.13 and higher since it is based on the Docker secrets feature.
By default, all of the Raft logs used by Swarm managers are encrypted, you can also use TLS based communications
between nodes in order to protect the Swarm cluster from external malicious access. In both cases (Raft logs
encryption and TLS communication), Swarm use keys (1 key per case) and these keys are stored in the memory of
each manager but you have the possibility to get these keys using the autolock feature.
When you start a manager, it is started by default without autolock:
docker swarm init --autolock=false

By default Docker generates a key and store it on disk, so that managers can restart automatically without a human
intervention. But if you want to lock the Swarm, you should use a similar command to this one :
docker swarm init --autolock=true

This command will generate a worker token then a locking/unlocking key that could be used later to unlock a
manager in order to start a stopped manager, to restart Docker or restore the swarm from a backup.

The generated key should be saved manually in a safe place.
In the case you get the ownership of the key, your Swarm manager will no longer be able to restart without your
intervention, since providing the key is now your responsibility. If you want to give back this responsibility to
Docker, you can do it using:
docker swarm update --autolock=false

Typing this command will disable the encryption of both keys, they are stored unencrypted on disk.

While disabling the lock feature, other Docker managers may be down and will not be able to change their
statuses from locked to unlocked, so they will still need the lock key. So keep them stored in a safe place.
Note that you can lock an already created service using the update command:
docker swarm update --autolock=false

And in order to unlock a Swarm, you may use

docker swarm unlock

and enter your unlock key.

Swarm Backup
A Swarm manager stores the state of the cluster, its logs and the keys used to encrypt the Raft logs in
/var/lib/docker/swarm/ :
ls -lrth /var/lib/docker/swarm/
total 20K
drwxr-xr-x 2 root root 4.0K Feb 7 00:01
drwxr-xr-x 2 root root 4.0K Feb 7 00:01
drwx------ 4 root root 4.0K Feb 7 00:01
-rw------- 1 root root 108 Feb 18 17:03
-rw------- 1 root root
68 Feb 18 17:03

worker
certificates
raft
docker-state.json
state.json

In order to restore an auto-locked Swarm, you will absolutely need these keys and make the backup from any of
your cluster managers.
Unlock key, retrieve it and store it in a safe location or read-lock your swarm to protect its encryption key then make
you backup, it is better if you stop Docker on the manager you are using because data could change during the
backup.
cp -r

/var/lib/docker/swarm/ /backup

Now you can start your manager.
Save these files in a safe place and before that make sure to have valid files, sometimes files may be corrupted. The
best way to check if a file is corrupted or not is to read it or to check its checksum and compare it to a valid file.
cksum 

Swarm Disaster Recovery
Unless you are experimenting and learning you can do this safely of your test servers but beware to have a valid
backup of you are trying to do it on a production server.
Follow the steps enumerated in the previous section Swarm Backup, make sure your files are not corrupted and save
them to a safe place.
In order to make a disaster recovery, create a new Swarm, shut down Docker on the machine where the swarm will
be recovered and remove the files inside /var/lib/docker/swarm , then copy your backup files inside.
If everything is ok, you can start a new Swarm using the following command:
docker swarm init --force-new-cluster

You can add new managers and workers for now.
If your node uses an encryption key (with autolock enabled), it will be the same of your old recovered one and you
can not change it unless your recovery is finished. After finishing the recovery, unlock the Swarm using the old
unlock key then you can create a new key using the key rotation feature.

Swarm Key Rotation
Scheduling a regular key rotation is a best practice. Creating a new key is not complicated, one command to do it:
docker swarm unlock-key --rotate

The previous command will generate a new unlock key and in order to use it to unlock a swarm manager you should
use docker swarm unlock . At this step, the generated key should provided.
Like the first key, the new one should be stored in a safe place. If lost, you will not be able to restart your manager.
The key rotation feature is also used, like we have seen together in the section Swarm Disaster Recovery, in
generating a new key after a Swarm disaster recovery.

Monitoring Swarm Health
HEALTHCHECK is a Docker instruction that tells the engine to test a container in order to check its state and if it is
still working.
Let's see in practice how this works. We are still using the same Swarm that we used trough all of this chapter. Let's
begin by removing the two running services and create a web server service.
docker service rm service_1 service_2

We will use Nginx for our web server:
docker service create --name web_server --replicas 1

-p 80:80 nginx:alpine

Go to your server IP address and check if you have the default Nginx home page.

We are going to use the official image of
the Dockerfile:

nginx:alpine

to create a new image where we add a health check. This is

FROM nginx:alpine
RUN apk add --no-cache curl
RUN echo "hello world" > /usr/share/nginx/html/index.html
HEALTHCHECK --interval=5s --timeout=5s CMD curl --fail -A "healthcheck-routine" http://localhost:80/ || exit 1

I built, committed and pushed this image to my Docker Hub and you can use it directly. You can also build it by
your own:
docker build -t eon01/nginx_healthcheck_example .

Or run it using this command:
docker run -it --name nginx_healthcheck_example -p 80:80 -d eon01/nginx_healthcheck_example

Make sure your port 80 is not used by another local server. Verify that your container is running using
Docker HEALTHCHECK will return the state of the running nginx, if you type this command:
docker inspect --format "{{json .State.Health.Status }}" nginx_healthcheck_example

docker ps

.

It will give you the state of your container, normally it should be: healthy. This state is based on the curl check run
by the instruction:
HEALTHCHECK --interval=5s --timeout=5s CMD curl --fail -A "healthcheck-routine" http://localhost:80/ || exit 1

Where :
interval=DURATION (default: 30s). This is the time interval between executing the healthcheck.
timeout=DURATION (default: 30s). If the check does not finish before the timeout, consider it failed.
retries=N (default: 3). How many times to recheck before marking a container as unhealthy.
You can customize your HEALTHCHECK using other configurations like:
HEALTHCHECK
--interval=2s
--timeout=1s
routine" http://localhost:8080/ || exit 1

--retries=1

CMD

curl

--fail

-A

"healthcheck-

In both cases, the curl command to verify the check is curl --fail -A "healthcheck-routine" http://localhost:8080/ || exit
1 which tells Docker to monitor the exit code of curl, if Nginx stops serving the index.html page, it will fall into 1
and exits. Docker will restart the container after timeout second(s) for retries times and each interval second(s).
In order to simulate a failure, we are going to move index.html and move it to index.html.orig, the index page will be
unreachable and our HEALTHCHECK will be monitoring this failure.
Type:
docker exec -it nginx_healthcheck_example sh

Inside the container:
cd /usr/share/nginx/html && mv index.html index.html.orig

Now (on your host machine), type watch docker ps , wait some seconds and see the STATUS changing from (healthy)
to (unhealthy). At this step, Docker will restart the container.
The HEALTHCHECK state will log the failure that we simulate and any output from the curl command. If you wan
to debug your container, you can use this command:
docker inspect --format "{{json .State.Health }}"

nginx_healthcheck_example

You should be able to see something like these logs:
{
"Status":"unhealthy",
"FailingStreak":77,
"Log":[
{
"Start":"2017-03-07T01:18:35.952544005+01:00",
"End":"2017-03-07T01:18:35.973369673+01:00",
"ExitCode":1,
"Output":" "[...]""
},
{
"Start":"2017-03-07T01:18:40.973509581+01:00",
"End":"2017-03-07T01:18:40.993919955+01:00",
"ExitCode":1,
"Output":" "[...]""
},
{

"Start":"2017-03-07T01:18:45.994059011+01:00",
"End":"2017-03-07T01:18:46.014129303+01:00",
"ExitCode":1,
"Output":" "[...]""
},
{
"Start":"2017-03-07T01:18:51.014382522+01:00",
"End":"2017-03-07T01:18:51.063646337+01:00",
"ExitCode":1,
"Output":" "[...]""
},
{
"Start":"2017-03-07T01:18:56.064089164+01:00",
"End":"2017-03-07T01:18:56.135445869+01:00",
"ExitCode":1,
"Output":" "[...]""
}
]
}

Docker health check is a useful feature, if the container fails health checks or terminates, the task terminates and the
orchestrator creates a new replica task that spawns a new container.

Using Secrets In Swarm Mode
Starting from the version 1.13, Docker users can use Docker Secrets when working in Swarm Mode.
A secret is a blob of data like:
password
SSH/SSL private key and certificates
A secret key that could be stored in your Dockerfile
Sensitive environment data that could be transmitted over a network
Sensitive data in the application source code that could be transmitted over a network
Other data such as the user-name or the name of a database, generic strings or even binary content ( < 500 kb)
..etc
In one of the above cases, Docker Secrets allow you to centrally manage sensitive data and transmit it in a secure
way and only to containers that need access to it. This sensitive data, is encrypted in both rest and transmit.
When you add a secret to a Swarm it is sent to the manager over a TLS connection.
A TLS connection uses the Transport Layer Security protocol that provides privacy and data integrity in containers
inter-communications.

Docker has its built-in certificate authority.
After that, the secret is stored in the Raft log, which is encrypted by default (> 1.13) and when this log is replicated
across all of the other managers, the secret is securely propagated.
If a new node joins the cluster, at the moment when the latter will run a given service, a manager will establish a
TLS connection to the new node and will securely send the service's secret data. A secret is then unencrypted and
mounted into the container in an in-memory file system.
These mounts have

/run/secrets/

as a target.

In the case when a node is no longer running a service, all of the latter's secrets disappear from the node: The node
will no longer have access to secrets. This process is made by the node itself after having a notification from a
manager.

The Docker Secret is never written to disk on any of the Swarm nodes, instead of disk, a secret is written to the
tmpfs volume and it is not possible to recover it from the Docker CLI but you should actually read it from the
service(s) where it has been injected.

tmpfs is a temporary file storage facility used in many Unix-like OSs. It is intended to appear as a mounted
file system, but stored in RAM instead of a disk.

Creating & Storing Secrets
Let's try some practical examples.
In some of the examples we run a Mysql database using this command:
docker run --name mysql -p 3306:3306 --env MYSQL_ROOT_PASSWORD=password -d mysql

and when we want to create a Swarm we need to run
docker service create --name mysql -p 3306:3306 --env MYSQL_ROOT_PASSWORD=password mysql

With Docker Secrets, the run command will be:
docker
service
create
--name
mysql
-p
env MYSQL_ROOT_PASSWORD_FILE=/run/secrets/mysqlrootpassword mysql

/run/secrets/mysqlrootpassword

3306:3306

--secret

mysql-root-password

--

is the file where the root password is encrypted. Here is how to create it:

Say our password is simply "password"
After creating your Swarm, open the terminal and type: echo "password"|docker secret create mysqlrootpassword You will get the id of the secret. In my case : vbsvaiunxjau6r6isnc3vxcyt
Now you can check the secret file under /run/secrets/ in the Mysql running container: docker exec

-it

mysql.1.j9tixgi8pmwa131ux5screqaq ls /run/secrets/mysql-root-password

When I created the Mysql service, it was deployed to a host that I called host_1:
ID
j9tixgi8pmwa

NAME
mysql.1

IMAGE
mysql:latest

To make a double check, we can type:
container.

NODE
host_1

DESIRED STATE
Running

CURRENT STATE
ERROR
Running 4 minutes ago

mysql -h host_1 -uroot -ppassword

PORTS

and we will be connected to the Mysql

In order to see the injected secrets, type:
docker secret ls

e.g.:
ID
wf048or4g6v7gn3v8pmj57vz0

NAME
mysqlrootpassword

CREATED
35 minutes ago

UPDATED
35 minutes ago

Sending Secrets Between Services
In my example, we have two Mysql services running in the same cluster of Swarm nodes, the first database is
mysql_1 running the port 3306 and the second one is mysql_2 and it is running the port 3307. In order to see how we
can send secrets, the first service will share the same password across the cluster in order to be used by the new
service mysql_2 as a root password.
To accomplish this, we need to create an overlay network to which both services will be attached.
docker network create -d overlay mysql_network

Remove the first Mysql service:
docker service rm mysql

Create a new secret:
echo "password"|docker secret create mysql1rootpassword -

Create the first service:
docker service create \
--name mysql_1 \
-p 3306:3306 \
--secret mysql1rootpassword \
--env MYSQL_ROOT_PASSWORD_FILE=/run/secrets/mysql1rootpassword \
--network mysql_network \
mysql

Create the second service:
docker service create \
--name mysql_2 \
-p 3307:3306 \
--secret source=mysql1rootpassword,target=mysql2rootpassword,mode=0400 \
--env MYSQL_ROOT_PASSWORD_FILE=/run/secrets/mysql2rootpassword \
--network mysql_network \
mysql

You may have noticed how using --secret source=mysql1rootpassword,target=mysql2rootpassword creates a new file in the
tmpfs of the containers running mysql_2 service and if not you can check it by logging into one of the containers and
executing cat /run/secrets/mysql2rootpassword .
Sending secrets between services is easy and secure and it is possible in the case when services are accessible to
each other (in our case, both services are in the same network).

Rotating Secrets
The good thing about using Docker Secrets is the modularity and the separation between code and secrets: Changing
a secret does not require changing code. In the other hand, changing secrets is a good security practice. Let's see how
to rotate secrets.
Our password is changing from "password" to "new_password", we create a new secret in this case:
echo "new_password"|docker secret create newmysqlrootpassword -

Remove the first secret:
docker service update --secret-rm mysqlrootpassword mysql_1

Update the first secret by the new one:
docker service update --secret-add source=newmysqlrootpassword,target=mysqlrootpassword mysql_1

Update the second Mysql service (note the combination of two steps in one command):
docker
service
update
--secret-rm
add source=newmysqlrootpassword,target=mysqlrootpassword,mode=0400 mysql_2

mysqlrootpassword

--secret-

Let's make some verifications: Type
CONTAINER ID
8b305cb625ed
a9e64fff8e06

IMAGE
mysql:latest
mysql:latest

docker ps

to get the list of containers:

COMMAND
"docker-entrypoint..."
"docker-entrypoint..."

PORTS
3306/tcp
3306/tcp

NAMES
mysql_2.1.
mysql_1.1.

Check the content of the secret file:
docker exec -it mysql_2.1.1piq2o2ascilmjighsq4qxld9 cat /run/secrets/mysqlrootpassword

If you get this

new_password

then everything was well done in both services.

Chapter XIII - Orchestration - Kubernetes
o
o

^__^
(oo)\_______
(__)\
)\/\
||----w |
||
||

Kubernetes is one of the most know orchestration systems. Kubernetes needs more than a chapter, it needs a book. I
recently created a project called Books For DevOps where you can find several hand curated books, some of them
are about Kubernetes. In this chapter, we are going to see how to deploy and use K8S but some concept definitions is
obligatory before moving to this part.

Introduction
First, let's have a look at the Kubernetes architecture.

Master Components
In order to schedule and orchestrate a set of containers and services, Kubernetes needs a master. The master
components are the controlling services that operate to manage the cluster and gives administrator an interface to
manage deployment, rollbacks and other operations. These master components can be installed on a single machine
and you can also create a distributed system of master components where multiple machines forms a master.
More specifically, master components are :
Etcd: a distributed key-value store for the critical data of a distributed system
API Server: the main management point of the entire cluster
Controller Manager Service : a daemon that embeds the core control loops shipped with K8S. It enables a nonterminating loop that regulates the state of the system by watching the shared state of the cluster through the
apiserver and making changes attempting to move the current state towards the desired state (e.g. the
replication controller, endpoints controller, namespace controller, serviceaccounts controller)
Scheduler Service: responsible for assigning workloads to specific nodes in a cluster. It is actually used to read
in a service's operating requirements then analyze the infrastructure environment in order to distribute a work
on the right node(s)

Pods
If Kubernetes is an operating system, then a pod is the process.
A pod is simply a group of one or more containers, the shared storage for those containers, and the different
configurations describing how to run the containers. Two or more containers in the same pod, share an IP address
and port space. They can find each other via localhost and communicate with each other using inter-process
communications (e.g. SystemV semaphores, POSIX shared memory).
Pods serve as unit of deployment, horizontal scaling, and replication. They enable data sharing and communication
among their containers and can be used to host vertically integrated application stacks (e.g. LAMP).

Deployments
With a declarative approach, deployment update pods. Deployment controller will change the actual state to the
desired state when a Deployment object desired state is updated by the user. A Deployment is described using a
YAML file, this is an example of a Deployment that creates a ReplicaSet to bring up 5 Apache Pods:
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: apache-deployment
spec:
replicas: 5
template:
metadata:
labels:
app: apache
spec:
containers:
- name: apache
image: httpd
ports:
- containerPort: 80

A Deployment supports operations like updating, rollback, checking rollout history, pausing and resuming.

Services
A Kubernetes service is an abstraction of the pod they consistently maintain a well-defined endpoint for pods. A pod
can be placed in a different node, it can also die and get resurrected but Kubernetes has a built-in service discovery
that allows a service to have the same endpoints of changing pods.
K8S Services could be
Internal (e.g. databases and private pods in a more general)
External (e.g. any service that needs an external endpoint with a specific port like Nginx)
Load balanced (e.g. services that are load balanced. Kubernetes can use some cloud providers managed load
balancers like AWS ELB or GCE load balacing service: This feature enables integrating a third-party load
balancer with the Kubernetes.)

Replication Controller
A Replication Controller is responsible for ensuring that a pod or homogeneous group of pods are always up and
reachable. When there are many or too few pods, it will regulate them by killing or starting new pods: a specified
number of pod replicas should be running at any given time.

Replicaset
ReplicaSet is the next-generation Replication Controller. ReplicaSet has a generalized label selector.

Nodes/Minions
A minion is a worker machine that could be a virtual or a bare metal machine that runs pods and are managed by the
Kubernetes master component.

Kubelet
The kubelet is the primary node agent that runs on each minion

The Container Runtime
Each node runs a container runtime. The latter is responsible for downloading images and running containers like
Docker.

Kube Proxy
kube-proxy is a daemon that runs on each minion and acts like a network proxy and load balancer for the services
living in that minion.

A Local Kubernetes Using Minikube
After the last definitions, we can start manipulating K8S, so let's start a local cluster.

Installation
We will proceed for the moment with Minikube. Minikube is a tool that makes it easy to run Kubernetes locally.
Minikube runs a single-node Kubernetes cluster inside a VM on a laptop. It is useful for development since it
packages and configures a Linux VM, the container runtime, and all Kubernetes components and it supports
Kubernetes features such as:
DNS
NodePorts
ConfigMaps and Secrets
Dashboards
Container Runtime: Docker and rkt
Enabling CNI (Container Network Interface)
You can get the installation instructions from the official Minikube repository. I am going to use the Linux
installation.
curl
-Lo
minikube
https://storage.googleapis.com/minikube/releases/v0.18.0/minikube-linuxamd64 && chmod +x minikube && sudo mv minikube /usr/local/bin/

In order to use it with OS X, you will need xhyve driver, VirtualBox or VMware Fusion installation With Linux you
will need VirtualBox or KVM installation and with Windows VirtualBox or Hyper-V installation is needed.
In allOSs, VT-x/AMD-v virtualization must be enabled in BIOS and kubectl (>1.0) should be installed.
The installation of kubectl depends on your system :
For OS X :
curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl
release/release/stable.txt)/bin/darwin/amd64/kubectl

-s

https://storage.googleapis.com/kubernetes-

-s

https://storage.googleapis.com/kubernetes-

-s

https://storage.googleapis.com/kubernetes-

For Linux :
curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl
release/release/stable.txt)/bin/linux/amd64/kubectl

For Windows users :
curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl
release/release/stable.txt)/bin/windows/amd64/kubectl.exe

After the installation, you should make the kubectl binary executable and move it to your PATH :

chmod +x ./kubectl
sudo mv ./kubectl /usr/local/bin/kubectl

Running Minikube
First command to run to bootstrap a K8S cluster is

minikube start

.

This command will start a local VM, downloa the Minikube ISO and check if the requirements are satisfied.
Starting VM...
Downloading Minikube ISO
89.51 MB / 89.51 MB [==============================================] 100.00% 0s

If your requirements are not satisfied, you will get an error message.
Example :
Error
starting
host:
Error
creating
host:
Error
create check: "VBoxManage not found. Make sure VirtualBox is installed and VBoxManage is in the path"

with

pre-

I had a problem first time working with Minikube and the error message was :
Error starting host:

The solution was
Executing

Error configuring auth on host: Too many retries waiting for SSH to be available.

minikube delete && minikube start

minikube delete && minikube start

Last error: Maximum number of retries (6

. The solution was mentioned in this issue.

, will :

Delete local Kubernetes cluster
Start local Kubernetes cluster
Start the VM
SSH files into VM
Set up certs
Start cluster components
Connect to cluster
Set up kubeconfig

Kubectl
Kubectl is now configured to use the cluster.
To connect to the target cluster Minikube has already run some commands like:
kubectl config set-cluster default-cluster --server=https://${MASTER_HOST} --certificate-authority=${CA_CERT}
kubectl
config
set-credentials
default-admin
--certificate-authority=${CA_CERT}
--client-key=${ADMIN_KEY}
certificate=${ADMIN_CERT}
kubectl config set-context default-system --cluster=default-cluster --user=default-admin
kubectl config use-context default-system

In order to make a verification, we can type
output to the following:

kubectl get nodes

--client-

and everything was setup right, we will get a similar

NAME
minikube

STATUS
Ready

AGE
2h

VERSION
v1.6.0

Let's run Nginx in a container:
kubectl run nginx --image=nginx --port=80 --replicas=1

Now we can type: kubectl get
(localhost) or simply the pods:
NAME
nginx-158599303-xxf1d

READY
1/1

pods

to view the group of containers that are deployed together on the same host

STATUS
Running

RESTARTS
0

AGE
1m

If we were running other pods, it is possible to "filter" the

get pods

output using:

kubectl get pods --selector="run=nginx" --output=wide

NAME
nginx-29nqv
nginx-g24b3

Without the

READY
1/1
1/1

STATUS
AGE
IP
NODE
Running
3h
172.17.0.4
minikube
Running 3h
172.17.0.5
minikube

--output=wide

NAME
nginx-158599303-29nqv
nginx-158599303-g24b3

:
READY
1/1
1/1

STATUS
RESTARTS
Running
0
Running
0

We can also show other options like labels, using
NAME
nginx-29nqv
nginx-g24b3

READY
1/1
1/1

DESIRED
1

kubectl get pods --show-labels

:

STATUS LABELS
Running pod-template-hash=158599303,run=nginx
Running pod-template-hash=158599303,run=nginx

Let's view the deployment list:
NAME
nginx

AGE
3h
3h

CURRENT
1

kubectl get deployments

UP-TO-DATE
1

AVAILABLE
1

:
AGE
22m

We can have more information about the pod, using kubectl get -o json pod nginx-158599303-xxf1d . Don't forget to
always adapt the commands to your specific case, change nginx-158599303-xxf1d by your pod name.
{
"apiVersion": "v1",
"kind": "Pod",
"metadata": {
"annotations": {
"kubernetes.io/created-by":
"{\"kind\":\"SerializedReference\",\"apiVersion\":
\"v1\",\"reference\":{\"kind\":\"ReplicaSet\",
\"namespace\":\"default\",\"name\":\"nginx-158599303\",
\"uid\":\"5a0dc0a9-2aca-11e7-8020-0800275cee00\",
\"apiVersion\":\"extensions\",
\"resourceVersion\":\"9004\"}}\n"
},
"creationTimestamp": "2017-04-26T21:50:24Z",
"generateName": "nginx-158599303-",
"labels": {

"pod-template-hash": "158599303",
"run": "nginx"
},
"name": "nginx-158599303-xxf1d",
"namespace": "default",
"ownerReferences": [
{
"apiVersion": "extensions/v1beta1",
"blockOwnerDeletion": true,
"controller": true,
"kind": "ReplicaSet",
"name": "nginx-158599303",
"uid": "5a0dc0a9-2aca-11e7-8020-0800275cee00"
}
],
"resourceVersion": "9075",
"selfLink": "/api/v1/namespaces/default/pods/nginx-158599303-xxf1d",
"uid": "5a103f82-2aca-11e7-8020-0800275cee00"
},
"spec": {
"containers": [
{
"image": "nginx",
"imagePullPolicy": "Always",
"name": "nginx",
"ports": [
{
"containerPort": 80,
"protocol": "TCP"
}
],
"resources": {},
"terminationMessagePath": "/dev/termination-log",
"terminationMessagePolicy": "File",
"volumeMounts": [
{
"mountPath": "/var/run/secrets/kubernetes.io/serviceaccount",
"name": "default-token-jk2cd",
"readOnly": true
}
]
}
],
"dnsPolicy": "ClusterFirst",
"nodeName": "minikube",
"restartPolicy": "Always",
"schedulerName": "default-scheduler",
"securityContext": {},
"serviceAccount": "default",
"serviceAccountName": "default",
"terminationGracePeriodSeconds": 30,
"volumes": [
{
"name": "default-token-jk2cd",
"secret": {
"defaultMode": 420,
"secretName": "default-token-jk2cd"
}
}
]
},
"status": {
"conditions": [
{
"lastProbeTime": null,
"lastTransitionTime": "2017-04-26T21:50:24Z",
"status": "True",
"type": "Initialized"
},
{
"lastProbeTime": null,
"lastTransitionTime": "2017-04-26T21:51:14Z",
"status": "True",
"type": "Ready"
},
{
"lastProbeTime": null,
"lastTransitionTime": "2017-04-26T21:50:24Z",
"status": "True",
"type": "PodScheduled"
}
],
"containerStatuses": [
{
"containerID": "docker://b..3",
"image": "nginx:latest",
"imageID": "docker://sha256:4..8",
"lastState": {},
"name": "nginx",
"ready": true,
"restartCount": 0,
"state": {

"running": {
"startedAt": "2017-04-26T21:51:13Z"
}
}
}
],
"hostIP": "192.168.99.100",
"phase": "Running",
"podIP": "172.17.0.4",
"qosClass": "BestEffort",
"startTime": "2017-04-26T21:50:24Z"
}
}

This JSON file contains information that can be included in a YAML file, the latter can be used with kubectl
command to create a pod: kubectl create -f pod.yaml .
Let's delete the deployment nginx and create the YAML specifications file that can be used to re-create the pod.
kubectl delete deployments/nginx

Note that you can get a list of deployments, using

kubectl get deployments

.

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: nginx
spec:
replicas: 2
template:
metadata:
labels:
run: nginx
spec:
containers:
- name: nginx
image: nginx
port:
- containerPort: 80

You can find this code in

Orchestration_Kubernetes_Running_Minikube.yaml

file.

In order to use it, type:
kubectl create -f Orchestration_Kubernetes_Running_Minikube.yaml --record

Using --record flag allows you to record the last command in the annotations of the created resources (in the case of
an update also). This could be useful to see the commands executed in each Deployment with a revision. To
understand this, type:
kubectl describe deployments

And you will get a similar output to this one:
Name:
nginx
Namespace: default
CreationTimestamp: Thu, 27 Apr 2017 11:21:43 +0200
Labels:
run=nginx
Annotations: deployment.kubernetes.io/revision=1
kubernetes.io/change-cause=kubectl create\
--filename=Orchestration_Kubernetes_Running_Minikube.yaml --record=true
Selector: run=nginx
Replicas: 2 desired | 2 updated | 2 total | 2 available | 0 unavailable
StrategyType: RollingUpdate
MinReadySeconds: 0

RollingUpdateStrategy: 1 max unavailable, 1 max surge
Pod Template:
Labels: run=nginx
Containers:
nginx:
Image: nginx
Port: 80/TCP
Environment: 
Mounts: 
Volumes: 
Conditions:
Type
Status
Reason
-------------Available
True
MinimumReplicasAvailable
OldReplicaSets: 
NewReplicaSet: nginx-158599303 (2/2 replicas created)
Events:
FirstSeen LastSeen Count From SubObjectPath
Type Reason Message
--------- -------- ----- ---- ----------------- ------ ------4m 4m 1
deployment-controller Normal ScalingReplicaSet Scaled up replica set nginx to 2

Notice the annotations:
Annotations: deployment.kubernetes.io/revision=1
kubernetes.io/change-cause=kubectl create \
--filename=Orchestration_Kubernetes_Running_Minikube.yaml --record=true

We can get more details about the created Nginx using

kubectl describe pods

:

Name:
nginx-158599303-29nqv
Namespace:
default
Node:
minikube/192.168.99.100
Start Time:
Thu, 27 Apr 2017 11:21:43 +0200
Labels:
pod-template-hash=158599303
run=nginx
Annotations:
kubernetes.io/created-by=
{"kind":"SerializedReference","apiVersion":"v1","reference":
{"kind":"ReplicaSet","namespace":"default","name":"nginx-158599303","uid":"xxxxx","a...
Status:
Running
IP:
172.17.0.4
Controllers:
ReplicaSet/nginx-158599303
Containers:
nginx:
Container ID:
docker://xxxx
Image:
nginx
Image ID:
docker://sha256:xxxx
Port:
80/TCP
State:
Running
Started:
Thu, 27 Apr 2017 11:22:22 +0200
Ready:
True
Restart Count:
0
Environment:

Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-27fkv (ro)
Conditions:
Type
Status
Initialized
True
Ready
True
PodScheduled
True
Volumes:
default-token-27fkv:
Type:
Secret (a volume populated by a Secret)
SecretName:
default-token-27fkv
Optional:
false
QoS Class:
BestEffort
Node-Selectors:

Tolerations:

Events:
FirstSeen
LastSeen
Count
Message
-------------------------23m
23m
1
default-scheduler
Successfully assigned nginx-158599303-29nqv to minikube
23m
23m
1
kubelet, minikube
pulling image "nginx"
22m
22m
1
kubelet, minikube
Successfully pulled image "nginx"
22m
22m
1
kubelet, minikube
Created container with id xxxx
22m
22m
1
kubelet, minikube
Started container with id xxxx

Name:
Namespace:
Node:
Start Time:
Labels:

nginx-158599303-g24b3
default
minikube/192.168.99.100
Thu, 27 Apr 2017 11:21:43 +0200
pod-template-hash=158599303

run=nginx
Annotations:
kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":
{"kind":"ReplicaSet","namespace":"default","name":"nginx-158599303","uid":"xxxxx","a...
Status:
Running
IP:
172.17.0.5
Controllers:
ReplicaSet/nginx-158599303
Containers:
nginx:
Container ID:
docker://xxxx
Image:
nginx
Image ID:
docker://sha256:xxxx
Port:
80/TCP
State:
Running
Started:
Thu, 27 Apr 2017 11:22:21 +0200
Ready:
True
Restart Count:
0
Environment:

Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-27fkv (ro)
Conditions:
Type
Status
Initialized
True
Ready
True
PodScheduled
True
Volumes:
default-token-27fkv:
Type:
Secret (a volume populated by a Secret)
SecretName:
default-token-27fkv
Optional:
false
QoS Class:
BestEffort
Node-Selectors:

Tolerations:

Events:
FirstSeen
LastSeen
Count
From
Message
----------------------------23m
23m
1
default-scheduler
assigned nginx-158599303-g24b3 to minikube
23m
23m
1
kubelet, minikube
pulling image "nginx"
22m
22m
1
kubelet, minikube
Successfully pulled image "nginx"
22m
22m
1
kubelet, minikube
Created container with id xxxx
22m
22m
1
kubelet, minikube
Started container with id xxxx

We can use less verbose commands like
NAME
nginx

or

DESIRED
2

kubectl get pods

CURRENT
2

kubectl get deployments

UP-TO-DATE
2

AVAILABLE
2

AGE
31m

RESTARTS
0
0

AGE
33m
33m

:

:

NAME
nginx-158599303-29nqv
nginx-158599303-g24b3

READY
1/1
1/1

STATUS
Running
Running

Now we can create a service object that exposes the last deployment:
kubectl expose deployment nginx --port=80

We can use the same get command used above to show the JSON specefication:
kubectl get -o json service nginx

This is the same spec that we wrote in a YAML file but it is in JSON format:
{
"apiVersion": "v1",
"kind": "Service",
"metadata": {
"creationTimestamp": "2017-04-27T10:10:29Z",
"labels": {
"run": "nginx"
},

"name": "nginx",
"namespace": "default",
"resourceVersion": "4669",
"selfLink": "/api/v1/namespaces/default/services/nginx",
"uid": "bdf36511-2b31-11e7-b29a-080027c0925b"
},
"spec": {
"clusterIP": "10.0.0.234",
"ports": [
{
"port": 80,
"protocol": "TCP",
"targetPort": 80
}
],
"selector": {
"run": "nginx"
},
"sessionAffinity": "None",
"type": "ClusterIP"
},
"status": {
"loadBalancer": {}
}
}

We can get a specefic part of these information using

--template=

tag with the

get scv

command.

Example:
kubectl get service nginx --template={{.spec.clusterIP}}

will give you the IP address of the service (nginx)
kubectl get service nginx --template={{.metadata.name}}

will give you the name of the service
etc ..
Another way to view information about a service is using the describe command like
An output similar to the following one, will show on your screen:

kubectl describe service nginx

Name:
nginx
Namespace:
default
Labels:
run=nginx
Annotations:

Selector:
run=nginx
Type:
ClusterIP
IP:
10.0.0.234
Port:

80/TCP
Endpoints:
172.17.0.4:80,172.17.0.5:80
Session Affinity:
None
Events:


In order to view the ReplicaSet objects, we can use
NAME
nginx-158599303

Or

DESIRED
2

kubectl describe replicasets

CURRENT
2

READY
2

kubectl get replicasets

AGE
3h

to obtain more details:

Name:
nginx-158599303
Namespace:
default
Selector:
pod-template-hash=158599303,run=nginx
Labels:
pod-template-hash=158599303
run=nginx
Annotations:
deployment.kubernetes.io/desired-replicas=2

to obtain a list:

.

deployment.kubernetes.io/max-replicas=3
deployment.kubernetes.io/revision=1
kubernetes.io/change-cause=kubectl create
--filename=Orchestration_Kubernetes_Running_Minikube.yaml
--record=true
Replicas:
2 current / 2 desired
Pods Status:
2 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
Labels:
pod-template-hash=158599303
run=nginx
Containers:
nginx:
Image:
nginx
Port:
80/TCP
Environment:

Mounts:

Volumes:

Events:


In order to scale up the service to 5 instances, we can use this command:
kubectl scale deployments nginx --replicas=5

Let's run another service. It is possible to create a service from a remote YAML file. This is the hello-world service:
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: hello
spec:
replicas: 7
template:
metadata:
labels:
app: hello
tier: backend
track: stable
spec:
containers:
- name: hello
image: "gcr.io/google-samples/hello-go-gke:1.0"
ports:
- name: http
containerPort: 80

You can find the file here :

https://kubernetes.io/docs/tutorials/connecting-apps/hello.yaml

and you can use it this way:

kubectl create -f http://k8s.io/docs/tutorials/connecting-apps/hello.yaml

Now the

kubectl get pods

NAME
hello-1987912066-1tnxq
hello-1987912066-3fwvt
hello-1987912066-503fc
hello-1987912066-7htlm
hello-1987912066-pvbmj
hello-1987912066-qc0xt
hello-1987912066-qvqlr
nginx-158599303-2rgff
nginx-158599303-g24b3
nginx-158599303-jkjm7
nginx-158599303-k0zt3
nginx-158599303-rskx3

command, will show :
READY
1/1
1/1
1/1
1/1
1/1
1/1
1/1
1/1
1/1
1/1
1/1
1/1

STATUS
Running
Running
Running
Running
Running
Running
Running
Running
Running
Running
Running
Running

RESTARTS
0
0
0
0
0
0
0
0
0
0
0
0

AGE
6m
6m
6m
6m
6m
6m
6m
31s
6h
39s
34s
37s

Publishing Services & Services Types
Now that we have Minikube & Kubectl and we were able to create services, let's run this service:

kubectl run hello-minikube --image=gcr.io/google_containers/echoserver:1.4 --port=8080

echoserver is a simple application that responds with the HTTP headers it received.
Let's publish the created service:
kubectl expose deployment hello-minikube --type=NodePort

You may have noticed the NodePort option. Kubernetes ServiceTypes allow you to specify what kind of service you
want. The default value is ClusterIP.
ClusterIP exposes the service on a cluster-internal IP so that the service will be reachable only from
within the cluster
NodePort exposes the service on each node's IP at a static port and create a ClusterIP service. The
latter will recieve the trafix routed from the NodePort service. In this type, the NodePort service will
be reachable from outside the cluster suing : .
LoadBalancer type exposes the service externally using a cloud provider’s load balancer. This could
be limited since Kubernetes load balancing is not implemetend with many cloud vendors. The
NodePort and the ClusterIP services will recieve traffic from the load balancer and they are
automatially created.
ExternalName will map the service to the contents of the externalName field. (e.g.
k8s.painlessdocker.com). ExternalName will return a CNAME record with its value and this feature
requires kube-dns > 1.7.
We considered using the NodePort and in this case Kubernetes master will allocate a port from a configured range
(30000 -> 32767), and each Node will proxy that port into your service. That port will be reported in the service’s
spec.ports[*].nodePort field.

In the case we have multiple Nodes, Kubernetes will use the same port number on every Node.
Now we can use Curl to get the service response, but we don't know the url neither the port and that's why this
command will help us:
minikube service hello-minikube --url

Now we can easily do :

curl $(minikube service hello-minikube --url)

CLIENT VALUES:
client_address=172.17.0.1
command=GET
real path=/
query=nil
request_version=1.1
request_uri=http://192.168.99.100:8080/
SERVER VALUES:
server_version=nginx: 1.10.0 - lua: 10001
HEADERS RECEIVED:
accept=*/*
host=192.168.99.100:30805
user-agent=curl/7.47.0
BODY:
-no body in request-

and we will get something similar to this:

You can use your browser also !
We can see the IP address and the interface (vboxnet1 in my case and it most probably the same for you ) using
minikube ip
and we will notice that the IP is 192.168.99.1 , the broadcast is 192.168.99.255 and the mask is
255.255.255.0 .
Use

ifconfig

vboxnet1

:
Link encap:Ethernet HWaddr 0a:00:27:00:00:01
inet addr:192.168.99.1 Bcast:192.168.99.255 Mask:255.255.255.0
inet6 addr: fe80::800:27ff:fe00:1/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:203 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:40551 (40.5 KB)

Using Kubernetes With Google Container Engine
Prerequisites
GKE - Google Container Engine is a service managed by Google that makes easy to run containers in Google cloud.
It uses Kubernetes.
In this part, we will run a Wordpress blog in a managed Kubernetes cluster so you need a Google Cloud account and
a project that you can create using GCP UI:

Now go to the project dashboard

Remember the project ID which is a unique name across all Google Cloud projects. It will be referred to later in this
codelab as PROJECT_ID.
Next, you will need to enable billing in the Developers Console in order to use Google Cloud resources like Cloud
Datastore and Cloud Storage. Now go and enable APIs in the API Manager:

Click on Compute Engine API to enable it:

You must also enable Google Container Engine API

Setting Up The Compute Zone
Start Google Cloud Shell by clicking on the terminal icon (the icon is in the top toolbar). This Shell is connected to a
virtual machine where GCP tools are installed.

We will be using gcloud command in order to create our cluster but first let's set the compute zone so all of the VMs
of this cluster will be created in the choosed region.
gcloud config set compute/zone 

In our cluster, we are going to choose the us-central1-b zone:
gcloud config set compute/zone us-central1-b

You will get a similar output to this one:
Updated property [compute/zone].

GCP has 3 regions: Americas, Europe & Asia, each regions has some zones. You can use your preferred
region and zone and you will find them in the official Google documentation.

Creating The Cluster
The gcloud command will allow us to create a new cluster, we will call it

my-test-node

:

gcloud container clusters create my-test-node --num-nodes 1

At the moment when you create the cluster, if you get a warninig like
WARNING: Accessing a Container Engine cluster requires the kubernetes commandline client [kubectl].

Then you should install kubectl component.
gcloud components install kubectl

This command creates a new cluster called "hello-world" with one node (VM). Since we'll only be launching one
container a single node is fine. If you wanted to launch multiple containers you could specify a different number of
nodes when you create the cluster. When launching the cluster
Creating cluster my-test-node ... done
Created
[https://container.googleapis.com/v1/projects/wordpress-161315/zones/us-central1-b/clusters/my-testnode]. kubeconfig entry generated for my-test-node.
NAME
my-test-node

ZONE
us-central1-b

MASTER_VERSION
1.5.3

MASTER_IP
35.184.199.238

MACHINE_TYPE
n1-standard-1

NODE_VERSION
1.5.3

NUM_NODES
1

STATUS
RUNNING

You can also verify if everything was ok with the cluster launch by listing the instance of your cluster:
gcloud compute instances list

You will get a similar output to this:
NAME
gke-my-test-node-default-pool-f295ca85-pvjc

ZONE
us-central1-b

MACHINE_TYPE
n1-standard-1

PREEMPTIBLE

INTERNAL_IP
10.128.0.2

EXTERNAL_IP
104.198.50.198

STATUS
RUNNING

Creating The Wordpress Services
Now that our cluster is running, it is time to deploy a Wordpress container to it. We'll be using the tutum/wordpress
image that contains all what we need to run a Wordpress site, including a MySQL database in a single Docker
container.
We are using this Wordpress image just to demonstrate how GKE works but it is not a good practice to run
several processes in the same Docker containers.

Creating Our Pod
Like we said a pod is one or more containers that "travel together", they could be administered together and could
have the same network requirements ..etc.

Let's create the pod using kubectl:

We are always using the same Cloud Shell:
gcloud container clusters get-credentials my-test-node --zone us-central1-b --project wordpress-161315

Remember the project ID:

wordpress-161315

.

Now run
kubectl run wordpress --image=tutum/wordpress --port=80

You will get a similar message:
deployment "wordpress" created

This command starts up the Docker image on one of the nodes we have in our cluster and we can see it using the
kubectl CLI:
kubectl get pods

You will see a ready-to-use pod:
NAME
wordpress-2410004867-dkc26

READY
1/1

STATUS
Running

RESTARTS
0

AGE
1m

Exposing Wordpress
By default a pod is only accessible to other machines inside the same cluster but in order to use the Wordpress
application, the service needs to be exposed. This will allow external traffic.
In order to expose the pod, we will use the same command that we have already used
time we are going to create a load balancer using --type=LoadBalancer flag.

kubectl expose <..>

but this

This flag will creates an external IP that the Wordpress pod can use to accept traffic.
kubectl expose pod wordpress-2410004867-dkc26 --name=wordpress

--type=LoadBalancer

You will need to replace the pod name wordpress-2410004867-dkc26 with the result you got from
pods

.

kubectl expose

creates

kubectl get

The service
The forwarding rules for the load balancer
The firewall rules that allow external traffic to be sent to the pod.

It may take some minutes to create the load balancer.
Note the value in the load Balancer Ingress field. This is how you can access your container from outside the cluster.
LoadBalancer Ingress:

104.198.26.134

Logging Into Our Cluster Machines
If you are a new GCP, you may wonder if we have access to the VMs of our cluster or not. The response is yes, we
can go the list of VMs:

select the browser mode and if you want to access your machine using your console click on view gcloud command
and it will shows something like this:
gcloud compute --project "wordpress-161315" ssh --zone "us-central1-b" "gke-my-test-node-default-pool-f295ca85-pvjc"

If not already generated before, the same command will generate a pair of public and private keys and a known hosts
file :
.ssh/google_compute_engine
.ssh/google_compute_engine.pub
.ssh/google_compute_known_hosts
You can now see the running containers if you type
CONTAINER ID
56082ad88e0d
ef0d27870b4c
3fe304b3ec19
f61268d8ca36
cf6d05737632
amd64
0df23fc8a6f5
2579f8dbedf4
6496a5d33b86
d33c866c969a
backend
5777cbc6bca6
eab12f0c765f
backend
75f8faef1369
logging
dc30769eb75f
autoscaler
13e0c16438a8
dashboard
5be92530ad6a
1e80866c0446
logging
80d7325ae045
proxy
c1f0995028cc
9421a6f0b4d8
proportiona"
0fa05812b004
dashboard

docker ps

IMAGE
tutum/wordpress
gcr.io/google_containers/pause-amd64
gcr.io/google_containers/addon-resizer
gcr.io/google_containers/heapster
"/pause"
gcr.io/google_containers/kubedns-amd64
gcr.io/google_containers/exechealthz-amd64
gcr.io/google_containers/pause-amd64
gcr.io/google_containers/defaultbackend
gcr.io/google_containers/kube-dnsmasq-amd64
gcr.io/google_containers/pause-amd64
gcr.io/google_containers/fluentd-gcp
gcr.io/google_containers/pause-amd64
gcr.io/google_containers/pause-amd64
gcr.io/google_containers/pause-amd64
gcr.io/google_containers/pause-amd64
gcr.io/google_containers/kube-proxy

:

COMMAND
NAMES
"/run.sh"
k8s_wordpress
"/pause"
k8s
"/pod_nanny --cpu=80m"
k8s_heapster-nanny
"/heapster --source=k"
k8s_heapster_heapster
gcr.io/google_containers/pausek8s_POD.d8dbe16c_heapster
"/kube-dns --domain=c"
k8s_kubedns_kube-dns
"/exechealthz '--cmd="
k8s_healthz_kube-dns
"/pause"
k8s_POD_kube-proxy
"/server"
k8s_default-http"/usr/sbin/dnsmasq --"
"/pause"
/bin/sh -c 'rm /lib/"
"/pause"
"/pause"
"/pause"
"/pause"
"/bin/sh -c 'kube-pro"

k8s_dnsmasq_kube-dns
k8s_POD_l7-defaultk8s_fluentd-cloudk8s_POD_kube-dnsk8s_POD_kubernetesk8s_POD_kube-dns
k8s_POD_fluentd-cloudk8s_kube-proxy_kube-

gcr.io/google_containers/dnsmasq-metrics-amd64
"/dnsmasq-metrics --v"
k8s_dnsmasq-metrics
gcr.io/google_containers/cluster-proportional-autoscaler-amd64
"/clusterk8s_autoscaler_kube-dns-autoscaler
gcr.io/google_containers/kubernetes-dashboard-amd64
"/dashboard --port=90"
k8s_kubernetes-

Using an HTTP Proxy to Access the Kubernetes API
In this section we are going to connect kubectl to Google Kubernetes using its remote API.

Go to the container engine web page and select connect:

Type the generated command, it should be similar to this one:
gcloud container clusters get-credentials my-test-node

--zone us-central1-b --project wordpress-161315

If you get this warning message:
WARNING: Accessing a Container Engine cluster requires the kubernetes commandline

Then you should type:
gcloud components install kubectl

If you get any failure during one of the above or below commands, you can use
Let's create a HTTP proxy to access the Kubernetes API

gcloud info --show-log

.

kubectl proxy

If you have a similar error to this one

error:

google:

could

not

find

https://developers.google.com/accounts/docs/application-default-credentials for more information.

default

credentials.

See

, you should authenticate to

Google Cloud services using:
gcloud auth application-default login

Then you should type again:
kubectl proxy

You can choose a specific port:
kubectl proxy --port=8081

The default port is 8080. You can now use your localhost to access the Kubernetes proxy UI. You will probably get
this error if you point your browser to the wrong address:
Unauthorized

These are the urls you can use:
http://127.0.0.1:8001/ui
http://localhost:8001/ui

Now you can use curl in order to interact with the API:
Get the API versions:
curl http://localhost:8080/api/

Get a list of pods:
curl http://localhost:8080/api/v1/namespaces/default/pods

Inspecting Services
You can see more details about the Wordpress we created using:
kubectl describe services wordpress

In the Cloud Shell console you will get a similar output to this:
Name:
Namespace:
Labels:
Selector:
Type:
IP:
Port:
NodePort:
Endpoints:
Session Affinity:
Events:
FirstSeen LastSeen
--------- -------8s
8s

wordpress
default
pod-template-hash=2410004867
run=wordpress
pod-template-hash=2410004867,run=wordpress
LoadBalancer
10.7.251.61
 80/TCP
 30795/TCP
10.4.0.9:80
None
Count
----1

From
SubObjectPath
---------------{service-controller }

Type
-------Normal

Reason
-----CreatingLoadBalancer

Message
------Creating load balancer

Inspecting Nodes
You can describe the different nodes you are using:
kubectl describe nodes

A similar output to this will appear right on your screen:
Name:
Role:
Labels:

Taints:
CreationTimestamp:
Phase:
Conditions:
Type
---NetworkUnavailable
OutOfDisk
MemoryPressure
DiskPressure
Ready

gke-my-test-node-default-pool-f295ca85-pvjc
beta.kubernetes.io/arch=amd64
beta.kubernetes.io/instance-type=n1-standard-1
beta.kubernetes.io/os=linux
cloud.google.com/gke-nodepool=default-pool
failure-domain.beta.kubernetes.io/region=us-central1
failure-domain.beta.kubernetes.io/zone=us-central1-b
kubernetes.io/hostname=gke-my-test-node-default-pool-f295ca85-pvjc

Sun, 12 Mar 2017 16:44:47 +0100

Status
-----False
False
False
False
True

LastHeartbeatTime
----------------Sun, 12 Mar
Sun, 12 Mar
Sun, 12 Mar
Sun, 12 Mar
Sun, 12 Mar

LastTransitionTime
-----------------Sun, 12 Mar
Sun, 12 Mar
Sun, 12 Mar
Sun, 12 Mar
Sun, 12 Mar

Reason
-----RouteCreated
KubeletHasSufficientDisk
KubeletHasSufficientMemory
KubeletHasNoDiskPressure
KubeletReady

Addresses:
10.128.0.2,104.198.50.198,gke-my-test-node-default-pool-f295ca85-pvjc
Capacity:
alpha.kubernetes.io/nvidia-gpu:
0
cpu:
1
memory:
3788484Ki
pods:
110
Allocatable:
alpha.kubernetes.io/nvidia-gpu:
0
cpu:
1
memory:
3788484Ki
pods:
110
System Info:
Machine ID:
833668693a8ae07719ffda0c786a12fc
System UUID:
83366869-3A8A-E077-19FF-DA0C786A12FC
Boot ID:
528cd889-61b2-476e-ac01-795c51d18cbb
Kernel Version:
4.4.21+
OS Image:
Container-Optimized OS from Google
Operating System:
linux
Architecture:
amd64
Container Runtime Version:
docker://1.11.2
Kubelet Version:
v1.5.3

Message
------RouteController created a route
kubelet has sufficient disk space available
kubelet has sufficient memory available
kubelet has no disk pressure
kubelet is posting ready status. AppArmor enabl

Kube-Proxy Version:
v1.5.3
PodCIDR:
10.4.0.0/24
ExternalID:
5258581285997695597
Non-terminated Pods:
(8 in total)
Namespace
Name
CPU Requests
----------------------default
wordpress
100m (10%)
kube-system
fluentd-cloud-logging
100m (10%)
kube-system
heapster138m (13%)
kube-system
kube-dns260m (26%)
kube-system
kube-dns-autoscale
20m (2%)
kube-system
kube-proxy-gke-my100m (10%)
kube-system
kubernetes-dashboa
100m (10%)
kube-system
l7-default-backend
10m (1%)

CPU Limits
---------0 (0%)
0 (0%)
138m (13%)
0 (0%)
0 (0%)
0 (0%)
100m (10%)
10m (1%)

Memory Requests Memory Limits
--------------- ------------0 (0%)
0 (0%)
200Mi (5%)
200Mi (5%)
301456Ki (7%)
301456Ki (7%)
140Mi (3%)
220Mi (5%)
10Mi (0%)
0 (0%)
0 (0%)
0 (0%)
50Mi (1%)
50Mi (1%)
20Mi (0%)
20Mi (0%)

Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.
CPU Requests CPU Limits
Memory Requests Memory Limits
------------ ------------------------ ------------828m (82%)
248m (24%)
731536Ki (19%) 803216Ki (21%)

Inspecting Namespaces
You can list the current namespaces in a cluster using:
kubectl get namespaces

The default namespaces on a fresh Kubernetes installation are:
NAME
default
kube-system

STATUS
Active
Active

AGE
1h
1h

default is default namespace for objects with no other namespaces
kube-system is the namespace for objects created by the Kubernetes system
You can set the namespace for all the kubectl commands using:
kubectl config set-context $(kubectl config current-context) --namespace=test-namespace

Change test-namespace by your preferred name then check if it was set using:
kubectl config view | grep namespace:

Note that we can find namespaces in the DNS entry created for a service (e.g. service-name.namespacename.svc.cluster.local).
..svc.cluster.local

In the case when a container just uses the service name , it will resolve to the service which is local to a
namespace which is useful to use the same configuration with multiple namespaces like dev, qa, staging,
production..
If you want to reach across namespaces, you need to use the fully qualified domain name (FQDN).

Viewing Kubernetes Configurations
In order to view your configurations, you need to type:
kubectl config view

And it will shows a description of your configuration.
e.g: The configuration of the Wordpress cluster we deployed should look like this:
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: REDACTED
server: https://35.184.192.231
name: gke_wordpress-161315_us-central1-b_my-test-node
contexts:
- context:
cluster: gke_wordpress-161315_us-central1-b_my-test-node
user: gke_wordpress-161315_us-central1-b_my-test-node
name: gke_wordpress-161315_us-central1-b_my-test-node
current-context: gke_wordpress-161315_us-central1-b_my-test-node
kind: Config
preferences: {}
users:
- name: gke_wordpress-161315_us-central1-b_my-test-node
user:
auth-provider:
config:
access-token: xxxxxxxxxxxxxxxxxxxxx
expiry: 2017-03-12T18:13:16.201747726+01:00
name: gcp

This configuration can be accessed using this file
ls ~/.kube/config

If you use multiple kubeconfig files at the same time and want to view merged config then you can still use the
kubectl config view . Say we have two configuration files:
~/.kube/config1
~/.kube/config2
The command will be :
KUBECONFIG=~/.kube/config1:~/.kube/config2 kubectl config view

Using Kubernetes With Amazon Web Services
In order to install and use Kubernetes in AWS, we are going to use a tool called kops (stands for Kubernetes
Operations). It allows to get a production grade Kubernetes cluster from the command line. Deployment is currently
supported on Amazon Web Services but more platforms are planned.

Installing Kops
In order to install Kops you will need a *nix system or OSx.
For OSx user, from Homebrew:
brew update && brew install kops

You can also use this method:
wget https://github.com/kubernetes/kops/releases/download/v1.4.1/kops-darwin-amd64
chmod +x kops-darwin-amd64
mv kops-darwin-amd64 /usr/local/bin/kops

And Linux users should download the binary:
chmod +x kops-linux-amd64
mv kops-linux-amd64 /usr/local/bin/kops

You can install from sources:
go get -d k8s.io/kops
cd ${GOPATH}/src/k8s.io/kops/
git checkout release
make

Windows users need to install a Linux VM or Vagrant.

Prerequisites
We need to setup a user, using the IAM we can create a new user, remember to save its access id/key and give it the
administrator access. You can create your own policy by giving the user only the rights that he needs. We will
proceed for this course with the full administrator access.

After that we need to setup the DNS. kops uses DNS for discovery inside the cluster and for the clients so that we
can reach the kubernetes API server. The cluster name should be a valid DNS name. This allows us to share our
cluster with its domain. We are going to use Route53 with a subdomain. You should either register a domain or
transfer an existing domain name to Route53 (if you would like to centralize all your domain operations in AWS
Route53).
If you don't want to spend money, you can register a free domain name like .tk.
Note that if you have a registered domain with another registrar (Namecheap, GoDaddy ..etc) and you want to keep
it instead of transferring it, you should setup the generated hosted zone values (NS records) in your original registrar
configuration.
Now create a hosted zone for your domain. You can use the AWS web interface or the AWS CLI. In my case, I
already hosted the domain devopslinks.com in Route53 and I will be using kubernetes.dev.devopslinks.com for the
remainder:
aws route53 create-hosted-zone --name kubernetes.dev.devopslinks.com --caller-reference 1

You should have a similar output to this:

{
"ChangeInfo": {
"Status": "PENDING",
"SubmittedAt": "2017-04-30T21:10:40.362Z",
"Id": "/change/xxxxxxxxxx"
},
"DelegationSet": {
"NameServers": [
"ns-1752.awsdns-27.co.uk",
"ns-1333.awsdns-38.org",
"ns-904.awsdns-49.net",
"ns-102.awsdns-12.com"
]
},
"HostedZone": {
"Name": "kubernetes.dev.devopslinks.com.",
"CallerReference": "1",
"Id": "/hostedzone/xxxxxxxxx",
"Config": {
"PrivateZone": false
},
"ResourceRecordSetCount": 2
},
"Location": "https://route53.amazonaws.com/2013-04-01/hostedzone/Z2AZ80TW24KT5C"
}

Creating a hosted zone is a paid service. You can find more details about Route53 here.
Note the 4 generated name servers :
ns-1752.awsdns-27.co.uk.
ns-1333.awsdns-38.org.
ns-904.awsdns-49.net.
ns-102.awsdns-12.com.
If you have your domain registered elsewhere, say GoDaddy, you should go to GoDaddy web interface, create 4 NS
records with the host kubernetes.dev. and set the one of the name servers to each of the created records.
We also need an S3 bucket.
To manage a cluster kops must keep track of the created clusters, along with their configuration, the keys they are
using and different other configurations. This information is stored in a bucket.
Create the bucket (we will call it kubernetes.dev.devopslinks.com):
aws s3 mb s3://kubernetes.dev.devopslinks.com

At this step, you should have setup the right user, S3 and Route53. We can proceed by typing this command:
kops create cluster \
--name=kubernetes.dev.devopslinks.com \
--state=s3://kubernetes.dev.devopslinks.com \
--zones=eu-west-1a \
--node-count=2 \
--node-size=t2.micro \
--master-size=t2.micro \
--dns-zone=kubernetes.dev.devopslinks.com

You should change adapt this commands to your preferences like the region, the size of the machines, the number of
the machines in a cluster ..etc This command will create the following resources:
EBSVolume
DHCPOptions
Keypair
SSHKey
EBSVolume
VPC

IAMRole
IAMInstanceProfile
SecurityGroup
Subnet
IAMRolePolicy
InternetGateway
SecurityGroupRule
SecurityGroupRule
Route
RouteTableAssociation
AutoscalingGroup

As you can see in the output of the last command:
To list clusters: kops get cluster
To edit this cluster kops edit cluster kubernetes.dev.devopslinks.com
To edit node instance group: kops edit ig --name=kubernetes.dev.devopslinks.com nodes
To edit the master instance group: kops edit ig --name=kubernetes.dev.devopslinks.com

master-eu-west-1a

Now we can configure the cluster with:
kops update cluster kubernetes.dev.devopslinks.com --yes

Don't forget to add

--state=s3://kubernetes.dev.devopslinks.com

--state=s3://kubernetes.dev.devopslinks.com

After launching this command, you can find important configurations in

~/.kube/config

file.

Cluster is starting. It should be ready in a few minutes.
As you may see in the output of the last command:
To list nodes: kubectl get nodes --show-labels
To ssh to the master: ssh -i ~/.ssh/id_rsa admin@api.kubernetes.dev.devopslinks.com
To read about installing addons: https://github.com/kubernetes/kops/blob/master/docs/addons.md

Chapter XIV - Orchestration - Rancher/Cattle
o
o

^__^
(oo)\_______
(__)\
)\/\
||----w |
||
||

In the Docker World, the orchestration is the most important part of the ecosystem. Docker Swarm, Kubernetes,
Apache Mesos .. all of these are orchestrators, every one of them has its own philosophy, use cases and architecture.
Rancher is a tool built to simplify Docker orchestration and management. Through this tutorial,we are going to
discover how to use it in order to create a scalable Wordpress application.

Rancher Architecture
In Rancher, everything (like containers, networks or images ) is an API resource with a process lifecycle.
Containers, images, networks, and accounts are all API resources with their own process lifecycles.
Rancher is built on the top of containers:
A web UI
An API
A server that manage Rancher Agents
A database
A machine microservice
The docker-machine binary

When you run Rancher using docker run rancher/server ... the Rancher API + Rancher Process Server + The
Database + Machine Microservice are processes that live inside this container.
Note that the docker-machine binary is also living in the same container but only runs when it is called by the API.

Rancher has also an Agent part that manage the life cycle of containers.
If docker-machine creates a machine successfully, some events are exchanged between the docker-machine and the
microservice. A bootstrap event is created and a docker-machine config command is executed to get the details needed
to connect to the machine’s Docker daemon.
If everything run without problems, the service fires up a Rancher Agent on the machine via
rancher/agent ... ```.

`̀

docker run

Rancher Agents open a WebSocket connection to the server in order to establish a 2-way communication. The
Rancher Agent manage its containers and reports every change using the Docker API.
During this tutorial, we are going to use an EC2 machine and this is how a different view of the layers of our
Rancher installation:

RancherOS
RancherOS is a small distribution released by Rancher team. It is an easy way to run containers at scale in
production, and includes only the services needed to run Docker.
It only includes the latest version of Docker and removes any unneeded library that a "normal" Linux distribution
could have.
In RancherOS, everything is a container, the traditional init system is replaced so that Docker run directly on the
Kernel.
A special component in this system is called User Docker which is the daemon that allow a user (a non-system user)
runs its containers.

We are going to run RancherOS in an EC2 machine using AWS CLI:
aws ec2 run-instances --image-id ami-ID --count 1 --instance-type t2.micro --key-name MySSHKeyName --security-groups sg-name

This is the list of AMI by region:
Region
ap-south-1
eu-west-2
eu-west-1
ap-northeast-2
ap-northeast-1
sa-east-1
ca-central-1
ap-southeast-1
ap-southeast-2
eu-central-1
us-east-1
us-east-2
us-west-1
us-west-2

Type
HVM
HVM
HVM
HVM
HVM
HVM
HVM
HVM
HVM
HVM
HVM
HVM
HVM
HVM

AMI
ami-fd1e6d92
ami-51776335
ami-481e232e
ami-c32efdad
ami-33aaf154
ami-15ed8d79
ami-e61fa282
ami-63b50900
ami-86b7bbe5
ami-a71ecfc8
ami-37b00f21
ami-c61632a3
ami-8998c3e9
ami-f6910496

Now you can login to your machine using the AWS common ssh command :
ssh -i "MySSHKeyName" rancher@xxxx.xx-xxxx-x.compute.amazonaws.com

Running Rancher
Since Docker is installed by default, we can start using it directly. Let's start a MariaDB server then use it with
Rancher Server.
docker
run
--name
mariadb
-e
MYSQL_ROOT_PASSWORD=password
e MYSQL_PASSWORD=cattle -p 3306:3306 -d mariadb

-e

MYSQL_DATABASE=cattle

-e

MYSQL_USER=cattle

-

docker
run
--name
rancher_server
-p
8080:8080
-e
CATTLE_DB_CATTLE_MYSQL_HOST=172.31.0.190
e
CATTLE_DB_CATTLE_MYSQL_PORT=3306
-e
CATTLE_DB_CATTLE_MYSQL_NAME=cattle
-e
CATTLE_DB_CATTLE_USERNAME=cattle
e CATTLE_DB_CATTLE_PASSWORD=cattle -v /var/run/docker.sock:/var/run/docker.sock -d rancher/server

-

Run the Rancher Server and change 172.31.0.190 by your MariaDB IP:

Now you can go to your

:8080

and add a Linux host with a supported version of Docker:

We are not going to use the public IP address but the private (eth0) one.

Let’s add a custom host.
A command is given to run on any reachable host in order to let it join the server. At this step, I already created an
EC2 machines (with RancherOS as an operating system) and I am going to use it and run this command:

sudo
docker
run
-d
--privileged
-v
/var/run/docker.sock:/var/run/docker.sock
v /var/lib/rancher:/var/lib/rancher rancher/agent:v1.2.1 http://172.31.0.190:8080/v1/scripts/02048221239BE78134BB:1483142400000:i5yrQRE4kFo7eIKz9

Make sure any Security Groups or firewalls allow traffic from and to all other hosts on UDP ports 500 and 4500.
After the Agent installation, we can notice that some containers are running in each Agent machine like the dns,
healthcheck, network-manager ..:
CONTAINER ID
8339a8af3bb8
95f3d3afdb7c
5643089ca485
fec564c53f13
c162c07d07f6
0313ff5cb812

IMAGE
rancher/net
rancher/net:holder
rancher/healthcheck
rancher/dns
rancher/net
rancher/scheduler

COMMAND
"/rancher-entrypoint."
"/.r/r /rancher-entry"
"/.r/r /rancher-entry"
"/rancher-entrypoint."
"/rancher-entrypoint."
"/.r/r /rancher-entry"

NAMES
r-ipsec-ipsec-router
r-ipsec-ipsec
r-healthcheck-healthcheck
r-network-services-metadata-dns
r-ipsec-ipsec-cni-driver
r-scheduler-scheduler

0f6f62a190c7
ec270864ff07
98a78058bc3c

rancher/network-manager"/rancher-entrypoint."
rancher/metadata
"/rancher-entrypoint."
rancher/agent
"/run.sh run"

r-network-services-network-manager
r-network-services-metadata
rancher-agent

Running A Wordpress Service
If everything is ok, we are going to create a Wordpress service. Go to Stack menu and click User.
Create the MariaDB database, name it wordpressdb (we are going to use it in order to link it to the Wordpress
container)

Then add your environment variables:

Now, using the same way, create the Wordpress container, map host port 80 to the container port 80 and link the
mariadb container as a mysql instance.

Then add your environment variables:

At this step, we can check the running services, use Stack->User then default to see the containers:

What we started is the equivalent of 2 docker commands, one to start the mariadb container and the other one to
start the wordpress container.
You can visit the Wordpress fresh installation using its IP address.
We can see the details of the different configurations a running container could have if you choose the container
from the same view then use the "View in API" menu.
This is an example of the wordpress container:
{
"id":"1s7",
"type":"service",
"links":{
"self":"…/v2-beta/projects/1a5/services/1s7",
"account":"…/v2-beta/projects/1a5/services/1s7/account",
"consumedbyservices":"…/v2-beta/projects/1a5/services/1s7/consumedbyservices",
"consumedservices":"…/v2-beta/projects/1a5/services/1s7/consumedservices",
"instances":"…/v2-beta/projects/1a5/services/1s7/instances",
"networkDrivers":"…/v2-beta/projects/1a5/services/1s7/networkdrivers",
"serviceExposeMaps":"…/v2-beta/projects/1a5/services/1s7/serviceexposemaps",
"serviceLogs":"…/v2-beta/projects/1a5/services/1s7/servicelogs",
"stack":"…/v2-beta/projects/1a5/services/1s7/stack",

"storageDrivers":"…/v2-beta/projects/1a5/services/1s7/storagedrivers",
"containerStats":"…/v2-beta/projects/1a5/services/1s7/containerstats"
},
"actions":{
"upgrade":"…/v2-beta/projects/1a5/services/1s7/?action=upgrade",
"restart":"…/v2-beta/projects/1a5/services/1s7/?action=restart",
"update":"…/v2-beta/projects/1a5/services/1s7/?action=update",
"remove":"…/v2-beta/projects/1a5/services/1s7/?action=remove",
"deactivate":"…/v2-beta/projects/1a5/services/1s7/?action=deactivate",
"removeservicelink":"…/v2-beta/projects/1a5/services/1s7/?action=removeservicelink",
"addservicelink":"…/v2-beta/projects/1a5/services/1s7/?action=addservicelink",
"setservicelinks":"…/v2-beta/projects/1a5/services/1s7/?action=setservicelinks"
},
"baseType":"service",
"name":"wordpress",
"state":"active",
"accountId":"1a5",
"assignServiceIpAddress":false,
"createIndex":1,
"created":"2017-04-04T21:05:53Z",
"createdTS":1491339953000,
"currentScale":1,
"description":"wordpress files",
"externalId":null,
"fqdn":null,
"healthState":"healthy",
"instanceIds":[
"1i14"
],
"kind":"service",
"launchConfig":{
"type":"launchConfig",
"capAdd":[
],
"capDrop":[
],
"dataVolumes":[
],
"dataVolumesFrom":[
],
"devices":[
],
"dns":[
],
"dnsSearch":[
],
"environment":{
"WORDPRESS_DB_NAME":"wordpress",
"WORDPRESS_DB_USER":"wordpress",
"WORDPRESS_DB_PASSWORD":"wordpress"
},
"imageUuid":"docker:wordpress",
"instanceTriggeredStop":"stop",
"kind":"container",
"labels":{
"io.rancher.container.pull_image":"always"
},
"logConfig":{
"type":"logConfig",
"config":{
},
"driver":""
},
"networkMode":"managed",
"ports":[
"80:80/tcp"
],
"privileged":false,
"publishAllPorts":false,
"readOnly":false,
"secrets":[
],
"startOnCreate":true,
"stdinOpen":true,
"system":false,
"tty":true,
"version":"0",
"dataVolumesFromLaunchConfigs":[
],
"vcpu":1
},
"lbConfig":null,

"linkedServices":{
"mysql":"1s6"
},
"metadata":null,
"publicEndpoints":[
{
"type":"publicEndpoint",
"hostId":"1h1",
"instanceId":"1i14",
"ipAddress":"54.246.163.197",
"port":80,
"serviceId":"1s7"
}
],
"removed":null,
"retainIp":null,
"scale":1,
"scalePolicy":null,
"secondaryLaunchConfigs":[
],
"selectorContainer":null,
"selectorLink":null,
"stackId":"1st5",
"startOnCreate":true,
"system":false,
"transitioning":"no",
"transitioningMessage":null,
"transitioningProgress":null,
"upgrade":null,
"uuid":"0b8a8290-0d10-47be-b182-45f1e57e80bc",
"vip":null
}

We can also inspect other services and hosts configurations in the same way.

Cattle: Rancher Container Orchestrator
What we started below is Rancher powered by its own orchestration tool called Cattle. Rancher offers the possibility
to use other orchestration tools like Kubernetes, Docker Swarm or Mesos.
Cattle is a container orchestration and scheduling framework, in its beginning, it was designed as an extension to
Docker Swarm but since Docker Swarm continues to develop, Cattle and Swarm started to diverge. Cattle is used
extensively by Rancher itself to orchestrate infrastructure services as well as setting up, managing, and upgrading
Swarm, Kubernetes, and Mesos clusters.
Cattle application deployments are organized into stacks which can be used to User or Infrastructure stacks. A
Cattle Stack is a collection of Services and the latter is primarily a Docker image with its networking, scalability,
storage, health checks, service discovery links, environment and all of the other confgurations. A Cattle Service
could be a load balancer or an external service. A Stack can be launched using a docker-compose.yml or ranchercompose.yml file or just start containers like we did for the Wordpress stack.
In addition to this, you can start several applications using an application catalog. More than 50 different apps are
available in the app catalog.
An application is defined by a docker-compose and a rancher-compose file and can be deployed easily with the
default configurations.
Let's take the example of Portainer:
Portainer is a lightweight management UI which allows you to easily manage your Docker host or Swarm
cluster. Portainer is meant to be as simple to deploy as it is to use. It consists of a single container that can run
on any Docker engine (Docker for Linux and Docker for Windows are supported). Portainer allows you to
manage your Docker containers, images, volumes, networks and more ! It is compatible with the standalone
Docker engine and with Docker Swarm.
This is the

docker-compose.yml

file of Portainer:

portainer:
labels:
io.rancher.sidekicks: ui
io.rancher.container.create_agent: true
io.rancher.container.agent.role: environment
image: rancher/portainer-agent:v0.1.0
volumes:
- /config
ui:
image: portainer/portainer:pr572
command: --no-auth --external-endpoints=/config/config.json --sync-interval=5s -p :80
volumes_from:
- portainer
net: container:portainer

This is its

rancher-compose.yml

file:

.catalog:
name: "Portainer"
version: "1.11.4"
description: Open-source lightweight management UI for a Docker host or Swarm cluster
minimum_rancher_version: v1.5.0-rc1

Rancher offers the possibility to start infrastructure Stacks. If you go to Stacks->Infrastructure, you can see a catalog
of tools like Bind9, Portainer, Rancher vxlan ..etc

Now we want to scale our Wordpress frontal app in order to handle more traffic. In this case, you should click Stack>User then the name of the Stack (it should be the default one if you followed this tutorial as it is). Now click on
Wordpress and scale your app to 3 containers:

At this step, you should see how Wordpress won't scale the right way.

Scaling Wordpress Using Rancher
Before starting this, we should know that scaling a container that already mapped to a host port is impossible, we
have seen this with the latest failed example of Wordpress. To make this work, we should remove the port mapping,
go to the Wordpress service and remove port mapping using the UI. Now let's create a load balancer service. Go to
Stacks->User->Default and add a load balancer:

We know that the Wordpress container will run the service wordpress. So we should choose it as a target, let's also
give the target a port (which is the default port of a wordpress service): We are not using port mapping but
wordpress service listens always on port 80 - this is the same thing as in Docker Swarm. Choose also the request
host, port, path ..etc.
In our case, the Wordpress Stack is running in a single Rancher host, since we added just one. You can add another
host and the same load balacing mechanism will be reproduced on two hosts: the wordpress service containers will
be available on both hosts and the load balancer will auto-discover Wordpress containers in all of Rancher hosts.

Chapter XV - Docker API
o
o

^__^
(oo)\_______
(__)\
)\/\
||----w |
||
||

Exploring Docker API
Docker API is the API served by Docker Engine and it allows to have a full control on Docker. This could be
interesting if you want to build application that use Docker easily. You can see the Engine API version you are
running by typing sudo docker version |grep -i api
The API is usually changed in each release of Docker, so API calls are versioned to ensure that clients don't break.
In order to interact with theis API you should use on of the known SDKs and this will depend on the language you
are using. Here is a list of the know SDKs to use Docker Engine API:
Language
C
C#
C++
Dart
Erlang
Gradle
Groovy
Haskell
HTML (Web Components)
Java
Java
NodeJS
Perl
PHP
Ruby
Rust
Rust
Scala
Scala

Library
libdocker
Docker.DotNet
lasote/docker_client
bwu_docker
erldocker
gradle-docker-plugin
docker-client
docker-hs
docker-elements
docker-client
docker-java
dockerode
Eixo::Docker
Docker-PHP
docker-api
docker-rust
shiplift
tugboat
reactive-docker

The following go code creates a container running Alpine as an OS that will print "hello world" then exits:
package main
import (
"io"
"os"
"github.com/moby/moby/client"
"github.com/moby/moby/api/types"
"github.com/moby/moby/api/types/container"
"golang.org/x/net/context"
)
func main() {
ctx := context.Background()
cli, err := client.NewEnvClient()
if err != nil {
panic(err)
}
_, err = cli.ImagePull(ctx, "docker.io/library/alpine", types.ImagePullOptions{})
if err != nil {
panic(err)
}

resp, err := cli.ContainerCreate(ctx, &container.Config{
Image: "alpine",
Cmd:
[]string{"echo", "hello world"},
}, nil, nil, "")
if err != nil {
panic(err)
}
if err := cli.ContainerStart(ctx, resp.ID, types.ContainerStartOptions{}); err != nil {
panic(err)
}
if _, err = cli.ContainerWait(ctx, resp.ID); err != nil {
panic(err)
}
out, err := cli.ContainerLogs(ctx, resp.ID, types.ContainerLogsOptions{ShowStdout: true})
if err != nil {
panic(err)
}
io.Copy(os.Stdout, out)
}

To do the same thing with Python:
import docker
client = docker.from_env()
print client.containers.run("alpine", ["echo", "hello", "world"])

Docker has an official documentation if you want to use the API.
e.g. To list containers, we can send a GET request to
"everything is 200 ok", the API will return a JSON:

/containers/json

, and the reponse will be either 400, 500 and if

[
{
"Id": "8dfafdbc3a40",
"Names": [
"/boring_feynman"
],
"Image": "ubuntu:latest",
"ImageID": "d74508fb6632491cea586a1fd7d748dfc5274cd6fdfedee309ecdcbc2bf5cb82",
"Command": "echo 1",
"Created": 1367854155,
"State": "Exited",
"Status": "Exit 0",
"Ports": [
{
"PrivatePort": 2222,
"PublicPort": 3333,
"Type": "tcp"
}
],
"Labels": {
"com.example.vendor": "Acme",
"com.example.license": "GPL",
"com.example.version": "1.0"
},
"SizeRw": 12288,
"SizeRootFs": 0,
"HostConfig": {
"NetworkMode": "default"
},
"NetworkSettings": {
"Networks": {
"bridge": {
"NetworkID": "7ea29fc1412292a2d7bba362f9253545fecdfa8ce9a6e37dd10ba8bee7129812",
"EndpointID": "2cdc4edb1ded3631c81f57966563e5c8525b81121bb3706a9a9a3ae102711f3f",
"Gateway": "172.17.0.1",
"IPAddress": "172.17.0.2",
"IPPrefixLen": 16,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": "02:42:ac:11:00:02"
}
}
},
"Mounts": [
{
"Name": "fac362...80535",

"Source": "/data",
"Destination": "/data",
"Driver": "local",
"Mode": "ro,Z",
"RW": false,
"Propagation": ""
}
]
},
{
"Id": "9cd87474be90",
"Names": [
"/coolName"
],
"Image": "ubuntu:latest",
"ImageID": "d74508fb6632491cea586a1fd7d748dfc5274cd6fdfedee309ecdcbc2bf5cb82",
"Command": "echo 222222",
"Created": 1367854155,
"State": "Exited",
"Status": "Exit 0",
"Ports": [],
"Labels": {},
"SizeRw": 12288,
"SizeRootFs": 0,
"HostConfig": {
"NetworkMode": "default"
},
"NetworkSettings": {
"Networks": {
"bridge": {
"NetworkID": "7ea29fc1412292a2d7bba362f9253545fecdfa8ce9a6e37dd10ba8bee7129812",
"EndpointID": "88eaed7b37b38c2a3f0c4bc796494fdf51b270c2d22656412a2ca5d559a64d7a",
"Gateway": "172.17.0.1",
"IPAddress": "172.17.0.8",
"IPPrefixLen": 16,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": "02:42:ac:11:00:08"
}
}
},
"Mounts": []
},
{
"Id": "3176a2479c92",
"Names": [
"/sleepy_dog"
],
"Image": "ubuntu:latest",
"ImageID": "d74508fb6632491cea586a1fd7d748dfc5274cd6fdfedee309ecdcbc2bf5cb82",
"Command": "echo 3333333333333333",
"Created": 1367854154,
"State": "Exited",
"Status": "Exit 0",
"Ports": [],
"Labels": {},
"SizeRw": 12288,
"SizeRootFs": 0,
"HostConfig": {
"NetworkMode": "default"
},
"NetworkSettings": {
"Networks": {
"bridge": {
"NetworkID": "7ea29fc1412292a2d7bba362f9253545fecdfa8ce9a6e37dd10ba8bee7129812",
"EndpointID": "8b27c041c30326d59cd6e6f510d4f8d1d570a228466f956edf7815508f78e30d",
"Gateway": "172.17.0.1",
"IPAddress": "172.17.0.6",
"IPPrefixLen": 16,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": "02:42:ac:11:00:06"
}
}
},
"Mounts": []
},
{
"Id": "4cb07b47f9fb",
"Names": [
"/running_cat"
],
"Image": "ubuntu:latest",
"ImageID": "d74508fb6632491cea586a1fd7d748dfc5274cd6fdfedee309ecdcbc2bf5cb82",
"Command": "echo 444444444444444444444444444444444",
"Created": 1367854152,
"State": "Exited",
"Status": "Exit 0",
"Ports": [],
"Labels": {},

"SizeRw": 12288,
"SizeRootFs": 0,
"HostConfig": {
"NetworkMode": "default"
},
"NetworkSettings": {
"Networks": {
"bridge": {
"NetworkID": "7ea29fc1412292a2d7bba362f9253545fecdfa8ce9a6e37dd10ba8bee7129812",
"EndpointID": "d91c7b2f0644403d7ef3095985ea0e2370325cd2332ff3a3225c4247328e66e9",
"Gateway": "172.17.0.1",
"IPAddress": "172.17.0.5",
"IPPrefixLen": 16,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": "02:42:ac:11:00:05"
}
}
},
"Mounts": []
}
]

Streaming Containers Logs Using Docker API
Like in all of Painless Docker chapters, our goal is not providing a documentation about Docker or its API, but
giving more practical examples. You can find all of the details about the API in the official documentations. Let's
move to more practical stuff.
In this part of the book we are going to create a central logging system, that will collect every line of log in a given
server and stream it to a web page. Think of it as the docker logs  .
We are going to use Docker API with Python Flask.
Flask is a microframework for Python based on Werkzeug and Jinja 2. Python and Flask is easy to read and
understand, you don't need to be a Python expert to use follow these steps.
For my Flask projects, I usually use a code template that you can find on Github. You need first to install these
packages:
git
python3
python-pip3
python-virtualenv
Now create an isolated Python virtual environment
Go to the created directory

cd docker-log-stream

Activate the virtual environment:

virtualenv -p python3 docker-log-stream

.

. bin/activate

Download the code template from Github:

git clone https://github.com/eon01/flasklate.git code

This is the app.py file that we are going to use in order to start our server on port 5000:
#!/usr/bin/env python
# -*- coding: utf-8 -*import logging, traceback, configparser, os
from flask import Flask
app = Flask(__name__)

# start configuration parser
parser = configparser.ConfigParser()
parser.read("app.conf")
# reading variables
logger_level = parser.get('logging', 'logger_level', raw = True)
handler_level = parser.get('logging', 'handler_level', raw = True)
log_format = parser.get('logging', 'log_format', raw = True)
log_file = parser.get('logging', 'log_file')
# set logger logging level
logger = logging.getLogger(__name__)
logger.setLevel(eval(logger_level))
# set handler logging level
handler = logging.FileHandler(log_file)
handler.setLevel(eval(handler_level))
# create a logging format
formatter = logging.Formatter(log_format)
handler.setFormatter(formatter)
# add the handlers to the logger
logger.addHandler(handler)

@app.route('/')
def hello_world():

.

return 'Hello, World!'

if __name__ == '__main__':
# Bind to PORT if defined, otherwise default to 5000.
port = int(os.environ.get('PORT', 5000))
app.run(host='0.0.0.0', port=port)

This file will read configuration variables from app.conf.
In order to test this, we need to install the requirements (dependencies): pip install -r requirements.txt Now we can
start our local web server using python app.py and on another terminal, you can test it using a simple curl: curl
http://0.0.0.0:5000 . If everything is fine, you will get 'Hello, world!' on your screen.
Now we need to install the Docker API for Python, just type
installed in your local development environment.

and hit enter. Docker will be

pip install docker

In general, we can use Docker for Python this way:
We create a client :

client = docker.from_env()

and then we can access to a list of methods in the created client:

['__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattr__', '__getattribute__', '__gt__', '__

We are more interested in :

'api', 'containers', 'df', 'events', 'from_env', 'images', 'info', 'login', 'networks', 'nodes',

'ping', 'plugins', 'secrets', 'services', 'swarm', 'version', 'volumes'

methods.

As you may see we can get a list of running containers in our host using
In order to go trough all of these containers one by one, we can use:

containers

:

containers = client.containers()

.

for container in containers:
# we can execute what we want here for each container

I order to get a container logs, we can use this:
for container in containers:
for line in container.logs(stream=True):
print (line)

We can do a better code by stripping the printed line:
for container in containers:
for line in container.logs(stream=True):
print (line.strip())

Since we can access a container name using

container.name

we can enhance our program:

for container in containers:
for line in container.logs(stream=True):
log_line = container.name + " : " + line.strip().decode("utf-8")
print (log_line)

In the original code file, change the
stream the log files:

hello_world

#!/usr/bin/env python
# -*- coding: utf-8 -*import logging, traceback, configparser, os

function by a new one that we could call

stream

and add the code to

from flask import Flask
import docker
app = Flask(__name__)

# start configuration parser
parser = configparser.ConfigParser()
parser.read("app.conf")
# reading variables
logger_level = parser.get('logging', 'logger_level', raw = True)
handler_level = parser.get('logging', 'handler_level', raw = True)
log_format = parser.get('logging', 'log_format', raw = True)
log_file = parser.get('logging', 'log_file')
# set logger logging level
logger = logging.getLogger(__name__)
logger.setLevel(eval(logger_level))
# set handler logging level
handler = logging.FileHandler(log_file)
handler.setLevel(eval(handler_level))
# create a logging format
formatter = logging.Formatter(log_format)
handler.setFormatter(formatter)
# add the handlers to the logger
logger.addHandler(handler)
@app.route('/')
def stream():
client = docker.from_env()
containers = client.containers.list()
for container in containers:
for line in container.logs(stream=True):
log_line = container.name + ":" + line.strip().decode("utf-8")
print (log_line)

if __name__ == '__main__':
# Bind to PORT if defined, otherwise default to 5000.
port = int(os.environ.get('PORT', 5000))
app.run(host='0.0.0.0', port=port)

Now add Docker API to the requirements file:

pip freeze > requirements.txt

In order to test this I used a Wordpress stack that I deployed to my machine using this docker-compose file:
version: '3.1'
services:
wordpress:
image: wordpress
ports:
- 8080:80
environment:
WORDPRESS_DB_PASSWORD: passw0rd
mysql:
image: mysql:5.7
environment:
MYSQL_ROOT_PASSWORD: passw0rd

You can start Wordpress docker-compose -f docker-compose.yml up , then start refreshing the index page at:
http://0.0.0.0:8080 . At the same time, after running the Python program using a simple python app.py command, you
should send a GET request to your Python program curl http://0.0.0.0:5000/ .

Chapter XVI - Docker Security
o
o

^__^
(oo)\_______
(__)\
)\/\
||----w |
||
||

Possible Threats
Docker is neither more nor less secured than VMs and the VM vs Docker security discussions are the answers to a
wrong question. Usually security threats are either PEBCAK (Problem Exists Between Chair And Keyboard) or
problems that have existed before.
Discussing the security best practices and checklists is more constructive and productive.
In this chapter, we are going to see what are the know security threats that you should be aware of when using
Docker and introduce good security practices. I am not a security researcher so all of the following parts are based
on my opinions, some researches and experiences.

Kernel Panic & Exploits
As we have seen in the first chapters of this book, one of the differences between Docker and VMs is that a container
share the Kernel with the host system and with the other running containers. If a container causes a Kernel panic,
both hosts and other containers could be taken down. This "direct access" to the Kernel, which is at the same time
one of the strengths of Docker, could cause serious damages.

Container Breakouts & Privilege Escalation
If you start the container X with the user Y, container X will have the same privileges on the host system as the user
Y. This is harmful when a process breaks out the container. In this case, if you were root in the container, you will be
the same on the host.
Container breakout will cause unauthorized access across containers, hosts and even your data centers.

Poisoned Images
It is possible that download and use a Docker image that run malwares (e.g. scanning the network for sensitive data,
downloading malware from a distant host, executing harmful actions .. etc). An attacker can also get access to your
data if you are using his poisoned image.

Denial-of-service Attacks
Like you know, containers share the same Kernel with each other and with the host and in this case any container
monopolizing the Kernel resources will make other containers starve out. Poisoned containers can eat up CPU time,
memory resources, disk IO.. etc and this could lead other containers and even the host system to a crash.

Compromising secrets
A Docker container could contain or send sensitive data. During transit and at rest, the secret data could be viewed
and stolen by an attacker. Microservices architecture have the same security threat but there are always good
solutions for these threats.

Application Level Threats

A container is primarily made to build, ship and run an application and even if the host and the container are
secured, the application layer could open some doors to attackers. e.g.
Flooding the network
Application level DDOS
XSS vulnerabilities and SQL injection
DNS hijacking

Host System Level Treats
If the host system is not up to date, you may have some security threats like: heart-bleed, shell-shock (Bashdoor),
glibc ..etc

Security Best Practices
Security By Design
Like for any other security threat, a system that has been designed from the ground up to be secure is always the best
and first practice. The application layer, the containerization layer, the host system, the software and the cloud
architecture ..etc are all elements of the same running stack and each of them could be a weak link. A system secure
by design will work to limit the damage when a container breakout occurs.

SetUID/SetGID
setuid/setgid binaries run with the privileges of the owner and can sometimes be compromised by an attacker to gain
elevated privileges. To remove the setuid bit, add this line in the Dockerfile:
RUN find / -perm +6000 -type f -exec chmod a-s {} \; || true

Without the
find:
find:
find:
find:

|| true

, we can have errors like:

`/proc/18/task/18/fd/5': No such file or directory
`/proc/18/task/18/fdinfo/5': No such file or directory
`/proc/18/fd/5': No such file or directory
`/proc/18/fdinfo/5': No such file or directory

This is a prevention from some privilege escalation threats and it will applied to files like:
/sbin/unix_chkpwd
/usr/bin/chage
/usr/bin/passwd
/usr/bin/mail-touchlock
/usr/bin/mail-unlock
/usr/bin/gpasswd
/usr/bin/crontab
/usr/bin/chfn
/usr/bin/newgrp
/usr/bin/sudo
/usr/bin/wall
/usr/bin/mail-lock
/usr/bin/expiry
/usr/bin/dotlockfile
/usr/bin/chsh
/usr/lib/eject/dmcrypt-get-device
/usr/lib/pt_chown
/bin/ping6
/bin/su
/bin/ping
/bin/umount
/bin/mount

Controlling CPU Usage
An attacker that gains access to a container could make DDoS attacks on the CPU that make other containers starve
out. By default, containers get an equal number of CPU cycles and quotas. This could be modified by changing the
container's CPU period and quota values.
docker run -d --cpu-period 50000 --cpu-quota 5000 ubuntu:16.04

The default CPU CFS (Completely Fair Scheduler) period is 100ms.
The

--cpu-period

The default

and

--cpu-quota

--cpu-quota

flags limit the container's CPU usage.

value is 0 and allows the container to take 100% of a CPU resource for 1 CPU.

The CFS (Completely Fair Scheduler), the default Linux Scheduler used by the kernel handles resource allocation
for executing processes.
When this value is set to 50000 the container is limited to 50% of a CPU resource.
Adjusting --cpu-period and
multiple CPUs context.

--cpu-quota

give the container administrator the control for container's CPU usage in a

Controlling Memory Usage
With the default setting, container can use 100% of the memory on the host. Choosing a limit for a container is a
security best practice in order to limit risks in the case of a container breakout.
Limiting the max amount of memory a container can use is done using the

-m

flag. e.g.

docker run -m 512m

ubuntu:16.04

Verifying Images
Be careful from downloading 3rd party images. Only use images from automated builds with linked source code or
use official images by pulling the digest to take advantage of checksum validate.

Set Container Filesystem to Read Only
Unless you need to modify files in your container, make the filesystem read only.
docker run --read-only ubuntu:16.04 touch test

If a hacker breaks out a container, the first thing he wants to do is to write the exploit into the application, this way
at its startup, it will start the exploit.
A read-only container will prevent everyone from from definitely leaving an exploit. The exploit will no longer exist
once the application restarts.

Set A User
Don't run your application as root in containers. Users in Docker are not namespaced.
RUN groupadd -r groupname && useradd -r -g groupname username
USER username

Do Not Use Environment Variables To Share Secrets
A sensitive data should not be shared using the ENV instruction otherwise it could be exposed to child processes,
other linked containers, Docker inspection output ..etc

Use Orchestrators Secret managers
Kubernetes or Docker Swarm offer their own secret management tools, if you want to store sensible data or send it
from one service to another then it is recommended to use these tools (e.g. Docker Secret for the Swarm mode)

Do Not Run Containers In The Privileged Mode
When you run a container using the --privileged flag, it gives all the capabilities to the container like accessing all
devices on the host as well as set some configuration in AppArmor or SELinux. The container will have almost all
the same access to the host as a normal process running without a container on the host.

Turn Off Inter-Container Communication
If you don't need inter-container communication on the same host, run containers with --icc=false --iptables in order
enable only communication between containers linked together explicitly. If these flags are not activated
unrestricted network traffic is enabled between all containers on the same host.
docker run -d --icc=files --iptables

Set Volumes To Read-Only
If you don't need to modify files in attached volumes make them read-only.
docker run -v /folder:/folder:ro alpine

Only Install Necessary Packages
Inside the container, install only what you need. Don't install unnecessary packages e.g ssh, cron, man-db ..etc In
order to see what packages are installed in a container, depending on your package manager, run: docker exec
 rpm -qa or dpkg -l ..etc

Make Sure Docker Is Up To Date
Docker has an active community and security updates are frequent. It is recommended from a security point of view
to always have the latest version in your production environments.

Use Vulnerability Analysis Scanners
Most known ones are the official Docker Cloud security scanner and Clair by CoreOS.

Properly Configure Your Docker Registry Access Control
Vine Docker images were hacked because their private registry was publicly accessible at

docker.vineapp.com

Security Through Obscurity
If

docker.vineapp.com

was

1xoajze313kjaz.vineapp.com

, the hacker would not be able to discover the private registry.

Secure And Control Your Code
Even if your hosts and containers are secure, the problem could come from the containerized application: e.g. You
are using PHP and the remote file inclusion/execution configuration is set to active or running system commands
from code is active .. etc

Use Limited Linux Capabilities
When limiting the Linux capabilities of a container, even in the case a hacker gets into the container, the host system
will be protected.
By default, Docker starts containers with a restricted set of capabilities

Use Seccomp
By default a container has around 44 disabled system calls (out of 300+). The 270 calls that still open may be
susceptible to attacks. For a high level of security, you can set Seccomp profiles individually for containers but be
sure to understand each syscall.
docker run --rm -it --security-opt seccomp=default.json hello-world

Chapter XVII - Docker, Containerd & Standalone
Runtimes Architecture
o
o

^__^
(oo)\_______
(__)\
)\/\
||----w |
||
||

The architecture of Docker has evolved many times since its creation. This chapter aims to explain what you
(developers, ops engineers and architects) should know about Containerd integration in Docker architecture. Let’s
start by defining Docker Daemon and then see how it is integrated in the new Docker architecture and Containerd.

Docker Daemon
Like the init has its daemon, cron has crond, dhcp has dhcpd, Docker has its own daemon dockerd. To list Docker
daemons, list all Linux daemons:
ps -U0 -o ‘tty,pid,comm’ | grep ^?

And grep Docker on the output:
ps -U0 -o ‘tty,pid,comm’ | grep ^?|grep -i dockerd
? 2779 dockerd

Notice that you see also docker-containerd-shim. We are going to see this in details later in this chapter. If you are
already running Docker, when you type dockerd you will have a similar error message to this :
FATA[0000] Error starting daemon: pid file found, ensure docker is not running or delete /var/run/docker.pid

Now let’s stop Docker:
service docker stop

and run its daemon directly using

dockerd

command.

Running the Docker daemon command using dockerd is a good debugging tool, as you may see, you will have the
running traces right on your terminal screen:
INFO[0000] libcontainerd: new containerd process, pid: 19717
WARN[0000] containerd: low RLIMIT_NOFILE changing to max current=1024 max=4096
INFO[0001] [graphdriver] using prior storage driver "aufs"
INFO[0003] Graph migration to content-addressability took 0.63 seconds
WARN[0003] Your kernel does not support swap memory limit.
WARN[0003] mountpoint for pids not found
INFO[0003] Loading containers: start.
INFO[0003] Firewalld running: false
INFO[0004] Removing stale sandbox ingress_sbox (ingress-sbox)
INFO[0004]
Default
bridge
(docker0)
is
assigned
with
an
IP
address
172.17.0.0/16.
Daemon
option
-bip can be used to set a preferred IP address
INFO[0004] Loading containers: done.
INFO[0004] Listening for local connections
addr=/var/lib/docker/swarm/control.sock proto=unix
INFO[0004] Listening for connections
addr=[::]:2377 proto=tcp
INFO[0004] 61c88d41fce85c57 became follower at term 12
INFO[0004] newRaft 61c88d41fce85c57 [peers: [], term: 12, commit: 290, applied: 0, lastindex: 290, lastterm: 12]
INFO[0004] 61c88d41fce85c57 is starting a new election at term 12
INFO[0004] 61c88d41fce85c57 became candidate at term 13
INFO[0004] 61c88d41fce85c57 received vote from 61c88d41fce85c57 at term 13
INFO[0004] 61c88d41fce85c57 became leader at term 13
INFO[0004] raft.node: 61c88d41fce85c57 elected leader 61c88d41fce85c57 at term 13
INFO[0004] Initializing Libnetwork Agent Listen-Addr=0.0.0.0 Local-addr=192.168.0.47 Adv-addr=192.168.0.47 Remote-addr =
INFO[0004] Daemon has completed initialization
INFO[0004] Initializing Libnetwork Agent Listen-Addr=0.0.0.0 Local-addr=192.168.0.47 Adv-addr=192.168.0.47 Remote-addr =
INFO[0004] Docker daemon
commit=7392c3b graphdriver=aufs version=1.12.5
INFO[0004] Gossip cluster hostname eonSpider-3e64aecb2dd5
INFO[0004] API listen on /var/run/docker.sock
INFO[0004]
No
nonlocalhost DNS nameservers are left in resolv.conf. Using default external servers : [nameserver 8.8.8.8 nameserver 8.8.4.4]
INFO[0004] IPv6 enabled; Adding default IPv6 external servers : [nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844]
INFO[0000] Firewalld running: false

Now if you create or remove containers for example, you will see that Docker daemon connects you (Docker client)
to Docker containers.

This is the global architecture of Docker :

Containerd
Containerd is one of the recent projects in the Docker ecosystem and its purpose is breaking up more modularity to
Docker architecture and more neutrality visà-vis the other industry actors (Cloud providers and other orchestrator
services).
According to Solomon Hykes, containerd is already deployed on millions of machines since April 2016 when it was
included in Docker 1.11. The announced roadmap to extend containerd get its input from the cloud providers and
actors like Alibaba Cloud, AWS, Google, IBM, Microsoft, and other active members of the container ecosystem.
More Docker engine functionality will be added to containerd so that containerd 1.0 will provide all the core
primitives you need to manage containers with parity on Linux and Windows hosts:
Container execution and supervision
Image distribution
Network Interfaces Management
Local storage
Native plumbing level API
Full OCI support, including the extended OCI image specification
To build, ship and run containerized applications, you may continue to use Docker but if you are looking for
specialized components you could consider Containerd.
Docker engine 1.11 was the first release built on runC (a runtime based on Open Container Intiative technology) and
containerd.

Formed in June 2015, the Open Container Initiative (OCI) aims to establish common standards for software
containers in order to avoid a potential fragmentation and divisions inside the container ecosystem.
It contains two specifications:
runtime-spec: The runtime specification
image-spec: The image specification
The runtime specification outlines how to run a filesystem bundle that is unpacked on disk:
A standardized container bundle should contain the needed information and configurations to load and run a
container in a config.json file residing in the root of the bundle directory.
A standardized container bundle should contain a directory representing the root filesystem of the container.
Generally this directory has a conventional name like rootfs.
You can see the json file if you export and extract an image. In the following example, we are going to use busybox
image.
mkdir my_container
cd my_container
mkdir rootfs

docker export $(docker create busybox) | tar -C rootfs -xvf -

Now we have an extracted busybox image inside of rootfs directory.
tree -d my_container/
my_container/
└── rootfs
├── bin
├── dev
│
├── pts
│
└── shm
├── etc
├── home
├── proc
├── root
├── sys
├── tmp
├── usr
│
└── sbin
└── var
├── spool
│
└── mail
└── www

We can generate the config.json file:
docker-runc spec

This is the generated configuration file (config.json):
{
"ociVersion": "1.0.0-rc2-dev",
"platform": {
"os": "linux",
"arch": "amd64"
},
"process": {
"terminal": true,
"consoleSize": {
"height": 0,
"width": 0
},
"user": {
"uid": 0,
"gid": 0
},
"args": [
"sh"
],
"env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"TERM=xterm"
],
"cwd": "/",
"capabilities": [
"CAP_AUDIT_WRITE",
"CAP_KILL",
"CAP_NET_BIND_SERVICE"
],
"rlimits": [
{
"type": "RLIMIT_NOFILE",
"hard": 1024,
"soft": 1024
}
],
"noNewPrivileges": true
},
"root": {
"path": "rootfs",
"readonly": true
},
"hostname": "runc",
"mounts": [
{
"destination": "/proc",
"type": "proc",
"source": "proc"
},

{
"destination": "/dev",
"type": "tmpfs",
"source": "tmpfs",
"options": [
"nosuid",
"strictatime",
"mode=755",
"size=65536k"
]
},
{
"destination": "/dev/pts",
"type": "devpts",
"source": "devpts",
"options": [
"nosuid",
"noexec",
"newinstance",
"ptmxmode=0666",
"mode=0620",
"gid=5"
]
},
{
"destination": "/dev/shm",
"type": "tmpfs",
"source": "shm",
"options": [
"nosuid",
"noexec",
"nodev",
"mode=1777",
"size=65536k"
]
},
{
"destination": "/dev/mqueue",
"type": "mqueue",
"source": "mqueue",
"options": [
"nosuid",
"noexec",
"nodev"
]
},
{
"destination": "/sys",
"type": "sysfs",
"source": "sysfs",
"options": [
"nosuid",
"noexec",
"nodev",
"ro"
]
},
{
"destination": "/sys/fs/cgroup",
"type": "cgroup",
"source": "cgroup",
"options": [
"nosuid",
"noexec",
"nodev",
"relatime",
"ro"
]
}
],
"hooks": {},
"linux": {
"resources": {
"devices": [
{
"allow": false,
"access": "rwm"
}
]
},
"namespaces": [
{
"type": "pid"
},
{
"type": "network"
},
{
"type": "ipc"
},
{
"type": "uts"

},
{
"type": "mount"
}
],
"maskedPaths": [
"/proc/kcore",
"/proc/latency_stats",
"/proc/timer_list",
"/proc/timer_stats",
"/proc/sched_debug",
"/sys/firmware"
],
"readonlyPaths": [
"/proc/asound",
"/proc/bus",
"/proc/fs",
"/proc/irq",
"/proc/sys",
"/proc/sysrq-trigger"
]
}
}

Now you can edit any of the configurations listed above and run again a container without even using Docker, just
runC:
runc run container-name

Note that you should install runC first in order to use it. For Ubuntu 16.04, you can just type this command:
sudo apt install runc

You could also install it from sources:

mkdir -p ~/golang/src/github.com/opencontainers/
cd ~/golang/src/github.com/opencontainers/
git clone https://github.com/opencontainers/runc
cd ./runc
make
sudo make install

runC, a standalone containers runtime, is at its full spec, it allows you to spin containers, interact with them, and
manage their lifecycle and that’s why containers built with one engine (like Docker) can run on another engine.
Containers are started as a child process of runC and can be embedded into various other systems without having to
run a daemon (Docker Daemon).
runC is built on libcontainer which is the same container library powering a Docker engine installation. Prior to the
version 1.11, Docker engine was used to manage volumes, networks, containers, images etc.. Now, the Docker
architecture is broken into four components: Docker engine, containerd, containerd-shm and runC. The binaries are
respectively called docker, docker-containerd, docker-containerd-shim, and docker-runc.
To run a container, Docker engine creates the image, pass it to containerd. containerd calls containerd-shim that
uses runC to run the container. Then, containerd-shim allows the runtime (runC in this case) to exit after it starts the
container : This way we can run "daemon-less" containers because we are not having to have the long running
runtime processes for containers.
Currently, the creation of a container is handled by runc (via containerd) but it is possible to use another binary
(instead of runC) that expose the same command line interface of Docker and accepting an OCI bundle.
You can see the different runtimes that you have on your host by typing:
docker info|grep -i runtime

Since I am using the default runtime, this is what I should get as an output:
Runtimes: runc
Default Runtime: runc

To add another runtime, you should follow this command:
docker daemon --add-runtime "="

Example:
docker daemon --add-runtime "oci=/usr/local/sbin/runc"

There is only one containerd-shim by process and it manages the STDIO FIFO and keeps it open for the container in
case containerd or Docker dies. It is also in charge of reporting the container’s exit status to a higher level like
Docker.

Container runtime, lifecycle support and the execution (create, start, stop, pause, resume, exec, signal & delete) are
some features implemented in Containerd. Some others are managed by other components of Docker (volumes,
logging ..etc). Here is a table from the Containerd Github repository that lists the different features and tell if they
are in or out of scope.

If we run a container:
docker run --name mariadb -e MYSQL_ROOT_PASSWORD=password -v /data/lists:/var/lib/mysql -d mariadb
Unable to find image 'mariadb:latest' locally
latest: Pulling from library/mariadb
75a822cd7888: Pull complete
b8d5846e536a: Pull complete
b75e9152a170: Pull complete
832e6b030496: Pull complete
034e06b5514d: Pull complete
374292b6cca5: Pull complete
d2a2cf5c3400: Pull complete
f75e0958527b: Pull complete
1826247c7258: Pull complete
68b5724d9fdd: Pull complete
d56c5e7c652e: Pull complete
b5d709749ac4: Pull complete
Digest: sha256:0ce9f13b5c5d235397252570acd0286a0a03472a22b7f0384fce09e65c680d13
Status: Downloaded newer image for mariadb:latest
db5218c494190c11a2fcc9627ea1371935d7021e86b5f652221bdac1cf182843

Then if you type ps aux you can notice the docker-containerd-shim process relative to this container running with
the following parameters and arguments:
db5218c494190c11a2fcc9627ea1371935d7021e86b5f652221bdac1cf182843
/var/run/docker/libcontainerd/db5218c494190c11a2fcc9627ea1371935d7021e86b5f652221bdac1cf182843
and runC binary ( docker-runc )
This is the full line with the right format:
docker-containerd-shim  /var/run/docker/libcontainerd/ docker-runc

db5218c494190c11a2fcc9627ea1371935d7021e86b5f652221bdac1cf182843 is the id of the container that you can
see at the end of the container creation.

ls -l /var/run/docker/libcontainerd/db5218c494190c11a2fcc9627ea1371935d7021e86b5f652221bdac1cf182843
total 4
-rw-r--r-- 1 root root 3653 Dec 27 22:21 config.json
prwx------ 1 root root
0 Dec 27 22:21 init-stderr
prwx------ 1 root root
0 Dec 27 22:21 init-stdin
prwx------ 1 root root
0 Dec 27 22:21 init-stdout

Final Words
Docker is really a powerful tool not only because it's changin the IT industry but also because it is creating a new
landscape. The futur of the Cloud Computing, the Serveless Computing, the distributed systems, IoT and the
production systems will be using this technology and its ecosystem, so it is good that you went trough this course
and learned important things that you will need for your expertise.
I would like to thank everybody who encouraged me to start working on this, from my family to my friends and of
course my readers. Thanks a lot !
Don't forget to join my newsletter DevOpsLinks, Shipped and the community Job Board JobsForDevOps. You can
follow me on Twitter for future updates.
I hope you have been enjoying this course.
Aymen El Amri.

Table of Contents
Introduction
Preface
Chapter I - Introduction To Docker & Containers
Chapter II - Installation & Configuration
Chapter III - Basic Concepts
Chapter IV - Advanced Concepts
Chapter V - Working With Docker Images
Chapter VI - Working With Docker Containers
Chapter VII - Working With Docker Machine
Chapter VIII - Docker Networking
Chapter IX - Composing Services Using Compose
Chapter X - Docker Logging
Chapter XI - Docker Debugging And Troubleshooting
Chapter XII - Orchestration - Docker Swarm
Chapter XIII - Orchestration - Kubernetes
Chapter XIV - Orchestration - Rancher/Cattle
Chapter XV - Docker API
Chapter XVI - Docker Security
Chapter XVII - Docker, Containerd & Standalone Runtimes Architecture
Final Words



---

Source Exif Data:

File Type                       : PDF
File Type Extension             : pdf
MIME Type                       : application/pdf
PDF Version                     : 1.4
Linearized                      : No
Author                          : Aymen El Amri
Create Date                     : 2018:01:09 14:05:01+00:00
Producer                        : calibre 3.6.0 [https://calibre-ebook.com]
Title                           : Painless Docker Basic Edition: A Practical Guide to Master Docker and its Ecosystem Based on Real World Examples
Description                     : Unlock the Power of Docker & its Ecosystem
Creator                         : Aymen El Amri
Subject                         : 
Publisher                       : GitBook
Date                            : 2017:08:21 00:00:00+09:00
Language                        : en
Identifier Scheme               : mobi-asin
Identifier                      : B0751GBJLY
Metadata Date                   : 2018:01:09 23:05:01.842000+09:00
Timestamp                       : 2018:01:09 23:04:26.883000+09:00
Author sort                     : El Amri, Aymen
Page Count                      : 375

EXIF Metadata provided by EXIF.tools