Docker Overview

Docker and the container ecosystem

• Open source projects: There are several open source projects started by Docker, which are now maintained by a large community of developers.
  • Moby Project is the upstream project upon which the Docker Engine is based. It provides all of the components needed to assemble a fully functional container system.
  • Runc is a command-line interface for creating and configuring containers, and has been built to the OCI specification.
  • Containerd is an easily embeddable container runtime. It is also a core component of the Moby Project.
  • LibNetwork is a Go library that provides networking for containers.
  • Notary is a client and server that aims to provide a trust system for signed container images.
  • HyperKit is a toolkit that allows you to embed hypervisor capabilities into your own applications, presently it only supports the macOS and the Hypervisor.framework.
  • VPNKit provides VPN functionality to HyperKit.
  • DataKit allows you to orchestrate application data using a Git-like workflow.
  • SwarmKit is a toolkit that allows you to build distributed systems using the same raft consensus algorithm as Docker Swarm.
  • LinuxKit is a framework that allows you to build and compile a small portable Linux operating system for running containers.
  • InfraKit is a collection of tools that you can use to define infrastructure to run your LinuxKit generated distributions on.
• Docker CE and Docker EE: This is the core collection of free-to-use and commercially supported Docker tools built on top of the open source components.
  • Docker Compose: A tool that allows you to define and share multi-container definitions.
  • Docker Machine: A tool to launch Docker hosts on multiple platforms.
  • Docker Hub: A repository for your Docker images.
  • Docker Store: A storefront for official Docker images and plugins as well as licensed products.
  • Docker Swarm: A multi-host-aware orchestration tool.
  • Docker for Mac
  • Docker for Windows
  • Docker for Amazon Web Services
  • Docker for Azure
• Docker, Inc.: This is the company founded to support and develop the core Docker tools. Docker, Inc. are offer consultative services to companies who wish take their existing applications and containerize them as part of Docker’s Modernize Traditional Apps (MTA) program.

Building Container Images

Introducing the Dockerfile

A Dockerfile is simply a plain text file that contains a set of user-defined instructions. When the Dockerfile is called by the docker image build command.

FROM alpine:latest
LABEL maintainer="Russ McKendrick <[email protected]>"
LABEL description="This example Dockerfile installs NGINX."
RUN apk add --update nginx && \
    rm -rf /var/cache/apk/* && \
    mkdir -p /tmp/nginx/

COPY files/nginx.conf /etc/nginx/nginx.conf
COPY files/default.conf /etc/nginx/conf.d/default.conf
ADD files/html.tar.gz /usr/share/nginx/

EXPOSE 80/tcp

ENTRYPOINT ["nginx"]
CMD ["-g", "daemon off;"]

Reviewing the Dockerfile in depth

FROM

The FROM instruction tells Docker which base you would like to use for your image.

LABEL

The LABEL instruction can be used to add extra information to the image.

However, using too many labels can cause the image to become inefficient as well, so I would recommend using the label schema detailed at http://label-schema.org/. You can view the containers’ labels with the following Docker inspect command:

docker image inspect -f {{.Config.Labels}} <IMAGE_ID>

RUN

The RUN instruction is where we interact with our image to install software and run scripts, commands, and other tasks.

However, much like adding multiple labels, this is considered to be considered inefficient as it can add to the overall size of the image, which for the most part we should try to avoid.

COPY and ADD

COPY files/nginx.conf /etc/nginx/nginx.conf
COPY files/default.conf /etc/nginx/conf.d/default.conf

We are copying two files from the files folder on the host we are building our image on.

ADD files/html.tar.gz /usr/share/nginx/

ADD automatically uploads, uncompresses, and puts the resulting folders and files at the path we tell it to.

EXPOSE

The EXPOSE instruction lets Docker know that when the image is executed, the port and protocol defined will be exposed at runtime. This instruction does not map the port to the host machine, but instead, opens the port to allow access to the service on the container network.

ENTRYPOINT and CMD

If you think of some of the CLI commands you might use, you have to specify more than just the CLI command. You might have to add extra parameters that you want the command to interpret. This would be the use case for using ENTRYPOINT only.

ENTRYPOINT ["nginx"]
CMD ["-g", "daemon off;"]

As the CMD is executed, giving us the equivalent of running the following command:

nginx -g daemon off;

Another example of how ENTRYPOINT can be used is the following:

docker container run --name nginx-version dockerfile-example -v

This would be the equivalent of running the following command on our host:

nginx -v

Other Dockerfile instructions

User

The USER instruction lets you specify the username to be used when a command is run. The USER instruction can be used on the RUN instruction, the CMD instruction, or the ENTRYPOINT instruction in the Dockerfile. Also, the user defined in the USER instruction has to exist, or your image will fail to build. Using the USER instruction can also introduce permission issues, not only on the container itself, but also if you mount volumes.

WORKDIR

The WORKDIR instruction sets the working directory for the same set of instructions that the USER instruction can use.

ONBUILD

The ONBUILD instruction lets you stash a set of commands to be used when the image is used in future, as a base image for another container image.
The ONBUILD instruction can be used in conjunction with the ADD and RUN instructions.

ENV

The ENV instruction sets environment variables within the image both when it is built and when it is executed. These variables can be overridden when you launch your image.

Building container images

You can use the other options to limit how much CPU and memory the build process will use. In some cases, you may not want the build command to take as much CPU or memory as it can have. The process may run a little slower, but if you are running it on your local machine or a production server and it’s a long build process, you may want to set a limit. There are also options that affect the networking configuration of the container launched to build our image.

Typically, you don’t use the --file or -f switch, as you run the docker build command from the same folder that the Dockerfile is in. Keeping the Dockerfile in separate folders helps sort the files and keeps the naming convention of the files the same.

It also worth mentioning that, while you are able to pass additional environment variables as arguments at build time, they are used at build time and your container image does not inherit them. This is useful for passing information such as proxy settings, which may only be applicable to your initial build/test environment.

The .dockerignore file is used to exclude those files or folders we don’t want to be included in the docker build as, by default, all files in the same folder as the Dockerfile will be uploaded.

Using a Dockerfile to build a container image

docker image build --file /path/to/your/dockerfile --tag local:dockerfile-example .

Typically, the --file switch isn’t used.

The most important thing to remember is the dot (or period) at the very end. This is to tell the docker image build command to build in the current folder.

Using an existing container

However, it is not recommended or considered to be good practice.

docker image pull alpine:latest
docker container run -it --name alpine-test alpine /bin/sh

To save our container as an image, you need to do something similar to the following:

docker container commit alpine-test local:broken-container

To save the image file, simply run the following command:

docker image save -o broken-container.tar local:broken-container

In the early days of Docker, distributing images that had been prepared in this way was common practice.

Building a container image from scratch

Docker has done some of the hard work for us already, and created an empty TAR file on the Docker Hub named scratch; you can use it in the FROM section of your Dockerfile. You can base your entire Docker build on this, and then add parts as needed.

FROM scratch
ADD files/alpine-minirootfs-3.8.0-x86_64.tar.gz /
CMD ["/bin/sh"]

Using environmental variables

ENV <key> <value>
ENV username admin

Alternatively, you can always use an equals sign between the two:

ENV <key>=<value>
ENV username=admin

With the second ENV example, you can set multiple environmental variables on the same line, as shown here:

ENV username=admin database=wordpress tableprefix=wp

Using multi-stage builds

FROM golang:latest as builder
WORKDIR /go-http-hello-world/
RUN go get -d -v golang.org/x/net/html 
ADD https://raw.githubusercontent.com/geetarista/go-http-hello-world/master/hello_world/hello_world.go ./hello_world.go
RUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -o app .

FROM scratch 
COPY --from=builder /go-http-hello-world/app .
CMD ["./app"]

Storing and Distributing Images

Docker Registry

Deploying your own registry

Docker Registry is distributed as an image from Docker Hub, which makes deploying it as easy as running the following commands:

docker image pull registry:2
docker container run -d -p 5000:5000 --name registry registry:2

Now that we have a copy of the Alpine Linux image, we need to push it to our local Docker Registry, which is available at localhost:5000. To do this, we need to tag the Alpine Linux image with the URL of our local Docker Registry, along with a different image name:

docker image tag alpine localhost:5000/localalpine
docker image push localhost:5000/localalpine

Docker Registry currently supports the following storage options:

• Filesystem: All images are stored on the filesystem at the path you define. The default is /var/lib/registry.
• Azure: This uses Microsoft Azure Blob Storage.
• GCS: This uses Google Cloud storage.
• S3: This uses Amazon Simple Storage Service (Amazon S3).
• Swift: This uses OpenStack Swift.

Docker Trusted Registry

One of the components that ships with the commercial Docker Enterprise Edition (Docker EE) is Docker Trusted Registry (DTR). Think of it as a version of Docker Hub that you can host in your own infrastructure. DTR adds the following features on top of the ones provided by the free Docker Hub and Docker Registry:

• Integration into your authentication services, such as Active Directory or LDAP
• Deployment on your own infrastructure (or cloud) behind your firewall
• Image signing to ensure your images are trusted
• Built-in security scanning
• Access to prioritized support directly from Docker

Microbadger

Microbadger is a great tool when you are looking at shipping your containers or images around. It will take into account everything that is going on in every single layer of a particular Docker image and give you the output of how much weight it has in terms of actual size or the amount of disk space it will take up.
Another great feature is that Microbadger gives you the option of embedding basic statistics about your images in your Git repository or Docker Hub.

Managing Containers

Docker container commands

Interacting with your containers

attach

The first way of interacting with your running container is to attach to the running process.

docker container attach nginx-test

Pressing Ctrl + C will terminate the process and return your Terminal to normal.

docker container attach --sig-proxy=false nginx-test

Pressing Ctrl + C will detach us from the nginx process, but this time, rather than terminating the nginx process, it will just return us to our Terminal, leaving the container in a detached state.

exec

docker container exec nginx-test cat /etc/debian_version

This will spawn a second process. The second process will then terminate, leaving our container as it was before the exec command was executed.

docker container exec -i -t nginx-test /bin/bash

The -i flag is shorthand for --interactive, which instructs Docker to keep stdin open so that we can send commands to the process. The -t flag is short for --tty and allocates a pseudo-TTY to the session.

Logs and process information

logs

docker container logs --since 2018-08-25T18:00 nginx-test

The logs command shows the timestamps of stdout as recorded by Docker, and not the time within the container.

docker container logs --since 2018-08-25T18:00 -t nginx-test

The -t flag is short for --timestamp; this option prepends the time the output was captured by Docker.

top

The top command is quite a simple one; it lists the processes running within the container you specify.

stats

The stats command provides real-time information on either the specified container or, if you don’t pass a NAME or ID container, on all running containers.

Resource limits

By default, when launched, a container will be allowed to consume all the available resources on the host machine if it requires it.
Typically, we would have set the limits when we launched our container using the run command:

docker container run -d --name nginx-test --cpu-shares 512 --memory 128M -p 8080:80 nginx

However, we didn’t launch our nginx-test container with any resource limits, meaning that we need to update our already running container:

docker container update --cpu-shares 512 --memory 128M --memory-swap 256M nginx-test

Container states and miscellaneous commands

Pause and unpause

docker container pause nginx1

Running docker container ls will show that the container has a status of Up, but it also says Paused.

Note that we didn’t have to use the -a flag to see information about the container as the process has not been terminated; instead, it has been suspended using the cgroups freezer. With the cgroups freezer, the process is unaware it has been suspended, meaning that it can be resumed.

Miscellaneous commands

The create command is pretty similar to the run command, except that it does not start the container, but instead prepares and configures one.

The port command displays the port along with any port mappings for the container.
The diff command prints a list of all of the files that have been added (A) or changed (C) since the container was started.

Docker networking and volumes

Docker networking

docker image pull redis:alpine
docker image pull russmckendrick/moby-counter
docker network create moby-counter

docker container run -d --name redis --network moby-counter redis:alpine
docker container run -d --name moby-counter --network moby-counter -p 8080:80 russmckendrick/moby-counter

docker container exec moby-counter ping -c 3 redis
docker container exec moby-counter cat /etc/hosts
docker container exec moby-counter cat /etc/resolv.conf
# nameserver 127.0.0.11

docker container exec moby-counter nslookup redis 127.0.0.11

docker network create moby-counter2
docker run -itd --name moby-counter2 --network moby-counter2 -p 9090:80 russmckendrick/moby-counter
docker container run -d --name redis2 --network moby-counter2 --network-alias redis redis:alpine

Docker volumes

docker volume create redis_data
docker volume inspect redis_data
docker container run -d --name redis -v redis_data:/data --network moby-counter redis:alpine
docker volume ls

docker container stop redis
docker volume prune

Docker Compose

Our first Docker Compose application

Docker Compose uses a YAML file, typically named dockercompose.yml, to define what your multi-container application should look like.

version: "3"

services:
    redis:
        image: redis:alpine
         volumes:
            - redis_data:/data
        restart: always
    mobycounter:
        depends_on:
            - redis
        image: russmckendrick/moby-counter
        ports:
            - "8080:80"
        restart: always
        
volumes:
    redis_data:

To launch our application, we simply change to the folder that contains your docker-compose.yml file and run the following:

docker-compose up

You may have also spotted the Docker Compose namespace in our multi-container application has prefixed everything with mobycounter. It took this name from the folder our Docker Compose file was being stored in.

Docker Compose commands

when we used Ctrl + C to return to our Terminal, the Docker Compose containers were stopped.

Up and PS

docker-compose up -d

This will start your application back up, this time in detached mode.

Once control of your Terminal is returned, you should be able to check that the containers are running using the following command:

docker-compose ps

When running these commands, Docker Compose will only be aware of the containers defined in the service section of your docker-compose.yml file; all other containers will be ignored as they don’t belong to our service stack.

Config

docker-compose config

If there are no issues, it will print a rendered copy of your Docker Compose YAML file to screen; this is how Docker Compose will interpret your file. If you don’t want to see this output and just want to check for errors, then you can run the following command:

docker-compose config -q

Pull, build, and create

The docker-compose build command can also be used to trigger a build if there are updates to any of the Dockerfiles used to originally build your images.

Scale

The scale command will take the service you pass to the command and scale it to the number you define.

docker-compose scale worker=3
# or
docker-compose up -d --scale vote=3

Kill, rm, and down

If you want to remove everything, you can do so by running the following:

docker-compose down --rmi all --volumes

This will remove all of the containers, networks, volumes, and images (both pulled and built) when you ran the docker-compose up command; this includes images that may be in use outside of your Docker Compose application. There will, however, be an error if the images are in use, and they will not be removed.

Docker App

Docker App is a self-contained binary that helps you to create an application bundle that can be shared via Docker Hub or a Docker Enterprise Registry.

There is a slight change to the docker-compose.yml file we will be using. The version needs to be updated to 3.6 rather than just 3.

docker-app init --single-file mobycounter

This command takes our docker-compose.yml file and embeds it in a .dockerapp file.

version: latest
name: mobycounter
description: An example Docker App file which packages up the Moby Counter application
namespace: masteringdockerthirdedition
maintainers:
  - name: Russ McKendrick
    email: [email protected]
    
---
version: "3.6"

services:
    redis:
        image: redis:alpine
        volumes:
            - redis_data:/data
        restart: always
    mobycounter:
        depends_on:
            - redis
        image: russmckendrick/moby-counter
        ports:
            - "${port}:80"
        restart: always
        
volumes:
    redis_data:
    
---

{ "port":"8080" }

As you can see, it split into three sections; the first contains metadata about the application.

The second section contains our Docker Compose file. You may notice that a few of the options have been replaced with variables. The default value for the port variable is defined in the final section.

Once the .dockerapp file is complete, you can run the following command to save the Docker App as an image:

docker-app save

You can view just the Docker Apps you have active on your host by running this:

docker-app ls
docker-app inspect masteringdockerthirdedition/mobycounter.dockerapp:latest

Now that we have our finished application, we need to push it:

docker-app push
docker image rm masteringdockerthirdedition/mobycounter.dockerapp:latest
docker-app render masteringdockerthirdedition/mobycounter:latest --set port="9090" | docker-compose -f - up
docker-app render masteringdockerthirdedition/mobycounter:latest --set port="9090" | docker-compose -f - ps

Windows Containers

An introduction to Windows containers

The work by Microsoft on the Windows kernel has introduced the same process isolation as found on Linux. Also, like Linux containers, this isolation extends to a sandboxed filesystem and even a Windows registry.

There is also another layer of isolation provided by Windows Containers. Hyper-V isolation runs the container processes within a minimal hypervisor when the container is started. This further isolates the container processes from the host machine. However, there is a cost of a small amount of additional resources needed for each container running with Hyper-V isolation, while these containers will also have an increased start time as the hypervisor needs to be launched before the container can be started.

You can’t manage Hyper-V isolated containers using the standard Hyper-V management tools. You have to use Docker.

Docker Machine

An introduction to Docker Machine

Docker Machine’s biggest strength is that it provides a consistent interface to several public cloud providers. The following locally-hosted hypervisors are supported, such as Oracle VirtualBox and VMware Workstation or Fusion.

Deploying local Docker hosts with Docker Machine

docker-machine create --driver virtualbox docker-local
docker-machine env docker-local

This command returns the following:

export DOCKER_TLS_VERIFY="1"
export DOCKER_HOST="tcp://192.168.99.100:2376"
export DOCKER_CERT_PATH="/Users/russ/.docker/machine/machines/docker-local"
export DOCKER_MACHINE_NAME="docker-local"
# Run this command to configure your shell:
# eval $(docker-machine env docker-local)

Docker Swarm

Introducing Docker Swarm

There are two very different versions of Docker Swarm. There was a standalone version of Docker Swarm; this was supported up until Docker 1.12 and is no longer being actively developed.

Docker version 1.12 introduced Docker Swarm mode. This introduced all of the functionality that was available in the standalone Docker Swarm into the core Docker engine, along with a significant number of additional features.

A Swarm is a collection of hosts, all running Docker, which have been set up to interact with each other in a clustered configuration.

Roles within a Docker Swarm cluster

Swarm manager

The Swarm manager is a host that is the central management point for all Swarm hosts. Swarm manager is where you issue all your commands to control those nodes. You can switch between the nodes, join nodes, remove nodes, and manipulate those hosts.

Each cluster can run several Swarm managers. Swarm managers use the Raft Consensus Algorithm to maintain a consistent state across all of the manager nodes.

Swarm worker

The Swarm workers are those that run the Docker containers.

Creating and managing a Swarm

Adding a Swarm manager to the cluster

docker $(docker-machine config swarm-manager) swarm init \
    --advertise-addr $(docker-machine ip swarm-manager):2377 \
    --listen-addr $(docker-machine ip swarm-manager):2377

Joining Swarm workers to the cluster

docker $(docker-machine config swarm-worker01) swarm join \
     --token $SWARM_TOKEN \
     $(docker-machine ip swarm-manager):2377

Listing nodes

docker node ls

Managing a cluster

Finding information on the cluster

docker node inspect swarm-manager --pretty

Promoting a worker node

You can promote a worker node to a manager node.

docker node promote swarm-worker01

Demoting a manager node

docker node demote swarm-manager

Draining a node

docker node update --availability drain swarm-manager

This will stop any new tasks, such as new containers launching or being executed against the node we are draining. Once new tasks have been blocked, all running tasks will be migrated from the node we are draining to nodes with an ACTIVE status.

To add the node back into the cluster, simply change the AVAILABILITY to active by running this:

docker node update --availability active swarm-manager

Docker Swarm services and stacks

Services

The service command is a way of launching containers that take advantage of the Swarm cluster.

docker service create \
    --name cluster \
    --constraint "node.role == worker" \
    -p:80:80/tcp \
    russmckendrick/cluster

First of all, we can list the services again by running this command:

docker service ls

As you can see, it is a replicated service and 1/1 containers are active.

We haven’t had to care about which of our two worker nodes the service is currently running on.

Run the following commands to scale and check our service:

docker service scale cluster=6

Stacks

Docker has created functionality that allows you to define your services in Docker Compose files.

version: "3"
services:
    cluster:
        image: russmckendrick/cluster
        ports:
            - "80:80"
        deploy:
            replicas: 6
            restart_policy:
                condition: on-failure
            placement:
                constraints:
                    - node.role == worker

The stack can be made up of multiple services, each defined under the services section of the Docker Compose file.

In addition to the normal Docker Compose commands, you can add a deploy section; this is where you define everything relating to the Swarm element of your stack.

docker stack deploy --compose-file=docker-compose.yml cluster
docker stack ls
docker stack services cluster
docker stack ps cluster
docker stack rm cluster

Load balancing, overlays, and scheduling

Ingress load balancing

Docker Swarm has an ingress load balancer built in, making it easy to distribute traffic to our public facing containers.

This means that our application can be scaled up or down, fail, or be updated, all without the need to have the external load balancer reconfigured.

Network overlays

Docker Swarm’s network overlay layer extends the network you launch your containers in across multiple hosts, meaning that each service or stack can be launched in its own isolated network.

Each overlay network has its own inbuilt DNS service, which means that every container launched within the network is able to resolve the hostname of another container within the same network to its currently assigned IP address.

Scheduling

At the time of writing, there is only a single scheduling strategy available within Docker Swarm, called Spread. What this strategy does is to schedule tasks to be run against the least loaded node that meets any of the constraints you defined when launching the service or stack.