[LINUX] Service mesh learned with Docker Swarm

Introduction

In this article, you will build a cluster with Docker Swarm and learn about service meshes.

The environment uses AWS ELB (Elastic Load Balancing) and two EC2 (Amazon EC2) instances to build a Docker Swarm cluster.

The advantage of introducing Docker Swarm compared to Kubernetes is the low installation cost. Docker Swarm has lower hurdles than Kubernetes, so I think it's ideal for systems that are trying to bring container technology into production.

** Experience the significance of service meshes through Docker Swarm. ** **

Docker Swarm Docker Swarm is an orchestration tool provided by Docker. With Docker Swarm, you can aggregate multiple hosts and easily deploy and scale your containers.

Docker Swarm consists of managers and nodes, where ** Swarm ** stands for swarm. To build a Docker Swar cluster, you need to do the following:

--[x] Install Docker on the manager and node. -[x] Allow the required communication in the firewall so that communication between the manager and the node is possible.

--Rules allowed by the firewall

Service mesh

Service meshes are premised on microservices architecture and provide a mechanism to control communication between applications. This controlled communication is woven like a mesh, which improves reliability.

For example, in the case of a two-unit host, the normal access path is as follows.

When the DNS name of the load balancer is accessed with a browser, the load balancer that receives the HTTP request distributes the HTTP request to the public port of the host that is the back end in round robin.

For example, even if one of the hosts fails and the container goes down, the service can be continued by the following route.

In summary, if you simply have a configuration that separates applications at the host level, you will not be able to provide services if one of the hosts fails. Therefore, it is possible to eliminate a single point of failure by utilizing container technology and making it a service mesh.

Build Docker Swarm

Describes the procedure for building Docker Swarm. To build Docker Swarm, work in the order of manager and node.

In the example of this article, web1 will be the manager and web2 will be the node, and we will deploy using the Nginx image.

As a prerequisite, Docker is already installed and the firewall described above is also allowed. In addition, since this article uses the minimum configuration, there will be one manager and one node.

manager

First of all, initialize Docker Swarm. ** Swarm mode ** is enabled by running the docker swarm init command.

If you have multiple IP addresses, specify the IP address of the interface used for communication with other nodes in the argument of ʻadvertise-addr`.

--Initialization of Docker Swarm # docker swarm init --advertise-addr <IP address>

Swarm initialized: current node (zirc78nsch77ox8di6021ux4n) is now a manager.

To add a worker to this swarm, run the following command:

    docker swarm join --token SWMTKN-1-459p0pkyqgxgkhaggjhavd419ldujenqttm1mqmwup0qz9m5qv-1kj3jy6ozwrr2fkj1qvas294a <Manager IP address>:2377

To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.

After executing the command, make a note of the output result docker swarm join --token SWMTKN-1- (abbreviation) <manager IP address>: 2377.

--node confirmation # docker node ls

ID                            HOSTNAME            STATUS              AVAILABILITY        MANAGER STATUS      ENGINE VERSION
zirc78nsch77ox8di6021ux4n *   web1          Ready               Active              Leader              18.09.9-ce

For managers, ** Leader ** is output to MANAGER STATUS.

--Network check # docker network ls

NETWORK ID          NAME                DRIVER              SCOPE
53703efd3d07        bridge              bridge              local
aacf6f5e0eb4        docker_gwbridge     bridge              local
1f0d0e4ae3e7        host                host                local
xip5tlqmokpb        ingress             overlay             swarm
2d36f1c8c80f        none                null                local

When Swarm is initialized, two new networks (docker_gwbridge, ingress) will be created.

node

Next, in order to join the node to the Docker Swarm cluster, execute the following command on the node side.

The value used as the argument of --token is an example. You can copy and paste the output of the above docker swarm init command.

You can also redisplay the token by executing the docker swarm join-token worker command on the manager side.

--Cluster participation # docker swarm join --token SWMTKN-1- (Omitted) <Manager IP address>: 2377

This node joined a swarm as a worker.

For confirmation on the manager side, execute the docker node ls command again to confirm that the node is recognized.

--node confirmation

ID                            HOSTNAME            STATUS              AVAILABILITY        MANAGER STATUS      ENGINE VERSION
zirc78nsch77ox8di6021ux4n *   web1          Ready               Active              Leader              18.09.9-ce
n2o22ptdmyhan8qg0ijmo0qn5     web2          Ready               Active                                  18.09.9-ce

Deploy Docker Swarm

In this article, we will deploy with docker-commpose, which is easy to manage. You can also deploy with the docker service create command. I will omit the explanation of service and stack.

Deployment work is done on the manager side. Go to any directory and create the following docker-commpose.yml file.

• docker-commpose.yml

version: "3"
services:
  web:
    image: nginx
    deploy:
      replicas: 2
      #resources:
      #  limits:
      #    cpus: "0.1"
      #    memory: 100M
      restart_policy:
        condition: on-failure
    ports:
      - "80:80"
    networks:
      - webnet
networks:
  webnet:

After creating the docker-commpose.yml file, execute the following command to deploy it. test is an example, so specify any name.

# docker stack deploy -c docker-commpose.yml test

Updating service test_web (id: egubeieuri00rzmm9imsna93o)

After deploying, you can check the service status with the following command. Since ** REPLICAS ** is 2, two containers are running.

# docker service ls

ID                  NAME                MODE                REPLICAS            IMAGE               PORTS
r04mfg1se3nh        test_web            replicated          2/2                 nginx:latest        *:80->80/tcp

You can see that the container is running by running the docker container ps -a command on the manager and node side.

--Manager side

CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS               NAMES
4a26c3ca6df7        nginx:latest        "nginx -g 'daemon of…"   3 seconds ago       Up 2 seconds        80/tcp              test_web.1.mnnz40tdykzd2intwz5hf68bs

--Node side

CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS               NAMES
614c19349bf0        nginx:latest        "nginx -g 'daemon of…"   2 minutes ago       Up 2 minutes        80/tcp              test_web.2.6om5oazbavassohd4akucbum2

Operation check of Docker Swarm

Check the operation of Docker Swarm.

Access the DNS name of the load balancer from your browser and check that the load is distributed normally.

--Access the DNS name of the load balancer

If you check the container log, you can see that the load is distributed by round robin. Check the ** CONTAINER ID ** with the docker container ps -a command.

# docker logs -f ＜CONTAINER ID＞

10.255.0.3 - - [06/Feb/2020:14:20:51 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36" "<Global IP of access source>"

As a test, stop the container running on web2 (node side).

# docker stop ＜CONTAINER ID＞

CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS                     PORTS               NAMES
423f673b0513        nginx:latest        "nginx -g 'daemon of…"   19 hours ago        Exited (0) 5 seconds ago                       test_web.2.kc7yypyasgvjtolsb1zwmhjoy

At this time, the container is not running on web2. Make sure you can access the DNS name of your load balancer.

As an example, access the public IP of web2 stopped from the browser.

You can check the access to the public IP of the stopped web2 (node) in the access log of the container on Web1 (manager).

10.255.0.3 - - [07/Feb/2020:02:19:01 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36" "-"

You can also access the container from the stopped web2 (node side) local by executing the following command.

# curl localhost 80

<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>
curl: (7) Couldn't connect to server

In the log of web1 (manager), it is output as follows.

10.255.0.2 - - [07/Feb/2020:02:41:47 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.61.1" "-"

If you dump the 4789 port in web1 (manager) and look at it, you can see the communication of the overlay network.

# tcpdump -nli eth0 port 4789 -Av

Docker Swarm scale

You can scale by executing the following command while the service is already running. The following command changes the number of containers to 4 containers.

# docker service scale test_web=4

test_web scaled to 4
overall progress: 4 out of 4 tasks 
1/4: running   [==================================================>] 
2/4: running   [==================================================>] 
3/4: running   [==================================================>] 
4/4: running   [==================================================>] 
verify: Service converged 

After scaling, run the docker container ps -a command and you'll see that the number of containers is increasing.

CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS               NAMES
41962946aa48        nginx:latest        "nginx -g 'daemon of…"   3 minutes ago       Up 3 minutes        80/tcp              test_web.4.pytl36ejcd81ybthisxfo47br
423f673b0513        nginx:latest        "nginx -g 'daemon of…"   7 hours ago         Up 7 hours          80/tcp              test_web.2.kc7yypyasgvjtolsb1zwmhjoy

** When deploying, there is no need to replace application assets for each server, just one manager. ** **

The nature of Docker Swarm

Docker Swarm is idempotent.

Specifically, Docker Swarm maintains a specified number of replicas, so if one of the clustered hosts goes down, the container will be started on that host.

The following is the output of the docker container ps -a command. As an example, containers are running on web1 and web2 respectively.

• web1

e5ccbfa9739b        nginx:latest                      "nginx -g 'daemon of…"   3 minutes ago       Up 3 minutes        80/tcp              test_web.1.mrgkhbd7juer72v6bv0l42fxq

• web2

4820c7bbe9c1        nginx:latest                      "nginx -g 'daemon of…"   3 minutes ago       Up 3 minutes        80/tcp              test_web.2.wfe1n11s8940rdl8r1c47o6nc

The web2 host has been stopped. When it detects that web2 is down, it creates a container on web1.

0a88f53039a3        nginx:latest                      "nginx -g 'daemon of…"   5 seconds ago       Created                                 test_web.2.p06zas3c3kt9ekjojhhfnl3co
e5ccbfa9739b        nginx:latest                      "nginx -g 'daemon of…"   4 minutes ago       Up 4 minutes        80/tcp              test_web.1.mrgkhbd7juer72v6bv0l42fxq

Two containers are running on web1.

0a88f53039a3        nginx:latest                      "nginx -g 'daemon of…"   37 seconds ago      Up 31 seconds       80/tcp              test_web.2.p06zas3c3kt9ekjojhhfnl3co
e5ccbfa9739b        nginx:latest                      "nginx -g 'daemon of…"   5 minutes ago       Up 5 minutes        80/tcp              test_web.1.mrgkhbd7juer72v6bv0l42fxq

Docker Swarm release

Describes how to cancel Docker Swarm.

First, execute the following command on the manager side to delete the service.

# docker service rm test_web

test_web

Then work on the node and then the manager.

node

--Node disconnection # docker swarm leave

Node left the swarm.

manager

Confirm that the STATUS of the node is ** DOWN **.

--node confirmation # docker node ls

ID                            HOSTNAME            STATUS              AVAILABILITY        MANAGER STATUS      ENGINE VERSION
zirc78nsch77ox8di6021ux4n *   web1          Ready               Active              Leader              18.09.9-ce
n2o22ptdmyhan8qg0ijmo0qn5     web2          Down                Active                                  18.09.9-ce

Delete the node. Specify the node name in the optional argument.

--node deleted # docker node rm --force web2

web2

Confirm that the node is no longer recognized by the manager.

--node confirmation # docker node ls

ID                            HOSTNAME            STATUS              AVAILABILITY        MANAGER STATUS      ENGINE VERSION
zirc78nsch77ox8di6021ux4n *   web1          Ready               Active              Leader              18.09.9-ce

Finally, disconnect the manager itself.

--Manager disconnection # docker swarm leave --force

Node left the swarm.

Docker Swarm dismantling

If you don't use Docker Swarm, do the following on the manager side:

Make sure the node does not exist.

--node confirmation # docker node ls

Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again.

Confirm that the network created by Docker Swarm has been deleted. The docker_gwbridge network remains.

# docker network ls

NETWORK ID          NAME                DRIVER              SCOPE
987cfc73d87c        bridge              bridge              local
aacf6f5e0eb4        docker_gwbridge     bridge              local
1f0d0e4ae3e7        host                host                local
2d36f1c8c80f        none                null                local

Execute the following command to delete all unused resources.

--Resource deletion # docker system prune

WARNING! This will remove:
        - all stopped containers
        - all networks not used by at least one container
        - all dangling images
        - all dangling build cache
Are you sure you want to continue? [y/N] y
Deleted Networks:
docker_gwbridge

Deleted Images:
untagged: nginx@sha256:ad5552c786f128e389a0263104ae39f3d3c7895579d45ae716f528185b36bc6f
deleted: sha256:2073e0bcb60ee98548d313ead5eacbfe16d9054f8800a32bedd859922a99a6e1
deleted: sha256:a3136fbf38691346715cac8360bcdfca0fff812cede416469653670f04e2cab0
deleted: sha256:99360ffcb2da18fd9ede194efaf5d4b90e7aee99f45737e918113e6833dcf278
deleted: sha256:488dfecc21b1bc607e09368d2791cb784cf8c4ec5c05d2952b045b3e0f8cc01e
untagged: nginx@sha256:70821e443be75ea38bdf52a974fd2271babd5875b2b1964f05025981c75a6717
deleted: sha256:5ad3bd0e67a9c542210a21a3c72f56ef6387cf9b7f4c2506d2398d55a2593ed0
deleted: sha256:b69e2ed46519bc33e7c887967e4f61a2ee53aef165b70f75e208937fb42e7b4c
deleted: sha256:4cb7f732537bf0f65cd9f8f7b63bbe71abcf9d0df396f58621ef3be0b2487b27
deleted: sha256:556c5fb0d91b726083a8ce42e2faaed99f11bc68d3f70e2c7bbce87e7e0b3e10

Total reclaimed space: 253.4MB

knowledge

version of docker-compose

When using Docker Swarm, the version that can be used with docker-commpose is ** 3 **.

Precautions when building with docker-compose

An image is required when building (stacking) with docker-commpose.

From Compose file version 3 reference

Note: (Version 3) This option is ignored when deploying the stack in Swarm mode using the Compose file. This docker stack command only accepts pre-built images.

Therefore, the environment of ** Docker registry ** is required.

For information on how to set up the Docker registry locally, refer to the official URL below.

• Deploy a stack to a swarm

As an example, if you specify the build and execute the deployment as shown below,

version: '3'

services:
  app:
    build: ./src

The following error message is output and it is not possible to deploy.

failed to create service stackdemo_backend: Error response from daemon: rpc error: code = InvalidArgument desc = ContainerSpec: image reference must be provided

Deployment error

As an example, if you deploy with 2 hosts and set the value of ** replicas ** in the docker-commpose.yml file to 2, it will basically be distributed in 1 container for 1 host.

If two containers are started on one host during deployment, it is possible that some error occurred and the containers could not be started on one host. For Linux, the error can be confirmed from the syslog.

About the result of verification with the configuration of this article (1 Master and 1 Node)

The result of verification with the configuration of this article (1 Master and 1 Node) is described as a reminder.

Simply put, one Master and one Node cannot achieve true redundancy.

To further improve reliability, it is necessary to perform redundancy with two Masters. There is an official description of Master redundancy.

-** Behavior when one container is stopped or deleted in the configuration of this article (1 Master and 1 Node) ** If you stop or delete one of the Master and Node containers, it will self-regenerate until you reach the number defined in ** replicas ** in the docker-compose.yml file. You can also access the public IP via the service mesh while this momentary interruption occurs. However, if it is local, a momentary interruption will occur.

-** Behavior when the Node host itself is stopped in the configuration of this article (1 Master and 1 Node) ** Containers running on Node will be created on the Master side until the number defined in ** replicas ** of the docker-compose.yml file is reached. (The movement is similar to Vmware's Vmotion, but the reality is different.) After that, even if you start Node, the container that was running on Node will not be automatically restored, so manual action is required. By the way, if you stop or delete the container created earlier in Master, the container will be created again on the Node side.

-** Behavior when the Master host itself is stopped in the configuration of this article (1 Master and 1 Node) ** Master of Docker Swarm is stopped, but it can be accessed as an application because it can be accessed by the service mesh.

in conclusion

Orchestration tools are an essential technology for modern application development, where uncertainty and rapid change are required.

As an application, serverless technology can be used to set thresholds for CPU and memory resources, and when the thresholds are exceeded, the server can be scaled out.

reference

• Docker Swarm -What is a service mesh