Enter from docker-compose up for the time being, and learn Docker while learning the basic design of Web server (Nginx) ①

(1) Environment construction-The simplest Web server construction

If you comment on points that are difficult to understand or inaccurate, it will lead to motivation. Prior to this time, I wrote an article with a service called Zenn Original article Enter from docker-compose up for the time being and learn Docker while learning the basic design of Web server ① | Zenn

Introduction

I hope it will be useful for the following people

1. Create your first web app
2. Create a web app and deploy it now
3. I reached the deployment for the first time, but I do not understand well what Nginx is doing or the setting contents by copying

I think it's close to 3 in the 5th month of self-study programming As a personal message, there are more people in the state of 1. I hope that the contents of this article will arrive. ** I feel that it is convenient to understand Docker somehow! **, ** Expand interest in how web apps work **, I hope I can share that awareness.

The same content as this article can be reproduced by using EC2 on AWS or by using a contracted VPS, but if you use docker, you can challenge ** completely free of charge **! Easier and don't be scared of unplanned billing

What you can find in this article

Expand your horizons to the network

--Know the minimum configuration required for the web application to work --Touch the three important roles of Nginx (pronounced Engine X): web server, load balancer, and reverse proxy. --Know the basic settings of Nginx

Get to know the basics of Docker

--Know that you can easily reproduce the existing development environment --You can try the minimum commands required for development on Docker. --You can know the contents described in docker-compose.yml and the mechanism of volume by moving your hand. --Prepare multiple docker-compose.yml to simulate different environments (development-> production)

Web application development environment-know the difference in production environment

--Understanding why asset compilation is needed in a production environment --Understand why asset compilation is not required in your development environment

Therefore, at the end of this article, I will deploy the application on Docker while touching on the difference from the development environment in the virtual production environment.

Appendix

appendix is supplementary content I wrote the contents that can be known in that section at the beginning, so If you don't need to know it again, you can skip it. If you don't know what you're doing, you can actually move your hand and keep it in the corner of your head, which may be useful later **

What you need, skills

--Editor (verified with VS Code) Visual Studio Code - Code Editing. Redefined

--Knowledge of basic Linux commands (cd, ls, vi ..., even if you don't have it, you can copy and paste it)

• Docker desktop Docker Desktop for Mac and Windows | Docker

――What is Docker? The following is recommended for those who say

[Series] The most understandable container in the world & Introduction to Docker ~ Part 1: What is a container? ~ | SIOS Tech. Lab It is recommended that you understand the concept quickly and move your hands here.

The application part uses Rails for FW (framework), You don't need to know Rails

(I have no development experience other than Rails, so there may be inappropriate content in other FWs)

Verification environment

macOS Catalina docker desktop: 2.3.05 (docker engin: 19.03.12, docker-compose: 1.27.2) Nginx: 1.18 Rails: 6.03 PostgreSQL: 11.0

Architecture (design) overview

In the environment to be built with Docker from now on ** Nginx is acting as a reverse proxy, delivering static content on behalf of ʻapp: Rails (= proxy) and forwarding only requests for dynamic content to ʻapp: Rails It has become **

I would like to understand that little by little

It is a configuration that you often see (Database and some others omitted) Services called web (Nginx) and app (rails) are running on docker in independent containers. It is an understanding that each dependency etc. is defined by docker-compose

Goal 5 minutes, build Rails environment with Docker

Nginx --Build Rails environment I will refer to the following wonderful article (laugh) Nginx, Rails 6, PostgreSQL environment (and even Bootstrap) can be built immediately! We are improving little by little, so we welcome your comments.

Building Rails6 development environment in 5 minutes with one command on Docker --Rails6 + Nginx + PostgreSQL + Webpack (Bootstrap installed) --Qiita

Source code prepared for this article based on the above https://github.com/naokit-dev/try_nginx_on_docker.git

Git clone source code

#Create a directory to place the application (application root)
mkdir try_nginx_on_docker

#Go to application root
cd $_

#Get source code
git clone https://github.com/naokit-dev/try_nginx_on_docker.git

#Move source code to application root
cp -a try_nginx_on_docker/. .
rm -rf try_nginx_on_docker

I think it will have the following configuration

.(try_nginx_on_docker)
├── Dockerfile
├── Gemfile
├── Gemfile.lock
├── README.md
├── docker
│   └── nginx
│       ├── default.conf
│       ├── load_balancer.conf
│       └── static.conf
├── docker-compose.prod.yml
├── docker-compose.yml
├── entrypoint.sh
├── setup.sh
└── temp_files
    ├── copy_application.html.erb
    ├── copy_database.yml
    └── copy_environment.js

Part of the source code docker-compose.yml 4 containers are defined

version: "3.8"

services:
  web:
    image: nginx:1.18
    ports:
      - "80:80"
    volumes:
      - ./docker/nginx/static.conf:/etc/nginx/conf.d/default.conf
      - public:/myapp/public
      - log:/var/log/nginx
      - /var/www/html
    depends_on:
      - app

  db:
    image: postgres:11.0-alpine
    volumes:
      - postgres:/var/lib/postgresql/data:cached
    ports:
      - "5432:5432"
    environment:
      PGDATA: /var/lib/postgresql/data/pgdata
      POSTGRES_USER: ${POSTGRES_USER:-postgres}
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-password}
      POSTGRES_INITDB_ARGS: "--encoding=UTF-8 --locale=ja_JP.UTF-8"
      TZ: Asia/Tokyo

  app:
    build:
      context: .
    image: rails_app
    tty: true
    stdin_open: true
    command: bash -c "rm -f tmp/pids/server.pid && ./bin/rails s -p 3000 -b '0.0.0.0'"
    volumes:
      - .:/myapp:cached
      - rails_cache:/myapp/tmp/cache:cached
      - node_modules:/myapp/node_modules:cached
      - yarn_cache:/usr/local/share/.cache/yarn/v6:cached
      - bundle:/bundle:cached
      - public:/myapp/public
      - log:/myapp/log
      - /myapp/tmp/pids
    tmpfs:
      - /tmp
    ports:
      - "3000-3001:3000"
    environment:
      RAILS_ENV: ${RAILS_ENV:-development}
      NODE_ENV: ${NODE_ENV:-development}
      DATABASE_HOST: db
      DATABASE_PORT: 5432
      DATABASE_USER: ${POSTGRES_USER}
      DATABASE_PASSWORD: ${POSTGRES_PASSWORD}
      WEBPACKER_DEV_SERVER_HOST: webpacker
    depends_on:
      - db
      - webpacker

  webpacker:
    image: rails_app
    command: ./bin/webpack-dev-server
    volumes:
      - .:/myapp:cached
      - public:/myapp/public
      - node_modules:/myapp/node_modules:cached
    environment:
      RAILS_ENV: ${RAILS_ENV:-development}
      NODE_ENV: ${NODE_ENV:-development}
      WEBPACKER_DEV_SERVER_HOST: 0.0.0.0
    tty: false
    stdin_open: false
    ports:
      - "3035:3035"

volumes:
  rails_cache:
  node_modules:
  yarn_cache:
  postgres:
  bundle:
  public:
  log:
  html:

Environment

source setup.sh

If the setup is completed normally After launching the container with the following command in the application root directory

docker-compose up

#When starting in the background-d option
docker-compose up　-d

When you access localhost or localhost: 80 from your browser

You can see Yay! You ’re on Rails! ** Anyone can easily build a development environment! The first advantage of Docker **

Let's check the container running here (If you do docker-compose up without the -d option, a new terminal will be opened. For VS Code, control +@\ * mac environment)

docker ps

Just make sure you have four containers running: web (Nginx), app (Rails), webpacker (webpack-dev-server), db (PostgreSQL)

If you can confirm it, close the container once

docker-compose down

Try delivering static content with Nginx

I don't use Rails app yet Here we will challenge the following

--Check the minimum setting of Nginx

--Try starting the container alone (nginx only) while using Docker-compose

--Try delivering simple static content (HTML) with Nginx alone

The simplest Nginx configuration

Edit docker-compose.yml to change Nginx settings

services:
  web:
    image: nginx:1.18
    ports:
      - "80:80"
    volumes:
    #Rewrite here./docker/nginx/default.conf... -> ./docker/nginx/static.conf...
      - ./docker/nginx/static.conf:/etc/nginx/conf.d/default.conf
      - public:/myapp/public
      - log:/var/log/nginx
    depends_on:
      - app
...

** A little about Docker volume ** ./docker/nginx/static.conf:/etc/nginx/conf.d/default.conf ofvolumes:is<host side path>: <container side path> This mounts static.conf on the host (local) side as a volume so that it can be treated as default.conf in the container. Just keep in mind that ** a volume mount like this is needed ** as the storage behaves independently on the host side and inside the container.

The setting of Nginx is described in docker / nginx / static.conf, and the contents are as follows.

server { #From here
      listen 80;　# must
      server_name _; # must
      root /var/www/html; # must
      index index.html;
      access_log /var/log/nginx/access.log;
      error_log  /var/log/nginx/error.log;
}　#One server block to here=Specifications of one virtual server handled by Nginx

** server **: Define a virtual server based on the contents enclosed in "{}" (server block)

Here, the following 3 items must be set

** listen **: Specify the listening IP and port (xxx.xxx.xxx.xxx:80, localhost: 80, 80) ** server_name **: The name assigned to the virtual server. Nginx searches for virtual servers that match the host name (example.com) or IP (xxx.xxx.xxx.xxx) included in the request. ("_" Means to match all conditions. Other wildcards and regular expressions can be used) ** root **: Specifies the document root, the directory where the content is located

By the way, for log, the same path as above is defined in the file ʻetc / nginx / nginx.conf, so Even if there is no description here, both error_log and access_log should be recorded under / var / log / nginx / For example, by setting ʻaccess_log /var/log/nginx/static.access.log;, it seems that it is possible to record the log specific to the relevant virtual server (server block).

Start Nginx container alone

In the previous docker-compose up, all containers of nginx, rails, webpack-dev-server, db are started, but you can also start only a specific container by using the docker-compose option. Is possible

--no-deps: Start ignoring dependencies between containers (here web: nginx only) -d: Start the container in the background, the shell can continue typing -p: Port mapping \ : \ web: nginx container defined by compose

Start the Nginx container with the following command

docker-compose run --no-deps -d -p 80:80 web

(Port mapping is also specified in compose, but it is necessary to specify it again, and port 80 on the host side is mapped to port 80 in the web container)

Check the difference from running docker-compose with no options

docker ps

Unlike before, I think only the nginx container is running

Create HTML content

Call the shell inside the container

docker-compose run --no-deps web bash

Below is the work inside the web container

# index.Create html
touch /var/www/html/index.html

# index.Add the contents of html
echo "<h1>I am Nginx</h1>" > /var/www/html/index.html

# index.Check html
cat /var/www/html/index.html
<h1>I am Nginx</h1>

Now that you have created ʻindex.html directly under the document root in the container, let's close the shell with ʻexit.

Operation check

When you access localhost from your browser, You can see that it is delivered as HTML as shown below.

** Nginx here has a simple behavior of searching the document root for content that matches the request and returning the match **

If you can confirm it, close the container once

docker-compose down

Appendix --Nginx behavior when no virtual server matches the request

--Understanding the concept of Nginx's default server

Nginx defines which virtual server to route to based on the information in the Host field included in the request from the client.

** So what happens if none of the virtual servers match the request? ** ** It seemed to be important when thinking about the design, so let's check it here.

In the server block of the previous config file, I defined server name to be applicable to any request, but I will rewrite it with a random name that does not match the request

server_name undefined_server;

Start the Nginx container again

docker-compose run --no-deps -p 80:80 web

When I access localhost from a browser, even though there is no virtual server that matches the request Unexpectedly, you should see the same "I am Nginx" as before

default server

Nginx is used to process with the default server when the request does not correspond to any virtual server, and it is specified to treat the virtual server described at the top as the default server at the very beginning.

In the configuration above, the default server is the first one — which is nginx’s standard default behaviour. It can also be set explicitly which server should be default, with the default_server parameter in the listen directive: How nginx processes a request

Alternatively, you can explicitly specify default_server in the listen directive.

listen      80 default_server;

In this experiment, "undefined_server" doesn't match the request, but there is nothing else to match. Probably routed as default server

** If none of the virtual servers match the request => routed to default server **

I feel that it helps to isolate the error, such as when it does not connect to the backend server well.

Let's close the container once

docker-compose down

appendix --A little understanding of Docker volume

--Learn about container independence --Containers --- Learn about the difference between volume as a mechanism for sharing (persistently sharing) storage between containers, especially named volume and anonymous volume.

** About the significance that volume is necessary (= persistence is necessary) in the first place ** Docker creates and manages volumes to persist the data in the container

I'm not sure so I'll check it

Call the shell inside the web container

docker-compose run --no-deps web bash

Below is the work inside the web container

#Create a directory for verification
mkdir /var/www/test

#Create a file for verification
touch /var/www/test/index.html

#check existence
ls /var/www/test/

The point is that / var / www / test is now a path in docker-compose.yml that is not managed as a volume.

Let's close the shell once with ʻexit` (the container will also be closed)

Start the web container again and call the shell

docker-compose run --no-deps web bash

I will try to find the previous file

cat /var/www/test/index.html

ls /var/www

What do you think, I don't think you can find either the directory /var/www/test or the file /var/www/test/index.html

** When the container is closed, the data in the container is not retained ** This is the principle Volumes are available to circumvent this mechanism

Close the terminal with ʻexit`

Stop all containers

docker-compose down

** Volume type **

There are the following types of volumes in Docker, but they are the same in that they persist the data in the container.

1. host volume? (I don't know the name)
2. anonymous volume (I feel like I'm going through anonymous volume? Anonymous volume)
3. named volume

Take a look at docker-compose.yml

version: "3.8"

services:
  web:
    image: nginx:1.18
    ports:
      - "80:80"
    volumes:
      - ./docker/nginx/static.conf:/etc/nginx/conf.d/default.conf #host volume
      - public:/myapp/public # named volume
      - log:/var/log/nginx # named volume
      - html:/var/www/html # named volume

...

volumes: #Define sharing between different containers here
  public:
  log:
  html:

host volume I mentioned it in the nginx settings part Mount the host side path as a volume at ./docker/nginx/static.conf:/etc/nginx/conf.d/default.conf ** Image of copying files in the host to the container side **

named volume html: / var / www / html part I am mounting the volume with the name "html" In addition, by defining it with this name in the "volumes" block, which is in the same row as the "services" block. ** Allows sharing of volumes between multiple containers **

And ** this volume is persisted independently of the host side **

Finally ** anonymous volume ** In the official doc, the only difference from named volume is ** with or without a name **. Rather than actually having no name, ** the part corresponding to the name of the named volume is given as a hash for each container **. It's a bit confusing, but it's possible to use it when you don't need to mount the host side, but you need to persist it ** and you don't expect sharing in multiple containers **. (It's still hard to imagine, but in the content that follows, I'll come across a scene where it has to be anonymous volume)

Here, I will verify it for a little deeper understanding. Change / var / www / html, which was originally defined as named volume, to anonymous volume I will repeat the procedure for creating an HTML file performed in this section.

docker-compose.yml

version: "3.8"

services:
  web:
    image: nginx:1.18
    ports:
      - "80:80"
    volumes:
      - ./docker/nginx/static.conf:/etc/nginx/conf.d/default.conf
      - public:/myapp/public
      - log:/var/log/nginx
      - /var/www/html #Specify only the path on the container side and change to anonymous volume

...

volumes:
  public:
  log:
  # html:Comment out here

Start Nginx as a web server

docker-compose run --no-deps -d -p 80:80 web

Call the shell

docker-compose run --no-deps web bash

This is important, but if you look at the container that is currently running in another terminal

docker ps

** You can see that the container with two containers running and the shell running is different from the container with port mapping at 80:80 **

Create HTML in the container as it is

# index.Create html
touch /var/www/html/index.html

# index.Confirm the existence of html
ls /var/www/html

Let's access localhost from the browser as before

The browser then shows a 403 error If you check the Nginx error log

tail -f 20 /var/log/nginx/error.log

...directory index of "/var/www/html/" is forbidden...

An error is logged if the directory cannot be found

By changing to named volume-> anonymous volume ** The contents under / var / www / html / will not be shared between the two containers ** Since the container that received the request on port 80 from the local can no longer refer to ʻindex.html` It is possible that such an error has occurred

** Persistence, but do not share volume with other containers **, I think you could touch the characteristics

If you can confirm, close the shell with ʻexit`

Let's close the container every time

docker-compose down

(The changed contents of docker-compose.yml can be left as it is)

...

The contents of the appendix got hotter than I expected and it became longer, so I will break it here (to keep my motivation)

Continue to ②