[PYTHON] Rebuild Django's development environment with Docker! !! !! !!

Introduction

I wrote this article before. Build a Django development environment with Docker! (Docker-compose / Django / postgreSQL / nginx)

The original article was written a long time ago (about a year?), But now that I have accumulated some knowledge, I thought I could make it a little better, so I decided to take on the challenge!

I will not talk about the details such as installing Docker, so if you are interested, please see the original article! (Here I think something will be helpful.)

Also, as a clear difference from the previous article, Django thinks as an API distribution server and assumes Vue.js and React for the front end implementation, so static file distribution and template settings etc. are done. Please note that we do not do anything like creating docker-compose.yml and docker-compose.yml.prod to separate the production environment from the development environment.

table of contents

-[Table of Contents](#Table of Contents) -[Directory structure](# directory structure) -[Building Django environment](#django environment building) -[Building PostgreSQL environment](Building #postgresql environment) -[Try using docker-compose](Try using # docker-compose) -[Building nginx environment](#nginx environment building) -[Add nginx to docker-compose.yml](Add nginx to # docker-composeyml)

Directory structure

In the end, the directory structure looks like this!

backend
└── containers
    ├── django
    │   ├── Dockerfile
    │   ├── Pipfile
    │   ├── Pipfile.lock
    │   ├── config
    │   │   ├── __init__.py
    │   │   ├── settings.py
    │   │   ├── urls.py
    │   │   └── wsgi.py
    │   ├── entrypoint.sh
    │   ├── manage.py
    ├── docker-compose.yml
    ├── nginx
    │   ├── Dockerfile
    │   └── nginx.conf
    └── postgres
        ├── Dockerfile
        └── sql
            └── init.sql

Each container is placed in the containers directory under the backend directory.

The postgres directory has been added compared to the previous article.

Now let's start with the top django directory!

Build a Django environment

First, let's create three directories: backend, containers, and django! Then change to the django directory.

$mkdir -p backend/containers/django
$cd backend/containers/django

First, let's create a Pipfile in the django directory. The version of each package is the latest version at the time of writing this article, and there is no particular preference. Feel free to change it. (Operation is not guaranteed)

Also, if the package to install is arbitrary and the purpose is only to build the environment, there is no problem if only django, gunicorn and django-environ are included.

backend/containers/django/Pipfile

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]

[packages]
django = "==3.0.6"
djangorestframework = "==3.11.0"
djangorestframework-simplejwt = "==4.4.0"
djangorestframework-gis = "==0.15"
django-cors-headers = "==3.2.1"
django-environ = "==0.4.5"
djoser = "==2.0.3"
gunicorn = "==20.0.4"
psycopg2-binary = "==2.8.5"

[requires]
python_version = "3.8.2"

Let's create a virtual environment with pipenv with python3.8 series, enter the virtual environment and check the version and installed packages.

$pipenv install
$pipenv shell
(django) $python -V
Python 3.8.2
(django) $pip list
Package                       Version
----------------------------- -------
asgiref                       3.2.7
Django                        3.0.6
django-cors-headers           3.2.1
django-environ                0.4.5
django-templated-mail         1.1.1
djangorestframework           3.11.0
djangorestframework-gis       0.15
djangorestframework-simplejwt 4.4.0
djoser                        2.0.3
gunicorn                      20.0.4
pip                           20.0.2
psycopg2-binary               2.8.5
PyJWT                         1.7.1
pytz                          2020.1
setuptools                    46.1.3
sqlparse                      0.3.1
wheel                         0.34.2

If you see the word (django) on the left edge of the shell, you're in a virtual environment. (For the sake of simplicity, the character string of (django) will not be described from the next time onwards)

You have Django installed properly!

Once you've verified that you're in the django directory, use Django's commands to create your project.

$django-admin startproject config .

This completes the Django project creation. You should see a config directory and a file named manage.py created in your django directory.

At this point, start the debug server with the python manage.py runserver localhost: 8000 command, connect to loaclhost: 8000 from your browser, and you should see that screen rather than your parent's face.

Next, let's modify the files around Django as follows.

Please comment out SECRET_KEY as it will be used later.

backend/containers/django/config/settings.py

import os
from datetime import timedelta
import environ

env = environ.Env()
env.read_env('.env')

BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))

SECRET_KEY = env.get_value('SECRET_KEY')

DEBUG = env.get_value('DEBUG')

ALLOWED_HOSTS = ['localhost', '127.0.0.1']

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',

    'rest_framework',
    'rest_framework_gis',
    'corsheaders',
    'django.contrib.gis',
]

MIDDLEWARE = [
    'corsheaders.middleware.CorsMiddleware',
    'django.middleware.security.SecurityMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.common.CommonMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',
    'django.middleware.clickjacking.XFrameOptionsMiddleware',
]

ROOT_URLCONF = 'config.urls'

TEMPLATES = [
    {
        'BACKEND': 'django.template.backends.django.DjangoTemplates',
        'DIRS': [os.path.join(BASE_DIR, 'templates')],
        'APP_DIRS': True,
        'OPTIONS': {
            'context_processors': [
                'django.template.context_processors.debug',
                'django.template.context_processors.request',
                'django.contrib.auth.context_processors.auth',
                'django.contrib.messages.context_processors.messages',
            ],
        },
    },
]

WSGI_APPLICATION = 'config.wsgi.application'

DATABASES = {
    'default': {
        'ENGINE': env.get_value('DATABASE_ENGINE', default='django.db.backends.sqlite3'),
        'NAME': env.get_value('DATABASE_DB', default=os.path.join(BASE_DIR, 'db.sqlite3')),
        'USER': env.get_value('DATABASE_USER', default='django_user'),
        'PASSWORD': env.get_value('DATABASE_PASSWORD', default='password'),
        'HOST': env.get_value('DATABASE_HOST', default='localhost'),
        'PORT': env.get_value('DATABASE_PORT', default='5432'),
    }
}

AUTH_PASSWORD_VALIDATORS = [
    {
        'NAME': 'django.contrib.auth.password_validation.UserAttributeSimilarityValidator',
    },
    {
        'NAME': 'django.contrib.auth.password_validation.MinimumLengthValidator',
    },
    {
        'NAME': 'django.contrib.auth.password_validation.CommonPasswordValidator',
    },
    {
        'NAME': 'django.contrib.auth.password_validation.NumericPasswordValidator',
    },
]

LANGUAGE_CODE = 'ja'

TIME_ZONE = 'Asia/Tokyo'

USE_I18N = True

USE_L10N = True

USE_TZ = True

STATIC_URL = '/static/'
STATICFILES_DIRS = (os.path.join(BASE_DIR, 'static'),)

REST_FRAMEWORK = {
    'DEFAULT_AUTHENTICATION_CLASSES': [
        'rest_framework_simplejwt.authentication.JWTAuthentication',
    ],
}

SIMPLE_JWT = {
    'AUTH_HEADER_TYPES': ('JWT',),
    'ACCESS_TOKEN_LIFETIME': timedelta(minutes=30),
}

CORS_ORIGIN_ALLOW_ALL = False
CORS_ORIGIN_WHITELIST = (
    'http://localhost:8080',
    'http://127.0.0.1:8080',
)

When I first created the Pipfile, I added a package called django-environ, which describes the information I want to hide, such as SECRET_KEY and DB connection information, in the .env file, etc., and the source code. It is a package for managing separately from setting.py etc.

env = environ.Env()
env.read_env('.env')

Use it like this.

Let's create a .env file right away.

backend/containers/django/.env

DEBUG=True
SECRET_KEY=<YOUR_SECRET_KEY>
DATABASE_ENGINE=django.contrib.gis.db.backends.postgis
DATABASE_DB=<YOUR_DB_NAME>
DATABASE_USER=<YOUR_DB_USER>
DATABASE_PASSWORD=<YOUR_DB_PASSWORD>
#entrypoint.Used in sh
#compose.It will resolve the name with the service name described in yml
DATABASE_HOST=postgres
DATABASE_PORT=5432
DATABASE=postgres

-<YOUR_SECRET_KEY>: KEY originally described in setting.py -<YOUR_DB_NAME> (USER / PASSWORD): Please enter any one. Details will be explained when launching the PostgreSQL container.

After that, I will fix the files related to Gorigori. Since the path from the backend directory is also described, create, add, and modify necessary files as appropriate.

backend/containers/django/Dockerfile

#Pull the ubuntu image and install Python
FROM ubuntu:20.04

SHELL ["/bin/bash", "-c"]

#install python
RUN apt-get update -y \
    && apt-get upgrade -y \
    && apt-get install -y python3.8 python3.8-dev \
    && source ~/.bashrc \
    && apt-get -y install vim

#Set working directory
WORKDIR /usr/src/app

#Set environment variables
#Prevent Python from writing to pyc files and discs
ENV PYTHONDONTWRITEBYTECODE 1
#Prevent Python from buffering standard I / O
ENV PYTHONUNBUFFERED 1
ENV DEBIAN_FRONTEND=noninteractive

#Dependency installation and pipenv installation
RUN apt-get install -y curl \
    && curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py \
    && apt-get install -y python3.8-distutils \
    && python3.8 get-pip.py \
    && pip install -U pip \
    && apt-get install -y build-essential libssl-dev libffi-dev python-dev python3-dev libpq-dev

#Installation of pipenv
RUN pip install pipenv

#Copy the Pipfile from your local machine to the container
COPY Pipfile ./

# Pipfile.Ignore the lock and install the packages listed in the Pipfile on your system
#Then uninstall pipenv
RUN pipenv install --system --skip-lock \
    && pip uninstall -y pipenv virtualenv-clone virtualenv

#Dependencies when installing geospatial libraries
RUN apt-get update -y \
    && apt-get upgrade -y \
    && apt-get install -y libgeos-dev binutils libproj-dev gdal-bin libgdal-dev \
    && apt-get install -y python3-gdal

RUN apt-get install -y netcat \
    && apt-get install -y expect

#Copy shell script
# COPY ./entrypoint.sh /usr/src/app/entrypoint.sh

COPY . /usr/src/app/

#Run shell script
# ENTRYPOINT ["/usr/src/app/entrypoint.sh"]

Let's see if the container works at this stage.

$docker build . -t pipenv_sample
...
$docker run -it pipenv_sample
root@e6bdfb335bee:/usr/src/app#

If you can log in to the container as root user like this, you're probably successful!

# python3 -V
Python 3.8.2
# pip list
Package                       Version
----------------------------- ----------
appdirs                       1.4.3
asgiref                       3.2.7
certifi                       2020.4.5.1
distlib                       0.3.0
Django                        3.0.6
django-cors-headers           3.2.1
django-environ                0.4.5
django-templated-mail         1.1.1
djangorestframework           3.11.0
djangorestframework-gis       0.15
djangorestframework-simplejwt 4.4.0
djoser                        2.0.3
filelock                      3.0.12
GDAL                          3.0.4
gunicorn                      20.0.4
numpy                         1.17.4
pip                           20.1
pipenv                        2018.11.26
psycopg2-binary               2.8.5
PyJWT                         1.7.1
pytz                          2020.1
setuptools                    46.1.3
six                           1.14.0
sqlparse                      0.3.1
virtualenv                    20.0.20
virtualenv-clone              0.5.4
wheel                         0.34.2

The Python version is 3.8.2 and the packages are installed properly!

Log out with Control + d.

Finally, let's write and save a shell script for connecting to the postgres container that runs at startup.

backend/containers/django/entrypoint.sh

#!/bin/sh

if [ "$DATABASE" = "postgres" ]
then
    echo "Waiting for postgres..."

    while ! nc -z $DATABASE_HOST $DATABASE_PORT; do
      sleep 0.1
    done

    echo "PostgreSQL started"
fi

exec "$@"

By the way, uncomment the two comments near the last line of the Dockerfile you edited above.

backend/containers/django/Dockerfile

#this
COPY ./entrypoint.sh /usr/src/app/entrypoint.sh

COPY . /usr/src/app/

#this
ENTRYPOINT ["/usr/src/app/entrypoint.sh"]

That's it for the Django directory.

--Directory structure

tree

backend
└── containers
    ├── django
    │   ├── Dockerfile
    │   ├── Pipfile
    │   ├── Pipfile.lock
    │   ├── config
    │   │   ├── __init__.py
    │   │   ├── settings.py
    │   │   ├── urls.py
    │   │   └── wsgi.py
    │   ├── entrypoint.sh
    │   ├── manage.py
    └── docker-compose.yml

Building a PostgreSQL environment

Next, we will launch the PostgreSQL container.

First, move to the containers directory and create a postgres directory, etc.

$cd ../../
$mkdir -p postgres/sql/

You should now have a postgres directory and a sql directory inside it.

Let's add a Dockerfile inside the postgres directory.

backend/containers/postgres/Dockerfile

FROM mdillon/postgis:11

RUN localedef -i ja_JP -c -f UTF-8 -A /usr/share/locale/locale.alias ja_JP.UTF-8
ENV LANG ja_JP.UTF-8

In my case, I wanted to create a map application, so I will use the PostGIS image, which is an extension of PostgreSQL, but I don't mind any PostgreSQL image. (Operation has not been verified)

Next, store the sql file you want to execute when the container starts in the sql directory.

backend/containers/postgres/sql/init.sql

CREATE EXTENSION postgis;

This time, I stored only the sql file for enabling the extension, but if you have any data you want to initially register, please feel free to store it.

Finally, add the .env_db file.

In it, write the same <YOUR_DB_NAME> (USER / PASSWORD) that you wrote when you created the Django container.

A DB will be created automatically with the contents described here.

backend/containers/postgres/.env_db

#If you write in env, DB will be created automatically
POSTGRES_DB=<YOUR_DB_NAME>
POSTGRES_USER=<YOUR_DB_USER>
POSTGRES_PASSWORD=<YOUR_DB_PASSWORD>

This completes the construction of the postgres environment.

--Directory structure

backend
└── containers
    ├── django
    │   ├── Dockerfile
    │   ├── Pipfile
    │   ├── Pipfile.lock
    │   ├── config
    │   │   ├── __init__.py
    │   │   ├── settings.py
    │   │   ├── urls.py
    │   │   └── wsgi.py
    │   ├── entrypoint.sh
    │   ├── manage.py
    ├── docker-compose.yml
    └── postgres
        ├── Dockerfile
        └── sql
            └── init.sql

Try using docker-compose

docker-compose is a convenient tool to use when starting multiple containers at the same time and connecting them.

Let's move to the containers directory and create a configuration file.

$cd ../
$touch docker-compose.yml

backend/containers/docker-compose.yml

version: "3.7"
services:
  django:
    #Container name
    container_name: django
    #Directory containing docker files to build
    build: ./django
    #Command to be executed after normal startup
    command: python3 manage.py runserver 0.0.0.0:8000
    volumes:
      #Directory to mount
      - ./django:/usr/src/app/
    ports:
      #Host side port: Container side port
      - 8000:8000
    env_file:
      #File to set in environment variable
      - ./django/.env
    depends_on:
      #Service to connect
      - postgres

  postgres:
    container_name: postgres
    build: ./postgres
    volumes:
      #DB data is saved by creating a volume
      #You may take the directory and mount and leave the actual data directly on the host OS
      # /var/lib/postgresql/DB data is stored in data
      - sample_postgis_data:/var/lib/postgresql/data
      # down -Specify the file to be executed at the first startup including when there is no volume with v etc.
      - ./postgres/sql:/docker-entrypoint-initdb.d
    env_file: ./postgres/.env_db
    ports:
      #The port on the host side batting with the local psql, so set it to something other than 5432.
      - 5433:5432

volumes:
  sample_postgis_data:

After writing, let's launch docker-compose.

$docker-compose up -d --build

Let's connect to localhost: 8000 from a browser.

If you see the initial Django screen from your brother's face, you're good to go.

You can log in to the container with the following command.

$docker exec -it <Service name> bash

In other words, in this case, you can connect with the following.

$docker exec -it django bash
Or
$docker exec -it postgres bash

Now that we've confirmed that the django container has started, let's also check the postgres container.

$docker exec -it postgres bash
#psql -U <YOUR_DB_USER> -d <YOUR_DB_NAME>
psql (11.2 (Debian 11.2-1.pgdg90+1))
"help"Get help with.

<YOUR_DB_NAME>=#
<YOUR_DB_NAME>=# SELECT postgis_version();
            postgis_version
---------------------------------------
 2.5 USE_GEOS=1 USE_PROJ=1 USE_STATS=1
(1 line)

I was able to confirm that it was created with the one specified by the DB and that postgis is also enabled.

Stop the container and delete the image with the following command.

$docker-compose down -v

Building nginx environment

Finally, let's build the nginx environment.

First, create an nginx directory and create a Dockerfile as well.

$mkdir nginx
$cd nginx/
$touch Dockerfile

backend/containers/nginx/Dockerfile

FROM nginx:1.17.10

RUN rm /etc/nginx/conf.d/default.conf
COPY nginx.conf /etc/nginx/conf.d

next

backend/containers/nginx/nginx.conf

upstream config {
    #If you specify the service name of the container, the name will be resolved.
    server django:8000;
}

server {
    #Stand by on port 80
    listen 80;

    location / {
        proxy_pass http://config;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $host;
        proxy_redirect off;
    }

    #Statically route static file requests
    location /static/ {
        alias /usr/src/app/static/;
    }
}

I think you have all the necessary files.

Add nginx to docker-compose.yml

Add the nginx service as shown below and change the command of the django service from runserver to gunicorn.

backend/containers/docker-compose.yml

version: "3.7"
services:
  django:
    #Container name
    container_name: django
    #Directory containing docker files to build
    build: ./django
    #Command to be executed after normal startup
    command: gunicorn config.wsgi:application --bind 0.0.0.0:8000
    volumes:
      #Directory to mount
      - ./django:/usr/src/app/
    ports:
      #Host side port: Container side port
      - 8000:8000
    env_file:
      #File to set in environment variable
      - ./django/.env
    depends_on:
      #Service to connect
      - postgres

  postgres:
    container_name: postgres
    build: ./postgres
    volumes:
      #DB data is saved by creating a volume
      #You may take the directory and mount and leave the actual data directly on the host OS
      # /var/lib/postgresql/DB data is stored in data
      - sample_postgis_data:/var/lib/postgresql/data
      # down -Specify the file to be executed at the first startup including when there is no volume with v etc.
      - ./postgres/sql:/docker-entrypoint-initdb.d
    env_file: ./postgres/.env_db
    ports:
      #The port on the host side batting with the local psql, so set it to something other than 5432.
      - 5433:5432

  nginx:
    container_name: nginx
    build: ./nginx
    volumes:
      - ./django/static:/usr/src/app/static
    ports:
      - 80:80
    depends_on:
      - django

volumes:
  sample_postgis_data:

Let's start the container.

$docker-compose up -d --build

If you can confirm that the nginx container can deliver from port 80, the environment construction is complete.

Let's connect to localhost.

You can see that screen seen from my sister's face.

Thank you for your hard work. After that, boil or bake, please do whatever you like!

--Final directory structure (same as the one mentioned at the beginning)

backend
└── containers
    ├── django
    │   ├── Dockerfile
    │   ├── Pipfile
    │   ├── Pipfile.lock
    │   ├── config
    │   │   ├── __init__.py
    │   │   ├── settings.py
    │   │   ├── urls.py
    │   │   └── wsgi.py
    │   ├── entrypoint.sh
    │   ├── manage.py
    ├── docker-compose.yml
    ├── nginx
    │   ├── Dockerfile
    │   └── nginx.conf
    └── postgres
        ├── Dockerfile
        └── sql
            └── init.sql