Searching for an efficient way to write a Dockerfile in Python with poetry

Introduction of how to reduce the size of Docker image for machine learning in Python A very useful M3 blog twitter I saw it at //twitter.com/m3_engineering/status/1306859277131427840), so I'll take a piggyback ride and briefly write down the difference from the Dockerfile I'm using. This is a Dockerfile for dashboard, not for machine learning, so please take a look at it as a reference. I don't think the following content is excellent, so I would appreciate it if someone with a better idea could give me a tsukkomi. And as always, I hope someone on the other side of the internet will help me. In addition, streamlit is very convenient.

Conclusion

Same part

--Version control with poetry --Use multi-stage build --docker build works with cache on dev, prod

Different part

--Since builder does not use apt install, builder also uses slim image --Use poetry export instead of poetry install (do not put poetry in the executable image). Then create requirements.txt and requirements-dev.txt. --Switch between dev and prod with docker-compose.yml --If you open it with remote-container using vscode, you don't need python on the development environment side.

code

build

Not long ago, I defined the environment variable as STAGE = in .env, but I stopped it because rewriting is troublesome.

# dev build
STAGE=dev docker-compose build
# prod build
docker-compose up build

Dockerfile

FROM python:3.8.5-slim as builder

WORKDIR /work/src
RUN pip install --upgrade pip && pip install poetry
COPY pyproject.toml poetry.lock ./
RUN poetry export --without-hashes -f requirements.txt > requirements.txt
RUN poetry export --without-hashes --dev -f requirements.txt > requirements-dev.txt
RUN pip install -r requirements.txt

### for dev. 
FROM python:3.8.5-slim as dev
ENV PYTHONUNBUFFERED=1
WORKDIR /work/src
COPY --from=builder /usr/local/lib/python3.8/site-packages /usr/local/lib/python3.8/site-packages
# need to copy if python package is installed in /usr/local/bin
COPY --from=builder /usr/local/bin/streamlit /usr/local/bin/streamlit
COPY src/ ./
# for dev packages
COPY --from=builder /work/src/requirements-dev.txt requirements-dev.txt
RUN pip install -r requirements-dev.txt
COPY tests/ ./
EXPOSE 8501  
CMD ["streamlit", "run", "hello.py"]

### for prod
FROM python:3.8.5-slim as prod
ENV PYTHONUNBUFFERED=1
WORKDIR /work/src
COPY --from=builder /usr/local/lib/python3.8/site-packages /usr/local/lib/python3.8/site-packages
# need to copy if python package is installed in /usr/local/bin
COPY --from=builder /usr/local/bin/streamlit /usr/local/bin/streamlit
COPY src/ ./
EXPOSE 8501  
CMD ["streamlit", "run", "hello.py"]

docker-compose.yml

version: '3'

services:
  app:
    build: 
      context: .
      target: ${STAGE:-prod}
    image: "internal-dashboard_app_${STAGE:-prod}"
    container_name: "internal-dashboard_${STAGE:-prod}"
    volumes:
      - ./src:/work/src
    ports:
      - "8501:8501"
    restart: always
    command: ["streamlit", "run", "hello.py"]

Impressions

I'm always wondering how to deal with the path problem in tests. that's all.

Recommended Posts

Searching for an efficient way to write a Dockerfile in Python with poetry
A simple way to avoid multiple for loops in Python
How to create a heatmap with an arbitrary domain in Python
A clever way to time processing in Python
I want to write to a file with Python
How to write what to do when an application is first displayed in Qt for Python with Designer
How to convert / restore a string with [] in python
I want to write in Python! (2) Let's write a test
I want to work with a robot in python.
A program to write Lattice Hinge with Rhinoceros with Python
For those who want to write Python with vim
A real way for people using python 3.8.0-2 from windows to work with multibyte characters
Created a Python library to write complex comprehensions and reduce in an easy-to-read manner
Write to csv with Python
How to convert an array to a dictionary with Python [Application]
Create a child account for connect with Stripe in Python
Probably the easiest way to create a pdf with Python3
Experiment to make a self-catering PDF for Kindle with Python
[Python] How to write an if statement in one sentence.
How to define multiple variables in a python for statement
How to make a Python package (written for an intern)
A standard way to develop and distribute packages in Python
Introducing a good way to manage DB connections in Python
Turn an array of strings with a for statement (Python3)
Recursively get the Excel list in a specific folder with python and write it to Excel.
It's a hassle to write "coding: utf-8" in Python, so I'll do something with Shellscript
I created an environment for Masonite, a Python web framework similar to Laravel, with Docker!
Seeking a unified way to wait and get for state changes in Selenium for Python elements
An easy way to view the time taken in Python and a smarter way to improve it
Write a binary search in Python
How to write a Python class
[Python] Write to csv file with Python
Write an HTTP / 2 server in Python
Write A * (A-star) algorithm in Python
Write a pie chart in Python
Write a vim plugin in Python
Write a depth-first search in Python
An introduction to Python for non-engineers
An alternative to `pause` in Python
Write a batch script with Python3.5 ~
How to make a string into an array or an array into a string in Python
A game to go on an adventure in python interactive mode
Try to extract a character string from an image with Python3
[Introduction to Python] How to use the in operator in a for statement?
[For beginners] How to register a library created in Python in PyPI
How to use an external editor for Python development with Grasshopper
Dockerfile with the necessary libraries for natural language processing in python
Write documentation in Sphinx with Python Livereload
Spiral book in Python! Python with a spiral book! (Chapter 14 ~)
Try logging in to qiita with Python
Memo to ask for KPI with python
Write the test in a python docstring
Send an email to a specific email address with python without SMTP settings
Write a C language linked list in an object-oriented style (with code comparison between Python and Java)
How to work with BigQuery in Python
Write a short property definition in Python
Read a Python # .txt file for a super beginner in Python with a working .py
[Python] Create a linebot to write a name and age on an image
How to get a stacktrace in python
Easy way to use Wikipedia in Python
Write a Caesar cipher program in Python