Searching for an efficient way to write a Dockerfile in Python with poetry

Introduction of how to reduce the size of Docker image for machine learning in Python A very useful M3 blog twitter I saw it at //twitter.com/m3_engineering/status/1306859277131427840), so I'll take a piggyback ride and briefly write down the difference from the Dockerfile I'm using. This is a Dockerfile for dashboard, not for machine learning, so please take a look at it as a reference. I don't think the following content is excellent, so I would appreciate it if someone with a better idea could give me a tsukkomi. And as always, I hope someone on the other side of the internet will help me. In addition, streamlit is very convenient.

Conclusion

Same part

--Version control with poetry --Use multi-stage build --docker build works with cache on dev, prod

Different part

--Since builder does not use apt install, builder also uses slim image --Use poetry export instead of poetry install (do not put poetry in the executable image). Then create requirements.txt and requirements-dev.txt. --Switch between dev and prod with docker-compose.yml --If you open it with remote-container using vscode, you don't need python on the development environment side.

code

build

Not long ago, I defined the environment variable as STAGE = in .env, but I stopped it because rewriting is troublesome.

# dev build
STAGE=dev docker-compose build
# prod build
docker-compose up build

Dockerfile

FROM python:3.8.5-slim as builder

WORKDIR /work/src
RUN pip install --upgrade pip && pip install poetry
COPY pyproject.toml poetry.lock ./
RUN poetry export --without-hashes -f requirements.txt > requirements.txt
RUN poetry export --without-hashes --dev -f requirements.txt > requirements-dev.txt
RUN pip install -r requirements.txt

### for dev. 
FROM python:3.8.5-slim as dev
ENV PYTHONUNBUFFERED=1
WORKDIR /work/src
COPY --from=builder /usr/local/lib/python3.8/site-packages /usr/local/lib/python3.8/site-packages
# need to copy if python package is installed in /usr/local/bin
COPY --from=builder /usr/local/bin/streamlit /usr/local/bin/streamlit
COPY src/ ./
# for dev packages
COPY --from=builder /work/src/requirements-dev.txt requirements-dev.txt
RUN pip install -r requirements-dev.txt
COPY tests/ ./
EXPOSE 8501  
CMD ["streamlit", "run", "hello.py"]

### for prod
FROM python:3.8.5-slim as prod
ENV PYTHONUNBUFFERED=1
WORKDIR /work/src
COPY --from=builder /usr/local/lib/python3.8/site-packages /usr/local/lib/python3.8/site-packages
# need to copy if python package is installed in /usr/local/bin
COPY --from=builder /usr/local/bin/streamlit /usr/local/bin/streamlit
COPY src/ ./
EXPOSE 8501  
CMD ["streamlit", "run", "hello.py"]

docker-compose.yml

version: '3'

services:
  app:
    build: 
      context: .
      target: ${STAGE:-prod}
    image: "internal-dashboard_app_${STAGE:-prod}"
    container_name: "internal-dashboard_${STAGE:-prod}"
    volumes:
      - ./src:/work/src
    ports:
      - "8501:8501"
    restart: always
    command: ["streamlit", "run", "hello.py"]

Impressions

I'm always wondering how to deal with the path problem in tests. that's all.