Pin python managed by conda

When I did conda install hogehoge, an unplanned package (or python) was updated and the whole thing died, so I'll write a memorandum.

The end

What I wanted to do: I wanted to create my own image by adding some libraries based on the docker image of pytorch.

What I did: I wrote conda install obediently in Dockerfile What happened: The version of python changed during conda install, and the conda environment was damaged and could not be used properly.

Genin: Mismatch between conda and python versions What I had to do (Conclusion): Write what you want to fix in a file called pinned (this time solved withpython == 3.6. *) (Or include the conda that matches the updated version of python; not done this time; at the bottom of the reference link)

reference

Is it possible to lock versions of packages in Anaconda? - Stack Overflow How to pin Conda | Damian's blog Notes on changing the python version of conda --Qiita

Details of what I did

I wrote the following Dockerfile (the old image I'm using may be part of the problem ...).

FROM pytorch/pytorch:0.4.1-cuda9-cudnn7-devel

RUN apt update && apt upgrade -y

#This will be a problem later
# joblib etc
RUN conda install -y \
    joblib \
    matplotlib \
    scikit-learn \
    tqdm

Then build.

$ docker build -t mypytorch .

At this time, conda install displayed these as packages to be updated (the part that will be a problem later).

The following packages will be UPDATED:

    ca-certificates:  2018.03.07-0            --> 2020.1.1-0
    certifi:          2018.4.16-py36_0        --> 2019.11.28-py38_0
    freetype:         2.9.1-h8a8886c_0        --> 2.9.1-h8a8886c_1
    libedit:          3.1.20170329-h6b74fdf_2 --> 3.1.20181209-hc058e9b_0
    libgcc-ng:        7.2.0-hdf63c60_3        --> 9.1.0-hdf63c60_0
    libpng:           1.6.34-hb9fc6fc_0       --> 1.6.37-hbc83047_0
    libstdcxx-ng:     7.2.0-hdf63c60_3        --> 9.1.0-hdf63c60_0
    mkl:              2018.0.3-1              --> 2020.0-166
    mkl_fft:          1.0.4-py36h4414c95_0    --> 1.0.15-py38ha843d7b_0
    mkl_random:       1.0.1-py36h4414c95_1    --> 1.1.0-py38h962f231_0
    ncurses:          6.1-hf484d3e_0          --> 6.2-he6710b0_0
    numpy:            1.14.5-py36h1b885b7_4   --> 1.18.1-py38h4f9e942_0
    numpy-base:       1.14.5-py36hdbf6ddf_4   --> 1.18.1-py38hde5b4d6_1
    openssl:          1.0.2o-h20670df_0       --> 1.1.1d-h7b6447c_4
    pip:              10.0.1-py36_0           --> 20.0.2-py38_1
    python:           3.6.5-hc3d631a_2        --> 3.8.1-h0371630_1
    readline:         7.0-ha6073c6_4          --> 7.0-h7b6447c_5
    scipy:            1.1.0-py36hc49cb51_0    --> 1.4.1-py38h0b6359f_0
    setuptools:       39.2.0-py36_0           --> 45.2.0-py38_0
    six:              1.11.0-py36h372c433_1   --> 1.14.0-py38_0
    sqlite:           3.23.1-he433501_0       --> 3.31.1-h7b6447c_0
    tk:               8.6.7-hc745277_3        --> 8.6.8-hbc83047_0
    wheel:            0.31.1-py36_0           --> 0.34.2-py38_0
    zlib:             1.2.11-ha838bed_2       --> 1.2.11-h7b6447c_3

Details of what happened

Now that the build is complete, we will triumphantly make a container.

$ docker run -it --rm mypytorch bash

When I try to use conda type commands in the container, it doesn't work.

root@ce88f6b1ba73:/workspace# conda
Traceback (most recent call last):
  File "/opt/conda/bin/conda", line 7, in <module>
    from conda.cli import main
ModuleNotFoundError: No module named 'conda'

Others ʻipythondoesn't work, or it can't be helped, so even if you try frompython` as below, it won't work.

root@ce88f6b1ba73:/workspace# python
Python 3.8.1 (default, Jan  8 2020, 22:29:32)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'torch'
>>>

So, I have created an image of something that I want to move but nothing that moves.

What I had to do

This was the problem. The version of python has been updated to 3.8.1.

The following packages will be UPDATED:

...
    python:           3.6.5-hc3d631a_2        --> 3.8.1-h0371630_1
...

It seems that it will not work if only python is updated without changing the version of conda. There are two ways to solve it.

1. Update conda and python together
2. Fix the version of python

This time, we will fix the version. To fix the version, just write what you want to fix in conda-meta / pinned under the management directory of anaconda / miniconda (how it will be handled when multiple environments are created has not been verified).

This time, since python is fixed to 3.6 series, it will be as follows (here, the installation destination is / opt. This depends on your environment).

$ echo "python=3.6.*" >> /opt/conda/conda-meta/pinned

It will work if this is in the Dockerfile. In other words, it becomes as follows.

FROM pytorch/pytorch:0.4.1-cuda9-cudnn7-devel

RUN apt update && apt upgrade -y

#Fixed version before conda install
RUN echo "python=3.6.*" >> /opt/conda/conda-meta/pinned

# joblib etc
RUN conda install -y \
    joblib \
    matplotlib \
    scikit-learn \
    tqdm

I don't think it's an accident that happens very often, but I hope it helps someone.