[PYTHON] Bluemix NLTK, MeCab, Watson API container build memorandum
Build an IBM container (Docker container) that contains the tools required for IBM Watson API and Python-based Japanese natural language processing, and register it in the IBM Bluemix repository so that it can be reused. Since it is a container for the development environment, of course, we will devise ways to keep the container running.
Build Docker container build environment
Build the target Docker container in your own PC environment and register it in the Bluemix container repository. Since Docker of Ubuntu Linux is convenient as the environment to build, first create an environment where Ubuntu Linux runs with Vagrant in your own PC environment. Here, I think the following link will be helpful for how to prepare the environment of Vagrant.
Once you have a Vagrant environment, boot the Ubuntu Linux fake machine and install Docker.io.
Then install the Cloud Foundry cf command and the Bluemix bx command to log in to your Bluemix account. Of course, getting a Bluemix account is a must. The following links will help you to meet these prerequisites.
- Create a free Bluemix account
- Install Bluemix CLI commands (https://console.ng.bluemix.net/docs/cli/index.html#downloads)
- Install Bluemix cf ic plugin
Get and edit Dockerfile template
As a Japanese processing tool, NLTK made in Python is convenient, so we will create a container that can execute the latest Python. Since the official image of Docker is used as a template of Dockerfile in the latest environment of Python, clone the Dockerfile on Ubuntu of Vagrant with the following command. There are all Python 2.7 series and 3 series templates in this template, but this time I am using Python 2.7.
root@ubuntu-xenial:~/docker# git clone https://github.com/docker-library/python
Go to the Python 2.7 directory and edit the Dockerfile.
root@ubuntu-xenial:~/docker/python# ls
2.7 3.6 Dockerfile-slim.template README.md
3.3 Dockerfile-alpine.template Dockerfile-windowsservercore.template update.sh
3.4 Dockerfile-debian.template generate-stackbrew-library.sh
3.5 Dockerfile-onbuild.template LICENSE
root@ubuntu-xenial:~/docker/python# cd 2.7
root@ubuntu-xenial:~/docker/python/2.7# ls
alpine Dockerfile onbuild slim wheezy windows
The part to be edited is listed below, so add it to the original Dockerfile.
1 FROM buildpack-deps:jessie
2 ADD ./app /app
3
<Omitted>
10
11 # runtime dependencies
12 RUN apt-get update && apt-get -y upgrade && apt-get install -y --no-install-recommends \
13 tcl \
14 tk \
15 mecab \
16 libmecab-dev \
17 mecab-ipadic \
18 mecab-ipadic-utf8 \
19 mecab-utils \
20 emacs-nox \
21 nmon \
22 && rm -rf /var/lib/apt/lists/*
The explanation of the additional parts is listed below.
- 2nd line This is the directory containing the shell that stops the termination of the container, which will be explained later.
- Line 12 Add apg-get -y upgrade. Ubuntu 14.04 has a serious vulnerability in OpenSSL and we will upgrade it as a workaround.
- Lines 15-19 Add MeCab and dictionary for morphological analysis.
- Line 20 is the addition of a tool for editing files. Please replace it if you like.
- Line 21 is the addition of a monitoring tool that displays the operating status of the OS. Since you can know the operating status of the host machine of the Docker container, it is convenient because you can know the CPU status of the host machine when the response is slow.
77 RUN pip install --no-cache-dir mecab-python
78 RUN pip install --no-cache-dir watson-developer-cloud
79 RUN pip install --no-cache-dir BeautifulSoup
80 RUN pip install --no-cache-dir nltk
I will list the following corrections.
- Line 77 Addition of MeCab's Python API library.
- Line 78 Addition of WatsonAPI Python library.
- Line 79 Addition of web scrubbing tool.
- Line 80 Addition of natural language toolkit.
80 RUN pip install --no-cache-dir nltk
81
82 CMD ["bash", "/app/loop_wait"]
Finally, execute the command to execute sleep in an infinite loop so that the container does not end.
Ingenuity for continuous container execution
A description of ADD ./app / app above. Create the app directory in the 2.7 folder, create the following files, and give the execution right chmod + x loop_wait.
!/bin/bash
while true; do sleep 60; done
By executing this file from the CMD on the last line 82 of the Dockerfile, the container will continue to run.
Build a container and register for a Bluemix ribosite
Build the container in your vagrant environment. Run the following command in the location of the Dockerfile file and the app directory. Install the required packages in the container, collect the files, compile Python 2.7.13, and build the container. It may take up to 10 minutes to build this container.
root@ubuntu-xenial:~/python/2.7# docker build -t mynode:python .
When the build is completed, the following message will be displayed, so you can know the success of the build.
Step 9 : RUN pip install --no-cache-dir virtualenv
---> Running in 3259e634ff72
Collecting virtualenv
Downloading virtualenv-15.1.0-py2.py3-none-any.whl (1.8MB)
Installing collected packages: virtualenv
Successfully installed virtualenv-15.1.0
---> b95cc31c623c
Removing intermediate container 3259e634ff72
Step 10 : CMD python2
---> Running in 3fb0181e560c
---> 25903811171b
Removing intermediate container 3fb0181e560c
Successfully built 25903811171b
Then get the name of the Bluemix repository. The cf command is the Cloud Foundry CLI tool on which Bluemix is based, and the subsequent ic is a subcommand that points to the Docker plugin.
root@ubuntu-xenial:~/docker/python/2.7# cf ic namespace get
takara_node
Tag your local container with your Bluemix repository.
root@ubuntu-xenial:~/docker/python/2.7# docker tag mynode:python registry.ng.bluemix.net/takara_node/python
Register the built container in the Bluemix repository with the following command.
root@ubuntu-xenial:~/docker/python/2.7# docker push registry.ng.bluemix.net/takara_node/python
You can risk Bluemix's registered common container and your own registered container with the following command. You can see that the python container is registered in the last line.
root@ubuntu-xenial:~/docker/python/2.7# cf ic images
REPOSITORY TAG IMAGE ID CREATED SIZE
registry.ng.bluemix.net/ibmnode latest acec21732cb5 11 days ago 192.5 MB
registry.ng.bluemix.net/ibmnode v4 acec21732cb5 11 days ago 192.5 MB
<Omitted>
registry.ng.bluemix.net/ibmliberty webProfile7 6b2b8341fa32 5 weeks ago 276 MB
registry.ng.bluemix.net/takara_node/mynode1 latest 721caef84790 12 days ago 190.9 MB
registry.ng.bluemix.net/takara_node/python latest 17f60ad04c10 2 hours ago 334.4 MB
In this state, if you look at the catalog in the Bluemix portal, you should see the above container. At the bottom right, you'll see a purple container called python.
Container startup and login
You can also start the container from the portal screen of the web, but you can also start the container with the following CLI command.
root@ubuntu-xenial:~/python/2.7# cf ic run --name python -m 512 registry.ng.bluemix.net/takara_node/python:latest
The first time it takes about 1 minute to build the container. During the build, you can know that the build is in progress with the following cf ic ps command.
root@ubuntu-xenial:~/python/2.7# cf ic ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
edd1f16c-625 registry.ng.bluemix.net/takara_node/python:latest "" 15 seconds ago Building 14 seconds ago python
You can also check this from the Bluemix portal.
Container building state
Build complete status
Container details screen
Login to container
The following command will allow you to log in to the container and execute commands from the shell.
root@ubuntu-xenial:~/docker/python/2.7# cf ic exec -it python /bin/bash
root@instance-003624a8:/# ps -ax
PID TTY STAT TIME COMMAND
1 ? Ss 0:00 bash /app/loop_wait
18 ? Ss 0:00 /bin/bash
24 ? S 0:00 sleep 60
25 ? Ss 0:00 /bin/bash
31 ? R+ 0:00 ps -ax
Other Docker commands
Use stop to stop the container.
root@ubuntu-xenial:~/python/2.7# cf ic ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
cea800cf-862 registry.ng.bluemix.net/takara_node/python:latest "" 6 hours ago Running 6 hours ago python
26dba932-cc4 registry.ng.bluemix.net/takara_node/mynode1:latest "" 12 days ago Running 12 days ago 169.46.19.213:3000->3000/tcp mynode1
root@ubuntu-xenial:~/python/2.7# cf ic stop cea800cf-862
cea800cf-862
Use rm to delete the container.
root@ubuntu-xenial:~/python/2.7# cf ic rm cea800cf-862
cea800cf-862
Use rmi to delete the image of the container.
root@ubuntu-xenial:~/python/2.7# cf ic images
REPOSITORY TAG IMAGE ID CREATED SIZE
<Omitted>
registry.ng.bluemix.net/takara_node/python latest 8cd32e284563 7 hours ago 310.2 MB
root@ubuntu-xenial:~/python/2.7# cf ic rmi registry.ng.bluemix.net/takara_node/python
Untagged: sha256:8cd32e2845639a3fad5bd6e26c7a57c3f69539cebbd846535bb99b634aa8b324
To delete the image used for the build in the local environment, use the docker command.
root@ubuntu-xenial:~# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
mynode python 8cd32e284563 7 hours ago 852 MB
registry.ng.bluemix.net/takara_node/python latest 8cd32e284563 7 hours ago 852 MB
root@ubuntu-xenial:~# docker rmi mynode:python
Untagged: mynode:python
root@ubuntu-xenial:~# docker rmi registry.ng.bluemix.net/takara_node/python:latest
Build test
Confirm that MeCab is installed
root@instance-003624a8:/# mecab -v
mecab of 0.996
Confirmation of introduction of NLTK of Python
root@instance-003624a8:/# python
Python 2.7.13 (default, Feb 17 2017, 23:23:18)
[GCC 4.9.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import nltk
>>> nltk.download()
NLTK Downloader
---------------------------------------------------------------------------
d) Download l) List u) Update c) Config h) Help q) Quit
---------------------------------------------------------------------------
Downloader>