Three things I was addicted to when using Python and MySQL with Docker

Overview

From a Python container launched with Docker I was addicted to various things when using the MySQL container, so I will summarize it.

environment

host

The guests

I was addicted to

1. I can't find the mysql command and can't connect on the command line

When developing using Docker I think there are many cases where Alpine Linux is used to reduce the weight of the container. While lightweight and quick to boot, many commands have to be installed on their own.

When connecting to a MySQL container from an Alpine Linux container on the command line Of course, the mysql command is not included, so Install as follows using the package management tool apk.

Install client


$ apk update
$ apk add mariadb-client

Although it is mariadb, it can be used with MySQL without any problem. I didn't keep the DB server on localhost, so I installed only the client.

Now you can operate MySQL on the command line by entering the command as follows.

Connection command (Please replace the host name and user name)


$ mysql -h {host} -u {user} -p

2. MySQLdb is not available and the script cannot access the DB

When developing a web application in Python I think that you often use SQLAlchemy as an ORM to access the DB. (The exception is Django because it has its own ORM)

According to the Flask-SQLAlchemy documentation (http://flask-sqlalchemy.pocoo.org/2.1/config/#configuration-keys), I initially wrote:

config.py


SQLALCHEMY_DATABASE_URI = 'mysql://username:password@server/db' 

An error will occur if you access the DB with this setting.

Error log


$ ImportError: No module named 'MySQLdb'

So I try the installation as below, but it doesn't work.

Installing MySQLdb


$ pip install MySQL-python

Error log


$ ImportError: No module named 'ConfigParser'

A quick look at this point shows that MySQLdb isn't available in Python 3. I've seen some ways to use it in some articles There is no reason to stick to MySQLdb this time, so consider other libraries.

When you look at SQLAlchemy documentation, it looks like [PyMySQL](https: // github.com/PyMySQL/PyMySQL) is the runner-up I chose this because compatibility seems to be no problem (although the reading comprehension in English is ambiguous). I installed and changed the config as follows, and confirmed the normal operation.

Install PyMySQL


$ pip install PyMySQL

config.py


SQLALCHEMY_DATABASE_URI = 'mysql+pymysql://username:password@server/db' 

3. Failed to process Japanese character string

If you are using Python, you will be stuck with Japanese processing. This time, the following error occurred when inserting a Japanese character string into the DB.

error


sqlalchemy.exc.InternalError: (pymysql.err.InternalError) (1366, "Incorrect string value: '\\xE9\\x98\\xBF\\xE4\\xBA\\x95...' for column 'name' at row 1")

The following is specified in the shebang line of the script to be used.

xxx.py


# -*- coding: utf-8 -*- 

If there is any other problem, I think it is the character code setting on the MySQL side. Check the settings on the mysql command line as follows:

Check MySQL settings


MySQL [(none)]> SHOW VARIABLES LIKE 'chara%'; 
+--------------------------+----------------------------+
| Variable_name            | Value                      |
+--------------------------+----------------------------+
| character_set_client     | utf8                       |
| character_set_connection | utf8                       |
| character_set_database   | latin1                     |
| character_set_filesystem | binary                     |
| character_set_results    | utf8                       |
| character_set_server     | latin1                     |
| character_set_system     | utf8                       |
| character_sets_dir       | /usr/share/mysql/charsets/ |
+--------------------------+----------------------------+
8 rows in set (0.01 sec)

It's really suspicious that latin1 is mixed. There is no particular merit as it is, so I would like to change it.

I think the most decent way is to set my.cnf This time I'm running the MySQL server as a container, so it's awkward to change. So, I thought about setting this when setting up the container in the first place.

As a result of the survey, the following pages were very helpful. Specify the character code in MySQL of Docker official image Official image of MySQL

Many configuration options can be passed as flags to mysqld. This will give you the flexibility to customize the container without needing a cnf file.

(Japanese translation) You can pass many options to mysqld as flags. This gives you the flexibility to customize your container without the need for a cnf file.

Because docker-compose is used in my environment Add the following description to docker-compose.yml.

docker-compose.yml


command: mysqld --character-set-server=utf8 --collation-server=utf8_unicode_ci

Check the MySQL settings again.

Check MySQL settings


MySQL [(none)]> SHOW VARIABLES LIKE 'chara%';
+--------------------------+----------------------------+
| Variable_name            | Value                      |
+--------------------------+----------------------------+
| character_set_client     | utf8                       |
| character_set_connection | utf8                       |
| character_set_database   | utf8                       |
| character_set_filesystem | binary                     |
| character_set_results    | utf8                       |
| character_set_server     | utf8                       |
| character_set_system     | utf8                       |
| character_sets_dir       | /usr/share/mysql/charsets/ |
+--------------------------+----------------------------+
8 rows in set (0.01 sec)

character_set_database and character_set_server have changed to ʻutf-8`. I was able to insert Japanese safely in this state.

Summary

There are some problems due to creating a DB server in a container, Since the environment can be created just by clicking with the docker-compose command Once you get used to it, you can speed up development.

By the way, I introduced three points that I'm addicted to this time. There is often the possibility that it will increase as you use it in the future.

Recommended Posts

Three things I was addicted to when using Python and MySQL with Docker
What I was addicted to when using Python tornado
I was addicted to scraping with Selenium (+ Python) in 2020
A note I was addicted to when running Python with Visual Studio Code
A story that I was addicted to when I made SFTP communication with python
What I was addicted to when migrating Processing users to Python
Things to keep in mind when using Python with AtCoder
Things to keep in mind when using cgi with python.
I know? Data analysis using Python or things you want to use when you want with numpy
What I was addicted to when introducing ALE to Vim for Python
What I was addicted to with json.dumps in Python base64 encoding
Connect to MySQL with Python within Docker
A note I was addicted to when creating a table with SQLAlchemy
I was addicted to confusing class variables and instance variables in Python
What I was addicted to Python autorun
When I tried to scrape using requests in python, I was addicted to SSLError, so a workaround memo
A story I was addicted to when inserting from Python to a PostgreSQL table
I was addicted to creating a Python venv environment with VS Code
Use Python from Java with Jython. I was also addicted to it.
The file name was bad in Python and I was addicted to import
Two things I was happy about with Python 3.9
What I was addicted to when I built my own neural network using the weights and biases I got with scikit-learn's MLP Classifier.
[Python] When I tried to make a decompression tool with a zip file I just knew, I was addicted to sys.exit ()
I tried to automate internal operations with Docker, Python and Twitter API + bonus
Docker x visualization didn't work and I was addicted to it, so I summarized it!
I want to handle optimization with python and cplex
What I was addicted to when combining class inheritance and Joint Table Inheritance in SQLAlchemy
When I tried to create a project using Python on Docker with PyCharm, it didn't work, but it worked with Docker Compose.
[IOS] GIF animation with Pythonista3. I was addicted to it.
I tried to make GUI tic-tac-toe with Python and Tkinter
[Fixed] I was addicted to alphanumeric judgment of Python strings
I was surprised to receive a nice review when I wrote Python to CheckIO and its explanation
What I was addicted to when dealing with huge files in a Linux 32bit environment
I want to make a voice changer using Python and SPTK with reference to a famous site
Build Mysql + Python environment with docker
I played with PyQt5 and Python3
I want to debug with Python
When using MeCab with virtualenv python
Precautions when using six with Python 2.5
I was addicted to multiprocessing + psycopg2
Things to watch out for when using default arguments in Python
The record I was addicted to when putting MeCab on Heroku
I want to solve APG4b with Python (only 4.01 and 4.04 in Chapter 4)
A note I was addicted to when making a beep on Linux
[Python] Error and solution memo when using venv with pyenv + anaconda
I tried to make a periodical process with Selenium and Python
How to deal with errors when installing Python and pip with choco
I tried to make a todo application using bottle with python
I tried to easily detect facial landmarks with python and dlib
I wrote AWS Lambda, and I was a little addicted to the default value of Python arguments
When I tried to use pip with python, I was told that XML_SetHashSalt could not be found.
Note that I was addicted to accessing the DB with Python's mysql.connector using a web application.
I was surprised at how to save objects with python, which is lean and very energy-saving.
When I put Django in my home directory, I was addicted to static files with permission errors
Two things I was addicted to building Django + Apache + Nginx on Windows
From Python to using MeCab (and CaboCha)
[Data science basics] I tried saving from csv to mysql with python
When I tried to run Python, it was skipped to the Microsoft Store
I was addicted to running tensorflow on GPU with NVIDIA driver 440 + CUDA 10.2
Behind the flyer: Using Docker with Python
I made a music bot using discord.py and Google Drive API (tested with Docker → deployed to Heroku)