A story about trying a (Golang +) Python monorepo with Bazel

This article is the third day of OpenSaaS Studio Advent Calendar 2019.

Introduction

The former is a mono-repo configuration using Bazel in Golang and Python microservices. I wondered if Python could be mixed with the mono repo if I used Bazel, which supports multiple languages, but it was difficult, so I will leave a difficult point.

Premise

motivation

Clogged point

Both Python 2 and 3 are required

I used only Python3 series in the service, but I needed Python2 series to push the created image. Since I used Container Registry as the image registry of the container, Bazel needs Google Cloud SDK to push the image and it is linked to it. I think that Python2 system was needed in the form of a sequel. Description of Google Cloud SDK

Recent versions of macOS include the appropriate version of Python required for the Google Cloud SDK. The Cloud SDK requires Python 2 with a release number of Python 2.7.9 or later. If you install an additional Python interpreter, it must not interfere with the installation of the Google Cloud SDK.

What happened

Bazel has a mechanism that allows you to specify the Python interpreter to use when building, so I used that.

The interpreter definition and build definition look like this.

load("@rules_python//python:defs.bzl", "py_runtime_pair")

py_runtime(
    name = "py2_local_bin_runtime",
    interpreter_path = "/usr/local/bin/python2",
    python_version = "PY2",
)

py_runtime(
    name = "py3_6_local_bin_runtime",
    interpreter_path = "/usr/local/bin/python3.6",
    python_version = "PY3",
)

py_runtime_pair(
    name = "py_local_bin_runtime_pair",
    py2_runtime = ":py2_local_bin_runtime",
    py3_runtime = ":py3_6_local_bin_runtime",
)

toolchain(
    name = "py_mac_toolchain",
    exec_compatible_with = [
        "@bazel_tools//platforms:osx",
        "@bazel_tools//platforms:x86_64",
    ],
    toolchain = "py_local_bin_runtime_pair",
    toolchain_type = "@rules_python//python:toolchain_type",
)
register_toolchains(
    "//:py_mac_toolchain",
    "//:py_linux_toolchain",
)
py_binary(
    name = "mysite",
    srcs = ["manage.py"],
    deps = [
        "//mysite/mysite:py_default_library",
        "//mysite/polls:py_default_library",
    ],
    main = "manage.py",
    python_version = "PY3",
)

I can't get the library corresponding to Python3

When using Bazel rules for Python, it seems that the pip associated with the python command is executed when downloading the library. As I wrote above, the default python was 2 series, so I could download the Python 3 series library. The code is around here

What happened

Changed to python-> python3.6 so that you can use 3 series by forking .. However, when I checked the latest version, it was Pip3 compatible by default, so there may be no problem now.

Suffering from the entry point of the created image

Bazel has a default rule (py3_image) for creating a Docker image of Python3 series, and when I built it in combination with python: 3.6-slim, the created image could not be started. The build definition looks like this

container_pull(
    name = "py3_base",
    registry = "index.docker.io",
    repository = "library/python",
    tag = "3.6-slim",
)
py3_image(
    name = "mysite_image",
    srcs = ["manage.py"],
    deps = [
        "//mysite/mysite:py_default_library",
        "//mysite/polls:py_default_library",
    ],
    main = "manage.py",
    base = "@py3_base//image",
)

Actually, the entry point of the image created by py3_image is / usr / bin / python by default, but in python: 3.6-slim, the python command is not set in that path and it can not be started. did. The code is around here

What happened

I used this image because it started normally with python: 3.6. (Image optimization is assumed to be performed later if necessary)

Python version is different for each service

I had three Python services and the versions of Python I was using were all different from 3.5, 3.6 and 3.7. The method of using the Python interpreter described above can only separate the 2nd and 3rd systems, so this mechanism cannot handle it.

What happened

I decided to move all Python versions used in the service to 3.6. It took a long time to verify so far, and I thought that complicating the configuration around the build would deviate from the original policy, so I decided to reduce the number of things to manage. If you want to coexist with multiple versions of Python3, you may be able to separate WORKSPACE and specify the required Python interpreter for each. (unconfirmed)

├── service_for_python3.6
│   └── WORKSPACE <-Python3 here.Set up 6 interpreters
├── service_for_python3.7
│   └── WORKSPACE <-Python3 here.Set up 7 interpreters
└── WORKSPACE

Import of __init__.py is broken

(Probably) The same event as this issue occurred in the google-cloud-XXX series library. It seems that the problem was caused by the different directory structure when installing the library with pip install and when building with Bazel.

What happened

The issue also has a comment, but it can now be executed by specifying the attribute legacy_create_init.

Docker image and build environment platform are different

I'm using tensorflow-gpu for some services and couldn't download the library because the build environment didn't have a GPU.

What happened

It is unsolved. It's been a while since I started the verification, so I rounded it up here. Bazel may be able to handle it by specifying a platform associated with the GPU (if it exists) or by building this service on a build farm that has a GPU.

Summary

Recommended Posts

A story about trying a (Golang +) Python monorepo with Bazel
A story about making 3D space recognition with Python
A story about making Hanon-like sheet music with Python
A story about an amateur making a breakout with python (kivy) ②
A story about an amateur making a breakout with python (kivy) ①
A story about trying to implement a private variable in Python.
A story about a python beginner stuck with No module named'http.server'
A story about adding a REST API to a daemon made with Python
A story about machine learning with Kyasuket
A story about developing a soft type with Firestore + Python + OpenAPI + Typescript
[Note] A story about trying to override a class method with two underscores in Python 3 series.
A story about trying to run multiple python versions (Mac edition)
A story about Python pop and append
[Python3] A story stuck with time zone conversion
A story stuck with handling Python binary data
A story about implementing a login screen with django
A story about running Python on PHP on Heroku
A story about modifying Python and adding functions
A story about using Resona's software token with 1Password
A story about predicting exchange rates with Deep Learning
A story about how theano was moved with TSUBAME 2.0
A memo about building a Django (Python) application with Docker
A story about how Windows 10 users created an environment to use OpenCV3 with Python 3.5
A story about a Python beginner trying to get Google search results using the API
A story about trying to introduce Linter in the middle of a Python (Flask) project
Stumble story with Python array
A memorandum about correlation [Python]
Make a fortune with Python
A memorandum about Python mock
Create a directory with python
A note about [python] __debug__
A note about hitting the Facebook API with the Python SDK
A story about how to specify a relative path in python.
A story about competing with a friend in Othello AI Preparation
[python] A note when trying to use numpy with Cython
A story about installing matplotlib using pip with an error
A story about how to deal with the CORS problem
A story about making a tanka by chance with Sudachi Py
Machine learning A story about people who are not familiar with GBDT using GBDT in Python
[Django] A story about getting stuck in a swamp trying to validate a zip with form [TDD]
[Python, Selenium, PhantomJS] A story when scraping a website with lazy load
[Python] What is a with statement?
Solve ABC163 A ~ C with Python
A python graphing manual with Matplotlib.
A refreshing story about Python's Slice
A story about trying to automate a chot when cooking for yourself
Let's make a GUI with python.
Python: A Note About Classes 1 "Abstract"
A sloppy story about Python's Slice
Create a virtual environment with Python!
I made a fortune with Python.
The story of making a standard driver for db with python.
Building a virtual environment with Python 3
Solve ABC168 A ~ C with Python
Make a recommender system with python
[Small story] Get timestamp with Python
Generate a Pre-Signed URL with golang
[Python] Generate a password with Slackbot
Solve ABC162 A ~ C with Python
Solve ABC167 A ~ C with Python
Solve ABC158 A ~ C with Python