Package python runtime and pypi library with chef / omnibus and Docker

Preface

It was once developed under the name omnibus-ruby to provide Ruby linetime and gem packages [^ 1] for various operating systems (?) Chef / omnibus. omnibus), but it feels like a cross-platform packaging creation framework that can be used [^ 2] regardless of language.

However, even if you google, there are few people who use omnibus itself and most of the examples are Ruby, but this time I had the opportunity to package the Python runtime and pypi library, so leave the information. I think this.

In addition, I referred to this very much in the implementation. Thank you very much. http://qiita.com/sawanoboly/items/0902a97ef71d99ad60a6

Build environment preparation

This is also built in the Docker environment like the article introduced above, but this time as well. However, how to create a Docker image is a little different (more omission w)

First of all, the quickest way to create an omnibus build environment is to run the officially prepared omnibus cookbook. https://github.com/chef-cookbooks/omnibus

omnibus Officially, it is introduced to build a build environment using kitchen-vagrant and kitchen-docker, but I want to do it on CircleCI, so kitchen -vagrant is NG, kitchen-docker is Docker's bad practice, so you can put ʻopensshto connect with SSH, create an extra user for SSH login, and do various extra things, so that Use the Test-Kitchen Driver calledkitchen-docker_cli`, which was created to avoid and use Docker natively as much as possible. https://github.com/marcy-terui/kitchen-docker_cli

So, set it like this.

yaml:.kitchen.yml


driver:
  name: docker_cli

transport:
  name: docker_cli

provisioner:
  name: chef_zero

platforms:
  - name: centos-6.6
    driver_config:
      image: centos:centos6.6
    run_list:
      - yum-epel::default
      - omnibus::default
      - omnibus-lamvery::yum_clean
  - name: ubuntu-14.04
    run_list:
      - apt::default
      - omnibus::default
      - omnibus-lamvery::apt_clean

suites:
  - name: default
    attributes:
      omnibus:
        build_user:          root
        build_user_group:    root
        build_user_home:     /root
        install_dir:         /opt/lamvery

ʻOmnibus-lamvery :: yum_clean, ʻapt_clean is a recipe that is almost meaningless, just like ʻapt-get clean, yum clean all` to reduce the capacity even a little w

Now all you have to do is do the following:

bundle exec kitchen converge

Check with the baked Docker container and docker ps, and register it in Docker Hub.

docker commit <container-id> <username>/<image>
docker push <username>/<image>

Click here for the result https://hub.docker.com/r/marcy/omnibus-ubuntu-14/ https://hub.docker.com/r/marcy/omnibus-centos-6/

It's like doing a docker pull on CircleCI and running a build with docker run.

Build the package

Below, if you explain everything, there is no sharpness, so I will briefly describe it. Click here for the source to see the whole picture. https://github.com/willyworks/omnibus-lamvery/tree/master/omnibus Mount this with -v when starting the Docker container for build (docker run) and build it.

Python Unless you are particular about build options, a collection of recipes (?) For building various software called omnibus-software with omnibus. There is Python inside, so if you include it in the Gemfile and bundle you should be able to use it.

Gemfile


source 'https://rubygems.org'

gem 'omnibus', github: 'chef/omnibus'
gem 'omnibus-software', github: 'opscode/omnibus-software'

This time I wanted to change the options, so I defined it at hand. As a precaution when specifying options, although it is not unique to Python, omnibus is a framework for creating a full stack package that is completed by itself, so it is NG if it depends on system global libraries. By the way, the check runs at the end of the build, so it ends with an error. In Python, specifying a configure option like --with-system-ffi will result in NG.

PyPi + library

This is easy. First, install pip using Python provided by the package as shown below.

software/pypi.rb


name "pypi"

build do
  command "curl -kL https://bootstrap.pypa.io/get-pip.py | #{install_dir}/embedded/bin/python"
  command "ln -fs #{install_dir}/embedded/bin/pip #{install_dir}/bin/pip"
end

Then all you have to do is install the library with the resulting pip command.

software/lamvery.rb


name "lamvery"

build do
  command "#{install_dir}/embedded/bin/pip install lamvery==#{version}"
  command "ln -fs #{install_dir}/embedded/bin/lamvery #{install_dir}/bin/lamvery"
end

By the way, you can get the latest version of the library provided at that time as follows.

require 'rexml/document'
require 'open-uri'

doc = REXML::Document.new(open('https://pypi.python.org/pypi?:action=doap&name=lamvery').read)
p doc.elements['rdf:RDF/Project/release/Version/revision'].text #=> "0.14.0"

Distribution of deliverables

This time, I uploaded it to Bintray. In the example I referred to, I used the CLI client made by Go, but I tried hitting the REST API directly. Bintray's REST API is simple and nice. The response code was strange and I was a little addicted to it. .. .. I couldn't find a gem that would be nice to use in the future, so it might be a good idea to make one.

For reference, the code of the rough rake task uploaded to Bintray looks like this. https://github.com/willyworks/omnibus-lamvery/blob/master/Rakefile

Summary

If you try to do something around here, there is a good chance that sawanoboly will do it first and it will be saved (repost)

[^ 1]: rpm, deb, etc. It seems that you can also make pkg for Mac and msi for Win (I only do rpm and deb) [^ 2]: Rather, it is OK even if the language is not included

Recommended Posts

Package python runtime and pypi library with chef / omnibus and Docker
Python installation and package management with pip
Use cryptography library cryptography with Docker Python image
[Python] Get Python package information with PyPI API
Build PyPy and Python execution environment with Docker
Registering with PyPI from modern Python library self-made
Use python with docker
Manage Python runtime packages and development environment packages with Poetry
Programming with Python and Tkinter
Encryption and decryption with Python
Python and hardware-Using RS232C with Python-
Prepare python3 environment with Docker
python with pyenv and venv
Library for specifying a name server and dig with python
Works with Python and R
Create a C ++ and Python execution environment with WSL2 + Docker + VSCode
Create a simple Python development environment with VS Code and Docker
Communicate with FX-5204PS with Python and PyUSB
Shining life with Python and OpenCV
[Package cloud] Manage python packages with package cloud
Robot running with Arduino and python
Install Python 2.7.9 and Python 3.4.x with pip.
Neural network with OpenCV 3 and Python 3
AM modulation and demodulation with python
[Python] font family and font with matplotlib
Scraping with Node, Ruby and Python
Easy Slackbot with Docker and Errbot
Scraping with Python, Selenium and Chromedriver
Scraping with Python and Beautiful Soup
JSON encoding and decoding with python
[GUI with Python] PyQt5-Drag and drop-
Reading and writing NetCDF with Python
I played with PyQt5 and Python3
[Hyperledger Iroha] Query with Python library
Reading and writing CSV with Python
Multiple integrals with Python and Sympy
Coexistence of Python2 and 3 with CircleCI (1.0)
Easy modeling with Blender and Python
Docker environment update: add Python package
Application development with Docker + Python + Flask
Sugoroku game and addition game with python
FM modulation and demodulation with Python
I set the environment variable with Docker and displayed it in Python
Communicate between Elixir and Python with gRPC
Data pipeline construction with Python and Luigi
Calculate and display standard weight with python
How to package and distribute Python scripts
Monitor Mojo outages with Python and Skype
Build Jupyter Lab (Python) environment with Docker
About package management with conda and pip
[Automation] Manipulate mouse and keyboard with Python
Behind the flyer: Using Docker with Python
Passwordless authentication with RDS and IAM (Python)
[Python] Package and distribute your own modules
POST variously with Python and receive with Flask
Capturing images with Pupil, python and OpenCV
Fractal to make and play with Python
A memo with Python2.7 and Python3 on CentOS
Let's try gRPC with Go and Docker
Connect to MySQL with Python within Docker
Use PIL and Pillow with Cygwin Python