Pass OpenCV data from the original C ++ library to Python

It's the 21st day of the OpenCV Advent Calendar. I will post it late.

Summary of this article

I think there are many people who want to use various C ++ code using OpenCV in python (fewer). I am one of them. So, in this article, I will make it possible to pass data such as cv :: Mat or std :: vector << cv :: Rect >> to python in numpy format. Of course, if you just call the C ++ code, there is nothing wrong with it. The point of this time is to convert Mat and vector to numpy and pass it, just like using OpenCV in Python.

Boost.Python is a well-known mechanism for calling libraries written in C ++ with python, but if you try to convert a complex class like Mat in C ++ with Boost.Python, it is quite difficult. On top of that, if you try to port not just Mat, but Point, Rect, and all the other classes, you'll die. On the other hand, OpenCV functions can be easily called from python in the first place, and Mat or Rect vector can be easily handled in python in the common format of numpy. The motivation is that if you use this mechanism, you can easily call your own C ++ library from python.

In the end, it looks like this.

c++~ side


cv::Mat aaaa() {
  return cv::Mat();
}
std::vector<cv::Rect> bbbb() {
  return std::vector<cv::Rect>(4);
}

python side


import xxxx
mat = xxxx.aaaa()
vec = xxxx.bbbb()

Also, I'm sorry, I haven't confirmed the operation on windows this time. I remember giving up in an instant, saying that it wouldn't work for a while. Maybe if you do your best. Ubuntu has a track record of working in the past. As the current test environment, we have confirmed the operation on the Mac at hand.

How it works

This time we will use Boost.Python. Boost.Python is a nice library that wraps C ++ classes and functions and makes them available to Python.

http://d.hatena.ne.jp/moriyoshi/20091214/1260779899

The mechanism and merits of Boost.Python are described in detail in, so I will omit it here. I'm not familiar with Boost.Python in the first place ...

By the way, suddenly, OpenCV is in modules / python / common.cmake at build time.

add_custom_command(
   OUTPUT ${cv2_generated_hdrs}
   COMMAND ${PYTHON_EXECUTABLE} "${PYTHON_SOURCE_DIR}/src2/gen2.py" ${CMAKE_CURRENT_BINARY_DIR} "${CMAKE_CURRENT_BINARY_DIR}/headers.txt"
   DEPENDS ${PYTHON_SOURCE_DIR}/src2/gen2.py
   DEPENDS ${PYTHON_SOURCE_DIR}/src2/hdr_parser.py
   DEPENDS ${CMAKE_CURRENT_BINARY_DIR}/headers.txt
   DEPENDS ${opencv_hdrs})

ocv_add_library(${the_module} MODULE ${PYTHON_SOURCE_DIR}/src2/cv2.cpp ${cv2_generated_hdrs})

As shown in, gen2.py is called to convert each OpenCV function so that it can be used in Boost.Python. Furthermore, using cv2.cpp below it, Mat, Rect, etc. are converted from C ++ to numpy. For example, the conversion of Mat is as follows.

PyObject* pyopencv_from(const Mat& m)
{
    if( !m.data )
        Py_RETURN_NONE;
    Mat temp, *p = (Mat*)&m;
    if(!p->u || p->allocator != &g_numpyAllocator)
    {
        temp.allocator = &g_numpyAllocator;
        ERRWRAP2(m.copyTo(temp));
        p = &temp;
    }
    PyObject* o = (PyObject*)p->u->userdata;
    Py_INCREF(o);
    return o;
}

As you can see by reading it, Py_INCREF is used to increase the reference count, and g_numpyAllocator in the middle is used to allocate memory.

    UMatData* allocate(int dims0, const int* sizes, int type, void* data, size_t* step, int flags, UMatUsageFlags usageFlags) const
    {
        if( data != 0 )
        {
            CV_Error(Error::StsAssert, "The data should normally be NULL!");
            // probably this is safe to do in such extreme case
            return stdAllocator->allocate(dims0, sizes, type, data, step, flags, usageFlags);
        }
        PyEnsureGIL gil;

        int depth = CV_MAT_DEPTH(type);
        int cn = CV_MAT_CN(type);
        const int f = (int)(sizeof(size_t)/8);
        int typenum = depth == CV_8U ? NPY_UBYTE : depth == CV_8S ? NPY_BYTE :
        depth == CV_16U ? NPY_USHORT : depth == CV_16S ? NPY_SHORT :
        depth == CV_32S ? NPY_INT : depth == CV_32F ? NPY_FLOAT :
        depth == CV_64F ? NPY_DOUBLE : f*NPY_ULONGLONG + (f^1)*NPY_UINT;
        int i, dims = dims0;
        cv::AutoBuffer<npy_intp> _sizes(dims + 1);
        for( i = 0; i < dims; i++ )
            _sizes[i] = sizes[i];
        if( cn > 1 )
            _sizes[dims++] = cn;
        PyObject* o = PyArray_SimpleNew(dims, _sizes, typenum);
        if(!o)
            CV_Error_(Error::StsError, ("The numpy array of typenum=%d, ndims=%d can not be created", typenum, dims));
        return allocate(o, dims0, sizes, type, step);
    }

And so on. It's hard to do this yourself, so this time let's use the code around here as it is and convert C ++ format Mat etc. to python.

Dependent libraries

I'm sorry, I've put a lot in my environment, so I'm not sure if the following is enough, but if it's not enough, please comment as much as possible. Comments are welcome that it worked if you put this in.

OpenCV3 with python
numpy
boost
boost-python

Creating a project

As a sample, let's create a new project. This time, we will call it sample_project. Create sample_project in a suitable location.

mkdir sample_project
cd sample_project

Create a directory containing the library code. This time with boost_opencv.

mkdir boost_opencv

Create a folder to put the python test script code.

mkdir scripts

About diversion of OpenCV

This time it will be the main one. However, diversion is very easy. Copy cv2 from earlier to your library directory. There is no big reason, but rename the code of cv2.cpp as a header file.

cd boost_opencv
wget  https://raw.githubusercontent.com/opencv/opencv/3.1.0/modules/python/src2/cv2.cpp
mv cv2.cpp cv2.hpp

I want to use only a part of this time, and I will not use the part automatically generated by gen2.py explained earlier, so I will modify it in various ways. First of all, I will comment out the following include parts scattered in various parts of the code.

#include "pyopencv_generated_types.h"
#include "pyopencv_generated_funcs.h"
#include "pyopencv_generated_ns_reg.h"
#include "pyopencv_generated_type_reg.h"

Next, comment out the entire initialization part. Initialization will be done separately, so you can scrape it. At the beginning of shaving

1351 #if PY_MAJOR_VERSION >= 3
1352 extern "C" CV_EXPORTS PyObject* PyInit_cv2();
1353 static struct PyModuleDef cv2_moduledef =
1354 {
1355     PyModuleDef_HEAD_INIT,
1356     MODULESTR,
1357    "Python wrapper for OpenCV.",

So, I will comment out everything from here to the end. However, this is information at this time, so I think it will change over time. In that case, I think that you can skip here once and remove the part where the error occurred in the build step later. This completes the diversion section.

Library code sample

Here, as a simple example, we will prepare a function that returns Mat and a function that returns a vector of Rect. Here, we will return a 3x3 zero initialization matrix and a vector with four Rects.

boost_opencv/main.cpp


#include <boost/python.hpp>
#include "cv2.hpp"

PyObject* GetImage()
{
  cv::Mat cv_img(3, 3, CV_32F, cv::Scalar(0.0));
  return pyopencv_from(cv_img);
}

PyObject* GetObject(int index) {
  vector_Rect2d rects;
  rects.push_back(cv::Rect2d(0,0,0,0));
  rects.push_back(cv::Rect2d(0,0,0,0));
  rects.push_back(cv::Rect2d(0,0,0,0));
  rects.push_back(cv::Rect2d(0,0,0,0));
  return pyopencv_from(rects);
}

BOOST_PYTHON_MODULE(libboost_opencv)
{
  using namespace boost::python;
  import_array();
  def("GetImage", &GetImage);
  def("GetObject", &GetObject);
}

The most important here is pyopencv_from. Here's the code that converts Mat and vector to numpy. By the way, vector_Rect2d is a vector of Rect2d and is defined somewhere in cv2.cpp. Also, don't forget to call import_array () as a way to use numpy. Also, as for Boost.Python, you need to put the correct library name in the argument of BOOST_PYTHON_MODULE.

Build preparation

Now that the basic preparation is complete, let's write CMakeLists.txt. First, make the root CMakeLists.txt as follows.

CMakeLists.txt


cmake_minimum_required(VERSION 3.7)

project(sample_project)

set(CMAKE_MODULE_PATH ${CMAKE_MODULE_PATH} ${CMAKE_CURRENT_SOURCE_DIR}/cmake-modules)
set(CMAKE_LIBRARY_OUTPUT_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/lib)

find_package(OpenCV REQUIRED)
find_package(Boost REQUIRED COMPONENTS python)

find_package(PythonLibs REQUIRED)
# in the case of homebrew python
#set(PYTHON_INCLUDE_DIRS "/usr/local/Cellar/python/2.7.12_2/Frameworks/Python.framework/Versions/2.7/include/python2.7/")
#set(PYTHON_LIBRARIES "/usr/local/Cellar/python/2.7.12_2/Frameworks/Python.framework/Versions/2.7/lib/libpython2.7.dylib")

find_package(PythonInterp REQUIRED)
find_package(Numpy REQUIRED)

add_subdirectory(boost_opencv)

As a caveat, when loading PythonLibs, you can do it as it is with ubuntu, but if you have installed python using Homebrew on Mac, this does not seem to work. I entered the pass directly because it was troublesome ...

Well, if you leave it as it is, you will be told that Numpy cannot be found, so let's bring in the Numpy include path and library path. As long as NUMPY_INCLUDE_DIR is set in the end, any method is fine, but I used FindNumPy.cmake from Caffe.

mkdir cmake-modules
cd cmake-modules
wget https://raw.githubusercontent.com/BVLC/caffe/master/cmake/Modules/FindNumPy.cmake

In addition, FindNumPy.cmake of caffe seems to have the setting for caffe at the end, so I'm not sure, but let's comment it out.

# caffe_clear_vars(__result __output __error_value __values __ver_check __error_value)

Then the boost_opencv / CMakeLists.txt for the library looks like this: As it is, I will omit the details.

boost_opencv/CMakeLists.txt


project(boost_opencv)

include_directories(${OpenCV_INCLUDE_DIRS} ${Boost_INCLUDE_DIRS} ${PYTHON_INCLUDE_DIRS} ${NUMPY_INCLUDE_DIR})
add_library(boost_opencv SHARED main.cpp) 
target_link_libraries(boost_opencv ${OpenCV_LIBS} ${Boost_LIBRARIES} ${PYTHON_LIBRARIES})

Build

This is just that. Just create a build directory and cmake and make.

mkdir build
cd build
cmake ..
make

Then, libboost_opencv.so (libboost_opencv.dylib on Mac) will be created in the lib directory. All you have to do is import this with python and call it.

Library call

This is really simple, but I'll post it for now. There is no rule, but prepare a suitable script in the scripts directory.

scripts/test.py


import sys
sys.path.append('../lib')
import libboost_opencv
print(libboost_opencv.GetImage())
print(libboost_opencv.GetObject(0))

By the way, python does not import dylib made on Mac, so I changed the extension to so and it read it. Result is,

[[ 0.  0.  0.]
 [ 0.  0.  0.]
 [ 0.  0.  0.]]
[[ 0.  0.  0.  0.]
 [ 0.  0.  0.  0.]
 [ 0.  0.  0.  0.]
 [ 0.  0.  0.  0.]]

So I was able to do it. Now you can convert C ++ Mat and Rect vectors to numpy and pass them to python. I'm happy.

Recommended Posts

Pass OpenCV data from the original C ++ library to Python
How to use the C library in Python
I wanted to use the Python library from MATLAB
Python C / C ++ Extensions: Pass some of the data as np.array to Python (set stride)
[Python] How to import the library
[pepper] Pass the JSON data obtained by python request to the tablet.
Pass a list by reference from Python to C ++ with pybind11
Tips for Python beginners to use the Scikit-image example for themselves 9 Use from C
Python C / C ++ Extension Pattern-Pass data to Python as np.array
Just add the python array to the json data
[Python] How to read data from CIFAR-10 and CIFAR-100
[Python] Flow from web scraping to data analysis
I want to make C ++ code from Python code!
Make the library created by Eigen in C ++ available from Python with Boost.Numpy.
Find the position in the original image from the coordinates after affine transformation (Python + OpenCV)
Changes from Python 2 to Python 3.0
Introduction to OpenCV (python)-(2)
The wall of changing the Django service from Python 2.7 to Python 3
Send log data from the server to Splunk Cloud
Python points from the perspective of a C programmer
Send data from Python to Processing via socket communication
[Python] How to use the graph creation library Altair
Write data to KINTONE using the Python requests module
The first step to getting Blender available from Python
Execute Python function from Powershell (how to pass arguments)
I felt that I ported the Python code to C ++ 98.
[Introduction to Python] Basic usage of the library matplotlib
[Python] How to call a c function from python (ctypes)
Python OpenCV tried to display the image in text.
"Cython" tutorial to make Python explosive: Pass the C ++ class object to the class object on the Python side. Part 2
Python Machine Learning Programming Chapter 1 Gives Computers the Ability to Learn from Data Summary
"Cython" tutorial to make Python explosive: When C ++ code depends on the library. Preparation
How to pass arguments when invoking python script from blender on the command line
Pass an array from PHP to PYTHON and do numpy processing to get the result
Call C language functions from Python to exchange multidimensional arrays
Python --Read data from a numeric data file to find the covariance matrix, eigenvalues, and eigenvectors
I tried using the Python library from Ruby with PyCall
[Python] It might be useful to list the data frames
Cheating from PHP to Python
How to debug the Python standard library in Visual Studio
[python] How to use the library Matplotlib for drawing graphs
"Cython" tutorial to make Python explosive: Pass a C ++ class object to a class object on the Python side. Part ①
Anaconda updated from 4.2.0 to 4.3.0 (python3.5 updated to python3.6)
Various ways to calculate the similarity between data in python
Library path setting to pass GAE / Python local unit tests
I tried face recognition from the video (OpenCV: python version)
[Introduction to matplotlib] Read the end time from COVID-19 data ♬
Build a Python environment and transfer data to the server
"Cython" tutorial to make Python explosive: When C ++ code depends on the library. Write setup.py.
Switch from python2.7 to python3.6 (centos7)
The story of copying data from S3 to Google's TeamDrive
Connect to sqlite from python
Hit REST in Python to get data from New Relic
Meteorology x Python ~ From weather data acquisition to spectrum analysis ~
I tried changing the python script from 2.7.11 to 3.6.0 on windows10
Program to determine leap year from the Christian era [Python]
[Introduction to Python] How to get data with the listdir function
Extract the value closest to a value from a Python list element
Ported from R language of "Sazae-san's rock-paper-scissors data analysis" to Python
[Python] Hit Keras from TensorFlow and TensorFlow from c ++ to speed up execution
How to pass and study the Python 3 Engineer Certification Basic Exam