Debug with VS Code using boost python numpy

I think there is a need to extend Python with c ++ and convert Numpy ndarray to c ++ vector. Boost :: python and boost :: python :: numpy are useful in such cases. (It seems that pybind11 is also popular these days. I wanted to know early ...)

Also, boost python can actually be built as an executable file, so you can debug it with VS Code, for example.

I will explain from 1 from the necessary packages.

environment

Preparation

First, install the required packages.

The packages required for this article were wget, build-essential, cmake, python3-dev, and python3-pip. Please install as appropriate according to your environment.

$ apt install wget build-essential cmake python3-dev python3-pip 

Also, check the installation location and version of Python with the following command.

$ which python3
/usr/bin/python3
$ python3 --version
Python 3.8.2

If you want to use boost :: python :: numpy, you need to install numpy. (I had a hard time forgetting this.)

$ pip3 install numpy

Then download and install boost. Install by referring to Chapter 5 of the following official document.

How to start boost in Unix environment boost download page

Download boost, unzip it, and go inside the unzipped folder.

$ wget (download URL for boost)
$ tar --bzip2 -xf boost_1_73_0.tar.bz2
$ cd boost_1_73_0

Prepare for installation with ./bootstrap.sh in the unzipped folder, and build and install with ./b2.

$ ./bootstrap.sh --with-libraries=python --with-python=python3 --with-python-version=3.8
$ ./b2 install -j8

By specifying --with-libraries = python, only boost :: python will be built and installed, saving you time. Also, specify the python version confirmed earlier with --with-python-version.

(In the case of another environment such as Anaconda or pyenv, it seems that additional settings are required. Since this article is built in a container, I am doing it with a stance that I can pollute the environment as much as I want.)

By default, boost is installed under / usr /. You can find the boost include directory and library path by using the find command.

$ find /usr/ -name "*boost*" | grep include
/usr/include/c++/9/bits/boost_concept_check.h
/usr/local/include/boost
/usr/local/include/boost/chrono/typeof/boost
...

$ find /usr/ -name "*boost*" | grep lib
...
/usr/local/lib/cmake/boost_numpy-1.73.0/libboost_numpy-variant-static-py3.8.cmake
/usr/local/lib/cmake/boost_numpy-1.73.0/boost_numpy-config-version.cmake
/usr/local/lib/cmake/boost_numpy-1.73.0/libboost_numpy-variant-shared-py3.8.cmake
/usr/local/lib/libboost_numpy38.a
/usr/local/lib/libboost_python38.so.1.73
/usr/local/lib/libboost_numpy38.so.1.73
...

Let's make a module that actually uses ndarray

Let's create a sample module that actually has a function that returns the size of the first dimension of ndarray.

sample.cpp

#define BOOST_BIND_GLOBAL_PLACEHOLDERS
//To avoid collisions with placeholders(Investigation required)
#include <boost/python/numpy.hpp>
namespace py = boost::python;
namespace np = boost::python::numpy;

auto size(np::ndarray &a)
{
  auto N = a.shape(0);
  return N;
}

BOOST_PYTHON_MODULE(sample)
{
  Py_Initialize();
  np::initialize();
  py::def("size", size);
}

To compile it with g ++, run the following command.

g++ --shared -fPIC \
	-I$BOOST_INCLUDE_DIR -I/usr/include/python3.8/ \
	sample.cpp -Xlinker -rpath -Xlinker $BOOST_LIB_DIR \
	-lboost_python38 -lboost_numpy38 -o sample.so

$ BOOST_INCLUDE_DIR is the boost include path and $ BOOST_LIB_DIR is the boost library path, which may vary depending on your environment. In my environment, it was / usr / local / include / boost / and / usr / local / lib /, respectively, as I checked using the find command earlier.

I use CMake because I have to change the command depending on the environment and it is difficult to type this command every time.

CMakeLists.txt

cmake_minimum_required(VERSION 3.12)
project(boost-python-test)

find_package(Python3 COMPONENTS Development)
find_package(Boost COMPONENTS python38 numpy38 REQUIRED)

#For debugging: Show resolved path
message("## Boost_LIBRARIES:  ${Boost_LIBRARIES}")  
message("## Boost_INCLUDE_DIRS: ${Boost_INCLUDE_DIRS}")
message("## Python3_INCLUDE_DIRS: ${Python3_INCLUDE_DIRS}") 
message("## Python3_LIBRARIES :${Python3_LIBRARIES}")

add_library(sample SHARED sample.cpp)
target_include_directories(sample PRIVATE ${Python3_INCLUDE_DIRS})
target_link_libraries(sample PRIVATE ${Boost_LIBRARIES} ${Python3_LIBRARIES})
# target_link_libraries(sample PRIVATE ${Boost_LIBRARIES})It works just by itself. The reason is unknown.
set_target_properties(sample PROPERTIES PREFIX "") #prefix'lib'To omit

(For how to use g ++ and how to use cmake

--@ shohirose's How to use CMake (1) -Introduction to CMake without permission Part 1 Basic usage | Kamino memo

Etc. are detailed. If you look at this, you can see everything from the compilation mechanism to the find_package mechanism used above. Thanks! )

Then, place these sample.cpp and CMakeLists.txt in the following directory structure.

.
|-- CMakeLists.txt
|-- build
`-- sample.cpp

Build within build.

$ cd build
$ cmake ..
$ cmake --build .

There is a file (shared library) called sample.so in the build that does so. And if you write and execute the following python script in the build, it will ʻimport` this shared library and you can see that it works fine.

test.py

import sample
import numpy as np

a = np.array([1,2,3,4], dtype=np.float32)
print(a)
size=sample.size(a)
print(size)

$ python3 test.py 
[1. 2. 3. 4.]
4

Debug with VS Code

As long as you call it as a python module, it's hard to debug and use, If you can build it as an executable file, you can use VS Code's CMake-Tools etc. to speed up debugging.

Building a C ++ development environment with Remote Containers of Visual Studio Code Prepare the VS Code remote debugging environment by referring to. (There is also a repository that I improved the repository of the environment in the above article.)

First, create the following directory structure on that environment.

.
|-- CMakeLists.txt
|-- build
`-- main.cpp

CMakeLists.txt

cmake_minimum_required(VERSION 3.12)
project(boost-python-test)

find_package(Python3 COMPONENTS Development)
find_package(Boost COMPONENTS python38 numpy38 REQUIRED)

add_executable(main_app main.cpp)
target_include_directories(main_app PRIVATE ${Python3_INCLUDE_DIRS})
target_link_libraries(main_app PRIVATE ${Boost_LIBRARIES} ${Python3_LIBRARIES})

If you call Py_Initialize () and np :: initialize () at the beginning of the main function as shown below in main.cpp, you can build it as an executable file.

main.cpp

#include <iostream>
#define BOOST_BIND_GLOBAL_PLACEHOLDERS
//To avoid collisions with placeholders(Investigation required)
#include <boost/python/numpy.hpp>
namespace py = boost::python;
namespace np = boost::python::numpy;

int main()
{
  Py_Initialize();
  np::initialize();
  double v[] = {1.3 , 2.4 , -5.3};
  int v_size = 3;
  py::tuple shape = py::make_tuple(v_size);
  py::tuple stride = py::make_tuple(sizeof(double));
  np::dtype dt = np::dtype::get_builtin<double>();
  np::ndarray output = np::from_data(&v[0], dt, shape, stride, py::object());
  np::ndarray output_array = output.copy();
  
  std::cout << "c++ array address::" << std::endl
          << std::hex << &v[0] << std::endl;
  
  auto *p = reinterpret_cast<double*>(output.get_data());
  std::cout << "ndarray address ::" << std::endl
          << std::hex << p << std::endl;
  return 0;
}

In this state, you can use VS Code's CMake-Tools debug feature to set breakpoints and look at variables. If you use DEBUG CONSOLE of VS Code as shown in the image below, you can handle C ++ as if it were an interpreted language, and debugging will be faster. Convenient!

Supplement: Past articles and reference sites

There are past articles on Qiita and articles about Boost :: python on an external site, There wasn't an article that explained what I was stuck with, or how to use so-called modern cmake as mentioned in this article. So I made this article.

Boost.NumPy Tutorial for Extending Python in C ++ (Practice) This article uses boost-numpy, but boost-numpy has been integrated into boost :: python and is now deprecated.

Create a library for Python in C ++ Passing numpy arrays between Python and C ++ These articles were also very helpful.