Tags: howtos, programming

When writing Aleph, my library for persistent homology and computational topology, I decided to add a few Python bindings one idle afternoon. Using the magnificent pybind11 library, this was easier than I anticipated. Much to my chagrin, though, it turns out that using such a bindings with Python interpreter is more complicated if you want to do it the right way.

Of course, having built a single .so file that contains the code of your module, the easy way is to modify the PYTHONPATH variable and just point it to the proper path. But I wanted to do this right, of course, and so, together with Max, I started looking at ways how to simplify the build process.

The situation

I am assuming that you have installed pybind11 and wrote some small example that you want to distribute now. If you are unsure about this, please refer to my example repository for this blog post.

The module may look like this:

#include <pybind11/pybind11.h>

#include <string>

class Example
{
public:
  Example( double a )
    : _a( a)
  {
  }

  Example& operator+=( const Example& other )
  {
    _a += other._a;
    return *this;
  }

private:
  double _a;
};

PYBIND11_MODULE(example, m)
{
  m.doc() = "Python bindings for an example library";

  namespace py = pybind11;

  py::class_<Example>(m, "Example")
    .def( py::init( []( double a )
            {
              return new Example(a);
            }
          )
    )
    .def( "__iadd__", &Example::operator+= );
}

Pretty standard stuff so far: one class, with one constructor and one addition operator exposed (for no particular reason whatsoever).

Building everything

Building such a module is relatively easy with CMake if you are able to find the pybind11 installation (the example repository has a ready-to-use module for this purpose). Since we want to do this the right way, we need to check whether the Python interpreter exists:

SET( PACKAGE_VERSION "0.1.1" )

FIND_PACKAGE( pybind11 REQUIRED )

FIND_PACKAGE(PythonInterp 3)
FIND_PACKAGE(PythonLibs   3)

Next, we can build the library using CMake. Some special treatment for MacOS X is required (obviously) in order to link the module properly.

IF( PYTHONINTERP_FOUND AND PYTHONLIBS_FOUND AND PYBIND11_FOUND )
  INCLUDE_DIRECTORIES(
    ${PYTHON_INCLUDE_DIRS}
    ${PYBIND11_INCLUDE_DIRS}
  )

  ADD_LIBRARY( example SHARED example.cc )

  # The library must not have any prefix and should be located in
  # a subfolder that includes the package name. The setup will be
  # more complicated otherwise.
  SET_TARGET_PROPERTIES( example
    PROPERTIES
      PREFIX ""
      LIBRARY_OUTPUT_DIRECTORY "${CMAKE_CURRENT_BINARY_DIR}/example"
  )

  # This is required for linking the library under Mac OS X. Moreover,
  # the suffix ensures that the module can be found by the interpreter
  # later on.
  IF( APPLE )
    SET_TARGET_PROPERTIES( example
      PROPERTIES
        LINK_FLAGS "-undefined dynamic_lookup"
        SUFFIX     ".so"
    )
  ENDIF()


  # Place the initialization file in the output directory for the Python
  # bindings. This will simplify the installation.
  CONFIGURE_FILE( example/__init__.py
    ${CMAKE_CURRENT_BINARY_DIR}/example/__init__.py
  )

  # Ditto for the setup file.
  CONFIGURE_FILE( example/setup.py
    ${CMAKE_CURRENT_BINARY_DIR}/example/setup.py
  )
ENDIF()

The salient points of this snippet are:

• Changing the output directory of the library to a subordinate directory. We will later see that this simplifies the installation.
• Configuring (and copying) __init__.py and setup.py files and make them available in the build directory.

__init__.py is rather short:

from .example import *

This will tell the interpreter later on to import all symbols from the example module in the current directory.

The setup.py is slightly more complicated:

from distutils.core import setup

import sys
if sys.version_info < (3,0):
  sys.exit('Sorry, Python < 3.0 is not supported')

setup(
  name        = 'cmake_cpp_pybind11',
  version     = '${PACKAGE_VERSION}', # TODO: might want to use commit ID here
  packages    = [ 'example' ],
  package_dir = {
    '': '${CMAKE_CURRENT_BINARY_DIR}'
  },
  package_data = {
    '': ['example.so']
  }
)

The important thing is the package_data dictionary. It specifies the single .so file that is the result of the CMake build process. This ensures that the file will be installed alongside the __init__.py file.

Testing it

First, we have to build our package:

$ mkdir build
$ cd build
$ cmake ../
$ make
$ cd example
$ ls
example.so  __init__.py  setup.py
$ sudo python setup.py install

Afterwards, the package should be available for loading:

$ python
Python 3.6.5 (default, Apr 12 2018, 22:45:43)
[GCC 7.3.1 20180312] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import example
>>> example.Example(0.0)
<example.example.Example object at 0x7f54e7f77308>
>>> example.Example("Nope")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: __init__(): incompatible constructor arguments. The following argument types are supported:
    1. example.example.Example(arg0: float)

Invoked with: 'Nope'

Everything seems to work as expected.

Conclusion

This is certainly not the easiest or most modern way to install your own Python module. However, it is the easiest one in case you already have a large project that exports Python bindings. In a truly optimal world, we would use setuptools directly and build wheel packages—but this will have to wait for another article.

In the meantime, the code for this example is available on GitHub.

Happy packaging, until next time!