Kodi for Android: Building and Using Binary Python Extensions - Part 4

 Jan. 20, 2017     0 comments

This is the 4th article in my series about building and using binary Python modules in Kodi on Android. In the previous article I described how to import a binary Python module in Kodi for Android. In this article I will briefly cover 2 C++ libraries used for writing binary Python modules: Boost.Python and Pybind11.

In my 2nd article of this series I demonstrated a simple "Hello World" type Python/C extension module. A pure Python/C API is quite verbose (for example, you need to manually count references for your PyObject instances) and complex. Fortunately, there are a number of convenience tools that simplify writing binary Python modules. I'm not going to mention them all and will briefly describe only two of such tools.

Boost.Python and Pybind11 are C++ libraries that allow to wrap regular C++ code into Python modules. Naturally, those binary modules need then to be compiled for a target platform. Historically, Boost.Python was developed first and Pybind11 can be considered as its descendant (at least, ideologically), but I'll describe Pybind11 first because it is simpler to use.

Pybind11

Pybind11 is a header-only C++ library that allows you to write Python extension modules using modern C++11/14 standard. Being header-only, Pybind11 does not require linking against pre-built libraries, which is a great advantage for platforms that require cross-compilation. However, it supports only Python 2.7 and above, so you cannot use it to write extension modules for Kodi versions earlier than 17 (Krypton).

Let's see how our "Hello World" module looks when written with Pybind11:

#include <pybind11/pybind11.h>
#include <string>

std::string get_greet()
{
	return std::string("Hello World!");
}

// Pybind11 wrapper
PYBIND11_PLUGIN(hello)
{
	pybind11::module mod("hello");

	mod.def("get_greet", &get_greet);

	return mod.ptr();
}

As you can see, the code consists of 2 parts: the main C++ code and a Python module wrapper. Usually, the main code and a wrapper are put in separate files, but in our simple example it's OK to combine them together.

The wrapper is implemented with PYBIND11_PLUGIN macro that looks like a regular function. Inside this "function" we define a module object, add members to it (in our case a single function object) and return a pointer to this module object ( PyObject*) to the Python runtime.

Building such module for Kodi on Android is no different than building our simple Python/C module described in the 2nd article. You only need to add Pybind11 headers to your project's LOCAL_C_INCLUDES paths.

Boost.Python

Boost.Python was developed quite a while ago and supports older versions of Python, including Python 2.6 which is needed for Kodi 16 (Jarvis) and below. However, unlike Pybind11, it requires linking against a pre-built shared or static boost_python library. So we need to cross-compile this library to our target platform first.

First, download Boost sources from the official site and unpack them to the directory of your choice, for example c:\boost_1_59_0. I recommend to use Boost v.1.59 because on later versions a build command hangs for no apparent reason.

Boost uses its own build system — Boost.Build. First, you need to build Boost.Build utility b2 for your host platform used for cross-compiling. For this you need a C/C++ compiler for your platform. Since Windows has no system-default compiler, I recommend to install the latest version of MS Visual Studio Community. To build b2 utility run the bootstrap script: bootstrap.bat on Windows or bootstrap.sh for *nix. When compilation finishes you will find b2 executable in your Boost directory.

Now you need to create Boost.Build configuration for our cross-compiler. Boost.Build configuration file is called user-config.jam and you need to place it into your home directory. On Windows it is usually c:\Users\<your.username> (%USERPROFILE% environment variable) and on *nix it is /home/<your_username> ($HOME environment variable). The user-config.jam file is a simple text file with the following contents:

AndroidNDKRoot = c:/crystax-ndk-10.3.2 ;
HOST = windows-x86_64 ;
TOOLCHAIN = arm-linux-androideabi ;
GCC_ver = 5 ;
ARCH = arm ;
ABI = armeabi-v7a ;

ANDROID_API = 19 ;

PythonInclude = c:/hello-android/include/Python2.6 ;
PythonLibDir = $(PythonInclude) ; # Fake path

using python : 2.6 : : $(PythonInclude) : $(PythonLibDir) : ;

using gcc : android : 
    $(AndroidNDKRoot)/toolchains/$(TOOLCHAIN)-$(GCC_ver)/prebuilt/$(HOST)/bin/$(TOOLCHAIN)-g++ : 
    <compileflags>--sysroot=$(AndroidNDKRoot)/platforms/android-$(ANDROID_API)/arch-$(ARCH) 
    <compileflags>-mthumb 
    <compileflags>-Os 
    <compileflags>-fno-strict-aliasing 
    <compileflags>-O2 
    <compileflags>-DNDEBUG 
    <compileflags>-g 
    <compileflags>-I$(AndroidNDKRoot)/sources/cxx-stl/gnu-libstdc++/$(GCC_ver)/include 
    <compileflags>-I$(AndroidNDKRoot)/sources/cxx-stl/gnu-libstdc++/$(GCC_ver)/libs/$(ABI)/include 
    <compileflags>-L$(AndroidNDKRoot)/sources/cxx-stl/gnu-libstdc++/$(GCC_ver)/libs/$(ABI) 
    <compileflags>-lgnustl_static
    <compileflags>-D__GLIBC__ 
    <compileflags>--std=c++11 
    <archiver>$(AndroidNDKRoot)/toolchains/$(TOOLCHAIN)-$(GCC_ver)/prebuilt/$(HOST)/bin/arm-linux-androideabi-ar 
    <ranlib>$(AndroidNDKRoot)/toolchains/$(TOOLCHAIN)-$(GCC_ver)/prebuilt/$(HOST)/bin/arm-linux-androideabi-ranlib 
    ;

Edit configuration variables at the beginning of the file according to your actual setup. Please note that a Boost.Build configuration directive must end with a space+semicolon  ;. Also forward slashes / are used as path separators regardless of your OS.

If you already have user-config.jam file in your home directory (from some other project) I recommend to back-up it into a separate location and replace with the preceding configuration to avoid possible problems with Boost libraries cross-compilation.

Now go to your Boost directory and run the following command:

b2 -q -j2 toolset=gcc-android cxxflags=-fPIC target-os=linux threading=multi link=static runtime-link=shared --layout=system --stagedir=android --with-python

This command will build the necessary libboost_python.a library and place it into android\lib sub-directory inside your Boost directory. If you need to build other Boost libraries for your project, specify them using --with-<lib_name> option.

The following example shows our "Hello World" module implemented with Boost.Python:

#include <boost/python.hpp>
#include <string>

std::string get_greet()
{
    return std::string("Hello World!");
}

// Boost.Python wrapper
BOOST_PYTHON_MODULE(hello)
{
    boost::python::def("get_greet", get_greet);
}

As you can see, there are minimal differences with the Pybind11 example in the preceding chapter. The module wrapper is also implemented with a macro but, unlike in Pybind11, in Boost.Python the module object is defined implicitly, although you can access it via boost::python::scope class if necessary.

To build a binary module based on Boost.Python you need to add the respective header files and libraries to our Android NDK build configuration like in the following Android.mk file:

LOCAL_PATH := $(call my-dir)

include $(CLEAR_VARS)

LOCAL_MODULE    := hello  # name your module here.
LOCAL_SRC_FILES := ../src/hello.cpp
LOCAL_C_INCLUDES := $(LOCAL_PATH)/../include/Python2.6 c:/boost_1_59_0
LOCAL_LDLIBS := -L$(LOCAL_PATH)/../lib -lkodi -lboost_python
LOCAL_WHOLE_STATIC_LIBRARIES := gnustl_static

include $(BUILD_SHARED_LIBRARY)

The preceding configuration assumes that you have copied the cross-compiled boost_python library into your project's \lib directory.

Conclusion

In this series of articles I described how to build binary Python extension modules for Kodi addons on Android platform. Although building such modules is not a simple task, you may need to use such modules for various reasons, for example to speed-up a CPU-intensive task or to bring some existing Python library with binary components to Android. I hope you will find this information useful.

And as I have warned you at the beginning of the 1st article, there's a strong chance that your binary module won't work. Use this information at your own risk.

  Android, Boost, C++, Kodi, Plugin, Python

submit to reddit