Skip to end of metadata
Go to start of metadata

Exposing classes and functions defined in C++ libraries to Python is now possible in comma by creating C++/Python bindings with Boost.Python.

Example

To illustrate this new capability, bindings for the C++ class format and its member function size() declared in csv/format.h have been defined in python/comma/cpp_bindings/csv.cpp:

// python/comma/cpp_bindings/csv.cpp
#include <boost/python.hpp>
#include <comma/csv/format.h>
BOOST_PYTHON_MODULE( csv )
{
    boost::python::class_< comma::csv::format >( "format", boost::python::init< const std::string& >() )
        .def( "size", &comma::csv::format::size );
	// add other csv bindings here
}
 

and added to cmake:

# fragment of python/comma/cpp_bindings/CMakeLists.txt

add_cpp_module( csv csv.cpp comma_csv )
# add other modules here

Build comma with BUILD_SHARED_LIBS=ON and BUILD_CPP_PYTHON_BINDINGS=ON, then open a python interpreter and enter the following commands:

>>> import comma.cpp_bindings.csv as csv
>>> f = csv.format('d,2ub,s[5]')
>>> f.size()
15

The function size() outputs binary size corresponding to the format string that was passed to the format object f on construction.

Under the hood

The bindings are placed inside a shared library and saved in the file csv.so, which is then installed in comma/cpp_bindings along with the other Python modules. On Ubuntu, it will usually be /usr/local/lib/python2.7/site-packages/comma/cpp_bindings/csv.so. Note that the name of the module used as a parameter for BOOST_PYTHON_MODULE macros has to match the name of the shared library declared in the cmake file, e.g. csv in the above example. 

Limitations

The bindings are exposed as a shared library and hence one is limited to building comma with shared libraries or ensuring that all required static libraries have been compiled with -fPIC. Attempting to link with static libraries without position independent code may cause linking to fail or link with shared libraries instead.

 

  • No labels

4 Comments

  1. Pretty awesome Andrey! Does this mean a bunch of the native comma/snark python stuff should be updated to use these bindings?

    Is it an option to add the -fPIC flag by default for static builds of comma/snark? Or will that create other problems...

    We reasoned a while back that building static was generally preferable, because Seva had to keep fixing people's comma/snark installs when old so files got left lying around, or there were mismatches between comma/snark versions.

    The comment above might be encouragement to go back to dynamic builds, where maybe -fPIC is preferable?

    1. Q: Does this mean a bunch of the native comma/snark python stuff should be updated to use these bindings? 

      A: No, for a couple of reasons: 1) it would require linking with shared libraries and make distributing via rpm problematic, 2) it would take time to do it as it is not as straightforward as it sounds.

      Q: Is it an option to add the -fPIC flag by default for static builds of comma/snark?

      A: I think they are already compiled with -fPIC. The problem is that boost and other system libraries are not. However, you could try building the bindings even with BUILD_SHARED_LIBS=OFF. Most likely cmake will try to link the bindings with shared libraries anyway, with all the problems/limitations this entails.

  2. I can't stress enough how much the utility of Comma/Snark would increase by having better Python support...

    ... many academics (like me, and I clearly have a vested interest in going this direction!) do not have the time/skill/patience to wrestle with C++. I believe better Python support would catalyze adoption (if that is a goal).

    For completeness, if you are unaware: Cython, SWIG and the 'so new it is not even bleeding yet' pybind11 are other options. Seeing as boost is already a dependency, the boost binding solution is sensible. In Boost and pybind11, some clever mucking around can translate Eigen to Numpy and stl objects to native Python objects. Writing the wrappers is tedious but it also offers the chance to conceptually translate good C++ interfaces to a more Pythonic (god I hate that word) interface.

     

    (thumbs up)

    1. We have made some steps in this direction with the comma csv-style classes and functions in Python. See here: https://github.com/acfr/comma/wiki/python-csv-module. I think work in this direction will continue if there is a demand and funding.