Every programming language has its benefits and its drawbacks. Python, for example, provides a dynamic and clean environment for the developer, which boosts productivity by hiding much of the “dirty work”: memory management, type inference (duck typing, in this case), and more. Such benefits usually come at the cost of efficiency, which is something that other programming languages (such as C++) excel in.

The next logical step would then be a way to combine the benefits of such different programming languages in a single system. For instance, we could use C++ solely for the cpu-intensive parts, while the rest of the software could very well be written entirely in Python. The missing piece would then be a way to integrate the two. The common way of doing so is through SWIG, which is a tool that is able to wrap C++ to many different programming languages, Python included. Using SWIG right out of the box, we will be able to invoke C++ code directly from a Python interpreter. But what if we wanted to extend C++ classes in Python, making native C++ seamlessly call Python code?

Let me better illustrate the situation: suppose a big portion of your software consists of a heavy computation, but there are some flavours of it – the main differences can be refactored to a fairly sophisticated work that needs to happen just before the computation (“pre-compute”) or right after it (“post-compute”). The straightforward solution would then be to implement the core logic in C++, and allow Python code to run both pre-compute and post-compute.

Let us module the problem as follows, handling only the pre-compute part:

// File: wrapped.h #include <iostream> struct Job { virtual void pre () { std::cout << "C++!" << std::endl; } virtual int compute () { pre(); return 42; // quite cpu-intensive } virtual ~Job {} };

You may obviously assume that such polymorphic Jobs are executed elsewhere.

Now, employing SWIG we will be able to use this code in Python:

>>> import wrapped >>> j = wrapped.Job() >>> j.compute() C++! 42 >>>

We have successfully invoked C++ code from Python, which is all good and well.

But when we try to create a specialised MyJob class in Python, which derives from the given Job and does something interesting within pre(), we discover a problem. The C++ dynamic dispatch mechanism is unable to invoke the new Python code from within the old C++ code:

>>> import wrapped >>> class MyJob(wrapped.Job): ... def __init__(self): ... super(MyJob, self).__init__() ... def pre(self): ... print "Python!" ... >>> j = MyJob() >>> j.compute() C++! 42 >>>

Recalling how C++ vtables actually work, and having even the most basic understanding of what SWIG does, this behaviour is fairly obvious: since SWIG (by default) does not generate any classes deriving from Job in its wrapping process, the matching C++ code has already been compiled containing a reference only to the original Job::pre() function in its vtable.

As you could guess by now, a feasible solution for the problem at hand could be automatic (SWIG-)generated deriving classes. These classes should just override all such virtual methods with new methods whose implementation would be to either invoke the most deriving C++ implementation, or the most recent Python implementation (if one exists). This can be achieved through the use of Python directors in SWIG. The underlying implementation is interesting and I would suggest having a glimpse at it, but the usage is fairly simple and only requires enabling the “directors=1” option in SWIG.

Enabling the Python directors feature for our setup yields the desired behaviour, which actually allows Python and C++ perfectly interleave:

>>> import wrapped >>> class MyJob(wrapped.Job): ... def __init__(self): ... super(MyJob, self).__init__() ... def pre(self): ... print "Python!" ... >>> j = MyJob() >>> j.compute() Python! 42 >>>

As a final note I would like to mention that, at least in my eyes, the presented workflow — of interleaving dynamic languages (such as Python, or TCL) and C++ — attempts to combine the best of the two worlds while keeping the worst out. It is a great, efficient, and effective synergy.

For the sake of completeness, the final SWIG interface file is hereby presented, along with its invocation command-lines:

// File: wrapped.i // To generate Python bindings: // swig -python -c++ -o wrap.cc wrapped.i // g++ -fPIC -dynamiclib -lpython -I/usr/include/python2.7 -L/usr/lib/python2.7 wrap.cc -o _wrapped.so %module(directors="1") "wrapped" %{ #include "wrapped.h" %} %feature("director"); %include "wrapped.h"