CPython Extensions for IronPython

A Proof of Concept Using Python.NET

Note This article describes one way to use CPython extensions from IronPython / .NET by embedding CPython itself via Python.NET. Since this article was written a much more practical approach has been implemented as an Open Source project by Resolver Systems. This project fakes Python25.dll and has a large subset of the Python C API implemented in C#. A large percentage (around a thousand tests at the time of writing) of the Numpy test suite passes when used from IronPython and other whole extension libraries are usable: Ironclad Introduction

Ironclad Project Page (with Downloads)

Introduction This module came out of The C Extensions for IronPython Project started by Resolver Systems. This module allow you to access CPython modules from IronPython. It works by embedding a CPython interpreter, using a an assembly provided by the Python.NET Project where most of the hard work was done!. Despite some serious limitations it works! The example code included uses matplotlib with numpy and Tkinter from IronPython. It provides two different usage patterns: An import hook allowing you to import CPython binary modules with normal import statements

An Import function allowing you to import modules (pure Python or binary modules) into the embedded CPython interpreter See the usage section for examples. Download The download includes Python.Runtime.dll from Python.NET, for a UCS2 build of Python. This is the right assembly for Python 2.4 on Windows. For Linux you probably want a UCS4 build. The Python.NET distribution contains runtimes for Python 2.4 and 2.5 in both UCS2 and UCS4. This technique does depend on having the appropriate version of Python installed (although you could also ship the relevant Python dll). Download cext 0.1.4 The source for this module lives in the FePy Subversion Repository: embedding.py

cext.py (the import hook) This code only works with IronPython 2 because of an annoying bug when passing Arrays as arguments in IronPython 1. This can be worked around, but targeting IronPython 2 may be better as we could look at using the DLR to help with making PyObjects behave like IronPython data types (see below). Bug reports, contributions and suggestions welcomed.

The C Extensions for IronPython Project Resolver Systems recently announced a new project to get CPython extensions working seamlessly with IronPython. This is needed by customers of Resolver, but is being run as an Open Source project. Announcement: Project to get some CPython C extensions running under IronPython

C Extensions for IronPython Mailing List The approach in this module, is to embed the CPython interpreter in an assembly and access CPython extensions through the hosted interpreter. This module is far from the final solution and it's not even clear that it is the best approach to take. At Resolver we have also been experimenting with directly loading CPython assemblies and replacing the CPython API with function pointers that use delegates to call back into managed code. This would also give us binary compatibility and avoid some of the problems caused by hosting a real CPython interpreter. So far we have Python binary extensions loading, calling into our code and then crashing! This is a great first step because it means the basic approach works, and know all we need to do is implement the whole Python C API in managed code...

Usage The distribution includes the following files: embedding.py : The main module

: The main module cext.py : The import hook

: The import hook test.py : A simple test of the basic functionality - run this with IronPython 2

: A simple test of the basic functionality - run this with IronPython 2 echo.py : A test module that is imported into CPython by test.py

: A test module that is imported into CPython by Python.Runtime.dll - Suitable for a UCS2 build of Python 2.4 Note Due to some bug in the Orcas Beta, the import hook doesn't work if you have any of Visual Studio 2008 betas (Orcas) installed. There are two ways of using this project. The first way is with the Import function from the embedding module. The Import Function The test.py shows this approach in action. The final part of this module generates a plot using matplotlib :

pylab = Import ( 'pylab' )

pylab . plot ( [ 1 , 1 , 1.5 , 2.5 , 3 , 3 , 3.1 ] )

pylab . show ( ) This generates a simple plot: As you can see, the the pylab module imported from CPython behaves in (apparently) the same way as it does when running directly in CPython. Caution! The actual code in test.py imports the sys module from the hosted interpeter, and attempts to adds to sys.path . Ufortunately this has no effect! (Although the path seems to be set correctly for my computer anyway.) The reason it doesn't work highlights something to be aware of if you want to use this module. sys = Import ( 'sys' )

sys . path . append ( 'c:\\Python24\\Lib\\' ) In the code above, the sys module is successfully imported as a proxy object. When you access sys.path , this proxy object recognises that you are accessing a Python list (on the CPython side) and copies it across to IronPython for you. This means that the append is executed on the copy, not on the original. d'oh The solution would be, either to not copy the list and to proxy access to it as well, or to provide functions on the CPython side allowing you to manipulate sys.path . The Import Hook Installing the import hook allows you import Python binary extensions using normal import statements! Python binary extensions are .pyd files on Windows and .so files on other platforms. To install the import hook, execute the following code: import cext

cext . install ( ) You can then do things like import cElementTree . The goal is that eventually this will be build into FePy and enabled by an option, so that you can import CPython modules without having to take any special steps. Under the hood, the import hook uses the embedding.Import function. When you use the import hook to import binary modules you may want to do things like setting the import path on the hosted interpreter. Obviously normal import statements (like import sys ) will import the IronPython version. To access the builtin modules of CPython you will still need to use embedding.Import .

Implementation Details The Python Runtime Assembly This provides a very thin wrapper around the CPython embedding API. You have to acquire and release the Global Interpreter Lock (GIL) around every operation and it works with PyObjects which are managed wrappers around CPython types. The code below imports the PythonEngine and initializes it. It also defines two decorators GIL acquires and releases the GIL, and wraps all operations with CPython objects.

acquires and releases the GIL, and wraps all operations with CPython objects. handle_exception handles exceptions that occur in CPython and reraises them as IronPython exceptions import clr

clr . AddReference ( 'Python.Runtime' )

from Python . Runtime import PythonEngine



engine = PythonEngine ( )

engine . Initialize ( )



def lock ( ) :

h = engine . AcquireLock ( )

def unlock ( ) :

engine . ReleaseLock ( h )

return unlock





def GIL ( function ) :

def f ( * args , ** keywargs ) :

unlock = lock ( )

try :

ret = function ( * args , ** keywargs )

finally :

unlock ( )

return ret

return f





def handle_exception ( function ) :

def f ( * args , ** kw ) :

try :

return function ( * args , ** kw )

except PythonException , e :

exc = PyExcConvert ( PyObject ( e . PyType ) )

value = ConvertToIpy ( PyObject ( e . PyValue ) )

raise exc ( value )

return f Importing modules is done with the Import function. This could be the only function you need to import from the embedding module to access CPython extensions. @ GIL

def Import ( name ) :

module = engine . ImportModule ( name )

if module is None :

raise ImportError ( "Importing module named %s failed" % name )

return PythonObject ( module ) Proxied Objects and Data Structures When you import a module it returns a proxied object. By default all objects you access are proxied objects unless they are a fundamental datatype - which will be converted from a PyObject into the equivalent IronPython type. Proxied objects let you get and set attributes on them, plus call them. The proxied class is PythonObject : class PythonObject ( object ) :

def __init__ ( self , real ) :

self . _real_ = real



@ GIL

@ handle_exception

def __getattribute__ ( self , name ) :

real = object . __getattribute__ ( self , '_real_' )

if name == '_real_' :

return real



return ConvertToIpy ( real . GetAttr ( name ) )





@ GIL

@ handle_exception

def __setattr__ ( self , name , value ) :

if name == '_real_' :

object . __setattr__ ( self , '_real_' , value )

return



self . _real_ . SetAttr ( name , ConvertToPy ( value ) )





@ GIL

@ handle_exception

def __call__ ( self , * args , ** keywargs ) :

if not keywargs :

return ConvertToIpy ( self . _real_ . Invoke ( ConvertToPy ( args ) ) )

return ConvertToIpy ( self . _real_ . Invoke ( ConvertToPy ( args ) ,

StringDictToPy ( keywargs ) ) ) The ConvertToPy and ConvertToIPy functions do the converting between CPython and IronPython types. It can convert integers, long integers, strings, booleans and None, lists, tuples and dictionaries. This means that if you call a Python function (or set an attribute) with a reference to an IronPython data structure it will be copied into CPython types. Any return values will be copied from CPython types to IronPython types. See the limitations section below for some of the consequences of this. It does mean that once you have imported a module you can call functions / methods and instantiate classes from inside CPython. You can also fetch and set attributes and all the right things should happen.

The Test File test.py tests the basic functionality of the embedding.py module. It also serves as a usage example. When run with IronPython 2, it imports echo.py into CPython and passes data back and forth to check that it survives the journey. These are done with asserts, so if any fail then it will bomb out with an error. If it works you should see: Received into CPython: 1 Type: <type 'int'> Received into CPython: 3.2000000000000002 Type: <type 'float'> Received into CPython: u'Hello from IronPython' Type: <type 'unicode'> Received into CPython: 10000000000L Type: <type 'long'> Received into CPython: None Type: <type 'NoneType'> Received into CPython: True Type: <type 'bool'> Received into CPython: False Type: <type 'bool'> Received into CPython: [] Type: <type 'list'> Received into CPython: () Type: <type 'tuple'> Received into CPython: {} Type: <type 'dict'> Received into CPython: ({u'something': 3.2000000000000002}, u'hello', u'from', u'ironpython', [1, 2, 3]) Type: <type 'tuple'> setting something something else setting something something else fetching something fetching something Received into CPython: 123 Type: <type 'int'> Caught an exception from CPython correctly. test.py also tests other features of this module, like catching exceptions from CPython (etc).