Python Programming, news on the Voidspace Python Projects and all things techie.

Psyco 2 Binaries for Windows and Python 2.4, 2.5 and 2.6

Pysco is a specializing compiler (a kind of JIT) for Python written by Armin Rigo. The difficulty of maintaining and extending psyco was one of the motivating factors behind the inception of PyPy. Psyco itself was maintained to remain compatible with recent versions of Python but still didn't optimise more recent features like Python generators and floats.

Development of psyco was recently taken over by Christian Tismer. Christian made many improvements to Psyco, but Windows binaries were never released. Available here are compiled binaries of psyco 2 for Windows for Python 2.4, 2.5 and 2.6.

(Compiled from SVN head on January 11th 2010.)

From the official psyco project page on Sourceforge, prebuilt binaries are only available for Python 2.5. I've also built binaries for psyco 1.6 for Python 2.4 and 2.6:

Binaries for Python 2.2 and 2.3 (pysco 1.5.1) are available from my Python modules page.

A Rambling Recording on Member Lookup in Python (podcast)

I was thinking about the Python object model, in part as a result of my post on The Python Class Statement. Python is a really easy language to learn, but it also has advanced features like its protocols, descriptors and metaclasses, that make the full object model pretty complex - and that's before you start looking at the corner cases.

It would be really nice to write up a single document describing the Python object model, including all of its intricacies. That sounds too much like hard work, so instead I recorded a rambling hand-wavy description of member lookup in Python. I don't go into full blown detail, but then this is a podcast - it won't seriously mislead you and no-one is going to use it as a reference guide...

This was recorded using the Blue Fire iPhone app whilst I was wandering around outside. I chopped out about half my pauses and coughing using Audacity, so if you think the quality is rough you should have heard the first version.

Topics covered include:

Member lookup on instances and classes

How the interpreter looks up protocol ('magic') methods

__getattr__ and its mysterious cousin __getattribute__

and its mysterious cousin Descriptors, bound methods, properties and friends

In the podcast I mention the new technique I have for dynamically mocking magic methods. Magic methods, when they are called for you by the interpreter, are usually looked up directly on the class. Unfortunately Python is not entirely consistent, some magic methods are still looked up on the instance first before the class. This is gradually being fixed in Python (in 2.7 they pretty much all fixed), but the inconsistency is a pain for mocking the magic methods.

Mock now allows you to mock the magic methods by assigning an appropriate function, that takes self as the first argument, to the magic method on the mock instance. By default mocks do not have the magic methods implemented except the ones it uses itself. When you assign to them it dynamically grows them on just that instance - all other mock instances are unaffected. Magic methods can then be looked up on the class or the instance, either way works (and you can delete them):

>>> from mock import Mock >>> m = Mock () >>> m <mock.Mock object at 0x429770> >>> m . __repr__ = lambda self : 'A Mock Object' >>> m A Mock Object >>> m . __repr__ () 'A Mock Object' >>> del m . __repr__ >>> m <mock.Mock object at 0x429770>

You can also use Mocks for magic methods. Here's an example of mocking out the built-in open function when used as a context manager:

@patch ( '__builtin__.open' ) def test_with_statement ( self , mock_open ): mock_open . __enter__ = Mock () mock_open . __exit__ = Mock () mock_open . __exit__ . return_value = False with open ( 'filename' ) as handle : handle . read () mock_open . assert_called_with ( 'filename' ) mock . __enter__ . assert_called_with () mock . __enter__ . return_value . read . assert_called_with () mock . __exit__ . assert_called_with ( None , None , None )

The version of mock with magic method support hasn't yet been released, but you can pull it from the google code SVN repo. When I have time to write docs it will be released as 0.7.0.

There's a bit of trickery involved in making this work. If you're interested in how it's done look at the implementation of __new__ and __setattr__ .

Notes on the Python Class Statement

Python classes are created at runtime, usually when you execute a script, or import the module they are defined in. Class creation is done primarily with the class statement. The class statement is executed by the Python runtime to create the class. Functions and names assigned in the body of the class statement become methods and attributes of the class.

You can easily see that the code inside the body of the class is executed, and that it can contain arbitrary code, by putting a print statement inside the class body:

>>> class ClassName ( object ): ... print 'hello world' ... hello world

Any assignments that happen in the body of the class definition create class members. Class and function definitions both cause names to be assigned, so classes defined inside the body of another class statement can be accessed as class attributes and functions defined inside the body of a class become methods.

Here's a trivial example with simply assigning a value to the name X:

>>> class SomeClass ( object ): ... X = 3 ... >>> SomeClass . X 3

We can combine the fact that arbitrary code is executed with the assignment rule to conditionally define class members:

>>> import sys >>> class SomeClass ( object ): ... if sys . platform == 'darwin' : ... X = 3 ... else : ... X = 4 ... >>> SomeClass . X 3

What happens in class creation (in Python 2 - the rules change slightly in Python 3 as the metaclass mechanism is improved) is that the class body is executed, the collection of names and values are passed as a dictionary (along with the class name and a tuple of the base classes) to the metaclass which is 'called' (if the metaclass is a type - which it usually is - the metaclass is instantiated) and the resulting class object is assigned to the name in the scope in which it was defined. The resulting class is an object like everything else in Python. Unless the class uses __slots__ the dictionary of members becomes the class __dict__ . This dictionary is protected by being wrapped in a dictproxy . Although you can fetch members directly from the dictproxy you can't directly assign or delete members, instead you have to go through the normal attribute setting / deleting mechanisms:

>>> class SomeClass ( object ): ... X = 3 ... >>> SomeClass . __dict__ < dictproxy object at 0x50b7b0 > >>> SomeClass . __dict__ . keys () [ '__dict__' , 'X' , '__module__' , '__weakref__' , '__doc__' ] >>> SomeClass . __dict__ [ 'X' ] 3 >>> SomeClass . __dict__ [ 'Y' ] = 4 Traceback ( most recent call last ): File "<stdin>" , line 1 , in < module > TypeError : 'dictproxy' object does not support item assignment >>> del SomeClass . __dict__ [ 'X' ] Traceback ( most recent call last ): File "<stdin>" , line 1 , in < module > TypeError : 'dictproxy' object does not support item deletion >>> SomeClass . Y = 4 >>> del SomeClass . X >>> # X has now gone from the __dict__ and Y appeared >>> SomeClass . __dict__ . keys () [ '__module__' , 'Y' , '__dict__' , '__weakref__' , '__doc__' ]

An interesting example of assignment creating class members is what happens when you put a list comprehension inside a class body. An implementation detail of list comprehensions is that variables used in the list comprehension 'leak' into their surrounding scope. A list comprehension in a class body creates an unexpected class member:

>>> class SomeClass ( object ): ... [ foo for foo in ( 1 , 2 , 3 )] ... >>> SomeClass . foo 3

The same isn't true of generator expressions where the variable doesn't leak:

>>> class AnotherClass ( object ): ... list ( bar for bar in ( 1 , 2 , 3 )) >>> AnotherClass . bar Traceback (most recent call last): File "<stdin>" , line 1 , in <module> AttributeError : type object 'AnotherClass' has no attribute 'bar'

The variable leaking from list comprehensions is a side-effect and should not be relied on.

Whilst the code in the class statement is being executed it creates a temporary namespace. Code can refer to names already assigned as if they were local variables.

>>> class SomeClass ( object ): ... X = 3 ... b = [ a * X for a in ( 1 , 2 , 3 )] ... >>> SomeClass . b [3, 6, 9]

A common use for this is to create aliases, where you give the same member two or more names. In this example cost is an alias to the calculate_price method:

>>> class SomeClass ( object ): ... def calculate_price ( self , quantity ): ... return quantity * 10.0 ... cost = calculate_price ... >>> instance = SomeClass () >>> instance . calculate_price ( 20 ) 200.0 >>> instance . cost ( 20 ) 200.0

It is also the standard way of creating properties before Python 2.6:

>>> class SomeClass ( object ): ... _value = None ... def get ( self ): ... return self . _value ... def set ( self , value ): ... self . _value = value ... value = property ( get , set ) ...

The value property is created using the get and set functions from the scope that forms the class members.

Unfortunately we have a problem with generator expressions. Generator expressions create their own scope, causing names to be looked up lexically and ignoring the temporary class scope.

>>> class AnotherClass ( object ): ... X = 3 ... b = list ( a * X for a in ( 1 , 2 , 3 )) ... Traceback (most recent call last): File "<stdin>" , line 3 , in AnotherClass File "<stdin>" , line 3 , in <genexpr> NameError : global name 'x' is not defined

If you're interested in how metaclasses are involved in class creation then you should read: Metaclasses in five minutes. (Hopefully readable even for non-gurus.)

An interesting reference on why the class statement in Python contains executable code is this article by Guido van Rossum, the creator of Python: How Everything Became an Executable Statement.

Release: ConfigObj 4.7.0 and validate 1.0.1

I've just released ConfigObj 4.7.0 and validate 1.0.1. ConfigObj is an easy to use configuration file reader and writer module. ConfigObj is easy to use, with a host of powerful features including:

Nested sections (subsections), to any level

List values

Multiple line values

String interpolation (substitution)

Integrated with a powerful validation system including automatic type checking/conversion repeated sections and allowing default values

When writing out config files, ConfigObj preserves all comments and the order of members and sections

Many useful methods and options for working with configuration files (like the 'reload' method)

Full Unicode support

The headline feature of 4.7.0 is an ~25% performance improvement in reading and validating configuration files. The cost of this performance improvement is losing compatibility with Python 2.2 . Python 2.3 is now the minimum version of Python supported by ConfigObj.

The next new feature is the addition of the extra_values section attribute and the get_extra_values function. extra_values is populated by validation and lists all config file members (in that section) that weren't specified in the configspec file. get_extra_values is a convenience function that returns a list of all values and sections that appear in the config file but aren't in the configspec.

Another important change is that passing in arguments to the ConfigObj constructor via an options dictionary is not possible any more. This was a hangover from previous versions and if you were using it you can just change you code to the following (that works with older versions of ConfigObj as well):

config = ConfigObj ( filename , ** options )

4.7.0 also brings in a minor change in syntax. Previously spurious commas in list values would be ignored (e.g. value = first, , second ). They are now invalid syntax.

There have been several other minor bugfixes. See the list below for details.

All Changes in Version 4.7.0

Minimum supported version of Python is now 2.3

~25% performance improvement thanks to Christian Heimes

String interpolation now works in list value members

After validation any additional entries not in the configspec are listed in the extra_values section member

section member Addition of the get_extra_values function for finding all extra values in a validated ConfigObj instance

function for finding all extra values in a validated ConfigObj instance Deprecated the use of the options dictionary in the ConfigObj constructor and added explicit keyword arguments instead. Use **options if you want to initialise a ConfigObj instance from a dictionary

dictionary in the ConfigObj constructor and added explicit keyword arguments instead. Use **options if you want to initialise a ConfigObj instance from a dictionary Constructing a ConfigObj from an existing ConfigObj instance now preserves the order of values and sections from the original instance in the new one

BUGFIX: Checks that failed validation would not populate default_values and restore_default_value() wouldn't work for those entries

and wouldn't work for those entries BUGFIX: clear() now clears 'defaults'

BUGFIX: empty values in list values were accidentally valid syntax. They now raise a ParseError . e.g. "value = 1, , 2"

. e.g. "value = 1, , 2" BUGFIX: Change to the result of a call to validate when preserve_errors is True. Previously sections where all values failed validation would return False for the section rather than preserving the errors. False will now only be returned for a section if it is missing

when is True. Previously sections where all values failed validation would return False for the section rather than preserving the errors. False will now only be returned for a section if it is missing Distribution includes version 1.0.1 of validate.py

Removed __revision__ and __docformat__

The only change in version 1.0.1 of validate was removing a test whose behaviour was dependent on platform / Python version and was impossible to make pass consistently.

PyCrypto 2.1.0 Binaries for Windows 32bit Python 2.6, 2.5 and 2.4

PyCrypto is a Python cryptography package originally created by Andrew Kuchling and now maintained by Dwayne C. Litzenberger. For a while I've been hosting Windows binaries for version 2.0.1. Dwayne has now done a new release, version 2.1.0.

I've built installers for 32 bit Windows for Python 2.6, 2.5 and 2.4. The 2.6 installer was built with Visual Studio 2008. The Python 2.5 and 2.4 installers were built with Visual Studio .NET 2003. They were built without GMP support (only needed for Crypto.PublicKey._fastmath ). The installers come without guarantees, but all the tests pass.

Archives