Python Programming, news on the Voidspace Python Projects and all things techie.

Reasons to Love Python

I've been dabbling in C# recently, and I'm afraid I like it. As statically typed languages go it's a good one. However, on coming back to Python I remember how much I like it. It is not just that it is dynamically typed, but it is all the language features that make it concise and expressive.

A programming language is a medium of expression. —Paul Graham

Here are a few of my favourites (in no particular order):

Generator Expressions

Python generators are very cool, and have been around since Python 2.2 (2002). They behave very similarly to the Yield Return introduced in C# 2.0 in 2005.

Also in Python 2.2 were list comprehensions (similar to LINQ over objects introduced in .NET 3.5 in 2007) that allow you to combine a loop and filter in a single expression:

result = [ f ( value ) for value in iterable if value > 0 ]

The expression in the square brackets is evaluated immediately and is identical to:

result = [ ]

for value in iterable :

if value > 0 :

result . append ( f ( value ) )

Python 2.4 (2004) introduced a novel extension to list comprehensions: generator expressions. Generator expressions are similar to list comprehensions, but instead of square brackets they use parentheses.

generator = ( f ( value ) for value in iterable if value > 0 )

The major difference is that instead of being evaluated eagerly they are evaluated lazily. The generator expression returns a generator object that is not executed until you consume it by iterating over it. This means they can be more memory efficient as you can consume individual items from the generator without creating the whole list up front.

One side effect is that they look nicer as the arguments to a function that takes an iterable:

sum ( f ( value ) for value in iterable if value > 0 )

Generators are naturally first class objects that you can pass around your code. David Beazley has an excellent set of slides on using generators for systems programming (and will be giving a tutorial on them at PyCon UK): Generator Tricks for Systems Programmers

One of the examples he gives is a pipeline of generator expressions for parsing Apache log files and summing the amount of data served:

wwwlog = open ( "access-log" )

bytecolumn = ( line . rsplit ( None , 1 ) [ 1 ] for line in wwwlog )

bytes = ( int ( x ) for x in bytecolumn if x != '-' )

print "Total" , sum ( bytes )

None of the generators are consumed until the final call to sum . As it iterates a line at a time (not keeping the log file in memory) it can handle huge log files - and as a bonus it runs faster than a typical solution with loops!

Everything is an Object

Everything in Python is an object (even None ) - functions, classes, methods and modules are all first class objects.

This allows for wonderful programming techniques including higher order functions, factory functions (for classes and functions), creating modules at runtime from databases, meta-programming and so on. It makes testing massively easier and is particularly useful when combined with Python's powerful introspection (inspect any object at runtime to determine its capabilities).

Tuples, Tuple Unpacking and Multiple Return Values

Tuples are one of the built-in container types. Although experts may 'decry' the description, they are effectively immutable lists.

The syntax is straightforward: (1, 2) is a tuple containing two numbers. They can be indexed or iterated over like lists and are a fantastic way of grouping values where otherwise you might create a custom class just to hold a pair of values! They are not unlike the new anonymous types that were introduced along with LINQ in C# - but those you can only use inside the scope where you create them.

Tuples in Python turn up implicitly in a few places, and one of these is where a function or method returns multiple values:

def function ( a , b ) :

return a * b , a - b

The return value from function is a tuple of two values. Being able to return multiple values is enormously useful - especially if you use it with tuple unpacking. Python can 'unpack' tuples in assignment statements:

product , difference = function ( a , b )

It is fair to say that out parameters are only needed in C# because it can't return multiple values (again the alternative is creating a custom class and returning an instance of that instead).

You can also unpack tuples into function calls:

parameters = ( a , b )

product , difference = function ( * parameters )

And the converse is collecting all positional arguments into a tuple (similar to the C# params method signature):

def function ( * args ) :

a , b = args

return a * b , a - b

Keyword Arguments

Another small language feature, but invaluable for creating flexible and usable APIs: keyword arguments with default values.

def function ( arg1 = 3 , arg2 = None , arg3 = 'fish' ) :

. . .



function ( arg1 = 2 )

function ( arg1 = 2 , arg3 = 'nothing' )

function ( 2 , None , 'something' )

Arguments with default values can be called positionally (as normal) or by keyword. This allows you to call a function or method only overriding values that differ from the default.

In fact it is so useful that support for it has been built into IronPython for working with .NET objects. Keyword arguments used in a constructor are the equivalent to constructing the instance and then setting properties afterwards:

form = Form ( Text = "Form Title" )

Decorators

One consequence of first class functions is that 'wrapping' functions (one example of higher order functions) becomes possible. Decorators were introduced to provide a convenient syntax for doing that. I didn't follow the debate as I wasn't interested in the feature and I didn't think I would use it. I was wrong - they turn out to be massively convenient for all sorts of things.

A simple example of wrapping a function (without decorator syntax) for exception handling and logging:

def wrapper ( function ) :

def inner ( * args , ** keywargs ) :

try :

return function ( * args , ** keywargs )

except Exception , e :

logger . log ( "Exception occurred in function '%s': %s" %

( function . __name__ , e ) )

return inner



wrapped = wrapper ( function )

The wrapper function takes a function as an argument. It defines an inner function that calls the original function (keeping a reference to it through a closure) with exception handling and logging. It returns the inner function that can be used in the place of the original. (The *args, **keywargs syntax captures all the positional and keyword arguments that inner is called with and calls function with the same arguments.)

Having created the wrapper function we can actually do the wrapping with the decorator syntax:

@ wrapper

def function ( a , b ) :

. . .

We use this all the time at Resolver Systems - for profiling, mocking out names and 'auto-unmocking' them within the scope of a single function, invoking methods onto control threads and so on.

The Interactive Interpreter

No discussion of useful Python features can be complete without a mention of Python's REPL ; the interactive interpreter.

You can explore new libraries or check out language features in a matter of seconds. You can even do real work from it - Tim Golden (a Python DBA and WMI guru) says that when working with databases he often uses the interactive interpreter and "slurp the data in, transform it, push it back out and walk away".

I've saved the most important (and most controversial) two for almost last:

Explicit Self

Explicit self certainly stirs up some debates. I like it - it makes Python scoping very explicit. When declaring an instance method in a class body, you declare the instance as the first argument to be passed in to the method - and by convention you name this argument self .

class SomeClass ( object ) :



def instance_method ( self , arg1 , arg2 ) :

. . .

The useful thing about this is that you can see at a glance which instance attributes your method is using - they are all prefixed with self .

It also makes calling up to base class methods straightforward and consistent without requiring additional syntax:

class SomeClass ( BaseClass ) :



def instance_method ( self , arg1 , arg2 ) :

BaseClass . instance_method ( self , arg1 , arg2 )

Of course if you really can't cope with the explicit self then you can always use my Selfless Metaclass that uses bytecode hackery to remove the need to declare it.

Indentation Based Block Structure

Some people really don't like this, which to be honest baffles me a bit. Here's what I wrote for the Why Separate Sections by Indentation Instead of by Brackets or End question on the Python wiki:

In order to separate blocks of code (like for loops, if blocks and function definitions) the compiler / interpreter needs something to tell it when a block ends. Curly braces and end statements are perfectly valid ways of providing this information for the compiler. For a human to be able to read the code indentation is a much better way of providing the visual cues about block structure. As indentation also contains all the information for the compiler, to use both would be redundant. As indentation is better for humans, it makes sense to use that for the compiler too. It has the advantage that Python programs tend to be uniformly and consistently indented, removing one hurdle to understanding other people's code. Python does not mandate how you indent (two spaces or four, tabs or spaces - but not both), just that you do it consistently. Those that get used to the Python way of doing things tend to start seeing curly braces as unnecessary line noise that clutters code. On the other hand, 'the whitespace thing' is possibly the single biggest reason why some developers refuse to even try Python.

Interesting to note that both Haskell and F# also allow you to delimit block structure by indentation (plus Python influenced languages like Boo and Cobra).

I haven't even mentioned language features that come as a consequence of being a dynamically typed language: duck typing (protocols instead of interfaces), heterogeneous container types (no need for generics) and many more...

PyCon UK: IronPython Tutorial, Socials and Volunteers Needed

The PyCon UK conference (national UK Python conference) draws inexorably nearer - just over a week to go now. We've currently got over 200 people registered and about 80 for the tutorials so it should be a great conference.

If you're coming, don't forget to register for the social events on the Thursday and Friday evenings:

Today Christian, Menno and I have been working on our IronPython Tutorial. The handout notes and example code are basically done, just a little polishing off to do.

The tutorial is based around a simple Twitter client, which Menno has named Twatter! We've been testing Twatter on Windows, Linux and the Mac (it doesn't look bad on the Mac).

At each step we will be introducing a new topic and explain the principles. We'll give the attendees the skeleton of the code and then assist you to add new functionality. It is a very simple application, but we will manage to cover topics including (all using .NET APIs from IronPython):

Windows Forms

Databases

Web services and network access

Handling XML

Threading

Preparations for the conference itself are going well. We have great sponsorship this year from quite a few companies. Resolver Systems is paying for the drinks on the Saturday night dinner!

What we still need is more volunteers to help with the practical arrangements. We particularly need volunteer Session Chairs who will introduce the speakers and make sure they finish on time! We also need help putting up the signs on the Thursday and Friday. If you can help please sign up on the volunteer wiki pages or email us using the address from the main PyCon UK website.

Archives