Python Programming, news on the Voidspace Python Projects and all things techie.

Krakow, the Seaside and IronPython

For the last two days I've been at the Polish Academic Student Conference in Krakow.

I gave a talk on IronPython and Windows forms yesterday. Basically it was a longer and modified version of the talk we gave at PyCon. It went ok (talking whilst typing is difficult), and hopefully I will be able to link to a video of the talk soon. There were actually a few more people at the talk than at the PyCon one, and of course very few of them had experience of Python.

You can view the longer presentation at :

Many thanks to the organisers of the conference who have been very good at looking after the speakers.

Of course the best thing about conferences is all the great people you get to meet.

This is Jan Szumiec the organiser of the conference, Lukas Renggli a lead developer of Seaside and me.

Seaside is a very interesting web framework that uses continuations, and is written in Smalltalk (the Squeak flavour of smalltalk).

Lukas is fun, and a few of us have had a smalltalk tutorial from him. Having now written some Smalltalk I feel like I understand Ruby a bit better! (Andrzej will be pleased.)

Personally I think the syntax looks like a train wreck , but the language is very flexible and very interesting. It would probably be possible to port Seaside to stackless Python, someone definitely ought to give it a go. Another interesting thing about Smalltalk (or Squeak in particular) is that you develop in a live environment (an image based environment); editing a class changes all instances of that object. It is a very different development environment. It wouldn't be too hard to reproduce this environment for Python. You would need to produce an object browser (using the garbage collector to keep track of your instances) and re-evaluate class definition and swap out the class of instances when you make changes. Writing the browser would be the hardest part - and you would need to work on the persistence layer to persist and reload images. If I wasn't so busy I might be tempted...

Ola Bini, one of the core JRuby developers is here, and is also a great guy.

The JRuby guys have gone through a lot of the same trials that the IronPython guys have. They have solved the problem of creating classes that Java can consume (which the IronPython team haven't solved) by creating proxy classes on the fly (which call back into the JRuby engine). IronPython needs this ! Whilst I'm here I've been looking at the System.Reflection API and it certainly isn't an insoluble problem.

Note In actually got it wrong about the JRuby capabilities. They generate the proxy objects for handling subclassing of Java classes, which you can also do in IronPython. I've been looking at Reflection.Emit though, and it looks like there is a fairly straightforward route to generating new class / method objects (so long as I don't bother to handle all the nasty corner cases). I would do it by generating two assemblies per module, one with the stub classes and one with the IronPython classes. The assembly containing the IronPython classes would have to be called through the IronPython engine and have a clean API for creating and accessing instances. In order to generate the .NET classes you would need to annotate methods with a type signature for both calling and return values. Attributes on the IronPython side would need to be properties on the .NET side. I've done some playing with Reflection.Emit and I'd love to do more - but I really shouldn't.

Andrzej on Test Driven Development

My colleague Andrzej has done a new blog entry on Test Driven Development :

He goes through adding a new feature to Tabbed Images the TDD way.

The basic steps are:

Specifying the requirements

'Spiking' to work out how to implement the feature,

Writing a user story in English

Turning the user story into a functional test

Finally, implementing the feature (including appropriate unit tests)

Embedding IronPython

I've put another IronPython article up. It is a simple example of embedding the IronPython engine into a dotnet application.

Embedding IronPython In this example (taken from the Tabbed Image Viewer) IronPython is embedded in C# to create a custom executable.

Tonight I'm leaving to attend a Polish Academic Student Conference , to talk about IronPython. Hopefully see you there.

Oh - and if you're interested in the possibility of being an intern at Resolver this summer, stay tuned for an announcement about how to apply soon.

Compiling Dynamic Languages

I haven't heard much buzz about this amongst the Python community :

This is a video presentation from Raj Bandyopadhyay exploring the compiling of dynamic languages, by translating Python into OCaml (which can then be compiled to native code).

From the presentation :

Our thesis is that dynamic languages like Python can be effectively compiled by translation to statically typed functional languages like OCaml. Not only that, but this approach is highly amenable to formal verification. This, in fact, entails developing a formal semantics for the dynamic language. The talk described ongoing efforts to demonstrate this strategy in the concrete case of compiling Python using OCaml. After explaining why statically typed functional languages like OCaml may play an important role in compiling dynamic languages, we describe our progress to date in understanding the semantics of Python and in devising a correct translation into OCaml. At the time of writing this abstract, preliminary performance measurement were quite encouraging.

I have heard the comment that he has only implemented the 'easy parts' of the language, which currently yields about a 2x speed-up. By the time he has implemented some of the harder (read 'more dynamic') parts of the language this speedup may have evaporated.

Deadman Vista

This photo was taken from literally one of the 'high points' of our US road trip.

As the picture says. what's in a name ? This was one of the scenic view points in the Ouachita (pronounced Washita) Mountains.

At some point I may get round to making a gallery of some of the other pictures from our trip...

The only disappointment from the journey was that we didn't see a single gun. What is America, after all, if not the land of freedom and lethal weapons ?

Doctest or Unittest ?

Martijn Faassen has posted an interesting comment on doctests to my entry on Python and testing.

He feels that doctests are generally better than unittests. I think I disagree (but it is a matter of personal taste, and hardly a big deal so long as you are testing).

The reasons he gives are :

Doctests serve as developer documentation

Writing API documentation first also forces you to think clearly about your API design

Doctests are more readable

There is a lot that is good is that. A long while ago (at least in my history as a developer) I wrote a short piece about Document Driven Design. I'm a great fan of documentation, I quite enjoy writing it and I can't stand using projects with missing, incomplete or out of date documentation.

However, I am also a great fan of the test coverage you get when you do TDD : at Resolver this means a three to one ratio of test code to production code. This is incompatible with having doctests in your production code files (unless you're insane...).

Having doctests in a separate file seems to be a bit odd, unless like Matijn you believe they are more readable than unittests. I don't see how this would fit easily with mocking out dependencies to do true unit testing though.

At Resolver we rarely use comments in our code. The principle is that if the code isn't readable (you can't tell what it is doing just by looking at it), then it needs refactoring. We occasionally comment things that are unavoidably complex.

Another good principle is that you should test all the important functionality of your 'units'. In this way, your tests act as a specification for your code. We try to use a naming convention like :

testThisMethodShouldPerformThisFunction

Given just the tests (or even the test names!) you should have a complete specification for the code, so they also serve as documentation of a kind. It is important that your tests are also readable.

The downside is that we have no user documentation for Resolver yet. As Resolver is due to go to our first customers soon, this is a bit of a lack...

More Projects Using ConfigObj

Every now and then I google around to see if ConfigObj is being used somewhere I haven't seen yet. This time I've found several projects using ConfigObj that I haven't seen before :

CryptoBox A very interesting looking Debian based Live-CD which supports storing data using an encrypted harddisk; usable even by non technical users. I've seen this before - but they've gone upmarket with their website.

Py-notify A Python package providing tools for implementing Observer programming pattern. These tools include signals, conditions and variables.

Coherence Coherence is a framework written in Python enabling your application to participate in digital living networks, at the moment primarily the UPnP universe.

Twisted-Goodies Twisted-Goodies is a package of miscellaneous add-ons and improvements to the Twisted asynchronous processing framework.

Jabbim A Jabber/XMPP client, written with Python and QT 4.2.

FluidVoice Fluid Voice is a way to locally connect a set of terminals where at least one of them has a (satellite or cellular) phone connection to the rest of the world. Any of the terminals on the network can access the telco connections seamlessly.

Vista Simulator

Want to try Vista, but don't want it to take over your machine ? Try the following Vista simulator advice, from BOFH :

"... I run a Vista simulator."

"Virtual Server?" the Boss asks.

"Nah, I just turned on all the flashy crap in XP, changed the background image, took some memory out of my box and clocked down the CPU. Then broke Media player. Works like a charm."

"So you don't like it?"

"No. But it has does have one advantage."

"What's that?"

"It causes a clean reinstall of XP which is generally good from a defrag point of view."

Python Community, Rails Community, Beautiful Code and the Testing Culture

PyCon was great: I simply don't have enough superlatives to express how much I enjoyed it. It was also the largest PyCon yet with almost six hundred attendees.

Not wanting to put a dampener on this achievement, but to put it into perspective, this year's Rails conference has twelve hundred attendees.

This post is a bit of a ramble about the Python community, the Rails community and testing. It is of course inspired by (or perhaps more properly the fault of) my colleague Andrzej. He is one of a select breed (but by no means the only one): he is a member of both the Python community and the Ruby community.

He discovered Ruby (which I conflate in this post with Rails; without apology or explanation) a few years ago as he was moving towards more agile development techniques. Needless to say he fell in love with it and has done commercial and private web development with it ever since. He also started programming with Python almost a year ago when he took a job with Resolver. (So his practical experience of Python is basically with IronPython.)

Andrzej isn't a language zealot, he can appreciate the strengths and weaknesses of both languages - and let's face it, they have more in common than they have differences.

Note I attended a BoF (well, most of it, do I use too many parentheses ?) on Python advocacy. I think the wild success of the deliberate and aggressive Ruby marketing has awakened the Python community to the need to get our message heard. Andrzej attributes the success of the Ruby marketing to the fact that they picked a very deliberate target audience: Java programmers. Their message was clear, programming with Ruby will save you from your Java nightmare. This was negative, but struck a chord and was very successful (and equally applies to Python of course). Unfortunately we can't just pick the same strategy, by the time you see a bandwagon it's too late.

Andrzej hasn't done web development with Python, although he does read with interest about Django and the other web development frameworks. At PyCon we attended the talk on WSGI, because it has no Ruby equivalent and I've obviously failed to explain it, and a talk on SQLAlchemy. It's fair to say that SQLAlchemy and related Python projects are at least in part inspired by the success and expressiveness of ActiveRecord for Rails.

One of the features provided by Rails is 'migrations'. I don't understand all the details, but it is basically a tool that allows you to dynamically refactor your database schema and related code. Andrzej says that having used it he couldn't imagine undertaking a large project without this. At the SQLAlchemy talk he asked if migrations were available. The response was along the lines of 'no, but some of the migrations features are available as a plugin'. The reason they're not in the core is because apparently they're dangerous, meaning that some programmers who didn't have backups lost their data...

Andrzej wonders if that reflects, at least in part, the difference between the Python and the Ruby philosophies ?

He also hasn't yet seen a compelling reason to shift his web development from Ruby to Python, and perhaps has a compelling reason not to.

That aside, despite appreciating both languages, Andrzej feels that he learns more from the Ruby community. I mentioned earlier that Andrzej isn't a language zealot. He is a zealot for agile development techniques. What he appreciates about both Ruby and Python is that they are languages that assist and encourage in the production of beautiful and elegant code. He cares about the beauty of his code, ugly code offends him.

This is something that I started to appreciate with Python, and why I fell in love with it. It is also what makes the art of software creation so beguiling. Not only can we solve problems, but we can do so in ways that are elegant. Beautiful code is easier to understand, is a clearer expression of the programmers intention (a concise encapsulation of concepts), and is therefore more reliable and easier to maintain.

This is why domain specific languages are important. They emphasise that code should be the clearest expression of meaning possible, and this is often exemplified in the declarative style. You write code that says what you mean, and then provide a framework which interprets this in ways the computer can understand. See this blog entry, about a PyCon talk, for an example of writing a DSL with Python.

For Andrzej, and also now for me since joining Resolver a year ago, agile practises like test driven development are an important part of creating beautiful code. This is what Andrzej means when he says that he learns more from the Ruby community than from the Python community. Part of this probably stems from his longer involvement in the Ruby community, but he feels that he learns more about effective testing (and explorations of other subjects about methodologies that are not specific to any individual programming language) when reading Ruby blogs than Python blogs.

I have to say that in the Python blogosphere it does seem like there is not a huge emphasis on testing and test driven development, and the nuances of when to test and when to mock. That isn't to say there is no-one talking about it, just not a focussed exploration of these issues. There are exceptions (and feel free to disagree with this point).

Grig Gheorghiu is a notable and important exception to this. (Andrzej and I were fortunate to meet Grig at PyCon, although only briefly, and I regularly point Andrzej to interesting posts on Grig's blog.) Not only does he blog regularly about testing, but he has also just started the Testing in Python mailing list. Hopefully this will become a focal point for the Python community in discussing and refining testing tools and practises.

At this point let me explain how I think testing is relevant to beautiful code, more specifically how I think test driven development is relevant to beautiful code.

The basic principle of TDD is that you write the test for your features / units before writing the code. You are driven by your failing tests ! This simple fact changes the way you code. It forces you to think about your ideal API before implementing. It also forces you to implement your code in modular, decoupled and testable units.

The fact that you get incomparable test coverage is almost a side issue.

Good test coverage is invaluable when refactoring. You can make changes in one part of your code, and be certain that you have completed your refactoring when the rest of your test suite all passes...

This does mean that you will throw some tests away when you refactor. It doesn't mean that you write tests first for experimental code that you know you are going to throw away or change drastically. Exploring areas of a problem that you are uncertain about is spiking. Rough out the code, solve the problem, then write the tests for a cleaner implementation.

Test driven development is more work than not testing, testing after implementation, or only testing for bug regressions. Many developers just don't see the benefits, and never try it. I'd certainly never tried it until I started with Resolver, and I think that it is diffusely to appreciate the benefits without giving it a go. At Resolver we have 20 000 lines of production code and 70 000 lines of test code; I'm afraid I'm a fully paid up convert now. I couldn't imagine developing without good test coverage, and for any serious development work TDD forms an important part of my thinking process.

Andrzej feels like he finds more discussion of these issues (tests as specification being one area that I've read interesting posts on from the Ruby world) in the Ruby blogosphere than the Python blogosphere, particularly using tools like RSpec. I talked briefly to Ian Bicking about this at PyCon (he also has interesting things to say about testing). Ironically (notice I don't say incorrectly...) he feels that testing is something that the Python community get much more than the Ruby community...

Possibly part of the issue is not that testing isn't discussed much in the Python community, but that the Python community is so diverse. Ruby (at least in its current popularity) has a young community, and as such they are very focussed. Perhaps they have more synergy on the kind of agile topics that Andrzej is interested in. I think the question that Andrzej is interested in is not just on the broad subject of testing, but how testing fits in as part of agile development techniques that make your code more solid, and importantly more beautiful.

The diversity of the various Python communities is of course the great strength of Python. Python is used for web development, writing desktop applications, scientific computing, games writing, scientific computing, on mobile devices, within industry, system administration and much more.

Perhaps testing is thought about a great deal within the Python community as a whole, (the number of testing talks at PyCon would indicate that this is true) but maybe the 'testing community' within Python is still young. I hope that the new mailing list is a sign that it is maturing, and I also hope that the Resolver developers are able to play some part in it.

Archives