Python Programming, news on the Voidspace Python Projects and all things techie.

Global Python Sprint Weekend in London

The next Python bug days are going to be a bit different. Instead of just a bug day we're going for 'Sprint Weekends'.

May 10th-11th (four days after 2.6a3 and 3.0a5 are released)

June 21st-22nd (~week before 2.6b2 and 3.0b2 are released)

The goal is to have the usual coordination via IRC (on #python-dev at irc.freenode.net), but also (hopefully) Python user groups can meet across the globe to sprint collaboratively. This is a great way for people to have fun hacking together and learning the non-esoteric art of contributing to core Python development. User groups who are interested in this can register by responding to this thread on the Python-Dev mailing list (that post also has all the details).

This is being coordinated by Trent Nelson, who suggests that Saturday be for groups to meet up in person, with Sunday geared more towards an online collaboration day via IRC, where we can take care of all the little things that got in our way of coding on Saturday - like finalising/preparing/reviewing patches, updating tracker and documentation, writing tests (you do test first, right Trent?).

Trent, Simon Brunning and I are organising a London Python sprint. There is also a meetup a few days before on May 6th. Venue of the sprint weekend (or the Saturday at any rate) still to be decided, but we'll keep you up to date.

UK MSDN Flash: Getting Started with IronPython

The Microsoft UK developer newsletter MSDN Flash, has published an article of mine on Getting Started with IronPython. This is one of their first experiments in publishing articles by non-Microsoft employees!

Here's an extract:

There is increasing developer interest in dynamic languages, and Python is a particularly good one. It is widely used, semantically concise and easy to learn. As a dynamic language, types are enforced at runtime rather than compile time. Developers used to statically typed languages may find this removal of the "safety net" worrying, but in practise it turns out not to be a problem, particularly if you are using best practises for your testing cycles. There is a performance cost for these runtime checks, but dynamic typing makes it easy to do things that are either cumbersome or not even possible with statically typed languages. Benefits include first class (and higher order) functions and types, late binding, easy introspection, duck-typing and metaprogramming. For more on dynamic typing, read Strong Typing versus Strong Testing. The most important thing about IronPython is how well it is integrated with the .NET framework, and how easy it is to work with managed objects. This makes mixing C# and IronPython very straightforward. Beyond writing full applications, other practical uses for IronPython include embedding it into applications to provide a scripting environment, using it as a standalone language for system administration and utilising it as a tool for exploring assemblies. This last use case is best done with the interactive interpreter, which is also a great place to start experimenting.

For the rest you'll have to read the full article.

Report from the RuPy Conference

This was the second year for the RuPy Ruby and Python conference in Poznan, Poland. Last year there were 80 attendees and this year the conference had grown to 250. The conference is not only a great concept (great to see Ruby and Python grouped together rather than pitted against each other), but there were a high number of quality talks.

My talk was on IronPython and Silverlight 2 and it went great. I felt confident with the material, the audience laughed in all the right places and all my demos worked! (There were 80~100 in the audience - I'm rubbish at guessing numbers.) I think the talk was improved by showing two good looking Silverlight applications from the gallery. These were Tetrislight and the Image Snipper, both of which do a much better job of showing what Silverlight is capable of than my 'developer oriented' examples! (But you'll learn more from mine of course.)

As usual with a Python crowd the biggest concern was the Linux compatibility story (I believe that Moonlight will prove to be a very good 'Silverlight 2 on Linux', but time will tell). As for the adoption story, Silverlight 1 is currently being downloaded at a rate of about 1.5 million per day and is already being offered as an 'optional update' via Windows update. I'm sure that Microsoft will do the same with Silverlight 2, meaning that not long after the final release it will probably find its way onto ~90% of the world's desktop computers! Microsoft have a deal to put Silverlight 2 on the official Olympics website this summer. This gives them both a hard deadline for final release and a great driver for Silverlight 2 adoption.

The infamous Zed Shaw was there and gave a very tame talk on how to get statistically meaningful results when testing web app performance. He is a great speaker, and he did have a dig at Rails with an utterly unfair performance comparison between Rails and web.py as his example. Naturally web.py , which does a lot less, could handle ten times as many requests per second.

Interestingly, he used R to turn the raw data into meaningful results and plots. R is an odd looking language, but it did look like a good fit for the task. Of course many Pythonistas would prefer him to use matplotlib...

I missed Jay Field's talk on Domain Specific Languages. Jay is renowned in the Ruby community, both for his work with DSLs and his focus on testing. The reason I missed the talk was because I was talking to Zed (he is loud - but he isn't an asshole). As he currently works at a large financial institution I thought I would show him Resolver One. However, as that organisation is Bear Stearns I doubt much will come of it.

I did ask Zed if Python is a Ghetto. He said no.

I spent quite a bit of time talking to Carl Friedrich Bolz , who is a core PyPy developer and one of the writers on the PyPy Status Blog. One of the things that Carl has done is write a framework so that PyPy Garbage Collectors can be written in Python - plus he wrote the first two GC implementations. Armin has just written a new one that is more efficient - as it can avoid copying large objects - by subclassing one of the existing ones.

Carl did an excellent talk on the current state of PyPy and particularly the recent work he has been doing on the Just In Time compiler. The PyPy JIT is a tracing JIT (like Tamarin), and as well as speeding up operations on the basic types it will be able to optimise method calls (etc) which are one of the bugbears of Python performance. Techniques it uses include not generating intermediate objects that will always be thrown away and even keeping frame objects in registers where possible (without any semantic changes to Python). Very clever stuff.

For those few who still don't understand the purpose of PyPy, it is much more than just 'Python implemented in Python'. PyPy is a language agnostic compiler toolchain. Typically it is used for compiling interpreters that are written in RPython, a static subset of Python. Interpreters are analysed and compiled by PyPy, which can emit C, Java or .NET bytecode (and various other actual and possible backends).

It also compiles garbage collection and a Just in Time compiler for the interpreter (although the GC is not needed for the .NET and JVM backends). Other interesting things can be done, like experimenting with new GC and object space implementations. So PyPy is a compiler toolchain - with swappable front, middle and backends! As well as this, there is a highly compliant Python interpreter written in RPython. When translated to C it currently runs at around half the speed of CPython, but it is already faster for highly 'memory bound' operations (because the GC is better).

As the JIT makes real progress (it is the subject of Carl's thesis - so real progress is being made), some interesting things become possible. First of all, PyPy is likely to become faster than CPython. I've long joked that so far PyPy's greatest contribution to the Python community is to kill off Psyco and Stackless . It finally looks like it won't be very long before I have to stop saying this.

One fun, and plausible if not very likely, consequence is that it would be entirely possible to write a Ruby interpreter in Python (RPython of course)! That might make a few flame wars moot! Porting Zope to different Pythons is an article that explores some of the more likely consequences.

On the Saturday night was the geek party, which was great fun and I had entirely too much to drink (largely at the behest of Jonathan and Susan, so it was not my fault). Jay joined the reverie and we staggered back from the centre of Poznan at 4.30 in the morning.

Speaking of Jonathan, he gave his talk on Test Driven Development. This is an excellent talk and it gets better every time he delivers it. Yet again he failed to get his Ubuntu laptop connected to the projector though, and his fonts were screwed up. No matter though, the large audience appreciated it and asked a lot of questions. If any of you have objections to my earlier blog entry on the value of test first, you need to hear Jonathan.

On the Sunday morning Konrad woke me up at 9am. I got up, which was a mistake! It did mean I was able to demo Resolver One (during the post-lunch lightning talks, which were new to RuPy this year), but I was fairly befuddled whilst doing it (actually the demo went ok but the setting up was a bit 'confused').

There was a 3 hour 'Rails 101' session, which if I had been less knackered I would have enjoyed. Instead I spent the time talking to Carl. I nearly persuaded him that he ought to be working at Resolver Systems. It is not an entirely far-fetched suggestion as he is something of an Excel/VBA expert and a great fan of what we have done with Resolver One.

All in all it was a great conference, and if you are available next year I highly recommend it.

Report from the Expert Zone Conference

I've just returned home after a week away attending and speaking at the Expert Zone and RuPy conferences. I'm beginning to get sick of hotel rooms (poor me) and it will even be nice to return to work.

Whilst at Expert Zone I met Roy Singham, one of the founders and the chairman of Thoughtworks. We talked about Python and Thoughtworks in London. Roy is concerned that Thoughtworks has recently acquired a reputation as a Ruby shop. In fact Thoughtworks still make about 70% of their income from Java and .NET projects, and whilst they have a very strong Ruby team in the US they also have a strong Python team in the UK (and many developers who like both - which is as it should be).

Roy wants to emphasise that Thoughtworks likes Python and prefer dynamic languages in general - not just Ruby. To this end he has appointed Simon Brunning as the officially unofficial 'Python representative' at Thoughtworks. In this vein he has also agreed to be one of the sponsors for PyCon UK which is great.

Expert Zone is a community developers conference in Sweden, hosted by Cornerstone. The conference covers a broad range of topics, but the attendees (~250 this year) reflect the fact that Microsoft has a strong presence in Sweden.

Many of the talks were in Swedish, but I attended Joe Armstrong's talk on Erlang (for the third time in two months - poor me again!) and Ola Bini talking about JRuby. It was interesting to note how similar Ola's talk on JRuby was to mine on IronPython. His JRuby and Swing demo was virtually identical to my IronPython and Windows Forms demo. The same was true of his example of embedding JRuby in Java and mine on embedding IronPython in C#. Particularly gratifying was to note that his definition of a pure object oriented language fitted Python without exception. (Unlike languages like C# and Java where types, functions and methods may not be first class objects and there are primitives that aren't objects either.)

My talk ("IronPython and Dynamic Languages on .NET" - a cut down version of my ACCU Talk) went ok. Unfortunately the room was changed at the last minute, meaning at least a couple of people who wanted to attend the talk missed it. There were about thirty people present and they were a very different audience to the ACCU crowd. At ACCU the audience was split roughly fifty-fifty between Python and .NET programmers, at Expert Zone only one of my listeners had used Python. Although they seemed to be largely listening intently, they were much quieter than the ACCU rabble. This could be a cultural thing, but is more likely to be the fact that English wasn't their first language. Unfortunately I 'optimised' my talk for an hour when I actually had fifty minutes (my fault). This meant that I had to speak fast (bad) and although I could show embedding IronPython I had to skip showing them Silverlight.

For those who attended, or are just interested, here are some links:

Whilst at Expert Zone I was fortunate to meet some great people - like Julie Lerman (independent writer and expert on the Entity Framework), Christian Weyer (who will also be at DDD Ireland) and Dan North from Thoughtworks.

Thanks to the organisers, particularly Mats Rydin and Niclas Nilsson - organiser and chair of the dynamic languages track respectively.

On Testing: Some Programmers Refuse to Get it

Well I hadn't intended this to be my first blog entry after returning from RuPy and Expert Zone, in fact I should be going to bed. However, this blog entry on whether large test suites prove Python is not suitable for large projects is a subject dear to my heart.

The basic tenet (not advanced by the blog author himself but as a response to an article on Resolver One he wrote), is laughable - but I've been speaking about dynamic languages to .NET developers recently and this belief is not unusual. In fact I have a blog entry on static and dynamic typing waiting in the wings, but meanwhile...

Statically typed languages need type safety in order to be able to compile. The compiler must know the types of objects at all points in your program, and so if the compiler accepts your program you have 'type safety'. This is a very thin layer of safety, and applications in statically typed languages have occasionally been found to have bugs. In fact type errors ('typos') are the very easiest bugs to detect. Tools like PyLint (the most powerful), and PyFlakes (the fastest) will catch many of the errors a compiler will catch and are easy to integrate with Python IDEs.

More importantly, Test Driven Development is much more about a design approach than it is about catching errors. Almost none of our tests are there to catch the kind of errors a compiler would catch - the app. simply won't run with those errors in place so it is not necessary to test for them - our tests are there to test application logic and functionality. A good sound principle is: if it isn't tested it's broken. My favourite quote on the subject comes from the Ruby community:

If you write applications without tests then you are a bad person, incapable of love. —Wilson Bilkovich (The Rails Way)

There are lots of people who write large Python applications without automated test suites. I think they are crazy. There are lots of people who write large applications in statically typed languages without automated test suites. I think they're crazier - but for two different reasons...

In fact the best refutation is to tell you the story of how Resolver One came to be written in Python.

The first two developers started writing Resolver One around November 2005. They were a Java developer and a C++ developer and for various reasons had chosen the .NET framework for Resolver One. They assumed they would use C#, as that is what you do when writing a desktop application with .NET, but Resolver One needed an interpreted language embedded in it. They started to evaluate scripting language 'engines' that were available at the time, which didn't give them a great deal of choice, and started to experiment with IronPython which was at version 0.7.

Not only was the .NET integration very good, but something else became apparent and influenced them to see how far they could get actually writing Resolver One in IronPython. They had already decided to use Test Driven Development as their basic development methodology. They were convinced that this was the best way to ensure code quality, and having practised it now for two years at Resolver Systems I'm convinced that they are right. Dynamic languages are substantially easier to test with. Mocking objects is trivially easy and you can monkey patch at runtime for testing purposes being two of the major reasons for this.

So the decision to use Python came well after (almost as a consequence of...) the decision to practise TDD.

I very much enjoyed the recent UK ALT.NET conference, where a bunch of .NET developers got together to discuss topics like agile development and testing strategies. Most of them develop in C#, and although it was great to spend time with them it was very obvious that a lot of the problems they encountered I simply didn't have as a Python programmer. Testing statically typed languages is harder, and ironically to maintain the same test coverage if Resolver One was written in C# we would need a lot more test code.

Test Driven Development produces better code, and this alone is reason to do it (my colleague Jonathan Hartley does an excellent talk on the subject by the way), but it has many other benefits as well:

It forces you to think about feature design and API design before writing code

A large test suite means that you can refactor mercilessly without fear - the tests will tell you when you have got everything working again

Your tests act as a specification and as documentation on how code is intended to be used

They help you to resist 'developer gold plating' by focussing your mind on features you actually need (rather than might need or would like)

Reducing the need for a separate test phase and increasing the feedback between writing code and learning about bugs you have introduced

It breaks down large tasks into smaller and more easily understood steps

A comprehensive test suite that individually exercises units of your application is an invaluable debugging tool

Anyway, some programmers don't get it and don't want to get it. To my mind testing is much more important than which language you use. Some people just prefer statically typed languages, and that's fine. But some people don't like testing, and if you have any concern for code quality that isn't fine.

Archives