Python Programming, news on the Voidspace Python Projects and all things techie.

Python is Nearly Safe - or is it ?

I'm sure that by now you've seen the Joel Spolsky blog entry, Language Wars.

He debates the benefits of using 'safe and mature' platforms for projects.

These debates are enormously fun and a total and utter waste of time, because the bottom line is that there are three and a half platforms (C#, Java, PHP, and a half Python) that are all equally likely to make you successful. ... Python get a half because it's on the border, about to cross the line from an "interesting" choice to a "safe" choice.

Of Ruby he says :

Ruby is a beautiful language and I'm sure you can have a lot of fun developing apps it in, and in fact if you want to do something non-mission-critical, I'm sure you'll have a lot of fun, but for Serious Business Stuff you really must recognize that there just isn't a lot of experience in the world building big mission critical web systems in Ruby on Rails, and I'm really not sure that you won't hit scaling problems, or problems interfacing with some old legacy thingamabob, or problems finding programmers who can understand the code, or whatnot. So while Ruby on Rails is the fun answer ... that's not a safe choice for at least another year or six.

Needless to say the Ruby world is up in arms about this, which is especially fun for me because I work with two Ruby enthusiasts and can tease them about it.

Fredrik Lundh comments that just as Python is becoming generally recognised as a mature and safe platform, we're about to break it. Could he have a point ? Possibly. Will people looking for a long term development platform, be scared off by the choice between the Python 2.X series, which is soon to become 'apparently obsolete' , and Python 3, which will essentially be a new and untested platform ?

We shall see I guess.

IronPython, Threads and More GUI Pain

Ahh, the joys of threading.

Up until recently Resolver has been a basically single threaded application. The main exception is the test suite, where we run the application on a separate thread to the tests so that we can interact with it even when it is blocked. We also have the odd asynchronous callback (implemented in .NET with a thread), but nothing too serious.

Today though, this might have all changed. I say might, because the glorious fun that threads inevitably cause mean we haven't been able to check-in yet. My colleagues will be delighted to read this, by the way. They're all away at the moment except for the two interns, so we're busy causing devastation behind their backs.

It all started innocuously enough. A simple user story to display a 'throbber' style activity indicator when a document enters a potentially time consuming process. We set up the 'enter processing' and 'exit processing' events fine, and got the throbber displayed and undisplayed when these were fired. All that was left was to wire these events to the method that does the job.

Unfortunately, if it is being processed on the main application thread then it blocks the GUI from updating and you can't see the darn throbber anyway. sigh

The obvious answer is to kick off the processing on another thread. A mutex around the method should stop any potential risk of it being called simultaneously, and two threads is hardly multi-threaded so it shouldn't be too difficult right ?

Hmmm... The first problem was that the throbber needs to be enabled on entering the method, and then disabled on exiting. The method is now non-blocking, so we can't just wait for it to exit and then disable, it has to be done from within the thread.

You can't access Windows Forms controls (or interact with them anyway) from another thread. Luckily IronPython and .NET between them provide a way round this :

import clr

clr . AddReference ( 'IronPython' )

from IronPython . Runtime . Calls import CallTarget0



control . Invoke ( CallTarget0 ( DisableThrobber ) )

DisableThrobber is a function that does the work. The Invoke method on a control, causes the callable you pass in to be invoked on the thread the control is running on. You have to use CallTarget0 so that IronPython can turn your callable into a delegate.

Fine...

This is where it got fun for us. This process can be kicked off when one of our controls loses the focus. If the user has the focus in this control and then exits the application, then first there is a LostFocus event causing the process to kick off . When this was all synchronous there was no problem, there would be a slight delay before the close dialog appeared, but we didn't realise this was happening. So I guess it was a 'bug' that was already there. As it had no bad, or even visible, side effects; we never noticed it.

Joy of joys, because this process is now run in another thread, the close dialog appears first which blocks the GUI thread. The control.Invoke is blocked until the dialog is left. If you confirm your exit, the invoke is now happening on a control which has been disposed and everything blows up. Hurrah.

This is where we left it and went home.

The fix should be simple, unwire the LostFocus event in the FormClosing event. They are on different controls, managed by different adapters, so this is not quite trivial but still easy. Well, we'll find out tomorrow I guess. The summary is that, yet again, what should have been a really simple job has spiralled. double sigh

One alternative is to manage the throbber gif from another process altogether and just synchronise the positions. Even threads are less evil than this though.

Python and Curly Braces

How long has this been in ?

A Busy Schedule

I've got a lot of stuff on at the moment, particularly with the various Python projects I run.

For those of you who might be waiting for an update on any of these, this is a rough ordering of what I have to do :

Movable Python Write new documentation. I am also intending to create distributions of SciPy (and possibly other common extension modules) especially for Movable Python .

ConfigObj Fix a couple of bugs reported on sourceforge. Add PEP 292 Interpolation Ensure ConfigObj is picklable Allow interpolation from the current section A minor change to allow ConfigObj to work with IronPython

rest2web A new release is just sitting in subversion, waiting for documentation updates. I also need to create a manpage. I may also add a couple of additional features, like a new filemask keyword for the restindex and fix a bug that prevents you using hard tabs for indentation in the templates.

pythonutils With a new version of StandOut just out, and hopefully a new version of ConfigObj by the time I get to this, it will be time to update pythonutils. Should just be a case of putting together a new release.

Alongside this I am in discussion with Manning about a possible IronPython book. It depends on whether they think there is sufficient market for one (and whether or not I am the right person to write one), so it may never happen. Stay tuned !

I also have several more entries in the IronPython tutorial that I have planned, but haven't yet done.

Unfortunately, amidst all this activity, Firedrop2 is being neglected. Firedrop2 is a great blog client, and with a little more work could be even better. Unfortunately it does all I need at the moment, so it is dropping off the bottom of my priority list.

All Our Code is Broken (or Python 3000 is Coming)

Python 3000 is really on its way. Guido's move to google has finally provided the spur (or at least the time) for real development to happen. Python 3k is no longer vapourware !

It even has its own mailing list and branch in the Python Subversion Repository.

You can see from Guido's report on the recent google sprint some of the concrete work that has been happening to make Python 3 a reality.

So what's the big deal ? Well, Python 3 is the opportunity for Guido (and the other Python core developers) to fix what they (and others) see as mistakes in Python by breaking backwards compatibility with the Python 2.X series. That means that in Python 3 terms, your code is probably broken.

If you look through PEP 3100 for everything marked 'to be removed', you might see some of your old friends disappearing or changing.

One of the major changes is that the unicode data-type will die. In Python 3 all strings will be unicode and there will be a new data-type bytes for byte strings. Opening a file in 'text' mode will return a unicode string, and will rely on a new I/O layer for 'guessing' the appropriate encoding of a text file. This will actually make text handling a lot easier if you understand the basics of unicode. You will still have problems if you want to load in an arbitrary file and treat it as text without knowing the encoding, but to be fair you already have this problem even if you're not aware of it. See this article on guessing text encoding for one approach to handling this.

Another controversial change is that print becomes a function rather than a keyword. This change alone is enough to guarantee that all Python text books have to be rewritten and a huge proportion of code is just plain broken. Personally, I think print feels like a keyword and that this is gratuitous breakage, but I'm not in the mood to fight against the wind and there is a lot in Py3k that is very good. Guido sets out his reasoning here.

Part of his reasoning is, programs that use print are much harder to refactor to modify the behaviour of print. For example to add logging. Of course this can be done without any refactoring, by using an approach similar to StandOut.

StandOut modifies the behaviour of print, adding logging and filtering based on message priority, by making sys.stdout and sys.stderr a bit more 'clever'. Of course 'clever' is not always a compliment in Python, and for a large application this kind of magic may not be appropriate. However, it can be done without breaking everything... The advantage of this approach is that sys.stderr is diverted: so exception tracebacks are also logged without having to be explicitly handled (and are logged even if your explicit handling fails).

A long time ago we were promised that there would be tools to help us migrate our code from Python 2.X to Python 3.X. In recent discussions there was scant mention of these tools. Some of my legacy code may fare better if quietly consigned to the annals of history, but I have not been looking forward to converting all the code that I did want to keep current.

Fortunately news of such a tool has surfaced from Jeremy Hylton . It is a remarkably simple approach (at least in theory, lots of detail to sort I imagine). Code is parsed to an AST with a 2.X parser, and then transformed into a Python 3 AST (or CST or whatever) and rewritten. Great ! It even looks like a tokenizer can be used to decorate the AST with the real tokens used in the original source. This means that whitespace, comments, quotes and multiple lines may be able to be preserved in the 're-generated' code. Very cool.

As a kind of last word, there is little you can do to try and keep your code 'Python 3' compatible whilst still writing for the current interpreter. Python 3 doesn't yet exist and the details are still being worked out. Many of the changes will be syntax errors in Python 2 anyway.

StandOut 3.0.0 Released

There is a new release of StandOut, the flexible output object.

This version is a complete rewrite, with a full set of unit tests.

standout is a module that provides a single class: StandOut - the flexible output object. It provides a simple way of adding logging to a program, and an easy way of adding verbosity levels.

By assigning a priority level to each message, it makes it trivially easy to allow your users to choose their own verbosity level. Verbosity levels for normal output and the log file can be different.

Alternatively you can just use StandOut for logging stdout and stderr to a file.

As an added bonus it includes a software unbuffered mode. In this mode all writes to stdout and stderr are flushed immediately.

Quick download links :

The zip file includes full documentation and unit tests.

See StandOut Homepage for the online documentation.

Basic usage, for logging all output on the standard output and standard error streams to a file :

from standout import StandOut

stout = StandOut ( 'log.txt' )





print 'This will be logged to the file'







stout . close ( )

StandOut is currently used in both Movable Python and rest2web.

Code Reuse

Plus a nifty bit of blog title re-use too.

I've discovered a few of my modules are in use in other projects :

ConfigObj Configuration file reader and writer (ini files), with an emphasis on ease of use and flexibility. Flazzle A language flashcard program with both a commercial and an Open Source version. PyKeylogger A key logger ! I hope this has nothing to do with the Python Virus. Dynagen Dynagen is a text front-end to Dynamips, a Cisco 7200 emulator written by Christophe Fillot. It emulates a 7206 hardware platform, and runs standard 7200 IOS images.

scraper.py A simple HTML scraper class, for easy parsing of dirty HTML. Rather than presenting the structure of the HTML as an object, it provides methods to override for processing the HTML as it is being parsed. Cedar Hepforge A development environment for high energy physics software development projects. Hepforge uses scraper.py in the CEDAR project.

akismet.py This modules provides a Python interface to the Akismet comment spam blocking web service. The Trac Spamfilter Plugin has taken this module and adapted it. It still bears more than a passing resemblance to my code, and they were nice enough to credit me.

Commercial software and a keylogger ! Serves me right for the lax license terms.

Archives