Python Programming, news on the Voidspace Python Projects and all things techie.

Resolver One Beta 7 and Pricing Plans

We've just released a new beta of Resolver One. This is the last beta before our 1.0 release on January 16th.

The changes in this version are:

Added Find option to the toolbar, with appropriate F3 etc accelerator keys.

Added support for remaining font options (italic, underline, etc).

Resolver One checks for updated versions when it starts up.

Most formatting is now included when exporting to Microsoft Excel format.

Fixed defect: Bloomberg not updating with real-time data

Fixed defect: Numbers input with a leading currency symbol are set to 2 decimal places with strip zeros set, while numbers formatted through the menu option to a currency format have strip zeros cleared.

We've also announced the pricing details for Resolver One:

The Non-commercial version of Resolver One will be free for personal and open source use

The commercial version will cost $199, but for a short time will be available for $99

We also have licensing options for 'Resolver One Financial Edition', including connectivity with Bloomberg and Thomson market data feeds, guaranteed level of support etc.

Perhaps more importantly though, Resolver Systems has a Twitter Account.

Changing the Default Browser with IronPython (and an aside about the registry)

Aargh... I've just spent the morning on one of those tasks that should be easy, but somehow never is. This blog entry is partly an expression of my frustration, partly an example (because it mostly works) and also because I hope that someone, somewhere knows what the problem is...

We're trying to functionally test part of an 'aesthetic clean-up' of Resolver. Part of this includes adding links to the local documentation and also the online support forum to the help menu. Launching the default system browser with a URL is easy:

>> > from System . Diagnostics import Process

>> > Process . Start ( 'http://www.voidspace.org.uk/' )

But how do you functionally test this? We thought an easy solution would be to temporarily switch the system default browser to a little application that writes the arguments it is called with to a file. Of course, changing the system default browser requires working with the registry.

By the way, before you start on a 'the registry is an evil pile of crud what on earth were they thinking' trip, Raymond Chen has posted an interesting blog entry on Why are INI files deprecated in favor of the registry?. Whatever you think of the implementation (obscure, opaque binary blob that can only be accessed with obscure and awkward tools - bah), the reasoning behind it has some merit. Notably:

Multiple writers to an INI file can result in data loss. If two programs try to update a config file simultaneously then one will overwrite the other. You can't specify granular permissions on parts of a config file...

Anyway, that aside, working with the registry invariably sucks. We worked out that the correct registry entry to modify is HKEY_CLASSES_ROOT\http\shell\open\command (set the default key to the path of the application to call) along with HKEY_CLASSES_ROOT\http\shell\open\ddeexec\Application (set the default key to the application name without the .exe ).

The IronPython code that does this is:

import os

from Microsoft . Win32 import Registry

from System . Diagnostics import Process

from System . IO import Path

from System . Windows . Forms import Application



launcherPath = Path . Combine ( os . path . dirname ( Application . ExecutablePath ) , "..\\bin\\TestEcho.exe" )



regNodeCommand = "HKEY_CLASSES_ROOT\\http\\shell\\open\\command"

regNodeApplication = "HKEY_CLASSES_ROOT\\http\\shell\\open\\ddeexec\\Application"

keyName = ""



oldDefaultCommand = Registry . GetValue ( regNodeCommand , keyName , None )

oldDefaultApplication = Registry . GetValue ( regNodeApplication , keyName , None )

try :

Registry . SetValue ( regNodeCommand , keyName , launcherPath + ' "%1"' )

Registry . SetValue ( regNodeApplication , keyName , 'TestEcho' )







finally :

Registry . SetValue ( regNodeCommand , keyName , oldDefaultCommand )

Registry . SetValue ( regNodeApplication , keyName , oldDefaultApplication )

One minor issue is that if you call Process.Start('http://www.voidspace.org.uk') (no trailing slash), then the default browser gets called with http://www.voidspace.org.uk/ (trailing slash added)...

This works - sort of. First of all, changing these registry entries also causes HKCU\Software\Clients\StartMenuInternet to be set to our executable . This sets the 'internet' button on the start menu to our dummy application. Not a problem, as I don't know anyone who actually uses this button.

Worse, although it works it also bombs out with an exception: WindowsError: The system cannot find the file specified . Searching the google we discovered that lots of other people have had the same problem, with no clear solution. Possible causes we read about include:

It is a problem that only happens on a few machines (haven't tried it on another machine yet)

It is something to do with dde, a horrible obsolete technology - but it would explain why we need to set the ddeexec key (and obviously our little test app knows nothing of dde)

key (and obviously our little test app knows nothing of dde) We haven't set all the required registry entries (found one piece of example code that changes about 15 keys!)

Reports that this error should only happen with .NET 1.1 (?)

It could be related to a browser (our application) that doesn't have an associated profile

We will do some more exploring tomorrow. Trapping which file it is looking for will probably give us a clue, but we could just trap the error and move on...

Chasing Memory Leaks in Python Applications

When you program in a high-level language like Python (assuming you aren't hacking on the core or writing C extensions), you tend to cheerfully assume that hunting down memory leaks is a problem for other people. Unfortunately that's wishful thinking, leave stray references around or caches uncleared and your application will leak memory.

We've recently spent quite a while tracking down a serious memory leak in Resolver. When closing documents, no memory was being freed. For small documents that wasn't a problem (which was why we didn't notice it), but for large documents it really is a problem - and as we added larger datasets to our test suite it died with out of memory errors.

We managed to verify pretty quickly that on closing a document we were closing the window correctly and removing all references to it, but the main document itself was being kept alive. A quick fix was to 'manually' clear the data associated with the document. This freed most of the memory but didn't deal with the root problem. We actually had to resort to stack and heap dump analysis, following references back from the document to see what was keeping the document alive. Kamil Dworakowski, explains the issue in his blog entry Anti-pattern: static subject to observer mapping. (He has also promised a blog entry about doing heap dump analysis to follow references with IronPython.)

The real problem is the way that IronPython 'does' event handling (so it is a problem specific to IronPython). When you add an event handler (function or method) to an event, IronPython wraps this with a delegate (which is the type of object that .NET requires its event handlers to be). It then stores a mapping of events to handlers, so that if you remove the event handler again it can compare the Python functions in its mapping and remove the right delegate.

These event handlers for GUI events are often methods on our controller classes. The controllers mediate between the view and the data model, so they are document observers and hold a reference to the document. The IronPython internal mapping of event handlers was keeping the controller classes alive, which in turn was keeping the documents alive. Unfortunately IronPython can't just use a weak reference to the event handler functions. If you add a function as an event handler (lambda or otherwise), then you may not be keeping an explicit reference to it yourself and a weak reference would cause it to be garbage collected - so it is a difficult problem to solve.

The only answer on our side is to implement proper 'Dispose' methods on our view and controller classes (and call these explicitly on document close), in which we unhook all the events we have registered for. Finding all of these was great fun for Kamil. He would analyse the stack dump, following references around to see which events were keeping the the document alive, and unhook them. Then do a new dump, see which events are now keeping the document alive - repeat until no more memory leak.

A Year with Python, IronPython and Ruby

Well I guess I'm a bit late to the retrospective party, but I've just seen that the TIOBE Programming Community Index has just declared Python the "Programming language of 2007". Over the last year Python has risen 2% and two places by this metric, which attempts to gauge programming language popularity using search engines. Although you may question their technique, it is probably useful for tracking trends for individual languages rather better than it ranks languages against each other. Having said that, Python now stands at position six having overtaken both C# and Perl.

About this time last year, the two rising stars in TIOBE were Ruby and D (which styles itself as a better C++) both were storming up the charts and seemed unstoppable. My esteemed colleague, Andrzej Krzywda, felt that Ruby was still in the early phases of adoption and would continue to grow. I felt that the meteoric growth of Ruby was a rush of early adopters, and that it would stabilise.

According to TIOBE, Ruby has actually dropped 0.17% (and one position) since the peak of early 2006. D has has also stabilised and the new rising stars are Lua (used as a scripting engine for World of Warcraft amongst other things) and Logo! I have seen recent articles resuscitating Logo as 'actually a fairly capable implementation of Lisp', so perhaps this is plausible.

Although TIOBE seems to favour my interpretation, there are other (and probably better) metrics. Book sales is probably a good metric, and at the height of the Ruby growth in December 2005 Ruby Surpassed Python in Book Sales . In his end of year blog entry A Year in O'Reilly Books (2007) Tim O'Reilly makes some interesting comments but doesn't provide much hard data:

C# continues to gain significantly on Java in terms of book sales. What's notably missing from the bestseller lists: books on programming languages (besides Javascript). The top programming language books in last week's bookscan report were Learning Python, followed closely by the just-released Head First C#. Books on Java, Perl, PHP, and yes, even Ruby, are well down the list.

The Indeed.com Job Trends shows a relatively flat trend for both Ruby and Python over the last two years.

What I think is happening is that dynamic languages are continuing to gain in popularity. Both Ruby and IronPython have helped through their high visibility. Microsoft have made a great deal of the IronPython integration in flagship products like Silverlight, Visual Studio 2008 and their new MVC ASP.NET framework. Virtually every .NET developer I speak to these days has heard of it. Whilst few .NET developers will switch to CPython, some will, and the increased visibility can only be beneficial in general. It is also encouraging that this year Jython seems to have made significant progress.

Ruby almost certainly is growing, but more incrementally and building on its current base. The future is bright for dynamic languages in the coming years.

To finish with, here is another interesting metric: Django compared to Ruby on Rails in Google Trends. This also shows Ruby on Rails having flattened, whilst Django has risen steadily to the point where they have converged. Whilst the Django term is polluted with Jazz, the trend is an interesting one.

Last and final footnote. One of the improvements to Firedrop2 that Davy Mitchell has just made, is that the Blog Statistics Page now works!

New Mac Pro Announced

Well, if I had bought a Mac Pro in the last few weeks I'd be really annoyed right now. Apple have stolen the thunder from the CES by announcing a new range of Mac Pros.

Yesterday a new Mac Pro with two 3.0GHz quad core Xeon processors and 1Gb of 667 MHz RAM would have cost me Â£2700. Today the basic spec of a Mac Pro is two 2.8 GHz quad core processors with 2Gb of 800MHz RAM, and costs Â£1700!

Very nice, almost affordable and likely to be a high performance computer for quite some time.

New Computer Musings and Other Random Hardware Stuff

I've just bought a new mouse, a Logitech LX7. I've had bad experiences with wireless mice and keyboards in the past, but we use Logitech gear at Resolver and they have served us well. Unfortunately the USB receiver with this mouse doesn't work (secondhand from ebay), but it does work with my office receiver - and I really like the mouse. Very high resolution, smooth and a lovely feel. Whether or not I sort this one out I'll be going for the same model again.

I've also just bought a new monitor (yes another one). I'm still only using three monitors, but my main one is now a 24" Acer AL2416WB Widescreen monitor with a resolution of 1920x1200 pixels. This thing is awesome, it makes my 22" widescreen look small. It is wide enough to show two pages of a word document side by side at a decent font size, and it only cost £200. My colleague Kamil also bought one, and although he doesn't exactly say so in his review - he likes it!

I mentioned this to my colleague Jonathan, who is an old-school gamer. He is looking for a big gaming monitor and would prefer a CRT to an LCD. He says the black is blacker and the colours are brighter. This gives a superior 'immersive effect' when gaming with the lights off! For gaming the refresh rate is also important, and at least the last generation of LCDs used to show trails when viewed in the dark with lots of movement on screen. Additionally, CRTs are capable of displaying different resolutions natively, whereas LCD panels have only one 'real' resolution. Big(-ish) quality CRTs must cost more than LCD panels, just because less people want them now - which sounds like buying a valve amplifier.

I will also be replacing my computer at some point in the next year. My current one has reached the ripe old age of eighteen months old, and in the next few months will gracefully retire as a media-centre for the new living room of the house we hopefully move into later this month. I have quite a bit of money burning a hole in my gadget fund, and would like a nice specced computer (I'll tell you about my gadget fund in a later blog post if I have the time, but the summary is that I have to spend it on essential computer gear or pay tax on it).

As always with computers for the geek, I have the choice between building and buying. The last few times I have priced up building I have always found pre-built computers available for the same (or even less) and buying has obvious advantages. This time however, there aren't many manufacturers making computers with the sort of spec I would like - twin quad core processors with as fast memory as possible.

My boss has given me the sage advice that the sweet-spot for processors is at about the £200 mark. In fact this seems to hold. Looking at average UK processor prices (for the Intel Core 2 or Xeon quad-core range which seem to be the best at the moment):

Q6600 2.4GHz Quad Core 8MB Cache - £160

Q6700 2.67 GHz Quad Core 8MB Cache - £340

QX6850 3.0 GHz Quad Core 8MB Cache - 600

So there is a big jump in price from 2.4 to 2.67 and from 2.67 to 3.0. Even if it's irrational, it seems wrong to spend a lot of money and not get a system with a faster processor than my current 2.4GHz AMD dual core.

Important factors in overall computer speed seem to be (in no particular order):

Processor clock speed

Total number of cores

Amount of memory (the cheapest way to improve performance is to add memory - page faults are very slow!)

Processor cache size

Front Side Bus (FSB) speed

Memory speed

Speed of the hard drive that the OS and virtual memory live on makes a lot of difference (15000 rpm drives are available relatively easily now)

Possibly, whether you are running a 64bit OS or a 32bit OS

For gaming (which I do a bit of - Team Fortress 2 rocks), and some applications, the graphics card matters a lot too. The latest Edge magazine review for Crysis states that "to get visually the best from this game you need two of the latest graphics card strapped together" - which of course is ridiculous. It does look good though.

This is all fine, but as soon as you move to a twin-processor and quad core spec, you are seriously reducing the number of motherboards available to choose from.

There are also a few questions that I don't know the answer to.

Memory type - is there a massive speed advantage to be had by using DDR3 or dual channel RAM (and are there any retail motherboards that support them for twin Socket LGA771 processors)

Is there any advantage to be gained using 'ECC' ram (with an extra byte on the bus for checksum), and the same question as above about the motherboards

800Mhz DDR2 RAM seems to be the fastest that I can find. In which case does it matter whether you have an FSB speed of 1066MHZ or 1333MHZ if your data can only ever clock 800MHZ? (In fact what is the difference between FSB speed on the motherboard/processor and the speed your RAM can run at - I assume that in practise they will both run at whichever is slowest out of these two numbers)

The only two hardware supplies I have found (but I haven't looked too hard) that will offer computers at this spec are Dell and Apple. Apple offer the Mac Pro with two quad-core processors for £2700. This is with a meagre 1gig of RAM, but can be upgraded much more cheaply than buying the RAM from Apple. I don't have this amount of money yet, but am not massively far off (and it would come with a slower hard drive and of course only 667MHz RAM, but does have a reasonable video card).

Buying from Dell gives me much more choice in configuration. For a fair way south of two grand I can get a computer shell with:

A motherboard that will take two quad-core Xeon processors

One 3.0GHZ Xeon Quad-core with 8MB L2 cache

One GB of DDR 2 at 667MHZ, with 4GB from Crucial for about £180

A 73gig 15000rpm Hard drive for the OS

A 160gig second drive which is a little meagre but comes included anyway

This would give me a pretty good system that I can build on. I'd need to choose a gfx card of course, which is another minefield...

As to the question of whether running eight cores will give me much improvement, it will be interesting to see. Given that most of the time I have many applications open at the same time, it ought to help.

Whilst we're on about processor cores, I've found something to do with some of my current spare cycles. I often leave my computer on overnight, stress testing my network connection of course, and the World Community Grid seems like an excellent use to put those cycles to. Unlike similar grid computing projects, they have a whole range of different tasks you can donate your computer time to - like climate change prediction and finding dengue and malaria drugs.

This is the first blog entry using the new tagging feature of Firedrop2 implemented by Davy Mitchell. There are now tags (as well as categories) at the bottom of each entry, as well as in the RSS. Currently only in SVN, but I'll do a new release soon. This is something I've wanted for ages.

More on the Eee

Some updates on the Eee.

It turns out I was wrong about the eee wireless not remembering pass-phrases, I was just using the wrong application. 'Wireless Networks' just shows you available Wifi networks and doesn't remember keys/pass-phrases. The 'Network' application does remember them.

Also, not only is a Huawei E220 USB modem driver available, it comes with the OS! I was able to plug the modem into USB and setup a connection with the network wizard! This increases the utility of the Eee tenfold.

Apparently the alternative 'advanced mode' for the OS isn't available yet (UPDATE there is a communtiy distribution though - see this wiki). You can however customize the 'icewm' window manager as much as you like. By following the instructions on this forum thread I've got a start button and additional workspaces (desktops) so far, with plenty more customizations available.

Oh, and it looks like the version of Mono that apt-get installs is 'pre-IronPython'. I'll be looking at installing a more recent version and reporting back...

(For initial impressions, read Shiny New Asus Eee PC.)

Archives