Python Programming, news on the Voidspace Python Projects and all things techie.

Falling in Love with Sphinx (and redeeming doctest)

Sphinx 0.5 (half a Sphinx) has just been released. This is timely as one of the most valuable things that I learned at PyWorks (in the hallway track) was how to use Sphinx. Chris Perkins (who has some patches that will be integrated into the next release) showed me how to get started. My conclusion - Sphinx is frickin awesome!

Sphinx is the documentation system used to build the Python documentation. It takes reStructured Text source documents and can build HTML / Latex output, with reST directives for classes / functions / methods, cross-referencing between pages, auto-generation of indexes, online and offline search (very clever) for the HTML, syntax highlighting and more.

I think we can get a lot of benefit from Sphinx at Resolver Systems. We already use a homegrown tool (originally written by me one Christmas) that imports our spreadsheet object model from IronPython and builds reST documents from docstrings, which get turned into our API Documentation. Although Sphinx can do docstring extraction we'll probably still need to use our ApiDoc tool - both because we need it to work with IronPython code (that imports .NET classes) and because it gives us complete control over which methods and properties are documented as part of our public API (plus I don't think Sphinx extracts docstrings from properties).

Syntax highlighting, search and indexing are all things we'd like for our API docs though, we just need to change our tool to generate reST suitable for Sphinx. In order to get a feel for it, I've ported the Mock docs over in preparation for the forthcoming 0.5.0 release (still a few things to do before it's ready):

Getting started with Sphinx was made even easier by using virtualenv for creating clean development environments:

easy_install virtualenv virtual_env some_name cd some_name source bin/activate easy_install sphinx

By activating the new virtualenv environment, sphinx is then just installed for that environment (you will also need docutils installed which isn't picked up as a dependency by easy_install).

Note You can make some really cool workflows by also using virtualenvwrapper by Doug Hellmann. It provides useful commands like workon for automatically activating virtualenv environments.

You can then get started with the sphinx-quickstart command that will create source and build directories for your docs along with a configuration file based on your answers to its questions. After that you just need to learn the Sphinx specific directives! The source reST files for the Python docs are a good place to start for examples (or you can browse the sources of the mock docs).

Building the HTML output is amazingly easy: sphinx-build source output

Once I'd installed MacTex (nearly 1.2 GIGS to download - for goodness sake) then building the PDF was also trivially easy:

sphinx-build -b latex source latex-dir cd latex-dir make all-pdf

The -b switch to the sphinx-build tells Sphinx to use a builder other than the default HTML builder. As well as generating Latex output there is another really useful builder - the doctest builder.

A few weeks ago I wrote an ode to doctest: Doctest How I Loathe Thee. In my opinion doctest is a terrible unit testing tool, it does however excel at testing examples in documentation. Sphinx makes it brain-dead simple to do. Any literal blocks you include in your docs Sphinx will try to infer if they contain Python code, and if they do it provides syntax highlighting with Pygments (fully configurable of course). You can mark a block of code as a doctest instead with the following directive:

.. doctest:: >>> items = [1, 2, 3] >>> mock = Mock(magics='getitem contains', items=items) >>> 2 in mock True >>> 4 in mock False >>> mock[-1] 3

Not only will this be syntax highlighted, but you can run all of the doctests in your documentation with: sphinx-build -b doctest source output

Mock has been developed using Test Driven Development, so it has pretty full tests. Nonetheless, making sure that all the examples in the documentation are up to date and still work can be a major pain. Now it is trivially easy. I have a renewed love and respect for doctest.

Of course it would be nice to have Sphinx running under IronPython. That means that the components of Sphinx needs to run on IronPython. Jinja 2 has no C extension dependencies, so after easy installing it (and copying the package to somewhere importable for IronPython since it doesn't have fecking zlib or zipimport and so doesn't support easy install...) the simple example works fine:

C : \ compile > "C:\Program Files (x86)\IronPython 2.0\ipy.exe"

IronPython 2.0 ( 2.0 .0 .0 ) on . NET 2.0 .50727 .3053

Type "help" , "copyright" , "credits" or "license" for more information .

>> > from jinja2 import Template

>> > template = Template ( 'Hello {{ name }}!' )

>> > template . render ( name = 'John Doe' )

'Hello John Doe!'

>> >

What about docutils?

>> > import docutils . core

>> > docutils . core . publish_parts ( 'hello' )

{ 'version' : '0.5' , 'whole' : u ' < document source = "<string>" > \ n < paragraph > \ n

hello \ n ', ' encoding ': ' utf - 8 ' }

>> > docutils . core . publish_parts ( 'hello

=====



odd' )

{ 'version' : '0.5' , 'whole' : u ' < document ids = "hello" names = "hello" source = " < strin

g > " title=" hello " > \ n < title > \ n hello \ n < paragraph > \ n odd \ n ' ,

'encoding' : 'utf-8' }

OK, so that's a very shallow test, but its a good start.

Fun at PyWorks

Last week I returned from a fun time at the PyWorks Conference in Atlanta. PyWorks was a conference organised by the folks behind the Python Magazine (happy birthday by the way!), and was bolted alongside the already established PHP Works conference.

In total there were about 250 people present, the vast majority of whom were there for the PHP content. Whilst I'm interested in programming languages in general, there is very little about PHP that interests me (except perhaps how easy it is to get started with PHP), so I'm afraid to confess I didn't attend any of the PHP talks. There were some fantastic Python talks given, and it is a shame that so few people were there to hear them (perhaps 30-50 total people there primarily for the Python content). This was the first year for PyWorks and it will be interesting to see if they do it again.

In this picture from Saturday evening, Mark Ramm assists Chris Perkins in a visual demonstration of WSGI middleware...

The more refined gentleman to the right is Shane Caraveo from Activestate who gave a talk on GUI Applications with XULRunner and PyXPCOM from Python. For guaranteed fun at any conference invite Mark Ramm and ply him with tequila...

It was great to catch up with the Python folk I know from conferences and the blogosphere (Kevin Dangoor, Ed Leafe, Holger Krekel, Noah Gift, Doug Napoleone, Matthew Wilson, Jesse Noller and others). It was nice to meet new faces too, like Brandon Rhodes who roped a few of us into speaking at the local PyAtlanta User Group. There were about 30 people or so there, which was a great attendance for a regional user group. It was all the more poignant because that evening Noah officially handed the reigns for the PyAtlanta group over to Brandon, as Noah is leaving for a cushy new job in New Zealand. I gave a talk on Mock for testing. As I didn't prepare (well - I had three slides) I did it all with live coding and it didn't go horribly wrong!

Noteworthy talks (just the ones I can remember I'm afraid):

Matthew Wilson on decorators. Decorators are way cool.

Jesse Noller on multiprocessing The API is really nice and it has some cool features. It will be nice when I do some CPython work and get a chance to use it.

Ed showing off some of the new features in the Dabo Framework. Dabo is a framework for writing desktop applications using the wxPython UI toolkit . Ed was showing off two of the new features - a seriously cool looking forms designer and a way of delivering desktop applications across the internet (with features like auto-update built-in). As any desktop application ought to whip a web application in terms of usability and features this is a nice way of getting some of the benefits of web applications for desktop apps.

Holger Krekel on py.test I also spoke to Chris Perkins quite a bit about Nose. The more I see talks on these two unit testing frameworks, the more convinced I am that I like the unittest way of structuring tests (object oriented tests with assert methods). I'm also impressed with features like integrated coverage testing and particularly test auto-discovery. These should be easy to add to unittest (including getting them into the standard library version) and it is something that is on my list.

Noah Gift and Python for System Administration Even better I managed to get the copy of his book Python for Unix and Linux System Administration that he gave away (just beating Jesse to it). I'm really enjoying it and have found the section on IPython the most useful so far.

Outside of the talks I learned about virtualenv, Sphinx and a bit about Paver. Once Paver supports 'sub-pavers' then we may be able to use it to replace MSBuild for some of our build scripts. Currently we call out into a lot of small CPython scripts from MSBuild - so Paver could bring them all together for us.

My talk was on IronPython and Silverlight. All the material is on my IronPython Pages, but I still desperately need to update the Silverlight Pages for the latest release of Silverlight.

Many thanks to those who organised PyWorks, and the people I spent time with who ensured I had a good time!

My First Screencast: Column Level Formulae in Resolver One

I've finally created a screencast. It's 2mins36 seconds and is about using column level formulae (one of the new features of version 1.3) in Resolver One. (Bonus points for working out where the names come from - but not many because it's easy.)

Column level formulas are cool to avoid duplicating formulae, but you can use them to do some interesting things like data aggregation in combination with list comprehensions; which is the subject of this screencast.

It actually took me two days of work to create this screencast! About half a day to create the spreadsheets, half a day to write and practise the script, and then a day of recording the audio and video plus editing. The next one I do will take considerably less time. Surprisingly (to me) recording the audio and video separately, and then editing them to fit together, worked very well. The trick is to know the script well enough that you don't have to read it (and therefore don't sound like you're reading a script).

For the audio I used Audactiy, and for recording the video Camtasia. Camtasia is extremely simple to use, but pretty limited in the editing that you can do.

Needless to say, it isn't 'studio quality', but it was fun to make and hopefully informative. The docs have more information on Column Level Formulae, and Jonathan did a great screencast on the other new features in Resolver One 1.3.

Ian Ozsvald of Showmedo is also doing some screencasts for us. They're a lot better than the one I did! The one on Python Objects in the Spreadsheet Grid is particularly good.

Open Rights Group in the UK

I'm founding member 748 of the Open Rights Group in the UK. Probably my biggest claim to fame to date.

The ORG exists to "protect our bits" (and someone has to):

The Open Rights Group is a grassroots technology organisation which exists to protect civil liberties wherever they are threatened by the poor implementation and regulation of digital technology. We call these rights our "digital rights".

As an organisation they're three years old now, but the UK branch was only founded this year. They're supported by individuals like Neil Gaiman (who recently did a small talk for the ORG in Clerkenwell - which finally prompted me to pull my finger out and join up) and Charlie Stross.

Archives