Python Programming, news on the Voidspace Python Projects and all things techie.

Converting a Google Code Repository from svn to Hg on Mac OS X

Google code allows projects it hosts to have either a subversion or a mercurial repository. If you have a project there which has been around for a while there is a good chance it is still has a subversion repository. Using mercurial makes it easier for others to collaborate, and google provide instructions on converting a project from svn to hg.

If you are using the Mac OS X system Python then these instructions work fine. If you have installed a more recent version of Python from the Python.org binaries then the section on converting your svn history won't work.

This is because the Python svn bindings, that the hg-convert extension needs and come with the free Mac OS X developer tools are installed into the system Python. The instructions on the google page for building the Python svn bindings from source are out of date. The repository for them has moved and I failed find its new location.

This is fixable though, all you need to do is to run Mercurial with the system Python instead of your default Python. First this means installing Mercurial for the system Python though... On top of this hg-convert relies on being able to shell out to mercurial, so you need to make sure your newly installed mercurial is on the path (this can be done temporarily without messing with your 'normal' mercurial installation).

Here are the set of steps I did to achieve this.

Install mercurial with the system Python ( /usr/bin/python ). I did this by first installing distribute so I could easy_install mercurial: curl -O http://python-distribute.org/distribute_setup.py sudo /usr/bin/python distribute_setup.py sudo /usr/local/bin/easy_install -U mercurial Change the path so that mercurial can find itself: export PATH=/usr/local/bin:{$PATH} Edit the mercurial ~/.hgrc to enable the hgext.convert extension as described on the google code wiki page Now create the hg-client directory and run mercurial (which will now be from /usr/local/bin because of the path change): mkdir hg-client hg convert http://projectname.googlecode.com/svn hg-client Change the google code repo to mercurial in the admin/source settings Now you can push your local converted repo up, including all the history information: cd hg-client hg push https://projectname.googlecode.com/hg

I've done this twice now so I'm pretty sure it works.

A Brief Update

Over the summer I went even-longer-than-normal without blogging. In that time a lot has changed.

The wife and I had a great summer break visiting her family in Romania. This included a trip up the gloriously beautiful Ceahlau mountain. Whilst we were out there we discovered that Delia is pregnant. This was a very pleasant discovery, but as I'm sure you all know babies are the great enemies of open source - so we'll have to see what happens in the future... Fortunately I have Jesse Noller as a great inspiration for enjoying fatherhood whilst still contributing to Python.

(Greg Newman hypothesises that Michael Foord junior will look like this)

Immediately on returning from Romania I started working full time with Canonical as a Python developer on the ISD (Infrastructure Systems Development) team. Our team blog is here. We work mainly with Django, but we also use a whole range of 'standard' Python tools including pip, fabric, virtualenv, and mock. We also have some of our internal libraries open sourced like configglue and Canonical SSO.

I work from home (mostly - my team is joining the Ubuntu Developer Summit in Florida next week for a development sprint) communicating with the team over IRC and occasionally VOIP (mumble), which is how all Canonical development happens. Having done several years commmuting to London, followed by working from home for the last nine months or so, I have the taste for home working and wouldn't work any other way now. My team includes one guy from Poland, one from Germany, a couple of Englishmen, two Argentinians and at least one American. All great people, but it will be nice to actually meet them at the UDS.

My computers are all Macs (and have been for sometime), but I've been working inside a Ubuntu VM. So far I've enjoyed it but am not (yet) tempted to move to Ubuntu as my main OS... The job change also means that for the first time in about five years I'm not working at all with IronPython. I'm still involved (more news on that to come soon), but purely for hobby programming. (As a side note, I wonder how long it will take for me to lose my reputation as a 'windows guy'. Years probably.)

Unsurprisingly the Canonical development workflow is all based around Launchpad and Bazaar. I'm really impressed. In the past I've been annoyed by the web UI of Launchpad, finding source code for a project can be annoying and it doesn't provide documentation hosting. For small projects it is still possibly overkill, but for teams of several people doing development in new branches and using merge reviews Launchpad provides a very nice workflow. As I now have some projects in Mercurial, work with Bazaar and some projects still in subversion I do get confused about which version control system I am actually using at any one time...

configglue, one of our internal projects is interesting as it is a direct 'competitor' for ConfigObj. As it is owned by my team I will almost certainly be working on it as well as with it. configglue provides layered configuration files, a frequent feature request for ConfigObj. In ConfigObj you can do it "manually" via the merge() method, but the configglue way is nicer.

configglue has schema validation (which is its major feature really). It is code based whereas ConfigObj validation is via schema files. The configglue way is more verbose but I quite like it. We use it with django-configglue which allows django project settings to be kept in ini files rather than as code.

You may also be wondering how the unittest2 plugin system I blogged about a while ago is getting on.

I recently posted an update to the Testing in Python mailing list:

After recent conversations with Holger Krekel and Robert Collins I have decided to make a fairly major internal changes to unittest2 plugins. I'm getting rid of all global state, which includes the global hooks that are currently the core of the extension machinery.

Getting rid of global state always sounds goood, but it isn't without API consequences. With no global access to the hooks set it needs to be passed down and 'made available' to the runner, loader, suites and all test cases. This is annoying, but probably still better than global state. Removing the global state makes testing the plugin machinery massively easier however... It also allows multiple different plugin configurations within the same process (which is nice but only actually useful for a small set of use cases).

This refactoring also means changing the way plugins are initialized, merely importing the plugin is no longer enough (the metaclass goes - which makes me sad but is probably also a good thing).

So the refactor is still in progress. This means plugins won't land in Python 3.2, but they will have longer to mature and stabilise in unittest2, which is no bad thing. Meanwhile I'm still working on the final release of Mock 0.7.0. There will be one more beta before final release. I also have a few Python issues for unittest itself that I'm really hoping to work on before the Python 3.2 beta which is rapidly approaching!

unittest2, django and why import * is bad

So, first the good news. unittest2 is now integrated into django trunk and will be included in Django 1.3. This will allow you to use all the new unittest2 goodness like addCleanup , test skipping and the fancy new asserts in your django tests.

Many thanks to Russell Keith-Magee for his hard work on the integration.

unittest2 is bundled in Django 1.3, but if you are using Python 2.7, or have unittest2 installed locally, it will use these in preference. This is implemented in django/utils/unittest/__init__.py. The correct way to import unittest for your tests becomes:

from django.utils import unittest

unittest2 strives to be backwards compatible with older versions of unittest as much as possible, but as with any code change there were unforeseen side-effects. One of these was breaking the Django continuous integration. Django has a Hudson continuous integration server for trunk. By coincidence I have been working on setting up continuous integration with Hudson for some of our Django projects at work. I blogged about it on our team blog:

Back to the matter at hand, the broken Django trunk CI which makes for an interesting story. The junit-xml for Hudson was being generated by django-test-extensions. This information along with the traceback is enough to diagnose the problem:

Traceback (most recent call last): File "./runtests.py" , line 313 , in <module> failures = django_tests ( int ( options . verbosity ), options . interactive , options . failfast , args ) File "./runtests.py" , line 182 , in django_tests failures = test_runner . run_tests ( test_labels , extra_tests = extra_tests ) File "/.../django-trunk/django/test/simple.py" , line 277 , in run_tests result = self . run_suite ( suite ) File "/.../python2.6/site-packages/test_extensions/testrunners/xmloutput.py" , line 14 , in run_suite return XMLTestRunner ( verbosity = self . verbosity ) . run ( suite ) File "/.../python2.6/site-packages/test_extensions/testrunners/xmlunit/unittest.py" , line 811 , in run t ( result ) File "/.../django-trunk/django/test/testcases.py" , line 291 , in __call__ super ( TransactionTestCase , self ) . __call__ ( result ) File "/.../django-trunk/django/utils/unittest/case.py" , line 397 , in __call__ return self . run ( * args , ** kwds ) File "/.../django-trunk/django/utils/unittest/case.py" , line 359 , in run self . _addSkip ( result , str ( e )) File "/.../django-trunk/django/utils/unittest/case.py" , line 300 , in _addSkip addSkip ( self , reason ) File "/.../django-trunk/django/utils/unittest/result.py" , line 124 , in addSkip self . skipped . append (( test , reason )) AttributeError : '_XmlTextTestResult' object has no attribute 'skipped'

Now this is pretty weird. Test skipping is something that is new in unittest2, but there shouldn't be problems with using test result objects built to the old API as unittest2 will happily work with them. If there is a skipped test and the test case is given a test result that doesn't support skipping it will report the test as a success and emit a warning. There is something odd going on here though, the traceback includes the addSkip method in result.py in the embedded django copy of unittest2 . That means we do have a 'new' TestResult object, but somehow it doesn't have the skipped attribute. The skipped attribute is created in TestResult.__init__ , so it seems like we have an uninitialised result object. Curious.

So we need to dig into the django-test-extensions code (in xmlouput.py ) to see what is going on.

The import code for this module looks like this:

import time , traceback , string from unittest import TestResult from xmlunit.unittest import _WritelnDecorator , XmlTextTestRunner as his_XmlTextTestRunner from django.test.simple import * from xml.sax.saxutils import escape

The code that defines the TestResult object in use (same module) starts like this:

class _XmlTextTestResult ( unittest . TestResult ): def __init__ ( self , stream , descriptions , verbosity ): TestResult . __init__ ( self ) # more code...

Can you see the problem? unittest is never actually imported. This code works because of the import * that pulls in everything from django.test.simple , which happens to include the unittest module. See also that the TestResult is explicitly imported from unittest and used directly.

What has changed is that the version of unittest imported in django.test.simple is now unittest2 and not the vanilla standard library version. This means that _XmlTextTestResult(unittest.TestResult) is defining a class that inherits from unittest2.TestResult . However the __init__ method explicitly calls up to TestResult.__init__ which has been imported from the standard library version of unittest. This means that the unittest2.TestResult initialiser is never called and we end up with a unittest2 test result object with no skipped attribute!

(Calling an unbound method on a class requires that the first argument you pass in - self - be a real instance of the class. (A restriction that has been removed in Python 3.) It works here because the unittest2 classes do inherit from their equivalents in the standard library unittest.)

This dear reader is why the wildcard import form, import * , is bad and you should never use it. As a side note, if django.test.simple had always defined a __all__, exporting only the public names it defined, then this code would never have worked and so wouldn't have been left in such a 'crazy' state. If you have a module that defines a public API it is good practise to define __all__ . At the very least it provides documentation as to what you consider the public API of your module to be.

It was too late to fix django.test.simple in this particular way as django-test-extensions is now depending on being able to import unittest from there. Jannis Leidel (jezdez) fixed the django continuous integration by forking django-test-extensions.

Note Even in its fixed state the junit-xml generator in django-test-extensions doesn't understand new features like test skipping, which is supported by the JUnit XML spec. In preference I would reccommend pyjunitxml. See my blog entry on Continuous Integration with Django and Hudson on how we set that up at work.

(Apparently last night the 'django guys' added __all__ to the django.test.simple module. That means if you're doing crazy tricks like django-test-extension was your code will deservedly be broken with Django 1.3.)

Archives