Python Programming, news on the Voidspace Python Projects and all things techie.

Decoding json on Silverlight with System.Json (plus implicit conversion operators in IronPython)

As I explained we're writing a Silverlight application that communicates with our Django server and exchanges a lot of json with it.

Unfortunately, due to what is apparently just an oversight, the codecs module is incomplete for IronPython on Silverlight. This means that recent versions of simplejson don't work.

What we've been using is an older version of simplejson, that pulls in the obsolete and huge sre module in as one of its dependencies. On top of bloating up our application and adding several seconds to the startup importing it all the performance is not exactly blazing.

Fortunately part of the Silverlight SDK is System.Json.dll. Using this API from IronPython is pretty simple. The following code defines a loads function (load string - the same API as simplejson) that takes a string and parses it.

import clr clr . AddReference ( 'System.Json' ) from System import Boolean from System.Json import JsonValue , JsonType String = clr . GetClrType ( str ) def loads ( inString ): thing = JsonValue . Parse ( inString ) return handle_value ( thing ) def handle_value ( inValue ): if inValue is None : return None elif inValue . JsonType == JsonType . String : return clr . Convert ( inValue , String ) elif inValue . JsonType == JsonType . Boolean : return Boolean . Parse ( str ( inValue )) elif inValue . JsonType == JsonType . Number : return get_number ( inValue . ToString ()) elif inValue . JsonType == JsonType . Object : return dict (( pair . Key , handle_value ( pair . Value )) for pair in inValue ) elif inValue . JsonType == JsonType . Array : return [ handle_value ( value ) for value in inValue ] # Should be unreachable - but if it happens I want to know about it! raise TypeError ( inValue ) def get_number ( inString ): try : return int ( inString ) except ValueError : return float ( inString )

As with my custom json emitter there is a lot it doesn't do. It handles the following types:

None

lists

dictionaries

strings

floats and integers

booleans

If you want it to handle decmials or dates you'll have to add that yourself. How it works is mostly straightforward, but there is one little piece of 'magic' in there. When you call ToString() (or str() they do the same thing) you get the original json back. For numbers and booleans this is fine as we can easily turn them into the objects they represent. For strings this is a nuisance as we get double quoted, escaped strings. The correct way to get the value we want is to use implicit conversion. In C# this looks something like:

string value = jsonArray [ "key" ];

Declaring the result as a string calls the appropriate JsonValue implicit conversion operator for us. IronPython doesn't have implicit conversion operators as we don't declare types and with a dynamic type system you rarely need to cast. Up until version 2.6 it wasn't possible to call implicit operators, but in IronPython 2.6 we gained a new function in the clr module; clr.Convert :

>>> print clr.Convert.__doc__ object Convert(object o, Type toType) Attempts to convert the provided object to the specified type. Conversions that will be attempted include standard Python conversions as well as .NET implicit and explicit conversions. If the conversion cannot be performed a TypeError will be raised.

The implicit conversion to string is done with the code:

clr . Convert ( jsonValue , clr . GetClrType ( str ))

I cover some of the other goodies new in IronPython 2.6 (like CompileSubclassTypes ) in Dark Corners of IronPython.

I haven't yet written the code to do json encoding, but the JsonValue class (and friends) can be used for encoding as well as decoding, so I expect the code will pretty much be the opposite of the snippet shown above...

Django: Tear down and re-sync the database

Django includes the useful management command syncdb for creating the database tables and columns used by your application. If you add new tables (model classes) then re-running syncdb will add them for you. Unfortunately if you modify columns of existing tables, or add new columns, then syncdb isn't man enough for the job.

For modifying the schema of production systems migrations are the way to go. I played a bit with South for Django, which is pretty straightforward. For a system still in development, and changing rapidly, migrations are overkill. We have a script for populating the database with test data, which we update as the schema evolves. (In parallel with this we have a script that imports the original data from the legacy application we are replacing - again updating the script as our app is capable of handling more of the original schema.)

For development what we really want to do is to tear down our development database and re-run syncdb. Running syncdb requires manual input, to create a superuser, so preferably we want to disable this so that the whole process can be automated. I found various recipes online to do this, but mostly using an obsolete technique to disable superuser creation.

In the end I used a combination of this recipe to programatically clear the databases (using the sql generated by sqlclear) and this recipe to disable super user creation.

Note The code also skips clearing the authentication table as we are using Django authentication unmodified. Comment out the line that does this if you aren't using Django authentication or want to clear it anyway.

#!/usr/bin/env python import os import sys import StringIO import settings from django.core.management import setup_environ , call_command setup_environ ( settings ) from django.db import connection from django.db.models import get_apps , signals app_labels = [ app . __name__ . split ( '.' )[ - 2 ] for app in get_apps ()] # Skip clearing the users table app_labels . remove ( 'auth' ) sys . stdout = buffer = StringIO . StringIO () call_command ( 'sqlclear' , * app_labels ) sys . stdout = sys . __stdout__ queries = buffer . getvalue () . split ( ';' )[ 1 : - 2 ] cursor = connection . cursor () for query in queries : cursor . execute ( query . strip ()) from django.db.models import signals from django.contrib.auth.management import create_superuser from django.contrib.auth import models as auth_app # Prevent interactive question about wanting a superuser created. signals . post_syncdb . disconnect ( create_superuser , sender = auth_app , dispatch_uid = "django.contrib.auth.management.create_superuser" ) call_command ( 'syncdb' )

It wasn't all plain sailing. We're using MySQL (God help us) and our development machines are all running Mac OS X. On Mac OS X MySQL identifiers, including table names, are case insensitive. Whilst I would object strongly to a case sensitive programming language this actually makes working at the sql console slightly less annoying so it isn't a problem in itself.

We define our data model using standard Django model classes:

from django.db import models class NewTableName ( models . Model ): NewColumnName = models . CharField ( max_length = 255 , db_column = "OriginalSpaltennamen" ) class Meta : db_table = 'UrsprunglichenTabellennamen'

The Meta.db_table specifies the table name that will actually be used in the database. We use the original table and column names where possible as the end users will have to modify some existing tools to work with the new system and this minimizes the changes. As you can see both the original table and new table names are mixed case.

For some reason, which I never got to the bottom of, where the model classes have foreign key relationships syncdb will create these tables with all lowercase names. This could be Django, MySQL or the Python connector to MySQL (or any combination of these) and I never worked out why.

Unfortunately sqlclear will only generate sql to drop tables where the casing specified in the model exactly matches the casing in the database. I worked round it by changing all our Meta.db_table entries to be all lowercase. Not what you would call ideal but acceptable.

Now everytime we update our database schema we can simply run this script. It drops all existing tables and then re-creates them with all the changes.

Note Carl Meyer suggests using call_command('syncdb', interactive=False) instead of the signals.post_syncdb.disconnect code. It's certainly shorter but I haven't tried it yet. In the comments Stavros Korokithakis points out that the reset admin command will reset individual apps and regenerate them. If you have several apps in a project this script is still simpler, but if you only need to reset one then you might as well just use ./manage.py reset <appname> . It takes the --no-input switch if you want to supress the user prompts.

Archives