Now we've announced our new Open Platform's Content API and microapp framework, I can talk a bit more about how the Guardian Zeitgeist actually runs and how it's using the APIs and the microapp framework.

People are generally pretty familiar with APIs but microapps may need a bit more explaining, so here's a very quick overview. First a screenshot of the Zeitgeist running on the Guardian...

But wait, it's really running off Google App Engine. This is what it looks like naked on Google's own servers...

The microapp framework allows the Guardian to assign areas of a single page or, various types of pages to a microapp, which can be hosted elsewhere. When the Zeitgeist page on the Guardian is loaded, it requests the content from the Zeitgeist application and places the results into the page, rather like this...

Because I'm running Zeitgeist on Google's App Engine I'm not that worried about scaling. Even so, I can set cache headers at my end telling the Guardian to only grab new data from me once every 10 mins and I can adjust this as I need to. If my app goes down or start throwing errors the Guardian can just use it's latest cached version.

This is good for me as a developer, even an in-house one because I'm not tied to the Guardian's own internal platform. My weapon of choice happens to be Python, but it could be coded in anything that serves up HTML. I can also easily deploy updates outside of our internal deploy cycle, which is nice.

Ok, so what does the Zeitgeist do? Well I've already covered that a bit in this blog post here: What's hot? Introducing Zeitgeist,

Zeitgeist is a visual record of what people are currently finding interesting on guardian.co.uk at the moment. While other bits of the site are curated by editors (like the front page, or individual sections) or metadata (like blogs, which display in reverse-chronological order), Zeitgeist is dynamic, powered by the attention of users. We hope that this makes for an interesting alternative springboard into the content on this site, and those who have been playing with it behind the scenes can confirm that it's a great starting point when you've got a few minutes spare and just want something to read but you're not sure where to start.

This is roughly what's going on under the hood ...

The Brain

The powerhouse behind Zeitgeist is a Python App Engine thingy called "The Brain", this is the one that does all the serious number crunching. Every hour our analytics system emails over the latest data from the site, across all sections, searches and so on. The app has mail handlers waiting for the reports to arrive, it then parses the data, fetches additional information (such as keywords, sub-sections and so on) from the Guardian Open Platform API, for each item row in the reports.

It then ranks the news (based on complex secret sauce algorithms, discussed more over here) which form the base Zeitgeist scores. The results are then stored again and indexed so we can find the top stories based on section (i.e. technology, politics), type (article, gallery) or both.

Here's the scary backend graph of it working...

... and that's basically The Zeitgeist.

The Layouterer

Because we generally try to keep the data layer and the presentation separate we have a different Google App that generates various HTML views of the data. There are cron tasks set up to request information from The Brain via an API serving JSON, which is then stores locally and converts into HTML. We hope to get this Zeitgeisty API endpoint into the Open Platform "soon".

There's two reasons for doing it this way ...

We want The Brain to just concentrate on the heavy number crunching. If we want to add a new HTML view of the data we don't need to touch The Brain, its left doing it's own thing. Also it means the CS (frontend) team can work with the code and not worry about breaking anything databasey.

If for some reason the backend Brain decided to crash and burn, the Layouterer still has latest copies of the results that it can continue to serve up to The Guardian. Giving us time to diagnose any problems and restore data if we need to.

The Searcher

But wait, there's more, the "Love Notes" ...

We also keep an eye on what's happening out there on the Internet. The Searcher watches for search terms that are being used to arrive at The Guardian, it also monitors Social Networking sites, OneRiot and Twitter for mentions of Guardian articles. It's very general at the moment, but again it can be worked on separately from The Brain without fear of breaking anything.

When it notices a lot of activity around a certain article, it sends a note to The Brain to keep an eye out for it. The Brain can then boost the Zeitgeist score of articles based on this information.

This also works the other way round, if The Brain spots items that are rapidly trending: getting hotter at a much faster rate than normal, it sends a request to The Searcher to see if it can find out what's going on.

The same thing happens if it notices that something is still ranking highly after a number of days, or if an old article reappears in the reports.

By sending these "Love Notes" between each-other the systems can keep updating hotness values independently of the reports arriving. The Searcher is also a place where we can experiment more and ultimately, hopefully, be used to report information back to us - the news organization. Letting us know when things are doing exceptionally well and more importantly an insight into why they are doing well.

The $1.24 Wrap-up

What we have here is a great combination of the Open Platform microapp framework allowing us to inline content hosted elsewhere, the APIs to get further information such as keywords to generate a better understanding of trending topics and Google App Engine to do the heavy lifting without needing the infrastructure in-house.

The Zeitgeist itself is still an evolving experiment, one that hopefully gives the reader of the Guardian's website an interesting way to dip into the news, and through various back-end tools give us as a news organisation more information and insight into how people interact with our content.

The cost? Well Google App Engine sets us back a whole $1.24 a day, or 84 pence (and that's without a huge amount of hardcore optimization). Which I think is easily worth it. The Guardian's Content API is free to play with and you don't even need to sign up or get an API key to start, although we'd love it if you did :)

Database and World icons from Tim Morgan, Creative Commons Attribution License.