Maintaining a complex web page full of different components with their own requirements can be challenging. Often you will start with a simple web page but slowly you start adding new components and integrating with other services.

When you have a dynamic, feature-rich page caching becomes more difficult and testing an entire page becomes impractical. Your tests can become difficult to maintain and can become slow.

Micro components

By breaking your page up into separate components you get the following advantages:

Easier to maintain distinct pieces of functionality

Reusability

More flexible caching

Ability to independently deploy components so you can enhance and fix bugs without a “big bang” release.

Inspired by the LinkedIn talk by Yevgeniy Brikman, Rob and I wanted to deliver a proof of concept of this approach for SpringerLink’s abstract/fulltext page. (e.g http://link.springer.com/article/10.1007/s10817-012-9258-1).

We wanted to break up our page into smaller components, pull them in via HTTP asynchronously.

Our proof of concept

The components

We developed 4 simple web pages in the Play framework which represent the main features on an article page.

http://localhost:9000/abstract/{ID} - Displays the HTML abstract for a given ID

- Displays the HTML abstract for a given ID http://localhost:9000/fulltext/{ID} - Displays the HTML for the entire document for a given ID

- Displays the HTML for the entire document for a given ID http://localhost:9000/downloads/{ID} - Displays total number of downloads and downloads by country for a given ID

- Displays total number of downloads and downloads by country for a given ID http://localhost:9000/time - Returns the server’s local time

Putting it together

Now that these components are created you can imagine developers being able to work on them without worrying about the rest of the website. They can be tested and deployed completely independently from the big web app.

Now we have the problem that we want to bring all these components to construct a web page

Scala is really great at asynchronous programming, so this is as easy as just making 4 HTTP calls and wrapping them in Future . You can then compose them together, plug it into a template and then you have the page fully rendered.

Play is a great choice for this because it allows you as a developer to have your controllers return Future’s and it will handle the rest.

Here is the code for the aggregating controller

As you can see, there is no real complexity in breaking up the page into pieces, fetching them asynchronously and then composing it together into a view.

Edge side includes

Edge side includes are server side tags which contain a URL. Proxies like Varnish and Akamai can read and then fetch the HTML from its cache and then output it to the page seamlessly for the user.

High quality diagram

Given our components are now pulled in via HTTP, we wanted it so if you were hitting the website via Varnish the web server would create ESI tags rather than making potentially slow HTTP calls itself.

The simplest way to accomplish this is to edit the getBodyOf function:

The net result for this is the server can output ESI tags and delegate the hard work to the cache which can store these potentially heavy requests, rather than doing the heavy work itself.

You’ll notice that we check for the presence of a Varnish request, the reason for this is we wanted to make sure that development could be done without the need of Varnish running on a local dev box.

By doing this the page remains dynamic, yet highly cacheable. We can serve a cached version of the page depending on your access rights.

What about client side?

You can find plenty of tutorials on bringing in data asynchronously via the client. This approach works fantastically on a number of successful websites, like Facebook and LinkedIn.

However you can face a number of tricky issues around this approach, Twitter famously reverted from a client side approach to moving this complexity back to the server

“To improve the twitter.com experience for everyone, we’ve been working to take back control of our front-end performance by moving the rendering to the server”

I feel client side rendering and pulling of data is great for auxiliary content but it does add complexity to the client which I feel is a bit easier to handle on the server. This post by Karl Seguin goes over some of these problems.

In addition, managing things like SEO and testing are a bit easier on the server side and for the main content on our site we need to make sure these things are bullet-proof.

That said, the micro-component approach doesn’t restrict you using the client side, it actually allows you to be flexible in how you want content to be delivered to your users.

You may prefer in some cases to output a bespoke HTML tag which your client side code knows to look for and then subsequently retrieve the data.

Next steps

The code above is just a proof of concept and as such is a bit messy so it would be nice to encapsulate the behaviour of “Retrieve the contents of this URL by this means” in some kind of re-usable way.

I would like to write a nice declarative way of writing things like

“Fetch the contents of URL-A” “Output ESI tags for URL-B” “Output client side include tags for URL-C”

and then be able to compose these calls into a view for rendering.

The code I didn’t show was of the components, which you need to make sure set the appropriate cache headers.

Conclusions

In spite of our page being very dynamic the main bulk of the response can be brought in from a cache, which brings in the benefits of decreased load and faster performance.

This architecture has the happy side-effect that you have to make your components small and independent, which in the long run makes a more maintainable, re-usable platform.

With modern languages the old objections of this approach being too complicated are largely irrelevant. Scala’s monadic composition makes this very easy but would be just as doable in other languages such as Javascript with Node.js

You can accomplish all of these things client side too and there is no real right answer as to where this logic should live; but I hope this post gives you a heads up on a different approach which might be more suitable for your needs.

By Chris James