The Dojo Toolkit is a modular open source JavaScript library, designed to ease the rapid development of JavaScript or Ajax-based applications and web sites. InfoQ had a Q&A with Dylan Schiemann, CEO of SitePen and co-creator of the Dojo Toolkit, about AJAX, Comet, Bayeux, RIAs and the newly released Dojo Toolbox.

InfoQ: Dylan, could you give us a small overview of how you started developing Dojo, what was your initial vision and what has Dojo grown to become?

In early 2004, Alex Russell began looking to hire a collaborator on DHTML (now known as Ajax) projects at Informatica and I became that collaborator. During our work on these projects, many members of the community were contacted with the goal of creating a toolkit that could be used freely across companies, without reinventing the wheel for every new effort. It was apparent that toolkits built to support Netscape 4 had too many inherent inefficiencies, so we decided to take the best ideas from a number of early projects (netWindows, BurstLib, f(m), etc.) and create a new toolkit that would make life easier when using JavaScript. While we saw the value in what we were doing, we had no idea of the reach or impact would have on the development community.

InfoQ: Could you give us an architectural overview of the toolkit? What do we get by downloading Dojo version 1.1.1?

Dojo is structured in 4 main projects: Dojo Core, Dijit, DojoX, and Dojo Utilities. Simply including the standard dojo.js gives you: A very fast, lean (26KB gzipped) core packed full of APIs for XHR, events, DOM, query, animations, namespacing, and other highly used convenience functions.

Optional modules for dates, colors, back button/history, currency, and more In Dojo 1.2, we’ve actually provided an optional facility to get your base build down to 6KB and build up from there. The Dojo core also includes additional code that may be optionally included such as a behavior system, common regular expressions, date and currency APIs, and more. Dijit is both a widget system and a collection of common widgets, meaning that you get a nice set of widgets to use by default, and a powerful system for creating your own. All widgets in Dijit are fully accessible, internationalizable, and can be instantiated either through markup in an unobtrusive manner, or through a simple JavaScript instantiation. Dijit includes common form element upgrades, a calendar, editor, tree, sliders, and more. Everything in Dojo and Dijit are deemed production-ready and API compatibility will be one of our higher priorities as we work towards Dojo 2.0. DojoX, or Dojo extensions, contain features that are less widely, complex, or not yet production quality. Examples include encryption, additional Dojo data stores, dojo.gfx for native vector graphics support, charting, offline, grids and other widgets, and much more. Features in DojoX are not guaranteed to be backwards compatible across 1.x releases. DojoX contains a significant code base with great new additions arriving on a regular basis, most recently an XMPP client, JSON-Schema, JSON-Referencing, and a secure Ajax module, all of which will ship with Dojo 1.2 and are available in the current nightly builds. Dojo Utilities offers, as its name suggests, a number of useful utilities for JavaScript developers, including A highly-flexible build system for combining JavaScript files, satisfying dependencies, and merging HTML and CSS files, based on a slightly customized version of Rhino

ShrinkSafe, a compression tool that removes comments, extraneous whitespace, and more, which also makes use of Rhino

DocTool, a highly advanced system for generating API documentation from source code, that also can resolve complex mixin inheritance structures and more

DOH, the Dojo Objective Harness, a unit test tool that works both in the browser and on the command-line Dojo is organized in a very modular manner, making it easy to just get the features you need so you don’t expend system resources for everything else in Dojo. Dojo is also an open-source foundation, and is home to other projects including Cometd, DWR, Open Record, and the Psych Desktop.

InfoQ: Dojo seems to have had a lot of momentum this year, especially with the release of three new books by O’Reilly, Pragmatic Programmers and Addison Wesley, this summer. Why do you think that is?

First, I think the books were possible because we released a stable version 1.0, and we were able to find some great authors interested in writing about Dojo. Feature-wise, companies such as IBM, Google, Sun, Blue Coat Systems, Nexaweb, and many others, along with our great individual contributor base, made significant and focused efforts towards creating a toolkit that is fast, powerful, and feature-rich. I think the best answer to the question is that we just have a great community that works hard and our hard work is starting to pay off after spending several months in 2007 on completely rewriting Dojo to be a faster leaner toolkit. A lot of this also comes from how we have grown our community. Being very open and transparent, using favorable licensing (BSD or AFL), and also using a CLA process for making sure there are no intellectual property issues with using source code. Dojo scores, by Dion Almaer’s standards, a perfect 100 points on the how open is your project scale.

InfoQ: A few days ago SitePen released Dojo Toolbox, an AIR application for Dojo developers. Would you like to talk to us about that?

Adobe, in an interesting move, released a platform that is WebKit-based, supports both Open Web technologies (HTML, JavaScript, CSS, Canvas, SVG, etc.) and its proprietary Flash/Flex stack. This gives developers some of the benefits of desktop applications including local file system access, better caching, and a single-deployment environment. They’ve achieved a great way for web developers to create installable web apps. Adobe approached SitePen first about refining Dojo’s support for AIR prior to the release of AIR, and then with the proposal to co-sponsor development of an application, which led to Dojo Toolbox. Dojo has such a large feature-set that creating a printable PDF of the API documentation or an easy-to-view structure is always a challenge, so we decided to bring the 18MB of Dojo documentation offline. We also knew that providing an easy way to create optimized versions of a Dojo build would be handy and AIR was a great solution. All of this was done under the BSD license. In the few short weeks this has been released, we’ve received a number of great feature requests and plan to increase the app’s functionality as a result. We’ve also created something that is modular enough that people can create their own customized toolbox builds or contribute their own modules back to the toolbox for consideration in the official release.

InfoQ: Another popular way in the Java world for developing web application besides pure JavaScript libraries like Dojo, is GWT. The way it works is fundamentally different since it acts as a code generator that tries to keep developers away from checking all the different browser quirks. How do you think these two measure up?

I think too often there’s a tendency to say that you have to choose X over Y, rather than giving developers the flexibility to choose combinations of what they like or what works best. We like giving people choices, and in fact, people are currently working on a project to bring Dojo to GWT. There’s a talk Alex Russell gave at this year’s Google I/O event titled “Can We Get There From Here”. One of his points was while GWT is open, the more it does, the harder it is to deconstruct. On the complexity scale, with HTML/CSS being the most transparent, Dojo is a little more complex than plain HTML or simple JavaScript with features such as namespaces, while Silverlight, GWT, and Flex are much more difficult to decompile. With Dojo, we’ve made an explicit decision that a compile step is purely optional. GWT does have a reload button, but that requires using an IDE. Their code optimizer gives it an additional performance boost that we hope to replicate with our JS Linker utility, but we decided that source code would remain extremely easy to comprehend for JavaScript developers. I urge people to experiment and determine the most productive choice for them, taking into account factors that matter to their business.

InfoQ: There are several other JavaScript frameworks/toolkits out there that have had varying degree of success and adoption in the enterprise. What do you think differentiates Dojo from the rest?

Dojo receives wide, practical, “we need this feature now” improvement from the likes of IBM, Sun, AOL, and many other companies including the clients of SitePen. Our open licensing, attention to the details important to enterprise, from IP issues to accessibility and internationalization support, makes us a great open source solution for companies looking to optimize their development power. Financial institutions, militaries, governments, and many other large organizations use Dojo throughout their web development models. Much of this work is Ajax “Dark Matter”, used behind the firewall.

InfoQ: This year the organization responsible for the new ECMAScript, version 4 (AKA JavaScript 2) published a revised version of the language overview, that spawned much controversy and debate. How do you feel about the proposed changes to the language and their effects to JavaScript frameworks like Dojo?

Kris Zyp of SitePen currently participates in the ECMAScript working group. To date, it has had little impact on the Dojo Toolkit itself because we’re pragmatic and wait until something is ready before we use it. In general, we’ve tried to be a voice of reason that is impartial to politics and focus on what JavaScript developers need most from a language. We’re pushing for features that we believe matter and recommending against changes that we think make development more difficult.

InfoQ: When creating web UIs that heavily utilize JavaScript, there are many pitfalls that are not always apparent to the more inexperienced. Usability, search engines, “back button” behavior, browser performance etc. Which are the most common mistakes you see Dojo developers doing?

A single browser page, or any web or software application for that matter, has access to a finite amount of system resources. In addition to the issues you’ve mentioned, a common trap is for new developers to not understand the implications of their work, e.g. implementing 100 dynamic charts in a page with a sub-second XHR polling implementation. With Dojo 1.0 and beyond, we’ve tried to be less “magical” so developers can better understand that each and every feature comes at some price, that the fastest toolkit is 0KB, and that most things that are difficult require challenging decisions and trade-offs. Another common mistake developers make is assuming that browsers are consistent and rational. Long-held opinions on the best way to do something are often incorrect and surprising. Simply “looking at the data” with tools like Firebug reveal performance bottlenecks and tell us more about the true performance behavior of browsers..

InfoQ: It has been suggested that the creation of elaborate JavaScript frameworks is a sign that the traditional "Web Platform" has reached its maximum potential and should be replace by something more modern. What do you think about that?

I think this is simply a case of the rapid rate of invention over the past four years drastically outpacing the capacity of browser vendors to create better deployment environments. Browser vendors are often slammed for trying a non-standards-based approach, which is unfair because something can’t become a standard if there’s no room for experimentation and innovation. Flex and Silverlight have done a decent job with providing a good development experience, but don’t count out the Open Web yet. We believe that an evolutionary approach to constantly improving what we have today is a viable option. Take WebKit for example, and its success as a great environment for apps in the browser, inside Adobe AIR, on many mobile devices obviously including the iPhone, or Opera on the desktop, mobile devices, and the Nintendo Wii. JavaScript toolkits have done a great job of updating the web faster than browser vendors can, as have things like Google Gears. As browser vendors add new features and new capabilities, toolkits like Dojo defer to the native implementation, and then look for the next thing that needs to be improved across the web.

InfoQ: How do you think pure-JavaScript frameworks measure up against plugin-based RIA solutions like Flex, OpenLaszlo or Curl?

I think they hold up quite well. OpenLaszlo is in many ways more about being a compiler like GWT that can deploy to Flash or the Open Web. What’s interesting is that we’ve started to take things in another direction. For native vector graphics, dojox.gfx has supported SVG, Canvas, and VML for a while. Recently, we added support for Silverlight as a renderer because VML is terribly broken in the IE alpha builds, and we’ve been experimenting with Flash and Flex for rendering vector graphics and charts as well. We also support Flash’s local storage facility in browsers that don’t have native local storage. Again, I mention that it’s about choice and also pragmatism.

InfoQ: It is commonly believed that there are certain things that you cannot do with JavaScript and have to turn to Flex/Flash. Some of those things are very pervasive, like charting. With the latest activity on charting and other areas it seems that Dojo is limiting the reasons to use other technologies. Do you think that as JavaScript libraries become more sophisticated and the support from browsers improves, the need for page components implemented from within plugins will be minimized?

With Dojo, we believe that anything that cannot be done natively can still be done using Dojo, if so desired. Our native vector graphics and charting support is competitive with non-native options, but we give users choice rather than trying to lock developers into one solution over another. The future is hard to predict, but the role of the plug-in is generally to accomplish something that cannot be accomplished using native options and JavaScript. With native vector graphics support in all modern browsers, and dojox.gfx to hide the differences in implementations across browser, the need to use a plug-in for this feature is now just an option rather than a necessity. This is all part of the natural evolution of the Open Web.

InfoQ: It has been said that IE 7 was a gift to designers since it had a better support for CSS, while IE 8 will be a gift to JavaScript developers. With all the publicity and the details that have been made available how do you feel about the upcoming version of IE? Is it a significant step forwards? How do you think it will affect JavaScript frameworks like Dojo?

I think it is too soon to tell. The first release felt like two steps forward (6 connections instead of 2, secure cross-domain requests), one step back (severe bugs with VML), so we’re cautiously optimistic for the next release.

InfoQ: After 2005 when AJAX became widely known, the latest buzzword is Comet. There are still people debating on terminology so I would like to ask you to give us your own definition.

Comet is a collection of techniques, transports, protocols, servers, and clients to deliver low-latency data transit between the server and the client. The XHR is the least common denominator of Comet, and everything else is an attempt to improve upon the XHR to deliver faster, real-time applications. My definition intentionally avoids some points of the debate, as I prefer to be inclusive and broad, much as Ajax has come to represent everything dynamic in a browser to most people.

InfoQ: An important components for enabling Comet applications seems to be the Bayeux protocol which facilitates routing JSON encoded events between clients and servers in a publish subscribe model. Would you like to elaborate on that? It seems that Bayeux is gaining de facto industry acceptance since it is unique, but do you feel it will be heading for a more formal standardization, e.g. through W3C?

To date, I think the informal nature has been highly useful, though if there’s interest, I’m sure a standards-body will eventually pick it up. Bayeux is the standard protocol for servers and clients that are part of the Cometd project. XMPP is of course another popular protocol. I’m also quite interested in the recent draft WebSocket addition to the HTML5 efforts, and the API abstraction of this already implemented within the Orbited project.

InfoQ: In the Java realm there are several ways to develop Comet applications at this moment. With the emergence of the Servlets version 3 spec (JSR 315) which defines a standard way of doing Comet, how do you think this genre of application will grow?

In the Java space, a variety of open-source Comet server options such as Jetty, DWR, and GlassFish, as well as free versions of Lightstreamer and Caplin Liberator are available. The new servlets specification has already experimental support in Jetty and GlassFish, and I’m sure when the final specification is ready, everyone will make use of this. One interesting thing to emerge is the distinction between on-board and off-board Comet by Joe Walker at JavaOne. The language that your Comet server of choice is written in is less important than making sure you have client access (client being a web-based client, or a web server based client communicating with your Comet server) in your language of choice

InfoQ: It has been suggested that since web applications target clients that understand JavaScript (browsers) it would make sense to have end-to-end JavaScript solution. What do you think about that? Do you see server-side JavaScript becoming mainstream?

The emergence of Jaxer (by Aptana), Persevere (SitePen), AxiomStack (Axiom), Phobos (Sun), and others have shown that is becoming common. JavaScript is now the world’s most ubiquitous programming language, available just about everywhere. I think that for anyone that started out on the client-side, it’s a great way to get into the server-side without learning the syntax of PHP, Python, Java, or other languages (though most users of Dojo currently come to Dojo in the more traditional direction). I don’t know what the future holds, but I think it’s a great opportunity to use the same JavaScript code base everywhere.

InfoQ: Although you can often find developers handling large Java code bases, it is not the same with JavaScript. Its dynamic nature and the fact that frameworks like Dojo target a varying set of clients (browsers) must impose unique restrictions and difficulties in the development process. Besides using Trac for issue tracking what are the tools and procedures that you have found particularly effective for this specific genre of software?

I think you’d be surprised at the size of some JavaScript code bases! At SitePen, we recently switched from Trac to Redmine as it has kept the simplicity of Trac while adding the features we’ve missed for a long time (dependencies, projections, multiple projects). We still use Subversion as it has great support across a variety of tools. At SitePen, we use pretty much every text editor or IDE there is depending on which developer you ask. We use the Dojo DocTool and the utilities found in Dojo. Some of the less common development tools we use include: Windmill, Tito Web Studio, Charles, and Versions. I can’t say we have a standard across the board, here’s the best way to maintain and refactor a large codebase tool like you would have for Java, but I’m not sure the need is as great either.

InfoQ: People have speculated that when Steve Yegge talks about what will be the next big language (if any) he refers to JavaScript. The fact that he has ported a significant part of the Rails framework from Ruby to JavaScript makes this speculation more solid. Having worked with this language do you see this kind of potential?

Absolutely and I think this ties back to the question about JavaScript on the server. Persevere is another example of this type of approach. In short, Persevere is an open source set of tools for persistence and distributed computing using intuitive standards-based JSON interfaces of HTTP REST, JSON-RPC, JSONQuery, and HTTP Channels.

InfoQ: How do you see Dojo evolving? What are your next steps and what can we expect in the following months?

In the short term, we’ll continue to improve performance, widgets such as the grid, charting, tree and editor, the overall look and feel of Dijit, form and data validation, the stability of our Comet client, our validation system, and much more. We’re also starting to look towards Dojo 2.0, probably in 2010. It’s really about keeping Dojo useful, promising and efficient for developers who are interested in coding impressive web apps and providing excellent experiences for their users.

InfoQ.com features more interesting articles on RIAs, JavaScript and Web 2.0!