Accelerate the performance of Dojo TreeGrid

Load large data in less time

The Dojo v1.4 TreeGrid is a useful widget when you want to present hierarchical data on a web page. But if you have a large data set, TreeGrid 's performance is extremely slow. In this article, learn how to work around this problem by customizing TreeGrid and QueryStore . The article describes problems you might encounter when using the two widgets, explains the reasons for the errors, and then helps you create a solution. You can also download the sample code used in this article.

Dojo grid and large data sets

Of course, you can use the grid to effectively display relatively small data sets in the browser. You get the niceties of sorting, column resizing, and so on. However, there's a practical limit to the number of records you can deal with at any given time, which eventually leads to the issue of paginating results.

You can forget about pagination—those days are finally over. Dojo's grid uses lazy Document Object Model (DOM) rendering as the grid scrolls. For a relatively small data set with less than a few hundred records, lazy DOM rendering builds out the DOM for a preloaded data set when you scroll. For example, if you had 100 records but can only view 20 at a time, you wouldn't need to build the nodes for any of the records in 21 through 100 until you scrolled to that particular section. Sorting by a column and related tasks works in memory as expected for small data sets; you can effectively get things done in JavaScript.

For very large data sets it quickly becomes impractical, or even impossible, to use JavaScript for tasks such as sorting by a column. If the data set is gargantuan, it's not practical to try to maintain it in the browser with an ItemFileReadStore , which loads all data into a browser.

Dojo's approach to dealing with large data sets is simple, elegant, and boils down to two things:

A dojo.data implementation that can request data from the server in arbitrary page sizes. This article uses QueryReadStore .

. A grid that's capable of scrolling such that it requests and loads a particular page on demand.

The rest of this article shows you how to build a lazy-loaded tree grid with QueryReadStore .

QueryReadStore, TreeGrid, and TreeModel

This section provides a very basic introduction to QueryReadStore , TreeGrid , and TreeModel .

QueryReadStore

As stated in the dojocampus.org doc (see Related topics), QueryReadStore is very similar to ItemReadStore . They both use JSON as their exchange format. The difference is in the way they query data. ItemReadStore makes one fetch from the server and handles all sorting and filtering in the client. That's fine for hundreds of records. But for hundreds of thousands of records, or slow Internet connections, ItemReadStore is less feasible.

QueryReadStore makes a request to the server for each sorting or query, making it ideal for large data sets with small windows of data, as in dojox.grid.DataGrid . Listing 1 shows how to create a QueryReadStore .

Listing 1. Create a QueryReadStore

var url = "./servlet/QueryStoreServlet"; var store = new dojox.data.QueryReadStore({ url: url, requestMethod: "get" });

TreeGrid and TreeModel

A tree widget, such as Tree and TreeGrid , presents a view of hierarchical data. The TreeGrid widget itself is merely a view of the data. The real power comes with TreeModel , which represents the actual hierarchical data that the Tree widget will display.

Typically, data ultimately comes from a data store, but the Tree widget interfaces with a dijit.tree.Model (an object matching a certain API of methods that the tree needs). Therefore, the Tree widget can access data in various formats, such as with a data store where items reference their parents.

TreeModel is in charge of certain tasks, such as connecting to the data source, lazy loading, and querying from the Tree widget about items and the hierarchy of items. For example, a task might be getting children of an item for the Tree widget. TreeModel is also in charge of notifying the Tree of changes to the data.

To get started, you need to define the query of the ForestStoreModel (an implementation of TreeModel ) to return the multiple top-level items for the TreeGrid . You will also create the model adapter for the tree to access the store. You need the parameters in Listing 2 to construct a TreeModel .

Listing 2. Code to create TreeModel

var query = { type: "PARENT" }; var treeModel = new dijit.tree.ForestStoreModel({ store: store, // the data store that this model connects to query: query, // filter multiple top level items rootId: "$root$", rootLabel: "ROOT", childrenAttrs: ["children"], // children attributes used in data store. /* For efficiency reasons, Tree doesn't want to query for the children of an item until it needs to display them. It doesn't want to query for children just to see if it should draw an expando (+) icon or not. So we set "deferItemLoadingUntilExpand" to true. */ deferItemLoadingUntilExpand: true });

Relationship between QueryReadStore, TreeGrid, and TreeModel

To understand how to use a TreeGrid , keep in mind the following tree components that feed each other:

QueryReadStore is in charge of fetching data from the server.

is in charge of fetching data from the server. ForestTreeModel is in charge of querying the hierarchy of items from QueryReadStore and notifying the TreeGrid to update.

is in charge of querying the hierarchy of items from and notifying the to update. TreeGrid is in charge of displaying the data and handling user events only.

Figure 1 shows the relationship between the three. Listing 3 shows how to create a tree.

Figure 1. Relationship between QueryReadStore, TreeGrid, and TreeModel

Listing 3. Create a Tree

// define the column layout for the tree grid. var layout = [ { name: "Name", field: "name", width: "20%" }, { name: "Age", field: "age", width: "auto" }, { name: "Position", field: "position", width: "auto" }, { name: "Telephone", field: "telephone", width: "auto" } ]; // tree grid var treegrid = new dojox.grid.TreeGrid({ treeModel: treeModel, structure: layout, // define columns layout /* A 0-based index of the cell in which to place the actual expando (+) icon. Here we define the "Name" column as the expando column. */ expandoCell: 0, defaultOpen: false, columnReordering: true, rowsPerPage: 20, sortChildItems: true, canSort: function(sortInfo) { return true; } }, "treegrid"); treegrid.startup();

How QueryReadStore works with TreeGrid

This section explores rendering the data in the TreeGrid , sorting with QueryReadStore , and expanding parent nodes.

Rendering

The QueryReadStore will send a request to the server for the top-level nodes. The first request the store will make to the server will be a request for the top-level nodes (the children of the root). This dojo.data request follows a specific JSON format.

The tree grid will make the initial request using the query provided to the model, and the store will combine that with the target. Listing 4 shows the request format, the REST URL pattern, and the server response to the request. In the example, the request is made to a REST pattern.

Listing 4. Query request and server response

// query { query: {type: "PARENT"}, start: 0, count: 20 //rowsPerPage attribute of the tree grid } // request sample url?type=PARENT&start=0&count=20 // server response { "identifier": "id", "label": "name", "items": [ { "id": "id_0", "name": "Edith Barney", "age": 39, "position": "Marketing Manager", "telephone": 69000044 "children": true, "type": "PARENT" }, { "id": "id_1", "name": "Herbert Jeames", "age": 43, "position": "Brand Executive Manager", "telephone": 69000077, "type": "Child" }, ... ], "numRows": 10000 // total records in the grid }

The item "Edith Barney" is of type "PARENT" , and the "Herbert Jeames" item type is "Child" . The query in the tree model is {type: "PARENT"}, so the tree grid will only display the "Edith Barney" record initially.

You don't need to actually include the children; the presence of a children property will indicate to the Tree that the node is expandable and an expansion icon will be included. The children attribute of item "Edith Barney" is not a false value, so the tree model will treat it as a node with children. There will be a plus (+) sign in the Edith Barney row in the tree grid. Figure 2 shows the result.

Figure 2. Rendering result of the created TreeGrid

Now that you have the data rendered in the TreeGrid , you'll want to verify whether the TreeGrid features, such as sorting and expanding parent nodes, work correctly with the QueryReadStore .

Sorting

To exercise the sorting function, click on the header of the Name column and sort by name ascending. It is the same for sorting the other columns in ascending or descending mode.

Listing 5. Sorting

// the tree model will pass a request to the query store like this: { query: {type: "PARENT"}, start: 0, count: 20, sort: { attribute: "name", descending: false } } // The query store will request server like this: url?type=PARENT&start=0&count=20&sort=name

Figure 3 shows the sorted result.

Figure 3. Sorted result

Expanding parent nodes

For large tree data sets you probably want to only load the necessary data for the visible nodes of the tree. When a user expands a node, it starts to load the children of that node. Ideally, you only want to make one HTTP request per expansion for optimal performance.

When a user clicks on one of the nodes in the example, the tree will ask the store to load the item and the store will request the resource. But strange things may happen, such as in our case when the plus icon disappeared and the sub-rows didn't show.

Figure 4. Expanding does not work correctly

So, why doesn't it work?

Look through the source code of TreeGrid and TreeModel in Listing 6 to see how expanding sub-rows work.

Listing 6. How TreeGrid and TreeModel expand sub-rows

//TreeGrid.setOpen setOpen: function(open){ ... treeModel.getChildren(itm, function(items){ d._loadedChildren = true; d._setOpen(open); }); ... } //TreeModel.getChilren getChildren: function(/*dojo.data.Item*/ parentItem, /*function(items)*/ onComplete, /*function*/ onError){ // summary: // Calls onComplete() with array of child items of given parent item, all loaded. var store = this.store; if(!store.isItemLoaded(parentItem)){ // The parent is not loaded yet, we must be in deferItemLoadingUntilExpand mode, // so we will load it and just return the children (without loading each child item) var getChildren = dojo.hitch(this, arguments.callee); store.loadItem({ item: parentItem, onItem: function(parentItem){ getChildren(parentItem, onComplete, onError); }, onError: onError }); return; } // get children of specified item var childItems = []; for(var i=0; i<this.childrenAttrs.length; i++){ var vals = store.getValues(parentItem, this.childrenAttrs[i]); childItems = childItems.concat(vals); } }

The tree model will ask the store to load the parent item first. The parent item is not fully loaded because its children attribute is not an array that lists all the children of this item.

Somehow, however, this case will pass the store.isItemLoaded(parentItem) check logic and you have childItems=[true] . Because true is not a valid item, the tree grid will skip this invalid child item, which results in nothing happening. Look at the code for the store.isItemLoaded method in Listing 7 to find out what's going on.

Listing 7. store.isItemLoaded

isItemLoaded: function(/* anything */ something){ // Currently we don't have any state that tells if an item is loaded or not // if the item exists it's also loaded. // This might change when we start working with refs inside items ... return this.isItem(something); }

The comments in this method provide the explanation. QueryReadStore does not support partially loading items at this time.

Customize QueryReadStore to support partial loading

In this section, you'll customize QueryReadStore and TreeGrid to implement partial loading and fix regression bugs.

You can download all the customized code. It's a web project, run in Eclipse, that's easily configured.

Create CustomQueryStore

To make the expand function of tree grid work with QueryReadStore , you must customize QueryReadStore to make it work with a partially loaded item. First, you must extend the class to add the missing methods. As shown in Listing 8, simply subclass QueryReadStore to CustomQueryStore .

Listing 8. Subclass QueryReadStore

/* treegrid-demo\WebContent\script\dojo-1.4.3\demo\data\CustomQueryStore.js */ dojo.provide("demo.data.CustomQueryStore"); dojo.require("dojox.data.QueryReadStore"); dojo.declare("demo.data.CustomQueryStore", dojox.data.QueryReadStore, { /* @Override */ isItemLoaded: function(/* anything */ something) { // TODO } });

The next step is to change the rule for checking if an item has been loaded, as shown in Listing 9.

Listing 9. isItemLoaded method

/* treegrid-demo\WebContent\script\dojo-1.4.3\demo\grid\CustomTreeGrid.js */ /* @Override isItemLoaded method */ isItemLoaded: function(/* anything */ something) { // Currently we have item["children"] as a state that tells if an item is // loaded or not. // if item["children"] === true, means the item is not loaded. var isLoaded = false; if (this.isItem(something)) { var children = this.getValue(something, "children"); if (children === true) { // need to lazy loading children isLoaded = false; } else { isLoaded = true; } } return isLoaded; }

QueryReadStore does not have a loadItem method, so you will create it. This method will request a server for an array that lists all the children of the item.

Listing 10. Override loadItem, getValues, and setValues

/* treegrid-demo\WebContent\script\dojo-1.4.3\demo\grid\CustomTreeGrid.js */ /* @Override loadItem method */ loadItem: function(/* object */ args) { if (this.isItemLoaded(args.item)) { return; } var item = args.item; var scope = args.scope || dojo.global; var sort = args.sort || null; var onItem = args.onItem; var onError = args.onError; if (dojo.isArray(item)) { item = item[0]; } // load children var children = this.getValue(item, "children"); // load children if (children === true) { var serverQuery = {}; // "parent" param var itemId = this.getValue(item, "id"); serverQuery["parent"] = itemId; // "sort" param if (sort) { var attribute = sort.attribute; var descending = sort.descending; serverQuery["sort"] = (descending ? "-" : "") + attribute; } // ajax request var _self = this; var xhrData = { url: this.url, handleAs: "json", content: serverQuery }; var xhrFunc = (this.requestMethod.toLowerCase() === "post") ? dojo.xhrPost : dojo.xhrGet; var deferred = xhrFunc(xhrData); // onError callback deferred.addErrback(function(error) { if (args.onError) { args.onError.call(scope, error); } }); // onLoad callback deferred.addCallback(function(data) { if (!data) { return; } if (dojo.isArray(data)) { var children = data; var parentItemId = itemId; var childItems = []; dojo.forEach(children, function(childData) { // build child item var childItem = {}; childItem.i = childData; childItem.r = this; childItems.push(childItem); }, _self); _self.setValue(item, "children", childItems); } if (args.onItem) { args.onItem.call(scope, item); } }); } } /* @Override geValues method */ getValues: function(item, attribute) { // summary: // See dojo.data.api.Read.getValues() this._assertIsItem(item); if (this.hasAttribute(item, attribute)) { return item.i[attribute] || []; } return []; // Array } /* @Override seValue method */ setValue: function(/* item */ item, /* attribute-name-string */ attribute, /* almost anything */ value) { // summary: See dojo.data.api.Write.set() // Check for valid arguments this._assertIsItem(item); this._assert(dojo.isString(attribute)); this._assert(typeof value !== "undefined"); var success = false; var _item = item.i; _item[attribute] = value; success = true; return success; // boolean }

Now the example has a new CustomQueryStore . Replace the QueryReadStore in the JavaScript, as shown in Listing 11.

Listing 11. Use CustomQueryStore to send a request

var url = "./servlet/QueryStoreServlet"; var store = new demo.data.CustomReadStore({ url: url, requestMethod: "get" });

The code returns the result shown in Figure 5.

Figure 5. Returned result by customQueryStore

Customize TreeGrid to fix regression bugs

It's time to check on other features to make sure there are no regression bugs after applying the CustomQueryStore .

Expand a row first, then click on the Name header to perform the sorting. As shown in Figure 6, an error occurs.

Figure 6. Error occurs when sorting CustomQueryStore

After investigating, you find that the tree grid would always hold the states of expando functions. Before the sorting action, the fourth row is expanded and the tree grid remembered it. After the sorting action, the items were different, but the tree grid would still try to open the expando function of the fourth row. An error occurred because now the item for the fourth row does not have any children. You need to clear the expando states when the tree grid performs a sorting query.

Just as you customized QueryReadStore , you need to customize the TreeGrid . Then, using the CustomTreeGrid to replace TreeGrid should fix the sorting defect.

Listing 12. Customize TreeGrid

dojo.provide("demo.grid.CustomTreeGrid"); dojo.require("dojox.grid.TreeGrid"); dojo.declare("demo.grid.CustomTreeGrid", dojox.grid.TreeGrid, { /* @Override */ sort: function() { this.closeExpando(); this.inherited(arguments); }, closeExpando: function(identities) { if (identities) { if (dojo.isArray(identities)) { // close multiple expando dojo.forEach(identities, function(identity) { this._closeExpando(identity); }, this); } else { // close single expando var identity = identities; this._closeExpando(identity); } } else { // close all expando var expandoCell = this.getCell(this.expandoCell); for (var identity in expandoCell.openStates) { this._closeExpando(identity); } } }, _closeExpando: function(identity) { var expandoCell = this.getCell(this.expandoCell); if (expandoCell.openStates.hasOwnProperty(identity) === true) { var open = expandoCell.openStates[identity] || false; if (open === true) { // clean up expando cache this._cleanupExpandoCache(null, identity, null); } } } });

Together, the TreeGrid and QueryReadStore are a powerful combination for lazy loading data. Large, extensive hierarchical data can be displayed without large, upfront data transfers. You can leverage QueryReadStore 's partial loading support to perform lazy loading with a single request per expansion.

Performance comparison

For this article, we compared the TreeGrid performance using ItemFileReadStore versus QueryReadStore . Figure 7 and Table 1 show that the customized QueryStore and grid uses only about 1/30 of the time used by ItemFileReadStore and basic TreeGrid .

Figure 7. Performance comparison between ItemFileReadStore and CustomQueryStore

Table 1. Performance comparison

Used data store Server data load times (s) Grid rending time (s) Total ItemFileReadStore 1.45 12.65 14.1 CustomQueryStore 0.14 0.37 0.51

Summary

In this article, you learned how to create a customized solution to accelerate the performance for Dojo TreeGrid to load large data. You explored how QueryReadStore , TreeModel , and TreeGrid work together to fetch and render data. Customizing QueryStore and TreeGrid can solve the problem of Dojo currently not supporting large data very well. Results in the article showed that the performance of the solution is about 1/30th of the time that ItemFileReadStore and TreeGrid take.

Downloadable resources

Related topics