Welcome to Part Three of our guide to Node.js REST clients. This third and final blogpost will wrap up the series with a look at topics like querying, caching, API authentication, and lessons learned the hard way.

If you haven’t already, please start with Part One, the RESTful principles important for REST clients. Or skip to Part Two on building the client’s Public API and designing a component-based architecture.

Queries

Robust querying support reduces the effort required to use your API, especially if you implemented it with familiar conventions.

For instance, say your client user needs to interact with a particular collection resource…hopefully a plausible scenario! They would probably write something along these lines:

account.getGroups(function(err,groups) { ... callback logic ... }); 1 2 3 4 account . getGroups ( function ( err , groups ) { . . . callback logic . . . } ) ;

Assuming the groups are not in cache, a request needs to be automatically sent to the server to obtain the account’s groups, for example:

GET https://api.stormpath.com/v1/accounts/a1b2c3/groups 1 2 GET https : //api.stormpath.com/v1/accounts/a1b2c3/groups

This is a great start, but ultimately a limited one. What happens the client user wants to specify search parameters? We need more powerful request query capabilities.

Query Parameters with Object Literals

An improvement is to accept an object literal to populate query parameters. This is a common technique in Node.js that specifies query parameters or skips the optional object and passes in the proceeding callback function.

account.getGroups({ name: ‘foo*’, description: ‘*test*’, orderBy: ‘name desc’, limit: 100 }, function onResult(err, groups) { ... }); 1 2 3 4 5 6 7 8 9 account . getGroups ( { name : ‘ foo* ’ , description : ‘* test* ’ , orderBy : ‘ name desc ’ , limit : 100 } , function onResult ( err , groups ) { . . . } ) ;

This would result in the following HTTP request:

GET https://api.stormpath.com/v1/accounts/a1b2c3/groups?name=foo*&description=*test*&orderBy=name%20desc&limit=100 1 2 GET https : //api.stormpath.com/v1/accounts/a1b2c3/groups?name=foo*&description=*test*&orderBy=name%20desc&limit=100

The client simply takes the name/value pairs on the object and translates them to url-encoded query parameters. We accept an object literal rather than query strings because no one wants to mess with URL encoding. This is a perfect place to offload work from your client user to the client.

Fluent API Queries

While the above approach is nice and convenient, it still requires some specific knowledge of the API’s specific query syntax. In the above example, to find any group that starts with the name foo , you need to know to use a wildcard matching asterisk at the end of the search term, i.e.

account.getGroups({ name: 'foo*', // for comparison, in SQL, this looks like // 'where name like foo%' ... etc ... 1 2 3 4 5 account . getGroups ( { name : 'foo*' , // for comparison, in SQL, this looks like // 'where name like foo%' . . . etc . . .

While it is easy enough to learn these syntax additions, it is even easier for client users if they have a Fluent API to help them construct queries in a fully explicit and self-documenting way. IDEs with intelligent auto-completion can even help tell you what methods are available while writing your query, helping you write queries much faster!

Consider the following example. The resulting query to the server is no different than the one above that used an object literal, but the query author now does not need to know any syntax-specific modifiers:

account.getGroups().where() .name().startsWith(“foo”) .description().contains(“test”) .orderBy(“name”).desc() .limitTo(100) .execute(function onResult(err, result) { ... handle result ... }); 1 2 3 4 5 6 7 8 9 account . getGroups ( ) . where ( ) . name ( ) . startsWith ( “ foo ” ) . description ( ) . contains ( “ test ” ) . orderBy ( “ name ” ) . desc ( ) . limitTo ( 100 ) . execute ( function onResult ( err , result ) { . . . handle result . . . } ) ;

As expected, this results in the same HTTP request:

GET https://api.stormpath.com/v1/accounts/a1b2c3/groups?name=foo*&description=*test*&orderBy=name%20desc&limit=100 1 2 GET https : //api.stormpath.com/v1/accounts/a1b2c3/groups?name=foo*&description=*test*&orderBy=name%20desc&limit=100

As you can see, the query author just chains method calls and then the client implementation constructs the relevant query parameter map and executes the request.

In addition to being easier to read and perhaps better self-documenting, this approach has some other very compelling benefits:

Because functions represent known queryable properties, it is not possible to specify query criteria that is not supported as a function.

Similarly, all possible functionality, like ordering and size limiting, are discoverable just by inspecting functions and code – there is less of a need to read external documentation! This is a huge help to those who just want to write queries quickly and move on – definitely a nice usability benefit.

It is much harder, if not impossible, to form a syntactically incorrect query. Query authors will likely ‘get it right’ the first time they write a query, thereby reducing bugs in their own code and increasing their overall happiness with your library.

There is, of course, a downside to supporting a fluent API for querying: implementation effort. It definitely requires a little more time and care to develop a builder/chaining API that client users can interact with easily. However, because of the self-documenting and syntatic checking nature, we feel fluent APIs are one of those features that really takes your library to the next level and can only make users happier.

Even with this downside though, there is one side benefit: when you’re ready to add a fluent query API to your library, your implementation can build directly on top of the object literal query capability described above. This means you can build and release your library in an iterative fashion: build in the object literal query support first, ensure that works, and then release your library.

When you’re ready, you can create a fluent query implementation that just generates the same exact object literals that a client user could have specified. This means you’re building on top of something that already works – there is no need to re-write another query mechanism from scratch, saving you time.

Caching

Our SDK utilizes a CacheManager , a component used by the DataStore to cache results that come back from the server. While the cache manager itself is a very simple concept, caching is extremely important for performance and efficiency. This is especially true for clients that communicate with REST servers over the public Internet. Cache if you can!

The CacheManager reflects a pluggable design. Users can configure the client to use the default in-memory implementation, or they can configure (plug in) out-of-the-box support for Memcache or Redis. This is a big benefit to many production-grade apps that run on more than one web server. Additionally, the CacheManager API itself is very simple; you can also implement and plug in new ones easily if the existing three implementations do not suit your needs.

Regardless of the specific CacheManager implementation selected, the client instance can access one or more Cache objects managed by the CacheManager . Typically, each Cache object represents a single region in the overall cache memory space (or cache cluster) where data can be stored. Each cache region typically has a specific Time-To-Live and Time-To-Idle setting that applies to all data records in that cache region.

Because of this per-region capability, the Stormpath SDK stores resource instances in a region by type. All Accounts are in one region, all Groups in another, etc. This allows the client user to configure caching policies for each data type as they prefer, based on their application’s data consistency needs.

So, how does this work internally in the Client?

Because of RESTful philosophies (covered in Part One of this blog series), every resource should have a globally-unique canonical HREF that uniquely identifies it among all others. Because of this canonical and unique nature, that means a resource’s HREF is a perfect candidate for cache key. Cache entries under a href key will never collide, so we’ll use the HREF as the cache key.

This allows a client DataStore to obtain a cached version of a resource before attempting to send a REST API server HTTP request. More importantly, the DataStore should be able to get an element out of the cache by passing in a resource HREF. In the example below, the callback function takes an error or the raw object stored in the cache.

var cache = cacheManager.getCache(regionName); cache.ttl //time to live cache.tti //time to idle cache.get(href, function(err, obj) { ... }); 1 2 3 4 5 6 7 8 var cache = cacheManager . getCache ( regionName ) ; cache . ttl //time to live cache . tti //time to idle cache . get ( href , function ( err , obj ) { . . . } ) ;

At Stormpath, we only store name/value pairs in the cache and nothing else. The data in our cache is the same stuff sent to and from the server.

client.getAccount(href, function(err, acct) {...}); // in the DataStore: var cache = cacheManager.getCache(‘accounts’); cache.get(href, function(err, entry) { if (err) return callback(err); if (entry) { ... omitted for brevity ... return callback(entry.value); } //otherwise, cache miss – execute a request: requestExecutor.get(href, function(err, body) { //1. cache body //2. convert to Resource instance //3. invoke callback w/ instance } } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 client . getAccount ( href , function ( err , acct ) { . . . } ) ; // in the DataStore: var cache = cacheManager . getCache ( ‘ accounts ’ ) ; cache . get ( href , function ( err , entry ) { if ( err ) return callback ( err ) ; if ( entry ) { . . . omitted for brevity . . . return callback ( entry . value ) ; } //otherwise, cache miss – execute a request: requestExecutor . get ( href , function ( err , body ) { //1. cache body //2. convert to Resource instance //3. invoke callback w/ instance } }

When getAccount is called, the client first interacts with cacheManager to get the cache region and then requests the cached object. If the object is found, it executes the callback. If the object isn’t found, it executes a request to the server using requestExecutor .

The fallback logic is fairly straightforward: Check the cache before issuing a request to the server. The beautiful thing is that your client users don’t have to change any of their code – they can just navigate the object tree regardless of whether the data is held in cache.

Recursive Caching

Recursive caching is, in a word, important.

When you request a resource from the server, you can use something called reference expansion or link expansion to not only obtain the desired resource, but any of its linked/referenced resources as well.

This means any expanded response JSON representation can be an object graph: the requested resource plus any other nested JSON objects for any linked resources. Each one of these resources needs to be cached independently (again, keyed based on each resource’s href).

Recursive caching is, then, basically walking this resource object graph, and for each resource found, caching that resource by its href key. Our implementation of this ‘graph walk’ utilizes recursion because it was simpler for us to implement and it’s fairly elegant. This is why it is called recursive caching. If we implemented it via iteration instead, I suppose we would have called it iterative caching.

Authentication

Secure authentication is near and dear to our hearts at Stormpath, which is why we recommend defaulting API authentication in your client to a digest-based scheme instead of HTTP basic. Although basic authentication is the simplest form of HTTP authentication, it is probably the one with the most security risks:

The secret (‘password’) is sent in an HTTP header essentially in plain text. This means HTTP basic can very easily expose raw passwords to potential attackers in any part of infrastructure that does not use TLS (previously known as SSL).

HTTP Basic authentication allows embedding the id and password in the URL, and this is never a good idea: URLs are logged all the time for analytics and reporting, exposing the secret password to anyone.

HTTP Basic authentication does not prevent Man-in-the-Middle (MitM) attacks at any point before or after network transmission. This is true even if you’re using TLS.

Because of these perils, we only advocate supporting HTTP Basic authentication if you have no other choice, and instead use what is known as a Digest-based authentication scheme. While digest authentication schemes are out of scope of this blogpost, OAuth 1.0a and Stormpath’s sauthc algorithm are good, very secure examples.

That being said, clients should offer basic authentication as an optional strategy for environments where digest schemes are incompatible. For instance, Google App Engine manipulates HTTP request object headers before sending requests out on the wire – the exact behavior that digest algorithms protect against. Our original clients didn’t work on GAE for this reason until we implemented optional basic auth.

Note: Basic authentication can only ever be implemented over TLS. It’s never okay to use basic without TLS, it’s simply too easy to find the raw password value.

Because of the need for this occasional customization, clients can be configured to specify an alternative Authentication Scheme as a constant object. For example:

AuthenticationScheme.SAUTHC1

AuthenticationScheme.Basic

AuthenticationScheme.OAUTH10a

… etc …

For example, if a client user needed to use BASIC authentication:

var client = new Stormpath.Client({ authcScheme: ‘basic’ //defaults to 'sauthc1' otherwise }); 1 2 3 4 var client = new Stormpath . Client ( { authcScheme : ‘ basic ’ //defaults to 'sauthc1' otherwise } ) ;

If they don’t specify a config value, default to the most secure option your API server supports.

Plugins

A well-defined API permits the client to support plugins/extensions without the support of type safety or interfaces. Duck typing helps too.

For example, Stormpath’s RequestExecutor relies on the Request.js module (as do many other Node.js applications). However, if anyone wanted to modify our client to use a different HTTP request library, they could implement an object with the same functions and signatures of the RequestExecutor API and just plug it in to the client configuration.

This flexibility becomes important as your client supports a broader variety of environments and applications.

Promises and Async.js

Callback Hell: The bane of Node.js library maintainers and end-users alike. Luckily, there are a couple good options to keep your Node.js code readable. For instance, Promises promote traceability even in async environments executing concurrently.

var promise = account.getGroups(); promise.then(function() { //called on success }, function() { //called on error }, function() { //called during progress }); 1 2 3 4 5 6 7 8 9 10 var promise = account . getGroups ( ) ; promise . then ( function ( ) { //called on success } , function ( ) { //called on error } , function ( ) { //called during progress } ) ;

However, excessive use of Promises has been linked to degraded application performance – use them in a calculated fashion. Since Promises are a fairly new design approach in Node.js, most Node.js applications today use callbacks instead. If you want to stick with the dominant approach, and still avoid highly-nested functions, take a look at the fantastic async.js module.

Check out this waterfall style control flow!

async.waterfall([ function(callback) { callback(null, 'one', 'two'); }, function(arg1, arg2, callback) { // arg1 now equals 'one' and arg2 now equals 'two' callback(null, 'three'); }, function(arg1, callback) { // arg1 now equals 'three' callback(null, 'done'); }], function (err, result) { // result now equals 'done' } ); 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 async . waterfall ( [ function ( callback ) { callback ( null , 'one' , 'two' ) ; } , function ( arg1 , arg2 , callback ) { // arg1 now equals 'one' and arg2 now equals 'two' callback ( null , 'three' ) ; } , function ( arg1 , callback ) { // arg1 now equals 'three' callback ( null , 'done' ) ; } ] , function ( err , result ) { // result now equals 'done' } ) ;

This code is readable – it looks like more readable imperative-style programming code, but because async.js is managing invocation, the code is still asynchronous and conforms to Node.js performance best practices.

As we mentioned previously in Part One, all of the Stormpath SDK Collections inherit async.js iterator functions so you can use collections in the same way. Convenient!

Stormpath Node.js Client

The Stormpath Node.js client is Apache licensed and 100% open source. Now that you’ve gotten an idea for how we built ours, try cloning it for more real-world examples.

$ git clone https://github.com/stormpath/stormpath-sdk-node.git $ cd Stormpath-sdk-node $ npm install $ grunt 1 2 3 4 5 6 $ git clone https : //github.com/stormpath/stormpath-sdk-node.git $ cd Stormpath - sdk - node $ npm install $ grunt

API Management With Stormpath

Stormpath makes it easy to manage your API keys and authenticate developers to your API service. Learn more in our API Key Management Guide and try it for free!

Happy coding!