About the module

pooledtokeninterceptor is a NodeJS module that uses a pool of tokens to access API endpoints thereby increasing the number of requests that can be made at a given time despite the imposed limits. The module works by assigning access tokens to request in a round robin pattern and returns a promise from each of them. The module then checks the response and resolves the promise based on the following types of calls:

'WTC' or Wait to Complete - This type of call checks the response code to determine if the limit has been exceeded. It then waits for the rate limit to be reset, makes another requests and resolves the promise on success. 'FOK' or Fill or Kill - This type of call simply distributes tokens from the supplied pool to requests in order to maximize the number of successful requests that can be made. All failed responses are returned as it is.

How to use

Get the module and Q from npm:

npm install pooledtokeninterceptor npm install q

Q is used to process the promises.

Add the module and Q:

var interceptorModule = require ( ' pooledtokeninterceptor ' ) ; var Q = require ( ' q ' ) ;

Setup a configuration object for the token pool:

var tokenConfig = { tokenLocation : ' header ' , tokenIdentifier : ' Token ' , tokens : [ ' token1 ' , ' token2 ' ] , failureResponseCode : 403 , headerLimitResetVar : ' x-ratelimit-reset ' , resetOffset : 4 , } ;

Initialize it and make a request:

var options = { url : ' https://api.github.com/search/repositories?q=Material+language:Java&sort=stars&order=desc ' , headers : { ' User-Agent ' : ' request ' } } var interceptor = interceptorModule ( tokenConfig ) ; Q . fcall ( interceptor . authenticatedRequest ( options , ' FOK ' ) ) . then ( console . log ) ;

The module is built on Request so other fancier options can also be used based on what it accepts.

The module can also be used to make multiple requests as follows:

var promisesArray = [ ] ; for ( var i = 0 ; i < 45 ; i ++ ) { promisesArray . push ( request . authenticatedRequest ( options , ' WTC ' ) ) ; } Q . all ( promisesArray ) . then ( function ( responses ) { console . log ( " Number of responses: " + responses . length ) ; for ( var i = 0 ; i < responses . length ; i ++ ) { console . log ( responses [ i ] . statusCode ) ; } } )

The above example will make 45 requests to the GitHub api. If one token is supplied, only 30 requests can ideally be made based on the imposed limits. With the 'WTC' option, the module will collect the 15 failed responses, wait for the limits to be reset and retry the requests for each of them. Promises are only resolved when all the requests have been successfully completed.