qbp - queue, batch, process

qbp offers a lot of options and featues around asychronous processing.

Reach out on Twitter or GitHub. Let me know if you like it or have questions.

Contents

Usage

await qbp ( items , ( item ) => each ( item ) ) ;

This is the core of qbp's functionality. It will loop through items and will concurrently pass every item to your each function and await its completion.

qbp returns the queue object (which is explained below), completed which is an array of successfully completed items, and errors which is an array of objects with an error attribute and an item attribute. These are items that threw an error in the each function. The error attribute is the error that was thrown.

var { queue , completed , errors } = await qbp ( items , ( item ) => each ( item ) ) ;

You can obviously forego await for then() .

qbp ( items , ( item ) => each ( item ) ) . then ( ( { queue , completed , errors } ) => nextStep ( ) ) ;

The constructor breaks down like this:

qbp ( [ array , ] [ function ,] [options])

Pro Tip: The way I'm parsing parameters allows you to enter these in any order. It just looks for an array, a function, and an object. You can also choose not to add any of the parameters.

Queue Object

The queue object is returned by qbp as well as passed to all of the functions that qbp calls. The queue lets you interact with the process as it is running. Here are the attributes and functions of the queue object.

You can set a name option when calling qbp . If you set the name option, this will have that value. This can be useful when you have multiple queues you've created and maybe some of them are using the same progress function, for example.

You can call this function if you want to remove all items yet to be processed from the queue.

qbp ( ( ... args ) => each ( ... args ) , { error : ( ... args ) => handleError ( ... args ) } ) function handleError ( err , item , { queue , setStatus } ) => { queue . empty ( ) }

Call this function to halt processing without removing items from the queue like queue.empty() does.

Start processing the queue again after being paused.

Pass this function one or an array of items to add them to the queue. You can await this function to wait for the items to finish processing before continuing. Keep in mind that the function expects an array passed in to be an array of items to be processed. If the item you want processed is an array, you would want to wrap it in another array before adding it.

var { queue } = qbp ( ( ... args ) => each ( ... args ) ) async function processNewItems ( newItems ) { var { queue , completed , errors } = await queue . add ( newItems ) for ( var i = 0 ; i < completed . length ; i ++ ) { var successfulItem = completed [ i ] } for ( var i = 0 ; i < errors . length ; i ++ ) { var { error , item } = errors [ i ] } }

You can pass or even change the each function for the queue at any time by calling queue.each() .

var { queue } = qbp ( items ) var { completed , errors } = await queue . each ( ( item ) => { } )

Pass this function an integer to set how many threads you want the queue to process simultaneously.

You can tell the queue to start rate limiting at any time by calling this function and passing in your rate limit options. See Rate Limiting.

Call this to stop rate limiting.

This is an object containing attributes: items for a total count of items added to the queue, complete for total number of items completed, threads for now many threads are currently running simultaenously, queued for how many items have yet to be processed, and batch which is the current batch size.

Progress Updates

You can setup a progress function to get called to inform your users how the processing is going.

function start ( ) { await qbp ( items , ( ... args ) => each ( ... args ) , { progress : ( ... args ) => progressUpdate ( ... args ) } ) } function progressUpdate ( update ) { var { total , complete , queued , percent , threads , itemsPerSecond , secondsRemaining , batch , dateTime , statuses , queue } = update }

With this configuration the progress function will be called basically everytime there is a change. But if you would rather just get updates on a regular interval, you can set the progressInterval option.

await qbp ( items , ( ... args ) => each ( ... args ) , { progress : ( ... args ) => progressUpdate ( ... args ) , progressInterval : 5000 } )

This will have the progress function called every 5 seconds. It will also be called when the queue is empty regardless of the interval.

Throttling

There are a few options for limiting how many items can be processed simultaenously. The simpliest is by limiting the number of threads that can run concurrently.

await qbp ( items , ( ... args ) => each ( ... args ) , { threads : 2 } )

By default, qbp will process all of your items concurrently. But if that's too much all at once, setting the threads option will limit how many items are processed concurrently, in this case only 2 items will be processed at the same time.

Rate Limiting

If you need to limit processing your items by a number of items within a timeframe then this should help. Rate limiting adjusts the number of threads running simultaneously and makes sure each item takes a minimum amount of time to process in order to meet limit expectations.

In the options object you pass to qbp you can set rateLimit which is the maximum number of items allowed to process within the a time frame, rateLimitSeconds which is that timeframe in seconds, rateUpdate which is a function that can get called with updates every time the rate limit check runs, and rateFidelity which is how often you want the rate limiting check to run within rateLimitSeconds . So a fidelity of 16 (which is the default value) in with a rateLimitSeconds of 3600 would check and adjust the rate limiting every 225 seconds, running 16 times in 3600 seconds.

At any time you can programmatically add rate limiting to a queue by calling queue.rateLimit() and passing in the rateLimit , rateLimitSeconds , and rateFidelity values.

Also, at any time you can stop rate limiting by calling queue.stopRateLimit() .

Important note! Rate limiting hijacks your threads setting and automatically adjusts it to get the most items processed while remaining within your limits.

var { queue } = qbp ( ( item ) => each ( item ) , { rateLimit : 500 , rateLimitSeconds : 100 , rateFidelity : 20 , rateLimitOnEmpty : true , rateUpdate : ( update ) => rateUpdate ( upate ) , empty : ( ) => itemsEmpty ( ) } ) ; async function processNewItems ( newItems ) { queue . add ( newItems ) ; } async function each ( item ) { } function rateUpdate ( update ) { var { queue , projectedCount , projectedRate , minimumThreadTime , currentThreads , targetThreads , currentRatePerSecond , currentThreadRate } = update } function itemsEmpty ( ) { }

Batching

Need to get a few items at a time? Use the batch option.

async function start ( ) { var { queue } = await qbp ( items , ( batch ) => each ( batch ) , { batch : 5 } ) ; } async function each ( batch ) { var ids = [ ] ; for ( var i = 0 ; i < batch . length ; i ++ ) { ids . push ( batch [ i ] . id ) ; } await httpSvc . getDetails ( ids ) ; }

Item Statuses

This is a meant to be a system for easily creating a queued process that runs in the background but will provide updates as it progresses. Think of a resource intensive process, like crawling a website. You want to limit how many of these scans can happen simultaneously, and maybe you want to display a queue of the user's requests that updates as their requests progress.

var { queue } = qbp . ( ( ... args ) => handleItem ( ... args ) , { threads : 5 , progress : ( ... args ) => progressUpdates ( ... args ) } ) async function processItem ( newItem ) { await queue . add ( newItem ) } async function handleItem ( item , { setStatus } ) { setStatus ( { name : item . name , process : ' Preparing ' , time : new Date ( ) } ) await _db . insert ( item ) if ( checkSomething ) { await migrateItem ( item ) } else { await doSomethingElse ( item ) } } async function migrateItem ( item ) { setStatus ( { name : item . name , process : ' Migrating ' , time : new Date ( ) } ) await _db . move ( item ) } function progressUpdate ( { statuses } ) { console . log ( ' Current Item Statuses: ' ) for ( var i = 0 ; i < statuses . length ; i ++ ) { var { stage , status , item } = statuses [ i ] } }

Mixing

I found myself nesting queues whenever I needed to loop through multiple arrays. So I added a mix function to help with this. Here's what I was doing.

await qbp ( teachers , ( teacher ) => { await qbp ( classRooms , ( classRoom ) => { await qbp ( students , ( student ) => { await addStudent ( teacher , classRoom , student ) ; } ) ; } ) ; } ) ; async function addStudent ( teacher , classRoom , student ) { }

So instead, now we can use the qbp.mix() function.

await qbp . mix ( [ teachers , classRooms , students ] , ( ... args ) => addStudent ( ... args ) , { threads : 5 } ) ; async function addStudent ( teacher , classRoom , student , { queue } ) { }

So if you had an arrays such as:

var teachers = [ ' Mrs. Robinson ' , ' Mr. Knox ' , ' Mr. Anderson ' ] ; var classRooms = [ 102 , 203 ] ; var students = [ ' Billy ' , ' Jane ' ] ;

The each function would get called with every combination of those.

async function each ( teacher , classRoom , student , { queue } ) { }

One thing to keep in mind. You only have one queue object using this. The queue.add() function won't perform the mixing functionality that you get when you pass it in to the mix() function. But if you don't need to add any more items while processing, then this works perfect.

Error Handling

You can provide an error function in qbp's options to be called if any items throw an error during their processing.

await qbp ( items , ( item ) => each ( item ) , { error : ( ... errInfo ) => onError ( ... errInfo ) } ) ; functon onError ( error , item , { queue , setStatus } ) { }

Whenever you add items to the queue, whether it is when creating the queue or adding them with queue.add() you will recieve an array of items that completed succesfully and an array of items that had errors.

var { queue , completed , errors } = await qbp ( items , ( ... args ) => each ( ... args ) ) for ( var i = 0 ; i < completed . length ; i ++ ) { var successfulItem = completed [ i ] } for ( var i = 0 ; i < errors . length ; i ++ ) { var { error , item } = errors [ i ] }

Empty Function

You can set an empty option as a function when calling qbp and it will get called whenever the queue has finished processing all of the items it has been given.

var { queue } = await qbp ( items , ( ... args ) => each ( ... args ) , { empty : ( ... args ) => onEmpty ( ... args ) } ) function onEmpty ( queue ) { }

Testing