This lib is for managing a large number of tasks, by adding each task into the queue(sqlite), and digesting each item of the queue. These are the reasons for using taskm :

you got a large number of task to handle, an exception may cause the script to restart from the first

you got a large number of task to handle, and needed to know how many are pending , finished and fail

, and you got a large number of task to handle, and those tasks could be digested by more than one computer

( sqlite is used to manage queue)

Install

$ yarn add @pscraper/taskm or

$ npm install @pscraper/taskm

Example-1 (FIFO)

Find More Example: https://github.com/wahengchang/pscraper/tree/master/packages/demo/taskm

adding new task to queue (FIFO)

[Nodejs] Managing thousand of task, @pscraper/taskm

const TM = require('@pscraper/taskm')

const taskm = new TM()

await taskm.init() const uniqueId = "amazingId"

const options = {

title: 'amazingTitle',

meta: JSON.stringify({key: "key1"})

}

await taskm.add(uniqueId, options)

consuming task from queue

const TM = require('@pscraper/taskm')

const taskm = new TM()

await taskm.init() const item = await taskm.getFirst()

// do something

await taskm.markFinished(item.id)

Example-2 (Task Distribution)

adding new task to queue

const TM = require('@pscraper/taskm')

const taskm = new TM()

await taskm.init() for(let i =0 ; i<999999; i++) {

const id = `mockId${new Date().getTime()}-${i}`

const title = `${id}-title-${i}`

inputList.push({id, title})

await taskm.add(id, {title})

}

consuming task from queue, usually is run on more than one PC parallel

const TM = require('@pscraper/taskm')

const taskm = new TM()

await taskm.init() const item = await taskm.getFirstRandom()

// do something

await taskm.markFinished(item.id)

Usage

[Nodejs] Managing thousand of task, @pscraper/taskm

Reference: