What are background processes?

When somebody tells me about background process, the picture that comes to my mind is something that is not seen by anybody, but that does some kind of work behind the scenes. In reality, it is pretty close to this definition. Background process in a web application scenario is some process which does some work away from the normal request-response loop. Some good examples of this are sending emails, generating reports, scheduling tasks, system monitoring or even sending out notifications to users as an effect of something that happened in the system.



The need for background processing

Typically in web applications, great efforts are made to maximize the responsiveness by getting only the critical data required to render the current page to the screen as fast as possible, thereby completing the user request and handing back the control to the user. Ideally, requests taking more than a second or few seconds should be cached or split and moved to a background process. Typically when a request comes in, web-server spawns a thread to handle it. As long as the request is not served, the thread remains alive and will not be returned to the thread pool. If web-server takes too long to respond, other requests will be queued, response time gradually degrades, thread pool gets exhausted and your web-server crashes.



Every web application, medium to large, requires some kind of background processing mechanisms to work smoothly. Usually, this is done via some kind of queueing mechanism. There are many message brokers available with popular ones being RabbitMQ, Kafka, ActiveMQ to name a few. Sidekiq (which is based on Redis) is used for job processing in Rails Applications. AWS also provides a message-queue service called SQS, which hardly requires any setup and is fully managed by AWS.



Options for background processing in NodeJS

Node.js is basically a good fit for background processing and there are many ways of doing it. For the people who came from Rails world, there is Node-resque. Then there is Bull and Kue, for handling jobs and messages, which makes use of Redis for queuing. We can also use RabbitMQ for message-queues, which has a wide set of configurations to match your requirements. You can read more about the options for running background tasks in NodeJS in the StackOverflow thread here.

In this article, we are going to discuss using RabbitMQ for background processing in NodeJS.



Setting up RabbitMQ

Installing RabbitMQ is the very first step to get started with background process in Node.js. In Linux Systems, you can directly use the following command to install `rabbitmq-server`:

sudo apt-get install rabbitmq-server

In MacOS the best way is to install it via Homebrew. You can do it with the following command.

brew install rabbitmq

A comprehensive download FAQ and options are given here in the RabbitMQ official download page. You can find the instructions to download the RabbitMQ installer for Windows systems also there.

After you install the rabbitmq-server, make you sure you did it correctly by typing rabbitmq-server in your terminal. In MacOS, this is what it shows in the terminal.

Rabbitmq-server

Basics of RabbitMQ

RabbitMQ is one the most popular open-source message broker. It supports multiple messaging protocols, has distributed deployment configuration support for high scalability and availability along with message queuing, delivery acknowledgment, flexible routing to queues and multiple exchange types.



The two main entities in RabbitMQ are exchanges and queues. Queues are the place where the messages are actually stored. All the messages are published on to an exchange. The queues are connected to exchanges through a binding. Bindings contain the info on how the queue is connected to the exchange. It has something called routing rule, by which the exchange decide whether to forward the message to a queue or not.





RabbitMQ flow

A queue can be bound to many exchanges with different routing rules. The exchanges can be of different types namely direct , topic,headers , and fanout . Fanout exchanges are typically used when the messages on the exchange need to be forwarded to all the queues registered in the exchange. Fanout exchange is like mindless broadcasting. In a direct exchange , a message goes to the queues whose binding key exactly matches the routing key of the message which gives a level of filtering of messages forwarded to a queue. The topic exchange is like a direct exchange, with advancements in routing, used for multicast routing of messages. You can have routing rules with partial matching of the topic name. A headers exchange is designed for routing on multiple attributes that are more easily expressed as message headers than a routing key.



Setting up a web server

First let’s quickly set up an expressjs project.

$ mkdir myapp

$ cd myapp

$ npm init

With the above commands, we created a directory called myapp , made that the current working directory and initialized an npm package. Now we will install expressjs by the following command.

npm install express --save

Now we will create a file named index.js

touch index.js

From the express tutorial here, we will make a simple web app which shows a “Hello World” message. Edit the index.js to have the following content.

Now run this web app with the following command :

node index.js

Now hit the URL http://localhost:3000 in your browser. If everything was done right, you should be seeing a Hello World! message.



Sending messages to exchange

There are a number of clients for RabbitMQ in many different languages. In this guide we use aqmp.node client in order to run node.js with express in background process. There are mainly two types of APIs that this client provides. One is callback based and the other is promise based. You can choose whichever fits your style or requirement. And in this tutorial, we will also use callback-based API.

The important step to make the most out of the tips to node.js task scheduling in background is an understanding of how to send messages to exchange. For this, we will install aqmp.node client first.

npm install amqplib

Now in the same directory, we will create a send.js file and add the following content.



Here in the above code, we load the aqmp callback_api, open a connection to RabbitMQ, create a channel to communicate with RabbitMQ, assert a default queue named hello and send a message to that queue. While asserting the queue, we pass an option { durable: true } which makes the queue to persist even after the connection is closed. While sending a message to the queue, we pass an option { persistent: true } which makes the message persist in the queue. Here we do not see the exchanges because we are making use of default exchange. You can even give the queue name as blank, in which case the RabbitMQ server will generate a random name for it.

Now run the following command to send a message to rabbitmq server.

node send.js

There is a rabbitmq-management plugin that comes with rabbitmq package. The rabbitmq-management plugin provides an HTTP-based API for management and monitoring of RabbitMQ server, along with a browser-based UI and a command line tool, rabbitmqadmin.



You can access the browser-based UI by typing in http://localhost:15672/ in your browser. After running your send.js , you can go to http://localhost:15672/#/queues . There you can see a queue named hello and total message count as one.



RabbitMQ Management UI — Queues

We have successfully sent a message to a default exchange onto a queue in RabbitMQ server. You can read more on this here.



Receiving messages from queue

We will first create a worker.js file in the current working directory with the following command.

touch worker.js

In this script, we open a connection to rabbitmq, then create a channel, then assert the queue and then consume the queue through the channel.



Modify worker.js to have the following content.



While consuming messages in a queue, in the real world we might need to have some kind of acknowledgment after processing the message so that even if the processing fails the message is not lost. In the code above we pass an option { noAck: true } to consume method, which means that the rabbitmq server won’t wait for the acknowledgment from the client side after processing. The server will delete the message from the queue as soon as it is delivered to the client. If you don’t want the server to delete message once it’s delivered, but rather you want the server to delete message from queue once you successfully processed it, you can pass the option { noAck: false } and in the code, inside consume, after you process the message, you can acknowledge it by channel.ack(msg) .



Now let’s run our worker to consume the queue by the following command.

node worker.js

You will see the following output in the terminal.







Now if you go to http://localhost:15672/#/queues you can see that the messages count in the queue is zero.



Putting it Altogether

In the above sections, we have seen how to send messages to rabbitmq and consume it. Now let’s make use of it in our web app.



Let’s chart out a basic workflow of our example website. Consider our website having an email subscription form. Users will give their email and click submit button. This should send an email to the user and display a thank you message.



Since the flow is set, let’s think of how we can do this. Sending an email is a time-consuming operation compared to other web requests. So let’s take that email sending-part out of the main request-response loop. We will send the email to the exchange and our worker will take that from the queue, process it, send the email and send an acknowledgment to the server so that server can delete the message from the queue.



We will use ejs for rendering HTML pages and nodemailer for sending emails. So let’s install those packages.

npm install ejs

npm install nodemailer

Then let’s create an HTML form to get the emails from the user which will be rendered on the landing page. Create a directory called views and file called index.html inside views directory. The content of index.html is given below.

Now we need to display the form on the landing page. For that, we modify our index.js to have the following content.

This will render the HTML form with a single input field for email and a submit button. Form submit is handled by /subscribe . On subscribe, we will send the email details to the rabbitmq queue.



Now run node index.js for the server. Make sure your rabbitmq-server is running. Go to http://localhost:3000 . You will see the following page.



Subscription form

Now type in an email and click submit. You will see a page with the message Thank you. You are successfully subscribed . . Now if you go to rabbit-mq browser UI at http://localhost:15672/#/queues you can see a new queue named email with a total message count of 1.



Email queue

Now let’s go back to our worker and make the worker consume the email queue and send an email. We will use nodemailer to send emails. Nodemailer supports ethereal.mail accounts for testing. We will leverage that for sending emails in this tutorial.



Edit worker.js to have the following content.



Now run the worker in another tab of the terminal with node worker.js . You can see the following message.



worker log

So in the above example, we have successfully isolated the email sending from the normal request-response loop. This can be extended to many other use-cases like report generation, fetching data from third party APIs, complex calculations, scheduling jobs …etc.

You can view all the files used in the project here.



Congrats — you have successfully published messages to rabbitmq-server and wrote code to implement the consumption and processing of that messages. Feel free to try the same exercise for other use-cases also.

Don’t hesitate to reach out with general feedback or with questions.