Photo by Suzy Hazelwood from Pexels

The guy who wrote this is gone. The guy who wrote this is now a consultant at the company. The guy who wrote this is retired.

NPM passed a total number of 1 million unique packages last June. It is common for our backend applications to depend on the top ten packages:

lodash

request

react

express

moment

async

… Etc

And those packages are being updated every month! Who has the time to keep track of what package is running in which services in production?

As soon as you have a product deployed into the production environment and serving customers, you will have a need to stay on top of packages.

A recent example is the Amazon RDS Certificate Authority change that required most Node developers who use Database SSL connection (a good idea in general) to update their mysql packages.

It is easy enough to update the package by doing a quick scan in package.json if you only have Dev andProd environments and a backend monolith. But, what if you have microservices and more environments?

Imagine a case where you are running 10 microservices across 5 environments (Dev/Staging/Training/Demo/Production): That would quickly turn into 50 deployments/branches to poke into to look at individual npm package.json!

Can we do better in monitoring for old packages while using self-hosted solutions? You bet!

If you follow this guide, you will be able to monitor running node packages in Prometheus and Grafana, it looks like the following:

Final result in Grafana, scroll by to see which microservice is lagging behind or should be upgraded!

The SETUP

Start with your own backend project, or clone this one: git@github.com:teamzerolabs/config-service-reference.git

In the same folder with the package.json, type npm install mindseye --save , or yarn add mindseye .

, or . mindseye is a prometheus exporter, it will collect information from the current package.json file and display it in a format that prometheus can understand.

Minimum Code Change

To use mindseye , simply do the following near your express entrypoint:

import express from 'express';

const http = require('http'); ... const { collectPackageInfo } = require('mindseye');

const promClient = require('prom-client'); collectPackageInfo({ registers: [promClient.register] });

And when you start your program now, it will display the following:

/tutorials/config-service-reference/node-starthere/$ yarn start

yarn run v1.21.1

$ node main.js

Hi, we will spin up a little server that loads books from a local database

MindsEye listening on port 9991 /metrics

Book Example app listening on port 3000!

If you go to localhost:9991/metrics in Chrome, you will see this:

# HELP npm_packages_installed npm packages installed for this project

# TYPE npm_packages_installed counter

npm_packages_installed{package_type="dep",package_name="express",package_version="4.17.1"} 1

npm_packages_installed{package_type="dep",package_name="mindseye",package_version="1.0.1"} 1

npm_packages_installed{package_type="dep",package_name="mysql2",package_version="2.1.0"} 1

npm_packages_installed{package_type="dep",package_name="sequelize",package_version="5.21.3"} 1

We now have the exporter format output that can be fed into monitoring system like Prometheus!

Configuring Prometheus to read from this API Endpoint

We have covered how to setup Prometheus and Grafana locally here. But a quick re-cap on setting them up:

git clone https://github.com/stefanprodan/dockprom

cd dockprom

ADMIN_USER=admin ADMIN_PASSWORD=admin docker-compose up -d

This will setup Prometheus on your local host at localhost:9090 , Grafana dashboard will be on localhost:3000

Follow the guide above for Prometheus to scrape our exporter endpoint at 9991.

And make you are able to view npm_packages_installed in the Prometheus Dashboard:

This example is from monitoring inside a kubernetes setup, we will cover how this is setup in future guides.

Finishing it with a Grafana Table

Once the metrics is scraped and ready in Prometheus, we can create better visualization with Grafana.

Start by adding a new Panel and select the type Table

In the Query Table, type in the following:

sum by (job, package_name, package_version) (npm_packages_installed{packge_type="dep"})

We want to know which job (running program) the packages belongs to.

packge_type="dep" means we only care about the production dependencies, not developer dependencies.

means we only care about the production dependencies, not developer dependencies. Note: If you are doing the scraping inside Kubernetes, you can switch job with app like the above.

The first result shows the time column , and will generate way too many pages, we can prevent this from happening by clicking on the instant option checkbox.

To hide the time column, click on Column Style, and select the Time category and change its display type to Hidden.

That’s it! Your package monitoring table is now ready for use.

You can now perform package audits and get a bird’s eye view without needing to drill into each and every one images running in production.

If you are using prom-client to instrument node applications already, there is a way to integrate that with mindseye as well. You can read more about what options are available at the git page: https://github.com/teamzerolabs/mindseye

In Conclusion

The best time to put in package tracking metrics was 4 years ago. The second best time put in package tracking metrics is today.

Don’t let your production code become stale and unmaintainable. Take charge and start monitoring those packages!

Need to monitor your production system or setup self-hosted Prometheus+Grafana in AWS? Team Zero Labs can help, send us an inquiry at info@teamzerolabs.com