Classified Image using SSD Coco Model (Yes! This is my truck.)

TL;DR: In this article we will learn 3 things (these are the struggles I had to endure while creating my project for Github):

How to upload large files into a GitHub project using git-lfs (Git Large File System). How to create a Node CLI (command-line Interface) How to do image classification with a Deep Neural Network

I am creating a section for each area, so read them all, or read the one you are just interested in.

Back Story

Before we begin, a bit of information how all of this came about. Where I work, we use in-house built cameras to do analytics (like detecting oil or gas leaks). When an alert occurs, snapshots are taken from the MPEG stream. Another programmer on my team then wrote Python code to run classification on this snapshot. I was curious if the same thing could be done with Node. I have never worked with Neural Networks before, so this was going to be challenging. I started off using tensorflow.js, but I had issues trying to convert our existing model into a “web-friendly” model as required by the tfjs-node package.

Then I found this excellent article “Node.js meets OpenCV’s Deep Neural Networks — Fun with Tensorflow and Caffe” on Medium written by Vincent Mühler. It introduced me to OpenCV via the node package opencv4nodejs. From there things started working with better results.

After I got my package all put together with a nice readme markdown, I had issues getting the project onto Github. The model files were too large! Then I learned about git-lfs (Git Large File System). A couple of days struggling with this (on limited bandwidth at the time — I was out camping), I got that to work. Then came npm (sigh) issues. I tried npm publish, but after packaging, the upload to the registry failed with “javascript heap out of memory”. Again, it was altogether too large!

I still haven’t got it on the npm registry. I need to explore different ways. If you’ve conquered this task with a large package, feel free to let me know how you did it.

Github and Large Files

First of all, GitHub has some limitations. From their documentation, “we place a strict limit of files exceeding 100 MB in size”. Well, then, this won’t work when the models are definitely bigger than that.

Enter git-lfs. This little gem allows you to track large files in git and GitHub. While free, it has limitations where GitHub will start charging if you exceed them. And, git-lfs also works with Gitkraken — bonus!

LFS Support in GitKraken

Notice the LFS at the end of the files? Perfect.

OK, so it didn’t all go so smoothly. Before you put large files into your repository, you have to initialise git-lfs and tell it what kind of files to track for your project. Read up on it here.

Creating a CLI with Node

I’m sure you’ve heard of CLI — Command-Line Interface. It allows a User to interact with a computer by means of a computer program. By creating a CLI with Node, the package can be run as if it were a natively compiled program that exists on the computer.

For instance, to run my Node package called “classify”, you would normally do the following (in the classify folder):

node index.js [arguments]

You can install the package globally into the Node ecosystem and it adds the package to the path. To install (from the classify folder) do the following:

npm install -g . classify

This installs the package in the current folder using the name “classify”.

Now you can issue, from the command-line, statements like this:

classify --image <path to image file> --filter ./filter.txt --confidence 50

CLI Output

All CLI’s should have an output so that the User can understand how to use it. In this case, “classify” will look like this:

classify Classifies an image using machine learning from passed in image path. Options --image imagePath [required] The image path.

--confidence value [optional; default 50] The minimum confidence level to use for classification (ex: 50 for 50%).

--filter filterFile [optional] A filter file used to filter out classification not wanted.

--quick [optional; default slow] Use quick classification, but may be more inaccurate.

--version Application version.

--help Print this usage guide.

Of course, there are packages to help you do that. I used command-line-usage and command-line-args to do the work for me.

But, before we get into that, just how does one run a JavaScript file using Node, without specifying Node on the command-line?

It has to do with the first line of all scripts on a Linux system. This is where the script interpreter is specified using the she-bang notation:

#!/usr/bin/env node

This tells the environment to use “node” as the interpreter of this script and should always be at the top of your JavaScript file when making a CLI.

command-line-usage

Fairly easy to use, it defines what the User will see.

Here’s the code:

Then, to output the results, the code looks like this:

console.log(usage)

command-line-args

Again, fairly easy to use. Just make sure you process and validate everything:

You’ll notice for the --version command that all we have to do is read in our package.json and output the version. That way, we only need to maintain it in one place.

The rest of the processing is checking if an option was used, if so, then validate it, etc.

Once we have collected all of the data needed for classification processing, we are ready to start the classification.

Image Classification with OpenCV

Now that we have collected and validated parameters collected from the User interaction with the CLI, the real fun can begin. The high-level processing isn’t as difficult as you might think:

Not too bad. You should notice this is where we use some of the User collected data to filter based on confidence and filter file if any.

But, the predict function is where most of the work is happening to return the predictions. And, it’s not just the predict function. There are several other supporting functions needed to extract data.

Let’s take a look at how that is done:

First of all, you have to know these models are trained. One is 300x300 and the other is 512x512. The 300x300 model is faster. There’s less data. The 512x512 is slower and overall has more accurate predictions, because there’s more data.

You will notice in the function above, that we have to resize the input image to match the size of the trained images in the model. If the image is not square, then we have to pad it. The colour white is more often used because it has less issues than, say, using black.

The image is then converted to a “blob” and passed to “net.setInput”. You will remember, we previously had this code:

// initialize model from prototxt and modelFile

let net

if (dataFile === 'coco300' || dataFile === 'coco512') {

net = cv.readNetFromCaffe(prototxt, modelFile)

}

In case you had forgotten. (Yes, I know it’s global and should have passed it in).

So, now you notice the last thing in the previous function is to extract the results:

This is when you are able to access the “index” (that relates to the classification classes), the “confidence” level of the classification, and the “rect” area of the identified object. These are our “predictions”, which are then filtered.

You may recall this bit of code from above (as a refresher):

// write updated image with new name

updateImage(imagePath, img, predictions)

The intent is to write a new image to file, with the filtered predictions, so the User can see the object and it’s confidence level. There are several functions that make this happen:

I’m not going to get into these functions as they are pretty common JavaScript (and well commented).

Caveats

You should always use some sort of filtering. Always use confidence level. I general use 50, but 30 is probably the lowest I’d go. Why? You ask? Because this is what happens:

Classification with no confidence filtering

If the image is quite “busy” you will get a lot of classifications. Most of them spurious. Most having a confidence level < 10. Try playing around with the confidence level. See what works best for you. Remember, this is the same image as the first image in this article (did I make you go back and look?)

Examples

Unclassified Train

Classified Train

Unclassified Royals

Classified Royals

Show some teeth Harry! You might have been 100% a person. All jokes aside, this has been fun. I’m still in learning mode. And there’s a devilish amount to learn. I hope what I have written helps you on your path if you have the same interest.

The complete project can be found on GitHub.