Click buttons — receive browser training

Might take a second to load that large image and parse it

As you press the buttons in the website, info will be populated to TFVIS, which stands for TensorFlow Visor. It’s basically a small slide-over menu that helps us show info as we train.

On the visor, we’ll see a random array of 42 hands from the dataset shaved off as test data. The data is in RGB, but if we open constants.js , we can move the number of channels down to 1 and just work in black and white if we’d like.

42 Random hands loaded and displayed in vis

Additionally, we’ll see the model layers, untrained results, training stats, and trained results in this side-menu.

The model you create:

I’ve got a simple one that’s perfect for this simple data, and an advanced one that would be more useful with real-world photos of fingers from many angles and backgrounds.

Teachable moment

You might be thinking: “Let’s just always use the most advanced model”, which is a common pitfall. If you select the advanced model, you’ll see several things. Firstly, it takes longer to train, and second, it doesn’t even do as well. One more thing you can notice is that if you train for too long, the advanced model will start to overfit the training data.

Overfitting means that the model gets SO GOOD at making predictions on the training data, that it’s far too advanced and, hence, terrible with data it’s never seen.

A good machine learning model can generalize.

^ overfitting with an overly complex equation.

While training the model, you’ll get a graph that gets updated every batch (in this case every 512 images), and another graph that gets updated once each epoch (all 2100 training images).

A healthy training round means the loss goes down, and the accuracy goes up, i.e.

The orange line in the accuracy graph is a representation of the validation data; i.e., how accurate was the model with the 420 remaining images that the model is NOT trained on. Seeing that orange line ride alongside the training accuracy is great! Our model should generalize well (as long as the new images are similar in complexity and style).

If we click “Check Model After Training” we can see some healthy results!

As you can see, Scissors is always accurate, and our worst class is Paper, which is only 95% accurate.

Your results will differ slightly since training is randomly initialized.

To dig a little further, we get a confusion matrix, too.

a healthy confusion matrix!

As you can see here, Paper has accidentally been categorized as Scissors 6 times. That makes sense—sometimes paper can look a bit like scissors. We can accept the results…or train more! Confusion matrices like the above help us figure out what kind of problems we need to fix.

Now we can finally test our model in the real world. Using a webcam, we can check our own hands for rock-paper-scissors! Keep in mind, we’ll want our photos to appear similar to our training images in order to have any reasonable functionality (no rotation, and a white background).