Last few months I was working on my master degree thesis, which was about Computer vision in mobile applications built with JavaScript. The problem I was working on was recognizing the trees from the photos of theirs leafs.

How did I solve this problem?

Could you solve all parts of an application in JavaScript?

Problem — Image classification

In my thesis, I have been working on recognizing the tree from the photograph of his letter. I came across a thesis about recognizing leafs, but only in a lab environment. I’ve decided to make a multiplatform mobile app that will allow you to recognize the tree in nature, outside the lab.

Computer Vision — OpenCV

The first idea was to solve this problem by using a computer vision and OpenCV library (JS interface node-opencv). During the programming of this section, I wrote an article “Computer vision in Node.js”, which is an introduction to Computer Vision in Node.js. The purpose of this solution was to find the main contour of the leaf and use Fourier descriptor methods to compare it with samples of other leafs.

Conclusion after this experiment?

It is not possible to specify a tree type or just leaf category based on a comparison of signals from FD because they are always too similar.

Neural network — Tensorflow

Neural networks are often used to classify images, so I decided to use it for my project. I chose Tensorflow from Google. To image classification, I used a pre-trained model Inception V3. For classify of our dataset, I re-train final layer of this model. Unfortunately, I did not get a suitable dataset of tree leaves that would be photographed in different out-of-laboratory environments. I decided to create my own dataset to validate my idea.

At present, the dataset consists of hundreds of photographs of few trees. While training of neural network, I get best results when I split the dataset in a 10% — validation, 10% — testing, 80% — training, and run training with 18,000 steps. After right setting of training parameters, I get my results for some photos from 73% to 85%. Average value is >= 90%. I think this is a really good value for photos with a random background! I write a Flask server with Tensorflow API handling in Python.

Architecture

This project consists of the following parts:

Mobile application — React Native

Database, File Storage, Auth — Firebase

OpenCV preprocessing, Email, etc. — Node.js server code (Heroku)

Tensorflow + Flask Server — Amazon EC2

Mobile application

I developed this mobile application in React Native. Very good decision! React Native applications have good performance, stability and development is very fast. I create a design for this app too.

On the following screenshots, you can see and understand the main parts of the application and how it works. When user photographing a leaf (screen no.4), it is necessary to follow the position of the leaf by placeholder in camera view.