In this blog post, you will learn how to create a complete machine learning pipeline that solves the problem of telling whether or not a person in a picture is showing the teeth, we will see the main challenges that this problem imposes and tackle some common problems that will arise in the process. By using a combination of Opencv libraries for face detection along with our own convolutional neural network for teeth recognition we will create a very capable system that could handle unseen data without losing significative performance. For quick prototyping, we are going to use the the Caffe deep learning framework, but you can use other cool frameworks like TensorFlow or Keras.

At the end of this post our trained convolutional neural network will be able to detect teeth on real video with a very good precision rate!

Obama’s teeth being detected by our conv net!

The overall steps that will involve creating the teeth detector pipeline are:

Finding the correct datasets then adapting those datasets to the problem Labeling the data accordingly ( 1 for showing teeth, 0 not showing teeth) Detecting the face region in an image Detecting the principal landmarks on the face Transforming the face with the detected landmarks to have a “frontal face” transformation Slicing the relevant parts of the image Easing the data for training by applying histogram equalization

Augmenting the data

Setting up the convolutional neural network in Caffe

Training and debugging the overall system

Testing the model with unseen data

Finding a dataset

Muct database

We are going to choose an open dataset called MUCT database http://www.milbo.org/muct/, this dataset contains 3755 unlabeled faces in total, all the images were taken in the same studio with the same background but with different lighting and camera angles.

Muct database image variations, source http://www.milbo.org/muct/

Because of manual labeling constraints only a subset of the dataset called muct-a-jpg-v1.tar.gz will be used, this file contains 751 faces in total, although this is a small number for training the machine learning model, it is possible to obtain good results using data augmentation techniques combined with a powerful convolutional neural network model, the reason for choosing this limited subset of data is because at some point in the process is necessary to do manual labeling for each picture, but note that it is always encouraged to label more data to obtain better results, in fact, you could have much better results than the final model of this posts by taking some time to label much more data and re-train the model.

LFW database

To have more variety on the data we are going to use the Labeled Faces in the Wild database too http://vis-www.cs.umass.edu/lfw/, this dataset contains 13.233 images of unlabeled faces in total, this database has a lot more variety because it contains faces of people in different situations all the images are gathered directly from the web. Similarly, for the LFW database, we are not going to use only 1505 faces for training.

LFW database image samples, source http://vis-www.cs.umass.edu/lfw/

Labeling the data

Labeling the data is a manual and cumbersome process but necessary, we have to label images from the two face databases, we will label all the faces with the value 1 if the face is showing the teeth or 0 otherwise, the label will be stored on the filename of each image file.

To recap: For the MUCT database, we are going to label 751 faces. For the LFW database, we are going to label 1505 faces. So we have a total of 2256 unique faces with different expressions, some of them are showing the teeth and some not.

To speed up manual labeling a bit, you can use this simple tool ImageBinaryTool for quick labeling using hotkeys, the tool will read all the images in a folder and will start asking you to put the binary value, if you push the Y key on your keyboard it will add to the existing filename the label _showingteeth and pass to the next image, if you want to use this tool for your purposes feel free to pull it from git hub and modify it to suite your needs.

Labelling images using the binary labelling tool

Note: Note that these labeled images are not our training set because we have such small data set (2256 images) we need to get rid of unnecessary noise in the images by detecting the face region by using some face detection technique.

Detecting the face region

Face detection

There are different techniques for doing face detection, the most well known and accessible are Haar Cascades and Histogram of Gradients (HOG), OpenCV offers a nice and fast implementation of Haar Cascades and Dlib offers a more precise but slower face detection algorithm with HOG. After doing some testing with both libraries I found that DLib face detection is much more precise and accurate, the Haar approach gives me a lot of false positives, the problem with Dlib face-detection is that it is slow and using it in real video data can be a pain. At the end of the exercise, we ended up using both for different kind of situations. I recommend reading the excellent post Machine Learning is fun by Adam Geitgey, most of the code shown here for face detection was based on his ideas.

By using the opencv libraries we can detect the region of the face, this is helpfull because we can discard unnecessary information and focus on our problem.

Face detection in action

Note: You can also use a convolutional neural network for face detection, in fact, you will get much better results if you do, but for simplicity, we are going to stick with these out of the box libraries.

In Python, we are going to create two files, one for OpenCV face detection and one for DLib face detection. These files will receive an input image and will return the area where the face is present.

OpenCV implementation

def mouth_detect_single ( self , image , isPath ): if isPath == True : img = cv2 . imread ( image , cv2 . IMREAD_UNCHANGED ) else : img = image img = histogram_equalization ( img ) gray_img1 = cv2 . cvtColor ( img , cv2 . COLOR_BGR2GRAY ) faces = self . face_cascade . detectMultiScale ( gray_img1 , 1.3 , 5 ) for ( x , y , w , h ) in faces : roi_gray = gray_img1 [ y : y + h , x : x + w ] eyes = self . eye_cascade . detectMultiScale ( roi_gray ) if ( len ( eyes ) > 0 ): p = x q = y r = w s = h face_region = gray_img1 [ q : q + s , p : p + r ] face_region_rect = dlib . rectangle ( long ( q ), long ( p ), long ( q + s ), long ( p + r )) rectan = dlib . rectangle ( long ( x ), long ( y ), long ( x + w ), long ( y + h )) shape = self . md_face ( img , rectan ) p2d = np . asarray ([( shape . part ( n ) . x , shape . part ( n ) . y ,) for n in range ( shape . num_parts )], np . float32 ) rawfront , symfront = self . fronter . frontalization ( img , face_region_rect , p2d ) face_hog_mouth = symfront [ 165 : 220 , 130 : 190 ] gray_img = cv2 . cvtColor ( face_hog_mouth , cv2 . COLOR_BGR2GRAY ) crop_img_resized = cv2 . resize ( gray_img , ( IMAGE_WIDTH , IMAGE_HEIGHT ), interpolation = cv2 . INTER_CUBIC ) #cv2.imwrite("../img/output_test_img/mouthdetectsingle_crop_rezized.jpg",gray_img) return crop_img_resized , rectan . left (), rectan . top (), rectan . right (), rectan . bottom () else : return None , - 1 , - 1 , - 1 , - 1

DLIB Implementation using histogram of gradients

def mouth_detect_single ( self , image , isPath ): if isPath == True : img = cv2 . imread ( image , cv2 . IMREAD_UNCHANGED ) else : img = image img = histogram_equalization ( img ) facedets = self . face_det ( img , 1 ) #Histogram of gradients if len ( facedets ) > 0 : facedet_obj = facedets [ 0 ] shape = self . md_face ( img , facedet_obj ) p2d = np . asarray ([( shape . part ( n ) . x , shape . part ( n ) . y ,) for n in range ( shape . num_parts )], np . float32 ) rawfront , symfront = self . fronter . frontalization ( img , facedet_obj , p2d ) symfront_bgr = cv2 . cvtColor ( symfront , cv2 . COLOR_RGB2BGR ) face_hog_mouth = symfront_bgr [ 165 : 220 , 130 : 190 ] #get half-bottom part if ( face_hog_mouth is not None ): gray_img = cv2 . cvtColor ( face_hog_mouth , cv2 . COLOR_BGR2GRAY ) crop_img_resized = cv2 . resize ( gray_img , ( IMAGE_WIDTH , IMAGE_HEIGHT ), interpolation = cv2 . INTER_CUBIC ) return crop_img_resized , facedet_obj . left (), facedet_obj . top (), facedet_obj . right (), facedet_obj . bottom () else : return None , - 1 , - 1 , - 1 , - 1 else : return None , - 1 , - 1 , - 1 , - 1

Landmark detection and frontalization

Faces on images can have a lot of variation, they can be rotated at certain degree or they can have different perspectives because the picture was taken at different angles and positions.

Simply detecting the face is not enough in our case because learning these multiple variations will require huge amounts of data, we need to have a standard way to see the faces this is we need to see the face always in the same position and perspective, to do this we need to extract landmarks from the face, landmarks are special points in the face that relate to specific relevant parts like the jaw, nose, mouth and eyes, with the detected face and the landmark points it is possible to warp the face image to have a frontal version of it, luckily for us landmark extraction and frontalization can be simplified a lot by using some dlib libraries.

#landmark detector shape = self . md_face ( img , facedet_obj )

md_face receives the face region and will detect 68 landmark points using a previously trained model, with the landmark data we can make a warp transformation to the face using the landmarks as a guide to make the frontalization.

Before and after landmark detection

Before and after landmark detection

to warp the face using the landmark data we use a python ported code that use the frontalization techinque proposed by al Hassner, Shai Harel, Eran Paz and Roee Enbar http://www.openu.ac.il/home/hassner/projects/frontalize/ and ported to python by Heng Yang, the complete code can be found at the end of this post:

p2d = np . asarray ([( shape . part ( n ) . x , shape . part ( n ) . y ,) for n in range ( shape . num_parts )], np . float32 ) rawfront , symfront = self . fronter . frontalization ( img , facedet_obj , p2d )

Frontalized image with image slicing

Image slicing

Now that we have frontal faces we can make a simple vertical division to discard the top face region and keep only the bottom region that contains the mouths:

rawfront , symfront = self . fronter . frontalization ( img , facedet_obj , p2d ) face_hog_mouth = symfront [ 165 : 220 , 130 : 190 ] #get half-bottom part

To generate all the mouth data you can run the script create_mouth_training_data.py

python create_mouth_training_data.py

Frontalized mouths black and white will be our training data

With those transformations in place, our net will receive inputs of the same part of the face for each image. The total output of this step will be 2256 mouths.

Histogram Equalization

A usefull technique for highlighting the details on the image is to apply histogram equalization, note that this step is already applied on create_mouth_training_data.py:

def histogram_equalization ( img ): img [:, :, 0 ] = cv2 . equalizeHist ( img [:, :, 0 ]) img [:, :, 1 ] = cv2 . equalizeHist ( img [:, :, 1 ]) img [:, :, 2 ] = cv2 . equalizeHist ( img [:, :, 2 ]) return img

Data Augmentation

As you recall, we have labeled only 751 images from the MUCT database and 1505 from the LFW database, this is just not enough data for learning to detect teeth, we need to gather more data somehow, the obvious solution is to label a couple of thousand images more, this is the ideal solution, having more data is always better but collecting it is time expensive, so for simplicity we are going to use data augmentation. We are going to make the following transformations to our set of mouth images to get almost 10x times more different images (23528 mouth images in total):

Mirroring the mouths

For each mouth image, we are going to create a mirrored clone, this will give us twice the data.

horizontal_img = cv2 . flip ( img , 0 )

Rotating the mouths

For each mouth image we are going to make small rotations, specifically -30,-20,-10,+10,+20,+30 degrees, this will give us 6x times the data approx.

for in_idx , img_path in enumerate ( input_data_set ): file_name = os . path . splitext ( os . path . basename ( img_path ))[ 0 ] print ( file_name ) augmentation_number = 8 initial_rot = - 20 #save original too path = output_folder + "/" + file_name + ".jpg" copyfile ( img_path , path ) for x in range ( 1 , augmentation_number ): rotation_coeficient = x rotation_step = 5 total_rotation = initial_rot + rotation_step * rotation_coeficient #print(total_rotation) mouth_rotated = image_rotated_cropped ( img_path , total_rotation ) #resize to 50 by 50 mouth_rotated = cv2 . resize ( mouth_rotated , ( IMAGE_WIDTH , IMAGE_HEIGHT ), interpolation = cv2 . INTER_CUBIC ) if generate_random_filename == 1 : guid = uuid . uuid4 () uid_str = guid . urn str_guid = uid_str [ 9 :] path = "" if 'showingteeth' in img_path : path = output_folder + "/" + str_guid + "_showingteeth.jpg" else : path = output_folder + "/" + str_guid + ".jpg" cv2 . imwrite ( path , mouth_rotated ) else : path = "" if 'showingteeth' in img_path : path = output_folder + "/" + file_name + "_rotated" + str ( x ) + "_showingteeth.jpg" else : path = output_folder + "/" + file_name + "_rotated" + str ( x ) + ".jpg" cv2 . imwrite ( path , mouth_rotated )

Setting up the Convolutional neural network in Caffe

Finally! we are at the point where all our training data has significant amounts of information to learn the problem, the next step will be the core functionality of our machine learning pipeline, we are going to create a convolutional neural net that will learn the knowledge of what a mouth showing a teeth is, the following steps are required to correctly configure this convolutional neural network in caffe:

Preparing the training set and validation set

Now that we have enough labeled mouths in place, we need to split it into two subsets, we are going to use the 80/20 rule, 80 percent (18828 mouth images in total) of our transformed data are going to be in training set and the 20 percent (4700 mouth images) are going to be in the validation set. The training data will be used during the training phase for our network learning and the validation set will be used to test the performance of the net during training, in this case, we have to move the mouth images to their respective folders located in training_data and validation_data.

training set folder

validation set folder

Creating the LMDB file

With the mouth images located in the training and validation folders, we are going to generate two text files, each containing the path of the corresponding mouth images plus the label (1 or 0), these text files are needed because Caffe has a tool to generate LMDB files based on these.

import caffe import lmdb import glob import cv2 import uuid from caffe.proto import caffe_pb2 import numpy as np import os train_lmdb = "../train_lmdb" train_data = [ img for img in glob . glob ( "../img/training_data/*jpg" )] val_data = [ img for img in glob . glob ( "../img/validation_data/*jpg" )] myFile = open ( '../training_data.txt' , 'w' ) for in_idx , img_path in enumerate ( train_data ): head , tail = os . path . split ( img_path ) label = - 1 if 'showingteeth' in tail : label = 1 else : label = 0 myFile . write ( tail + " " + str ( label ) + "

" ) myFile . close () f = open ( '../training_val_data.txt' , 'w' ) for in_idx , img_path in enumerate ( val_data ): head , tail = os . path . split ( img_path ) label = - 1 if 'showingteeth' in tail : label = 1 else : label = 0 f . write ( tail + " " + str ( label ) + "

" ) f . close ()

text file that has the path of the image plus the label, this will be required to generate the LMDB data

To generate both training and validation LMDB files we run the following commands:

convert_imageset --gray --shuffle /devuser/Teeth/img/training_data/ training_data.txt train_lmdb convert_imageset --gray --shuffle /devuser/Teeth/img/validation_data/ training_val_data.txt val_lmdb

Extracting the mean data for the entire dataset

A common step in computer vision is to extract the mean data of the entire training dataset to ease the learning process during backpropagation, Caffe already has a library to calculate the mean data for us:

compute_image_mean -backend = lmdb train_lmdb mean.binaryproto

This will generate a file called mean.binaryproto, this file will have matrix data related to the overall mean of all our training set, this matrix will be subtracted during training to each and every one of our training examples, this helps to have a more reasonable scale for the inputs.

Designing and implementing the convolutional neural network

Convnets are really good at image recognition because they can learn features automatically just by input-output associations, they are also very good at transformation invariances this is small changes in rotation and full changes in translation. In machine learning, there are a set of well-known state-of-the-art architectures for image processing like AlexNet, VGGNet, Google Inception etc. If you follow that kind of architectures is almost guaranteed you will obtain the best results possible, for this case and for the sake of simplicity we are going to use a simplified version of these nets with much less convolutional layers, remember that in this particular case we are just trying to extract teeth features from the mouths and not entire concepts of the real world like AlexNet does, so a net with much less capacity will do fine for the task.

train_val_feature_scaled.prototxt

name : "TeethNet" layer { name : "data" type : "Data" top : "data" top : "label" include { phase : TRAIN } transform_param { scale : 0.00390625 mean_file : "mean.binaryproto" mirror : false } data_param { source : "train_lmdb" batch_size : 256 backend : LMDB } } layer { name : "data" type : "Data" top : "data" top : "label" include { phase : TEST } transform_param { scale : 0.00390625 mean_file : "mean.binaryproto" mirror : false } data_param { source : "val_lmdb" batch_size : 256 backend : LMDB } } layer { name : "conv1" type : "Convolution" bottom : "data" top : "conv1" param { lr_mult : 1 } param { lr_mult : 2 } convolution_param { num_output : 20 kernel_size : 5 stride : 1 weight_filler { type : "xavier" } bias_filler { type : "constant" } } } layer { name : "pool1" type : "Pooling" bottom : "conv1" top : "pool1" pooling_param { pool : MAX kernel_size : 2 stride : 2 } } layer { name : "conv2" type : "Convolution" bottom : "pool1" top : "conv2" param { lr_mult : 1 } param { lr_mult : 2 } convolution_param { num_output : 50 kernel_size : 5 stride : 1 weight_filler { type : "xavier" } bias_filler { type : "constant" } } } layer { name : "pool2" type : "Pooling" bottom : "conv2" top : "pool2" pooling_param { pool : MAX kernel_size : 2 stride : 2 } } layer { name : "ip1" type : "InnerProduct" bottom : "pool2" top : "ip1" param { lr_mult : 1 } param { lr_mult : 2 } inner_product_param { num_output : 500 weight_filler { type : "xavier" } bias_filler { type : "constant" } } } layer { name : "relu1" type : "ReLU" bottom : "ip1" top : "ip1" } layer { name : "drop1" type : "Dropout" bottom : "ip1" top : "ip1" dropout_param { dropout_ratio : 0.5 } } layer { name : "ip2" type : "InnerProduct" bottom : "ip1" top : "ip2" param { lr_mult : 1 } param { lr_mult : 2 } inner_product_param { num_output : 2 weight_filler { type : "xavier" } bias_filler { type : "constant" } } } layer { name : "loss" type : "SoftmaxWithLoss" bottom : "ip2" bottom : "label" top : "loss" } layer { name : "accuracy" type : "Accuracy" bottom : "ip2" bottom : "label" top : "accuracy" include { phase : TEST } }

deploy.prototxt

name : "TeethNet" layer { name : "input" type : "Input" top : "data" input_param { shape { dim : 1 dim : 1 dim : 32 dim : 32 } } } layer { name : "conv1" type : "Convolution" bottom : "data" top : "conv1" param { lr_mult : 1 } param { lr_mult : 2 } convolution_param { num_output : 20 kernel_size : 5 stride : 1 weight_filler { type : "xavier" } bias_filler { type : "constant" } } } layer { name : "pool1" type : "Pooling" bottom : "conv1" top : "pool1" pooling_param { pool : MAX kernel_size : 2 stride : 2 } } layer { name : "conv2" type : "Convolution" bottom : "pool1" top : "conv2" param { lr_mult : 1 } param { lr_mult : 2 } convolution_param { num_output : 50 kernel_size : 5 stride : 1 weight_filler { type : "xavier" } bias_filler { type : "constant" } } } layer { name : "pool2" type : "Pooling" bottom : "conv2" top : "pool2" pooling_param { pool : MAX kernel_size : 2 stride : 2 } } layer { name : "ip1" type : "InnerProduct" bottom : "pool2" top : "ip1" param { lr_mult : 1 } param { lr_mult : 2 } inner_product_param { num_output : 500 weight_filler { type : "xavier" } bias_filler { type : "constant" } } } layer { name : "relu1" type : "ReLU" bottom : "ip1" top : "ip1" } layer { name : "drop1" type : "Dropout" bottom : "ip1" top : "ip1" dropout_param { dropout_ratio : 0.5 } } layer { name : "ip2" type : "InnerProduct" bottom : "ip1" top : "ip2" param { lr_mult : 1 } param { lr_mult : 2 } inner_product_param { num_output : 2 weight_filler { type : "xavier" } bias_filler { type : "constant" } } } layer { name : "softmax" type : "Softmax" bottom : "ip2" top : "pred" }

solver.prototxt

net : "model/train_val_feature_scaled.prototxt" test_iter : 5 test_interval : 100 base_lr : 0.1 lr_policy : "step" gamma : 0.1 stepsize : 500 display : 10 max_iter : 10000 momentum : 0.9 weight_decay : 0.0005 snapshot : 100 snapshot_prefix : "model_snapshot/snap_fe" solver_mode : CPU

CNN Architecture

Training and debugging the overall system

Training the neural network

With the architecture in place we are ready to start learning the model, we are going to execute the caffe train command to start the training process, note that all the data from the LMDB files will flow through the data layer of the network along with the labels, also the backpropagation learning procedure will take place at the same time, and by using gradient descent optimization, the error rate will decrease in each iteration.

caffe train --solver = model/solver_feature_scaled.prototxt 2>&1 | tee logteeth_ult_fe_2.log

Plotting loss vs iterations

A good way to measure the performance of the learning in our convolutional neural network is to plot the loss of the training and validation set vs the number of iterations.

Note: To create the plot is necessary to pre-process the .log file generated during the training phase, to do this execute:

python /Users/juank/Dev/caffe/tools/extra/parse_log.py logteeth.log .

this command will generate two plain text files containing all the metrics for the validation set vs iterations and the training set vs iterations. The next step is to plot the data using the provided Caffe tool for plotting:

python plot_diag.py

Loss vs Iterations, training with learning rate 0.01 after 5000 iterations

Note: It looks like we are stuck in local minima! you can tell this just by looking at this useful graph, note that the validation error won’t go down and it looks like the best it can do is 30% error on the validation set! this is not so good performance, a useful technique is to start with a bigger learning rate and then start decreasing it after a few iterations, let’s try with learning rate 0.1

Loss vs Iterations, training with learning rate 0.1 after 5000 iterations

Training with learning rate 0.1 (much better!) Look how we overcome the local minima at the beginning then we found a much deeper region on the loss space just by incrementing the initial learning rate at the very start, be careful because this doesn’t always works and is truly a problem dependant situation.

Deploying the trained convnet

Now that we have our network trained with a reasonably good performance on the validation set it is time to start testing it with new unseen data. To do this we are going to use the caffe library for python, and we are going to create a simple python script that will load the deploy.prototxt architecture of our convnet, along with this architecture we are going to feed it with the trained weights located on the .caffemodel file.

teeth_cnn.py

IMAGE_WIDTH = 32 IMAGE_HEIGHT = 32 class teeth_cnn : def __init__ ( self ): self . init_net () def init_net ( self ): self . mean_blob = self . mean_blob_fn () self . mean_array = np . asarray ( self . mean_blob . data , dtype = np . float32 ) . reshape (( self . mean_blob . channels , self . mean_blob . height , self . mean_blob . width )) self . mean_array = self . mean_array * 0.00390625 self . net = caffe . Net ( '../model/deploy.prototxt' , 1 , weights = '../model_snapshot/snap_fe_iter_8700.caffemodel' ) self . net . blobs [ 'data' ] . reshape ( 1 , 1 , IMAGE_WIDTH , IMAGE_HEIGHT ) self . transformer = caffe . io . Transformer ({ 'data' : self . net . blobs [ 'data' ] . data . shape }) self . transformer . set_mean ( 'data' , self . mean_array ) self . transformer . set_transpose ( 'data' , ( 2 , 0 , 1 )) self . transformer . set_raw_scale ( 'data' , 0.00390625 ) def mean_blob_fn ( self ): mean_blob = caffe_pb2 . BlobProto () with open ( '../mean.binaryproto' ) as f : mean_blob . ParseFromString ( f . read ()) return mean_blob def predict ( self , image , mouth_detector ): img = image mouth_pre , x , y , w , h = mouth_detector . mouth_detect_single ( img , False ) if mouth_pre is not None : mouth_pre = mouth_pre [:,:, np . newaxis ] mouth = self . transformer . preprocess ( 'data' , mouth_pre ) self . net . blobs [ 'data' ] . data [ ... ] = mouth out = self . net . forward () #pred = out['pred'].argmax() if ( out [ 'pred' ][ 0 ][ 1 ] > 0.70 ): return 1 , out [ 'pred' ], x , y , w , h else : return 0 , out [ 'pred' ], x , y , w , h else : return - 1 , 0 , 0 , 0 , 0 , 0

predict.py

BULK_PREDICTION = 0 #Set this to 0 to classify individual files #if bulk prediction is set to 1 the net will predict all images on the configured path test_set_folder_path = "../img/original_data/b_labeled" #all the files will be moved to a showing teeth or not showing teeth folder on the test_output_result_folder_path path test_output_result_folder_path = "../result" #if BULK_PREDICTION = 0 the net will classify only the file specified on individual_test_image individual_test_image = "../ana.jpg" #read all test images original_data_set = [ img for img in glob . glob ( test_set_folder_path + "/*jpg" )] mouth_detector_instance = mouth_detector () teeth_cnn_instance = teeth_cnn () if BULK_PREDICTION == 0 : img = cv2 . imread ( individual_test_image , cv2 . IMREAD_UNCHANGED ) result , prob , xf , yf , wf , hf = teeth_cnn_instance . predict ( img , mouth_detector_instance ) print ( individual_test_image ) print ( "Prediction:" ) print ( result ) print ( "Prediction probabilities" ) print ( prob ) else : files = glob . glob ( test_output_result_folder_path + '/not_showing_teeth/*' ) for f in files : os . remove ( f ) files = glob . glob ( test_output_result_folder_path + '/showing_teeth/*' ) for f in files : os . remove ( f ) #performance variables total_samples = 0 total_positives_training = 0 total_negatives_training = 0 true_positive = 0 true_negative = 0 false_positive = 0 false_negative = 0 for in_idx , img_path in enumerate ( original_data_set ): total_samples = total_samples + 1 head , tail = os . path . split ( img_path ) img = cv2 . imread ( img_path , cv2 . IMREAD_UNCHANGED ) result , prob , xf , yf , wf , hf = teeth_cnn_instance . predict ( img , mouth_detector_instance ) print ( "Prediction:" ) print ( result ) print ( "Prediction probabilities" ) print ( prob ) if ( result == 1 ): if 'showingteeth' in tail : total_positives_training = total_positives_training + 1 true_positive = true_positive + 1 else : total_negatives_training = total_negatives_training + 1 false_positive = false_positive + 1 path = test_output_result_folder_path + "/showing_teeth/" + tail shutil . copy2 ( img_path , path ) else : if 'showingteeth' in tail : total_positives_training = total_positives_training + 1 false_negative = false_negative + 1 else : total_negatives_training = total_negatives_training + 1 true_negative = true_negative + 1 path = test_output_result_folder_path + "/not_showing_teeth/" + tail shutil . copy2 ( img_path , path ) print "Total samples % d" % total_samples print "True positives % d" % true_positive print "False positives % d" % false_positive print "True negative % d" % true_negative print "False negative % d" % false_negative accuracy = ( true_negative + true_positive ) / total_samples recall = true_positive / ( true_positive + false_negative ) precision = true_positive / ( true_positive + false_positive ) f1score = 2 * (( precision * recall ) / ( precision + recall )) print "Accuracy %.2 f" % accuracy print "Recall %.2 f" % recall print "Precision %.2 f" % precision print "F1Score %.2 f" % f1score

Note: Note that this script will test our trained net with new single image if the parameter BULK_PREDICTION is set to zero, otherwise it will make a bulk prediction over an entire folder of images and will move the ones he thinks are showing the teeth to the corresponding folder, you can play with this behaviour based o your needs.

Testing the trained model with unseen data

Testing for a single image

First I’m going to test the net with some individual unseen images to measure individual results, to do this please modify the parameters shown below:

BULK_PREDICTION = 0 #Set this to 0 to classify individual files #if BULK_PREDICTION = 0 the net will classify only the file specified on individual_test_image individual_test_image = "../test_image.jpg"

Testing on individual images:

predicting teeth on a single image

predicting teeth on a single image

Testing for a bunch of images (Bulk testing)

Now I’m going to test over an entire folder of unseen images, we have to modify the parameters shown below:

BULK_PREDICTION = 1 #Set this to 0 to classify individual files #if bulk prediction is set to 1 the net will predict all images on the configured path test_set_folder_path = "../img/original_data/b_labeled" #all the files will be moved to a showing teeth or not showing teeth folder on the test_output_result_folder_path path test_output_result_folder_path = "../result"

the folder called b_labeled have images taken on different angles of the sampled MUCT dataset so, see this as the test set but with labels on it, I previously labeled these images using the manual labeling tool, this step is useful because we can calculate how good or how bad the net is behaving after the prediction phase.

You can look back at the entire script to know how the following code segment relates to the code, basically, we are calculating the F1score to know how good or bad our model is doing:

accuracy = ( true_negative + true_positive ) / total_samples recall = true_positive / ( true_positive + false_negative ) precision = true_positive / ( true_positive + false_positive ) f1score = 2 * (( precision * recall ) / ( precision + recall ))

So to start testing the net by classifying the b_labeled folder or classifying a single image, execute:

python predict.py

Note that this script will read all the images specified on the input folder and will pass one by one each image to our trained convolutional neural network and based on the prediction probability the image will be copied to the showing_teeth or not_showing_teeth folder. At the end of the execution of the process the accuracy, precision, recall and f1score are calculated:

Predicted Negative Predicted Positive Negative Cases TN: 283 FP: 18 Positive Cases FN: 54 TP: 396

Total samples 751

True positives 396

False positives 18

True negative 283

False negative 54

Accuracy 0.90

Recall 0.88

Precision 0.96

F1-score 0.92

The overall performance of the model is pretty good but not perfect, note that we have a couple of false positives and false negatives but is a reasonable ratio for the problem at the end.

By looking at the performance metrics we can start experimenting with different hyperparameters or different modifications of our pipeline and always have a point of comparison to see if we are doing better or not.

Testing our net with real video!

Now lets have some fun by passing a fragment of the Obama’s presidential speech to the trained net to see if Barack Obama is showing his teeth to the camera or not, note that in each frame of the video the trained convolutional neural network needs to make a prediction, the output of the prediction will be rendered on a new video along with the face detection boundary.

cv2 . namedWindow ( "preview" ) vc = cv2 . VideoCapture ( 0 ) #vc.set(3,500) #vc.set(4,500) #vc.set(5,30) if vc . isOpened (): # try to get the first frame rval , frame = vc . read () else : rval = False mouth_detector_instance = mouth_detector () teeth_cnn_instance = teeth_cnn () size = cv2 . getTextSize ( "Showing teeth" , cv2 . FONT_HERSHEY_PLAIN , 2 , 1 )[ 0 ] x , y = ( 50 , 250 ) while rval : rval , frame = vc . read () copy_frame = frame . copy () result , prob , xf , yf , wf , hf = teeth_cnn_instance . predict ( copy_frame , mouth_detector_instance ) print prob if result is not None : if ( result == 1 ): cv2 . rectangle ( frame , ( xf , yf ),( wf , hf ),( 0 , 255 , 0 ), 4 , 0 ) prob_round = prob [ 0 ][ 1 ] * 100 print prob_round cv2 . rectangle ( frame , ( xf - 2 , yf - 25 ),( wf + 2 , yf ),( 0 , 255 , 0 ), - 1 , 0 ) cv2 . rectangle ( frame , ( xf - 2 , hf ),( xf + (( wf - xf ) / 2 ), hf + 25 ),( 0 , 255 , 0 ), - 1 , 0 ) cv2 . putText ( frame , "Teeth!!" ,( xf , hf + 14 ), cv2 . FONT_HERSHEY_PLAIN , 1.2 , 0 , 2 ) cv2 . putText ( frame , str ( prob_round ) + " % " ,( xf , yf - 10 ), cv2 . FONT_HERSHEY_PLAIN , 1.2 , 0 , 2 ) #out.write(frame) print "SHOWING TEETH!!!" elif ( result == 0 ): cv2 . rectangle ( frame , ( xf , yf ),( wf , hf ),( 64 , 64 , 64 ), 4 , 0 ) prob_round = prob [ 0 ][ 1 ] * 100 print prob_round cv2 . rectangle ( frame , ( xf - 2 , yf - 25 ),( wf + 2 , yf ),( 64 , 64 , 64 ), - 1 , 0 ) cv2 . rectangle ( frame , ( xf - 2 , hf ),( xf + (( wf - xf ) / 2 ), hf + 25 ),( 64 , 64 , 64 ), - 1 , 0 ) cv2 . putText ( frame , "Teeth??" ,( xf , hf + 14 ), cv2 . FONT_HERSHEY_PLAIN , 1.2 , 0 , 2 ) cv2 . putText ( frame , str ( prob_round ) + " % " ,( xf , yf - 10 ), cv2 . FONT_HERSHEY_PLAIN , 1.2 , 0 , 2 ) cv2 . imshow ( "preview" , frame ) cv2 . waitKey ( 1 ) cv2 . destroyWindow ( "preview" )

Having fun

By running the script above you can test the trained network with any video you want:

Elon Musk teeth being detected by our conv net!

Source code

You can find all the source code https://github.com/juanzdev/TeethClassifierCNN