Outlook

In this tutorial we build an object-oriented model for an application that uses Microsoft Azure Face API to detect and highlight faces in a gallery of pictures:

This tutorial requires Pharo 6.1. Connecting to the Face API further requires a subscription key. You can also follow the entire tutorial without getting a subscription key. For that you need to load the code of the tutorial, as it contains the data returned by the Face API. Running the complete application requires Bloc, a new graphical framework for Pharo. More details on how to load the code for the tutorial are on the github page. The short version is:

Metacello new

baseline: ' CognitiveServiceDemo ';

repository: 'github:// chisandrei / cognitive-service-demo /src';

load.

Starting with a basic model

We start by designing a simple object-oriented model for our application. Given that we want to highlight faces within pictures, our model needs at least two entities, Picture and Face . A picture has then a list of faces, and a face knows the picture it belongs to. We also add a rectangle attribute to a face to indicate the contour of that face within the picture.

We could create our application without this model. However, we can leverage explicit entities to have a structure for our application that is closer to the actual domain. Once we have domain entities as explicit objects we can extend the IDE with custom views through which we can have high-level conversations about our domain.

Implementing the Picture class

To implement this model in Pharo we first create a class modelling a picture. As a side note, we use the prefix CSD for the classes defined in this tutorial. The classes in the reference implementation have the prefix CS .

Object subclass: #CSDPicture

instanceVariableNames: 'faces pictureForm'

classVariableNames: ''

package: 'Cognitive-Services-FaceAPI-Demo'

The picture has a list of faces stored within the faces attribute that we can initialise with an empty list. The picture has also a pictureForm attribute for storing the actual pixels; this will be an instance of the Form class. To interact with Picture objects let us only create for now accessors for the pictureForm attribute, and a getter for the faces attribute.

"initialization"

CSDPicture>>#initialize

super initialize.

faces := OrderedCollection new "accessing"

CSDImage>>#pictureForm

^ pictureForm "accessing"

CSDImage>>#pictureForm: aForm

pictureForm := aForm "accessing"

CSDImage>>#faces

^ faces

Now that we have the first version of our model we can play with it. To do that we need a picture. For this tutorial we’ll use the following picture available on wikipedia here.

Einstein with Habicht and Solovine (wikipedia)

We can load this picture into Pharo using Zinc HTTP client. We configure the client to do a single request, expect a jpeg image, and create a Form object from the response data. To access the URL we can copy it to the clipboard and then retrieve it directly from Pharo using the Clipboard class.

pictureUrl := Clipboard clipboardText asString. pictureForm := ZnClient new

beOneShot;

accept: ZnMimeType imageJpeg;

contentReader: [ :entity |

ImageReadWriter formFromStream: entity readStream binary ];

get: pictureUrl. picture := CSDPicture new

pictureForm: pictureForm.

Rather than just looking at static snippets of code, we’d like to execute them and interact with the resulting objects. In Pharo, the Playground is the tool that allows us to do this. We can execute first the code to download the image and inspect the resulting object. This opens a new pane with that object.

Inspecting a Form object using the Raw view.

In this case the result is a Form object. By default the Raw view is selected, showing us the actual technical implementation of this object. However, this only allows us to reason about how the picture is stored as a bitmap. This can be useful if we are working on implementing the Form class. If we are just consuming its content, seeing how the picture actually looks can provide more value. Luckily, every Form object provides a second view, Morph that does just this.

Looking at a Form object using the Morph view shows its graphical representation.

Now, we can immediately see that we got the right picture, and can proceed to create and inspect the picture object.

Inspecting a Picture object using the Raw view.

Creating a custom view for Picture objects

Like in the case of the form object, when inspecting the Face object, the inspector shows us by default the Raw view. This allows us to check the implementation of this object and see if it is what we expect. However, after assessing the implementation, we’d also like to see how the attached picture looks like. Unfortunately, this object does not provide us with such a view.

A first solution is to select in the Raw view the attribute pictureForm . This opens the form object in a new pane to the right where we can see the picture.

Navigating from a picture object to the form containing the actual pixels.

This requires a navigation in the inspector, for which we need to have knowledge about the internals of this object. Here we need to know that the picture is stored in the pictureForm attribute. For more complex objects, it might not be as obvious what attribute or set of attributes are directly relevant for understanding a domain-specific aspect. This can also be the case if other developers need to understand our code.

Hence, another solution is to create a new custom view for this object that shows us the graphical representation of the picture.

Given that Pharo has a moldable object inspector we can easily add this extension. We attach an extension to an object by creating a new method in the class of that object and adding a specific annotation to that method ( gtInspectorPresentationOrder: ).

"inspection"

CSDPicture>>#gtInspectorPictureIn: composite

<gtInspectorPresentationOrder: 25>

composite morph

title: 'Picture';

when: [ self pictureForm notNil ];

display: [ (AlphaImageMorph withForm: self pictureForm)

layout: #scaledAspect ]

This method takes as parameter a builder object that can instantiate various types of views. In this case, we select a view for displaying graphical components ( composite morph ). Next, we need to configure the view. For this we specify a title ( title: ‘Picture’ ) and use the method when: to indicate that this view should be available only if the picture has a pictureForm . We use the display: method to indicate what this view should display. We could directly return self pictureForm , however, we can improve the view by wrapping the form in a graphical component that automatically resizes it to fit within the available space. This is not something that we want in general for the Morph view of a Form , as that view needs to work for all Form objects.

Now, with just 7 lines of code we have a new way to interact with our object.

Inspecting a picture object with a view that shows its graphical representation.

Implementing the Face class

Next, let’s create a class for modelling a face within a picture. For now, this can be just a class with two attributes, rectangle and containerPicture , plus the corresponding accessors. We store a reference to the picture containing the face so we can recover the actual graphical representation of the face.

Object subclass: #CSDFace

instanceVariableNames: 'rectangle containerPicture'

classVariableNames: ''

package: 'Cognitive-Services-FaceAPI-Demo' "accessing"

CSDFace>>#rectangle

^ rectangle "accessing"

CSDFace>>#rectangle: aRectangle

rectangle := aRectangle "accessing"

CSDFace>>#containerPicture

^ containerPicture "accessing"

CSDFace>>#containerPicture: aPicture

containerPicture := aPicture

Now we can instantiate a Face object. Since we do not yet have a client that can automatically detect faces, we will do it manually. For example, the position of Einstein’s face is given by the rectangle (860@320) corner: (960@420) .

CSDFace new

rectangle: ((860@320) corner: (960@420));

containerPicture: picture.

To interact with this Face object let’s add the code to the previous snippet from the Playground and inspect it.

Inspecting a Face object using the Raw view.

Creating a custom view for Face objects

Again, when inspecting a Face object the inspector shows us by default the Raw view. While this view helps us to asses the implementation of the Face object, what would really help to see is the actual graphical representation of the face. This is a piece of information that is not directly visible in the Raw view. However, we have access to the container picture and the rectangle delimiting the face. Hence, we can write a small code snippet that extracts the graphical representation of the face from the picture.

self containerPicture pictureForm copy: self rectangle

We can now use the code editor from the Raw view to execute this snippet of code and inspect the resulting object in a new pane to the right.

Using a code snipper to extract the graphical representation of a Face object.

This gives us the desired result. However, to repeat this action on a new face object we have to remember the code and manually re-execute it. A different approach consists in creating a custom view for this object that directly show us the graphical representation of the face.

"accessing - dynamic"

CSDFace>>#faceForm

^ self containerPicture pictureForm copy: self rectangle "inspection"

CSDFace>>#gtInspectorFaceMorphIn: composite

<gtInspectorPresentationOrder: 20>

composite morph

title: 'Face';

display: [ (self faceForm scaledToSize: 256@256) asMorph ];

when: [ self hasFaceForm ] "testing"

CSDFace>>#hasFaceForm

^ self containerPicture notNil and: [

self containerPicture pictureForm notNil ]

Like in the case of the Picture object, we create a view displaying a graphical component. We label the view as Face and make sure that the view is only available when the Face object has a container picture attached. We can place the logic for extracting the form object representing the face in a dedicated method, so we can reuse it. Also, to handle faces that are two small or two large we scale the form to fit in a 256 by 256 rectangle.

With only 10 lines of code we have a new custom view for our Face object.

Inspecting a Face object with a view that shows the graphical representation of the face.

Prototyping a basic client for Azures Face API

Now that we have a basic model for handling pictures and faces let’s create a client for the Azures Face API and use it to obtain the position of faces within pictures. This part of the tutorial requires access to a subscription key for the Face API. One can be obtained for free. However, if you do not want to do this you can skip this section. If you downloaded the code of the tutorial it already contains the data that would be obtained by calling this API.

Let’s start by creating an HTTP client that can make a request to the Face API URL. At this point you will need to use you own key for the Face API. Once you have obtained a key you can insert it directly in the script or copy it from the clipboard. In our case, to not insert the subscription key directly in the code snippet, we copy it from the clipboard. We further configure the client to use a JSON parser to read the response data.



client := ZnClient new

url: '

headerAt: 'Ocp-Apim-Subscription-Key' put: subscriptionKey;

contentReader: [ :entity | STONJSON fromString: entity contents ]. subscriptionKey := Clipboard clipboardText asString.client := ZnClient newurl: ' https://westeurope.api.cognitive.microsoft.com/face/v1.0' headerAt: 'Ocp-Apim-Subscription-Key' put: subscriptionKey;contentReader: [ :entity | STONJSON fromString: entity contents ].

Next we configure the client to do a detect request according to the API specification. A detect request returns the list of faces found in the given picture. Since for now we do not want to keep track of faces using this service we set returnFaceId to false. However, we want the API to return us data about landmark positions in the face as well as several attributes. Since our target picture was downloaded from an URL we can just pass the API the actual URL.

faceAttributes := #(age gender headPose smile glasses emotion).

client

addPath: 'detect';

method: #POST;

queryAt: 'returnFaceId' put: false;

queryAt: 'returnFaceLandmarks' put: true;

queryAt: 'returnFaceAttributes' put: (String streamContents: [:s |

faceAttributes asStringOn: s delimiter: ',']);

contents: (STONJSON toString: {'url' -> pictureUrl} asDictionary);

contentType: ZnMimeType applicationJson.

If we execute the above code we will get a ZnClient object. We can then inspect the client object to see if our request header was properly configured.

Exploring the headers from the request objects using a dedicated view.

Since everything looks ok, we can proceed to execute the request

faceStructures := client execute.