GCP Goodies Part 10— Measuring happiness with Google Cloud Vision API

An attempt to measure happiness in a fully-remote company using Google Cloud Platform.

Mike Hoff @ Flickr.com CC 2.0

The story

Softwaremill.com is a fully remote company. We do work from our homes and see each other live from time to time in many different places around Poland. Every day though, we see each other and talk for a short while on a video conference using our day-to-day tools.

This is the last part of the “GCP Goodies” series and to make it a bit more fun for myself and to utilize some more services Google Cloud Platform offers, I have created a simple solution as an attempt to measure happiness at my workplace :).

You can read more about our company at excellent blog post series: “How we built a fully remote, self-organizing company before it was cool”

The tools we use

For our remote work, communication with the clients, as well as between ourselves, we mostly use Slack (for instant messaging and one-to-one video calls), BigBlueButton for daily video meetings and TeamSpeak for voice communication. As mentioned earlier, we meet almost every day for a short company video meetings where we use BigBlueButton for video only and TeamSpeak for voice. The meeting takes place at the same time every day and lasts, more or less, 15 minutes.

Besides standard chit-chat talk about current mood, weather etc., across the week we have a special question prepared for each day everyone may answer. The questions are usually non-work related and their purpose is to know each other better, mostly they are about people and their personal interest, things we do outside of work etc.

Happiness measurement attempt

Please, bear in mind that this blog post is still more about Google Cloud Platform than about any serious attempt to happiness measurement using some computing resources, but still, the results posted at the end of this article look promising ;).

The idea for this blog post has been simple. Every day, at specific time, we are going to launch a virtual machine on GCP and log into our video conference software (BigBlueButton) as some virtual user. Once our virtual user ‘sees’ all the people on a meeting, it will take a screenshot of all the faces every few minutes and save it to the Google Storage Service. Once we collect the screenshots for the whole week we will run a Python script to analyze how many people were smiling each day at our meetings.

Setup

Because BigBlueButton software uses flash on its website the handling of the web application with automated scripts was a bit more challenging than usual. At the end, I have decided to run the Chrome web browser on a Windows platform and use Python with selenium to do all the operations needed to log in into the meeting, create the screenshot, process it a bit and upload to Google Storage service.

Let’s start with creating Windows based virtual machine.

You can set windows password after the instance creation by clicking on an instance name and navigating to its details page.

To enable remote desktop connection to our VM we need to open the port 5901 for a TCP traffic. Navigate to VPC Network -> Firewall rules and add the following details:

The next step is to create a service account which will be used to communicate with other GCP services from the Windows VM (like sending screenshots to the bucket for further processing).

Click Create and save the generated JSON file somewhere for later use.

Virtual Video Conference User

Some preparations need to be done on the VM machine so the Selenium Python script will be able to enable flash and log in into our conference room. I have used Chrome RDP for Google Cloud Platform plugin to log into the server and install the needed components.

I highly recommend setting up static IP address and assign it to the virtual machine you will use. This way you will always connect to the same IP address and will have one less thing to remember when accessing the VM.

Setting up static IP address can be done under Networking -> VPC Networks -> Static IP Addresses

To install software needed for this tutorial on Windows machine, I highly recommend using choco .

We need to install Python and Chrome browser. Start by installing choco and then install Python in version 3.7 :

@"%SystemRoot%\System32\WindowsPowerShell\v1.0\powershell.exe" -NoProfile -InputFormat None -ExecutionPolicy Bypass -Command "iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))" && SET "PATH=%PATH%;%ALLUSERSPROFILE%\chocolatey\bin"

and then install Python (please note that you need to install version 3.7 or less, otherwise you will have problems installing Pillow library later on):

choco install python --version 3.7

refreshenv

If you want to do that from the powershell mode, simply enter powershell in the command line and you will switch to PS mode. The refreshenv command refreshes your console environment so you should be able to use your newly installed software straight away.

choco installed with the above command was in version 0.10.15 .

When in powershell mode, execute the command found here to install Chrome on windows server.

$LocalTempDir = $env:TEMP; $ChromeInstaller = "ChromeInstaller.exe"; (new-object System.Net.WebClient).DownloadFile('http://dl.google.com/chrome/install/375.126/chrome_installer.exe', "$LocalTempDir\$ChromeInstaller"); & "$LocalTempDir\$ChromeInstaller" /silent /install; $Process2Monitor = "ChromeInstaller"; Do { $ProcessesFound = Get-Process | ?{$Process2Monitor -contains $_.Name} | Select-Object -ExpandProperty Name; If ($ProcessesFound) { "Still running: $($ProcessesFound -join ', ')" | Write-Host; Start-Sleep -Seconds 2 } else { rm "$LocalTempDir\$ChromeInstaller" -ErrorAction SilentlyContinue -Verbose } } Until (!$ProcessesFound)

Once we have the software we need, we have to install python libraries used by our script, after choco Python installation, use pip or pip3 to install the libraries:

pip install selenium

pip install pynput

pip install google-api-python-client

pip install google-cloud-storage

pip install Pillow

Now we can test our script. I have created a windows shell script which simply navigates first to the correct directory and then executes our python script itself:

cd C:\Users\krzysztof_grajek

python screen.py

krzysztof_grajek directory is a home directory created by GCP when building the windows VM for us.

The screen.py script is self explanatory, the important bits are the key presses needed to enable flash in a browser for the web page we want to use:

actions = actions.send_keys(Keys.TAB * 21)

Similar one needs to be executed to log in once the flash takes over the control of the web page. In such cases, tab navigation and hitting enter button with selenium is your friend :) — dirty hack but works. The code is pasted below, you can find the sources for all the scripts in the repository for part-10 of this tutorial series here.

The last part which needs to be done on the windows side of things, is the scheduled job which will run every couple of minutes and start up our ps1 script. To do that, enter the following in the command line console:

schtasks /create /tn HappyTask /tr "powershell -file C:\Users\krzysztof_grajek\runhappy.ps1" /sc minute /mo 2 /ru System

That basically means, create a task executed every 2 minutes which will run the runhappy.ps1 shell script as a System user.

The ps1 script will start up the process handled by Python — we will start the Chrome browser, enable flash for our web page, navigate there, log in and take the screenshot. Once the screenshot is ready, we resize it by 100% in each direction and upload it to Google Storage service.

Please note that you need to have the credentials json file available on windows machine to send the data to the Google Storage bucket.

Scheduling Server work

Anytime the server VM is running the scheduled task created there will launch the script and send the screenshots to our bucket. There is only one small problem, the Windows machine is not cheap and we don’t need it to be run all day and send screenshots of the meeting when nobody is in there. What we want is to launch the VM when the meeting starts and stop it when the meeting ends. As the meetings at SoftwareMill are always at the same time of the day that job is easy.

The part for Scheduling Server Work is almost the same as the tutorial on Google Cloud website itself:

We will create a pair of Cloud Functions to start and stop our VM and then execute those functions with Cloud Scheduler service.

First, we need to distinguish which instance we want to start/stop. For this purpose we will label our instance so we won’t accidentally stop or start other instances we have.

Go to VM details and add label to our instance:

Navigate to Compute -> Cloud Functions and add new functions for starting and stopping our VM instance.

index.js content for starting the instance:

package.json content for starting the instance:

Similarly, do the same for stopping the instance function:

index.js content:

package.json content:

Once we have the functions ready we can test them by sending base64 encoded json with the data we want.

Prepare payload to test the function:

echo -n ‘{“zone”:”us-central1-a”, “label”:”env=dev”}’ | base64 {“data”:”eyJ6b25lIjoidXMtY2VudHJhbDEtYSIsICJsYWJlbCI6ImVudj1kZXYifQ==”}

Testing start/stop functions:

You should see on the Compute VM section your instance being stopped/started accordingly.

Once we have the start/stop functions working and we have tested that they indeed stop and start our instance, we can create a scheduler job to do that for us, let’s say — every day from Monday to Friday at specific hours.

Set up Cloud Scheduler job to start/stop an instance automatically:

Navigate to Tools -> Cloud Scheduler to create new job.

That will call our Cloud function every day from Monday to Friday at 10 am.

shutdown job:

Will do the same as the previous job but for the stop function, half an hour later.

You can test it with RUN NOW button.

Processing Results

At this point we should have everything set up and working to gather the data for us. The data are simple screenshots taken every few minutes during our video conference meetings throughout the week.

We will utilize one more Google Cloud service we haven’t used before on the GCP Goodies series — that will be Google Cloud Vision API.

With this simple service we will send every screenshot we have collected during the experiment, and for each face detected on the screenshot we will determine how happy the person detected was at that time.

The happiness levels are described by Google using values ranging from unknown to very likely: 'UNKNOWN', 'VERY_UNLIKELY', 'UNLIKELY', 'POSSIBLE', 'LIKELY', 'VERY_LIKELY'.

Once we gather all the results we will flatten them out and store them in a single CSV file, which we will process further in Google Data Reporting service.

First we get all the files stored in our bucket:

and for each file we get the likelihoods of joy and group them by day:

Detecting faces and returning the joy likelihoods is just a single call to the Google APIs:

We don’t need to download the files and send them again to another Google service (like Google Vision), we just need a path which we can use to point Google Vision API where the image we want to process is stored.

Please note, before you run the script, you need to pass the credentials somehow so that Python Google libraries we use will know you are the authenticated user and have access to the resources. For the purpose of this tutorial, just execute the following before running the script on your local machine:

export GOOGLE_APPLICATION_CREDENTIALS=”/path/to/your/happy.json

assuming that happy.json is the service account credentials file you had created earlier.

Results

After running the script you should be able to find a newly created .csv file in the same directory. The script contains the number of occurrences for specific joy level and day, e.g:

In other words, on the 22nd of October there was 314 detected faces for the whole meeting with level 1 ( UKNOWN ) and 11 occurrences of level 5 — which is joy_likelihood VERY_LIKELY .

Open up Google Data Studio Reporting and select Create and then Data source .

From the list of available options select File Upload

Select the file we have just created and wait for the processing to finish:

Click Connect in the top-right corner:

and then, Create Report :

Delete everything that has been added automatically to our report and add a new Bar chart

Modify the chart properties:

Set as a Breakdown Dimension our level to see the data grouped by Day and Level values. Then change the default Sort to use Day dimension in ascending way.

After playing around a bit you should end up with something similar to the graph below:

The levels 3,4 and 5 are 'POSSIBLE', 'LIKELY', 'VERY_LIKELY' correspondingly, therefore sum of those will show us the most happy day for the total of 6 days of this experiment.

And the winner is.. Thursday, 24th of October 2019.

To be frank, I was expecting that the clear winner will be Friday 25th but unfortunately it scored less, the reason for it, I believe, was the unfortunate random question we all can answer during each of those meetings. The one for Friday 25th of October was “What irritates you the most?”, whereas the Thursday one was “Are you fulfilled in life?” As you can see, the questions themselves and the topics we discuss everyday vary a lot :)

As a bonus I will post a couple of the screenshots created by our Google-based Happiness Monitoring Solution for you to see how happy we are :D