How to Painlessly Unite Art with Java, JavaScript, and Graphs or The Story Behind Creating an Interactive Theatre Production for the EV100 Celebrations AlexTavgen Follow Oct 10, 2018 · 12 min read

This year, a theatre production series called Tale of the Century was launched in Estonia. Throughout the year, 22 local theatres presented their interpretations of the past hundred years of Estonian history to the audiences. In the draw, the Russian Theatre was assigned the topic of the future of Estonia.

Each one of us has their own ideas about what the future might look like — what we’re afraid of and what we dream of. However, we didn’t want to create a play that would just tell the audience how the narrow circle of people at the theatre sees the future. That’s why we posed this question to the people who are actually going to live in the future — that is, the children and youths of Estonia, aged 3–19. We received responses both in Russian and in Estonian from cities and villages in all corners of the country. There were hundreds of them and we’re extremely grateful for all of them.

Each response we received was unique, but when going through the collected materials, we noticed some clear tendencies and patterns. An immense number of possible future worlds, both attractive and horrific, took shape from the children’s answers. However, there is something that unites these worlds.

The story takes us to the year 2118. Estonia is under a protective dome. We have learned a lot, including how to extend human life span. The main characters are an elderly couple called Linda and Timo. A hundred years ago, as kids, they had fantasies about the future in which they now live. In fact, they are the very children who helped create this play. During the play, these characters are dropped into different versions of Estonia in the year 2118 based on how the audience votes.

Of course, the performance does not begin when the curtain goes up, but much earlier. After processing all the responses, we identified the main possible paths for the future according to the children — from the info-technological to the ecological state. Since the future is not predefined but depends on the choices that we all make, the possible paths for the future were divided into short stories that were linked by the overarching story of the main characters and their family.

Artyom Gareev (the director of the play) demonstrates the main ideas on a miniature mock-up.

The stage design resembles a construction toy that can take any shape and perform any function in the hands of children. It’s also shaped like a graph — a special abstract mathematical object that allows us to describe and model many phenomena of the real world, like road networks, organizational models of businesses, the Internet, or the variety of choices that people make — choices that can be turned into different plotlines.

Decorations development

When represented on a graph, the possible plot developments form a complex and multi-branched structure.

Utopia often leads to dystopia and the road to hell is paved with good intentions. The scenes are bound together by the logic of dramaturgy. Taking into account all the possible options for the future that were inspired by the children’s fantasies, we had a lot of story lines. It was very important not to repeat any of them during a performance. For example, if the audience votes for a utopia, it will lead to a dystopia, or if the audience rejects the utopia, they need to make another choice. We have had situations where the votes have been almost equally divided and the final decision has depended on 2–3 votes.

Scene related to the Ecological Dystopia

Graph representation of storylines

Besides the dramaturgic task, there were multiple other technical challenges: the sophisticated controller systems that drive hundreds of LEDs on the stage, the artfully created costumes for each version of the future by one of the best Estonian theatre designers Rosita Raud (this is a challenge mainly for the actors who have to change their clothes fast when the voting has ended), and of course, the voting system that drives the story line and is built upon a graph model. This is what I’d like to tell you about in more detail.

All music was live played by MODULSHTEIN

MODULSHTEIN

In order to make it easier to understand, let’s enter a performance

We are in the multicultural story line. All languages and genders are mixed together. Everything is accepted. The performance segment begins with a news bulletin that explains the context of the following scene to the audience in a 5 languages mix.

Ironically, Jelena Solomina is a real host on a national television and a well-known public figure.

In the multicultural future, everything is fun. Semenova jr. organizes a meeting with her parents, where she introduces her boyfriend who is wearing a mixture of Arabic and ballet clothes. After introductions have been made, everybody goes to a party

At the end of the scene, voting opens. The main question is whether Estonia will go down this road or not. And if yes…

The utopia will turn into a dystopia. All freedoms are curtailed and new restrictions are introduced. Suddenly, special forces shut the party down and sort people into groups. Some people are welcomed and go to the right into the bright future, while others have to take their clothes off and go to the left into the execution machine.

The Technical Solution

Here are the requirements that we set down for our system:

- The system should collect votes, show the results in real time, and make decisions up to that.

- The system should select story lines depending on the voting results, open new votes, and switch between story line.

- The system should communicate with the audience using natural language.

- The system should be administered by the assistant director behind the scenes, with full statistics about ongoing states and votes.

In fact, this was another actor called EMA (“Mother” in Estonian), an artificial intelligence that has become a reality in the future. EMA leads the audience through the plot, manifesting itself in the different versions of the future. It also launches the voting, counts and presents the results, and moves the play along the plotlines based on the results of the voting. All of EMA was written from scratch in the 1.5 months before the premiere. Since the plotlines, various customizations, and graph transitions were constantly evolving as the play was taking shape according to the decisions of the team of scriptwriters and the director, it was not possible to start with the development earlier.

The auditorium could seat 600 people, so people had to be able to vote using their phones and we needed to be able to receive and process all these votes within less than a minute. The option of developing a mobile app was immediately discarded, because downloading and installing an application takes extra effort — no one is going to download your app if you’re not Facebook or Google. In addition, supporting different types and models of phones would have required separate development, which was unrealistic, since we had only 1.5 months and one developer. Moreover, the mobile client would have had to be in two languages ​​– Estonian and Russian. Therefore, we needed a web application that was optimized for mobile phones.

Another web application was needed as an administrative backend for controlling the system: starting/stopping, activating rounds of voting, monitoring the state of ongoing voting, and so on.

And thirdly, and most importantly, we needed to coordinate the interaction of EMA with the audience.

Here, it is necessary to introduce the concept of the “super” — something that was new to me. The “super” is the front curtain that goes down and hides the stage. It is reflective and allows you to display various projections, like news releases (which we’ll come back to later) or the EMA screen that displays information about the voting results and the choices made by the audience.

So we have a complex graph, a voting system, and three frontend systems, which need to interact with the viewers and the administrator (Svetlana Shushina, an incredible assistant director).

Graph

Let’s start with the graph. The data structure — in this case, different versions of Estonia’s future and the possible transitions between the scenes depending on the voting results — had to be stored somewhere. Standard relational databases are not well suited for storing graph data for a number of reasons. For us, the essential requirements were flexible data storage, storage of state transitions, links to audio files depending on the current state, and so on and so forth.

The obvious choice was Neo4j, the leading graph database management software in the world. Seven of the top ten technology companies in the world use it. It offers mature technology, speed, convenience, and an excellent community, which all really leave no other choice for this kind of solutions.

We modelled the scenarios built by the scriptwriters as a graph in Neo4j. Here is an example of what the plot twists and turns look like from inside Neo4j.

Being a native system for graph representation, the advantages that Neo4j provides are flexibility and the absence of a fixed structure. This made it easy to change and connect story lines and to add new versions of audio.

Piece of our graph representation

Backend

When choosing the technology for the server side, we hesitated between Python and Java. Eventually, we chose Java because of its speed and reliability while having clear requirements, and also because of Spring Boot, which supports Neo4j out of the box.

The server side was implemented in Spring Boot. The server collects votes, switches from the “super” to the following states when the voting is over, accepts commands from the administrator, provides statistics, and so on. It also handles user registration and dispatches voting data (the opening or closing of a vote and the available options) to the frontend clients.

Then we came to the most remarkable part of the task. The general idea was that the component that is displayed on the “super” — EMA — had to be able to talk to the audience. It had to prepare the audience for voting, announce the voting options, start and end voting, announce the results, and depending on the audience’s choice, suggest the options for the next vote, or give an introduction for the next scene. This meant that the audio had to be dynamic and prepared on the fly. Considering that the number of options is enormous, the duration of each audio file is different, and synchronization is important (for example, voting should open only after the announcement “Voting is open!”), this was not a trivial task at all.

So the system ended up having two control centers.

The first one is the server — it stores the current state of the plot, provides statistics, handles user registration, collects votes, and makes sure that each person can give only one vote.

The second one is the “super”, which starts and ends the voting, shows the progress and the results of the voting, and so on.

FrontEnd

The “super” was implemented in Vue.js. Initially, the plan was to dynamically build the audio using the Google Speech API, but we couldn’t guarantee that there would be no network latency. Therefore, another option was chosen and we recorded the audio files individually using Google Speech.

In total, we had 55 audio clips. The data about which audio files should be played in what state was stored on the graph together with the states. Since we were dealing with JavaScript, we strung the audio files and command calls on top of each other like grapes. Thus, we did not depend on the fact that the audio files were of various lengths and ended up with a fairly universal solution.

Initially, the mobile client website was written in VueJS, but in testing we realized that there was an incompatibility issue that affected some mobile phone models. We tried to solve the problem with Babel transpilers, but as multiple settings were added the volume of the page grew, and even though the web app started working on some phones, it broke on other ones. So finally, the mobile client was rewritten in Vanilla JS. As a result, we ended up with a distributed system.

The final volume of the system (the JAR file) was 146 megabytes. The telecommunication company Telia provided us with a virtual server at the symbolic price of 1 euro per month, for which we’re very grateful, and amazing quality of service.

First live run test were at Playtech office which turned to be pretty funny.