Why Unity

A couple of months back I started working on a strategy video game based off the Lebanese Civil War. I experimented with a bunch of engines including building my own crappy one and the one I ended up having the most affection towards was Unity.

Unity has a rich visual interface to create environments and place assets

2. Unity makes it easy to give different game objects various behaviors via the Entity Component System which treats objects as sorts of servers that send messages to other objects to interact with them

3. Lots of assets and useful code on the Asset Store

Why build a simulation? Why not use a benchmark?

At NeurIPS 2016 I had a brief discussion with a prominent Reinforcement Learning researcher where I made the case that new state of the art performance on Atari games was uninteresting and essentially over-fitting massive models on simple games.

I argued that it would be a lot more interesting and productive to develop new kinds of environments that test out new kinds of capabilities 0 → 1 instead of 99.8% → 99.99%.

But the researcher insisted that it would make comparing papers more challenging. I got flashbacks to my SAT days.

Even though I’ve been interested in Games my whole life and Reinforcement Learning my whole professional life, I never realized how big of a connection there was a until a couple of months ago

What is Unity ML agents

In short Unity ML agents is a Unity plugin that lets you hook up your game agents (e.g: enemy AI) to a Reinforcement Learning server which is responsible for 3 things

Sending the required RL data: states, actions, rewards Training the RL model Inputting actions back to the game from the model

Reinforcement Learning is a declarative technique for Game AI

Declarative because you don’t need to understand how the RL model works although it does help.

You really only need to do 2 things and Unity ML agents will take care of the rest.

Encode actions and states Set a reward function

So let’s open an example up and see how it all works

Unity ML agents at a glance

The first thing you’ll see in the README is 12 different environments where a couple seem to have several identical agents.

The reason we have simulations with multiple identical agents is that many popular RL algorithms are off policy which is a fancy way of saying that multiple agents can all collect data in the same cache which is called the experience replay buffer. An RL training algorithm would then run on that entire dataset.

Other notable highlights from the README are

Imitation learning Bootstrap the RL training by a video dataset of you playing the simulation Curriculum learning: Where you can chain more complex environments together after an agent excels at a more basic task Proximal Policy Optimization & Soft Actor Critic RL algorithms: examples of stable RL algorithms that’ll ensure your agents won’t be too “jerky”. You can download any algorithm you like from Github and integrate it in if you have specific needs

Moving on download Unity , open up the folder \ml-agents\UnitySDK\Assets\ML-Agents\Examples\Template\Scene.unity and then double click on Soccer2 in the projects tab.

Unity ML agents comes with 10 great examples but Soccer2 is both simple and interesting so it’s a great first example.

There are 4 agents in this simulation. 2 red and 2 blue each with a striker and a goalie. They were all created by dragging cube prefabs to the scene viewer and changing a couple of their properties like their size or color.

If we click on the BlueStriker, its properties will pop up to the right