17 mins read

Hey! Check out this Swarm Simulator. Wouldn’t that be great to have a fully automated solution to grow the swarm for you? How does it feel to be, you know, the coolest farmer in the neighbourhood using the most fancy farming gadgets? I guarantee you that after finishing this tutorial you will know exactly how it feels.

This is a detailed tutorial on how I wrote the code. I’m not very experienced with Elixir therefore you may find some bits weird or overcomplicated. I guess this is natural for learning process. I encurage you to leave comments on what improvements could be done. I will gladly apply them. A Pull Request on Github would be even more welcome!

Full source of the BOT on Github: https://github.com/RadekMolenda/SwarmSimulatorBOT

Technology

Here is what we are going to use:

Elixir

Phantomjs

and… A BOT (we will write one in this article)

Phantomjs we will use Elixir library for browser automation called hound it integrates nicely with phantomjs. Another reason for picking phantomjs is that it’s headless - setting it up on a server should be trivial.

Elixir is the best choice as I just decided to learn this amazing language. We will also benefit from some cool features of erlang like :timer.send_interval/3 . It seems like writing it in Ruby wouldn’t be as easy.

Setup

Let’s start with using the Elixir built in build tool mix

mix new swarmsimulatorbot && cd swarmsimulatorbot

This command should output something like:

* creating README.md * creating .gitignore * creating mix.exs * creating config * creating config/config.exs * creating lib * creating lib/swarmsimulatorbot.ex * creating test * creating test /test_helper.exs * creating test /swarmsimulatorbot_test.exs Your Mix project was created successfully. You can use "mix" to compile it, test it, and more: cd swarmsimulatorbot mix test Run "mix help" for more commands.

Implementation

We are ready to do some coding now (well almost as we need to install some dependencies before). As I mentioned earlier we will use hound for browser automation. Let’s add it to the codebase, function deps is defined at the end of mix.exs file

1 2 3 4 5 6 7 #file: mix.exs defp deps do [ { :hound , " ~> 0.8" } ] end

We can install the library by running the following command in shell

mix deps.get

We should also start hound process when the app starts we will do it by adding :hound to the application function.

1 2 3 4 5 #file: mix.exs def application do [ applications: [ :logger , :hound ]] end

As well as let hound know we will be using phantomjs driver

1 2 #file: config/config.exs config :hound , driver: " phantomjs"

Starting the hound and finding units

Now when the setup part is done. I would like to write some tests first. In this step I don’t really know what kind of functions and modules I would like to write. Being not experienced in Elixir is also disadvantage as it makes BDD even more difficult. Anyways - it doesn’t really matter we will just start writing and see where we get with it. Let’s open the autogenerated test file test/swarmsimulatorbot_test.exs and change it to look more or less like that:

1 2 3 4 5 6 7 8 9 10 11 12 13 #file: test/swarmsimulatorbot_test.exs defmodule SwarmsimulatorbotTest do use ExUnit . Case use Hound . Helpers doctest Swarmsimulatorbot test " Initial number of units should be three" do Swarmsimulatorbot . start assert length ( Swarmsimulatorbot . units ) == 3 Swarmsimulatorbot . stop end end

You can see there is quite a lot of things going on here. Thanks to use Hound.Helpers we will have access to useful hound helpers functions. You can also see that I want my Swarmsimulatorbot module to respond to three new functions: start/0 , units/0 and stop/0 .

After looking at hounds simple browser automation readme, I decided that starting hound session and performing some basic steps (like navigating to swarm simulator URL) will be done in start/0 . stop/0 will be responsible for stopping the hound session and units/0 should return all swarm units DOM elements after clicking Show all units link. Initialy there will be 3 units available and therefore length(Swarmsimulatorbot.units) should be 3.

There is one thing I don’t like about those tests tho - we will be making a real http request. I was trying to solve this issue by using exvcr library, but integrating it using my current my knowledge is far beyond my skills. I ended up leaving the tests like that but that’s definitely something I would like to improve in next iteration.

Let’s run our tests in shell

mix test

This should give us more or less the following output

1 ) test Initial number of units should be three ( SwarmsimulatorbotTest ) test /swarmsimulatorbot_test.exs:6 ** ( UndefinedFunctionError ) undefined function Swarmsimulatorbot.start/0 stacktrace: ( swarmsimulatorbot ) Swarmsimulatorbot.start () test /swarmsimulatorbot_test.exs:7 Finished in 0.07 seconds ( 0.07s on load, 0.00s on tests ) 1 test , 1 failure

Let’s fix this and next errors by implementing the correct functions in Swarmsimulatorbot module:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 #file: lib/swarmsimulatorbot.ex defmodule Swarmsimulatorbot do use Hound . Helpers @swarm_url " https://swarmsim.github.io/" def start do Hound . start_session navigate_to ( @swarm_url ) execute_script ( " localStorage.clear()" ); end def stop do Hound . end_session end def units do show_all_units find_all_elements ( :css , " .unit-table tr" ) end defp show_all_units do click_on_text ( " More..." ) click_on_text ( " Show all units" ) end defp click_on_text ( text ) do find_element ( :link_text , text ) |> click end end

Let’s repeat our tests in shell

mix test

The error message should look a bit more serious now

1 ) test Initial number of units should be three ( SwarmsimulatorbotTest ) test /swarmsimulatorbot_test.exs:6 ** ( exit ) exited in : GenServer.call ( Hound.SessionServer, { :change_session, #PID<0.175.0>, :default, %{}}, 60000) ** ( EXIT ) an exception was raised: ** ( MatchError ) no match of right hand side value: { :error, %HTTPoison.Error { id: nil, reason: :econnrefused }} ( hound ) lib/hound/request_utils.ex:43: Hound.RequestUtils.send_req/4 ( hound ) lib/hound/session_server.ex:67: Hound.SessionServer.handle_call/3 ( stdlib ) gen_server.erl:629: :gen_server.try_handle_call/4 ( stdlib ) gen_server.erl:661: :gen_server.handle_msg/5 ( stdlib ) proc_lib.erl:240: :proc_lib.init_p_do_apply/3 stacktrace: ( elixir ) lib/gen_server.ex:564: GenServer.call/3 ( swarmsimulatorbot ) lib/swarmsimulatorbot.ex:5: Swarmsimulatorbot.start/0 test /swarmsimulatorbot_test.exs:7 Finished in 0.1 seconds ( 0.08s on load, 0.1s on tests ) 1 test , 1 failure Randomized with seed 199589 20:17:32.354 [ error] GenServer Hound.SessionServer terminating ** ( MatchError ) no match of right hand side value: { :error, %HTTPoison.Error { id: nil, reason: :econnrefused }} ( hound ) lib/hound/request_utils.ex:43: Hound.RequestUtils.send_req/4 ( hound ) lib/hound/session_server.ex:67: Hound.SessionServer.handle_call/3 ( stdlib ) gen_server.erl:629: :gen_server.try_handle_call/4 ( stdlib ) gen_server.erl:661: :gen_server.handle_msg/5 ( stdlib ) proc_lib.erl:240: :proc_lib.init_p_do_apply/3 Last message: { :change_session, #PID<0.175.0>, :default, %{}} State: % {}

Apparently we forgot about one important element {:error, %HTTPoison.Error{id: nil, reason: :econnrefused}} should give us a clue what went wrong. Taking another look at hound documentation finally gives the explanation:

You’ll need a webdriver server running

Let’s start a webserver then by running the following command in another shell (we need to keep it running all the time from now on)

phantomjs --wd

mix test is finally green

Compiled lib/swarmsimulatorbot.ex . Finished in 1.2 seconds ( 0.07s on load, 1.1s on tests ) 1 test , 0 failures

Taking screenshots

We need to check the health of our swarm from time to time. I think the best way to do so is to make screenshots.

The test:

1 2 3 4 5 6 7 8 9 #file: test/swarmsimulatorbot_test.exs test " screenshot" do Swarmsimulatorbot . start f = " screenshots/test.png" Swarmsimulatorbot . screenshot ( " test.png" ) assert File . exists? ( f ) File . rm ( f ) Swarmsimulatorbot . stop end

I’m expecting mix test to complain about undefined function Swarmsimulatorbot.screenshot/1 . Let’s add a missing function then.

1 2 3 4 5 6 #file: lib/swarmsimulatorbot.ex def screenshot ( path ) do show_all_units take_screenshot ( " screenshots/ #{ path } " ) end

We will only do screenshots on all units page for now as it’s really enough to check our swarm conditions. We don’t want to polute our root directory with some file images - this is why we will keep them in screenshots directory. Hound take_screenshot/1 helper function will take care about the rest.

Make sure you run mkdir screenshots before running tests. mix test should end with 2 tests, 0 failures .

Now we are ready to play with our Swarmsimulatorbot.screenshot/1 function in iex console

iex - S mix iex ( 1 ) > Swarmsimulatorbot . start nil iex ( 2 ) > Swarmsimulatorbot . screenshot ( " hello-buggies.png" ) " screenshots/hello-buggies.png" iex ( 3 ) > Swarmsimulatorbot . stop :ok

And if all went well you should endup with the image saved in your screenshots directory similar to this one

pretty neat! Looking at swarm is quite entertaining but we obviously want more - we want our BOT to actively grow our Swarm.

Growing the Swarm

If we want to grow our swarm we need to think about the strategy. How about?. Iterate over each unit and try to grow as much units as you can. Seems like a perfect starting strategy. It should be quite easy to implement - on each unit page find the last ‘clickable’ button and click it. We will call our strategy dummy_grow/0

In our test we will expect population of drones would increase after first call of dummy_grow/0 function.

1 2 3 4 5 6 7 8 9 10 11 #file: test/swarmsimulatorbot_test.exs test " # dummy_grow grows the swarm" do Swarmsimulatorbot . start Swarmsimulatorbot . dummy_grow drone_text = Swarmsimulatorbot . units |> Enum . at ( 2 ) |> inner_text assert drone_text =~ ~r/Drone.*3/ Swarmsimulatorbot . stop end

We will fix test by implementing dummy_grow/0 function the following way.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 #file: lib/swarmsimulatorbot.ex def dummy_grow do units_size = length ( units ) - 1 Enum . each ( 0 .. ( units_size ), fn ( index ) -> units |> Enum . at ( index ) |> find_within_element ( :tag , " a" ) |> click active_buttons |> List . last |> click end ) end def active_buttons do find_all_elements ( :css , " a:not(.disabled).btn" ) end

And this change satisfies the tests. The drones are growing!

If you look closer at the dummy_grow/0 function you might have noticed there is some redundancy. It would be so much simpler just to iterate over Swarmsimulatorbot.units/0 and not call units/0 one more time in Enum.each/2 - that’s very true, but apparently angularJS (the framework used for building Swarm simulator) reloads the page after each click. An attempt to iterate over units/0 and clicking it one by one would endup in error due to rest of the units not being present in DOM after first unit click.

Rest of the code seems quite self-explanatory. We have used some new hound helper functions and some standard Elixir programming to implement dummy grow functionality.

Kind of a server

We have written just enough to start iex session call Swarmsimulatorbot.dummy_grow/0 couple of times and take some screenshots to see that the swarm is actually growing. It is growing, isn’t it?

Now what’s left is to automate calling dummy_grow/0 and screenshot/1 . I think this is the most interesting part. We will write it using processes.

Here is the idea: we will use two processes. One for keeping the browser session, clicking the buttons, growing our Swarm and taking screenshots. The second process will be responsible for sending two types of messages periodically to the first process:

:grow message - send every couple of seconds. It will force the second process to call dummy_grow/0 function :screenshot message - send every minute. It will force the second process to call screenshot/1 function

Here are the implementation details:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 #file: lib/swarmsimulatorbot.ex def start do spawn_link __MODULE__ , :init , [] end def init do Hound . start_session navigate_to ( @swarm_url ) execute_script ( " localStorage.clear()" ); loop end defp loop do receive do { :screenshot , path } -> screenshot ( path ) loop { :grow } -> dummy_grow loop { :stop } -> stop end end

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 #file: lib/swarmsimulatorbot/cli.ex defmodule Swarmsimulatorbot . Cli do @tick 1000 @screenshot_tick 60000 def start do spawn_link __MODULE__ , :main , [] end def main do bot_pid = Swarmsimulatorbot . start :timer . send_interval ( @tick , bot_pid , { :grow }) :timer . send_interval ( @screenshot_tick , bot_pid , { :screenshot , " growing.png" }) end end

So - what is going on here. First of all I have moved the start/0 logic to init/0 function and let start/0 to spawn a process for us (apparently it is some kind of a standard in Elixir world). The key change was adding private function loop/0 this is a ‘place’ where a process will listen for incoming messages. Recursive calling of loop/0 is just a way of saying we want to recieve messages all the time and we don’t want to stop after receiving the first message.

The Swarmsimulatorbot.Cli module is responsible for:

starting the Swarmsimulatorbot process periodically send messages to Swarmsimulatorbot process using :timer.send_interval/3 function

And believe me or not - this is the end of the part I. Let’s try this out in iex

iex - S mix iex ( 1 ) > Swarmsimulatorbot . Cli . start #PID<0.157.0>

And here is how my swarm stats looks like after about an hour of BOT running.

Sweet!

Unfortunatelly my BOT just stopped due to phantomjs timeout error.

07 : 49 : 20.798 [ error ] Process #PID<0.158.0> raised an exception ** ( MatchError ) no match of right hand side value: { :error , % HTTPoison . Error { id: nil , reason: :timeout }} ( hound ) lib / hound / request_utils . ex: 43 : Hound . RequestUtils . send_req / 4 ( swarmsimulatorbot ) lib / swarmsimulatorbot . ex: 41 : anonymous fn / 1 in Swarmsimulatorbot . dummy_grow / 0 ( elixir ) lib / enum . ex: 610 : anonymous fn / 3 in Enum . each / 2 ( elixir ) lib / enum . ex: 1478 : anonymous fn / 3 in Enum . reduce / 3 ( elixir ) lib / range . ex: 80 : Enumerable . Range . reduce / 5 ( elixir ) lib / enum . ex: 1477 : Enum . reduce / 3 ( elixir ) lib / enum . ex: 609 : Enum . each / 2 ( swarmsimulatorbot ) lib / swarmsimulatorbot . ex: 28 : Swarmsimulatorbot . loop / 0