Currently the federated web holds 35 projects with 9 different protocols (source: the-federation.info) all trying to connect and communicate with each other. But does that actually work? Most of the projects do have unit testing but only for certain areas, like the front-end or the database, but the protocol itself isn't tested. Well it definitely works-on-my-machine but does it work on others too?

Why do we never have time to do it right, but always have time to do it over?

- Murphy’s Law

Problem here is that you have to start up multiple server instances and watch their behaviour and that is what the federation testsuite from Feneas is all about!

With the testsuite you can write simple unit tests ensuring that your server is actually capable of sending/receiving federated payloads.

Testsuite

How does it work?

Say we have a project called GangGo which uses one or more protocols to participate in the federated web and we have a second one like Diaspora which we want to test federation with.

How can we ensure that if I start sharing from GangGo with a Diaspora contact, the Diaspora server actually receives the message? And how can we ensure that future code changes will not break that?

We built a system capable of starting multiple servers in parallel and comparing database changes and/or API results. In our example the testsuite will start a GangGo and a Diaspora instance already pre-configured and ready to send & receive! Then all we have to do is tell GangGo to start sharing and watch the other side for incoming messages.

And that could look something like this:

@test "$btf send public post" { [...] post "post=helloworld" "http://$(container_ip ganggo):9000/api/v0/posts" echo "expected 200, got $HTTP_STATUS_CODE" [ "$HTTP_STATUS_CODE" == "200" ] postID=$(json_value "ID") [...] } @test "$btf was the public post received" { [...] res=$(query diaspora "select count(*) from posts where guid = '$guid';") [ $res =~ "1" ] [...] }

The first test sends a public post to the API endpoint of the running GangGo server and the second test looks up public posts in the Diaspora database.

If the...

API call returns a HTTP_STATUS_CODE with 200 the test passes

with the test passes Diaspora database returns a count of one for the specified GUID the test will pass too

The testsuite uses Bats for executing the above script files and Docker to run multiple different servers. And thats all there is to it in respect of dependencies, cause everything else is installed via Docker. For example the Diaspora instance requires a Redis and Postgresql server in parallel but that can be configured via the test file.

Why do you use command line scripts (Bats)?

The reason for this is very simple. We have Ruby, Python, Golang, Rust, PHP and many more programming languages in the federated web but testing them together proves to be challenging! The first thing which comes to mind is the shell cause even if the project is not reachable via an API we can somehow send messages to it. For Diaspora we use e.g. rails_runner to talk to the server and with GangGo we use curl to request API changes/results.

Doing this in the shell seems to be the most sane solution for now ;)

Testsuite Server

Why should I use this service?

Assuming we have some unit tests for the project you are about to use in the testsuite repository. Then you can authenticate your personal project against our integration service. Once that happens, every time someone creates a merge request an automatic build is triggered and the testsuite will verify that the unit tests still succeed.

In GitLab an external pipeline will be added and it will update the status whether tests are failing or not. That looks something like this:

GitHub supports displaying similar checks:

How can I integrate my project?

Visit testsuite.feneas.org and click on the GitLab or GitHub symbol depending on where your project is located. The GitLab integration also supports self-hosted instances \m/

You will end up with a view like this:

Fill out the fields and pay attention to the hints below every field. Then your project should be integrated in no time.

On the bottom of the page you'll find the opt-in feature where you can also define whether it should start testing right away or if it should wait till someone requests it. The server will check the following fields for the opt-in flag:

Last commit message

Pull/merge request Title Description/body

Labels

Last but not least, hit the "Submit" button and if all data was set correctly your project should be listed on the index page of testsuite.feneas.org.

Now the testing can begin :)

Testproject

For everyone who is now curious about how exactly you can implement your project and/or write unit tests we've created a test project!

The project itself is not very interesting cause it's just a bash script capable of sending simple messages between other bash scripts.

The interesting part is how we implemented it in our testsuite.

You will find the documentation at feneas.git.feneas.org/federation/testsuite and you can have a look at some examples here:

Contributions

If you have questions, objections or suggestions you can write us via mail testsuite@git.feneas.org or create a ticket in GitLab.

The project repositories are located here:

Documentation

The documentation page is written in simple Markdown and is a work in progress. If you have some changes to make, do it here: git.feneas.org/feneas/federation/testsuite/tree/master/docs