First of all, what is FySideStory about?

I am writing this first article of FySideStory, which will have for purpose to explain my experiments on different subjects. Usually, those experiments came out because of the difficulties I encountered while working on FyS, which explain the naming.

As some people (myself for instance) prefer usually read a post and gaining something from it directly, like some tutorial, code snippet or stack-overflow, those articles are most likely to be more interesting as explaining actual technologies, how they were implemented and so on…

What about the testing?

After working as a java developer for some years in a bank, it appeared that the testing was something crucial and mandatory. Our project was getting a good coverage (around 90% to 95%) and only very few actual coding errors were source of incidents.

I don’t teach anything to anyone here, unit testing and testing in general is mandatory in any project. And this reasoning can actually be pushed to some extreme. Ensuring good behaviour by following Test Driven Development (TDD) practice, for example, we could talk about this subject some days too as I am quite interested in it myself.

After seeing the result of this coverage during my everyday work, I became addicted to unit test and were always aiming for the best coverage, like a game seeing some green value on SonarQube was making me feel good~

And when I made my come back in the C++ world, I understood how everything I took for granted, actually wasn’t. While working on my project, at the very beginning, I wanted to test my code, basically only the foundation where there, socket connected to each other with pipelining code.

Choose Testing Framework

As my project was implemented using boost::asio as network library, I already had a testing framework ready and waiting for me to use, boost::Test . Widely used in enterprises, I began its implementation in my source code thanks to CMake.

First it is needed to import the previously installed boost::Test framework in our project

<span data-offset-key="de1fv-0-0"><pre lang="shell"> find_package(Boost 1.55 REQUIRED COMPONENTS unit_test_framework) add_executable(test ${TEST_FILES}) target_link_libraries(test_exe ${Boost_LIBRARIES}) </pre></span> 1 2 3 4 5 < span data - offset - key = "de1fv-0-0" > < pre lang = "shell" > find_package ( Boost 1.55 REQUIRED COMPONENTS unit_test_framework ) add_executable ( test $ { TEST_FILES } ) target_link_libraries ( test_exe $ { Boost_LIBRARIES } ) < / pre > < / span >

Following the documentation (or any good tutorial out there: http://jeremy.ozog.fr/cpp/2015/01/18/debuter-boost-test-cmake/ ) we add the different MACRO in our test file in order to have an automatic implementation of the main that will execute our defined test.

At this moment I was testing a piece of code that was requiring a quite heavy setup.

This setup was allocating memory on the heap. Memory that I didn’t bother freeing. My test was not working. It was actually coring on boost side, and I was doing quite simple stuff despite the setup.

At this moment I got an overview of how it was going to be a hassle to do unit testing, the issue was a priori, that boost::Test was crashing because the memory allocated in the test was not freed. And I thought, “seriously ?”, I mean, even if the memory leak is something that has to be avoided, I didn’t think it should have that much importance in the case of a unit setup.

I say a priori because this issue just pissed me off right away and I looked for an alternative to boost::Test thinking “if it starts like that it’s not gonna be better”. A very bad reaction, but I really didn’t want to bother debugging something that, I think, should have work.

And that is how I ended up using another testing framework that I would recommend to anyone today.

Catch2 : The header only testing framework

Catch2 is very good in multiple aspects;

First , the one I was looking for the most at that moment. It is easy to setup, this is a header only framework without any external dependencies, which means that you basically only need to implement the header file into your testing files and define CATCH_CONFIG_MAIN in one of them in order to generate a main that is going to execute all your declared test.

Second, it worked fine and is easy to use, the exact same test that was coring with boost was working fine with catch2, which at that moment, was a big plus.

This framework doesn’t have extensive usage of fixtures but is using sections instead .

For those who don’t know, a fixture is an object that is going to be initialized at the beginning of the test through a setup() function (and is cleaned up at the end of each test via a tearDown() function), your test can then access this object. It is assured that a new instance of the object is provided to each test using the fixture. It is common practice to use a fixture for setting up data used in multiple tests.

Why using section may be better? Like the catch2 documentation says, when doing a big test, fixtures are getting bigger and bigger the test using them get split into different files decreasing the readability of the test. But as a personal experience, the worse is the fact that a fixture is a class, and sometimes people do things they shouldn’t because they can… for example thinking that it would be better to have a fixture than inherit something templated with complex hidden mechanism using type trait and whatnot in order to minimize the number of lines a test contains.

A personal, painful, experience

As a real-world example, at my work, I was on a project that was basically a C++ API that received messages, checked flags, status, and whatever in a couchebase db before filling/or not, a relational database.

But there were multiple really similar (almost identical) tables filled by this API. And a test began to regroup them all, making a fixture beginning to be generalized in order to be able to test everything at once. And we ended up with a code similar to this for the test :

Well this is quite an extreme example in which basically, everything happening in the test (the assertions, the code called an so on) is done it the hidden class parent class

MagicClass. What it does is hidden and even if you were to go directly in the code to see what it does. You should do multiple backs and forth in order to know which class are used because of the templates.

And everything was based on the assumption that the way we test DomDatabaseSomething, DomOtherDatabaseSomething and the other types are the same. But unfortunately, someday a change has been requested on the insert of DomMahDatabaseSomething. And I had one of those one-liner test failing, I basically had to change all the test as those types became different in a fundamental way.

But it removes code duplicate!

Factorizing the code is something that is necessary, no unnecessary duplication of code is a fundamental rule that has to be followed. But I think it is not a good thing in a testing (or at least the limit between necessary and unnecessary duplicate is blurrier).

If some tests need the same setup, you should use the same code. But if the setup is different (functionally or not) but similar, it is best to have this specific setup in a separated test.

When I read a test. I want to understand what it actually does while reading the test. By looking at the assertions it does and the way the different class has been setup. When done correctly, if those rules were followed in this particular case, I would only have to change the setup/assertion of the test concerning the insertion of Product.

But there I had to go deep in the code of the test (and believe me the “TestRuleHelper” type of fixture are very annoying) and get what was wrong for finally, rewriting the test completely. In the end, a new test file has been created and I just lost time trying to fix a “generic” test that had no reason to be in the first place.

Obviously, the usage of a MagicClass could also have been made with Catch2. But the section is in the form of a scope which makes it quite natural to have all the related tests together. In the end, it’s a personal preference, but give it a try.

What to use?

Well at the end this choice is up to each user’s affinity. I personally used 3 different one on which I am going, to do a summary of the pros and cons: Boost::Test: Setup: Hard

External Dependencies: Heavy

Personal Opinion: I am certainly not going to be impartial concerning this framework as my personal experience just made me suffer from retching to me it is not worth the time investing into making this works while knowing the

easy tools existing that does the work just fine. GTest: Setup: Easy

External Dependencies: Medium

Personal Opinion: Widely used, works very well and has a lot of advanced features (specific assertions working with regex and so on). Basically, just need to be compiled with the framework library. It is a perfectly good choice, and

if not for Catch2, it would be my choice by default. Catch2: Setup: Easy

External dependency: None

Personal opinion: My favourite so far, no external dependency is a real plus, header only and works like a charm.

In the end, it is your choice, you may be very comfortable with Boost::Test or Gtest and keep using it. All of the above framework provides the same kind of features and is widely used. Those frameworks are tools and like any tools, can be misused, I just think that the way Catch2 force you to organize your test make you less inclined in those pitfalls.



What’s next?