Non-Determinism

Sometimes it passes, sometimes it fails.

Randomness

Can a computer generate a truly random number? Not really, at least not without special hardware. Instead, computers simulate randomness by drawing from a pool of entropy like cursor movements, keystrokes, hard-disk access, etc.

These “random” sources are then combined in a single location (like /dev/random ) for applications that require pseudo-random behavior.

Regardless of whether the behavior is truly-random or pseudo-random, testing randomness can be quite difficult. Unsurprisingly, it is also an obvious source of non-deterministic behavior. Consider a Dice class:

Calling roll should return a random number between one and six, just like a real dice.

Next, we write tests to verify this behavior. Perhaps we also include a test that verifies that the dice is not somehow weighted (or maybe this code is for a gambling service and we want skewed results).

We run our tests locally, and they pass. They pass again in CI. A few days later, we notice an intermittent failure on the second test. The easiest solution would be to avoid randomness altogether. When that is not possible, we can instead provide a “seed” to the Random class.

Note: This is the same process we use when debugging randomly-ordered RSpec tests by manually setting the --seed option.

Pseudo-random number generators (PRNGs) are just functions that map one input to a seemingly-random, but deterministic sequence. Using a fixed seed provides us with a fixed sequence, and thus deterministic results.

Now, total will always yield 191 and the test will pass consistently! Although we have also introduced an inconsistency between test and production conditions. Fortunately, this scenario is not that common.

Unique constraints

Randomness will more likely show up as collisions between randomly generated data in a model factory with unique constraints.

The odds of collision with the ObjectId or similar UUIDs are exceptionally low, but the probability of duplication with the description are 1:1000. When randomly generated properties are necessary, the easiest solution is to pick from a larger pool of random.

Note: Faker is another great tool that can be used in test factories to create unique, structured data without the risk of collisions.