Hey everyone,

It’s been exactly a month ago since we launched the competitive experience on Battalion 1944. Over the past weeks, we’ve received tons of feedback from you and we wanted to shed some light on how we approach matchmaking, the tweaks that we’ve actively been making and what’s more to come!

Haven’t played Battalion 1944 yet?

In matchmaking there isn’t a silver bullet, a magic algorithm that solves all the issues (trust us, we wished there was 😔).

A few years back when we started looking into how we should solve matchmaking for our supported games, we decided to build a system that allowed us to iterate on a number of different algorithms in the form of pipelines, rather than a single, generic algorithm that would magically create matches only on the basis of skill and win rate.

This approach allowed us to create a multitude of algorithms ranging from very complex ones with multiple factors to tweak (time, region, language, skill, matches played, roles, etc.) to more simple ones (time, random). Each of these has served in one way or another for a different experience.

When it comes to Battalion 1944, the main goal of the matchmaking algorithm is to create a “balanced” and fair experience for the players involved.

The algorithm we are using is based on three primary factors (with a few adjustments based on minor variables like subscription and location):

Skill : The skill difference between the two teams calculated as the probability of winning, measured by our Matchmaking Rating (“MMR”).

: The skill difference between the two teams calculated as the probability of winning, measured by our Matchmaking Rating (“MMR”). Team composition : The composition of the team in terms of solos, parties, full stack and pre-made is a factor that impacts the quality of a match, or at least it’s perception of the match difficulty.

: The composition of the team in terms of solos, parties, full stack and pre-made is a factor that impacts the quality of a match, or at least it’s perception of the match difficulty. Time: How long has someone been waiting in queue in order to find a match.

Now that we’ve explained the main factors, how can we evaluate whether the algorithm is actually doing it’s job? We can analyze it based on each of the factors above.

1) Are matches balanced skill-wise?

The first question that we need to consider is the one linked to skill. Is it common to find uneven matches? How often does this happen?

The following graph shows the distribution of matches since launch, based on the probability of the weakest team winning.

The bulk of matches created, since June 1st, have a 50/50 probability of winning, as close to 80% of all matches.

These are matches where the weakest team is expected to win 5 matches every 10, in other words, these are statistically perfectly balanced. You can see that the number of perfectly balanced matches increased significantly on the 1st June; this is due to the fact that we tweaked the algorithm by modifying a step in the pipeline; balancing between the weight of stacking and the weight of skill.

The second largest category (balanced matches) represents roughly 21% of the matches created in Europe and North America since June 1st. This includes all matches where the weakest team is expected to win ~4 matches in every 10. The challenging category represents ~2% of the matches and includes all matches where the weakest team is expected to win ~3 matches in every 10.

Finally, you have the bottom two categories (unbalanced and < 5% chance) that represent less than ~0.5% of the matches (with an MMR difference of over 200).

The graph above shows the distribution of matches created (y-axis) by the probability of winning of the strongest team (x-axis).

To conclude, the analysis above shows that most matches are created with balanced teams in terms of skill. ~98% of the matches are well balanced and only 0.5% of the matches are bad matchups. At the same time, it shows that there are 5 matches in every 1,000 that are unbalanced and will not provide a positive experience for the weakest team. Why is that? Is there a way to improve this? Let’s look into it.

Why do these matchups happen at all?

This situation occurs when players in low liquidity skill levels (for example, high skill or very low skill) have been searching for a match for a long time. The new players joining the queue are immediately matched with the players that have been waiting, leading to an unbalanced matchup.

This is demonstrated clearly when looking at the same distribution graph on an hourly basis, as these matches tend to happen at times when the liquidity within the queue is lower and players are therefore waiting longer for a match.

Is there a way to avoid these matches from happening?

The reason why these happen is that when deciding whether to create a match, the algorithm used for “5v5 WARTIDE” takes into account the player or party that was in the queue for the longest period of time, rather than all of them. This means that when a party has been waiting for 10 minutes, while a new party joins the queue, the new party might get an unbalanced match because they are getting matched with someone that has been waiting in queue for a very long time.

There are a number of solutions that we can (and will) activate for this issue;

The first one is putting a limit to how unbalanced a match can be. The trade-off of this is; is it better to stay in queue for 15 minutes or to have an unbalanced match?

The second solution is taking into account the wait time of all parties instead of the one that spent the most time in queue. After all, why should 9 players that just joined the queue get matched with a level 10 because the player was in queue for a longer time? This leads to players with unusual skill levels never finding a match, as the players within range are matched into games of similar skill before they can be matched with the unusual skill level.

This trade-off is confirmed by the numbers whenever we look at matching quality by segments, as it shows that most of these unbalanced matches happen at a high level where there are fewer players, rather than at the mid to low skill level where most of the population plays.

2) Are matches balanced depending on team composition?

The second criterion we can use to evaluate the quality of potential matches created is the team composition by stacking balance. This criteria is based on the assumption we made above, that the team composition affects the expected result of a match (i.e. a stack has higher probability of winning than a team of solos). Data hints that, while there is some truth to this assumption, it has considerably less impact than it is perceived to have.

That said, perception and reality are in sync here. The matchup is indeed unbalanced, but to a smaller degree than people seem to think.

To avoid these matchups, we make sure that during the first few minutes in queue you cannot be matched with someone that has a party composition too different from yours (e.g. 5-stack vs. all solos), but as time passes, we increase the range of acceptable options. This adjustment is having a clear effect as only 0.7% of the matches have 5-stacks playing vs. solos.

This graph shows the distribution of matches with full stacks segmented by the team composition of the opposing team. As you can see, the majority of matchups for a team of five are against stacks while the minority are the extreme matchups with 5 vs solos.

At this time, unbalanced matchups occur at the extremes of the skill spectrum, either at high or low skill.

The reason is that when it comes to party composition, the algorithm is less aggressive than with skill.

Is it better to play against a 5-stack of players that are around your level, or against solo players that have a far higher level than yours?

As seen in the case of skill balance, there are a few matches that are outliers. These happen for the same reason as for skills. The algorithm for team composition looks at the player that spent the most amount of time in queue when deciding whether a match can be created, rather than looking at the time of all players. The problem here is the same and the solutions are similar (taking into account all players or putting a maximum difference), however, in this case we need to evaluate whether it makes sense to put the team composition as a hard limit i.e. never allow 5-stacks to play with solo players, ever?

An alternative solution to alleviate this issue, is to consider weighting the average towards higher skill rated for teams whenever they play as stacks to account for the advantage they have, ensuring that the final result is a fair match.

3) How long does it take to find a match?

The final thing we have to look at is time spent in queue. Rather than a measure of match quality alone, we need to consider time spent in queue in order to decide how aggressive the algorithm’s settings can be set. The questions we are trying to answer (for each segment of players) are:

How many players need to wait over a certain amount of time without finding a match? How many players drop out of the queue because they didn’t find a match in time?

Questions such as these provide an idea of the amount of time that each segment is willing to spend in queue, in addition to whether the current settings and algorithm work well.

The first graph we look at is the time spent in queue (below).

The time spent in queue by players before finding a match (median, 90th percentile and 99th percentile)

The first new is great news. The median time to find a match on Battalion 1944 in Europe and North America in June was 32.4 seconds, which is very low. This means that 50% of the matches are created in less than 30 seconds. Even the 90th percentile (131.2) and the 99th percentile (469.3) are relatively low, which means that 90% and 99% of the matches have been found within ~2 and ~7 minutes respectively.

This data provides a positive outlook and, at a first glance, we could even decide to be more aggressive in terms of matching requirements. But how does the data look for the segment that creates unbalanced matches (i.e. players with higher skill levels that queue at low liquidity times)?

If we look at the same data for players with an MMR greater than 1500, the picture changes slightly.

While the vast majority still experiences wait times of less than 2 minutes (with a median of 108 seconds) and 90% find matches within 469 seconds, the “slowest” 1% of the matches takes 1,166 seconds or more.

These waiting times are the factors that create issues from a balancing standpoint, as the more time a party waits in queue, the more unbalanced matches can be considered acceptable. As we increase the MMR, the wait times increase and the balancing worsens, accepting higher skill gaps and wider team compositions. This is even more evident when looking at it on an hourly basis. As you can see in the graph below at certain times (mainly very early in the morning local time) wait times increase significantly, leading to worsened balancing. These metrics are consistent with the balancing metrics as it is mostly at these times that the 0.5% of unbalanced matches happen.

Time spent in queue by hour for Wartide 5v5 in the Europe region

The graphs above explain the trade-off mentioned in the analysis of the first two criteria. Any change that would improve balancing creates risks of isolating players of higher skill levels that would need to wait in queue for even longer.

The final piece of analysis we do when looking at time spent is linked to understanding how much time people are happy with spending in queue before dropping out. In order to do this we look at the “queue drop rate”, in other words, how many times a player leaves the queue after entering it.

After all, if all players were happy to wait for 30 minutes, we’d be able to create the most balanced matchmaking system ever!

Approximately 13% of players that join the matchmaking queue eventually end up leaving it. This sounds surprising, right? To clarify this, ~90% of them do so within the first minute, and the median sits at 5 seconds — a less worrying result. These are all the “mis-clicks”, players that either did not want to actually enter the queue or that forgot to invite a friend to their party, which are not representative of the willingness to wait for a match.

The real issue becomes clear again when we segment by MMR. From the table below you can see how players that are below 500 MMR and above 2,000 drop at a far greater rate from the queue. This is due to the fact that these segments have to wait longer in order to find a match (as explained above) and hints to the fact that we are already alienating players by prioritizing balancing over time.

What are our next steps?

We have a few changes planned to test to see if we can improve the experience further.

A few things we will try out include:

Increase the minimum wait time threshold if you just joined the queue before matching you with a suboptimal match i.e. taking into account the median MMR of the players.

Match a higher imbalance depending on team composition e.g. match a 5-stack with a stronger team of solo and party queuers.

Put a threshold on the maximum imbalance that’s acceptable.

Segment the experienced players from the less so e.g. ensure that there is a good balance between new and experienced players based on number of matches played.

Increase the wait time for perfect team composition matchups.

As we evaluate these changes, we are being mindful about ensuring that we don’t make the experience worse.

We continue to analyze the data and run simulations based on historical data points, approaching this with experimentation and data while capturing the perceived value of what a good matchup is.