The EBU has confirmed that the Belarussian jury points announced during the Eurovision Song Contest 2019’s Grand Final were incorrect . Ben Robertson investigates what has happened, why it is easy to make this mistake, and what steps the EBU must ensure happen to improve the situation.

What Happened On Friday And Saturday Night?

Belarus’ jury vote reveal during the Eurovision Song Contest’s 2019 Grand Final was going to be one to keep an eye on. Earlier in the week it had been revealed that the EBU had disqualified the Belarussian jury from voting on the Grand Final. This was because members of the jury had revealed publicly the songs they preferred during the first Semi Final. Jury members should keep how they voted secret until after the Grand Final.

In confirming this to Eurovoix, the EBU also confirmed that for the Grand Final the Belarussian jury vote would be calculated by an ‘aggregated result approved by the auditors’:

The Belarussian jury voting has been revealed in an interview contravening Eurovision Song Contest rules. In order to be compliant with the ESC voting regulations, the EBU has taken action and has dismissed the Belarussian jury from the Grand Final on Saturday. An aggregated result approved by the auditors will be used in order to determine to whom the Belarussian votes will be allocated.

It was therefore a shock for many to see Belarus award twelve points during the jury voting on Saturday night to Israel. Not only was this Israel’s only twelve points, they were Israel’s only points of the jury voting. For this to be an aggregated result means somewhere else Israel must have placed high with the juries…which was visibly not the case.

Something clearly went wrong.

Members of the Eurovision community were quick to suggest what the problem was. Taking countries in the same pot as Belarus used to split up similar voting countries for the Semi Finals (namely Armenia, Azerbaijan, Russia and Georgia) and averaging their jury rankings resulted in a jury score that was a perfect match for how Belarus voted.

…if you took their combined last place and flipped it around to be their first place.

Israel was ranked 15th, 21st, 24th and 25th by those four respective countries. This average placment of 21.25 was the lowest scored by any country, but somehow rewarded Israel with 12 points on Saturday night.

A ‘human error’ can be inferred from these numbers; the calculated average was ranked from highest to lowest rather than lowest to highest. The lowest average ranking song (which would have been ‘Chameleon’ from Malta) should have received the 12 points, not Israel.

The EBU have now acknowledged that this is the mistake, and on Wednesday May 22nd revealed a corrected ranking for the Grand Final before updating the Grand Final scoreboard.

How The Results Have Changed

The Belarussian jury vote is only 1/92nd of the total vote in the Eurovision Song Contest, and thankfully there is no controversy over our winner… it’s still The Netherlands, although ‘Arcade’ is now a slightly more impressive winner with a total score of 498 points compared to the 492 points that were revealed on Saturday night.

The changes that exist are further down the table. Sweden becoming the top Scandinavian country – leapfrogging the televote winner Norway into fifth – is the most notable swap. The biggest alteration is in the mid-table. Cyprus and Malta improve two places to 13th and 14th respectively, forcing Slovenia and France down by the same amount.

From a production perspective, the most significant change is that North Macedonia win the jury vote. With 10 points from the Belarussian aggregated score, Sweden’s last minute steal of first place on Saturday night would not have happened. Tamara Todevska would have held the limelight and be leading the Song Contest at the half way mark, and the final head-to-head in the new voting announcement procedure would have been Todevska vs Lawrence.

A Need To Catch Human Error In The Future

That takes care of the facts from Eurovision 2019, but what lessons need to be learned?

The wrong scores were announced on the night, and these could have had a much larger impact. Imagine for a moment a closer result where The Netherlands was awarded the Eurovision Song Contest victory on screen, but the true winner was Italy…

The most troubling aspect has to be the number of checks that missed the error. As the EBU press release takes time to explain, it is Digame who produced the aggregated result; then voting monitor Ernst & Young approved the final results; and at the final level it goes to Executive Supervisor Jon Ola Sand and the EBU team who give the final all clear live on the Saturday night broadcast. Nobody in this chain spotted spotted the upside-down mistake… although when it was announced Eurovision fans online raised digital eyebrows on social media.

Credit must be given to @Euro_Bruno for his analysis of the jury scores which raised the issue in the community which was subsequently picked up by the world’s media.

This is the first time an entire jury score has been replicated in this way since 2016, when jury scores and tele votes were split in the Saturday night presentation, but it is not the first mistake in process that has happened.

From individual jurors we have seen evidence of incorrect ranking orders numerous times. Arguably the most notable example is Hilde Heick, a Danish juror from 2016, who intended to place eventual winner Ukraine second last, but instead voted ‘upside down’ and placed Ukraine second. 14 of the 23 point advantage ‘1944’ had over ‘Sound of Silence’ that year was due to that mistake. Other individual jurors, including a number from this year’s Song Contest, have voting patterns that suggest they have voted the wrong way around. Ranking ‘upside down’ has been a problem for years and it remains a possibility that there has been a material impact in this year’s Contest in terms of countries who qualified for the Grand Final:

While the process to determine the Belarussian ‘jury’ vote is different to the ranking from an individual juror, the same mistake has occurred… Eurovision voting rewards those who get the highest points, but jury rankings start from the smallest number. It is easy to see how human error can result in these mistakes. This must be the catalyst to ensure there is increase clarity to the process.

One immediate quick fix would be for each juror and jury chairperson to write down which song is their favourite and input it into the computerised voting form to ensure the ordering has been completed correctly as a ‘check’ on the ranking order

Ensuring Integrity to The Eurovision Song Contest Voting

The Eurovision Song Contest voting is now a battle of two halves, with both jury and televote having equal value. After the EBU’s corrections to the vote, this year we have an unusual situation where neither televote nor jury winner are in the final top 5… and there is nothing wrong with that.

What a split outcome requires though is the need for a demonstrably clean, transparent and fair voting system for the Song Contest. This year we didn’t get that until four days after the live show. The Belarussian jury vote was an embarrassment for the EBU, for the artists involved, and also for the Belarussian broadcaster.

There are also ethics which need to be openly discussed.

Is it ethically correct to use other juries in different countries to ‘simulate’ a Belarussian jury? The countries used to assemble the points may be in a group of similar countries, but their results showcase huge differences. Russia for example received a 1st, 6th and 26th place ranking from those used to construct the vote. Azerbaijan scored a 1st, 2nd and 26th place ranking. Even politically detached Sweden received a 1st place and 9th place…combined with a 15th and 18th place. By accepting that you should take replacement jury points from countries that have a similar voting history, does that mean the result you calculate just perpetuates the perceived issues of political voting?

Certainly one argument to make that would have been to not include a Belarussian jury at all. This has issues for the TV broadcast – as we would therefore not see a Belarussian spokesperson on screen. However making Maria Vasilevich read out a jury score that was quite obviously made up has its own moral dilemma.

The Black Box Of Exponential Weighting

This is not an abstract question for the 2019 presentation. There are a number of issues around voting and point calculations that need to be exposed to public scrutiny to ensure the long term integrity and confidence in the Eurovision Song Contest results.

Firstly, in 2018 the EBU revealed an ‘exponential weight model’ to calculate the points from the 5 jurors. While good in that it mimimises the ability of one juror to destroy the voting power of one song, no set formula for this exponential curve has been formally revealed by the EBU.

ESC Insight’s Ellie Chalkly took a closer look at the system when it was announced last year:

This change to the jury scoring system is a welcome step forwards to a more competitive Contest. It rewards positivity, it diminishes the power of a single juror to negatively impact a song, and it allows strong but divisive songs the opportunity to achieve a respectable jury score ahead of the televote the following night.

While these changes are welcome, the formula requires a number of constants that have not been revealed. Without these numbers it is impossible for a third party to confirm the model is working as advertised. Essentially there is a ‘black box’ between the juror scores and the jury points awarded. If the Belarussian jury vote can be incorrect, can we have confidence in the rest of the jury voting process?

Returning to the suggestion that a juror has ranked incorrectly during Semi Final One, we cannot be certain of the impact on the qualifiers because we do not know the exact workings of the black box that calculates the jury points.

It is impossible for a broadcaster and member of the public to themselves calculate and ratify that the result is valid alone. We can model what the exponential curve acts like, but this is not the same as a clear and transparent system. One key model of integrity is ensuring that the model is replicable by others.

The San Marino Televote

As noted, it was a simple average of the juries in the same pot as Belarus that resulted in a correct calculation of the Belarussian jury vote. Each year since 2016 we have had to use a similar aggregate method to calculate not a jury, but a televote. Because San Marino uses the Italian telephone system, it is not possible to guarantee a televote is only made up of San Marinese voters. Instead San Marino’s televote system is “an average result of a representative group of televote results of other countries.”

San Marino’s televote construction sounds eerily similar to how Belarus’ jury result has been constructed.

Yet whereas the Belarussian jury score appears to have been mathematically simple to construct, it has not been possible to re-construct a San Marinese televote model since 2016. Even their Head of Delegation does not understand how the vote is constructed. Poland’s Tulia were two points away from reaching the Grand Final, and received five points from this constructed San Marino televote. It is easy to imagine if a different group of countries were selected Poland could have scored two more points and qualified for Saturday.

Hiding the process does not protect the integrity of the televote, it simply creates more mystery and raises more questions around the validity of the vote.

Improve The Design By Expecting Mistakes

Mistakes and human error happen, we all accept that. The key is to expect them and design systems that can allow them to be discovered and corrected. Having to change the results after the live show has embarrassed the EBU, Digame and Ernst & Young. Improvements to the design of the jury vote are needed to make sure it delivers the robust result that is expected by the millions of viewers and listeners to the Eurovision Song Contest every year.

However the EBU can not and should not stop now the Belarussian jury vote has been addressed.Clarity and integrity throughout the Song Contest’s voting systems are needed. This year’s incidents can be the catalyst for much needed improvements both behind the scenes and in the package that is presented to the public.