Members' ticket sales: Behind the scenes

It's no secret that the last few Members' ticket sales have encountered a number of technical problems, prior to the sale in November 2018.

We understand just how frustrating it can be when these technical issues occur, which prevent supporters from being able to purchase tickets online.

Here we explain just what goes into the Members’ ticket sales and how we’re working to constantly improve the ticket-buying process.

What is the Members’ ticket sale?

Twice a year, we host an online pre-sale for tickets to our Premier League home games, which is open to all our official Members to give them priority access to seats at Anfield.

The first Members’ ticket sale week takes place in July and gives Members the opportunity to purchase tickets for games in the first half of the Premier League season, up until December 31.

The second Members’ ticket sale week takes place in November and gives Members the opportunity to buy tickets for games for the second half of the season, from January through to May.

There are certain criteria that Members must meet in order to be able to purchase these priority tickets, based on attendance at Premier League games during the previous season.

First, we open the sale to Members who have accrued a minimum number of games, then we extend this by lowering the minimum match requirement, and finally it’s made available to all Members until all tickets have been sold. We also hold a separate sale for disabled Members.

For those who haven’t been able to purchase a ticket to our home games in advance, there are additional opportunities to buy a seat ahead of each individual game if there are tickets still remaining or if tickets have been returned to the club. When tickets become available, they will be advertised on Liverpoolfc.com, so we recommend that fans check back regularly.

Why have there been technical issues in the past?

Tickets to a home game at Anfield are undoubtedly in increasingly high demand. So, during our extremely busy ticket sale week, our website sees a large increase in visitors, with lots of fans logging in around the same time to be first in line to apply for the matches that they’d like to attend.

Unfortunately, a few sales over the past couple of years have encountered some issues due to this increase in website traffic. While we anticipate that a large number of people will be logging on at a similar time to secure their tickets, not all the traffic our website sees is genuine, which can be problematic.

During the last sale in July 2018, some fans experienced issues trying to access the website due to a high volume of traffic from malicious bots. Our system had over one million hits per second when the sale started, with many of these being malicious attempts to access the site.

What have we done to minimise these issues?

Over the last few years, we have been making various changes behind the scenes to the way we run our Members’ ticket sales and the technology we use to improve the performance and security of our website.

We’ve added a queuing system to manage traffic volumes and give fans more clarity on where they are in the buying process. Those logging on to the site are given a unique queue ID, so they can see where they are in the queue and the estimated waiting time. Those at the front of the queue will be able to get on the site to make their purchase, and once they’ve logged off, the next supporter in the queue will be able to access it, rather than everyone trying to access it at the same time.

We’ve also moved some of our back-office systems to dedicated hosted data centres, which are specifically designed to house a company’s IT operations and can handle increased traffic and demand. Our back-office servers were moved from Anfield to a hosted data centre in December 2016 and our web servers were moved to a new data centre with increased load and performance in May 2017. This way, our systems and processes can be handled more appropriately by leading facilities that are purpose-built for housing computer systems and equipment.

Despite investing in these measures, we did still encounter some issues during the July 2018 sale. After a thorough investigation following these issues, the club introduced CloudFlare for the first time to stop any malicious attempts to access our system.

As part of the CloudFlare technology, we now have Captcha Challenge capability, which is designed specifically to stop bots. It requires the user to perform a specific task, like clicking on a box or clicking on displayed images. Bots don’t have the capability to perform these tasks, unlike humans do – so straight away the system will know if the access is genuine or not and can block it if needed.

What were the results from the November 2018 sale?

Overall, our efforts to improve the ticket-buying process have been successful, in that we didn’t have any major issues reported during the last sale regarding the technology or our website. This was despite each of the three main sales seeing record queues, peaking at 13,472 on Wednesday, 14,174 on Thursday and 30,611 on the Friday. There were 28 million requests to the system in total during the week.

We saw 1,338,241 requests to join the queuing system, but only 102,300 of these were allowed through. Over 92 per cent of all attempts to join the queue were blocked, mainly as the system recognised that people were running multiple ‘scripts’ – this basically means that people were using lots of repeated commands (known as ‘scripts’) on their devices that automate a particular task. In this case, that task was attempting to join the queue. If the system would have let all these requests through, there was a risk that the site could have crashed again, so instead the system blocked these attempts to protect other fans accessing the site legitimately.

1,235,941 malicious attempts to join the queue were stopped, enabling over 100,000 tickets to be sold. CloudFlare activated the Captcha Challenge capability a total of 435,675 times over the five days of the sale. This would have been because the traffic raised some suspicions, or the person typed their password in wrong too many times. It could also have appeared for genuine fans who were trying to purchase tickets from the same IP address, such as from their place of work. If there’s a large number of fans in the same company, then there’s a higher chance that they would encounter the captcha. Only 569 people attempted to complete the captcha, so just 0.001 per cent. The rest were most likely bots that could not complete it.

What are the plans for the next Members’ sale in July 2019?

We recognise that there are more improvements that we can make to continue enhancing our Members’ ticket sales. Our work here is not done, and we’re committed to constantly improving ticket availability for our supporters and overcoming any issues still remaining with our ticket-buying service.

Since the November Members’ sale, we moved our ticketing web servers to a new data centre to further increase our performance.

In order to improve the ticket-buying process for our disabled supporters, we’ve recently partnered with Mitel, a leading global business communications provider. We’ve implemented their innovative cloud-based communication and collaboration solutions to enable supporters to quickly and securely connect with the club from anywhere, and on any device through voice calls and online. Mitel’s technology will enable us to offer a better service for disabled supporters who need to call us to buy tickets. Better call menu options and queuing facilities will mean that they no longer have to repeatedly call us to try to get through to someone during busy periods. They will be able to dial us once and then supporters’ calls will be queued until they can be answered.

One of the next steps we’re taking with regards to the captcha technology is looking at whether we can reduce some of the limits that would trigger the captcha, so we can block more bad traffic without too many real fans having to complete it.

Our systems will also start looking into whether they can track which customer numbers are being included as part of the scripts – this way we can pinpoint where the excessive traffic is coming from and who’s overloading our site.

It has been suggested that we include a log-in before the queue to stop ineligible people joining. Adding a log-in to the queue would require a call to the back-office server to check the user name, password and eligibility. One of the main purposes of the queue is to protect the back-office servers from too much load and the system can only handle a certain number of people accessing the main application at a time. In the summer, the high volume of traffic from malicious bots would have brought the main back-office servers down, rather than just the queue. However, now that we have introduced CloudFlare, there are opportunities to explore this option and implement a scalable solution. The development is likely to be significant, so there’s a lot of work to be done.

But we’re not just looking at improving the technology that we use to run our Members’ ticket sales – we’re also looking at how the entire sales process currently works and how we can enable ticketing and Membership to work more effectively together to provide our fans with a better experience.

We are currently consulting with our supporters on any remaining issues to get everyone’s thoughts and to make sure that we find the best solution. We’ll be providing more information on any future changes that are planned ahead of the next Members’ sale. We’ll be putting any updates on our website before any changes take place to ensure our supporters are well informed.