SSP Worldwide's disaster recovery strategy has come under fire from disgruntled brokers, with many still struggling to access the insurance software house's cloud platform nearly two weeks after a datacentre power outage knocked it offline.

The downtime since 26 August 2016 has severely hampered the day-to-day operations of insurance brokers across the UK who rely on the SSP Pure Broking software-as-a-service (SaaS) system to trade.

The system is used, according to SSP, by 40% of the UK’s brokers to track insurance renewals and remind customers to reinsure their vehicles, as well as source quotes and purchase products from third-party insurance providers.

Several brokers told Computer Weekly the downtime has limited the amount of work they can do and is costing them business, while SSP has allegedly offered little indication as to when the situation is likely to be resolved.

“After today [8 September 2016] it will be two complete weeks that we have not been able to operate properly as an insurance broker because we cannot issue quotations and our ability to service renewals is limited,” Chris Howell, managing director of Torquay-based Seaway Insurance Consultants (GI) Ltd, told Computer Weekly.

“In general, our clients are very understanding and loyal, but we have lost several who were not willing to wait until we could get their renewal sorted out.”

Another broker, who spoke to Computer Weekly on condition of anonymity, said the situation has reached crisis point for their company.

“We have no idea when we are likely to get our system back,” they said. “It’s getting to the point now where it is costing us money and the longer this saga drags on, the more worrying it gets.”

Alexandra Jewett, company director at Isle of Wight-based Black Rock Express Insurance and Financial Services, said the problems are likely to be worse for brokers who rely on the system to run their entire business, as SSP Pure Broking also offers email and productivity services too.

“We are lucky that we have paper files, [but] I feel sorry for those brokers who are paperless as they cannot do anything. With the amount of backlog we are going to have to pay staff to work overtime to catch up and get us back on track,” she added.

Unprecedented problems Matt Hodges-Long, managing director of business continuity provider Continuity Partner, said he has never seen anything like the problems SSP is experiencing before: “This is probably the worst platform outage I’ve ever seen, just in terms of its duration and impact." Aside from the incovenience and stress this is causing to the broker community, Hodges-Long said the customers they sell car insurance to could also be at risk, because they may not have been notified their policies have expired. The information in SSP is used to update the Motor Insurance Database (MID), he added. This is used by police forces to ascertain the insurance status of cars on British roads, and if the information on it is out of data as a result of the downtime, this could see drivers being pulled over unnecessarily. Read more about datacentre disaster recovery Nato Support and Procurement Agency embarks on five-year datacentre deal to boost its disaster recovery capabilities

Jon Toigo argues against virtualisation advocates that say the software-defined datacentre, with its high availability and clustering, does away with the need for disaster recovery “For people whose policies have lapsed in the period it has been down, there is a real potential they wouldn’t have been reminded they need to renew their insurance so it auto-lapses. So there is a wider potential ramifications, beyond the brokers not being able to write new business,” Hodges-Long added. In a statement to Computer Weekly, SSP group CEO Laurence Walker said the company cannot confirm how long it will take to sort the situation out and get all those affected by the outage back online. “We are continuing to work as quickly as possible to rectify the situation and getting all of our customers restored remains our top priority. We have made progress, but at present we are unable to confirm precisely how long the task will take to complete for everyone,” he said. “Once again, we would like to express our disappointment about the delay, disruption and frustration experienced by our customers, for which we sincerely apologise,” added Walker.

Root cause analysis The SSP Pure Broking downtime began on 26 August 2016, when a power outage caused an HP Storage Area Network (SAN) in the company’s Solihull-based datacentre to malfunction. Efforts to restore the service, with HP’s help, ran into further problems on 1 September 2016 when a second SAN hardware failure occurred. SSP contacted customers to confirm it would be unable to restore the service – as promised – by Friday 2 September 2016, as a result of this. The company has since released further updates, where it revealed the outage has accelerated its efforts to decommission the Solihull site, and migrate customers to two of its newer, Tier-3 datacentres, at least one of which is in West London. The West London site, according SSP’s statement to Computer Weekly, is already being used to stand up new customer implementations of its software. “We have the capability to undertake real-time data replication across the two [newer] centres, enabling us to offer a 15-minute recovery point objective for data restoration,” said Walker. “Our Solihull datacentre has a more traditional daily backup, so to restore services for these customers we are working through a process of unpacking data from backup discs, recoupling it with application data, and then reconfiguring systems for individual customers.” While this process plays out, brokers should regain access to the platform in stages, Computer Weekly understands, but it remains unclear how many have benefited from this so far. At the time of writing, Computer Weekly was awaiting clarification from SSP about a subsequent datacentre outage at the West London site on 8 September 2016, which was referenced in a customer status update published on 8 September 2016. “We experienced some network issues connecting to our Powergate datacentre, which caused an intermittent service for part of this morning,” the update reads. According to reports on social media, this is thought to have caused further disruption for brokers who had previously said they had regained access to the platform.

Contingency planning According to Continuity Partner's Hodges-Long, the common misconception that entrusting data to the cloud negates the need to backup that information elsewhere is likely to have exacerbated things for SSP users. “Time and time and time again we see people say they don’t need to do that because it’s a SaaS platform,” he said. “We work with a number of brokers that use this and other platforms and there are ways to mitigate this. Even if it’s just a case of doing a regular download of upcoming renewals and contact details and storing that somewhere else but the SaaS platform. “Anyone who relies on a SaaS platform for their business processes should really plan for the eventuality that it may not be available to them, and need to do the proper due diligence on the providers of those SaaS platforms to see what protection they can offer,” he added.