From time to time, we’ve written about how bank IT is a systemic risk waiting to happen. Major financial firms have legacy code at the core of their systems that they can’t migrate off at acceptable costs and risk (numerous banks have had a go at this issue, and projects wind up being shelved; at best, they can port only some products or customers off the aging systems). Readers, even ones who are in IT but not in banking, sometimes scoff at what we have said.

The disaster at TSB should serve as a big wake up call. The very short version is that a UK bank, TSB, which had been merged into and then many years later was spun out of Lloyds Bank, was bought by the Spanish bank Banco Sabadell in 2015. Lloyds had continued to run the TSB systems and was to transfer them over to Sabadell over the weekend. It’s turned out to be an epic failure, and it’s not clear if and when this can be straightened out.

It is bad enough that bank IT problem had been so severe and protracted a major newspaper, The Guardian, created a live blog for it that has now been running for two days.

The more serious issue is the fact that customers still can’t access online accounts and even more disconcerting, are sometimes being allowed into other people’s accounts, says there are massive problems with data integrity. That’s a nightmare to sort out.

Even worse, the fact that this situation has persisted strongly suggests that Lloyds went ahead with the migration without allowing for a rollback. If true, this is a colossal failure, particularly in combination with the other probable planning failure, that of not remotely adequate debugging (while there was a pilot, it is inconceivable that it could have been deemed to be a success if the testing had been adequate).

Let’s turn the mike over to the Telegraph:

Customers of TSB continued to complain of being unable to access their accounts on Tuesday morning as the bank’s IT fiasco dragged on into its fifth day. TSB confirmed its 1.9m customers were still facing “intermittent” problems when attempting to log in to online services after a bungled switch over from a system the bank had been renting from its old owner Lloyd’s Banking Group. Customers had been warned the transfer of 1.3 billion customer records to a new system could affect services from 4pm on Friday to 6pm on Sunday – but the disruption continued overnight and into Monday and Tuesday.

A look at Twitter suggests that “intermittent” is not exactly accurate. These tweets are from this morning:

@TSB Finally get the option to log in and it says my username/password are incorrect! They definitely aren’t 😩 Is anyone else having this problem or have I been hacked as well as messed about? #TSB #TSBdown — Lola Moreno LaVey (@Lucky_Lola_) April 24, 2018

#tsbdown this is what im getting, hope im a zillionaire when i finally log in! Its beyond a joke now tho, dont know where i am with my money! pic.twitter.com/bbWB8a9p72 — steve-o (@smellyama) April 24, 2018

TSB this is bloody ridiculous. I’ve got important payments to make today with no online and phone access to my money. Two unwell kids to feed and I can’t pull them out of bed to go shopping 😡! #tsbdown #fixit — Laura Meres (@meres_laura) April 24, 2018

This is the only thing working so far.#tsbdown pic.twitter.com/q1gElV9OBQ — 🇪🇸 Eduardo Marin (@emarinuk) April 24, 2018

This one was a mere 15 minutes ago:

It’s my birthday and I have no access to my bank account – ‘well done’, TSB. It’s day number 5 and you are still ‘down for maintenance’… How much longer will that take? :( @TSB #TSB #TSBfail #TSBdown #happybirthday #JokeoftheDay pic.twitter.com/gkj1ZJmnuI — Sandra Kitlinska (@sandrakitlinska) April 24, 2018

And before you dismiss these tweets as mere upset customer noise, hard core techies see signs of meltdown-level problems:

Are @TSB developers seriously coding live changes in production?🤦‍♂️I’ve found loads of ugly debugging logs in the code which are being spat out in the browser console. I don’t dare to imagine what mess they have in the back-end. #TSB #tsbdown pic.twitter.com/qHh5sbnUIS — Martin (@apphancer) April 24, 2018

Before we go into the details, it appears that online banking is not working, or working for so few customers as to be effectively not working. Phone banking is swamped but on top of that, customers are reporting on Twitter that their user IDs and logins aren’t working for phone banking either.

Branch banking or ATMs do not seem to be on tilt, but like the person with the sick kids above, many people aren’t in a position to go to the branch as a backup, and if they did go, the branches are likely to have huge lines. And this tweet a mere seven minutes ago suggests some branches are down:

So @TSB your service is down again and that includes in branch (Leeds). My patience with you has expired! Expect @TheFCA involvement! #tsbdown — Stephanie Hayden (@flyinglawyer73) April 24, 2018

The overview is that Lloyds Bank, one of the former four British “clearing banks,” made a series of acquisitions. Its first large deal was for TSB. As a result of the crisis, it acquired HBOS, which itself was a merger of Halifax Building Society and the Bank of Scotland.

European banking regulators deemed the combined banks to be too big and concentrated. Rather than sell some products or some regional operations, Lloyds TSB decided to demerge TSB. That was already a bit, erm, gutsy, since TSB would presumably be well integrated from an IT perspective by now and therefore not necessarily so easy to hive off. From a 2013 Guardian story:

Twenty years after disappearing from the high street, the TSB bank will reappear in towns across the UK on Monday when more than 630 branches that were Lloyds units on Friday reopen with a new identity… Lloyds has been forced to split off and rebrand the TSB branches by the EU as a result of the £20bn of taxpayer money pumped into the bank during the 2008 bailouts. It has pledged to turn TSB back to its heritage as a “local” bank. The 631 branches were scheduled to be sold to the Co-operative Bank but the collapse of that deal earlier this year means it is likely the TSB network will be floated on the stock market as a separate bank. The TSB is being unveiled a week before an industry-wide current account switching service is launched to reduce the hassle of moving a bank account.

One would also assume that any buyer would make sure that the buyer’s and seller’s systems were sufficiently compatible. In the US going back to the 1990s, many a promising-seeming banking deal was scuppered because the integration issues looked too hairy. So as much as it’s easy to point fingers at Lloyds, Sabadell is the one that should have made the call as to whether they could successfully port the data and needed routines from the Lloyds/TSB systems. As indicated, that has long been a major due diligence issue in US bank mergers.

Sabadell appears to have decided to move the TSB customers onto an entirely new system.

As one of our house IT expert Richard Smith pointed out, based on a Computerworld interview with the Sabadell CIO:

If I read this right (and it’s not just hype) the target system is “brand new”, which is to say, untested (or nearly untested). That’s a whole other dimension of extra risk.

“We are in the process of cutting the rope, and the first step is having a core platform up and running. We are now working with Lloyds Bank on the data migration,” Abarca told Computer Weekly. “We have built a new technology platform for TSB, but this is not just a technology refresh or upgrade of an existing core banking system. It is a brand new core banking system.”

They *did* do a pilot but clearly it wasn’t effective, for reasons we are yet to understand:

Proteo4UK, as the new platform in the UK is known, was rolled out to the bank’s staff in November 2017 with a full range of banking services. It will move to a full roll-out in the first quarter of 2018.

That suggests that the Sabadell knew its IT systems were not compatible with TSB’s and rather than do the sensible and normal thing, which would be to nix the deal, it went ahead based on the naive belief that it could build something new that would work. It’s hard enough to shake down a “new” retail bank IT system work, let alone roll it out by porting a ton of data from an old banking system into it.

We may have a better sense soon, but there are indications not just of data mapping problems, which potentially may be hard to isolate but not necessarily hard to fix once found, but of data corruptions, such as wildly incorrect account information (zero balances, incorrect currencies, massively inflated mortgage amounts, and e-mails saying that there are no records of recent direct debits). If there are indeed problems on the books and records level, and as we suspect, the changes can’t be rolled back, this could produce a world of hurt for customers.

Richard Smith found an example he calls “coughing up blood”:

Just want to see my balance and these guys @tsb think I’m robbing a bean factory with a bomb, jesus pic.twitter.com/VRosgbZoML — Jack Thomson (@thejackthomson_) April 23, 2018

His comments:

Implies one or both of a. there wasn’t enough integration testing (the phase of testing where you check that the intra-system interfaces are working correctly). b. there are corrupt data items, somewhere deep down, that are making the system behave in completely unexpected ways

Needless to say, we’ll be revisiting this topic once the press has more intel on the nature and severity of the IT mess. And as always, reader sightings and observations are of great help. But at least so far, this looks as if this might not be a gunshot wound, as bad as that would be, but gangrene.