It will take JPMorgan Chase days, if not weeks, to get to the bottom of what caused a three-day outage of its consumer online banking system on Monday, Tuesday and Wednesday, cybersecurity experts say.

The bank has issued this terse apology to its online banking customers. Bank officials also have indicated to the Wall Street Journal and to the New York Times that they have ruled out an online attack and believe the outage stems from problems with a third-party software vendor.

However, cybersecurity experts say investigators have a lot of digging to do before they can confidentlyÃ‚Â isolate the specific trigger. Monash Research principal Curt Monash calls the bank’s explanation “ambiguous and vague.”

Given the scale and complexity of the bank’s online systems it is much too early to rule out some sort of malicious attack as possibly contributing to the outage.

“It’s like finding a needle in a field full of haystacks,” says Jerry Skurla, executive vice president at Nitro Security,Ã‚Â a security monitoring company. “You know the needle is there but you have to find out where it is, and more importantly, figure out how it got there. It takes time.”

Jill Eckhaus, CEO of datacenter association AFCOM, says enterprise networks are “extremely vulnerable” to outages, since a wide spectrum of users have multiple ways and devices they can use to access data stored in data centers. “This includes employees who work from home, cyber criminals, hackers, smart phones, etc.,” says Eckhaus. Securing all outside access points is a big challenge, she says.

Eckhaus says investigators will have to systematically assess and rule out the obvious, including “the nature of the corruption Ã¢â‚¬â€œ was the entire database corrupted? Was it only a specific file?” she say. “This will help rule out whether or not this was a cyberattack. It may well have been a virus that got into the database they received.”

Corporate networks today are built to almost never fail, and certainly not for days at a time. That’s especially true for online banking systems that process millions of transactions minute-t0-minute.

“These systems are incredibly resilient, yet also very complex due to the number of moving parts Ã¢â‚¬â€ software application, database, servers, storage, network, as well as the teams of people responsible for each,” says Brian Reagan, marketing manager at data storage company Xiotech. “There are levels of redundancy built into each layer, so a failure would involve multiple breakdowns.”

Online banking systems, in particular, are built to withstand most known failure scenarios, with back up systems at the ready to ensure crucial online services remain accessible to the public. JPMorgan Chase “lost millions every day that the site was down, so the problem had to be something big,” says Caleb Sima, CEO of web site security company Armorize Technologies.

Earlier ‘scheduled maintenance’ outage

JPMorgan Chase’s consumer online services also went off line for 15 hours on Aug. 7 through Aug. 8. Many customers, including Steve Karp, a 57-year-old computer consultant from Tuscon, Ariz., were taken by surprise. A bank spokesman told LastWatchdog on Aug. 9 that the outage was part of scheduled maintenance. But Karp, who checks his online accounts frequently, said he never received notification from the bank.

“I question how truthful Chase was about this matter,” says Karp.

Fast forward to this week’s unscheduled outage. Despite bank officials deflecting blame to an outside vendor, thorough tech forensics remains to be completed. Investigators will have to methodically examine whether they can rule out complicity of an insider, perhaps a disgruntled employee. And it will take extensive legwork to confidently rule out whether hackers who are expert at hiding their tracks might be to blame. There is precedent for both.

In 2008, a disgruntled city employee corrupted the City of San Francisco’s fiber optic network locking out the mayor and other administrators. And in 2007, hackers rigged the Bank of India’s Web site to infect the PCs of customers trying to access their accounts. The site went off line for several days while investigators and technicians restored the system to a clean state.

‘Advanced persistent threats’

Cybercriminals today routinely use cutting-edge tactics, referred to as “advanced persistent threats” (APT) that allow them to control parts of a network and dodge detection for long periods of time. Typically, when APT intrusions are finally discovered, it takes months more to completely clean up. “APT hackers are extremely brilliant at hiding,” says Skurla.

It will take a deep investigation to determine with a high degree of certainty that no security breach was involved. It is possible that a string of programming errors unrelated to any malicious activities is to blame, says Adam Powers, chief technical officer of Lancope, which supplies network monitoring systems.

“An organization can build out a beautifully redundant system but the software itself is always a potential failure point, ” he says.

Either way, JPMorgan Chase’s 16 million or so online banking customers should be concerned. Powers says bank patrons should ask, “has my information been corrupted? A corrupted financial database can lead to lost transactions, erroneous account balances, missed deposits, etc.”

By Byron Acohido

September 16th, 2010 | For consumers | Imminent threats | Top Stories