High-profile technology disasters by big companies are getting increasingly commonplace. Millions of people have had passwords compromised, sensitive data stored or sent as clear text, and credit card information stolen.

There’s a repeating pattern of gigantic, profitable companies dropping the ball in terms of security or usability. Then, because they’re huge, they end up with a media frenzy of bad press.

My own credit card information was compromised by one of these high profile failures. I got bit by Apple’s iCloud/Core Data sync in a big way, and my brand new iPhone 6+ was nearly rendered useless by the 8.0.1 update. I’ve been on the receiving end of a lot of this and I’ve got insights on this as a customer, a vendor, and as a CEO.

If you want to see some scary and depressing reading, try this google search: site:forbes.com data vulnerability.

If you are a CEO or CIO in a company that makes software, I would REALLY be grateful if you would take this article to heart. I am not promoting fussy ideas about having a perfectly polished app, or discussing ivory tower land ideals about The Right Way To Test. I am talking about avoiding unpleasant and possibly very damaging public scandals.

Full disclosure: we do software testing — as well as strategic consulting — for companies, for money, so take all this with a grain of salt.

Your testing philosophy is ineffective

These problems are the result of a broken test philosophy:

1. Focus on verifying how much of the required functionality in the app is implemented and working.

2. Minimizing testing costs through automation and engaging offshore vendors

3. Worse yet, relying on your developers to do your testing

Point 1 is a project management metric, not any kind of security testing or quality assurance. Point 2 is magical thinking on the part of your CIO.

Point 3 is a terrible waste of your development team’s time and is also ineffective. A developer will routinely verify that basic functionality is in place but almost never has the time or mental space to put together a reliable system-wide test to identify any regressions introduced by their changes. I will be writing an article specifically dealing with this soon.

Let’s say that your low-wage team is capable of putting together and executing a solid functional test plan for your app on the other side of the planet by the time you ship (spoiler: they’ll still be working on it), this approach is futile for preventing the kinds of high-profile disasters that are making the front pages of Forbes and the Wall Street Journal.

This strategy offers little if any protection against unexpected misbehavior, and actually dumbs down the group that might have a chance of catching a problem before it’s released in the wild. It might even be considered willfully turning a blind eye to the risks.

Why I Value QA

QA is a business analysis tool that helps you understand how much risk your company faces if you ship your app.

The purpose of QA is better intel: the ability to perceive with the best possible accuracy the fitness of an app in a gestalt — to know its capabilities, weaknesses and behaviors as well as possible. Then the stakeholders make an informed assessment of the risk and potential liability of shipping the current release.

A crack team of test analysts knows how to assault your app, find most if not all of the places where it breaks, and catalog all of those malfunctions and misbehaviors so that you can make an informed choice about what needs to get fixed and what won’t get fixed this time around.

Let me repeat that: you will ship bugs, but you’ll do it on purpose, because you’ll understand the impact and will decide that the risk is manageable. Apple may have had some higher profile issues lately, but they routinely ship with known, manageable issues because shipping is a feature and there will always be another build.

This approach works a lot better. Many of the issues getting press coverage could have been caught by experienced, skilled analysts doing routine exploratory testing. Our team has caught at least one issue that made headlines when we came into a pre-existing codebase created by another vendor. By the time it came to any media attention we’d already reported the issue and submitted the fix to the client. We do a lot more security work for them now, I might add.

Overconfidence

I’ve shared this philosophy with executives at multibillion dollar businesses, and the responses have uniformly been dismissive. No one sees the value in spending money on testing, and no one takes the risks seriously, because they’ve never run into any problems before. There’s also a pronounced tendency to overestimate how good the teams they have really are.

The fact is, you want solid testing for a lot of the same reasons you want good automotive insurance — it’s a simple, cost-effective step towards avoiding any number of basic but potentially expensive problems down the road. It also really doesn’t matter if you’ve never had an accident before or might even be a really good driver, this is about statistics and risk management. Carrying insurance is mandatory in most states and few things will get people sighing and rolling their eyes more than hearing someone say, “I got hit. They weren’t insured.”

In this day and age, skimping on testing is about as reckless as driving without insurance, especially if you are a large and profitable company. Software developing and configuration is complicated, and your developers can’t test everything themselves, and your cut-rate testers are certainly not catching these things, either.

Likewise, the bigger your company is, the bigger the scandal when you screw up. If Black Pixel messes up on one of its own apps you aren’t going to read about it on wsj.com any time soon. But if you’re a Fortune 100 company you’ll be lucky if you can avoid the bad press and backlash.

Takeaways

Software testing is getting recklessly shortsighted. Sometimes the consequences are minor, sometimes they are very dire, and they always attract unwanted attention and can impact the strength of your brand and your business. Although a combination of functional and exploratory testing isn’t a panacea, it is a sensible prophylactic measure that can save your company millions in damages or even legal action.

Testing isn’t a formality, it’s a business analysis tool. It’s something you want sharp analysts to be doing on your behalf — don’t just rely on your development team to do it themselves. They can’t do it effectively themselves and they certainly can’t comprehensively regression test an emergency hotfix (see the iOS 8.0.1 situation)

If you are a CEO (maybe even if you are not):

- Ask your CIO what the purpose of QA at your business is.

- Ask your CIO how your company’s current QA strategy addresses this purpose.

- Ask your CIO how their QA strategy protects your customers, board, and stockholders against public security scandals.