High-profile technology disasters at big companies are becoming increasingly commonplace. Millions of people have had passwords compromised, sensitive data stored or sent as clear text, and credit card information stolen.
There's a repeating pattern of gigantic, profitable companies dropping the ball in terms of security or usability. Then, because they're huge, they end up with a media frenzy of bad press.
My own credit card information was compromised by one of these high-profile failures. I got bit by Apple's iCloud/Core Data sync in a big way, and my brand new iPhone 6+ was nearly rendered useless by the 8.0.1 update. Because I've been on the receiving end of a lot of these issues, I have some insights as a customer, vendor, and CEO.
If you want some scary and depressing reading material, try this Google search: site:forbes.com data vulnerability.
If you are a CEO or CIO in a company that makes software, I would be really grateful if you would take this article to heart. I am not promoting fussy ideas about having a perfectly polished app, or discussing ivory tower ideals about "the right way to test." I am talking about avoiding unpleasant and possibly very damaging public scandals.
Full disclosure: We do software testing--as well as strategic consulting--for companies for money, so take all of this with a grain of salt.
Your Testing Philosophy is Ineffective
These problems are the result of a broken testing philosophy:
- Focus on verifying how much of the required functionality in the app is implemented and working,
- Minimizing testing costs through automation and engaging offshore vendors,
- And, worse yet, relying on your developers to do your testing.
The first problem is a project management metric unrelated to security testing or quality assurance, and the second problem is magical thinking on the part of your CIO.
The third problem is a terrible waste of your development team's time and is also ineffective. A developer will routinely verify that basic functionality is in place but almost never has the time or mental space to put together a reliable system-wide test to identify any regressions introduced by their changes. I will be writing an article specifically dealing with this soon.
Let's say that your low-wage team is capable of putting together and executing a solid, functional test plan for your app on the other side of the planet by the time you ship; this approach is futile for preventing the kinds of high-profile disasters that are making the front pages of "Forbes" and "The Wall Street Journal."
This strategy offers little, if any, protection against unexpected misbehavior, and actually dumbs down the group that might have a chance of catching a problem before it's released into the wild. It might even be considered willfully turning a blind eye to the risks.
Why I Value QA
QA is a business analysis tool that helps you understand how much risk your company faces if you ship your app. The purpose of QA is better intel. Armed with a firm understanding of an app's capabilities, weaknesses, and behaviors, stakeholders can make an informed assessment of the risk and potential liability of shipping the current release.
A crack team of test analysts knows how to assault your app, find most, if not all, of the places where it breaks, and catalog those malfunctions and misbehaviors so that you can make an informed choice about what needs to get fixed and what won't get fixed this time around.
Let me repeat that: You will ship bugs, but you'll do it on purpose, because you'll understand the impact and know that the risk is manageable. Apple routinely ships with known, manageable issues, because shipping is a feature and there will always be another build.
This approach works a lot better. Many of the issues getting press coverage could have been caught by experienced, skilled analysts doing routine exploratory testing. Our team has caught at least one issue that made headlines when we came into a pre-existing codebase created by another vendor. By the time the issue gained any media attention we'd already reported it and submitted the fix to the client. We do a lot more security work for that client now, I might add.
Overconfidence
I've shared this philosophy with executives at multibillion dollar businesses, and their responses have been uniformly dismissive. No one sees the value in spending money on testing, nor do they take the risks seriously, because they've never run into any problems before. There's also a pronounced tendency to overestimate how good their teams are.
The fact is, you want solid testing for a lot of the same reasons you want good automotive insurance; it's a simple, cost-effective step toward avoiding any number of basic but potentially expensive problems down the road. Also, it doesn't matter if you've never had an accident before or even if you're a really good driver. This is about statistics and risk management. Carrying insurance is mandatory in most states and few things will get people sighing and rolling their eyes more than hearing someone say, "I got hit. They weren't insured."
In this day and age, skimping on testing is about as reckless as driving without insurance, especially if you are a large and profitable company. Software development and configuration is complicated. Your developers can't test everything themselves, and your cut-rate testers certainly aren't catching these things, either.
Likewise, the bigger your company is, the bigger the scandal when you screw up. If Black Pixel messes up on one of its own apps you aren't going to read about it on wsj.com any time soon. But if you're a Fortune 100 company, you'll be lucky if you can avoid the bad press and backlash.
Takeaways
Software testing is getting recklessly shortsighted. Sometimes the consequences are minor and sometimes they are very dire, but they always attract unwanted attention and can impact the strength of your brand and your business. Although a combination of functional and exploratory testing isn't a panacea, it is a sensible prophylactic measure that can save your company millions in damages or even legal action.
Testing isn't a formality, it's a business analysis tool. It's something you want sharp analysts to be doing on your behalf, not just relying on your development team to do themselves. You can read more about the importance of a dedicated QA team in "Working with Great QA." Your development team can't test effectively on their own and they certainly can't comprehensively regression test an emergency hotfix (see the previously referenced iOS 8.0.1 situation).
If you are a CEO (maybe even if you are not), ask your CIO:
- What is the purpose of QA at your business?
- How does your company's current QA strategy addresses this purpose?
- How does that QA strategy protect your customers, board, and stockholders against public security scandals?
Daniel Pasco
Daniel is the CEO of Black Pixel. He tweets at @dlpasco.