Rethinking QA, Part II

By Daniel Pasco

Looking for designers or developers for your next project?

Black Pixel offers design, development, and testing services backed by unrivaled experience.

Hire Us

What working with bad QA looks like

Your QA team is a train wreck. You're part of a product development team, and your company's QA team is the bane of your group's existence. They are the lowest paid, least motivated, worst informed, and most technically inept people in your company.  



They are the last people to know anything about what the product is actually supposed to do. Their team is the only one that doesn't realize that the requirements for the app they are working on were changed last week.

They open multiple tickets on issues that all stem from the same root cause, failing to understand that the same issue is manifesting itself in different parts of your app.

They randomize your developers hourly with questions they should already know the answer to. They do not know what should be working in the app yet, and what should not. They also have a poor understanding of the platform your app runs on: they open bugs on your app because they have not granted it permissions to access your photos, so they are - wait for it - unable to access their photos in the app.

They test with the wrong builds, or run against the wrong environment. They use incorrect test configurations, and submit horrifically vague bug reports without any configuration information or steps that can be used to reproduce the reported issues.

Rest easy, citizen. These are the pros charged with setting the bar for quality on your team.

Why Does This Happen?

Maybe you've never had experiences like the ones mentioned above. You are fortunate.

We've worked with self-proclaimed QA experts where our own developers have handled most of the QA on an application, while the client's appointed QA team burned through cash without *any* relevant work being performed.

We've also seen people move their QA people to remote locations within their offices in order to keep them from bothering or interfering with their developers, indicating the extreme degree to which the QA staff had become a hindrance to the rest of the company.

These experiences are commonplace, and it's not the QA team's fault. The problem is institutional.

There are a rare number of effective QA teams out there, often limited to companies that are focused solely on the business of developing software and little else. Most of the companies that do not write code as their primary business - even some of the larger media companies with multiple apps under their belts - tend to miss the boat as far as this is concerned.

This means that if you're reading this and thinking to yourself "Well, at least we are doing QA right", the odds are good that you probably aren't.

How Does This Happen?

People understand that a QA team is something they're supposed to have in their company, but they don't always understand what a healthy QA organization looks like.

James Shore identified a corruption of Agile Software Development he called "Cargo Cult Agile",

The tragedy of the cargo cult is its adherence to the superficial, outward signs of some idea combined with ignorance of how that idea actually works.

BAM. That's the key issue - in a different area, but exactly the same problem.

In the story, the islanders replicated all the elements of cargo drops--the airstrip, the controller, the headphones--but didn't understand where the airplanes actually came from.

Instead of building something great, they try to figure it out themselves or engage similarly uninformed people to put these organizations together. And by "similarly uninformed", I don't mean, "small". A lot of these managers come from very large companies that endorse institutional inefficiency at a grand scale.

I see the same tragedy occurring with Agile. Many of the teams I know have adopted just two aspects of agile development: stand-up meetings and biweekly planning sessions. Nothing else.

But you know what? Stand-ups are one of the least important aspects of agile development. In a way, they're an admission of failure.

Stand-up meetings and biweekly planning sessions don't mean you're doing Agile. Likewise, although a lot of people know that you're supposed to have a test team, a QA director, and a test matrix, that doesn't mean at all that having them means you're assuring quality.

What it really comes down to how a given company answers the following two questions:

  • What is QA's purpose?
  • How does the QA team fulfill that purpose?

I'm going to compare and contrast a few different perspectives on this in my next posts, so stay tuned.


Up Next: Rethinking QA, Part III - The Downward Spiral

Daniel Pasco

Daniel Pasco

Daniel is the CEO of Black Pixel. He tweets at @dlpasco.