This post is part of a series of blogs on Atlassian QA. We will cover how the QA strategy has been implemented in different teams, the tools and techniques we use, and the personal experiences from members of the team.
Traditionally, software QA is often seen as an impediment to software development; a necessary evil that gets in the way of delivering features to customers. QA time is dutifully scheduled on the project plan, but as deadlines get uncomfortably closer, it’s the first item to be squeezed out to save time. It’s a tricky tradeoff – more time on testing means higher quality, so improving quality means slowing down development.
When I joined Atlassian over four years ago, I learned that the Atlassian QA goal was quite ambitious: not just to improve quality, but speed up development as well, with only a small number of people (roughly one QA engineer for every 10 developers). To me, this seemed impossible. Aside from the practical complications, how could it even make sense on the theoretical level?
To explore this concept, let’s fall back to an old favorite – the overextended metaphor.
The metaphor
Software development is a highway with lots of trucks driving along it, delivering goods to customers.
- The goods in the trucks are the customer value we’re delivering
- The truck drivers are our developers
- The highway is the development process
- QA engineers are the traffic cops, trying to keep the traffic flowing smoothly
- Rocks on the road are technical debt, workarounds and hacks from previous development work, frameworks that don’t work as you’d expect, hidden overrides, configuration options you didn’t know about, and fragile legacy code.
The problem
Unfortunately, for our more mature products, driving on the highway can be quite hazardous. The older the highway is, the more rocks there are on the road. We hire some of the best truck drivers in the world; people who are used to driving on nice roads at full speed. But when they drive at full speed on our highways, accidents happen and the customers don’t get the goods in the state they wanted.
As traditional software testers coming across this situation, our natural response was to lower the speed limit and spend our days sitting by the side of a road with a radar gun, issuing speeding tickets (i.e. raising bugs). This prevented a lot of accidents, got a lot of trucks to their destinations that would have otherwise had nasty crashes, and prevented damage to the goods in the trucks. It was a semi-successful approach, one that’s used in many software companies today.
However, it had some serious drawbacks:
- It was insufficient. We couldn’t catch every speeding truck, so accidents kept happening that impacted our customers.
- It didn’t scale. For every truck we pulled over, 10 more would speed by while we were writing the ticket. We would have needed to keep hiring until we had as many cops as truck drivers.
- We didn’t know the road very well, or knew what caused accidents. Many members of the QA team are experts in speed camera use, but don’t have driver’s licenses. Thus, we were playing it safe and trying to get people to drive slowly even on the good bits of the road,
- It made us responsible for accidents. People would ask, “Why didn’t you stop that speeding truck? We spend so much money on speed cameras and police salaries!” This put a lot of stress on us.
- It made us unpopular. As we were there to catch developers who broke the rules, it set up an antagonistic relationship between developers and testers. We worked so hard on improving quality, but we felt all we got was criticism for slowing things down. When a driver receives a speeding ticket, they rarely thank the cop for making their journey safer!
Worst of all:
It was easy to get so focused on writing speeding tickets that we forgot that our aim was getting the goods to the customers.
The solution
As we’ve described in previous posts, the Atlassian QA strategy is more about prevention and trust. As QA, we know the roads really well – we know what causes accidents, and where the accidents have been in the past. So instead of sitting by the road with a radar gun to catch speeding drivers, we sit in the cab with the drivers and say, “careful in this next bit, there’s a huge rock in the middle of the road after that blind corner.” In safer stretches, we say, “There’ve been no nasty accidents in this stretch in the past, full speed ahead!”
This has the following advantages:
- It’s more efficient. Instead of the trucks driving slowly along the full stretch of the highway, they only need to slow down specifically when there are known hazards.
- It provides QA scalability. We don’t need to be in the truck for the whole journey. We can pop in to each truck, give a word of advice about the upcoming road, then pop out again.
- It lets the drivers focus on the road. While driving, the developers don’t need to split their attention between the road and looking out for sneaky QA cops.
- It gives clearer cause and effect. In the traditional approach, there was a level of indirection – the cops – between the drivers and the often-unseen hazards. When we set the speed limits low due to black ice on the road, the drivers obediently slowed, but didn’t realize the hazard they’d avoided. In QA, we’re not highway engineers – we don’t have the skills or time to improve the roads. But by keeping the drivers constantly aware of the true hazards, it inspires them to organize or join road-clearing projects.
- It’s flexible. As highway engineers clear up stretches of road, we adapt our advice to match. We love being able to say, “This road was previously covered in rocks but now it’s totally safe, you can go fast here now.”
This helps the drivers get to their destinations more quickly because:
- They can drive quickly in the safe sections of road (i.e. low-risk stories are identified early, and little testing is needed for them).
- There are far fewer accidents (i.e. fewer stories have bugs or usability issues that require rework).
- They don’t get stopped by traffic cops writing tickets (i.e. when bugs are prevented, QA doesn’t need to write bug reports and reject their stories).
- The roads get improved over time as the time-consuming hazards are identified and fixed (i.e. parts of the system are rewritten so that the average dev is less likely to be tripped up by fragile components, unexpected behavior, or odd edge cases).
Back to reality
What does this just-in-time QA advice look like in practice? For the Jira QA team, you can read further details in this post.
By being intelligent about risks, by knowing the product well, and by demonstrating that we have the same goals as the developers, we can earn their trust more easily.
And in the end, we can work with them to deliver value to our customers, faster, and with higher quality.