Z is for ZERO (My A to Z of Software Testing, Part 2)

Having begun the My A to Z of Software Testing series last week with A is for Apathy it seems only appropriate (at least in my part of the universe) to jump straight to the other end of the (alphabetical) scale and dive into Z is for Zero. The particular aspect of Zero that I’m going to explore today is the concept of ZERO DEFECTS. The main reason that I am addressing this topic is that it is the main
discussion topic at our Test Engineering Alliance Meetup this week; so, in effect, this is my homework in preparation for our Meetup.

Just as a little bit of background (for those new to the term), ZERO DEFECTS was originally pioneered by Philip Crosby and then popularised via the TQM and Six Sigma concepts. In it’s purest sense, ZERO DEFECTS aims to create an environment where absolutely no defects escape when a product ships. In my world this means reducing creativity and stifling innovation to the point where everything is controlled to the ‘nth degree.

An excellent example of the adverse impact on software development occurred about 10 years ago when a major Australian Retail Bank struggled with several major software failures and reacted by locking down their SDLC to the point where a significant number of their development staff left due to the overly restrictive environment within which they were expected to work. This was obviously an over-reaction to a series of sub-standard software releases, but it does highlight how the SDLC can be misunderstood – even by major organisations. It is my opinion that this Bank has never shipped a major software release with zero defects and will never achieve this in the foreseeable future. This is not to say that their SDLC processes and procedures are flawed, it is just to recognise that they operate in a very complex and rapidly changing environment that makes it mathematically impossible to achieve a zero defects outcome while meeting budgetary and time constraints. 

Looking at the ZERO DEFECTS issue from another perspective, I was hired several years ago to assist a Scandinavian transport authority in implementing a smart ticketing system. The organisation was over 5 years into this ticketing transformation and was on the point of cancelling the project when they tried one last strategy – they called in a specialist team (including myself) and gave us 6 months to achieve a Go Live outcome. During my first week on the project I discovered that over 6,000 defects had been logged and fixed by the main vendor and several hundred were still outstanding. When we achieved Go Live, there were still over 100 known defects in the pipeline. The unenlightened of you will wonder how we could Go Live in this situation. The main reason we achieved our goal (with just 5 days to spare) was the excellent job we did in our root cause analysis of every defect from that first week that I arrived on the project. A detailed risk assessment of the status of the hardware and software implementation (underpinned by our root cause analysis work) meant that the board of the organisation had sufficient confidence that there was no significant reason to hold up implementation further. Another satisfied client and an even happier vendor – the vendor was staring down the barrel of a multi-million dollar lawsuit for failure to deliver on the contract!!

One final example for now… I was working with another major Australian bank in Melbourne about 8 years ago, when one of my Test Management colleagues came to me with a conundrum. Her Project Manager had made a statement, in their latest Project Status Meeting, that UAT would not commence until System Testing and Integration Testing had achieved zero defects for 3 consecutive test cycles (normally 3 calendar days of testing). My colleague was already feeling under pressure as both of her parallel project test phases were behind schedule and the number of defects was still climbing. She asked me if she should negotiate a non-zero number of (low impact) defects.

This was my advice to her – “This is not a negotiation, this is a total misunderstanding of what is achievable (on any project).” I continued by explaining that I could achieve zero defects within days by removing all test cases that had not already passed, thereby guaranteeing every test passing. Now, I’m obviously not advocating that we only run tests that we know will pass, I’m just pointing out that the only way you can ever guarantee achieving zero defects is to perform zero tests.

In summary, there are many (conflicting) priorities with software development initiatives and between them they conspire to make ZERO DEFECTS a financial (and long-term organisational) nightmare. An acceptable level of quality and risk should always be agreed during the first stage of any software testing initiative; however, if a specific number of DEFECTS is ever mentioned (especially zero) you are already on a hiding to nothing. The SDLC is not about defect numbers it is about agreed outcomes within agreed times-frames and to an agreed budget. 

Dateline: Melbourne, Monday March 14, 2016


2 thoughts on “Z is for ZERO (My A to Z of Software Testing, Part 2)

  1. The only way to get zero defects is to stop testing immediately. Management saying that they want this directly influences testers to test less.

    I sometimes wonder if they really mean “all known defects fixed”. Which is a more reasonable request even if a bit pricey on some projects. But I digress.

  2. Exactly!

    Good point that if you report zero defects you are hiding something. So once I was in a project where we had found some serious issues before go-live, but we couldn’t due to “acceptance criteria”. So we agreed with the customer to dial the severity down. .. Which really is what the whole thing is about, not the pedantics.

    It’s not everything that counts that can be measured,
    and not everything that can be measured that counts.

We're here to help

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s