Filip Słomski, Author at Espeo Software

Recently a client asked me how I prove application testing is worth the money? For me, it’s obvious that testing is a vital part of the development process. Without it, there is little chance for a successful, high-quality product. So how do I prove it to someone who isn’t familiar with the reality of the business?

It is a fact that in almost any aspect of life higher quality comes at a higher price. Do you want to have a great, luxurious car? You need to pay more. Do you want to have a tailor-made suit? You better dig deep into your pocket. Software development is no different in this matter. If you want to have a fast, reliable and threat-secure application you need to invest in testing. But – in the long run — is it really worth the expense? First, we need to understand how an application development cycle looks and how it affects the bug-fix cost.

The software development cycle usually starts with requirements & design phases. This is where you outline the scope of the application, its main functionalities, visuals, architecture, test scope and any other requirements that you may have. This is a very important phase because any issue at this point may result in complications later on. If a developer overlooks a requirement it may be considerably more expensive to fix it later. If you decide to change a requirement at a later stage then the other parts of the application may get affected in a way you could have not anticipated. Fixing an issue at this stage is almost costless – all you need is to rethink and rewrite it or add some new requirements.

The next phase is the actual coding part. Web application development teams usually consist of several people who measure the code in thousands of lines. Each line of code is a potential place for a defect. The more complex the software is the more interdependencies between modules, integrations and architecture pieces there are. Developers are also human and in this jungle of advanced logic, code standards, abstract end-user ideas and external system integrations it is very easy to overlook something. They also have to consider the fact that different browsers/devices often operate in a different manner. There are also changing requirements and bug fixes that require adjustments in code already written. It often happens that one part of the code affects some other unrelated functionalities. Finally, there is also a business pressure to meet the agreed deadlines which can sometimes make the developers work in a hurry. And, this is just the tip of the iceberg. An issue usually found at the development stage is fairly easy to fix as the developer is aware of the code he has just written and it doesn’t take much time to isolate the specified issue. Still, it’s much more costly than a simple change in the requirements phase.

In agile development methodologies testing happens alongside the development process. You may ask yourself a question: if testing takes place, there should be no issues on production, right? Well, not exactly. First of all, it is not feasible to test all the possible inputs and outputs of a complex application. Moreover, it is not possible to test the application for all browser and device combinations in all possible versions. Finally, the user’s creative ways are sometimes futile to predict – it is not possible to come up with all the things that a user may think of doing and prevent them from happening.

There is also a matter of maintaining the application. At times, the new software version may clash with the application and cause bugs. From time to time, the external integrations get updates without updating the code. This, in turn, may cause the app to malfunction. Although it may sound strange sometimes we also choose not to fix some of the bugs. If we know that a defect only causes a minor discomfort for a limited number of users it isn’t sometimes worth fixing because the costs would surpass the actual damage caused by this issue.

Despite the fact that a well-chosen test strategy may minimize the project risks and eliminate all critical issues, it still cannot guarantee an issue-free project. So what is the cost of fixing a bug at the testing stage? It’s even higher than in the development stage. Usually a tester, a developer and, sometimes, a PM decide on the priority of the bug. It is also much more difficult for a developer to diagnose the root cause of the issue. Is it architecture? Does it occur only for a certain set of data? Is it a bug within the code? Is it possible to reproduce it on my machine or is it only test environment related? A developer has to find an answer to these questions and it takes more time than investigating his own code and functionality.

The last phase of the development cycle is when an application is already deployed to production. The cost of fixing bugs on production is significantly higher, sometimes it can be incredibly large. Even setting aside the hard-to-measure costs of damage to company reputation, users leaving the application or customers not being able to complete their purchases – the cost of fixing the bug is higher. The issues that users find are usually much more difficult to reproduce in the development environment. They’re often connected to a specific production architecture or database state. On top of everything, the users usually don’t provide as many details as a qualified software tester which prolongs the investigation. The table below presents some real-life examples revealing how much an issue can affect a company’s revenue.

So is it even possible for software to be bug-free? Well, let me answer that question with another question. Do big companies with a huge amount of resources – like Apple, Google or Facebook provide bug-free software?

Knowing all that and based on the above explanations and estimated bug-fix costs provided by IBM data we can perform an example Return On Investment(ROI) analysis for testing.

Let’s try to analyze an ROI example for a 3-month project. First, let’s make the following assumptions:

1. Assuming that bug-fix cost during the design phase is $10 using the above IBM estimated costs we get:

Design bug-fix cost – $10
Development bug-fix cost – $65
Testing bug-fix cost – $150
Production bug-fix cost – $1000

2. There are 100 must-fix bugs during a 3-month period and they all cost approximately the same.
3. None of the issues were found during the design phase (which could have lowered the total cost even more)

Now we have to estimate what the cost of manual tests would be and test automation as well as the number of bugs to find in each phase. Assuming that basic manual application testing costs around $2000 per month (that’s the salary of a specialist) and adding some minor infrastructure fee for the test environment we receive a total investment of $6,500. When it comes to test automation let’s increase the specialist’s salary as well as the infrastructure & setup cost – then we get $11,000.

Let’s estimate that out of 100 total bugs 25 of them will be found by the developers (this is a constant number no matter whether we carry out tests or not). We can estimate that during the testing phase the testers will find 35 issues while performing manual tests and 50 when we add automation on top of that.

The rest of the bugs will be found by the end-users. Now we can count the total cost of quality as the sum of conformance (cost of ensuring that the app conforms to quality requirements) and non-conformance (money that needs to be spent for not conforming to the quality requirements). Finally, we can count the test ROI as the value brought by testing divided by the cost of actual testing.

This example certainly contains numerous assumptions but you can use it in an actual project. It shows how we can actually estimate ROI. I believe that it provides some insight to whether application testing costs are worth the money. Even though hiring a testing team is, in fact, an expense, when you look at your entire application development cycle you will view it as an asset. Testing will spare you a lot of future quality expenses. This also causes things like user frustration, clients leaving and even company reputation damage – factors which are difficult to measure in terms of money.

Types of Web Application tests

Manual tests are the cornerstone of each successful project, each functionality has to be tested –and often for many devices/browsers – whether it meets the specification requirements, each edge case needs to be tested thoroughly to prevent an end user from exploiting any security threats or causing a web page crash. Many of those manual tests can and should be automated in order to reduce the amount of time required for testing. Test automation is a development branch that requires special tools and frameworks where each test is fundamentally a script, so a skilled automation engineer should also follow all good code practices and standards.

Another area are performance tests which can be divided into several subareas and strategies. A very common approach is to just test the application response times, however, more detailed tests check things like: How does an application respond if there is high traffic? How many users can enter the page simultaneously before it crashes?

Does application performance degrade over time which implies some memory leaks? What is the page load time in Hong Kong in comparison to Atlanta? Thorough performance tests can provide answers to those questions and help find potential problems and bottlenecks.

Security tests are meant to find as many potential security issues as possible. They are conducted in order to locate and fix security vulnerabilities which could result in hacking the site, sensitive data being stolen or crashing the servers. They usually consist of the use of various automated tools that check for the most common threats but may also need some manual tests which require the tester to think as a hacker and predict possible issues.

There are some other testing approaches as well, accessibility testing focuses on adjusting the application so it is usable by people with issues like hearing impairment, color blindness, infirmity and old age or other problems; usability testing focuses on testing how easy and intuitive the page is; and regressions tests which constantly ensure that none of the already existing functionalities breaks and that quality does not deteriorate.

Requirements for successful Quality Assurance

As you can see testing web pages consists of many areas that often require a specialist to be able to prepare and conduct a valuable set of tests. In order to test application security one has to be up to date with current threats, be an expert in the area of system architecture and proficient as a developer as well. In order to provide a successful test automation a skilled developer with test experience and devops knowledge is a must.

Finally the testing requires effort from the whole team in order to be successful. This is very often overlooked and people think that simply adding the testers to the team is enough to improve application quality. However what everyone needs to be aware of is that it also slows down the development process due to the fact that it now incorporates testing.

So every new feature has to be manually tested, all defects need to be documented and fixed, also the automated tests need to be recorded and maintained as well as regularly updated to match the latest changes in order to produce benefits. Any additional testing activities will also usually require new testing environments and will further delay the release date. However, there is a huge advantage of doing these kinds of tests and I will discuss this in the next part of this article.

The Benefits of Testing

So what are the actual benefits of testing and should it be done? Well, let me give you an example. Let’s say that you have ordered a brand new car that you expected to be the best in all areas and you have just received it. At first glance everything looks fine but after one day of using it you notice that the top speed and acceleration is actually much worse than with your old car.

Moreover the door only occasionally locks which makes it very prone to theft. There are also a few issues inside the car, the gear box does not work properly, the radio is broken and in order to use the turn-signal you need to press 2 buttons that are located on the backseat of the car because someone thought that this may have been a good idea despite of what the standard is.

In fact this is not an extreme example, because some web pages look more like a car that could never even start its engine, never mind drive and yet there they stand, proud members of the online community. The benefits of testing affects both the team and the business-side. Testers will increase the team’s confidence before release and test automation constantly ensures the developers that their code changes do not introduce new issues.

Testers also make sure that the new functionalities are consistent with requirements and make it much easier to introduce all kinds of bigger changes in the application (technology stack updates or application changes) due to automated tests. From the business perspective the end-product is of higher quality, the risk of introducing a defect into the production environment is much lower, which is the same as the possibility of weak performance or security issues.

Summing up

If you don’t have time to do it right, when will you have time to do it over?

Without proper tests development costs rise over time as many features have to be done again or completely changed due to bad design decisions. Moreover software becomes more bug-prone, the usability suffers and the overall quality deteriorates which in some cases may lead to huge exodus of the application users.

To sum up quality assurance is crucial in web application development and it has been present in the industry almost since its beginning and is growing each year, being present in all informed companies development processes.