Category: Technology

Recently a client asked me how I prove application testing is worth the money? For me, it’s obvious that testing is a vital part of the development process. Without it, there is little chance for a successful, high-quality product. So how do I prove it to someone who isn’t familiar with the reality of the business?

It is a fact that in almost any aspect of life higher quality comes at a higher price. Do you want to have a great, luxurious car? You need to pay more. Do you want to have a tailor-made suit? You better dig deep into your pocket. Software development is no different in this matter. If you want to have a fast, reliable and threat-secure application you need to invest in testing. But – in the long run — is it really worth the expense? First, we need to understand how an application development cycle looks and how it affects the bug-fix cost.

The software development cycle usually starts with requirements & design phases. This is where you outline the scope of the application, its main functionalities, visuals, architecture, test scope and any other requirements that you may have. This is a very important phase because any issue at this point may result in complications later on. If a developer overlooks a requirement it may be considerably more expensive to fix it later. If you decide to change a requirement at a later stage then the other parts of the application may get affected in a way you could have not anticipated. Fixing an issue at this stage is almost costless – all you need is to rethink and rewrite it or add some new requirements.

The next phase is the actual coding part. Web application development teams usually consist of several people who measure the code in thousands of lines. Each line of code is a potential place for a defect. The more complex the software is the more interdependencies between modules, integrations and architecture pieces there are. Developers are also human and in this jungle of advanced logic, code standards, abstract end-user ideas and external system integrations it is very easy to overlook something. They also have to consider the fact that different browsers/devices often operate in a different manner. There are also changing requirements and bug fixes that require adjustments in code already written. It often happens that one part of the code affects some other unrelated functionalities. Finally, there is also a business pressure to meet the agreed deadlines which can sometimes make the developers work in a hurry. And, this is just the tip of the iceberg. An issue usually found at the development stage is fairly easy to fix as the developer is aware of the code he has just written and it doesn’t take much time to isolate the specified issue. Still, it’s much more costly than a simple change in the requirements phase.

In agile development methodologies testing happens alongside the development process. You may ask yourself a question: if testing takes place, there should be no issues on production, right? Well, not exactly. First of all, it is not feasible to test all the possible inputs and outputs of a complex application. Moreover, it is not possible to test the application for all browser and device combinations in all possible versions. Finally, the user’s creative ways are sometimes futile to predict – it is not possible to come up with all the things that a user may think of doing and prevent them from happening.

There is also a matter of maintaining the application. At times, the new software version may clash with the application and cause bugs. From time to time, the external integrations get updates without updating the code. This, in turn, may cause the app to malfunction. Although it may sound strange sometimes we also choose not to fix some of the bugs. If we know that a defect only causes a minor discomfort for a limited number of users it isn’t sometimes worth fixing because the costs would surpass the actual damage caused by this issue.

Despite the fact that a well-chosen test strategy may minimize the project risks and eliminate all critical issues, it still cannot guarantee an issue-free project. So what is the cost of fixing a bug at the testing stage? It’s even higher than in the development stage. Usually a tester, a developer and, sometimes, a PM decide on the priority of the bug. It is also much more difficult for a developer to diagnose the root cause of the issue. Is it architecture? Does it occur only for a certain set of data? Is it a bug within the code? Is it possible to reproduce it on my machine or is it only test environment related? A developer has to find an answer to these questions and it takes more time than investigating his own code and functionality.

The last phase of the development cycle is when an application is already deployed to production. The cost of fixing bugs on production is significantly higher, sometimes it can be incredibly large. Even setting aside the hard-to-measure costs of damage to company reputation, users leaving the application or customers not being able to complete their purchases – the cost of fixing the bug is higher. The issues that users find are usually much more difficult to reproduce in the development environment. They’re often connected to a specific production architecture or database state. On top of everything, the users usually don’t provide as many details as a qualified software tester which prolongs the investigation. The table below presents some real-life examples revealing how much an issue can affect a company’s revenue.

So is it even possible for software to be bug-free? Well, let me answer that question with another question. Do big companies with a huge amount of resources – like Apple, Google or Facebook provide bug-free software?

Knowing all that and based on the above explanations and estimated bug-fix costs provided by IBM data we can perform an example Return On Investment(ROI) analysis for testing.

Let’s try to analyze an ROI example for a 3-month project. First, let’s make the following assumptions:

1. Assuming that bug-fix cost during the design phase is $10 using the above IBM estimated costs we get:

• Design bug-fix cost – $10
• Development bug-fix cost – $65
• Testing bug-fix cost – $150
• Production bug-fix cost – $1000

2. There are 100 must-fix bugs during a 3-month period and they all cost approximately the same.
3. None of the issues were found during the design phase (which could have lowered the total cost even more)

Now we have to estimate what the cost of manual tests would be and test automation as well as the number of bugs to find in each phase. Assuming that basic manual application testing costs around $2000 per month (that’s the salary of a specialist) and adding some minor infrastructure fee for the test environment we receive a total investment of $6,500. When it comes to test automation let’s increase the specialist’s salary as well as the infrastructure & setup cost – then we get $11,000.

Let’s estimate that out of 100 total bugs 25 of them will be found by the developers (this is a constant number no matter whether we carry out tests or not). We can estimate that during the testing phase the testers will find 35 issues while performing manual tests and 50 when we add automation on top of that.

The rest of the bugs will be found by the end-users. Now we can count the total cost of quality as the sum of conformance (cost of ensuring that the app conforms to quality requirements) and non-conformance (money that needs to be spent for not conforming to the quality requirements). Finally, we can count the test ROI as the value brought by testing divided by the cost of actual testing.

This example certainly contains numerous assumptions but you can use it in an actual project. It shows how we can actually estimate ROI. I believe that it provides some insight to whether application testing costs are worth the money. Even though hiring a testing team is, in fact, an expense, when you look at your entire application development cycle you will view it as an asset. Testing will spare you a lot of future quality expenses. This also causes things like user frustration, clients leaving and even company reputation damage – factors which are difficult to measure in terms of money.

See also:

• Good reasons not to skimp testing
• Client’s engagement in an application development project? What involvement is needed for successful delivery?
• Web Application Testing: Terms of Quality in Web Applications
• Great application development starts with a product design workshop

 
[contact-form-7 id=”13387″ title=”Contact download_8_reasons”]
 

In software engineering, unit testing refers to testing individual units of source code. They’re part of the application code, written during development. Unit tests are powerful tools for developers, who can use them repeatedly to quickly and automatically fast check if the units of code work as they should.

However, writing them takes time, and since you can deliver an application without unit tests, some entrepreneurs unwisely consider them unnecessary. Time is money after all and some clients and IT companies don’t want programmers to spend time writing tests. They assume that manual tests are sufficient, so they can hire someone who’ll test their system or they can do it themselves — which will reduce short-term costs. Some clients are unaware that unit tests are a thing. In either case, knowing the benefits of unit testing will help you understand that having them actually saves considerable time and money.

Why unit tests matter

Before I get into the details, I’d like to share a story. I recently worked on a lottery gaming application where my job was to write one of two REST APIs. Work on the first service started about a month earlier than mine. The previous team did not write any automatic tests for the first API.

But the customer was content with the production method because he received the first several features quickly. When work on the second service started, unit and integration tests were part of the code production. Initially, the product owner had to wait longer for results and was not happy. There was some pressure to give up writing tests and leave them for later if there was enough time. I had to convince the product owner that this would be a huge mistake that will cost in the long run — especially when it comes to long-term maintenance. 

Over time, the benefits of unit tests — and the contrast between the two APIs grew. In the service with tests, when a developer added a new feature, manual testers detected almost no errors. In this version, the team could do their work systematically. By contrast, the project without tests often came back for corrections, disrupting the flow of the project. Over time, more developers joined the second service team. They could quickly adapt and implemented new features. The client also attempted to enlarge the first team, but it was ineffective and they ultimately abandoned it.

Throughout the whole development process of the second API, the developers were more productive and proud of code they wrote. Even though it took longer to start, jumping into a development project without unit tests leads to major headaches later.  

Don’t skimp on unit tests

As I mentioned above, the client wanted to give up writing automated tests, as he thought that manual tests are enough. This is due to the superficial treatment of the testing process. Many people think that once a tester tests an application’s functionality, he can move on. There is no need to do it again unless they need to change the feature behavior. It seems easy, but there are some dangers and gaps in this reasoning.

First of all, many people know exactly what they want and tend to think that there won’t need to be changes. But no matter how detailed the initial app specification is, after receiving the feature and using it, ideas of how to improve it always emerge. This is a normal and healthy part of Agile development.  In addition, something that may seem like a small change may actually require changes in many areas. What’s more, the main functions of the system consist of smaller functionalities — and these are often shared. Making changes in one area may affect others — often in places we don’t expect. 

A tester may need to test a feature many times as its behavior changes. A developer can use a test repeatedly for months or even years as long as there is a part of the system that it checks. The effort put into writing it will pay for itself in the long run.

How unit tests work

Unit tests usually run for a minute or two, so you can use them very often. Developers can launch them several times when adding a new functionality to the system. Thanks to this, they can detect defects early so they can find and correct them quickly. Manual testers can only test a new function once it goes live. Finding the cause of the error and fixing it at this stage often takes longer and is more difficult because it concerns a larger area of code. 

What’s more, manual testers don’t always check the code right after they’ve finished their work. Sometimes a developer can go several days or even weeks without them. To fix this type of error after such a long time requires digging down into the code again — time that could go toward moving the project forward. However, the key question is how can rapid error detection affect the stability of the project? Project complexity always increases over time. The more complex the project is, the more internal code dependencies it has. Introducing new functions often requires changes to the existing code. Errors are inevitable and can often occur in unexpected places.

If we can automatically check if the basic unit functions are preserved, and errors that were found are fixed at the developing stage, total functionality delivery time is much shorter.  We reduce the impact that system complexity has on the introduction of new functionalities and changes. We gain greater predictability regarding the time and resources needed to implement new features or to change existing ones. 

Unit tests are not only useful for checking the existing units. The very process of writing them also improves the stability of the project. Unit tests affect how developers write the application code. They force the programmer to divide the code into small units with a clearly defined, single responsibility. This process requires the programmer to plan the implementation and reveals edge cases. If individual responsibilities are not separated into units, they cannot be reused. If a unit meets several requirements, it is more difficult to understand which dependencies apply to each one, and that significantly impedes changes.

As one unit test checks one specific behavior of a single unit, information about the error found is very precise. The developer sees exactly what expectation has not been met and which unit has failed. Unit tests can also tell what is the expected result, which makes it easier for the developer to fix the fault. This benefit is particularly important when developing larger applications that more than one person worked on. The person who introduces the new functionality may not know the whole system. Information obtained from the unit tests are then particularly useful. In effect, system development is more efficient.

Unit testing has another important advantage for long-term projects: it’s easier for new developers to join it. This is not a secret, that in the IT industry, employee migrations between companies are common. In some cases, development teams might have to grow as projects grow. Unit tests give them comfort: despite the lack of knowledge of the system, they can test their changes. If an error is detected, the information received is so detailed that in many cases it is possible to fix it quickly without long analysis or asking others for help. Unit tests also give comfort to the project owner — changes in the development team are less time consuming and less stressful.

Project stability

There is another advantage of unit testing that affects project stability. Assumptions in the project change or evolve over time. Technology itself is also developing: new tools, new versions of libraries as well as new ideas regarding the quality of software development comes out. All these changes make refactoring necessary if we want to maintain code quality.

Unit tests allow us to quickly pre-check whether refactoring changes, often of global scope, negatively affect the functions of the system. The developer immediately receives information about the impact of the change. Based on the list of detected faults, he can make corrections or withdraw changes if he considers that the scope of work is too large and unprofitable.

Refactoring in a project that does not have automated tests is often rejected in advance because it is risky. It’s difficult to estimate its scope and time needed to complete it. 

Final thoughts on Unit tests

As we can see, unit tests have many advantages that make them profitable, despite their initial cost. Although those preparations need to be made before developers can start using them. Additional lines of code that always need to be written when adding a new feature, and fact, that just like the main application code, they also need to be modified when requirements change.

We need to remember programmers will use unit tests throughout the development process. They often fire the tests several times a day when they write changes. Unit tests are invaluable tools for programmers to add features and make modifications. These tests may detect errors at an early stage, support maintaining application consistency and code quality.

They are invaluable for long-term projects because they reduce the impact of their complexity on making improvements. The saving of time that we think will be gained by not writing them is apparent. The knowledge of their benefits allows us to understand that thanks to them we save time and our nerves.

See also:

• From Idea to working prototype. Startup App development Guide Initial stage
• How to build a successful MVP
• 7 common mistakes when developing an MVP
• How we work remotely! Find out our tips for effective cooperation on software development outsourcing!
• Our product design workshops – how to guarantee the most effective application development process