Category: Software

These last few weeks have been a challenge for devops teams everywhere. For months everyone watched as the novel Covid-19 coronavirus swept across the world. Governments have called for people to stay home to help stop the spread of the disease. Nobody knew exactly how it would affect business and our everyday lives. Hardly anyone expected a novel virus that has forced us into isolation and many of our routine activities online. This sudden surge in traffic puts a lot of strain on systems and exposes major weaknesses that otherwise would go unnoticed. Ensuring that your systems are ready for unforeseeable events is an essential part of your devops strategy.

From daily meetings to your kids’ school lessons to virtual museum visits traffic is up across the board. Keeping these services up and running effectively is now more important than ever. Perhaps the biggest surge in use is in video conferencing services, remote collaboration tools, streaming services and online payments. Having a plan in place for what to do when a surge in traffic threatens to crash the site is vital to prevent downtime and service disruptions. 

1. Reasons of rapid increase in traffic

As many companies scrambled to figure out how to go fully remote in the face of orders to isolate, many daily activities moved to the network. Some examples include: 

1. Meetings — both for work and personal

Video conferencing and messenger services replace face-to-face meetings.

2. Grocery shopping and eating out

We’re ordering more groceries online and arranging food delivery through couriers.

3. Payments

In Poland, for example, payment providers have increased the no-pin transaction limit to the Polish zloty equivalent of $25.00 to gain 80% of overall payments without touching a pinpad or banknotes. This is very hygienic but threatens to overload the payment system. A lot of shopping has moved to e-commerce sites, straining capacity.

4. Daily news

Demand for the latest updates has increased readership on news sites.

5. Streaming videos

Cinema closures have driven demand for streaming services.

6. Outings

We’re visiting museums and galleries virtually.

7. School

Lessons are going ahead in many parts of the world via e-learning platforms.

Whenever you have a rush of traffic in a short amount of time, it tests whether or not the developers who designed the platforms did their jobs well. If a system crashes under the strain of increased traffic, chances are there wasn’t enough planning and foresight in development.

2. Latest examples of system downtime

How do millions of people sitting at home affect the use of network services? One local example happened to an online news portal which announced at first that all schools in Poznań will be closed for two weeks due to quarantine. As the news broke, it ended with website unavailability because resources were too low and the company was unable to react quickly. It was only handling over 13,000 users at the same time — in a city with half a million people. Should it be a barrier for your business?

Another more dramatic case happened last week as the stock trading application Robinhood failed due to a surge in traffic. This failure prevented users from accessing their accounts and selling their stocks as prices fell, leaving many with huge losses. The loss of user trust and credibility — not to mention the drastic losses for users themselves is immense. Here are just a few other scenarios that can happen. 

3. System downtime – what could possibly go wrong?

1. An increased number of visits can kill the server

Literally, when the resources of a single machine are running out you can talk about unavailable content. Shared hosting is definitely not a solution here. The best would be to use a cloud service provider such as Amazon Web Services, Microsoft Azure or Google Cloud Platform which has enough resources for a devops team to scale up if needed. Alternatively, you can use a powerful dedicated server in a well-known data center.

2. Poorly designed databases may not withstand a sudden increase in data

Let’s say that the number of orders in the store has increased and with each subsequent order the database responds more slowly.

3. Poorly written code needs a lot of computing power for simple tasks

With increased traffic, costs can rise disproportionately to profits. To avoid this, write tests during development and carry out stress tests before going to production. 

4. Self-hosting instead of using the cloud

Many companies and publishers keep resources on their own. Nowadays, the cloud offers the flexibility to respond to urgent needs. In this model you only pay for what you use, you can start the next server at any time and quench it when the traffic drops. It’s also possible to automate this process. So why not use it?

5. Saving on infrastructure can lead to system failure.

Work on small, cheap resources cannot defend themselves in such a situation. Suppose someone is hosting a website on his own and has a small reserve on bandwidth. Increasing bandwidth is not possible overnight. Instead, use a cloud provider or a data center. 

4. System downtime – all too common mistakes

1. No support when the website is on fire

It’s common that companies order a website or e-commerce shop and later just let it run without any devops support. When increased traffic occurs, no one is able to help. At Espeo we offer support for our software in the production environment to not leave you alone in such a case.

2. Old technologies make the product inaccessible

An example would be one of the Polish e-learning platforms that still uses Adobe Flash extensions. Browsers no longer officially support these and the end of life is happening later this year. Now as the schools have been closed, it turned out that using the service exceeds the skills of most young users.

3. Weak security

Today, the standard is to use the HTTPS protocol (using SSL certificates). It provides a secure connection between the user and the provider. No implementation of encrypted transmission may result in users’ rejection. Especially when we deal with payments and providing personal data needed for the order.

On the other hand, sometimes websites are vulnerable to attacks because the code is written using open source solutions that are not updated on time. At Espeo we are putting a lot of effort to keep systems updated. Our services among others consist of scanning of running resources, servers monitoring so we can prevent attacks and keep software stable.

4. “Tests are not needed” sentiment

Many software houses cave to pressure from clients eager to rush a program to production. But it’s a huge mistake to think that you don’t need to test software. Simulating an outage is far easier and cheaper to test for than a system outage. It takes some long-term planning and upfront costs, but it’s much more cost-effective to test for these crises. At Espeo our quality assurance specialists test each project. Depending on the scope our devops team can handle a lot of different tests to prevent problems in the future.

5. Bad architecture

“A Single instance will deal with everything” is a bad concept. Keeping everything in one place will fail sooner or later. The key is to multiply resources and keep the database and website apart from each other. At Espeo advise clients to set environment with load balancers, take advantage of scalability and master-slave database replication.

5. Final thoughts on how to avert a system downtime crisis

Long story short, be prepared! Assume the worst scenario and prepare a solution for it before it happens. In Espeo during the development process for our clients, we put a lot of effort to use our experience to design solutions right the first time. 

The biggest part of preparing for a crisis is to make sure you have all the necessary features in place to respond quickly. As the coronavirus has shown us, these very unexpected events can have a huge impact on business and on the software we rely on every day. Making sure it can handle a rapid increase in traffic — and then quickly go back to normal will save you time, money, and reputation.

Want to learn more about devops and testing services? Drop us a line and we’ll get back to you shortly.

See also:

• Good reasons not to skimp on QA testing
• Key tips for intercultural communication in IT outsourcing companies
• 10 key trends in electronic payments for 2020
• Outsourcing your online payment gateway project: What skills to look out for when building your remote team
• How to make a great product with a remote development team

Recently a client asked me how I prove application testing is worth the money? For me, it’s obvious that testing is a vital part of the development process. Without it, there is little chance for a successful, high-quality product. So how do I prove it to someone who isn’t familiar with the reality of the business?

It is a fact that in almost any aspect of life higher quality comes at a higher price. Do you want to have a great, luxurious car? You need to pay more. Do you want to have a tailor-made suit? You better dig deep into your pocket. Software development is no different in this matter. If you want to have a fast, reliable and threat-secure application you need to invest in testing. But – in the long run — is it really worth the expense? First, we need to understand how an application development cycle looks and how it affects the bug-fix cost.

The software development cycle usually starts with requirements & design phases. This is where you outline the scope of the application, its main functionalities, visuals, architecture, test scope and any other requirements that you may have. This is a very important phase because any issue at this point may result in complications later on. If a developer overlooks a requirement it may be considerably more expensive to fix it later. If you decide to change a requirement at a later stage then the other parts of the application may get affected in a way you could have not anticipated. Fixing an issue at this stage is almost costless – all you need is to rethink and rewrite it or add some new requirements.

The next phase is the actual coding part. Web application development teams usually consist of several people who measure the code in thousands of lines. Each line of code is a potential place for a defect. The more complex the software is the more interdependencies between modules, integrations and architecture pieces there are. Developers are also human and in this jungle of advanced logic, code standards, abstract end-user ideas and external system integrations it is very easy to overlook something. They also have to consider the fact that different browsers/devices often operate in a different manner. There are also changing requirements and bug fixes that require adjustments in code already written. It often happens that one part of the code affects some other unrelated functionalities. Finally, there is also a business pressure to meet the agreed deadlines which can sometimes make the developers work in a hurry. And, this is just the tip of the iceberg. An issue usually found at the development stage is fairly easy to fix as the developer is aware of the code he has just written and it doesn’t take much time to isolate the specified issue. Still, it’s much more costly than a simple change in the requirements phase.

In agile development methodologies testing happens alongside the development process. You may ask yourself a question: if testing takes place, there should be no issues on production, right? Well, not exactly. First of all, it is not feasible to test all the possible inputs and outputs of a complex application. Moreover, it is not possible to test the application for all browser and device combinations in all possible versions. Finally, the user’s creative ways are sometimes futile to predict – it is not possible to come up with all the things that a user may think of doing and prevent them from happening.

There is also a matter of maintaining the application. At times, the new software version may clash with the application and cause bugs. From time to time, the external integrations get updates without updating the code. This, in turn, may cause the app to malfunction. Although it may sound strange sometimes we also choose not to fix some of the bugs. If we know that a defect only causes a minor discomfort for a limited number of users it isn’t sometimes worth fixing because the costs would surpass the actual damage caused by this issue.

Despite the fact that a well-chosen test strategy may minimize the project risks and eliminate all critical issues, it still cannot guarantee an issue-free project. So what is the cost of fixing a bug at the testing stage? It’s even higher than in the development stage. Usually a tester, a developer and, sometimes, a PM decide on the priority of the bug. It is also much more difficult for a developer to diagnose the root cause of the issue. Is it architecture? Does it occur only for a certain set of data? Is it a bug within the code? Is it possible to reproduce it on my machine or is it only test environment related? A developer has to find an answer to these questions and it takes more time than investigating his own code and functionality.

The last phase of the development cycle is when an application is already deployed to production. The cost of fixing bugs on production is significantly higher, sometimes it can be incredibly large. Even setting aside the hard-to-measure costs of damage to company reputation, users leaving the application or customers not being able to complete their purchases – the cost of fixing the bug is higher. The issues that users find are usually much more difficult to reproduce in the development environment. They’re often connected to a specific production architecture or database state. On top of everything, the users usually don’t provide as many details as a qualified software tester which prolongs the investigation. The table below presents some real-life examples revealing how much an issue can affect a company’s revenue.

So is it even possible for software to be bug-free? Well, let me answer that question with another question. Do big companies with a huge amount of resources – like Apple, Google or Facebook provide bug-free software?

Knowing all that and based on the above explanations and estimated bug-fix costs provided by IBM data we can perform an example Return On Investment(ROI) analysis for testing.

Let’s try to analyze an ROI example for a 3-month project. First, let’s make the following assumptions:

1. Assuming that bug-fix cost during the design phase is $10 using the above IBM estimated costs we get:

• Design bug-fix cost – $10
• Development bug-fix cost – $65
• Testing bug-fix cost – $150
• Production bug-fix cost – $1000

2. There are 100 must-fix bugs during a 3-month period and they all cost approximately the same.
3. None of the issues were found during the design phase (which could have lowered the total cost even more)

Now we have to estimate what the cost of manual tests would be and test automation as well as the number of bugs to find in each phase. Assuming that basic manual application testing costs around $2000 per month (that’s the salary of a specialist) and adding some minor infrastructure fee for the test environment we receive a total investment of $6,500. When it comes to test automation let’s increase the specialist’s salary as well as the infrastructure & setup cost – then we get $11,000.

Let’s estimate that out of 100 total bugs 25 of them will be found by the developers (this is a constant number no matter whether we carry out tests or not). We can estimate that during the testing phase the testers will find 35 issues while performing manual tests and 50 when we add automation on top of that.

The rest of the bugs will be found by the end-users. Now we can count the total cost of quality as the sum of conformance (cost of ensuring that the app conforms to quality requirements) and non-conformance (money that needs to be spent for not conforming to the quality requirements). Finally, we can count the test ROI as the value brought by testing divided by the cost of actual testing.

This example certainly contains numerous assumptions but you can use it in an actual project. It shows how we can actually estimate ROI. I believe that it provides some insight to whether application testing costs are worth the money. Even though hiring a testing team is, in fact, an expense, when you look at your entire application development cycle you will view it as an asset. Testing will spare you a lot of future quality expenses. This also causes things like user frustration, clients leaving and even company reputation damage – factors which are difficult to measure in terms of money.

See also:

• Good reasons not to skimp testing
• Client’s engagement in an application development project? What involvement is needed for successful delivery?
• Web Application Testing: Terms of Quality in Web Applications
• Great application development starts with a product design workshop

 
[contact-form-7 id=”13387″ title=”Contact download_8_reasons”]