Categories
Software Technology

Top causes of system downtime and how devops prepare to avert a crisis

These last few weeks have been a challenge for devops teams everywhere. For months everyone watched as the novel Covid-19 coronavirus swept across the world. Governments have called for people to stay home to help stop the spread of the disease. Nobody knew exactly how it would affect business and our everyday lives. Hardly anyone expected a novel virus that has forced us into isolation and many of our routine activities online. This sudden surge in traffic puts a lot of strain on systems and exposes major weaknesses that otherwise would go unnoticed. Ensuring that your systems are ready for unforeseeable events is an essential part of your devops strategy.

From daily meetings to your kids’ school lessons to virtual museum visits traffic is up across the board. Keeping these services up and running effectively is now more important than ever. Perhaps the biggest surge in use is in video conferencing services, remote collaboration tools, streaming services and online payments. Having a plan in place for what to do when a surge in traffic threatens to crash the site is vital to prevent downtime and service disruptions. 

Top causes of system downtime and how devops prepare to avert a crisis

Table of contents:

  1. Reasons of rapid increase in traffic
  2. Latest examples of system downtime
  3. System downtime – what could possibly go wrong?
  4. System downtime – all too common mistakes
  5. Final thoughts on how to avert a system downtime crisis
 

1. Reasons of rapid increase in traffic

As many companies scrambled to figure out how to go fully remote in the face of orders to isolate, many daily activities moved to the network. Some examples include: 

1. Meetings — both for work and personal

Video conferencing and messenger services replace face-to-face meetings.

2. Grocery shopping and eating out

We’re ordering more groceries online and arranging food delivery through couriers.

3. Payments

In Poland, for example, payment providers have increased the no-pin transaction limit to the Polish zloty equivalent of $25.00 to gain 80% of overall payments without touching a pinpad or banknotes. This is very hygienic but threatens to overload the payment system. A lot of shopping has moved to e-commerce sites, straining capacity.

4. Daily news

Demand for the latest updates has increased readership on news sites.

5. Streaming videos

Cinema closures have driven demand for streaming services.

6. Outings

We’re visiting museums and galleries virtually.

7. School

Lessons are going ahead in many parts of the world via e-learning platforms.

Whenever you have a rush of traffic in a short amount of time, it tests whether or not the developers who designed the platforms did their jobs well. If a system crashes under the strain of increased traffic, chances are there wasn’t enough planning and foresight in development.

 

2. Latest examples of system downtime

How do millions of people sitting at home affect the use of network services? One local example happened to an online news portal which announced at first that all schools in Poznań will be closed for two weeks due to quarantine. As the news broke, it ended with website unavailability because resources were too low and the company was unable to react quickly. It was only handling over 13,000 users at the same time — in a city with half a million people. Should it be a barrier for your business?

Another more dramatic case happened last week as the stock trading application Robinhood failed due to a surge in traffic. This failure prevented users from accessing their accounts and selling their stocks as prices fell, leaving many with huge losses. The loss of user trust and credibility — not to mention the drastic losses for users themselves is immense. Here are just a few other scenarios that can happen. 

 

3. System downtime – what could possibly go wrong?

1. An increased number of visits can kill the server

Literally, when the resources of a single machine are running out you can talk about unavailable content. Shared hosting is definitely not a solution here. The best would be to use a cloud service provider such as Amazon Web Services, Microsoft Azure or Google Cloud Platform which has enough resources for a devops team to scale up if needed. Alternatively, you can use a powerful dedicated server in a well-known data center.

2. Poorly designed databases may not withstand a sudden increase in data

Let’s say that the number of orders in the store has increased and with each subsequent order the database responds more slowly.

3. Poorly written code needs a lot of computing power for simple tasks

With increased traffic, costs can rise disproportionately to profits. To avoid this, write tests during development and carry out stress tests before going to production. 

4. Self-hosting instead of using the cloud

Many companies and publishers keep resources on their own. Nowadays, the cloud offers the flexibility to respond to urgent needs. In this model you only pay for what you use, you can start the next server at any time and quench it when the traffic drops. It’s also possible to automate this process. So why not use it?

5. Saving on infrastructure can lead to system failure.

Work on small, cheap resources cannot defend themselves in such a situation. Suppose someone is hosting a website on his own and has a small reserve on bandwidth. Increasing bandwidth is not possible overnight. Instead, use a cloud provider or a data center. 

 

4. System downtime – all too common mistakes

1. No support when the website is on fire

It’s common that companies order a website or e-commerce shop and later just let it run without any devops support. When increased traffic occurs, no one is able to help. At Espeo we offer support for our software in the production environment to not leave you alone in such a case.

2. Old technologies make the product inaccessible

An example would be one of the Polish e-learning platforms that still uses Adobe Flash extensions. Browsers no longer officially support these and the end of life is happening later this year. Now as the schools have been closed, it turned out that using the service exceeds the skills of most young users.

3. Weak security

Today, the standard is to use the HTTPS protocol (using SSL certificates). It provides a secure connection between the user and the provider. No implementation of encrypted transmission may result in users’ rejection. Especially when we deal with payments and providing personal data needed for the order.

On the other hand, sometimes websites are vulnerable to attacks because the code is written using open source solutions that are not updated on time. At Espeo we are putting a lot of effort to keep systems updated. Our services among others consist of scanning of running resources, servers monitoring so we can prevent attacks and keep software stable.

4. “Tests are not needed” sentiment

Many software houses cave to pressure from clients eager to rush a program to production. But it’s a huge mistake to think that you don’t need to test software. Simulating an outage is far easier and cheaper to test for than a system outage. It takes some long-term planning and upfront costs, but it’s much more cost-effective to test for these crises. At Espeo our quality assurance specialists test each project. Depending on the scope our devops team can handle a lot of different tests to prevent problems in the future.

5. Bad architecture

“A Single instance will deal with everything” is a bad concept. Keeping everything in one place will fail sooner or later. The key is to multiply resources and keep the database and website apart from each other. At Espeo advise clients to set environment with load balancers, take advantage of scalability and master-slave database replication.

 

5. Final thoughts on how to avert a system downtime crisis

Long story short, be prepared! Assume the worst scenario and prepare a solution for it before it happens. In Espeo during the development process for our clients, we put a lot of effort to use our experience to design solutions right the first time. 

The biggest part of preparing for a crisis is to make sure you have all the necessary features in place to respond quickly. As the coronavirus has shown us, these very unexpected events can have a huge impact on business and on the software we rely on every day. Making sure it can handle a rapid increase in traffic — and then quickly go back to normal will save you time, money, and reputation.

Want to learn more about devops and testing services? Drop us a line and we’ll get back to you shortly.

See also:

Categories
Blockchain Financial Services Technology

How enterprise blockchain applications can improve your firm’s internal processes

Enterprise Blockchain Applications – fill in the form and download your free copy.

How enterprise blockchain applications can improve your firm’s internal processes

Enterprise blockchain technology is a relatively new tool and companies are still working out ways to put its unique features to good use.

While there are plenty of dubious claims for what enterprise blockchains can do, getting up-to-date advice on how it will fit exactly is still difficult to find. Find out more in this e-book and learn if a blockchain is suitable for your business.

Categories
Finance Financial Services Technology

How to ensure proper payment gateway integration with popular methods

Payment gateways are becoming more and more important players to improve the shopping experience and effective payment gateway integration is ever more vital. Credit cards, one-click payments, instalments, recurring payments, refunds, withdraws, digital wallets, invoice payment links — it’s good to have these technologies integrated into one place. Customers don’t want to remember multiple login credentials, and merchants are not happy digging through complicated system integration guides. Both sides want faster, easier and more secure operations.

Let’s see how we can bring modern payment solutions closer to our customers.

Payment gateway integration with popular methods

Table of contents:

  1. Deep analysis of integration documentation
  2. Security considerations
  3. Performing sandboxed payment gateway integration tests
  4. Receiving notifications from payment systems
  5. Adapting your system to API changes
  6. Logging communication with APIs for quick troubleshooting
  7. Final thoughts on Payment Gateway Integration
 

1. Deep analysis of integration documentation

Every payment method provides various integration guides. Starting from tutorials and frequently asked questions, going through software development kits tailored for the most popular programming languages and ending with a comprehensive API reference for developers. Dedicated people have to carefully study all these materials.

Most payment methods will have limited availability by countries and regions. They can be subject to different regulations that can cause transaction limits, limited functionality or additional formal requirements.

An example can be offering recurring payments for markets around the globe. Subscription models are very convenient for customers and merchants, but they usually require additional development work. Acquirers will expect you to distinguish all transactions made within a specific recurring payment plan from ordinary, one-time e-commerce transactions. A subscription plan is an agreement between a customer and a merchant. Your job is to keep track of that agreement by providing the initial transaction ID. Mexico’s Banamex also requires providing a unique contract number for every payment plan. If in doubt, always contact a support person from the payment provider’s side.

 
 

2. Security considerations

Payment solution providers will expect you to ensure the best security level possible. For example, PayPal provides extensive “Security guidelines and best practices” in their developers portal. One of the most common security measures is using the latest encryption protocols, maintaining anti-phishing processes and regular security audits. 

The basic requirement for a modern payment platform is using secure communication protocols. In 2020, the minimum version for transport layer security — TLS — is 1.2. Industry security experts have long considered older protocols — especially secure socket layer, or SSL — vulnerable. Payment methods will not accept connections using vulnerable protocols.

PayPal, along with many other providers, dropped support for TLS versions older than 1.2 back in 2016. It happened after the company discovered several serious vulnerabilities like POODLE and Heartbleed. Hackers could exploit these and breach the system. Also, the SHA-1 cryptographic algorithm is discouraged in favor of a stronger SHA-256. Security best practices change rapidly as new software versions emerge. Staying ahead of all these changes and setting them up is a huge challenge. 

Internet protocols change frequently in response to threats. The best practice is not to hardcode a specific version, but to let the software negotiate the highest version available. This can happen automatically, but still, your DevOps team has to be up to date with recent security news. Most payment providers will help you by sending informative and warning emails. Do not ignore any of them and set up any development or maintenance work immediately instead of rushing just before a deadline.

Apart from staying up to date with payment method news, you should take your own actions to reduce systems vulnerability. Check your integration standards against industry best practices at least once a year. Introduce anti-phishing processes and monitor spoof sites which could lead your clients to pass their sensitive data to scammers. Warn your customers and accept spoof site reports from them.

The payment industry often relies on professional security tools from reliable sources like iDefense or Veracode. These companies provide software for automatic known vulnerability scans, which it categorizes by severity. These scan reports often come with proposed solutions and even time estimates. Combined with development and continuous integration plugins, these tools bring invaluable help to your product team.

Security considerations are vital if you deal with customer credit card details. This includes not only the classic scenarios when a customer enters data directly in your form but also some electronic wallet solutions like Masterpass. This is even more important while using its express checkout functionality where a customer consents to share data without even logging into their Masterpass wallet. Other wallet providers like Apple Pay try to mitigate the risk by using payment tokens instead of actual card numbers.

 

3. Performing sandboxed payment gateway integration tests

During onboarding, all payment companies will provide sandboxing environments for development and testing. First, you receive a test environment account with a public/private key pair, and then login credentials. Your developers and quality assurance team can test all the integration scenarios.

Some providers, like Apple Pay or Trustly, will also perform additional acceptance tests for both frontend and backend integration, domain verification, and so on. Part of EVO Snap the mandatory certification process is to run test scripts in your application and send results to the certification team. Then, after receiving approval, you will be provided with live credentials.

Need a consultation or help? Work with us.

Make sure no data gets between your test and live environments. It would be very unprofessional for your customers to see some random test records. Even worse, your development team should not see any production data unless explicitly allowed.

In addition to making transactions with test cards in a sandbox environment, you might need to perform some transactions with real cards in a production environment.

 

4. Receiving notifications from payment systems

Integrating your payment system with external solutions is a two-way integration. Not only your application is going to send payment requests, but it also has to accept and respond to any notifications and feedback from a payment solution. The most obvious example is a notification confirming or rejecting payment. If a particular payment method does not receive a valid answer from your system, it will likely raise a warning.

Read the documentation carefully to learn what contact attempts you can expect from a payment method and how to respond.

 

5. Adapting your system to API changes

Every payment method provider can introduce changes in their APIs. Some of them do not break compatibility with existing integrations, some others do. These changes may occur due to different circumstances — either new business functionalities, new market regulations or simply removing obsolete procedures.

Of course, such updates are always announced in advance. Watch out for any announcements from your payment method providers. Do not ignore them! If they sound too technical for you, forward them immediately to your payment gateway integration development teams and ask for explanations. Usually, companies allow their clients at least six months to adapt. But time flies, so schedule an update as soon as possible.

 

6. Logging communication with APIs for quick troubleshooting

Sooner or later something will go wrong. Your customers will file complaints and your team will have to respond quickly. It is crucial to prepare for such cases by logging all communication with payment method APIs, so the team can perform a proper investigation.

What to log? Your software should at least query the exact API version of each payment method. This is basic information while reporting problems to a payment method provider. Every piece of information exchanged between your application and the API also has to be logged. Developers have to make sure that all the logs include transaction identifiers so that the support team can easily find all information related to that particular transaction.

However, excessive application logs are very difficult to browse. Your team has to use a log aggregator or browser which allows them to quickly find relevant information. Most log aggregators, like Logentries or Papertrail, are paid software-as-a-service platforms. The cost will be related to the velocity of the logs. While choosing a proper solution, analyze different payment plans. You need to be prepared for sudden spikes in application usage. Other solutions include hosted or self-deployed ELK Stack.

While it may sound unnecessary to invest money in yet another development or support tool, it is going to pay off quickly. For payment gateway providers, it is crucial to have a fast incident response time. Your support team needs a reliable tool to identify the root cause as soon as possible.

 

7. Final thoughts on Payment Gateway Integration

Payment gateways bring huge value to the market by simplifying payments for customers and merchants. Payment gateway integration with the payment methods your customers are using improves the UX and overall usefulness of the app. It is a payment gateway vendor’s job to ensure proper integration with payment method providers where every solution has a separate set of requirements. With more and more payment solutions coming, customers and merchants will put even more trust in payment gateways.

The article was written by Piotr Horzycki, Java & PHP developer at Espeo Software. To read more pieces by Piotr, you can visit his website or his LinkedIn profile.

See also:

Categories
Blockchain Financial Services Technology

Benefits of blockchain in emerging markets

Blockchain in emerging markets promises to drive down to the cost of remittances and trade finance, improving financial inclusion for many of the world’s nearly two billion people currently without a bank account.

One thing that categorizes developing countries is inadequate access to banking. Individuals and small businesses would like greater access to financial services currently unavailable trough traditional finance. High levels of mobile coverage and new disruptive payment apps such as Bitpesa offer new solutions.

Blockchain technology is a digital, distributed, immutable transaction ledger that eliminates the need for intermediaries. By doing so, it provides several opportunities for cost savings while opening new market segments for existing financial institutions and new players alike. 

While developed markets experience exponential growth and advancements, the chances of emerging markets competing fairly with them on a global scale seem bleak sometimes. Nevertheless, the introduction of blockchain in emerging markets has reopened the prospects of a revival in these nations. Some of blockchain’s use cases can improve payments in developing countries by reducing remittance costs, enhancing financial inclusion, and eliminating corruption loopholes. 

But before that, let us briefly discuss how blockchain technology drives efficiency in existing businesses and how it creates new markets in developing countries.   

How blockchain drives efficiency in existing businesses 

Much of the attention surrounding blockchain technology is from developed countries, especially in the payment sector, where the technology will likely have a significant impact because of its power to minimize payment costs. This has led some to reassess micropayments as a viable model, for example.

As a result, distributed ledger technology is heavily linked with financial institutions that deal with process efficiency services. These companies have started using blockchain-based solutions to solve specific issues or improvements in their business models, such as data reconciliation, supply chain tracking, clearing, and internal settlements. 

Meanwhile, some international banks and financial intermediaries have partnered with blockchain firms to explore applications that apply to their business models and learn how this revolutionary invention may improve their legacy infrastructure. They are also considering consortia to leverage development and potential transition expenses and to raise the standard of blockchain technology.   

Many corporate projects so far have embraced private blockchains, such as the Linux Foundation’s Hyperledger Fabric, as companies try to weigh the pros and cons of the revolutionary technology and retain the integrity of their existing business models. 

The Post-Trade Distributed Ledger Group brings together international banks, custodians, central security depositories, and central banks from all over the world to share information and concepts on how blockchain technology can positively influence the post-trade landscape. 

Creation of new markets

Blockchain is a disruptive technology that can re-engineer economic models and facilitate the creation of markets and products that were formerly nonexistent or unproductive. Most of these new market prospects are related first to its offer as an alternative to fiat currency, solving issues of currency inflation and political instability. Second, its power to achieve a digital identity in a fast and cost-effective way improves financial inclusion of previously underserved markets. 

Blockchain technology also creates opportunities for startups and established businesses form non-financial sectors with a strong consumer base, like telecommunications or e-commerce establishments. Such players are rapidly innovating to create new business models and services, and are transforming the value chain landscape and challenging banks.  

These initiatives have mostly originated from established markets, targeting developing countries directly or indirectly. Though they are not entirely based in developing countries, the best-funded ones are from developed markets for now.  

A considerable chunk of the total venture capital has been invested in the digital wallet and capital market service segment. Regardless of their source, these startups are targeting the economic activities of emerging markets, such as remittances and trade finance. 

This is an exceptional phenomenon, implying that developing countries can be reasonable testing grounds for new ventures, where high demand for financial inclusion and relative inadequacy of infrastructure can speed up the use of new technologies — particularly blockchain. The prospect of outspreading banking services in such markets is high, with two billion adults lacking access to financial and credit services globally. Cross-border payments and remittances are a case in point: it has a market value of over $4 trillion with transaction charges that range from 5% to 30%. 

How blockchain in emerging markets can improve payments

Minimizing Remittance Costs

Citizens of developing nations who have migrated to developed countries for work drive the international remittance system. Time and again, these individuals send money to their families and friends back home using financial intermediaries such as Western Union, PayPal, MoneyGram, Payoneer, etc. These intermediaries impose high transaction charges. 

According to the World Bank’s latest Migration and Development Brief, remittances to developing countries hit a record high in 2018. The recorded remittance to emerging markets in 2018 was $529 billion, an increase of 9.6% from 2017. The brief also revealed that the global remittance fee for Sub-Saharan African countries was an average of $20 per $200, which was the highest in the world. 

With the emergence of cryptocurrencies, the cost of remittances can be considerably lower. While Bitcoin remains the largest cryptocurrency by trading volume, it can be difficult and unpredictable to use for remittances. Stellar meanwhile offers faster transactions and low fees making it ideal for remittance systems.

Already, there are several platforms using blockchain in emerging markets in Africa and Southeast Asia that support cross-border and peer-to-peer payment solutions, like BitPesa

BitPesa is a blockchain firm offering foreign exchange and business-to-business crypto-based payment services in Kenya and many parts of East and Central Africa. The startup has managed to leverage the existing financial models by partnering with the M-Pesa mobile money network, a subsidiary of Telecom Company Safaricom and provider of mobile payments and significant incumbent player (almost three-quarters of Kenya’s adult population have an M-Pesa wallet). 

Improving financial inclusion

Low financial inclusion is a significant problem in developing countries. According to the World Bank, there are over two billion people globally without a bank account, and a big percentage of this number comes from developing countries. In nations like Pakistan, Chad, Somali, Burundi, Niger, Yemen, and Cameroon, less than 15% of the adult population has access to banking services. Even those with bank accounts lack access to premium banking services, so they qualify as unbanked. The lack of access to banking services prevents them from partaking in global commerce.  

With crypto services like BitPesa in Kenya, BitSpark in Hong Kong, OkCoin in China, OkLink in India, Rebit, and Coin.ph in the Philippines, billions of the unbanked population have access to financial services through cryptocurrencies. These startups are providing crypto banking services via mobile phone applications. The telecommunication industry has been able to attain a higher market penetration compared to the banking industry. 

These blockchain-based companies are capturing the existing widespread use of telecoms to deliver their services to unbanked and underbanked people. The eventual result is better financial inclusion. 

Besides, there is also an added advantage of empowering small and medium-scale businesses. Local merchants can tap into global trade. Financial institutions in developing countries are reluctant to offer loans for small-scale enterprises even when appropriate collaterals are in place.

With blockchain, platforms like BitPesa and OkLink can provide crypto-backed loans to small and medium scale businesses. This will go a long way in getting them started in foreign trade, which is an integral part of national commerce. 

Another feature of financial inclusion that many developing countries face is the lack of global payment systems. International commerce is mainly denominated in US dollars, and it calls for specialized payment and documentation systems. This is an obstacle for many merchants in these countries as they lack access to foreign exchange and the means to send and receive money in foreign currency. 

BitPesa is championing the provision of solutions to these issues across Africa. In Indonesia, TenX has a digital wallet that enables users to receive Visa card payments. 

Eliminate corruption loopholes

Corruption is one of the significant issues facing developing countries. The absence of economic democratization and corrupt officials has created a framework that has left the mutual prosperity of these nations at the mercy of a few individuals. The middle-class has shrunk, and over 70% of the people survive below the poverty level.  

In emerging markets, misappropriation of government funds by corrupt officials is a significant issue. Refusal to adhere to project contracting best practices leaves state projects to be run by groups that channel the allocated funds to their pockets. The use of digital currencies, particularly those embracing smart contracts, will enable a more transparent contract system. With blockchain records being accessible to everybody, citizens will be able to track the way their funds are being used. 

Conclusion

Blockchain in emerging markets will lead to lower remittance costs, better financial inclusion, and put an end to corruption loopholes. Driven by high demand, especially in catering for the needs of financially excluded markets, and a hedging plan through cryptocurrencies in situations of currency inflation and political instability, blockchain technology appears ripe for adoption. It will undoubtedly be interesting to see to what extent emerging markets apply blockchain solutions to the problems facing them.

Related posts: