Building resilient systems in the Financial Services industry

Introduction

With customers becoming more and more demanding, they expect their financial services to be 24/7 available. Unfortunately the current monolithic architectures in the financial services industry are not very fault resilient. Banks and insurers have invested heavily in redundant infrastructure to increase the availability of their application landscape, but these investments only provide a solution for a limited set of failures (mostly the hardware failures). Any uncovered failures (like e.g. network issues, application failures due to software bugs…​) and maintenance activities (like software releases, operating system upgrades, database maintenance activities…​) still lead to service unavailability.

With customers nowadays being used to 24/7 available, online services, like Facebook and Google, don’t understand anymore why banks and insurers are not able to provide the same levels of availability. This provides an extra argument (next to the need for increased agility) for the financial sector to evolve their monolithic architectures to microservice based architectures, which are designed for failure. Such an architecture breaks the functionality in highly decoupled, small, simple and distributed chunks, which can fail without affecting the entire system.

Such a microservice based architecture is not a means to avoid failure. In fact, such an architecture, as it consists of many small components with a lot of communication in between, has even a higher probability for isolated failures. Microservices should therefore be explicitly designed for failure. This results in large scale resilience at the cost of some small-scale unreliability.

Such a resilient design requires safety mechanisms at various levels.

Mechanisms to automatically handle and react to failure

Microservices should be designed to be:

• Resilient, i.e. guard themselves from failures of dependent microservices, thus avoiding the ripple effect (i.e. avoid that one microservice brings down the entire system).
• Self-healing, i.e. quickly identify failure and recover as soon as possible (by automatically restoring service).

Different techniques exist to construct such resilient and self-healing services:

• Load balancer: a load balancer is typically used for spreading the load across multiple instances of a service, but can also help for making services more resilient. The balancer can perform regular service health checks and remove failed service instances from the pool.
• Circuit breaker pattern: this pattern is used to detect failures and encapsulate them (open the circuit after X unsuccessful communication attempts and close the circuit again, when a service is restored), thus avoiding failures to reoccur constantly (and impacting dependent services).
• Timeouts: when a service is slow, a timeout mechanism should be present. This ensures that services don’t wait too long for a response, thus avoiding to increase pending threads (and reach the maximum number of concurrent threads).
• Bulkheads / Throttling: this technique limits the number of concurrent calls to a component, by only accepting a limited number of calls per time unit. This allows to limit the number of resources (threads) waiting for a reply from the component.
• Elastic scalability: dynamic and rapid spinning up and down of services (often via Docker containers) allows to react quickly and pro-actively on failure. E.g. a system can react on increased response times (which could lead to failures) by spinning up extra nodes in a distributed system. Several cluster management tools like Apache Mesos, Google’s Kubernetes and Docker Swarm, exist to manage this dynamically.
• Graceful degradation: the typical behavior of a service, communicating with a dependent service which is down, is to stop processing the request and reply with an internal server error. This will accelerate the ripple effect and result in bad user experience. Instead services should implement graceful degradation, i.e. provide a plan B as soon as a service detects failure of a dependent service. Typical examples of graceful degradations are:
  • Use a cached response (which might not be up-to-date anymore), instead of real-time data retrieval.
  • Provide a canned response, i.e. a templated or scripted standard response
  • Call a different service, i.e. a backup service
  • Perform a simplified calculation in the service itself (instead of calling the dependent service)
  • Switch the entire application to read-only mode
• Build idempotent services: an idempotent service is a service, which gives the same result when called multiple times with the same input parameters. Such services are more resilient, as a service call can be retried (e.g. in case of time-out) without the risk of bringing the system in an inconsistent state. An idempotent service can be achieved by:
  • Avoid sending "deltas", but instead sending "current-value" type messages. This is not always possible, e.g. sending a payment or a securities order will always be a "delta" operation.
  • Filter out duplicates. This can be achieved by adding a unique identifier in the messages and tracking this unique identifier. If an identifier was already processed successfully, the request is rejected. This approach has the disadvantage that a service should store state, which makes the service more complex and subject to failure. This is especially the case if there are multiple instances of the service, as the list of successful requests should then be shared across the different instances, which could be located on servers in different regions.
• Retry / Recycling logic: a service should implement retry/recycling logic, i.e. when a call to a dependent service fails, the request should be automatically retried/recycled later. After a number of unsuccessful attempts, the request should be moved to an error queue/cache (dead letter queue/cache).

Mechanisms to identify and debug failures

Microservices should also be designed to detect and even predict abnormalities. This allows to proactively react to potential failures. If a failure still occurs, a maximum of information should be collected, so that the failure can be easily analysed and debugged.

• Logging: all service activities, both successful and unsuccessful, should be logged. As a microservice architecture is composed of a lot of small components, the system should foresee a central log microservice, so that the log entries of the different microservices can be aggregated in 1 central view.
• Tracing: as a request will be treated by multiple microservices, it is often difficult to trace back the path of a (failed) request. This is solved by associating a correlation token (also called trace or tracking ID) to a request. This token is passed from microservice to microservice and included in all log entries. By filtering the central log on the correlation identifier, it is possible to find back all actions performed in the context of the request.
• Clear error messages: when a service generates an error, it is important to return a clear error message. Such an error message should consist of: a unique error code (uniquely identifying the type of error), a clear error message for the end-user to know what happened and what he should do next, a link to documentation providing additional information on the error and a unique error identifier. This unique error identifier allows to search directly in the central log.
• Monitoring: breaking a system up in smaller services makes monitoring significantly more complex, especially since the correct working of each individual service is not sufficient to ensure that the integration points are also correctly working.
  Real-time monitoring of the application at various levels is therefore essential:
  • Technical monitoring like monitoring resource usage (memory, disk space, CPU…​), monitoring packets going over the wire, monitoring the JVM for thread usage, memory growth and kick off of the JVM Garbage Collector…​
  • Applicative monitoring like the number of service requests per second, service response times…​
  • Business monitoring like the number of payments per minute received
  • Semantic monitoring, which combines test execution and real-time monitoring to continuously evaluate the applications. Those tests mimic user actions via fake events and check if the system behaves as expected.
• Alerting: to avoid having to monitor a system continuously for anomalies, it is also important to categorize exceptions, so that alerts can be generated when important anomalies occur. This ensures that operators are (pro-actively) alerted, when they need to intervene manually.

Mechanisms to manually intervene on failures

An important part of designing systems for failure is also to provide the tools for operators to manually intervene on failures.

Such tools should provide following functionalities:

• Move a service request from the error queue back to the normal processing queue (e.g. after bug has been fixed).
• Cancel a service request from the error queue (i.e. if no longer needed to be processed).
• Adapt content of a service request (one by one or in bulk). This functionality should be fully audited, so that all adaptations can be fully tracked.
• Manually spin-up or spin-down instances of services. This can be used to deploy an updated version of services (after fix), increase/decrease service capacity, disable a specific service (by shutting down all instances of the service) or change the topology of the distributed service architecture.

Mechanisms to test for failure

The best way to validate the resilience (failure recovery logic) of a microservice based architecture is to introduce regularly diverse types of failures (fault injection) in the system, i.e. breaking things on purpose. This should be done both in the test environments as in the production environment, as often a production environment has specific settings not existing in other environments.

In follow-up of Amazon, many companies now organize Gameday exercises. During such an exercise, companies test the response of systems, software and people to a particular disastrous event and this directly in production and during a standard business day.
Many banks and insurers also execute once or twice a year DRP (Disaster Recovery Plan) tests, but unlike the major technology firms, these tests typically happen during weekends (often even with downtimes announced to customers) and are a lot more limited in scope (typically limited to testing the DRP of 1 or a few applications).

Other companies even go one step further and introduce production failures at random. Independent whether failures are injected at random or planned, these companies all use a set of tools to handle the creation of the failures, the roll-back and clean-up of the induced failures and the monitoring of the failure recovery. Most known examples of such tools are the Simian Army tools from Netflix or the Gremlin failure-handling testing framework (providing Failure as a Service).

Such tools allow to induce several types of failures, such as:

• Shut down virtual machines or kill instances and services (e.g. Chaos Money). This can be extended to simulate an outage of an entire availability zone (e.g. Chaos Gorilla).
• Induce artificial delays in the communication layer to simulate service degradation and check if the dependent services respond appropriately (e.g. Latency Monkey). By making these delays very large, companies can even simulate a service being down, without physically bringing these instances down.
• Simulate an unexpected service answer (i.e. answer not in line with the expected and documented answer).
• Simulate noisy neighbours in a cloud environment. The noisy neighbour problem is caused by the fact that in a cloud environment physical servers are split in multiple virtual machines, which are shared. Since it is typically very difficult to partition disks, a neighbor doing very large amounts of disk I/O can result in inferior performance for the other virtual machines on the same physical server.

This list can be extended with a lot of other types of failures, like JVM leaks, network issues, DNS failure, DoS attacks…​

Generalise design for failure

As explained above, designing a resilient microservice based architecture requires a lot of upfront design and introduces a lot of additional complexity. Unfortunately in today’s competitive financial landscape, such a design is no longer an option but a necessity.

The above mechanisms focus nonetheless only on non-functional failures. Just as important is dealing with functional failures, associated to application bugs and errors/flaws in the requirements.
A system should therefore also be designed to

• Identify and analyse bugs or requirement flaws as quickly as possible
• Isolate the effect of bugs as much as possible
• Create and deploy fixes as quickly as possible.

A few mechanisms supporting such a design are:

• Fast, easy and automated deployment (e.g. through Docker containers)
• Continuous business activity monitoring (in order to see drops in customer satisfaction or conversion rates as quickly as possible)
• Concurrent deployment of different versions of a service (allowing canary testing and A/B testing)
• Semantic monitoring
• …​

Setting up a resilient microservices architecture

Given the complexity of setting up such a resilient microservices architecture, one should avoid reinventing the wheel and setting up custom implementations for the above failure-management mechanisms. Instead banks and insurers should partner with a microservices framework and platform, to respectively build and run such an architecture. Such a product provides the above described mechanisms pre-packaged and glues together different micro-services in a fault tolerant way.

When banks and insurers can correctly implement such a resilient architecture, they will be able to create customer-centric financial services, which can compete with the services offered by the big technology giants.