Skip to main content

Microservices - Yet another buzzword or a real innovation for the Financial Services Industry

1. Introduction

Microservices are the new hot trend in the application architecture landscape. Like SOA (Service Oriented Architecture) and cloud a few years ago, "microservices" is the buzzword used by marketeers to position their solution as cutting-edge technology. The concept of microservices is however not new. Leading technology companies, like Google, Amazon and Facebook, have already migrated from a monolithic to a microservice based architecture (MSA) for over a decade, and more recently companies like Netflix, eBay, Twitter and PayPal have also evolved. Their track record with this architecture is impressive, e.g. Netflix is responsible for about 30% of the internet traffic and handles about one billion calls per day from 800 different types of devices.
Unfortunately, finding a good definition for the term "microservice" appears to be extremely hard. When going back to the basics, a microservice based architecture is the same as a SOA architecture with the exception that several additional constraints are imposed to the services. We could therefore call MSA the version 2.0 of SOA.
This (r)evolution of SOA is mainly driven by a number of recent technological and organisational evolutions, which made the microservices architecture possible (or at least more attractive). MSA is therefore heavily interconnected with these technologies and methodologies:
  • (Private) Cloud: the main driver for firms to move to the cloud is "elasticity" (a firm pays only for the resources it needs, but can scale immediately when the load increases). This "elasticity" requires an application architecture, which can effectively utilize this capability. Microservices can give an answer to this need.
  • APIs: the rapid adoption of APIs in recent years from a development technique to a business model driver, aims to reuse, share and monetize the organization’s assets. These principles fit perfectly with MSA, which will typically enable firms to create an (open) API ecosystem. Basically, MSA is breaking up an application or services in many different components, which are glued together through APIs (i.e. standardized formats for communication).
  • (Docker) Containers: containers allow firms to deploy software built in a variety of technology stacks in a homogeneous, simple and very quick way. Without the use of containers, the deployment of microservices would be extremely complex.
  • DevOps: the DevOps movement aims to improve the collaboration of development teams and operations by integrating those teams and automating the software delivery process. MSA loses a lot of its benefits and becomes very hard to manage (especially the deployment) without these DevOps principles.
  • Agile: Agile is a software development methodology where small (2-pizza) self-organizing teams develop software by building up the requirements and software gradually. Organizing a firm with MSA with a waterfall software methodology would lose almost all benefits associated to MSA (i.e. flexibility, short-time to market, independent development teams…​).
MSA combined with all those technologies and methodologies allows a rapid software delivery and an application landscape supporting rapid change and growth of the business.

2. Definition and Concepts

Microservices are all about "doing one thing and doing it well", i.e. build an application architecture with small, API-accessible, single-purpose components.
As explained above, a clear unique definition of microservices is hard to give, but all experts do agree about several common characteristics:
  • Small: microservices are by definition small. Of course, "small" is subjective, so finding the right size of a microservice is more an art than science.
  • Single-purpose: each service should be organized around business capabilities and run a unique process to serve a business goal.
  • Independent: microservices must be independent, modular components. This means:
    • Service should run fully independent, i.e. a service must be able to handle a request independently. This means it can also independently be replaced or upgraded.
      It also means that a microservice should be stateless, i.e. every request is handled with the information contained only within it.
    • Service can be autonomously developed, i.e. development can be done by fully autonomous teams.
    • Service should be independently deployable, i.e. it is possible to deploy a new version of a microservice without being forced to deploy any other component. This means that a microservice must be a full stack application, i.e. the business logic, data model and service interface (API or user interface) should all be included in the microservice.
  • API-accessible: microservices should communicate with the outside world, using well-defined, lightweight protocols.
  • Simple Integration: the integration of microservices should be as simple as possible. Typically, this is done by dumb, fast and asynchronous (publish - subscribe) messaging.
  • Design for failure: microservices should be designed so that they can react to failure, i.e. identify failure and fix it without impacting other services.
  • Decentralized governance: teams developing microservices are fully autonomous, choosing themselves the programming language, database, hardware…​ This means no central guidelines for these choices, but also a decentralization of data, i.e. no large central relational database, but instead each service managing its unique database.
    Nonetheless teams are encouraged to share code libraries with other teams and reuse existing code libraries.
  • DevOps approach
    • Automated Software Delivery: software delivery is as automated as possible. This includes concepts like Continuous Delivery, Continuous Integration and Automated Testing.
    • Deliver products, not projects: the teams building the microservices should own the services over its full life-cycle (you build it, you run it), i.e. also take the responsibility for the software in production. This brings the development teams much closer to the end-user.
With these characteristics in mind, it is clear that MSA is particularly suitable when multiple platforms and devices need to be supported, flexibility (i.e. frequent changes) is key and functionality can be easily split up in small components.
MSA should not be considered as a one-fit-all solution, i.e. it is not the ideal solution for all applications. A poorly designed MSA-based application landscape will still be unmaintainable.

3. Benefits and Drawbacks

3.1. Benefits

The usage of a microservices based architecture gives several advantages compared to a traditional architecture:
  • Flexibility (Agility): MSA makes systems as flexible as possible (Responsive to Change), i.e. allowing to push new functionality to users more rapidly:
    • allow continuous refactoring, by ensuring isolation of changes (i.e. eliminate dependencies). This means development teams can make changes without worrying about negative side effects (Own responsibility). Furthermore continuous delivery is possible, allowing to bring a changed service to production immediately when the change is ready, rather than having to wait for a few major releases per year (which is typically the case with a monolithic application).
    • allow to make changes by multiple development teams on the same system. Through the modularity of microservices, each individual service can be developed and deployed independently. This allows to easily scale development upon need.
  • Reusability: MSA facilitates reuse by creating composable, modular services. Due to the small, modular nature of microservices, the reusability will be considerably easier, compared to large monolithic applications.
  • High Availability: when considering availability, this can be split in unplanned unavailability (i.e. due to failures) and planned unavailability (e.g. for application maintenance or deploying releases). The MSA architecture will bring considerable improvements on both types, i.e.
    • Resilience: a resilient system is designed for failure. This is obtained by an intelligent handling of failures by microservices and a high-degree of redundancy in the architecture (i.e. fail independently, if 1 microservice goes down, the work can be taken over by others). The intelligent handling of failures is handled by isolating the fault, i.e. a fault in one service should not impact any other service. This means that in case of failure, only the functionality offered by that microservice will not be available. E.g. if a bank’s microservice to place a securities order fails, it will still be possible for the customer to see his securities positions.
      Many technology companies even deliberately induce failure (e.g. via Netflix’s Simian Army) to actively test their system’s resilience.
    • Planned unavailability: by design microservices typically do not need "planned unavailability". For example, deploying a new version of a microservice will typically not require any downtime. Also, maintenance is limited by the use of new technologies (e.g. through NoSQL databases) and by the small size of a microservice (if maintenance is required, it will only require a short downtime).
  • Elastic Scalability: MSA simplifies the elastic scaling of the application, based on demand, thus efficiently using the cloud infrastructure. Microservices should be both horizontally (scale-out) and vertically (scale-in) scalable. Horizontal scalability is inherent to the characteristics of a microservice, i.e. since microservices are small, independent and full stack systems, horizontal scaling can be realized by just changing the number of instances.
  • Shorter Learning Curve for designers and developers: given that microservices are small, single-purpose and isolated, it is easy for designers and developers to quickly understand a (new) microservice.

3.2. Challenges & Drawbacks

Even though the benefits are significant, there are of course also several challenges and drawbacks linked to the use of microservices.
Most common concerns are:
  • Difficult to perform end to end functional testing: end to end testing requires executing tests over multiple microservices, which are all developed and deployed independently, making it difficult to define and organize such tests.
  • Data consistency is difficult to guarantee: since each microservice has its own decentralized data and runs independently, keeping all data in sync can become very complex. E.g. it is not possible to perform different updates - handled by different microservices - in a single database commit (which can be rolled back as a whole in case of issue).
  • Data querying: many searches in end user applications, require joins between different data entities. When these data entities are managed by different micro-services such joins are no longer possible (as in different databases, but also such joins would lead to new dependencies). Alternative solutions like orchestration and aggregation in the application layer or replication of data (e.g. materialized views) are possible, but increase complexity considerably.
  • The overall complexity of MSA is much higher than a monolithic architecture, due to the number of "moving" parts. This makes supporting features, like load balancing, monitoring, logging, auditing…​ much more complex.
  • Application Governance: 'governance' refers to establishing and enforcing how people and solutions work together to achieve organisational objectives. In a SOA architecture, a heavy central SOA governance is imposed, which determines how services should be designed and developed and how these services should change over time. In a microservices architecture the design and build of different services is done decentral, meaning every team is fully in control of those decisions. This gives a huge challenge in large organisations to properly organise and coordinate the different teams, so that the overall architecture is still maintainable and properly understood and that all teams work towards common business goals.
  • Difficult to define a good MSA architecture: there is no rule book to setup MSA, so many companies attempting to setup an MSA architecture end up with a worse architecture than they initially started from. Especially choosing the right technology stack and size of a microservice tends to be a challenge (small enough and no smaller). Designing a microservice too big brings us back to the issues of a monolithic application, while a too small microservice results in too much overhead complexity.
  • Multitude of technology stacks: the independence of development teams to choose their preferred hardware, development language…​ leads quickly to a multitude of technology stacks. This can make it difficult to switch developers from one microservice to the other, but also increases operational complexity.
  • Performance: the high number of remote calls (instead of in-process calls) between microservices can have an impact on the overall technical performance of the system.
  • Increased Memory Consumption: with each microservice being a full technology stack and having its own data storage, there is quite some overhead, resulting in an increased usage of memory. By using a distributed architecture, consisting of cheap, commodity hardware this drawback can be tackled by just extending the hardware (i.e. size of the memory).

4. How does it impact the Financial Services industry?

Historically the Financial Services industry was at the head of technological evolutions, but in more recent years most banks and insurers are lagging behind, mainly because of their dependency on their large monolithic legacy systems. These highly inflexible systems have become maintenance nightmares. Even the smallest changes, require considerable investments and lead time. Additionally, these systems are often batch-based, with large windows of unavailability (for running batches, planned administration or for deploying changes).
At the same time, the financial services industry is becoming more and more digital. Many customers now do all their financials via the internet or mobile, meaning these digital interactions become their only experience with the bank or insurer. Customers are therefore expecting a similar user experience as offered by technology companies like Amazon, Netflix, Apple, Google…​ Furthermore, regulators are demanding an architecture which is more open to the outside world (i.e. an open API ecosystem) and more flexible to adapt to fast regulatory changes.
A microservices based architecture would allow a company to profit from the above described benefits (flexibility, scalability, availability, real-time interactions…​), allowing it to deliver new, innovative financial services quickly. This would allow banks and insurers to compete directly with the new FinTech players offering such customer-centric services.
Such an architecture would also permit companies not having to acquire large monolithic software packages, which often include a lot of unnecessary and unadapted functionality. Instead they could select the best-in-breed services (developed internally, downloaded from open source repositories, offered by external software vendors or partner with external FinTechs via an open API ecosystem).
The conclusion is that a shift in application architecture becomes almost inevitable. Banks and insurers are however hesitant to replace their well-proven and stable legacy systems by these new, innovative (and often considered to be experimental) architectures, for which the resulting operational risks (like e.g. security, auditability…​) are hard to predict. Furthermore, after several failures of large, expensive and time-consuming transformation programs (e.g. replacing custom legacy systems by packaged core banking platforms) aiming to replace the core legacy systems, companies have become hesitant to step again in such an adventure.
At the same time the financial services industry will also have to adopt the new organizational methodologies linked to an MSA architecture, i.e. DevOps and Agile software delivery. This is also a revolution in this industry, where a strong split between development teams and operational teams is still imposed by compliance and departments are still small "kingdoms" (meaning that cross-domain project teams are hard to accomplish).

5. Migration Approach

As indicated in the previous paragraph, a lot of banks and insurers have had bad experiences with large big-bang transformation programs. The migration approach from the traditional monolithic architecture (in best case already based on the principles of a SOA architecture) to an MSA architecture should be gradual.
This gradual non-invasive transition can be perfectly supported. In a first phase, the legacy system could even remain, but shielded of by an MSA architecture on top of the legacy systems. This would decouple the fast-changing front-end from the robust, slow-changing backend systems and allow companies to deliver new services, without threatening the stability of their operations.
Afterwards the legacy systems could (not necessarily) also gradually be replaced by microservices. This would also be done by ripping out one by one small pieces from the existing systems, until these systems are fully replaced by a collection of microservices. Initially these newly created microservices would still have to communicate with the existing legacy systems, but gradually this would be replaced by a full microservices based architecture.

6. Conclusion

Banks and insurer should (if not already the case) start setting up an MSA architecture, preferably with all associated technologies and methodologies (DevOps, Agile, Cloud and containerization). To avoid however a theoretical architecture exercise, this setup of the new architecture should be done in context of a pilot project, which will take care of the setup of the MSA building blocks and immediately consume those blocks. There is no need to setup immediately the state-of-the-art MSA architecture, but only what is needed to successfully deliver the selected pilot project. Gradual evolutions towards a more mature architecture can be obtained when delivering the next projects. Important however is the selection of a good pilot project, i.e. a non time-critical project, which is sponsored from the top management and which provides some satellite features, not requiring too much integration with existing core legacy systems.
Labels:

Comments

Post a Comment

Popular posts from this blog

Transforming the insurance sector to an Open API Ecosystem

1. Introduction "Open" has recently become a new buzzword in the financial services industry, i.e.   open data, open APIs, Open Banking, Open Insurance …​, but what does this new buzzword really mean? "Open" refers to the capability of companies to expose their services to the outside world, so that   external partners or even competitors   can use these services to bring added value to their customers. This trend is made possible by the technological evolution of   open APIs (Application Programming Interfaces), which are the   digital ports making this communication possible. Together companies, interconnected through open APIs, form a true   API ecosystem , offering best-of-breed customer experience, by combining the digital services offered by multiple companies. In the   technology sector   this evolution has been ongoing for multiple years (think about the travelling sector, allowing you to book any hotel online). An excelle...
55 comments
Read more

Are product silos in a bank inevitable?

Silo thinking   is often frowned upon in the industry. It is often a synonym for bureaucratic processes and politics and in almost every article describing the threats of new innovative Fintech players on the banking industry, the strong bank product silos are put forward as one of the main blockages why incumbent banks are not able to (quickly) react to the changing customer expectations. Customers want solutions to their problems   and do not want to be bothered about the internal organisation of their bank. Most banks are however organized by product domain (daily banking, investments and lending) and by customer segmentation (retail banking, private banking, SMEs and corporates). This division is reflected both at business and IT side and almost automatically leads to the creation of silos. It is however difficult to reorganize a bank without creating new silos or introducing other types of issues and inefficiencies. An organization is never ideal and needs to take a numbe...
6 comments
Read more

RPA - The miracle solution for incumbent banks to bridge the automation gap with neo-banks?

Hypes and marketing buzz words are strongly present in the IT landscape. Often these are existing concepts, which have evolved technologically and are then renamed to a new term, as if it were a brand new technology or concept. If you want to understand and assess these new trends, it is important to   reduce the concepts to their essence and compare them with existing technologies , e.g. Integration (middleware) software   ensures that 2 separate applications or components can be integrated in an easy way. Of course, there is a huge evolution in the protocols, volumes of exchanged data, scalability, performance…​, but in essence the problem remains the same. Nonetheless, there have been multiple terms for integration software such as ETL, ESB, EAI, SOA, Service Mesh…​ Data storage software   ensures that data is stored in such a way that data is not lost and that there is some kind guaranteed consistency, maximum availability and scalability, easy retrieval...
28 comments
Read more

IoT - Revolution or Evolution in the Financial Services Industry

1. The IoT hype We have all heard about the   "Internet of Things" (IoT)   as this revolutionary new technology, which will radically change our lives. But is it really such a revolution and will it really have an impact on the Financial Services Industry? To refresh our memory, the Internet of Things (IoT) refers to any   object , which is able to   collect data and communicate and share this information (like condition, geolocation…​)   over the internet . This communication will often occur between 2 objects (i.e. not involving any human), which is often referred to as Machine-to-Machine (M2M) communication. Well known examples are home thermostats, home security systems, fitness and health monitors, wearables…​ This all seems futuristic, but   smartphones, tablets and smartwatches   can also be considered as IoT devices. More importantly, beside these futuristic visions of IoT, the smartphone will most likely continue to be the cent...
18 comments
Read more

PSD3: The Next Phase in Europe’s Payment Services Regulation

With the successful rollout of PSD2, the European Union (EU) continues to advance innovation in the payments domain through the anticipated introduction of the   Payment Services Directive 3 (PSD3) . On June 28, 2023, the European Commission published a draft proposal for PSD3 and the   Payment Services Regulation (PSR) . The finalized versions of this directive and associated regulation are expected to be available by late 2024, although some predictions suggest a more likely timeline of Q2 or Q3 2025. Given that member states are typically granted an 18-month transition period, PSD3 is expected to come into effect sometime in 2026. Notably, the Commission has introduced a regulation (PSR) alongside the PSD3 directive, ensuring more harmonization across member states as regulations are immediately effective and do not require national implementation, unlike directives. PSD3 shares the same objectives as PSD2, i.e.   increasing competition in the payments landscape and en...
1 comment
Read more

Trade-offs Are Inevitable in Software Delivery - Remember the CAP Theorem

In the world of financial services, the integrity of data systems is fundamentally reliant on   non-functional requirements (NFRs)   such as reliability and security. Despite their importance, NFRs often receive secondary consideration during project scoping, typically being reduced to a generic checklist aimed more at compliance than at genuine functionality. Regrettably, these initial NFRs are seldom met after delivery, which does not usually prevent deployment to production due to the vague and unrealistic nature of the original specifications. This common scenario results in significant end-user frustration as the system does not perform as expected, often being less stable or slower than anticipated. This situation underscores the need for   better education on how to articulate and define NFRs , i.e. demanding only what is truly necessary and feasible within the given budget. Early and transparent discussions can lead to system architecture being tailored more close...
Post a Comment
Read more

Low- and No-code platforms - Will IT developers soon be out of a job?

“ The future of coding is no coding at all ” - Chris Wanstrath (CEO at GitHub). Mid May I posted a blog on RPA (Robotic Process Automation -   https://bankloch.blogspot.com/2020/05/rpa-miracle-solution-for-incumbent.html ) on how this technology, promises the world to companies. A very similar story is found with low- and no-code platforms, which also promise that business people, with limited to no knowledge of IT, can create complex business applications. These   platforms originate , just as RPA tools,   from the growing demand for IT developments , while IT cannot keep up with the available capacity. As a result, an enormous gap between IT teams and business demands is created, which is often filled by shadow-IT departments, which extend the IT workforce and create business tools in Excel, Access, WordPress…​ Unfortunately these tools built in shadow-IT departments arrive very soon at their limits, as they don’t support the required non-functional requirements (like h...
15 comments
Read more

An overview of 1-year blogging

Last week I published my   60th post   on my blog called   Bankloch   (a reference to "Banking" and my family name). The past year, I have published a blog on a weekly basis, providing my humble personal vision on the topics of Fintech, IT software delivery and mobility. This blogging has mainly been a   personal enrichment , as it forced me to dive deep into a number of different topics, not only in researching for content, but also in trying to identify trends, innovations and patterns into these topics. Furthermore it allowed me to have several very interesting conversations and discussions with passionate colleagues in the financial industry and to get more insights into the wonderful world of blogging and more general of digital marketing, exploring subjects and tools like: Search Engine Optimization (SEO) LinkedIn post optimization Google Search Console Google AdWorks Google Blogger Thinker360 Finextra …​ Clearly it is   not easy to get the necessary ...
9 comments
Read more

The UPI Phenomenon: From Zero to 10 Billion

If there is one Indian innovation that has grabbed   global headlines , it is undoubtedly the instant payment system   UPI (Unified Payments Interface) . In August 2023, monthly UPI transactions exceeded an astounding 10 billion, marking a remarkable milestone for India’s payments ecosystem. No wonder that UPI has not only revolutionized transactions in India but has also gained international recognition for its remarkable growth. Launched in 2016 by the   National Payments Corporation of India (NPCI)   in collaboration with 21 member banks, UPI quickly became popular among consumers and businesses. In just a few years, it achieved   remarkable milestones : By August 2023, UPI recorded an unprecedented   10.58 billion transactions , with an impressive 50% year-on-year growth. This volume represented approximately   190 billion euros . In July 2023, the UPI network connected   473 different banks . UPI is projected to achieve a staggering   1 ...
1 comment
Read more

AI in Financial Services - A buzzword that is here to stay!

In a few of my most recent blogs I tried to   demystify some of the buzzwords   (like blockchain, Low- and No-Code platforms, RPA…​), which are commonly used in the financial services industry. These buzzwords often entail interesting innovations, but contrary to their promise, they are not silver bullets solving any problem. Another such buzzword is   AI   (or also referred to as Machine Learning, Deep Learning, Enforced Learning…​ - the difference between those terms put aside). Again this term is also seriously hyped, creating unrealistic expectations, but contrary to many other buzzwords, this is something I truly believe will have a much larger impact on the financial services industry than many other buzzwords. This opinion is backed by a study of McKinsey and PWC indicating that 72% of company leaders consider that AI will be the most competitive advantage of the future and that this technology will be the most disruptive force in the decades to come. Deep Lea...
10 comments
Read more