Skip to main content

Microservices - Yet another buzzword or a real innovation for the Financial Services Industry

1. Introduction

Microservices are the new hot trend in the application architecture landscape. Like SOA (Service Oriented Architecture) and cloud a few years ago, "microservices" is the buzzword used by marketeers to position their solution as cutting-edge technology. The concept of microservices is however not new. Leading technology companies, like Google, Amazon and Facebook, have already migrated from a monolithic to a microservice based architecture (MSA) for over a decade, and more recently companies like Netflix, eBay, Twitter and PayPal have also evolved. Their track record with this architecture is impressive, e.g. Netflix is responsible for about 30% of the internet traffic and handles about one billion calls per day from 800 different types of devices.
Unfortunately, finding a good definition for the term "microservice" appears to be extremely hard. When going back to the basics, a microservice based architecture is the same as a SOA architecture with the exception that several additional constraints are imposed to the services. We could therefore call MSA the version 2.0 of SOA.
This (r)evolution of SOA is mainly driven by a number of recent technological and organisational evolutions, which made the microservices architecture possible (or at least more attractive). MSA is therefore heavily interconnected with these technologies and methodologies:
  • (Private) Cloud: the main driver for firms to move to the cloud is "elasticity" (a firm pays only for the resources it needs, but can scale immediately when the load increases). This "elasticity" requires an application architecture, which can effectively utilize this capability. Microservices can give an answer to this need.
  • APIs: the rapid adoption of APIs in recent years from a development technique to a business model driver, aims to reuse, share and monetize the organization’s assets. These principles fit perfectly with MSA, which will typically enable firms to create an (open) API ecosystem. Basically, MSA is breaking up an application or services in many different components, which are glued together through APIs (i.e. standardized formats for communication).
  • (Docker) Containers: containers allow firms to deploy software built in a variety of technology stacks in a homogeneous, simple and very quick way. Without the use of containers, the deployment of microservices would be extremely complex.
  • DevOps: the DevOps movement aims to improve the collaboration of development teams and operations by integrating those teams and automating the software delivery process. MSA loses a lot of its benefits and becomes very hard to manage (especially the deployment) without these DevOps principles.
  • Agile: Agile is a software development methodology where small (2-pizza) self-organizing teams develop software by building up the requirements and software gradually. Organizing a firm with MSA with a waterfall software methodology would lose almost all benefits associated to MSA (i.e. flexibility, short-time to market, independent development teams…​).
MSA combined with all those technologies and methodologies allows a rapid software delivery and an application landscape supporting rapid change and growth of the business.

2. Definition and Concepts

Microservices are all about "doing one thing and doing it well", i.e. build an application architecture with small, API-accessible, single-purpose components.
As explained above, a clear unique definition of microservices is hard to give, but all experts do agree about several common characteristics:
  • Small: microservices are by definition small. Of course, "small" is subjective, so finding the right size of a microservice is more an art than science.
  • Single-purpose: each service should be organized around business capabilities and run a unique process to serve a business goal.
  • Independent: microservices must be independent, modular components. This means:
    • Service should run fully independent, i.e. a service must be able to handle a request independently. This means it can also independently be replaced or upgraded.
      It also means that a microservice should be stateless, i.e. every request is handled with the information contained only within it.
    • Service can be autonomously developed, i.e. development can be done by fully autonomous teams.
    • Service should be independently deployable, i.e. it is possible to deploy a new version of a microservice without being forced to deploy any other component. This means that a microservice must be a full stack application, i.e. the business logic, data model and service interface (API or user interface) should all be included in the microservice.
  • API-accessible: microservices should communicate with the outside world, using well-defined, lightweight protocols.
  • Simple Integration: the integration of microservices should be as simple as possible. Typically, this is done by dumb, fast and asynchronous (publish - subscribe) messaging.
  • Design for failure: microservices should be designed so that they can react to failure, i.e. identify failure and fix it without impacting other services.
  • Decentralized governance: teams developing microservices are fully autonomous, choosing themselves the programming language, database, hardware…​ This means no central guidelines for these choices, but also a decentralization of data, i.e. no large central relational database, but instead each service managing its unique database.
    Nonetheless teams are encouraged to share code libraries with other teams and reuse existing code libraries.
  • DevOps approach
    • Automated Software Delivery: software delivery is as automated as possible. This includes concepts like Continuous Delivery, Continuous Integration and Automated Testing.
    • Deliver products, not projects: the teams building the microservices should own the services over its full life-cycle (you build it, you run it), i.e. also take the responsibility for the software in production. This brings the development teams much closer to the end-user.
With these characteristics in mind, it is clear that MSA is particularly suitable when multiple platforms and devices need to be supported, flexibility (i.e. frequent changes) is key and functionality can be easily split up in small components.
MSA should not be considered as a one-fit-all solution, i.e. it is not the ideal solution for all applications. A poorly designed MSA-based application landscape will still be unmaintainable.

3. Benefits and Drawbacks

3.1. Benefits

The usage of a microservices based architecture gives several advantages compared to a traditional architecture:
  • Flexibility (Agility): MSA makes systems as flexible as possible (Responsive to Change), i.e. allowing to push new functionality to users more rapidly:
    • allow continuous refactoring, by ensuring isolation of changes (i.e. eliminate dependencies). This means development teams can make changes without worrying about negative side effects (Own responsibility). Furthermore continuous delivery is possible, allowing to bring a changed service to production immediately when the change is ready, rather than having to wait for a few major releases per year (which is typically the case with a monolithic application).
    • allow to make changes by multiple development teams on the same system. Through the modularity of microservices, each individual service can be developed and deployed independently. This allows to easily scale development upon need.
  • Reusability: MSA facilitates reuse by creating composable, modular services. Due to the small, modular nature of microservices, the reusability will be considerably easier, compared to large monolithic applications.
  • High Availability: when considering availability, this can be split in unplanned unavailability (i.e. due to failures) and planned unavailability (e.g. for application maintenance or deploying releases). The MSA architecture will bring considerable improvements on both types, i.e.
    • Resilience: a resilient system is designed for failure. This is obtained by an intelligent handling of failures by microservices and a high-degree of redundancy in the architecture (i.e. fail independently, if 1 microservice goes down, the work can be taken over by others). The intelligent handling of failures is handled by isolating the fault, i.e. a fault in one service should not impact any other service. This means that in case of failure, only the functionality offered by that microservice will not be available. E.g. if a bank’s microservice to place a securities order fails, it will still be possible for the customer to see his securities positions.
      Many technology companies even deliberately induce failure (e.g. via Netflix’s Simian Army) to actively test their system’s resilience.
    • Planned unavailability: by design microservices typically do not need "planned unavailability". For example, deploying a new version of a microservice will typically not require any downtime. Also, maintenance is limited by the use of new technologies (e.g. through NoSQL databases) and by the small size of a microservice (if maintenance is required, it will only require a short downtime).
  • Elastic Scalability: MSA simplifies the elastic scaling of the application, based on demand, thus efficiently using the cloud infrastructure. Microservices should be both horizontally (scale-out) and vertically (scale-in) scalable. Horizontal scalability is inherent to the characteristics of a microservice, i.e. since microservices are small, independent and full stack systems, horizontal scaling can be realized by just changing the number of instances.
  • Shorter Learning Curve for designers and developers: given that microservices are small, single-purpose and isolated, it is easy for designers and developers to quickly understand a (new) microservice.

3.2. Challenges & Drawbacks

Even though the benefits are significant, there are of course also several challenges and drawbacks linked to the use of microservices.
Most common concerns are:
  • Difficult to perform end to end functional testing: end to end testing requires executing tests over multiple microservices, which are all developed and deployed independently, making it difficult to define and organize such tests.
  • Data consistency is difficult to guarantee: since each microservice has its own decentralized data and runs independently, keeping all data in sync can become very complex. E.g. it is not possible to perform different updates - handled by different microservices - in a single database commit (which can be rolled back as a whole in case of issue).
  • Data querying: many searches in end user applications, require joins between different data entities. When these data entities are managed by different micro-services such joins are no longer possible (as in different databases, but also such joins would lead to new dependencies). Alternative solutions like orchestration and aggregation in the application layer or replication of data (e.g. materialized views) are possible, but increase complexity considerably.
  • The overall complexity of MSA is much higher than a monolithic architecture, due to the number of "moving" parts. This makes supporting features, like load balancing, monitoring, logging, auditing…​ much more complex.
  • Application Governance: 'governance' refers to establishing and enforcing how people and solutions work together to achieve organisational objectives. In a SOA architecture, a heavy central SOA governance is imposed, which determines how services should be designed and developed and how these services should change over time. In a microservices architecture the design and build of different services is done decentral, meaning every team is fully in control of those decisions. This gives a huge challenge in large organisations to properly organise and coordinate the different teams, so that the overall architecture is still maintainable and properly understood and that all teams work towards common business goals.
  • Difficult to define a good MSA architecture: there is no rule book to setup MSA, so many companies attempting to setup an MSA architecture end up with a worse architecture than they initially started from. Especially choosing the right technology stack and size of a microservice tends to be a challenge (small enough and no smaller). Designing a microservice too big brings us back to the issues of a monolithic application, while a too small microservice results in too much overhead complexity.
  • Multitude of technology stacks: the independence of development teams to choose their preferred hardware, development language…​ leads quickly to a multitude of technology stacks. This can make it difficult to switch developers from one microservice to the other, but also increases operational complexity.
  • Performance: the high number of remote calls (instead of in-process calls) between microservices can have an impact on the overall technical performance of the system.
  • Increased Memory Consumption: with each microservice being a full technology stack and having its own data storage, there is quite some overhead, resulting in an increased usage of memory. By using a distributed architecture, consisting of cheap, commodity hardware this drawback can be tackled by just extending the hardware (i.e. size of the memory).

4. How does it impact the Financial Services industry?

Historically the Financial Services industry was at the head of technological evolutions, but in more recent years most banks and insurers are lagging behind, mainly because of their dependency on their large monolithic legacy systems. These highly inflexible systems have become maintenance nightmares. Even the smallest changes, require considerable investments and lead time. Additionally, these systems are often batch-based, with large windows of unavailability (for running batches, planned administration or for deploying changes).
At the same time, the financial services industry is becoming more and more digital. Many customers now do all their financials via the internet or mobile, meaning these digital interactions become their only experience with the bank or insurer. Customers are therefore expecting a similar user experience as offered by technology companies like Amazon, Netflix, Apple, Google…​ Furthermore, regulators are demanding an architecture which is more open to the outside world (i.e. an open API ecosystem) and more flexible to adapt to fast regulatory changes.
A microservices based architecture would allow a company to profit from the above described benefits (flexibility, scalability, availability, real-time interactions…​), allowing it to deliver new, innovative financial services quickly. This would allow banks and insurers to compete directly with the new FinTech players offering such customer-centric services.
Such an architecture would also permit companies not having to acquire large monolithic software packages, which often include a lot of unnecessary and unadapted functionality. Instead they could select the best-in-breed services (developed internally, downloaded from open source repositories, offered by external software vendors or partner with external FinTechs via an open API ecosystem).
The conclusion is that a shift in application architecture becomes almost inevitable. Banks and insurers are however hesitant to replace their well-proven and stable legacy systems by these new, innovative (and often considered to be experimental) architectures, for which the resulting operational risks (like e.g. security, auditability…​) are hard to predict. Furthermore, after several failures of large, expensive and time-consuming transformation programs (e.g. replacing custom legacy systems by packaged core banking platforms) aiming to replace the core legacy systems, companies have become hesitant to step again in such an adventure.
At the same time the financial services industry will also have to adopt the new organizational methodologies linked to an MSA architecture, i.e. DevOps and Agile software delivery. This is also a revolution in this industry, where a strong split between development teams and operational teams is still imposed by compliance and departments are still small "kingdoms" (meaning that cross-domain project teams are hard to accomplish).

5. Migration Approach

As indicated in the previous paragraph, a lot of banks and insurers have had bad experiences with large big-bang transformation programs. The migration approach from the traditional monolithic architecture (in best case already based on the principles of a SOA architecture) to an MSA architecture should be gradual.
This gradual non-invasive transition can be perfectly supported. In a first phase, the legacy system could even remain, but shielded of by an MSA architecture on top of the legacy systems. This would decouple the fast-changing front-end from the robust, slow-changing backend systems and allow companies to deliver new services, without threatening the stability of their operations.
Afterwards the legacy systems could (not necessarily) also gradually be replaced by microservices. This would also be done by ripping out one by one small pieces from the existing systems, until these systems are fully replaced by a collection of microservices. Initially these newly created microservices would still have to communicate with the existing legacy systems, but gradually this would be replaced by a full microservices based architecture.

6. Conclusion

Banks and insurer should (if not already the case) start setting up an MSA architecture, preferably with all associated technologies and methodologies (DevOps, Agile, Cloud and containerization). To avoid however a theoretical architecture exercise, this setup of the new architecture should be done in context of a pilot project, which will take care of the setup of the MSA building blocks and immediately consume those blocks. There is no need to setup immediately the state-of-the-art MSA architecture, but only what is needed to successfully deliver the selected pilot project. Gradual evolutions towards a more mature architecture can be obtained when delivering the next projects. Important however is the selection of a good pilot project, i.e. a non time-critical project, which is sponsored from the top management and which provides some satellite features, not requiring too much integration with existing core legacy systems.
Labels:

Comments

Post a Comment

Popular posts from this blog

Transforming the insurance sector to an Open API Ecosystem

1. Introduction "Open" has recently become a new buzzword in the financial services industry, i.e.   open data, open APIs, Open Banking, Open Insurance …​, but what does this new buzzword really mean? "Open" refers to the capability of companies to expose their services to the outside world, so that   external partners or even competitors   can use these services to bring added value to their customers. This trend is made possible by the technological evolution of   open APIs (Application Programming Interfaces), which are the   digital ports making this communication possible. Together companies, interconnected through open APIs, form a true   API ecosystem , offering best-of-breed customer experience, by combining the digital services offered by multiple companies. In the   technology sector   this evolution has been ongoing for multiple years (think about the travelling sector, allowing you to book any hotel online). An excellent example of this
45 comments
Read more

Are product silos in a bank inevitable?

Silo thinking   is often frowned upon in the industry. It is often a synonym for bureaucratic processes and politics and in almost every article describing the threats of new innovative Fintech players on the banking industry, the strong bank product silos are put forward as one of the main blockages why incumbent banks are not able to (quickly) react to the changing customer expectations. Customers want solutions to their problems   and do not want to be bothered about the internal organisation of their bank. Most banks are however organized by product domain (daily banking, investments and lending) and by customer segmentation (retail banking, private banking, SMEs and corporates). This division is reflected both at business and IT side and almost automatically leads to the creation of silos. It is however difficult to reorganize a bank without creating new silos or introducing other types of issues and inefficiencies. An organization is never ideal and needs to take a number of cons
6 comments
Read more

RPA - The miracle solution for incumbent banks to bridge the automation gap with neo-banks?

Hypes and marketing buzz words are strongly present in the IT landscape. Often these are existing concepts, which have evolved technologically and are then renamed to a new term, as if it were a brand new technology or concept. If you want to understand and assess these new trends, it is important to   reduce the concepts to their essence and compare them with existing technologies , e.g. Integration (middleware) software   ensures that 2 separate applications or components can be integrated in an easy way. Of course, there is a huge evolution in the protocols, volumes of exchanged data, scalability, performance…​, but in essence the problem remains the same. Nonetheless, there have been multiple terms for integration software such as ETL, ESB, EAI, SOA, Service Mesh…​ Data storage software   ensures that data is stored in such a way that data is not lost and that there is some kind guaranteed consistency, maximum availability and scalability, easy retrieval and searching
26 comments
Read more

IoT - Revolution or Evolution in the Financial Services Industry

1. The IoT hype We have all heard about the   "Internet of Things" (IoT)   as this revolutionary new technology, which will radically change our lives. But is it really such a revolution and will it really have an impact on the Financial Services Industry? To refresh our memory, the Internet of Things (IoT) refers to any   object , which is able to   collect data and communicate and share this information (like condition, geolocation…​)   over the internet . This communication will often occur between 2 objects (i.e. not involving any human), which is often referred to as Machine-to-Machine (M2M) communication. Well known examples are home thermostats, home security systems, fitness and health monitors, wearables…​ This all seems futuristic, but   smartphones, tablets and smartwatches   can also be considered as IoT devices. More importantly, beside these futuristic visions of IoT, the smartphone will most likely continue to be the center of the connected devi
16 comments
Read more

Neobanks should find their niche to improve their profitability

The last 5 years dozens of so-called   neo- or challenger banks  (according to Exton Consulting 256 neobanks are in circulation today) have disrupted the banking landscape, by offering a fully digitized (cfr. "tech companies with a banking license"), very customer-centric, simple and fluent (e.g. possibility to become client and open an account in a few clicks) and low-cost product and service offering. While several of them are already valued at billions of euros (like Revolut, Monzo, Chime, N26, NuBank…​), very few of them are expected to be profitable in the coming years and even less are already profitable today (Accenture research shows that the average UK neobank loses $11 per user yearly). These challenger banks are typically confronted with increasing costs, while the margins generated per customer remain low (e.g. due to the offering of free products and services or above market-level saving account interest rates). While it’s obvious that disrupting the financial ma
7 comments
Read more

PFM, BFM, Financial Butler, Financial Cockpit, Account Aggregator…​ - Will the cumbersome administrative tasks on your financials finally be taken over by your financial institution?

1. Introduction Personal Financial Management   (PFM) refers to the software that helps users manage their money (budget, save and spend money). Therefore, it is often also called   Digital Money Management . In other words, PFM tools   help customers make sense of their money , i.e. they help customers follow, classify, remain informed and manage their Personal Finances. Personal Finance   used to be (or still is) a time-consuming effort , where people would manually input all their income and expenses in a self-developed spreadsheet, which would gradually be extended with additional calculations. Already for more than 20 years,   several software vendors aim to give a solution to this , by providing applications, websites and/or apps. These tools were never massively adopted, since they still required a lot of manual interventions (manual input of income and expense transaction, manual mapping transactions to categories…​) and lacked an integration in the day-to-da
5 comments
Read more

From app to super-app to personal assistant

In July of this year,   KBC bank   (the 2nd largest bank in Belgium) surprised many people, including many of us working in the banking industry, with their announcement that they bought the rights to   broadcast the highlights of soccer matches   in Belgium via their mobile app (a service called "Goal alert"). The days following this announcement the news was filled with experts, some of them categorizing it as a brilliant move, others claiming that KBC should better focus on its core mission. Independent of whether it is a good or bad strategic decision (the future will tell), it is clearly part of a much larger strategy of KBC to   convert their banking app into a super-app (all-in-one app) . Today you can already buy mobility tickets and cinema tickets and use other third-party services (like Monizze, eBox, PayPal…​) within the KBC app. Furthermore, end of last year, KBC announced opening up their app also to non-customers allowing them to also use these third-party servi
9 comments
Read more

Beyond Imagination: The Rise and Evolution of Generative AI Tools

Generative AI   has revolutionized the way we create and interact with digital content. Since the launch of Dall-E in July 2022 and ChatGPT in November 2022, the field has seen unprecedented growth. This technology, initially popularized by OpenAI’s ChatGPT, has now been embraced by major tech players like Microsoft and Google, as well as a plethora of innovative startups. These advancements offer solutions for generating a diverse range of outputs including text, images, video, audio, and other media from simple prompts. The consumer now has a vast array of options based on their specific   output needs and use cases . From generic, large-scale, multi-modal models like OpenAI’s ChatGPT and Google’s Bard to specialized solutions tailored for specific use cases and sectors like finance and legal advice, the choices are vast and varied. For instance, in the financial sector, tools like BloombergGPT ( https://www.bloomberg.com/ ), FinGPT ( https://fin-gpt.org/ ), StockGPT ( https://www.as
2 comments
Read more

Can Augmented Reality make daily banking a more pleasant experience?

With the   increased competition in the financial services landscape (between banks/insurers, but also of new entrants like FinTechs and Telcos), customers are demanding and expecting a more innovative and fluent digital user experience. Unfortunately, most banks and insurers, with their product-oriented online and mobile platforms, are not known for their pleasant and fluent user experience. The   trend towards customer oriented services , like personal financial management (with functions like budget management, expense categorization, saving goals…​) and robo-advise, is already a big step in the right direction, but even then, managing financials is still considered to be a boring intangible and complex task for most people. Virtual (VR) and augmented reality (AR)   could bring a solution. These technologies provide a user experience which is   more intuitive, personalised and pleasant , as they introduce an element of   gamification   to the experience. Both VR and AR
9 comments
Read more

Eco-systems - Welcome to a new cooperating world

Last week I attended the Digital Finance Summit conference in Brussels, organized by Fintech Belgium, B-Hive, Febelfin and EBF. A central theme of the summit was the cooperation between banks and Fintechs and more in general the rise of ecosystems. In the past I have written already about this topic in my blogs about "Transforming the bank to an Open API Ecosystem ( https://www.linkedin.com/pulse/transforming-bank-open-api-ecosystem-joris-lochy/ ) and "The war for direct customer contact - Banks should fight along!" ( https://www.linkedin.com/pulse/war-direct-customer-contact-banks-should-fight-along-joris-lochy/ ), but still I was surprised about the number of initiatives taken in this domain. In my last job at The Glue, I already had the pleasure to work on several interesting cases: TOCO   ( https://www.toco.eu ): bringing entrepreneurs, accountants and banks closer together, by supporting entrepreneurs and accountants in their daily admin (and in the f
9 comments
Read more