One of the most vital and oldest departments of any national or state government in the world is taxation authority. Taxes are the government’s revenue, which is needed for social welfare, infrastructure projects, and sustaining the economy of the country. In this article, I will explain how to convert from batch-based to microservices using this tax system use case as an example.
Apart from being the oldest, this is one of the departments which needs to be very agile due to frequently changing national and international policies. However, in a majority of countries, the current taxation systems were developed a decade or two ago. Any new changes are implemented in silos and in a haphazard manner, which has made the whole system very rigid and very difficult to evolve alongside changing needs.
Traditionally, the tax calculations system is implemented using a monolithic architecture with a mix of SOA for service orchestration. In most cases, it is a very tightly coupled batch-based architecture as shown in the figure below. Because of this, the response time is prolonged. It takes at least 2-3 weeks for tax returns to be calculated and sent to the individuals/corporations. Also, adding a new taxation policy at the national level requires months of implementation and testing.
Before we convert this into microservices, here is a little refresher on the benefits of event-driven architecture:
In the last few years, there has been a significant drive towards moving from monolithic architecture to event-driven architecture (EDA).
“Event-driven architecture (EDA) is a software architecture paradigm promoting the production, detection, consumption of, and reaction to events.” -Wikipedia
In other words, event-driven architecture is an IT strategy that has evolved from SOA, where the flow of events is consumed and produced by microservices, functions, applications in real-time for business logic, insights, AI/ML, and IoT scenarios.
From a business perspective, an event-driven architecture can allow for better customer experiences and quicker times to market. Consider this: a business reacting to real-time data can be more proactive and responsive when it comes to its customers. Using real-time data in unique ways can give a business an edge over its competition, allowing for a quicker response to changing market demands and customer expectations. For example, an airline’s loyalty program could be notified when a check-in event occurs, resulting in a chain of events that includes a congratulatory beverage upon boarding. A check-in event could also trigger other systems that track the likelihood of a plane departing on time. In this ever-changing landscape, new use cases for real-time data can be the key to a competitive advantage that wasn’t previously considered due to technical limitations.
From a technical perspective, being event-driven provides better agility and easy adoption of cloud-native services such as Google Beam and AWS Lambda functions, thus providing an innovative backbone for new business use-cases.
Now let’s get back to the tax example.
The high-level component diagram shown above displays three main monolithic applications for doing the tax assessment. They are:
The monolithic, batch-based tax processes can convert into various event-driven microservices that support specific business functions, as shown in the figure below. Once that happens, they still need to communicate and exchange information – this is done by using an event broker. An event broker helps with the choreography of microservices, which can also be seen in the diagram:
One of the significant functionalities of an event broker is its ability to dynamically determine if a new consumer is added for the same subscription. Accordingly, it can also load balance the events between the multiple instances of the same consumer. Similar to how REST APIs can do auto-scale-up using API gateways, there can be multiple instances of the same event-driven microservice listening on the same queue and providing parallel processing.
This mechanism allows the flexibility to have in-process auto-scale-up and auto-scale-down depending on the load on the system. Alternatively, we can individually auto-scale only those services which are having a higher load.
An event broker should also be able to throttle the consumer traffic based on how many events a consumer is able to process. It should be able to do ‘weighted round robin’ delivery of the messages. Weighted round robin essentially means keeping track of the TCP acknowledgement from each of the consumers and intelligently pushing the traffic to the consumer based on its performance.
Now, these microservices can be further divided based on more fine-grained business functions. For example, the “Calculate Tax” microservice can be divided into:
Similarly, validation services would have different logic for different types of taxes.
Once the granularity of microservices has been decided, an event broker can be used to provide choreography of the microservices as shown in the figure below:
There is a separate microservice for various business functions. This allows one microservice to change, scale, and be reused without affecting any other services.
In a monolithic architecture, the example above would be a separate “if-else” function in the application logic. One can certainly argue that the execution is much faster in this case since there is no network latency (as there would between two microservices), and implementation is quick and easy from the developer’s point of view; however, implementing this as separate microservices has its advantages:
There is a delicate balance that needs to be maintained, taking into consideration all the factors outlined above, when deciding the granularity of a microservice.
The Architect's Guide to Building a Responsive, Elastic and Resilient EnvironmentSr. Architect Jonathan Schabowsky addresses the challenges of microservices architecture and shares his perspective on the modern messaging integration patterns architects can leverage.
While there may be many microservices with synchronous request-reply patterns which are powered by RESTful APIs, there would be many event-driven microservices that would publish and subscribe to events. Thus, these microservices would be loosely coupled using the power of the topic routing feature of the advanced broker.
Each of the microservices in the tax example emits an event – essentially a message with a topic attached to it. A topic defines the event; it identifies what the event represents. For instance, a calculated private property tax assessment event can have a topic as “tax/calculated/PT/private”. A topic can even be more descriptive if required, including things like version number, tax assessment year, etc. (eg. tax/v1.0/2019/calculated/PT/private).
To learn more about creating a sound topic hierarchy, I encourage you to read this post: Topic Hierarchy and Topic Architecture Best Practices
An event has two parts: topics and payload. A payload is a message itself which would be processed by the service. You can read this older post for more about payload data.
As mentioned earlier, a microservice can publish and consume one or more events. For instance, the “Retrieve Tax Returns” service would be consuming events from the topic “tax/assess/”. It would also publish events on two separate topics for different types of taxes such as: “tax/retrieve/PT” for property tax and “tax/retrieve/CT” for corporate tax.
These events will, in turn, be subscribed to by “Validate Property Tax” and “Validate Corporate Tax” respectively.
In this way, all microservices can be choreographed, as shown in the figure below, only starting its dance (or execution) when an event the service has subscribed to is published.
This architecture provides agility and flexibility, since adding a new microservice will not affect existing functions. New microservices can be added without bringing down the whole system, which would be the case in a monolithic architecture or even with a SOA-based service bus, which uses the orchestration of the services.
In the figure below, we see how audit/logging/error handling microservices can be added seamlessly:
As an enterprise starts getting more and more events, managing, discovering, and utilizing those events will eventually become a challenge for the organization. Enterprise architects will have to answer some tough questions such as:
If events and microservices are not governed appropriately, it won’t be long when the talks within the organization sound a little like this:
An event portal provides developers and architects tools to design, describe, and discover events within their system. It helps them see the relationships between applications and events, making event-driven applications and microservices easier to design, deploy, and evolve.
From our tax example above and using the PubSub+ Event Portal design space, we can design the following:
The PubSub+ Event Portal can be considered a design-time component, which allows architects to view and design applications while being mindful of their relationships. You can read more about it here or watch this demo to get started.
Event-driven methodology is still quite new and is rapidly making its way across industries. As more and more data is captured, new and innovative use cases are constantly being discovered. An enterprise has a lot to gain from moving towards an event-driven architecture. More agility, better flexibility, reduced time to market, and improved customer experiences are business objectives that go hand-in-hand with a proper foundation for event-driven microservices.
For more on microservices and PubSub+ Event Portal, bookmark these sections on our Resources Hub to keep up to date on a variety of helpful content: