The Advantages of Building an Event-Driven Application with an Event Portal
Stephen Tsoi is a Senior Technical Manager at the Hong Kong Jockey Club and has more than 20 years of IT experience in solution and architecture design.
All digital transformation projects have one thing in common: they produce lots of events, i.e. changes of state that can be recognized, transmitted, processed, and reacted to by applications across an enterprise.
In my previous article “Designing and Naming Topics for Event-Driven Architecture with PubSub+”, I used a payment system as an example to explain how to build an event flow using the design principles of event-driven architecture (EDA). In this post, I will explain how to build an application that can support up to 200,000 concurrently connected customers with an event portal (like an API portal for events) that enables the management of thousands of events, event flows, schemas, etc.
Challenges with REST
Many organizations develop numerous products riding on a REST interface to deliver a richer customer experience. With this blog post I’ll present an example of providing sports betting across a variety of devices.
Depending on the channel application, the UI design properties such as screen size, hardware, or device will vary. For example, a Web application needs to provide rich information, while a Mobile client must offer a compact user experience and easy interaction. An Automatic Teller Machine (ATM), on the other hand, may display less information, but provide more shortcuts to different services such as ePayment, purchasing, and account operation. These three different types of applications will have different UI requirements but share over 90% of data.
Traditional REST design, which revolves around a model-view-controller pattern, is tightly coupled with backend systems exposing services to provide data endpoints to various but specific channel systems. To cater to different UI components or screen requirements, developers need to separate the program logic into interconnected layers, and present the data fields for these UI components or screens. This results in the same data field overlapping in multiple service calls, and these service calls are hard to reuse. Any requirement change would affect both the backend and the UI events, even if it’s something as simple as adding or removing a single data field.
If the affected service is shared by multiple channels, or data is used in multiple services, the change must also be implemented on all affected channels. This then also results in significant effort being required for regression, functional and non-functional testing on all affected channels.
Additionally, code implemented on client applications usually require complex logic in order to handle various operations; this means that the same screen may need to call multiple backend services behind the scenes. Applications need constant enhancements for new services based on UI requirements, and as a result, an application might need to maintain hundreds of channel services with longer latency.
Architecture Design
With EDA, you can eliminate any dependencies between channels and the backend systems. EDA uses asynchronous request/reply and publish/subscribe instead of REST’s synchronous request-reply, so data can be pushed to the channel as it is updated or whenever it becomes available. This reduces latency, improves the freshness of data, and increases efficiency since clients do not have to constantly poll even if nothing is changing. Using choreography, the channel can also publish their request (ePayment, purchasing, account operation etc.) and get back the result from the corresponding reply topic.
All that means channel developers can focus on UI presentation, as applications can simply subscribe to the data topic from specific events they are interested in and display them. Depending on the requirement, backend services can also subscribe to multiple channel requests via separated event queues, without the need for a middle layer service. The overall design becomes simpler, and time (and cost) is saved from having to maintain an extra service layer.
Lastly, channel behavior becomes more consistent, especially when you have multiple customer touchpoints in omni or ‘phygital’ use cases that combine physical and digital interactions.
Infrastructure Design
With a traditional REST-based “polling” design, you need to set up a huge number of Web servers and application servers to serve 200,000 concurrent user connections on each channel. Even when the design can use a CDN to cache traffic on SaaS to decrease the capacity requirement on infrastructure, you still need to set up dozens of servers to accommodate 10,000 connections worth of traffic when the hit rate is 5%. The CDN also introduces extra latency, and the value will depend on the page TTL (time-to-live) value and information update frequency. For example, the information update frequency is 5 seconds on the backend, and the CDN TTL is 2 seconds. Then the latency will be up to 2 seconds. You can’t just reduce the cache value to 1 second because it will affect the cache rate which will increase the capacity requirement on the Web servers. Finally, you need to build same number of Web servers and Application servers for each channel solution if they need to cater to the same amount of peak traffic.
The EDA-based “pushing” solution allows the channel to directly connect to the event bus via open protocol MQTT, and send a request or receive information update when it available. The data source can directly export the service end point to the channel, and avoid extra latency on the cache layer. The backend microservice design can scale up when it needs. This infrastructure will be simpler, more flexible and cost effective than the REST polling design. Referring to the following infrastructure diagram, the EDA solution can decrease latency from 14 seconds to 2 seconds (and lower..), and the cost will be cheaper by at least 20% when using the event bus to replace the Web server, Application server and Load balancer. This infrastructure can also cater multiple channel usage.
Application Data Flow Design
As mentioned before, the EDA-based solution uses publish/subscribe to implement information fan-out and asynchronous request-reply. My previous article explained how to define the Topic and Queue naming to implement orchestration flow. However, the EDA orchestration flow is for distribution control. You also need to put in place an event portal, which is similar to the REST API management portal/tools, to help design the EDA flow, and allow the flexibility to reuse these events for other projects.
To help developers leverage the many assets that make up an event-driven system – APIs, applications, schemes, etc. – you need to implement a self-service “event portal”. All applications, events or schema, and corresponding mappings are created and cataloged in the event portal during the design stage. Any new enhancements could easily be mapped to corresponding events from the event portal to ensure downstream consumers of data can detect enhanced events. Service providers will know all subscribers and any respective impact from updates made on their service or interface. The event portal can be built in-house or by a 3rd party, although there are limited options in the market. In the following paragraphs, I will use an event portal to show how to build an ePayment system to support payment transaction from different channels or banks.
The ePayment system is used to support EFT (Electronic Funds Transfer), PPS (Positive Payment System), FPS (Fast Payment Service) and bank fund transfer which are built on channels or banks with different UI presentation layers. The payment service publishes the initial event to its associated message brokers which are connected via an event mesh, and then waits for the reply asynchronously.
As you can see in the following diagram, the methods’ triggering points and subsequent processes are decoupled and handled via individual services. Eventually, these services need to update the corresponding account records in the database via a backend account service.
With an event portal, we first need to create corresponding applications and events based on the above diagram, and then import the AsyncAPI interface for each event as a schema. We can view the following diagram after deploying it to the event mesh via Runtime Event Manager, and the event catalog will show all applications/events/schemas in list view. You can right-click the corresponding icon on the diagram to view detailed information about an application or event.
Benefits of Event-Driven Architecture
In the modern world, organizations need to quickly respond to the ever-changing market to maintain competitiveness and maximize efficiency/profitability. Both business and IT should work hand in hand to accomplish this.
Business Benefits of EDA
- Faster time to market
- Optimize cost of change
- Lower, most consistent latency
- Reusability of UI components
- Lower Risk
Technical Benefits of EDA
- Shared services and events
- Standardized interface
- High throughput
- Real-time event reaction
- Reuse of design patterns
EDA can work with other new and emergent technologies and methodologies such as Agile, DDA, and DevOps to meet the requirements above. Product delivery cycles can be decreased from 8-12 months down to 1-2 weeks depending on the technology/methodology combinations used.
EDA models help decouple systems, while the Event Portal provides a platform to create, maintain and govern thousands of applications, events, and schemas. Riding on agile development we can split the development lifecycle into smaller sprints to allow more flexibility and easily accommodate any required changes. With this, we now follow a 5-step process to implement changes: Discover, Design, Develop, Test, and Deploy. This process can be repeated weekly or even daily.
EDA Extends the Lifetime of Events
To recap, all digital transformations have one thing in common: they have the capacity to produce massive amounts of events. Getting these events ‘in-motion’ across the enterprise, out of siloed architectures, provides possibilities of leveraging them for as-of-yet unimagined use cases.
Under EDA design, all events brought forward are the super set of the original event; extra services can be built to collect and analyze activities about client applications and customer behavior in order to enhance user experience and interactions.
In the following transformation diagram, the red circle represents the main operation with multiple event flows. First, we define the product to sell, and then the product triggers subsequent initial odds. The customer logs into their account to place a wager (tickets) on specific products (markets) via various channels. The turnover of specific products drives subsequent odds changes. This circular flow then repeats. All these events combined together provide feedback and insights for business through big data analytics. For example, data scientists can analyze millions of the same pattern of events to design specific campaigns to promote bestselling products, remove unwelcome products or pages, fine-tune system capacity based on timeslot usage statistics, or provide tailor-made promotions based on customer activities.
This is possible with an event-driven, publish-subscribe communication pattern: adding additional subscribers/consumers of data do not impact any existing applications. These functions do not affect existing event flows or application performance while providing insights to focus on specific target groups and create better marketing campaigns and have better customer engagements.
Summary
With EDA, service components for business processes are decoupled, and initiation and downstream services in the flow are associated by real-time events and handled asynchronously. It is clear that EDA can help solve REST polling’s longer latency, lower throughput, and higher cost issue to address business requirements.
EDA is not just an architecture design or data flow change, it also requires the change of people’s mindset; often referred to as culture change. That is the most difficult part.
Management support is key, and training is necessary, but it needs buy in at all levels to really deliver material benefits.
Training should be general, and not dependent on an individual project. At our company, we have trained more than 300 colleagues to get EDA related certifications last year. Certified colleagues range from Users, Testers, Developers, Architects, PMs, and Management. So now when we talk about EDA on individual projects, they already know what it is, and that saves us a lot of time explaining or arguing the concept.
About the Author
Stephen has more than 20 years of IT experience in solution and architecture design, including most challenging areas such as low-latency risk management, ultra-high speed GPU calculation and scalable voice recognition system. He has been leading an architect team to formulate IT technical solutions, establish and develop architecture framework, technology policies, principles and standards for the Enterprise Integration portfolio.
He has designed and implemented low latency and high throughput event mesh infrastructure based on PubSub+ Platform which has connected application systems from various business located in different network segments.
Stephen holds two IT Master degrees, from University of New South Wales (Australia) and University of Wollongong (Australia), on top of a Professional Diploma in Computer Forensics from the Hong Kong University of Science and Technology. He has gained five Solace certifications, and trained hundreds of Solace certified colleagues in recent years. Stephen currently works for The Hong Kong Jockey Club, the largest single taxpayer in Hong Kong and one of the world’s top ten charity donors.
The post The Advantages of Building an Event-Driven Application with an Event Portal appeared first on Solace.