How to Align Your Business and Technical Goals with EDA

In today’s dynamic business environment, the collaboration between business and IT is crucial for organizations to adapt swiftly, stay competitive, and enhance efficiency and profitability. Event-driven architecture (EDA) plays a pivotal role in achieving these objectives by addressing both business and technical requirements.

Technical Objectives

In comparing the traditional request-driven architecture (RDA) with EDA, significant simplification and enhanced effectiveness and flexibility are evident to handle complex use cases and minimize changes for new enhancements. The key benefits are outlined below.

Simplify Architecture

Goal: Transition from Multiple Layers to Two Layers

Traditional architecture typically employs a multi-layered design, which includes three major layers: the backend to provide services, the middle layer to consolidate data, and the channel to distribute information. Each of these layers may contain various sub-layers. For instance, the channel layer could encompass a backend for data processing and caching, along with a frontend for data visualization.

Contrastingly, EDA design simplifies the structure by having the backend service directly expose service endpoints to client systems, thereby streamlining the architecture.

Improve Data Consistency; Reduce Interfaces

Goal: Shift from Point-to-Point to One-to-Many

Most of the legacy systems were built based on point-to-point integration and are tightly coupled with each other. This means that we need to cater to various communication protocols and maintain different information types via thousands of interfaces. Even if we only want to change one data field in the backend, it could affect dozens of other systems. More interfaces and more services also mean higher costs, higher risks and longer implementation time.

Harmonizing interfaces across systems and dispersing the source to multiple client systems markedly diminishes the number of interfaces and enables simultaneous distribution to systems.

Enhance Flexibility and Scalability

Goal: Move from Orchestration to Choreography

In orchestration, a central controller oversees all event flows, providing a unified mechanism for control. On the other hand, Choreography involves distributed control, where components operate independently and are loosely coupled.

Choreography design offers flexibility by allowing the easy introduction or removal of components to adapt to evolving data flow requirements. This approach minimizes the need for extensive code changes, as adjustments can be made through configurations. Additionally, components can scale autonomously to accommodate varying levels of traffic.

Decouple Systems

Goal: Transition from Request/Reply to Publish/Subscribe

EDA using fan-out mechanisms to replace point-to-point integration and avoiding the use of the “reply-to” attribute to dictate the reply target location on the consumer. System decoupling is not only on the geolocation, but also on the system interface, the size of payload and event flow.

Accelerate Performance

Goal: Switch from Synchronous to Asynchronous

Ensure a smoother customer experience by preventing the user interface from freezing while waiting for an entire service call to finish.

Strengthen Security

By integrating industry-standard security controls like MQTT OAuth support and event bus ACL setting, we enhance our security protocols effectively. These measures enable us to streamline access control down to the account level, establishing a robust security framework.

For instance, with MQTT OAuth, the client’s token during connection establishment facilitates a verification process within the event mesh. This verification involves cross-referencing the account’s unique key on the JWT with the topic name. Additionally, the backend consumer validates the account’s unique key against the topic and message payload, ensuring a comprehensive security check.

Business Objectives

Effectively packaging a technical solution as a business story is crucial to capturing stakeholders’ attention and securing the necessary budget for system enhancements. This session will demonstrate how to leverage technology to achieve key business objectives.

Ensure Low Latency

EDA offers a significant reduction in end-to-end latency, benefiting both Information distribution and real-time operation.

Information Distribution

In traditional REST polling architectures, channels typically rely on caching layers to format and tailor data, leading to increased latency. Alternatively, leveraging content delivery networks (CDNs) to cache information can reduce backend traffic impact, although this introduces additional latency based on page TTL and update frequency. Clients must regularly poll the web server for updates, adding to latency.

Under EDA, the backed source provider will immediately publish the delta information as a separated event to the event bus when the data is changed, and client will directly subscribe corresponding events via the event bus. There is not any cache layer between the client and data source provider, and the client performance will be increased due to less frequence data reloading.

Transitioning to EDA as showcased in the infrastructure diagram can notably reduce latency from 14 seconds to 2 seconds or less.

Real-Time Operation

Even without CDN involvement, requests still pass through channel web and application servers before reaching the backend, incurring processing time.

With EDA, the backend service provider directly exposes service calls to clients via the event bus, enabling immediate request/response interactions. This setup is poised to reduce latency by at least 20%.

Conversely, the messaging-related protocol (e.g. MQTT) outperforms REST due to its data exchange relying on persistent connections rather than initiating connection and authentication verification with each REST call.

Reduce Operating Expenses

EDA offers a significant cost advantage over REST. Here’s a breakdown of the key differences between the two approaches:

Event Bus vs. Web Server + Application server

• EDA: A single Event Bus can efficiently handle up to 200,000 concurrent connections.
• REST: To support the same number of concurrent connections, the REST solution would necessitate a substantial infrastructure. This includes at least 400 web servers, each equipped with 12 CPU cores and 32GB of RAM. Additionally, multiple application servers on each channel would be required. It’s noteworthy that one web server with the specified configuration can support up to 500 concurrent connections.

Network Equipment

• EDA: The Event Bus’s non-exclusive queue enables an Active-Active consumer design, eliminating the need for additional load balancers.
• REST: Implementing separate load balancers in front of the Web Server, Channel Application Server, and Backend Server becomes imperative to manage the load efficiently across the extensive web server network. Additional switches are essential for optimal load distribution and system reliability.

Bandwidth

• EDA: Employing the MQTT protocol with a persistent connection ensures streamlined end-to-end data transfer. MQTT’s lightweight nature minimizes message size, while authentication details are solely necessary during connection establishment. This approach optimizes bandwidth utilization.
• REST: Contrastingly, the REST solution mandates attaching authentication information to each message. Consequently, this method necessitates greater bandwidth across the entire flow, leading to escalated infrastructure bandwidth requirements and increased subscription costs.

Operating Expenses

EDA, encompassing electronic power and data center rack space, stands out as a more economical choice in contrast to the REST solution. The REST solution, requiring hundreds of servers including web servers, application servers, load balancers, switches, and more, significantly escalate operational expenses.

Internal Effort

Implementing EDA can streamline processes and reduce internal effort in various areas of product development and software development life cycles (PDLC/SDLC).

Simplified Architecture

Transitioning from multi-tier designs to two tiers in architecture design saves time and effort in design, implementation, and infrastructure setup.

System Decoupling

EDA design minimizes the need for extensive discussions on change impacts across teams, enabling faster decision-making and implementation.

Shared Services

By reducing interfaces, shared services within EDA solutions enhance efficiency and simplify interactions. Consistent design pattern can decrease the complexity of implementation and troubleshooting

Sample Cost Comparison

Let’s compare the cost of supporting 200,000 concurrent connections in real-time operations using two distinct approaches, REST and EDA, with the following assumptions:

• 3 channels (2 mobile apps and one browser)
• 200,000 concurrent connections
• 10 operations (e.g. account opening, account inquiry, etc.)
• REST message size: 100Kb, EDA MQTT message size: 10Kb (10% of REST message)

This comparison highlights the cost-effectiveness and operational efficiencies of EDA over the REST alternative, emphasizing the importance of system design in minimizing resource requirements and internal efforts.

Enable A/B Testing

EDA offers a strategic approach to optimizing the testing environment within large organizations. Traditionally, multiple non-production environments serve distinct testing purposes, such as functional tests in the development environment, integration tests in the SIT environment, end-to-end functionality tests in the UAT environment, and load tests in the SAT environment. Given the concurrent nature of numerous projects, the need for multiple instances of these environments arises.

In an EDA setup, a production-like event mesh infrastructure can be established to streamline all non-production environments. This setup provides the flexibility to distribute resources across different environments, enabling developers to conduct high availability (HA), disaster recovery (DR), and load tests on any environment, rather than being confined to SAT alone. By facilitating event routing among different environments, developers can assess application behaviors across various application versions.

Moreover, this infrastructure supports ReadyAPI for automated unit testing, utilizes JMeter to simulate client traffic, and integrates with CI/CD for seamless auto deployments. Leveraging this integrated toolset, organizations can efficiently manage all non-production environment requirements using a single suite of tools.

This consolidated approach not only reduces costs related to hardware and software licenses but also expedites implementation and testing processes. Ultimately, the business stands to benefit from enhanced agility, reduced expenses, and minimized risks, aligning closely with market demands and fostering operational efficiency.

Encourage Shared Services

Many organizations have developed products utilizing different technologies to enhance customer experience. A Web application focuses on providing comprehensive information, while a mobile client prioritizes user-friendly interactions. Similarly, an ATM may offer efficient shortcuts despite limited information. These products, with distinct UI requirements, share over 90% of data and offer common services like ePayment, ticket sales, and account operations.

In the context of the previous section “Ensure Low Latency for Real-Time Responses”, the REST solution requires multiple cache layers for each channel to address performance issues. This setup leads to numerous interfaces with simulated data fields, each catering to a specific channel’s page usage. Managing inconsistencies, especially when a single screen needs to call multiple services, adds to the code complexity. The development team continually creates new services based on UI demands.

EDA introduces a self-service platform removing dependencies between channels and the backend system. The backend system exposes services directly to the event catalog, breaking them down to the event level. Channel sites can create small functions to subscribe to these events and display them on the UI, ensuring data consistency across various channels.

For instance, in ticket selling, when a customer places a bet on any channel, that channel triggers a Place Bet event on the self-service platform. The backend service subscribes to these events using a persistent queue for processing. Successful bet placement triggers an odds change, disseminating the new odds seamlessly to various channels through the self-service platform.

This approach allows channel developers to focus on enhancing UI presentations.

Accelerating Time to Market

End users often strive for significant transformations within tight budget constraints, aiming for swift launches with minimal risks to meet market demands and enhance customer experiences. Traditional solutions relying on a mix of legacy technologies for point-to-point integration and interconnecting systems through multiple interfaces struggle to effectively achieve these objectives.

The primary issue lies in the complexity of:

• Managing various communication protocols and diverse information types across thousands of interfaces. Even a minor backend data field alteration can impact numerous systems, leading to increased costs, risks, and integration time.
• Addressing network and security compliance, alongside maintaining a sophisticated monitoring solution. Additional logic is required in applications for alerts and data analysis, affecting overall performance.
• Spending considerable time coordinating change impacts with multiple teams, followed by extensive end-to-end testing. Implementing even minor changes can take up to a year due to the thorough testing required.

EDA seamlessly integrates with innovative technologies like Agile, DDA, and DevOps to tackle these challenges. By combining EDA with these methodologies, product delivery timelines can be significantly reduced from 8-12 months to just 1-2 weeks, depending on the technology and methodology blend used.

EDA models are crucial in decoupling systems, while Solace Event Portal serves as a robust platform for creating, managing, and regulating various applications, events, and schemas. Embracing Agile practices allows development cycles to be broken into smaller sprints, enabling quick incorporation of necessary modifications. This structured approach follows a 5-step process for implementing changes: discover, design, develop, test, and deploy, with iterations possible on a weekly or daily basis.