The Flexibility of Topic Design and Naming Conventions with PubSub+

Event-driven architecture (EDA) has become a popular and valuable application design pattern. In this context, an “event” is some change of state. triggered by user actions, by systems, or by other event from upstream in the data flow. Each event appears individually, and is represented by a specific topic.

Events do not happen alone. For example, a business flow like a customer requesting the transfer of $100 from a bank to his/her account via FPS needs to be completed by a series of actions called event processes: validation of the login session, updating the database, and triggering a notification via a service of SMS or e-mail. The payload of these corresponding events must include the contents of original event, i.e. when the event is initiated from where it originates. In an EDA flow, you never know where the event will be routed, or what information will be needed by subsequent steps, so every step should be treated the same when it is initiated. As such you need to build up the relationship between events in entire business flow.

Solace PubSub+ Platform is an event streaming and management solution that enables event-driven architecture. At The Hong Kong Jockey Club we’ve been using PubSub+ for several years, and I have come to appreciate the robust features it offers in the realm of topic-based routing

I will use an example of an electronic payment solution to illustrate how these events could be linked to constitute a business flow via topic structure under the EDA design (asynchronous, isolation and decoupling).

Background

Since first implementing electronic payments via bank fund transfer 20 years ago, we have added support for more than 10 payment methods: web site, mobile phone, automatic teller machines in branches, etc. Today, we are refreshing our technologies and moving in the direction of microservices, EDA and cloud. This example will apply the following four payment methods to explain the topic structure design.

• Electronic Funds Transfer (EFT): Customer can make electronic fund transfer based on registered bank account number.
• Prospective Payment System (PPS): The payment will point to the bank’s website. After processing, the result will be sent back to our system (similar to PayPal)
• Fast Payment Service (FPS): Customer can transfer money for payments based on customer’s registered phone number or FPS ID. (note: this solution won the Best Payments and Collections Solution, Hong Kong within the Digital Solutions section of The Asset Triple A Treasury, Trade, Sustainable Supply Chain and Risk Management Awards 2020.)
• Bank Fund Transfer Service: The request of fund transfer is initiated from the bank.

Flow

The fund transfer requests are triggered against various channels (Web, mobile, ATM, etc.), and send to corresponding payment services (EFT/PPS/FPS/Bank service) via RESTful API or remote procedure calls (RPC). The payment service publishes the initial event to its associated message brokers which are connected via event mesh, and then waits for the reply asynchronously.

As you can see in the following diagram, the methods’ triggering points and subsequent processes are decoupled and handled via individual microservices. Eventually, these services need to update the corresponding account records in database via backend account service.

Topic Name Structure

PubSub+ gives you a lot of flexibility in how you design your topic structure, so you can create fine-grained topics that ensure data only goes where it’s needed, and use wildcards so subscribing applications can easily indicate interest in and receive all of the information they need.

It is important for enterprises to establish naming standards and well-defined topic naming structure to align the organization’s information data models or business nature.  Therefore, it is essential to define the standards for topic naming structure in event-driven architecture design for messaging.

The event topic names are in multiple levels hierarchical structure separated by the slash (/) delimiters. I’ll use the following topic naming standard in my example:

[Information domain] / [domain component] / [message type] / [schema version]/….. [APPLICATION-SPECIFIC]

I suggest you take the time to define standards for your organization, and this best practices guide is a great starting point. Don’t forget to follow the Solace Topic Architecture Best Practices.

Event Flow & Topic Structure

As mentioned above, all events within same event flow is the lifetime of “event” which each point-in-time of appearance is treated the same. During the lifetime of event, they might pass through various systems while they are processing, where the same event in data domain could mean different things to different systems/applications contexts. However, the topic that represents the event message must remain constant. Therefore, from topic naming perspective, these events’ data component values should base on the sole definition of the original purpose where the event originated from.

In our fund transfer example, the topic structure consists of two parts, namely mandatory and optional:

• Mandatory: arrangement/txn/evt/v01/[fund transfer method]/
• Optional: [component function]/[status]

The mandatory part will be put under the message header, and bring forward to all subsequent messages in the flow. This part should not be changed across the lifetime of the event flow.

In the following EFT event flow, the mandatory part is arrangement/txn/evt/v01/eft/. All component should put their functions and processes’ status after the mandatory part in the message topic. For example, the login session check service would subscribe the event from arrangement/txn/evt/v01/eft/init, and publish the result message to the arrangement/txn/evt/v01/eft/chk_login_session/success or arrangement/txn/evt/v01/eft/chk_login_session/fail.

This topic structure gives you the flexibility to add/remove any extra steps (account check service or SMS service) to or from the entire data flow when the business flow is changed. All you need to do is change the subscriber’s configuration, i.e. the topic subscription. Moreover, you can also consume all corresponding events within the same event flow by simply subscribing to arrangement/txn/evt/v01/[fund transfer method]/> without any changes to existing flow.

Original:

Add extra components:

To change the subscriptions of individual components, you don’t need to code anything in their logics for the topic routing. For example, in the backend account service, it handles different payment event flows to update the customer profile, and then returns the status to corresponding topic. It also needs to cater the requests for updating the customer profiles (e.g. phone number) or system configurations (e.g. account category amount range). As such, the incoming messages’ header has to cater the first part of topic name, while the consumer (backend account service) just needs to process the message and post the result back to the corresponding topic.

The flexibility of PubSub+’s topic naming support means that a new application can start receiving events associated with a given business process by simply subscribing to an existing topic.

We deployed the Solace messaging framework via PubSub+ brokers event mesh architecture to support the distribution of thousands of events and hundred of millions of messages a day. In the future, we will build the event category on PubSub+ Event Portal to support the EDA design.