Introduction to Smart Topics

What is an Event?

In event-driven architecture (EDA), any change in an object within your system is an event. Almost any real-world occurrence can generate such an event: a user interaction, some step in an automated process, a sensor reading, the physical movement of an asset or vehicle, etc.

These events are published onto an event mesh by an application to signal that a state change has occurred. These messages about events can then be consumed by other interested applications connected to the mesh. However, to properly route this information between the producers and consumers, some metadata needs to be attached to describe the contents.

What is a Topic?

In content of EDA, messaging, streaming, etc., a topic is a string attached to a message that describes its content so it can be made available to and used by systems that need it. Topics make events available and usable. Some brokers only support “flat” topics of limited length, and are deployed as static files tying them tightly to the broker.

What is a Smart Topic?

Solace, on the other hand, utilizes smart topics in which this string is not just a constant, but is made up of multiple levels. Each level can be used to dynamically encode information by the publisher, and levels can be filtered on using wildcards when subscribing. This allows for very efficient data movement through the event mesh as subscribers can indicate exactly which information they want to receive using the smart topic.

What is Smart Topic Architecture?

To ensure consistency between the publishers and subscribers within an event-driven system, it is a best practice to establish and follow a useful smart topic architecture. A smart topic architecture describes exactly how topics should be written and which information the various levels may contain, both of which are important as your system consists of more events and higher volumes. The collection of levels in a smart topic is known as a topic hierarchy. Defining this topic architecture is often referred to as cataloging, and can be done in Solace PubSub+ Event Portal.

For instance, a smart topic for a given application domain may follow a sentence-like structure making it easy to understand and filter, like this: Domain/Noun/Verb/Version/Properties

This will be further unpacked as we discuss semantics for production-ready topics.

Smart Topic Architecture Semantics

To effectively describe an event, a smart topic needs three main parts. First, an event is always an action (or state change) that occurred on an object. For instance, a flight may have started boarding or a trade might have been placed. All of these examples are some kind of update on some kind of object, thus the most basic of smart topic architectures is noun/verb.

Consider an airline that wants to modernize and event-enable their operations. There are many verbs that may apply to a flight such as boarding, delayed, or wheelsUp. Their most basic topic may look like flight/boarding. There may also be other verbs that apply to the flight, which would constitute separate events, for instance flight/wheelsUp or flight/delayed.

Subscribers may want to receive some of the notifications for flights, for instance a customer app might want to receive flight delayed notifications. However, they only want to receive notifications for the flights they are taking. To ensure that only the delayed events a consumer is interested in are received, smart topics can be used.

Smart topics extend the basic topic architecture described above with properties in the form Noun/Verb/Properties. For instance, the flight number can be added to the flight delayed event: flight/delayed/ex123. This allows the subscribing application that is only interested in an individual flight to subscribe to the smart topic flight/delayed/ex123. Additional property levels can be added as well. For instance, the origin and destination could be added to form a topic of flight/boarding/ex123/yow/yyz which would allow subscribers to subscribe to all flight notifications for a specific airport.

Putting it all together, we get the following smart topics:

• flight/boarding/<flight #>/<origin>/<destination>
• flight/delayed/<flight #>/<origin>/<destination>
• flight/wheelsUp/<flight #>/<origin>/<destination>

Then, subscribers can subscribe to a number of different topics depending on what they are interested in receiving:

• flight/*/ex123/> to receive all events for a given flight number, for customer applications
• flight/delayed/> to receive all wheels up events across the enterprise, for analytics
• flight/boarding/*/yow/> to receive all boarding events for a given airport, for in-airport announcements

Making Smart Topics Production Ready

Producer published topics are used for more than just consumer subscriptions. They are part of the DevOps lifecycle management for their asynchronous applications and globally dispersed development teams. Therefore, a Smart Topic delivers on two key needs: versioning and domains.

Versioning of Smart Topics

It is important to support blue/green deployments, where for some period of time both and old and new version of an application may be present, thus preventing downtime when upgrading. To properly handle this situation and prevent data loss, smart topics should be versioned to ensure that only the data the application is capable of processing is processed. Versioning in the smart topic instead of the payload also allows the various properties to be versioned as well so that properties can be added/removed when versioning the event.

Taking the smaret topic example above, we can add a version in. The version should be present before the properties so that the properties can change when versioning the event: flight/boarding/v1/<flight #>/<origin>/<destination>. Then, when a v2 comes out, this time adding in the aircraft type as a new property, and potentially versioning the payload, the smart topic can be flight/boarding/v2/<flight #>/<origin>/<destination>/<aircraft>.

Domains

In order to ensure the business can scale over time, both in terms of technical ability (e.g. more apps coming online in the event mesh), or in terms of expansions (e.g. mergers or acquisitions), the domain in which the data belongs should be included in the smart topic. This may consist of multiple parts.

• The application domain, which indicates which application the event is for. In the flight example above, this is as simple as flights.
• The business unit which owns the event. This should uniquely define the sector of the business responsible for producing the events, so that ownership can be established when connecting multiple business units together. In this case, operations, or ops for short, owns this event. A business unit may have multiple application domains, for instance ops/flights and ops/baggage.
• The organization name which will often be included to differentiate between brands within a conglomerate, or future-proof for acquisitions or mergers. In this case, the airline’s code might be added as a prefix on the topic. Let’s call that “ea” for sake of example.

So an example of a smart topic domain would be ea/ops/flights, and a full topic might be ea/ops/flights/flight/boarding/v1/<flight #>/<origin>/<destination>

Putting It All Together

When you combine these domain elements with the semantics described above, you get the following architecture for the smart topic: Domain/Noun/Verb/Version/Properties which can be used as a base for all topic architectures.

For more information about building smart topic hierarchy, read Topic Architecture Best Practices in the Solace documentation.

When to Include a Property in a Smart Topic

There may be a number of properties on a given event. For instance, in the flight boarding example above, the event might have many attributes, many of which would likely be captured in the payload of the event: gateClosedTime, takeoffTime, vehicleType, gateNumber, originAirport, destinationAirport, flightTime, etc. It may not be obvious which properties should be included in the smart topic. There are three main reasons to include a property in the topic hierarchy of a smart topic:

• Filtering: is the property something that could be used by subscribers to describe which information they are interested in?
• Access Control: is the property something that could be used by the event mesh to decide whether or not a producer/consumer has the right to publish/subscribe to the event?
• Governance: is the property something that could be helpful when administering the event mesh?

When to Not Include a Property in a Smart Topic

If the property is not useful for filtering, access control, or governance, then it should not be included in the smart topic hierarchy. For example, gateClosedTime is not easily matched with a wildcard, thus it is not useful for filtering, it is not useful for access control as there are likely no restrictions on publishing for certain gate close times, and it is not useful for governance as it would not provide any tangible benefits for administration of the event mesh. Therefore, it should not be included in the smart topic hierarchy.

Benefits of a Smart Topic Hierarchy

A smart topic hierarchy can accelerate development in the following ways:

• Routing, filtering, and governance can occur at multiple levels, as data flows across the event mesh. For example, a decision can be made on which events are allowed to cross geographical boundaries, but once a smart topic is in the geography, secondary decisions can be made on which consumers can receive the events.
• Gathering information across events is easier. For example, wildcards enable receiving all event types pertaining to an orderID, or all new orders regardless of originator or location.
• Introducing new applications and onboarding new employees is easier when you document events with an event portal, and assigning smart topics in accordance with well-defined topic hierarchy.
• Adding properties to the topic hierarchy allows for fine-grained filtering. This filtering ensures that consumers only receive the events that they need. It also ensures the system is more efficient by reducing the workload on applications and keeping egress data costs down.

Conclusion

Topic architecture is a key part of event-driven architecture, and combined with Solace smart topics can enable an efficient, well managed and easy to understand system.

Now that you understand Solace smart topics, learn how to build them with PubSub+ Manager.