Topic Subscription on Queues

In this blog post, I will be discussing a Solace feature called “Topic Subscription on Queues” and explore how this feature can be used to meet some complex messaging requirements elegantly. I will begin by covering few basic concepts, discuss the feature in detail and conclude with a list of benefits this feature offers.

The Messaging Model divide

JMS defines two distinct messaging models: Point-to-point and publish/subscribe. Point-to-point (P2P or PTP) is built on the concept of message queues with publisher addressing messages to specific queue and the receiving clients extracting messages from the queue. The queue acts as a holding area and messages are retained in the queue until they are consumed or expire. Queues remove the timing dependency between senders and receivers. As the name point-to-point implies, each message can be consumed by only one consumer, though multiple consumers can bind to a queue.

With publish/subscribe (PubSub), messages are addressed to topics and multiple consumers can subscribe to and receive the messages. The messaging system takes care of distributing the messages to subscribed consumers. Except for durable subscribers, messages are not retained for inactive or slow subscribers (more on durable subscribers later).

As a messaging application developer, you have to pick your side: PubSub for best effort fan-out subscriber use case or P2P for consumers requiring guaranteed delivery. But what if you wanted the best of both these messaging paradigms in your application? What if some of your subscribers require delivery guarantee while others don’t and you want publishers to be unaware of this dependency?

Now a note of caution before we proceed: To keep the discussion simple and the blog post focused, I am leaving out some related, but important JMS and Solace features deliberately. For example, in addition to P2P and PubSub, there are other messaging patterns such as request/reply; there are topic endpoints in addition to queues; there are subtle but important differences in the way JMS/Solace delivery types (persistent/non-persistent/direct) are mapped to Solace transports and so on. We will set them aside for now, but you are encouraged to explore them in Solace Documentation page.

A Simple Application

Before we answer that question of mixing messaging models, let’s see how these messaging models differ with a very simple hypothetical application.

Simple P2P Application

Let’s consider a P2P application with three publishers, queues and consumers. Let’s assume that not all consumers are live: C2 is slow to consume and can’t keep up with the publish rate; C3 is offline can’t receive messages now.

With P2P, messages published to a queue are persisted by Solace on the corresponding queue endpoints. Irrespective of the consumer status (Live, slow and offline), there is no message loss to any of the consumers. Messages will be delivered to each consumer as and when they are ready to consume. As long as the queue quota is not reached, messages keep accumulating on the endpoint until they are consumed or expire. To keep the discussion simple, queue properties that influence this behavior such as queue access-types (exclusive or non-exclusive), expiry, max redelivery are not discussed here.

Simple PubSub Application

Let’s now consider a simple PubSub application with two publishers (P1 and P2) that publish on two topics (Topic1 and Topic2). To illustrate the fan-out nature of PubSub, there are more than one subscribers per topic. Subscriber S3 also subscribes to both topics. To keep the discussion simple, none of the subscribers are assumed to be durable.

As messages are published to each topic, they are fanned-out to all subscribers. However, messages are delivered only to subscribers that are live. Since S2 is slow to consume, it’s likely to lose some messages; S3 and S5 will lose all messages during the time they are offline.

Mixing PubSub and P2P

Let’s assume in “Simple PubSub Application”, we want to make S2, S3 and S4 guaranteed consumers. What could be done to make them immune to message loss?

One option is to define queues for each of these subscribers requiring guaranteed delivery (S2, S3 and S4) and have the publisher double publish to a topic and respective queue as below:

Though this may be a workable solution, its less desirable for the following reasons:

• This adds more administrative overhead since a new queue needs to be created for each consumer/topic pair. For eg, in the case of S3, we will need to define two queues one each for Topic1 and Topic2.
• Publisher now has the additional workload of publishing to Topic as well as potentially multiple queues. As we add more consumers with guaranteed delivery requirement, this complexity increases.
• This also makes the publisher and subscriber tightly coupled that goes against one of the core messaging principles (publishers and subscribers should be loosely coupled)

Topic Subscription on Queues

Solace offers an interesting feature called “Topic subscription on Queues” which allows a set of topics to be added as “subscriptions” on queues. This feature expands the utility of queues to beyond being a mere P2P destination. Now queues can receive and persist messages published to all matching Topics it is subscribing to, in addition to receiving and persisting messages published to the queue destination. This feature allows any number of topics to be added as subscriptions on queues and supports topic wild carding as well.

With this feature in hand, each subscriber requiring guaranteed delivery (S2, S3 and S4) will define its own queues (Queue2, Queue3 and Queue4) and add corresponding topics as subscriptions on queues. Messages for these subscribers (S2, S3 and S4) are now safe stored even though they are published on topic destination.

And what’s interesting is that the publisher need not be aware of this consumer end differences and there is no change to the publisher end. This feature automatically “promotes” messages to guaranteed delivery for these (and only these) subscribers requiring delivery guaranty. Regular topic subscribers (S1 and S5) will continue to subscribe to the topic with no delivery guarantee. A related concept is message “message demotion” where publisher publishes messages with persistence, but the messages are demoted as “direct” if there are no subscribers requiring guaranteed delivery. Please see Understanding Solace Delivery Modes: Promotion and Demotion for more info on this topic.

Adding topic subscriptions on queues can be done either administratively (SolAdmin / CLI / SEMP) or programmatically from the application. This is covered in Topic to Queue Mapping Getting Started Guide.

What about Durable Subscribers?

Since JMS durable subscribers do offer queue like message reliability to topic subscribers, using that would be another option. If the consumer is interested in only a single topic (e.g S2 and S4 in the example above), they can create a durable subscriptions to topics. This is offered in Solace as well with durable topic endpoint (DTE). My colleague discusses the differences between durable and non-durable endpoints is covered in her blog post Understanding Solace Endpoints: Durable vs. Non-Durable. But it’s important to recognize that DTEs differ from queues in features and behavior. For example:

• DTEs support a single consumer while multiple consumers can bind to queues
• Durable endpoints are limited to a single topic subscription. Queues allow multiple topic subscriptions as well as topic wildcards.
• Any subscription change (eg: message filters) with durable endpoints deletes the messages.
• Message filters are applied inbound with DTE while they are applied outbound with queues. This means any messages not matching the filters will be discarded with DTEs while they are retained on the queue.

Benefits

This elegant solution makes the overall application design less complex and cleaner. The burden of managing the subscriptions and persistence requirements shifts to the consumer and the middleware domain. Publishers remain unaware of how the topics are subscribed to and consumed. For e.g., if Subscriber S3 decides not to receive Topic2, that subscription can be removed from Queue3 with no impact to publishers. Similarly, if a new consumer is to be added in future or additional topic needs to be added to existing queues, this can be done with no impact to the publisher.

Similarly, when topic subscriptions change on the queue, there is no impact to the consumers either. All messages published on subscribed topics will be delivered to the consumer in the order in which they are received. This is true even when there are multiple or wildcard topic subscriptions on the queue

Further, if the consumer requires to know the original topic where the messages were published to, this is possible too. The original destination (topic) where messages are published is available in the message header and the consumer can make use of that info.

This feature, combined with the topic hierarchy and wildcarding supported in Solace gives rise to very clean application design for some of the complex message exchange patterns. As long as the topic structure is rich enough, the complete de-coupling of publishers and consumers on destination and delivery type can “future-proof” the publisher from downstream changes.

If you’re interested in reading more about demystifying the best practices in creating a sound event topic hierarchy, I encourage you to check out this post: Topic Hierarchy and Topic Architecture Best Practices

Until next time!

Thanks for reading and I do hope this blog post provided sufficient info to help you understand and appreciate the “Topic Subscription on Queue” feature. I will discuss few use-cases with samples in a follow-up blog. In the meanwhile, you are welcome to explore Solace documentation page for additional info. If you have any comments or questions, please feel free to use “Leave a Reply” box below, or use our Developer Community.