When Topics Aren’t Just Publish/Subscribe: Explaining Topics & Subscriptions in Solace

I was at a retail customer recently helping them migrate from their existing infrastructure. Their message brokers have run out of capacity, so they are evaluating how Solace could simplify their messaging environment by providing all the scale they need in a single easy to manage appliance.

Things were going well when I was asked a question I often hear: “OK, so how do I create a Topic?”

This customer understood a “topic” to be an object on the message broker, created administratively and able to persist messages: in JMS terminology, he understood a “topic” to mean something like a Durable Subscription. (in Solace we map this to a Durable Topic Endpoint which is the broker artifact that persists messages).  Note that a JMS topic is not the same as a durable subscription –  unfortunately the same terms in messaging can have widely differing meanings between similar products.

This lead to the question “what is a Topic in Solace?”  The answer is surprisingly simple: “In Solace, a Topic is a property of a message.” That’s it.  It isn’t an artifact on the broker.  It isn’t something that survives the publisher, even with guaranteed messaging. It’s just a text string published in the message header, and it must be fully qualified and complete – no wildcards.  It’s therefore completely dynamic – the publisher can use whatever topic name it needs.

To receive messages, use a subscription, which is an artifact on the Solace message router.  It can be injected by a client, and attracts messages.  A property of the subscription is the Topic(s) that we subscribe to, which can contain wildcards (and hence match multiple Topics).  The full syntax for Topics can be found here, but to summarize, they are hierarchical (delimited by “/”) and support wildcards with “*” matching everything at that hierarchical level only and “>” matching all levels from this down.

Let’s take our retail example, where we have a publisher sending customer log-on events.  We could publish to a Topic like:

customer/<location>/event/logon

A listener might be interested in all customer events in the UK, in which case it would use a wildcard in a subscription that looks like:

customer/uk/event/>

This would match all topics in all hierarchical level from customer/uk/event/… down. (You can learn more about topics from these blog posts: Controlling Information Flow With Topics and Topic Namespaces and Governance)

If a publisher wanted to start using a new topic in this topic space, it can just publish messages to that topic – for instance, it could start publishing messages to customer/uk/event/logoff and our subscription would start getting those messages too – with no interaction needed with the message router.  With Solace there is no need to restrict your Topic design, other than having a well thought our hierarchy and a consistent approach so that subscriptions match what everyone expects them to match.

As Topics in Solace are dynamic, they can contain dynamic data.  At the logical if unlikely extreme, messages might not even have to contain a payload – the topic itself could be all the data that is required!

So far, so pub/sub.  In the world of persistence and messaging queuing, though, we also use subscriptions.  We can add subscriptions to queues, and then publish a persistent message to a Topic that the subscription will match.  This gives us all the power of Solace queues:

• A choice of redundancy vs. load balanced consumers (exclusive vs non-exclusive queues);
• “Endpoint Permissions” – whether a consumer can browse, consume or delete messages (See Solace documentation);
• Optional Time-To-Live semantics and optional Finite re-delivery along with optional removal to a Dead Message Queue;
• Last Value Queues for producer re-start synchronization (so a re-starting producer can find out how far it got in the message flow before it re-started)
• Low Priority Message Rejection;
• Disaster Recovery Replication;
• Egress Selectors;

Multiple queues can subscribe to the same Topic, and in this way our persistent message is sent to multiple destinations – persistent fan-out.  Our 1-1 point to point queuing has become 1-to-n.  If you were to use JMS persistent publish/subscribe, you wouldn’t have all the features I’ve listed for queues.  Some customers using other messaging solutions start to use JMS non-persistent messages (which can be “queued” at a Topic Endpoint but might not survive a broker re-start) in an attempt to improve performance: with Solace there is no need to make such a trade off.  This is why we recommend the use of queues.
In our retail case, we’re using precisely this: log-on requests go to one endpoint to feed a Big Data cluster to help understand customer activity; they are also sent to an endpoint feeding an audit process.  The example goes further – the audit endpoint subscribes to all customer events, giving us n-to-1 persistence.

This philosophy of using subscriptions goes even further, though.  Have you ever noticed some subscriptions in Solace that look like #P2P/…?  These are network Topics – default subscriptions created automatically.  For instance, every queue in Solace has a network Topic, of the form:

#P2P/QUE/<queue name>

You can try it: create a queue and send a message to its network Topic.

Actually, this is a simplified form of the network Topic, specific to queues.  Network Topics can contain a logical router ID, allowing the routing of messages around a network of message routers.  We use this with our Multi-Node Routing feature to allow request-reply messages to be sent to one router, forwarded to another router, replied to and sent back to the original location.  An example might be

#P2P/v:router1/server/4176/#00000001/#

The power of this is that we can use our topic infrastructure, outside of just subscriptions, with queues.  For instance, supposing a bank has a payments queue.  Only authorised users should be able to add messages to that queue, otherwise a malign user could create fraudulent payments.  To prevent this, we use the network Topic in an ACL to disallow publishing to that Topic, for instance:

payment-acl: publish
default: disallow
exception: #P2P/QUE/PAYMENT_Q
others-acl: publish
default: allow
exception: #P2P/QUE/PAYMENT_Q

Other places you’ll see subscriptions:

1. Message-VPN bridges use subscriptions to draw only the messages of interest across the bridge;
2. Replicated Topics for our Replication feature, which allows messages to be sent to a remote site for Disaster Recovery purposes. A replicated Topic is, in fact, a specialised subscription;
3. The Request-Reply message pattern. If you don’t use a reply-to destination, the API inserts a subscription to the client’s Network Topic and sets the message reply-to field to be this;
4. Topic endpoints. These help support JMS semantics, and are a specialised queue with a single subscription.

Why do we use Topics and Subscriptions so extensively?  Having a single mechanism to attract messages to destinations presents a simplified architecture to users and provides a single, unified approach.

So there you have it: a Topic is a simple string property of a message.  Subscriptions match Topics, allowing messages to be drawn to destinations.  And anything to which a message can be sent has a subscription, so you have the full power and performance of our topic routing hardware at your disposal, no matter what message pattern you need.