Topic Namespaces and Governance

In my last article I introduced the power and use of topics within an enterprise messaging platform and provided some information about how Solace’s appliance utilizes topic names to deliver information to interested parties.  In the second part of the series I want to provide suggestions about how organizations can leverage these features in a consistent way to maximize the value from their enterprise messaging capabilities.  Additionally we’ll focus on considerations and suggestions when designing a governance model to make sure the best practices are leveraged whenever possible.  These elements help drive adoption, reduce total cost of ownership and accelerate the learning curve as new developers interact with this technology.

Who Controls the Topic Namespace?

The importance of this question is directly proportional to the size of the organization and project it’s being asked about.  If you’re the sole person on a project performing development, SysAdmin, and Help Desk Guru tasks, then the answer should be obvious.  Things are more complicated when you belong to a larger organization where different people or teams are responsible for supporting, monitoring, deploying, and designing the application, and a lot more people have a vested interest in the composition of your topic hierarchy.

Having a ‘big picture’ view of current and future application direction will help you make choices that lead to the right number of levels of your topic namespace to satisfy everyone from the developer to the NOC who has to watch for misbehaving applications.  Making sure you have a grasp of what everyone expects from your application is a great way to make sure that the topic namespace has been designed in a way that meets everyone’s needs and makes their lives as easy as possible.

When should you start thinking about your Topic Namespace?

The sooner the better!  In reality, there are two kinds of scenarios: new applications, and existing applications. If the application you’re responsible for is already using publish/subscribe middleware it’s already got a set of defined topics.  Unless there is large benefit to be gained, such as faster time to market for new products using this infrastructure, decreased troubleshooting time (and it fails a lot), or the ability to provide more timely information to the powers that be, then re-engineering an existing topic namespace is rarely a project that gets the green light.

But if you’re developing a new application, or porting an application to a pub/sub architecture, you have a little more freedom.  Even if the program you’re developing has to consume data from another topic namespace, you can always segment your own information by starting your own ‘high level identifier’ such as /MyNewProject/my/topic/levels

What questions should I be thinking about as I design my Topic Namespace?

This list could probably fill a textbook, and the book would be different based on the application, the industry, the culture of your organization, and many other subjects.  Some basic ones that almost always come up:

• Who will be using this data?
  Yes – one of the main benefits of a PubSub architecture is that publishers don’t need to be aware of who consumes their data.  However, for most applications, the reason you’re publishing data is that you’re aware someone will need it.  In the brief manufacturing plant example I gave in my last post, a use case that was at the forefront of the design process was providing real-time dashboard updates for several levels of Operations.  This meant that each subsection (intake, processing, quality control, packaging, etc) needed its own level in the topic namespace to make it easier for the team responsible for the dashboard to get just the data they needed. Also, having an idea of what kind of information your consumers are interested in can help you make sure you’re properly addressing your needs.  Don’t forget to get input from your Ops team, or whoever is responsible for monitoring your application!  When you keep them happy, they keep your application happy, and that keeps you happy!
• What level of Granularity does my target consumer care about?
  This comes up in a lot of the Financial Industry firms I deal with.  If the consumer is an application that ‘eyeball’ traders will use, it’s perfectly acceptable to publish to more general information such as each message published to equities/us/nasdaq/aapl having the Open, High, Low, Bid, Ask, Volume, etc.  Because human eyeballs aren’t going to notice the micro-to-milli-seconds that it takes for their trading application to unpack the message and display the information on their screen.  If your intended consumer is a High Frequency Trading Algorithm, then nanoseconds can mean the difference between a trade making money and losing money.  Therefore you want a much more granular message structure such as:

• equities/us/nasdaq/aapl/bid
• equities/us/nasdaq/aapl/ask
• equities/us/nasdaq/aapl/volume
• equities/us/nasdaq/aapl/high
• equities/us/nasdaq/aapl/low
• Does the size of the topic matter?
  It depends!  If your application is streaming information over the internet, sending a single byte of information down can incur up to 1000x overhead from the header information.  If you’re on a LAN, and using PubSub for your ESB, and your usual message is a 20kB XML document, then even a couple hundred bytes of topic isn’t going to have an effect on your overall throughput. If you’re in a situation where latency and size on the wire really matter, this doesn’t mean you have to sacrifice richness of information in order to shrink your header! Many of my customers in this situation have a function in their application that takes your full expanded topic:  equitiies/us/nasdaq/aapl/volume and turns it into e/u/n/a11.  Remember, human readable topics are helpful when designing and developing programs, but the Messaging Subsystem is a machine, and doesn’t care what is in your topic space! Finally, the messaging middleware you use matters.  Solace uses purpose built hardware to match millions of topics against millions of subscriptions every second!  Most software messaging products don’t have the luxury of running their matches in such a dedicated, high performance environment, and you may need to consider additional approaches to maximize your performance in these situations.

There’s a lot more to consider, including, who else uses messaging in your organization?  Do they have a Topic Namespace approach you need to be aware of?  Is there a process in place to exchange information to make sure you’re not overlapping your topics? (perhaps to disastrous results!)  But these, and other questions will have to wait for another post.

Let me know about the additional questions you have about topic namespaces and governance and we’ll address them in the comments or if they’re more complicated and intricate we can write additional blog articles.