Messaging 101

What is messaging? Instant messaging like MSN/IRC/Yahoo? SMS stuff?  Email? Morse code? Even techies have different understandings of messaging depending on context, so I wasn’t too surprised when my banker father fired me those questions at me when I told him that Solace makes messaging appliances.

I believe most readers of this blog already know something about the type of messaging that Solace delivers, but for the sake of our guests who are new to it, here is another perspective to understand what we mean by messaging.Computer Systems from a layman’s perspective have 3 logical components:  Processing, Data Storage and Transport.

• Processing: The performance of mathematical and logical computations, e.g. calculating the area of a circle or executing an algo to compare patterns and decide whether to buy or sell a stock. Without getting into the deep systems architecture detail, this is typically handled by the CPU in a computer or server.
• Storage: There are two kinds of storage: persistent and transient. Persistent storage preserves data even if the computer/server shuts down by writing it to disk, tape, or more recently RAM grids. Examples include databases storing bank account information, historical trade information for audit, pdf files for loan collateral proof, or pictures on disk. Transient storage just temporarily stores data in RAM while it’s being actively used by some computing function, such as parameters (market data/control data) in the algos upon which the computations are done, values in memory to do analytics upon etc.
• Transport: Moving data around from storage to processing and vice versa. Early computers were monolithic, and transport was moving values up and down the stack or heap memory. Now with distributed computing the norm, it is not uncommon to have multiple servers now playing different roles in an automated task. Internet Protocol, TCP, UDP etc., provide low-level connectivity, but applications require a smarter and more flexible transport layer which handles things like guaranteed delivery, publish-subscribe semantics, sequencing, monitoring etc. That layer is broadly classified as messaging.

Messaging to Facilitate Transport

Take, for example, an algorithmic trading process. Market data moves from a gateway server to an algo server which processes each update and uses the information to determine whether or not to place buy or sell orders. Once such decisions are made, orders are created and sent via order entry gateways to exchanges. There are several passive systems in the flow such as audit systems which need to get a copy of the market and trade data and persist it for compliance reasons. Inline and post trade risk systems also play their part. Data needs to flow between servers and programs so it can be processed or stored. Whether these servers are distributed in a datacenter or around the globe, the data needs to move in a fast, low latency, reliable and scalable manner.  The system also needs to be sufficiently flexible and the parts sufficiently decoupled to enable ongoing changes at multiple levels across the system.

TCP/IP is a great backbone for encoding and moving data over the wires between servers, but application programmers need more than just a networking service, such as never losing a message even when there is no network connectivity. For example, even if the trade data capture (audit) server is down, the trade should go ahead, but the audit server should still get the data once it comes back online. Messaging is the technology that provides this smart layer as a service so developers can take transport for granted.

The Building Blocks for SOA and EDA

There are two common messaging paradigms: point-to-point or queue based, and publish/subscribe or subject/topic based. Point to point messaging simplifies data communication over TCP (or equivalent) where a sender sends a message to a queue and the receiver reads it off the queue. This is analogous to Tom sending an SMS to Harry – point to point. The messaging system manages connectivity, handles network issues, guarantees delivery, handles expiry, etc.

The more powerful paradigm is publish/subscribe messaging, which allows senders to broadcast messages on a “topic” or subject, and one or many receivers just listen to the topic. This is analogous to users tuning into radio. The sender is broadcasting music on a frequency (i.e. topic), and whoever is tuned in gets the music (i.e. message).

Topics are the foundation of a Service Oriented and Event-Driven Architectures. Topics allow each consumer application to choose what it wants to listen to from the “message bus”. This is orthogonal yet complimentary to some EAI tool invoking each service provider (a risk system, for example) for functionality. With a Topic based EDA, the risk system knows when it has to act by just listening to the topics it is interested in, rather than the ESB invoking it. This means more things happen in parallel. The fact that the individual receivers have now been transactionally decoupled from each other serves to increase the reliability and performance of the system as a whole while reducing the complexity of the data publisher.  SOA and EDA with a message bus is a very interesting discussion, but for another article.

Functions and Benefits of Messaging

Here are some of the primary functions and benefits of messaging:

• Persistence: Guaranteed messaging is the ability to store and forward messages to consumer applications such that no data is lost even if the receiver is not online. The data is spooled by the messaging system in case the consumer, e.g. the trade data capture server, is offline. With raw networks, applications need to have lots of intelligence and complexity baked into their code to handle this – messaging makes life easy by guaranteeing delivery.
• Platform Independence: A powerful messaging system is language independent and platform independent. It should allow its user the ability to send a message, say from an iPhone app, e.g. a bank funds transfer, to be received by interested backend subscribers, e.g. a Java/Oracle based banking system, and a C# .Net based risk system and more, for it to be processed and replied to.
• Sophisticated Functionality: The messaging system provides much richer functionality based on commonly used patterns for the application developer. Some of these are wildcarded topic subscriptions, topic to queue bridging, last value queues, dead letter queues and total time to live, message caching, eliding etc.

Just like TCP is a networking standard, there are a couple of notable messaging standards. Java Messaging Service (JMS) has been around for a decade now and is reasonably mature, though is limited to Java environments only. AMQP looks very promising, but is going through the iterations of standardizations and adoption.

In a nutshell, messaging makes the application developers leverage transport as a service that they can consume, to allow them to focus their efforts on business logic – that’s what they’re paid for, right? If you have a distributed computing problem, where you need to flexibly and scalable move data around , there’s a good chance that messaging may be the right solution for your need.you should be looking at messaging.