What a Load of JSON: XML and other Text Data in The Solace APIs

I recently wrote a series of blog posts that introduced the elements of Solace messages including message header properties and how to encode data in the payload.  Now I’d like to dig a little deeper and talk about using text messages in Solace. Text messages offer more than just simple string transfer: common formats such as JSON and XML allow the transfer of complex data as strings with readily available libraries to write, read and parse the data for you.

Performance

Text messages offer simpler, quicker and perhaps more agile development.  However, they are slower to write, read and parse, and they add extra latency and tend to be bigger than the equivalent binary or structured types.

XML, JSON… what format?

The first choice to make when using a text message is what format will you use.  Plain text works well for simpler data, but you will end up parsing it yourself, so using an existing format has big advantages.  Two common formats are XML and JSON.

Some common XML libraries are:

• For C: libxml2 and expat
• For .Net: XDocument
• For Java: JAXB
• For Javascript: DOMParser

JSON is, of course, native to Javascript.  Common libraries in other languages:

• For C: JSON-C
• For .Net: NET
• For Java: Google GSON

I’m not recommending any particular library or format here, just giving some examples to help you get started.  For the rest of this post I’ll use JSON, but the majority of points apply to XML and probably to most other text format packages as well. If you’re interested in learning more here’s an introduction to JSON.

A use case

To illustrate the concepts, let’s imagine the use case of writing a system to track, route and allocate delivery vehicles.  Let’s define our vehicle object:

public class VehicleRecord {
	private class LatLong {
		private float Lat;
		private float Lng;

		public void setLatLong(float lat,  float lng) {
			this.Lat = lat;
			this.Lng = lng;
		}
	}
	public class Parcel {
		private String sender;
		private String recipient;
		private LatLong deliveryLatLong;
		public Parcel() {
			…
		}
	}
	private LatLong currentLatLong;
	
    private String depotName;
	private LatLong depotLatLong;

	private String destinationName;
	private LatLong destLatLong;

	private String driver;
	
    private Vector<Parcel> manifest;
}

All we need to do is use Gson to convert this to JSON:

import com.google.gson.Gson;

Gson gson = new GsonBuilder().create();
gson.toJson(VehicleRecordObject);

Sending this as a message is then trivial:

VehicleRecord vehicleObject = new VehicleRecord();

// Populate…

TextMessage msg =
	JCSMPFactory.onlyInstance().createMessage(TextMessage.class);
msg.setText(gson.toJson(VehicleObject));
…
producer.Send(msg,  destination);

Reception

Once we’ve got this message in the onReceive callback, we need to deserialize it.  The most important part is to make sure we cast the BytesXMLMessage to a TextMessage to get the text payload:

vehicleRecord vehicleRcvd;

public void onReceive(BytesXMLMessage msg) {
	if (msg instanceof TextMessage) {
		try {
			vehicleRcvd = gson.fromJson(
				((TextMessage) msg).getText(), 
				VehicleRecord.class);
		} catch (…) {…}
	}
}

Now we have our original object in vehicleRcvd, with hardly any effort at all. Quick, easy, and powerful!

Java class as the message protocol definition

If you’re using the same language on both the sending and receiving side, it’s a simple matter to use the same definition of the object (in this case VehicleRecord) on both sides.

For Java, this is the class definition.  Your class definition serves as a message format specification while providing you with all the normal object access and manipulation, which is pretty cool.

What if I change my class?

What this doesn’t give you is any version control over this format.  If you change the format (for instance, add a field) applications will probably fail to parse the message unless you update all the applications at the same time, .

So it is well worth working out how you’ll deal with this at the start of the project.  You can decorate the message with a User Defined Header Field, or you can use the message topic.  I’d recommend using topics so you don’t have to worry about handling unknown formats in your business logic.  It’s a simple matter to use something like:

vehicletrack/direct/json/vehicleRecord/1.2/…

That way, a version compiled with version 1.2 of the vehicleRecord class will only receive that version.  With a little cleverness, you can incorporate the version in to your build scripts so the version topic level is automatically filled with the version number.

What if I have different APIs?

The C-style JSON parsers tend to generate a tree structure or allow iterative access to fields. This means you don’t have a direct one-to-one mapping between sent and received objects. With C# and C++ you will at least have classes to compare against.

Compression

Text formats tend to be very chatty.  However, this very chattiness lends itself very nicely to compression. Solace APIs and routers implement compression, which makes things easier.  It’s easy to enable:

• JCSMPChannelProperties.setCompressionLevel(9);

There are 9 compression levels, with 9 being highest effort compression.

I created 30 messages of different sizes from 700B to 5kB, for a total data set size of 85kB. With this test there was no real difference in the levels, with all giving a compression ratio of close to 20 times.

In comparison, using a mapMessage to create this object saved about 20% in size compared to the uncompressed message, at the expense of considerably more code to serialise and deserialise the message.  In this case, of course, most of the data are strings. There will, however, be more CPU use to parse the text with a TextMessage compared to a map message.

Closing curly brace

In all but the most CPU intensive or latency sensitive applications, text messages, along with text formats such as JSON and XML with their associated libraries, offer an easy to develop, easy to debug pattern for data payloads.   If you’re worried about bandwidth consumption, Solace’s compression facilities offer an extremely easy to use and high performance way to reduce the payload size.  Use text messages where you can!