Solace has been serving the financial industry with its industry leading event broker, PubSub+, for almost two decades, and is known for event-enabling enterprises by liberating their data through real-time streaming. Kx Systems has been a major player in the financial services space for years too, and its kdb+ time series database is known for providing users with efficient and flexible tools for ultra-high-speed processing of real-time, streaming and historical data. After establishing their footprint in the financial industry, both companies have expanded into other industries facing similar challenges such as IoT, retail, and aviation.
kdb+ has been optimized to ingest, analyze, and store massive amounts of structured data. Its columnar design and in-memory capabilities means it offers greater speed and efficiency than typical relational databases, and its native support for time-series operations accelerates the querying, aggregation and analysis of structured data.
Solace PubSub+ message brokers efficiently move information between all kinds of applications, users and devices, anywhere in the world, over all kinds of networks. With PubSub+, you can take advantages of rich hierarchical topics, open APIs and protocols, event mesh, and Replay.
With so many mutual customers, Solace and Kx decided it would be helpful to bridge the gap between PubSub+ and kdb+. With that in mind we teamed up with Kx’s head of fusion interfaces, Andrew Wilson, and lead developer Simon Shanks to develop a PubSub+ API for kdb+ that would benefit those using the two tools today and make it easier for new users to adopt either. The API is publicly available on Kx’s github.
Here are just a few of the ways kdb+ developers can leverage PubSub+:
You can learn how to install kdb+ here. As for PubSub+, you have multiple (free) ways to setup an instance:
Please post on Solace Community if you have any issues setting up an instance.
You can find the API on github. It’s freely available and supported by Kx as part of their Fusion Interfaces. In the README.md doc, you will find all the necessary information on how to install the API.
In the initial release, the API won’t support all of PubSub+’s features, but we’ll add functionality over time. For example, in the initial release, you will not be able to initiate replay from kdb+.
Currently, here is what you can do with the API:
The API comes with some very handy examples that show you how to use it. You can find all the examples in examples directory and find related documentation in examples/README.md.
Note that to run the scripts in examples directory, you will need to provide your PubSub+ connection details via command line parameters. I have a local instance of PubSub+ running with default vpn. You can also use -user and -pass parameters to provide your credentials.
You can check capabilities of your PubSub+ instance. For example, I would like to check whether my user is able to manage endpoints by checking for SESSION_CAPABILITY_ENDPOINT_MANAGEMENT
; by running this simple script: q sol_capabilities.q -opt SESSION_CAPABILITY_ENDPOINT_MANAGEMENT -host localhost:55555 -vpn default
Here is what the output would look like:
### Registering session event callback ### Registering flow event callback ### Initializing session [29226] Solace session event 0: Session up ### Getting capability : SESSION_CAPABILITY_ENDPOINT_MANAGEMENT 1b ### Destroying session
As you can see SESSION_CAPABILITY_ENDPOINT_MANAGEMENT
is set to 1b so we are allowed to manage our endpoints.
Once you confirm that you can manage endpoints, you can create a queue: q sol_endpoint_create.q -name "kdb_queue" -host localhost:55555 -vpn default
…and see in the Solace GUI that your queue kdb_queue
was created:
You can map topics to your queue from Solace’s UI but you can also do it from your kdb+ instance through the API.
q sol_topic_to_queue_mapping.q -queue "kdb_queue" -topic "EQ/US/IBM" -host localhost:55555 -vpn default
You can see from Solace UI, a topic subscription (EQ/US/IBM) has been added to our queue:
You can also delete the queue we just created by running this example: q sol_endpoint_destroy.q -name "kdb_queue" -host localhost:55555 -vpn default
I am going to keep the queue for the rest of the examples.
In this example, you will publish a guaranteed message (as opposed to a direct message) with the payload hello world to a topic, EQ/US/IBM and because our previously created queue, kdb_queue
has this topic mapped to it, we should see your message enqueued.
q sol_pub_persist.q -dest "EQ/US/IBM" -dtype "topic" -data "hello world" -host localhost:55555 -vpn default
You can then see 1 message queued in your queue.
Note that you can also publish to a queue, but that is a point-to-point messaging pattern. Publishing to a queue facilitates pub/sub messaging pattern.
Now that your message has been enqueued, you can consume directly from the queue: q sol_sub_persist.q -dest "kdb_queue" -host localhost:55555 -vpn default
Here is the output:
Registering session event callback ### Registering flow event callback ### Initializing session [29968] Solace session event 0: Session up ### Registering queue message callback [29968] Solace flowEventCallback() called - Flow up (destination type: 1 name: kdb_queue) ### Session event eventType | 0i responseCode| 0i eventInfo | "host 'localhost:55555', hostname 'localhost:55555' IP 127.0.0.1:55555 (host 1 of 1) (host connection attempt 1 of 1) (total connection attempt 1 of 1)" ### Flow event eventType | 0i responseCode| 200i eventInfo | "OK" destType | 1i destName | "kdb_queue" ### Message received payload | "hello world" dest | `kdb_queue destType | 1i destName | "kdb_queue" replyType | -1i replyName | "" correlationId| "" msgId | 1
As you can see, the message was received with the payload “hello world”.
I highly recommend checking out other examples in the examples directory that show you other features of PubSub+ that you can implement using this API. In the next few weeks, we will be publishing a series of blog posts demonstrating how you can enhance your existing kdb+ stack with PubSub+. Here are some of the usecases these blogposts will cover:
Let us know what you think about the API and if you have any suggestions for future improvements. Hope you like it!