When deploying your kdb+ estate, you may have struggled to efficiently and easily replicate your kdb+ instances around the globe or in the cloud. While there are many strategies that you can use, such as chained tickerplants or duplicating your feedhandler processes, all of them add complexity. In this post, I will introduce why you might want to replicate kdb+ instances, and explain how you can easily accomplish this natively with q and Solace PubSub+.
Your initial deployment of kdb+ applications may look something like this image, where you start out with your feedhandler pushing data into your kdb instance in New York and users across the globe are accessing the instance. As your user base grows, you may scale out your kdb+ instances horizontally, but you still incur latency costs as every query/response has to transit the WAN. So, as your user base grows, this architecture may not work as well.
There are a few issues we would like to solve here by replicating your kdb+ instances across the globe:
While there are various methodologies you may employ to achieve replication for your kdb+ instances, you would want to ensure the following:
As discussed in the previous post, using the Kx Fusion Interface you can natively send messages to Solace PubSub+. Combining this capability with subscribing to your tickerplant, you can send updates to a Solace PubSub+ Event Broker in one region, have Solace PubSub+ forward the message to brokers in another region, and then push the updates into your tickerplant in the other region. The benefit of this pattern is that it can even be extended to the public cloud.
Creating an event mesh – a cluster of brokers – sounds complicated but it’s actually extremely simple with Solace PubSub+. Follow these instructions in this video to link brokers together.
Consider a single table quote with the following schema:
quote |
|
time |
timestamp |
sym |
symbol |
bidPrice |
float |
askPrice |
float |
bidSize |
integer |
askSie |
integer |
quote:([]time:`timestamp$();sym:`symbol$();bid:`float$();ask:`float$();bsize:`int$();asize:`int$())
Now to set up a process to subscribe from your tickerplant and publish onto Solace PubSub+, you could do something like this:
sendToSolace:{[t;d] if[not 98h=type d;:(::)]; d:update `g#sym from d s:exec distinct sym from d; topics:{[t;s] "/" sv ("solace/kdb";string t;string s)}[t] each s; json:{[d;s] .j.j select from d where sym=s}[d] each s; .solace.sendDirect'[topics;json] } upd:sendToSolace;
In a nutshell, the code above will retrieve the name of the table and the sym from the record, construct a well-defined topic (example: solace/kdb/quote/AAPL
), and send the record(s) as a json record onto Solace PubSub+.
Now, in the other geography/location you will set up a Solace PubSub+ subscriber using something like the following code:
onmsg:{[dest;payload;dict] j:.j.k "c"$payload; h(".u.upd";`quote;(.z.P;exec `$sym from j;exec "f"$bid from j;exec "f"$ask from j;exec "i"$bsize from j;exec "i"$asize from j)) } .solace.setTopicMsgCallback`onmsg .solace.subscribeTopic[`$"solace/kdb/quote/>";1b];
The code above will subscribe to all messages that start with the topic solace/kdb/>
and push into the quote table.
Using Solace PubSub+ with kdb+ gives you a low-touch and frictionless way to achieve a robust replication strategy for your kdb+ stack. The Solace PubSub+ Event Broker is also completely free to use as a docker container. You can find all code referenced above in the kdb-tick-solace repo.