splunk-jms_1In this blog post, I am going to demonstrate how to integrate Solace with Splunk using JMS Messaging Modular Input (aka jms_ta). Splunk can ingest data from multiple sources using variety of Data Inputs; jms_ta is one them which is used to ingest data into Splunk from a JMS Broker. This blog builds on top of an existing blog Handling Solace Message Router Events in Applications. This is an excellent reference to build your understanding about Solace Message Router Events mechanism.

We will demonstrate how the following Solace Messaging Events gets ingested into Splunk via Modular Input JMS (jms_ta):

  • Client Connect / Disconnect
  • Subscription Add / Remove

* You can refer to Solace Events Reference online documentation for complete list of Events with detailed description of each.

Solace Base Setup

Splunk can be integrated with both Solace Messaging Router Hardware Appliance and Virtual Message Router (VMR). You can refer to instructions to install and configure a local VMR running on your machine.

Splunk Base Setup

  • You can use an existing Splunk installation or Install a fresh new instance
  • Install jms_ta by following instructions at JMS Messaging Modular Input
  • Copy following Solace JMS Jars at $SPLUNK_HOME/etc/apps/jms_ta/bin/lib
    • sol-jms-.jar
    • sol-jcsmp-.jar
    • sol-common-.jar
    • commons-logging-.jar
    • commons-lang-.jar
  • Restart Splunk so that JMS Modular Input is detected and is visible in Splunk Admin GUI

Solace Configuration

Now that we have Solace VMR up and running, let’s get started by creating necessary objects:

Create VPN (Name = SplunkVPN)

  • Enable VPN to publish “Client Event” & “Subscription Event” messages
  • Set Authentication type to “Internal”
  • Set the “Maximum Spool Usage” to 100 MB

Modify existing Client Profile (Name = default)

  • Enable “Allow Guaranteed Message Send” & “Allow Guaranteed Message Receive”

Create Client Username (Name = splunk)

  • Set password (splunk)

Create Durable Queue Endpoint (Name = Events)

  • Set “Access Type” to “Non-Exclusive”
  • Set “All Other Permissions” to “Consume”
  • Add following Topic Subscriptions for the events we are interested to monitor to “Events” queue
Event to monitor Topic String (wild carded)
Client Connect event #LOG/*/*/*/CLIENT_CLIENT_CONNECT/>
Client Disconnect event #LOG/*/*/*/CLIENT_CLIENT_DISCONNECT/>
Add Subscription event #LOG/*/SUB_ADD/>
Remove Subscription event #LOG/*/SUB_DEL/>

** Topic to Queue bridging is one of the most powerful Solace feature that helps wiretapping the data flow seamlessly. The Topic Subscriptions can be added / removed on the Queue without shutting down the ingress / egress ends of the Queue. A good example is let’s say you want to start monitoring the Client’s Flow Open and Close events, simply adding #LOG/*/CLIENT/*/CLIENT_CLIENT_OPEN_FLOW/SplunkVPN/>
#LOG/*/CLIENT/*/CLIENT_CLIENT_CLOSE_FLOW/SplunkVPN/>
topics to the Events Queue will enable the wiretap these additional events to flow into Splunk.

Configure JMS Access to message VPN

Enable “Client Access” for “SplunkVPN” under the “JNDI Properties” view

  • This allows the JMS clients (e.g. Splunk JMS Modular Input) to access the JNDI store, lookup and bind to objects. If this access is not enabled, then JNDI requests from clients will receive empty 503 Service Unavailable

Create Connection Factory (Name = cf/splunk)

  • Uncheck “Direct Transport” checkbox under Transport Properties
  • Uncheck “Text Messages Are XML” checkbox under Messaging Properties

Create JMS Queue

  • JNDI Name = solace/SplunkVPN/Events
  • Message VPN = SplunkVPN
  • Physical Name = Events (This is the physical Durable Queue endpoint)

Splunk Configuration

We will create a separate Splunk Index for storing Solace Events information separately. This is an optional configuration. You could choose to ingest Solace events into any existing index.

Navigate Splunk Menu to create a new Index.

splunk-jms_2

After successful index creation it should appear in the index list as per below.

splunk-jms_3

JMS Messaging Modular Input (jms_ta) configuration:

You would have a JMS Data Input ready for configuration if everything went well as described in JMS Messaging Modular Input.

splunk-jms_04

 

 

Click “Add new” to create a new JMS Data Input to consume Solace events from “Events” queue configured in the above section. Provide the details as per below information. The configuration is pretty simple and is based on standard JNDI JMS consumer configuration. Refer to the following pictures for the values to be used for configuration.

splunk-jms_05

splunk-jms_06

splunk-jms_07

Once the configuration is completed and saved, it should reflect in the Data Inputs section as below. Ensure that the Data Input is “Enabled” which indicates that Splunk JMS data input is now connected and listening on “solace/SplunkVPN/Events” queue. Validate that Client is connected to Solace VMR and has bound to “Events” queue using either of the Solace Admin tools – Solace Admin GUI OR Command Line Interface.

splunk-jms_08

Run and Validate the setup

We are now ready with all the configuration required on Solace VMR and Splunk. We will generate the events we are interested in by running following SDKPerf command. (Please refer to SDKPerf Information on Solace Developer Portal)

./sdkperf_java.sh -cip=:55555 -cu=splunk@SplunkVPN -cp=splunk -stl=test/topic/events/splunk

You will see SDKPerf output similar to shown below.

splunk-jms_09

Terminate SDKPerf command using “Ctrl+c”.

Above SDKPerf execution generated the events in following sequence and we should be able to see those events in Splunk search results on “Solace” index.

splunk-jms_10

Go to Splunk Search and enter new search query index=”solace”. You should see Solace Events in the search results.

splunk-jms_11

splunk-jms_12

splunk-jms_13

splunk-jms_14

If you don’t see any data through the search as explained above, then debug in following order:

  • Ensure that you have Enabled (i.e. started) JMS Messaging Data Input we configured above
  • Validate the client is bound to Events queue
  • For detailed troubleshooting, refer to Splunk JMS Modular Input logs at $SPLUNK_HOME/var/log/splunk/splunkd.log

Summary and Next Steps

We saw how easy it is to integrate Splunk with Solace using JMS. We used simple example of ingesting Solace Event logs into Splunk for the sake of brevity. However, it proves central premise of integration with Splunk. We can extend this integration for Business Event monitoring requirements as well. For example, within Financial Services, we could potentially “Wiretap (using Topic-to-Queue bridge)” business flows such as Payments Transactions, Trade Order Management & Settlement flows, Client facing channel interfaces such as Web and Mobile Click Stream and the list goes on. The “Wiretap” feature has immense potential to open up the data already flowing on the message bus (data river) without affecting existing business transactions and seamlessly ingesting them into both data-at-rest (e.g. HDFS) and data-in-motion (e.g. Splunk) analytics systems (data lakes).

There is another way to ingest Log & Monitoring data into Splunk and that is using Syslog. Solace Messaging Router generates Syslog messages, as defined in RFC 3164 to record events that occur within the platform. We can configure Solace Router to emit these events to a configured Syslog server. Splunk Syslog Forwarder is an easy channel (Data Input) to ingest this information into Splunk. Using Syslog to ingest monitoring events into Splunk is recommended compared to ingesting this information using JMS.

References

All of Solace documentation is accessible from the Solace developer portal. You are encouraged to check them out for further information and details.

Vidyadhar Kothekar

Vidyadhar has been working on Solace products since 2012. He has over 15 years of combined experience working with product engineering teams at a leading software vendor in integration space followed by implementation of those products and concepts at investment, retail and private banks. He has successfully led and delivered large scale integration solutions across front, middle and back office applications, both within Buy and Sell side financial services organizations.

In his current role as a Senior Architect at Solace, Vidyadhar is advising customers in APAC region with data movement strategy that eliminates impediments to real-time data flows and achieves unprecedented throughput, latency and availability for messaging backbone. He's responsible for solution and architecture consulting using Solace products and works with customers through discovery, pre-sales & post-sales activities as a trusted advisor on Solace technology and how Solace fits into larger ecosystems such as cloud, big data and the Internet of Things.