One of the most important aspects of system operations is monitoring system activity. As an operator or system administrator, real-time monitoring of the devices in your estate is essential – you want to be notified about any events regarding system health, so that appropriate measures can be taken to fix issues.
The Solace message router generates syslog messages for events that occur for a variety of reasons including system health, resource usage, service status and client events. These events provide an indication of system health, and whether any resource utilization thresholds have been reached, requiring action.
Note that Solace message routers only generate Syslog event log messages regarding router activity, and do not generate any alerts or notifications – monitoring agents should be used for this purpose. However, Solace does not allow the running of any third party components on the Solace message router for analyzing these messages and generating alerts, and these must be running on remote servers. Solace provides an open interface for transporting these syslog messages to a remote host where this can be accomplished. This remote host can be running a monitoring system or agent for parsing the syslog messages, detect events, and generate alerts such as emails, SMS, pager alerts, etc.
In this blog post, I will provide an overview of the types of events generated by a Solace Message router, and how the syslog facility in Solace message routers can be used to forward these over the network to a Syslog receiver.
This audience for this blog post is assumed to be familiar with the Solace messaging concepts, as well as basic Linux administration.
There are four distinct facilities of logging available on the Solace message router:
These log messages are written to the local file system of the Solace message router, and can be found at the location “/usr/sw/jail/logs” on the local file system.
For more information on Solace Syslog categories and destinations, refer to Syslog Message Format and Components.
The Syslog messages generated by the Solace message router for the system, command and event facilities can be forwarding to up to three destinations over the network. The log facilities are configurable on a per-destination basis.
The below diagram demonstrates the forwarding of Solace logs to a remote Linux host running a syslog receiver using rsyslog (http://www.rsyslog.com)
Monitoring agents can be running on the remote hosts which will parse and analyse the log files for particular patterns, generating alerts and notifications accordingly.
Syslog forwarding from the Solace message routers to an external syslog server uses connections from the management interface of the Solace message router. Hence, in order to set up Syslog forwarding, there should be connectivity established between the Syslog receiver and the management interface of the Solace message router.
When forwarding syslog messages, you can choose between TCP and UDP as the communication protocol. Solace recommends using TCP when forwarding Syslog messages as it is connection oriented, guarantees message ordering and provides congestion control.
For a particular syslog destination, you can decide to forward messages based on their facility – system, command or event.
Prior to configuring a Solace message router to forward logs, the following information must be available:
Sample CLI to configure a Solace message router to forward syslog messages on the event facility to a remote host with IP: 192.168.130.128 using the TCP transport are shown below:
sol-vmr-1> enable sol-vmr-1# configure sol-vmr-1(configure)# create syslog syslogDemo sol-vmr-1(configure/syslog)# host 192.168.130.128:10514 transport tcp sol-vmr-1(configure/syslog)# facility event This command can momentarily disrupt command and event logging, resulting in discarded logs Do you want to continue (y/n)? y sol-vmr-1(configure/syslog)# end
There are a number of syslog receivers, both free and commercial, available for collecting logs over the network. In this example, I use rsyslog (http://www.rsyslog.com) as a syslog receiver on the remote server – rsyslog is a logging utility available on most Linux systems. Modify the rsyslog configuration file at /etc/rsyslog.conf to write all incoming log messages for event.log to the local file system.
# Provides TCP syslog reception $ModLoad imtcp $InputTCPServerRun 10514 # Log Solace event logs to separate log file $template SolaceLocal3, "https://cdn.solace.com/opt/solace/logs/%HOSTNAME%/event.log" if $inputname == 'imtcp' and $syslogfacility-text == 'local3' then -?SolaceLocal3; &~
rsyslog provides a rich set of functions for further customization such as defining a format for the syslog messages, adding further template rules for additional facilities, etc. Refer to the rsyslog documentation for more details.
In order to verify that Solace logs are forwarded to the remote server, generate a syslog message by connecting and disconnecting a client to the Solace message router. The appropriate log message should appear on the remote host:
# ls -lrt /opt/solace/logs/sol-vmr-1/ total 12 -rw-r--r-- 1 root root 911 Sep 19 07:59 event.log
The log file on the remote host will contain the connect event for the client:
# less /opt/solace/logs/sol-vmr-1/event.log 2016-09-20T15:32:31+0800 <local3.info>; solsg1 SR007INFO: CLIENT: CLIENT_CLIENT_CONNECT: shrikanth_vpn elnino/2489/#000a0001 Client (1173) elnino/2489/#000a0001 username shrikanth_username OriginalClientUsername(shrikanth_username) WebSessionId (N/A) connected to 192.168.40.130:55555 from 192.168.40.3:58288 version(184.108.40.206) platform(Mac OS X-x86_64 (Java 1.8.0_101-b13) - JCSMP SDK) SslVersion() SslCipher() APIuser('srajgopalan' Computer: 'elnino' Process ID: 2489) authScheme(Basic) authorizationGroup() clientProfile(shrikanth) ACLProfile(default) SSLDowngradedToPlainText(No)
In this blog post, I have demonstrated how a Solace message router can be configured to forward syslog messages to a syslog receiver on a remote host. Thus, using Syslog forwarding, Solace can be integrated with external monitoring systems and agents, and the syslog messages can be used to generate custom alerts and notifications.
Once the log messages are received on the remote host, they need to be parsed and analyzed. Refer to SolOS Event Reference for a comprehensive list of the Syslog messages generated by Solace message routers, along with a description, recommended action and message format for each event.
Shrikanth is a former Solace employee who had been working with the Solace Professional Services Group since 2013, delivering consulting services to Solace’s global customer base. During his career with Solace, he accelerated customer adoption of the Solace platform by providing architecture and design services, developing bespoke applications, and delivering training and mentoring services.
Shrikanth holds a Master’s degree in computer engineering from the National University of Singapore and a Bachelors’ degree in electronics engineering from SRM University in India.