Monitoring System Activity with Solace Syslog Forwarding

One of the most important aspects of system operations is monitoring system activity. As an operator or system administrator, real-time monitoring of the devices in your estate is essential – you want to be notified about any events regarding system health, so that appropriate measures can be taken to fix issues.

The Solace message router generates syslog messages for events that occur for a variety of reasons including system health, resource usage, service status and client events. These events provide an indication of system health, and whether any resource utilization thresholds have been reached, requiring action.

Note that Solace message routers only generate Syslog event log messages regarding router activity, and do not generate any alerts or notifications – monitoring agents should be used for this purpose. However, Solace does not allow the running of any third party components on the Solace message router for analyzing these messages and generating alerts, and these must be running on remote servers. Solace provides an open interface for transporting these syslog messages to a remote host where this can be accomplished. This remote host can be running a monitoring system or agent for parsing the syslog messages, detect events, and generate alerts such as emails, SMS, pager alerts, etc.

In this blog post, I will provide an overview of the types of events generated by a Solace Message router, and how the syslog facility in Solace message routers can be used to forward these over the network to a Syslog receiver.

This audience for this blog post is assumed to be familiar with the Solace messaging concepts, as well as basic Linux administration.

Solace Syslog Facilities

There are four distinct facilities of logging available on the Solace message router:

System logs (system.log): Stores important events related to the health and status of the Solace message router, and are not related to any particular message-VPN or client connection, such as hardware component failures, messaging service status, resource utilization events, etc. All messages in the system log, are also copied to the event logs.
Event logs (event.log): Stores all system, VPN and client level events generated by the Solace message router- message-vpn events relate to the status and resource utilization of message-vpn resources such as connections, subscriptions and message-spool, whereas client events pertain to client connections for a particular message-vpn, such as connects/disconnects.
Command logs (command.log): Stores actions performed or errors encountered by commands issued at the Solace Command Line Interface (CLI) or SEMP.
Debug logs (debug.log): Stores internal software debugging information and is to be used by Solace support personnel only.

These log messages are written to the local file system of the Solace message router, and can be found at the location “/usr/sw/jail/logs” on the local file system.

For more information on Solace Syslog categories and destinations, refer to Syslog Message Format and Components.

Forwarding Solace logs

The Syslog messages generated by the Solace message router for the system, command and event facilities can be forwarding to up to three destinations over the network. The log facilities are configurable on a per-destination basis.

The below diagram demonstrates the forwarding of Solace logs to a remote Linux host running a syslog receiver using rsyslog (http://www.rsyslog.com)

syslog-integration-diagram

A remote Linux host is configured with a Syslog receiver, such that it can receive log messages sent over the IP network.
A Solace message router is configured to forward SolOS syslog messages to the remote hosts – these can be extended to more hosts if required.
Once the logs are received by the syslog receiver on the remote host, they are written to a location on the local file system.

Monitoring agents can be running on the remote hosts which will parse and analyse the log files for particular patterns, generating alerts and notifications accordingly.

Network considerations

Syslog forwarding from the Solace message routers to an external syslog server uses connections from the management interface of the Solace message router. Hence, in order to set up Syslog forwarding, there should be connectivity established between the Syslog receiver and the management interface of the Solace message router.

When forwarding syslog messages, you can choose between TCP and UDP as the communication protocol. Solace recommends using TCP when forwarding Syslog messages as it is connection oriented, guarantees message ordering and provides congestion control.

Which Logs should I forward?

For a particular syslog destination, you can decide to forward messages based on their facility – system, command or event.

System facility: Contains all log entries in system.log. Solace recommends forwarding the messages on the system facility in order to monitor all events related to the health and utilization of resources on a Solace message router.
Event facility: Contains all log entries in event.log – this includes VPN and CLIENT events. Administrators should forward messages on the event facility if they want to monitor the health and resource utilization of message-vpns, as well as individual client connections.
Command facility: Contains all log entries in command.log – this includes all the commands issued using SEMP or CLI, along with the corresponding action performed. Administrators should forward messages on the command facility if they want to monitor the CLI/SEMP commands issued to a Solace message router.

Configuring a Solace message router to forward logs:

Prior to configuring a Solace message router to forward logs, the following information must be available:

Name for the Syslog destination
IP address of the remote host where the logs are to be forwarded
Port number of the remote host (default 514)
Transport – tcp or udp.

Sample CLI to configure a Solace message router to forward syslog messages on the event facility to a remote host with IP: 192.168.130.128 using the TCP transport are shown below:

sol-vmr-1> enable
sol-vmr-1# configure
sol-vmr-1(configure)# create syslog syslogDemo
sol-vmr-1(configure/syslog)# host 192.168.130.128:10514 transport tcp
sol-vmr-1(configure/syslog)# facility event
This command can momentarily disrupt command and event logging,  resulting in discarded logs
Do you want to continue (y/n)? y

sol-vmr-1(configure/syslog)# end

Configuring a Syslog receiver:

There are a number of syslog receivers, both free and commercial, available for collecting logs over the network. In this example, I use rsyslog (http://www.rsyslog.com) as a syslog receiver on the remote server – rsyslog is a logging utility available on most Linux systems. Modify the rsyslog configuration file at /etc/rsyslog.conf to write all incoming log messages for event.log to the local file system.

# Provides TCP syslog reception
$ModLoad imtcp
$InputTCPServerRun 10514

# Log Solace event logs to separate log file

$template SolaceLocal3,  "/opt/solace/logs/%HOSTNAME%/event.log"

if $inputname == 'imtcp' and $syslogfacility-text == 'local3' then -?SolaceLocal3;
&~

rsyslog provides a rich set of functions for further customization such as defining a format for the syslog messages, adding further template rules for additional facilities, etc. Refer to the rsyslog documentation for more details.

Verification:

In order to verify that Solace logs are forwarded to the remote server, generate a syslog message by connecting and disconnecting a client to the Solace message router. The appropriate log message should appear on the remote host:

# ls -lrt /opt/solace/logs/sol-vmr-1/
total 12
-rw-r--r-- 1 root root  911 Sep 19 07:59 event.log

The log file on the remote host will contain the connect event for the client:

# less /opt/solace/logs/sol-vmr-1/event.log

2016-09-20T15:32:31+0800 <local3.info>; solsg1 SR007INFO: CLIENT: CLIENT_CLIENT_CONNECT: shrikanth_vpn elnino/2489/#000a0001 Client (1173) elnino/2489/#000a0001 username shrikanth_username OriginalClientUsername(shrikanth_username) WebSessionId (N/A) connected to 192.168.40.130:55555 from 192.168.40.3:58288 version(7.2.1.148) platform(Mac OS X-x86_64 (Java 1.8.0_101-b13) - JCSMP SDK) SslVersion() SslCipher() APIuser('srajgopalan' Computer: 'elnino' Process ID: 2489) authScheme(Basic) authorizationGroup() clientProfile(shrikanth) ACLProfile(default) SSLDowngradedToPlainText(No)

Conclusion

In this blog post, I have demonstrated how a Solace message router can be configured to forward syslog messages to a syslog receiver on a remote host. Thus, using Syslog forwarding, Solace can be integrated with external monitoring systems and agents, and the syslog messages can be used to generate custom alerts and notifications.

Using Syslog Forwarding to Integrate Solace with External Monitoring Systems