Thus, there was no requirement to provide a comprehensive set of monitoring and management tools since a lot of this functionality would fall under the control of the big-data management platform (Ambari, Cloudera Manager, etc.). This is generally a good idea as redundant functionality is not required. But as you move to more use cases the message broker needs to be able to provide enough detailed monitoring functionality to be able to do basic troubleshooting on system wide problems. Confluent Control Center and other 3rd party tools have been used to try and fill this gap.
Solace has a suite of management tools to aid in operationalizing Solace event brokers, from Solace PubSub+ Event Broker: Cloud, SolAdmin, Solace PubSub+ Broker Manager and Solace Enterprise Stats. These tools are written by Solace specifically to enhance operations and visualization of the Solace-enabled event mesh.
Apache Kafka also has a history of not being backward compatible, but this is improving. For example, Apache Kafka 1.0.0 and Apache Kafka 2.0 introduced a new wireline format. To provide pre-0.11 backward compatibility with latest Apache Kafka 2.0.0, you need to run the 2.0.0 broker in previous wireline formats. This means that the new features and functionality are not exposed. The upgrade procedure is explained here. This is complicated and would not fit into orchestrated environments like Kubernetes rolling upgrades. This is not what larger enterprises want. They want to run the latest broker and expose the new features to the customers that want them, while allowing the producers and consumers that do not require the new features to continue to use previous APIs without disruption. Solace message brokers can be upgraded to the next major release and expose new features while supporting existing clients and APIs.
Another operational difficulty is inter-cluster data transfer. Apache Kafka brokers have no way to distribute data outside the cluster and need an additional application to read from one cluster and write to another while preserving topic/partition integrity. This means you need to have another set of applications that needs to be managed and made highly available. Also, both clusters need to be running the same versions for the MirrorMaker to work. Next you need the exact same topic/partition layout on both sides or message publish order could be lost. Finally, it is not recommended to produce and consume from clusters that you are replicating. It is recommended to add a replication layer of clusters into your network – again preserving your topic/partition layout across all clusters.
Source: https://engineering.linkedin.com/kafka/running-kafka-scale
Below is a summary comparison of monitoring and management features between Solace and Apache Kafka. The color-coded rankings are as follows:
Fully supported by the broker natively
Requires 3rd party tooling to achieve this task, or partially supported
Not supported
| Feature - Configuration Management | Apache Kafka | Solace |
|---|---|---|
| Remote programmatic interface |  |  |
| Command line interface | | |
| Config file driven | |  |
| Scripted command line | | |
| Roles based management accounts | | |
| LDAP mgmt account integration | | |
| Kerberos mgmt account integration | | |
| Internal mgmt account database | | |
| Config Replication across cluster | | |
| Dynamic Configuration broker behavior without reboot | | |
| Feature - Monitoring | Apache Kafka | Solace |
|---|---|---|
| Programmatic access to statistics | | |
| Programmatic interface to event for alerts | | |
| Per Client TCP/IP statistics | | |
| Per Client Message rate statistics | | |
| Per Client Byte rate statistics | | |
| Per Client Connect/Disconnect timestamp logging | | |
| Client Disconnect reason logging | | |
| Rollup of VPN and Router Statistics for instantaneous and average message rates | | |
| Rollup of VPN and Router Statistics for instantaneous and average byte rates | | |
| Rollup of VPN and Router Statistics | | |
| Per queue Stats | | |
| Per Topic Stats | | |
| Broker CPU Stats | | |
| Broker Memory Stats | | |
| Broker Disk Utilization Stats | | |
| Feature - Upgrades | Apache Kafka | Solace |
|---|---|---|
| In service upgrade, does not require entire cluster down | | |
| Mixed release within a cluster | | |
| Mixed release across WAN | | |
| Broker/Router support for previous version | | |
Conclusion
Both Solace and Apache Kafka provide a standard programmatic interface to monitor their systems, Solace provides a deeper view into the behavior of the overall messaging system. Solace upgrades have been a simpler task with minimal service disruption whereas Apache Kafka often requires application recode as their heavy APIs evolved. Building anything other than a trivial network architecture with Apache Kafka causes a proliferation of Apache Kafka and supporting devices.