Automating Solace configuration management using SEMP and Ansible

Home > Blog > Automating Solace configuration management using SEMP and Ansible

There are two ways of managing PubSub+ Event Brokers: command line interface (CLI) and a web-based admin interface called PubSub+ Broker Manager.

For small deployments (say a handful of brokers) and development environments, using CLI and PubSub+ Broker Manager is sufficient. As the size of the Solace deployment grows, however, there are many scenarios in which customers would want to automate the process of creating and managing the configuration of Solace brokers. For example:

For agile, cloud-based applications that perform frequent deployments, it may be necessary to quickly spin up a Solace environment, perform testing, and tear it down.
Administrators would love to automate the process of creating, updating, and deleting a Solace environment for greater efficiency.

Solace Element Management Protocol (SEMP) provides a RESTful API for programmatically creating brokers and managing their configurations. Using SEMP, administrators can automate every aspect of Solace configuration management. Such automation increases productivity, removes the risk of human error associated with manual deployments, improves quality control, and accelerates time to market.

Since SEMP provides a RESTful API, it can be integrated with virtually any DevOps tool in the market – such as Ansible, Chef, Puppet, and Salt. For a nice introduction to the SEMP Management API, check out this blog, as well as the SEMP tutorials home page.

A few years back, I created a GitHub project on how to use Ansible to automate Solace deployments for integrating Solace with CI/CD pipelines. Since the project has been used frequently in the field as well as by many of our customers, I’d like to share more detail on how the Ansible playbooks work and how you can use as well as extend them for automating your Solace configuration.

Ansible

Ansible is an open source tool for automating tasks such as application deployment, IT service orchestration, cloud provisioning, and many more. Ansible is easy to deploy, does not use any agents, and has a simple learning curve. Ansible uses “playbooks” to describe its automation jobs, known as “tasks”. Ansible playbooks and tasks are described using YAML, making them easy to read, understand, and extend. Here is a quick background of the concepts in Ansible before we dive into how it can be used with Solace.

Playbooks

Configuration tasks using Ansible are laid out using playbooks – they can be used to manage configuration and deployment to remote machines. “Playbooks” are composed of “plays”, with each play defining the configuration work to be performed against a managed server. Within each play, you can define “tasks” – each task is a set of actions and instructions for configuration management, and these will be executed against a set of hosts.

By composing a playbook as a set of plays, it is possible to orchestrate deployment of Solace configuration across a single or multiple Solace event brokers, running certain steps on one set of brokers, and all steps on others, as required.

Roles

Roles, in Ansible, provide a method for breaking down a complex playbook into simple manageable components – each component is composed of a collection of tasks, variables, files, templates, and modules.

Modules

Ansible comes with a number of in-built “modules” – each module can be directly executed on a remote system to perform common operations, such as copying files, starting and stopping services, running Unix commands, managing EC2 instances, and so on. When writing Ansible plays, we can make use of these modules to perform common operational tasks. In this sample, the Ansible URI module is used to interact with Solace brokers by calling the SEMP API.

Ansible Playbook for Solace

This GitHub project contains a sample Ansible playbook for creating Solace messaging environment on an existing Solace broker, using the SEMP RESTful API. The Ansible playbook consists of a number of tasks, and each task uses Solace’s SEMP Management API to create a new Solace message-vpn on an existing Solace message broker along with associated objects:

Client Profiles
ACL Profiles
Client Usernames
Queue Endpoints

If the objects already exist, this is indicated in the playbook run’s output. It is not treated as a failure of the Ansible task, and the playbook execution continues to the next task. Object properties can be specified in a configuration file.

Variables

The details of the various configuration objects, such as their name and properties, are specified in the file vars/solace-env.yml. Here’s a snippet from this file showing example configuration for a message-vpn to be created:

#message-vpn
message_vpn:
   msgVpnName: "srAwesome_vpn"
   authenticationBasicType: "internal"
   enabled: true
   maxConnectionCount: 60
   maxSubscriptionCount: 200
   eventLargeMsgThreshold: 2048
   maxEgressFlowCount: 100
   maxEndpointCount: 100
   maxIngressFlowCount: 100
   maxTransactedSessionCount: 100
   maxTransactionCount: 100
   maxMsgSpoolUsage: 1000

All the variables for the environment to be created are specified in this file. This includes:

Message-VPNs
Client Profiles
ACL Profiles
Client Usernames
Queue Endpoints

Roles

The Ansible playbook is broken down into roles – with one role for creating each configuration object:

create-vpn
create-acl-profile
create-client-profile
create-client-username
create-queue
create-queue-subscription

These roles can be found in the directory “roles”. Each of the roles contains a set of tasks, for performing the configuration operation appropriate to that role. Let’s look at tasks for creating a message-vpn, in roles/create-vpn/tasks/main.yml:

- name: Compose SEMPv2 request payload for VPN "{{ vpn.msgVpnName }}"
  template: src=templates/create-vpn.json.j2 dest=files/create-vpn.json

- name: Create Message-vpn "{{ vpn.msgVpnName }}"
  uri:
   url: "http://{{ mgmt_host }}:{{ mgmt_port }}/SEMP/v2/config/msgVpns"
   method: POST
   user: "{{ semp_username }}"
   password: "{{ semp_password }}"
   headers:
    Content-Type: "application/json"
   body: '{{ lookup("file","files/create-vpn.json") }}'
   body_format: json
   force_basic_auth: yes
   status_code: 200
   return_content: yes
  register: result
  ignore_errors: True
  retries: 3
  delay: 1

- name: Error in VPN Creation?
  debug:
    var: result.json.meta.error.status
  when: result.json.meta.responseCode != 200

Three tasks are defined for each configuration object:

Load the variables for the task and compose the SEMP request body using Jinja2 templates (more on that in the next sub-section)
Send the SEMP POST request to the broker to create the configuration object
Verify if there was any error in the object creation by checking if the error code was 200 or not.

Templates

In order to ensure that the SEMP request body is composed dynamically from the variables specified, in the format required for the POST operation, the sample makes use of Jinja2 templating. The template JSON body for each of the different SEMP operations is defined in the directory templates/. The first task in the task-set is to load the appropriate template, replace the variables in the template, and create the SEMP request body under the files/ directory.

Once the SEMP request body is created, it is executed using the Ansible URI module and output is checked in the following tasks.

Inventory

The Ansible inventory contains the details of the hosts against which the playbooks are to be run – in our case, these are Solace brokers we want to create configurations in and appropriate management user details for access.

[solace]
sgdemo mgmt_host=192.168.42.11 mgmt_port=80 semp_username=ansible semp_password=ansible

Running the Ansible playbook for Solace:

Pre-requisites

Install Ansible.
The host running the samples should have network connectivity to the management IP of the Solace broker.
A Management user must be created on the Solace broker and given a global-access-level of read-write.

Checking Out and Building

To check out the project and build it, do the following:

Clone this GitHub repository.
Enter cd solace-ci-cd-demo.

Configuration

Edit the inventory.ini file to specify the Solace brokers on which the messaging environment is to be created. You can create one or more host groups against which the Ansible playbook will be run.
```
[solace]
sgdemo mgmt_host=192.168.42.11 mgmt_port=80 semp_username=ansible semp_password=ansible
```
Edit the configuration files: The configuration for Solace environments to be created is specified in vars/solace-env.yml. Edit the configuration objects as necessary. The current version of the sample supports the creation of:
- A single message-vpn
- One or more client profiles within the message-vpn
- One or more ACL profiles within the message-vpn
- One or more Client Usernames within the message-vpn
- One or more Queues within the message-vpn, with topic subscriptions on these queues

Running the Sample

In order to run the Ansible playbook, use:
ansible-playbook create-solace-vpn.yml -i inventory.ini

Summary

In this blog post, I have provided a quick overview of how you can use Ansible to automate common configuration management tasks using Solace’s SEMP API. You can use this sample as a starting point for your CI/CD automation, extending as necessary for your use case.

After the CI automation using Ansible, I also provide details of how you can integrate Ansible with a Jenkins job for end-to-end pipeline automation. Refer to the GitHub project documentation for more details on how you can set this up.

Please feel free to leave me any comments regarding this example, or leave comments for the entire community to get involved in. If you have any issues, check the Solace community for answers to common issues seen.