Designing, Documenting and Testing Event APIs for IoT Platforms

In this article we will dive deeper into how we designed and documented the Event APIs of our IoT reference platform that we outlined in our previous blog post about prototyping an IoT platform. We will:

• Describe the event flow between platform components
• Identify the core APIs for the use case
• Explain how we designed and documented Event APIs and provide examples and links to the full design and documentation.
• Briefly explain how we on-boarded partner applications and tested apps

Throughout the project we started to adopt AsyncAPI, so we will also briefly outline how AsyncAPI and Solace’s Event Horizon initiative will eventually provide full, tool-supported Event API life-cycle management. The APIs and tooling described here can be found on GitHub.

Introduction

In our earlier post we introduced a reference platform for IoT that we built with several partners. Solace’s event mesh connects all the components of the platform.

We had to ensure that all our and our partner’s contributions played together, and that development could be carried out independently by:

• Designing the contract between the components
• Documenting this contract
• On-boarding partners onto the event mesh
• Coming up with a way to independently test contributions

In other words, we had to design, document, and test event driven APIs. The diagram below depicts the event flow between our platform components:

1. Sensor data/telemetry flows from the device.
2. This data is consumed/processed by visual analytics – Altair Panopticon.
3. Telemetry is also fed into SL’s RTView-based real-time dashboards.
4. Commands and configuration events flow from a custom user command and control app implemented in Dell Boomi to the device.
5. Configuration events signal a status change – this is of interest to analytics systems. For example, setting devices into race mode indicates the start of a race on our slot-car track.
6. Notifications and alerts are created based on data analytics and are consumed to create service cases in Salesforce.

Looking at this use case, we identified three functional areas or APIs:

• Telemetry– everything related to sensor data from the device
• Command & Configuration – control and status messages from and to the device
• Notification – alert and notification events that can be consumed and used to generate actions

As you can see, we separated the data and control communication with the device into two APIs. This enables us to do more fine-grained routing of events through the IoT Event Mesh.

The actors in the diagram above fall into two different categories:

• Distributed devices – the Bosch XDK110
• Platform services including analytics, device configuration, dashboards, and monitoring

Compared to REST/HTTP, the roles of the actors in an Event API are harder to classify. In REST a client initiates a request, a server provides the response and this communication is always point to point. Event APIs are much more varied. They are inherently bi-directional, and they support publish/subscribe and request/reply message exchanges. This makes it  harder to assign client and server roles.

However, we still decided to identify the client side of an interaction and the service provider and describe the interaction from the client’s point of view similar to REST APIs. We considered topics as resource identifier and assigned the provider role to the actor that is managing the data.

In case of device interaction, this means we identify the device as the client that pushes data to a telemetry resource from which the provider consumes the events and processes them. For configuration data we consider the service provider to be the device or configuration management application as it is responsible for maintaining configuration and notifying the client – the device – when changes occur.

How to design event-driven APIs?

Expanding on the previous sections we identified the following aspects of an Event API that we would need to address:

• Actors – client and service provider
• Channels these actors communicate on – topics or queues
• Exchange patterns – publish, subscribe, request-response
• Intent or verbs – Is the event intended as a create, update, or delete? We considered it important to include the verb similarly to the HTTP request method.
• Payloads – the event data
• Attributes such as protocols, Quality of Service, security.

As mentioned previously, we decided to describe the interaction from the client’s point of view with a provider implementation mirroring this interaction.

Solace PubSub+ is a polyglot event broker and we leveraged a number of protocols for the showcase such as MQTT, REST, SMF (Solace API), and WebSockets based on the client’s or provider’s preferred technology/APIs. This is quite a powerful feature when it comes to rapid integration of many different technologies, but it has its drawbacks as well. The implication is that we can only rely on the event characteristics that all these protocols share in order to maintain interoperability.

The typical approach to solving this would be to create a convention for a message envelope that encloses message headers as well as the payload. However, this has some fundamental drawbacks as well. Each application needs to parse the payload to extract the headers and only then apply processing rules. This is both expensive in development and maintenance but also at runtime because it typically results in each message being parsed multiple times in the system.

An alternative approach is to make use of fine-grained topic structures to convey all required information at run time about the event. In practice, this requires that the event broker be able to support potentially many millions of topics – which is exactly one of the characteristics the Solace PubSub+ broker has. This is the approach we have chosen here.

Essentially, the topic becomes the way to communicate meta information such as the verb (or operation), the resource identifier and resource hierarchy or the expected payload type.

Therefore, we need to define conventions for the topic namespace and to design this namespace carefully. The way Solace PubSub+ implements topics helps us to create a flexible and extendable namespace. This blog post contrasts Solace’s topic implementation to a more traditional approach and explains it in more detail:

“With Solace, topics aren’t actually configured on the broker: they’re instead defined by the publishing application at publish time. They can be thought of as a property of a message, or metadata, and not like “writing to a file” or “sending to a destination.” Each and every message could be published with a unique topic.”

This is also why it’s easy to commission additional IoT devices into our solution that can communicate on dedicated topics immediately without additional broker configuration.

We defined the following convention for topic names (you can find more details and examples in this github project):

{method}/{representation}/{base-topic}/{version}/{resource-categorization}/{resource}/{id}/{aspect}

This convention is loosely based on typical REST resource identifiers and also incorporates some information that in the REST world is typically conveyed via HTTP headers:

• Method – indicates clearly the action that the recipient of the event shall take such as “CREATE”, “UPDATE”, and “REPLACE”. Equivalent to HTTP verbs.
• Representation – the data format used such as JSON or XML
• Base topic – similar to the base URL
• Version – the API version
• Resource categorization – this can be a hierarchical location, functional categorisation or similar
• Resource – such as device or IoT gateway
• Id – the unique resource Id
• Aspect – such as metrics, status, configuration

As we standardised on JSON as the payload format and didn’t intend to introduce versioning, we omitted these items from the topics we defined.

For a device emitting telemetry data the destination topic looks something like this:

CREATE/iot-control/bcw/solacebooth/handheld-solace/device/7bbb15d6-ca36-498a-9d52-7ddcef2a1d75/metrics

Armed with all of this we started describing our API design. We used Async API   for this, an initiative similar to Open API in the REST world.

Let’s have a quick look at the channels and messages defined in the Telemetry API.

It defines one channel that the client – the device – is supposed to publish events to (see the “publish” definition in the excerpt below). The intent of the client is to create a new telemetry event, hence the verb used is “CREATE”. You will recognize the topic design outlined above. The elements of the resource categorization and the device id are defined as parameters in the channel:

channels:
	  CREATE/iot-event/{region}/{location}/{production-line}/device/{deviceId}/metrics:
	    description: Topic to subscribe to connected events
	    parameters:
	      - $ref: '#/components/parameters/region'
	      - $ref: '#/components/parameters/location'
	      - $ref: '#/components/parameters/production-line'
	      - $ref: '#/components/parameters/deviceId'
	    publish:
	      summary: device publishes sensor readings. each event represents a new (create) reading for the timestamp.
	      operationId: publishTelemetry
	      message: 
	        $ref: '#/components/messages/telemetryMessage'

The telemetryMessage transferred via this channel is referenced above and you can see how it is defined in the components section.

components:
	  messages:
	    telemetryMessage:
	      name: telemetryMessage
	      title: Telemetry Message
	      summary: Telemetry Message
	      description: "Depending on the mode of the device, the telemetry message can contain all of the sensor readings or only a partial set. For units and ranges please see the datasheet for the XDK. You can find it on this site: https://xdk.bosch-connectivity.com"
	      contentType: application/json
	      payload:
	        $ref: "#/components/schemas/telemetryEvent"
	 
	  schemas:
	    telemetryEvent:
	      type: array
	      items:
	        type: object
	        title: Telemetry Reading/Sample

The components section also defines schemas and parameters using JSON Schema.

The second API we created was for the device to receive command and configuration events.

It was important to use a topic hierarchy that allowed us to target either a specific device or a group of devices on a production line, a location, or region. To achieve this, we designed a set of topics (or channels) that follows our design convention and creates a unique topic for each of the devices and device groups.

Here’s an excerpt from the channel hierarchy which strips out elements of the categorisation one by one to address a higher-level group. You can see that a device subscribes to a number of topics and once again these topics do not need to be defined on the broker upfront and the broker supports use of millions of topics.

channels:
  UPDATE/iot-control/{region}/{location}/{production-line}/device/{deviceId}/configuration:
    description: device subscribes to configuration events. each event represents an update to the existing configuration.
    parameters:
      - $ref: '#/components/parameters/region'
      - $ref: '#/components/parameters/location'
      - $ref: '#/components/parameters/production-line'
      - $ref: '#/components/parameters/deviceId'
    subscribe:
      summary: Subscribe to device specific configuration events
      operationId: subscribeDeviceConfiguration
      message: 
        $ref: '#/components/messages/configurationMessage'
  UPDATE/iot-control/{region}/{location}/{production-line}/device/configuration:
    description: device subscribes to configuration events. each event represents an update to the existing configuration.
    parameters:
      - $ref: '#/components/parameters/region'
      - $ref: '#/components/parameters/location'
      - $ref: '#/components/parameters/production-line'
    subscribe:
      summary: Subscribe to production line specific configuration events
      operationId: subscribeProductionLineConfiguration
      message: 
        $ref: '#/components/messages/configurationMessage'

You can find the full definitions for all three APIs on GitHub:

• Telemetry API
• Command & Configuration API
• Notification API

These are not very easy on the eye, so let’s see how we can create HTML or Markdown documentation.

Documentation

We first started out handcrafting our documentation on our internal wiki. However, as AsyncAPI progressed over time, it became the best option to maintain the API design and generate documentation as required. We have published the full documentation on GitHub in markdown format so it renders really nicely.

The AsyncAPI tooling is still in its early days, however, we found their template-based generator tool quite useful. It comes with HTML and Markdown templates.

You can explore how to design and create different documentation formats on the AsyncAPI playground for their upcoming 2.0.0 release of the specification.

It loads an example API:

Let’s load an API definition directly from our GitHub:

• copy this URL https://raw.githubusercontent.com/solace-iot-team/iot-platform-api-docs/master/api-definitions/iot-event.yml
• paste this URL into the textbox that reads “Type the URL of the AsyncAPI …” and then click “Import document”:

You can toggle the different views between HTML and markdown. The AsyncAPI Generator allows you to automate document generation as part of a build process in a similar manner to this web tool.

To explore our three APIs in detail, start with the readme on GitHub.

Onboarding

We manually onboarded applications onto the PubSub+ platform. Onboarding of an API client or provider required the following steps:

• Create an Access Control List (ACL) for each application: the ACL provides access to the topics an application utilising one or more APIs needs to publish and subscribe as per the API definitions. A future enhancement could be to generate these ACL profiles directly from the Async API definition.
• Create a client user name for the application to be registered. At this time Solace supports username/password and client certificate authentication. OAuth support utilising an external Authorization Server will be released shortly.

Here at Solace we have things in the pipeline that will automate this process:

• Event Portal – will provide application lifecycle capabilities that provision the information above into a Solace Event Mesh. Read more in the “What’s next?” section below.
• A set of IoT APIs that simplify integration with Device Management platforms and provide capabilities to provision device types and devices into the Event Mesh. This will be released as part of our IoT Labs repository.

API Testing

As described in the previous blog post in this series, we mainly used MQTT Box to test APIs. In the future we may be able to use the AsyncAPI Generator to generate stub applications and build automated tests based on these – at the time of writing the code generation templates are at a lower level of maturity compared to the document generation, so pursuing this avenue was not deemed feasible for our project.

What’s next?

Solace recently has announced its Event Horizon initiative aiming to create a developer experience and ecosystem around Event APIs similar to the synchronous API/REST experience.

The first step will be an Event Portal that enables the creation and discovery of events and the definition of event-driven application interfaces. Solace is also contributing to frameworks such as Spring CloudStreams and AsyncAPI adding code generators for CloudStreams which will accelerate Event API implementation and aid in testing.

In the IoT space we will add more open source capabilities for Device Management and IoT Platform integration. We also evolve the IoT APIs described in this blog post as we add more prototypes and examples on other platforms such as Azure IoT Hub and Azure Sphere.

Conclusion

A crucial factor in the development of our prototyping showcase was to design our event-driven IoT APIs upfront, similar to contract-first development with REST APIs. By making sure the topic structure/namespace we used was flexible and extendable, we avoided payload introspection and enabled fine-grained and flexible filtering for every subscriber.

We followed a typical API management lifecycle to design and document our IoT APIs through to on-boarding of applications and testing. AsyncAPI proved very useful in the design and document phases. As the Solace Event Portal, AsyncAPI and code generators evolve the full API life-cycle will eventually be tool-supported.

The next article (coming soon) will focus on the implementation of the custom RTOS C application we have developed for the Bosch XDK110 which implements the device APIs described here.

Stay tuned!