Crafting Robust and Secure MQTT Client Solutions

Introduction

The event mesh is the foundational infrastructure interlinking multiple event brokers to achieve an event-driven architecture (EDA) design. It is built on top of the network infrastructure to link various systems together regardless of whether they are located on the public cloud or on-premises. It provides a secure, low latency and high throughput platform to dynamically route millions of events across various systems, and allows client applications to seamlessly integrate with backend service.

In my previous video on “Building a Large Scale Low Latency EDA End-to-End Solution” I explained how to build an EDA-based solution that leverages Publish/Subscribe patterns for information fan-out and asynchronous Request/Reply mechanisms. These patterns help achieve low-latency communication by utilizing event mesh technologies. This article will compare the differences between REST and MQTT, and then explain in detail how to combine REST and MQTT to create robust and secure MQTT client solutions.

REST vs. MQTT

RESTful (representational state transfer) APIs are a widely used programming interface on the Internet that provide a standardized method for communication. They enable synchronous, point-to-point request and reply calls between the client and web server. In a REST architecture design, it is common to utilize content delivery networks (CDNs) to cache information, and API gateway (APIGW) to manage and secure traffic.

REST has emerged as the most popular protocol on the Internet for many years. According to the Postman API report from 2023, a staggering 86% of interviewed individuals reported using REST for their development activities. However, it is worth noting that there has been a 6% decline in this number from 2021.

MQTT (MQ Telemetry Transport) is a lightweight messaging protocol specifically designed for the Internet of Things (IoT) environment. It enables efficient and reliable communication between resource-constrained IoT devices and applications. MQTT supports two fundamental messaging patterns: publish/subscribe and asynchronous request/reply.

MQTT is a cost-effective protocol and it is also gaining popularity beyond the IoT environment. According to the Postman API report, 9% interviewed respondents reported using MQTT. MQTT usage is not limited to IoT scenarios anymore, especially since the release of version 5 in 2018.

In terms of usage patterns, MQTT is commonly employed to support both the publish/subscribe (9%) and EDA (3%) approaches [Refer to postman API report]. The Publish/Subscribe pattern enables efficient data dissemination, while Event Driven Architecture decouples various systems.

Both REST and MQTT are utilized on the client side, but their focus areas differ based on their respective characteristics. MQTT is used for time-critical Request/Reply and Publish/Subscribe, while REST can work with CDN to support huge non-time-critical information Request/Reply retrieval.

Performance

MQTT generally offers better overall performance compared to REST in specific scenarios:

• Lightweight Protocol: MQTT has a minimal message size. The overhead of an MQTT message can be as low as 2 bytes, which is significantly smaller compared to REST. This is because REST API calls typically include additional information such as the HTTP method, content type, authentication, and authorizations headers. These additional elements contribute to a larger message size, resulting in increased bandwidth usage.
• Permanent Connection: MQTT establishes a connection at the beginning, and subsequent message transmissions are based on this persistent connection. In contrast, REST requests establish a new connection on each API call. This fundamental difference between MQTT and REST has implications for message rate and information push capabilities.
• Latency: MQTT offers lower latency compared to REST due to its streamlined design and reduced infrastructure components.With MQTT, the client can subscribe directly using topics of interest in the backend using wildcard subscriptions.  And data will be pushed/published to those clients as events occur. This direct communication eliminates the need for intermediate components that are typically involved in REST infrastructure design. The absence of these additional components results in a simplified and more efficient communication path.

• Asynchronous API Call: MQTT supports asynchronous communication by design while REST API calls are primarily designed for synchronous communication. MQTT provides inherent support for asynchronous communication, enabling multiple in-sequence API calls without being blocked by slow process service calls.

Caching

A CDN provides caching capabilities for some REST API calls, which can effectively reduce network latency, improve performance and reduce the infrastructure cost. However, data latency is increased because of the extra cache layer on the CDN.

Security

Both REST and MQTT protocols support OAuth 2.0 for authentication and TLS channel encryption for secure communication. However, MQTT provides an additional feature known as stricter account-based security control, which I will explain later.

Reliability

The MQTT protocol offers two quality of service (QoS) levels, namely QoS0 (at most once, direct message), QoS1 (known as “at least once”, persistent message), and QoS2 (known as “exactly once delivery). REST does not have built-in support for release QoS level by default, but REST is essentially QoS1 because the HTTP request will generate a response indicating success (e.g. 200, 4xx, or 5xx).

Scalability

Load balancing technology can greatly enhance scalability by distributing incoming requests across services, allowing for efficient handling of concurrent connections. When it comes to scalability, both REST and MQTT can benefit from load balancers.

Both MQTT architecture and REST architecture support scalability by Load Balancer technology. One Web server can support 400-500 concurrent connections, and some MQTT brokers can support up to 200,000 concurrent connections.

The follow table summarizes the difference between REST and MQTT:

While MQTT protocol offers several advantages over REST, it does not necessarily replace REST entirely: they can be very complimentary. There are scenarios where it may not be feasible or cost-effective to handle a large number of repeat requests with the same reply content using MQTT, and where a REST + CDN solution can be a better option.

Architecture Design

Depending on the channel application, UI design properties such as screen size, hardware, or device will vary. For example, a web application needs to provide rich information, while a mobile client must offer a compact user experience and easy interaction. An Automatic Teller Machine (ATM), on the other hand, may display less information, but provide more shortcuts to different services such as ePayment, purchasing, and account operation.

These three different types of applications have distinct UI requirements, but they share over 90% of data, and provide the same set of services, such as Account Operation, ePayment service, and transaction service.

This solution is a comprehensive design that caters to the requirements of all aforementioned channels. It utilizes the traditional REST and CDN solution to handle non-time-critical functions, such as historical or full set of information retrieval. On the other hand, MQTT is employed for the delta information push and time-critical functions, such as account operations.

Furthermore, the entire solution is integrated with CIAM (Centralized Identity and Access Management) system. This integration allows the generation of JWT (JSON Web Token) and their verification through a REST call.

Solution Design

JSON Web Token (JWT) Generation

When a customer opens the main page or mobile app, the application initiates a token exchange request with the CIAM via APIGW. This request allows the application to obtain an authentication JWT. Subsequently, all subsequent REST API calls or MQTT connections within the application will utilize this token for authentication and authorization purposes.

Information Retrieval

The application initially makes a request-reply call to the source system in order to build the initial state in memory. Once the memory is established, the application continues to update it by receiving delta updates from the source system through pub/sub.

On the other hand, the client interacts with both the application server and the web server. The client retrieves the image from the web server while retrieving snapshot information such as a list of stock names and stock prices from the application server’s memory.

However, there may be a slight delay in obtaining the information due to the involvement of a CDN. The CDN caches the information based on the REST endpoint, and the cached values are dependent on the CDN’s specific setting.

As a result, there might be a few seconds of delay in receiving the updated information as the client’s request goes through the CDN’s caching mechanism.

When the client receives a delta update from the source system through the MQTT connection (green line in the diagram above), it will update the corresponding values on the screen. Simultaneously, the application server which also subscribes to the same set of topics will update its own memory.

The updates in these values are near to real-time because they originate directly from the source system. There is no cache layer between the client and the source system, ensuring that the updated values are delivered promptly without any caching delay. As a result, the client and application server can reflect the most recent and up-to-date information from the source system in near real-time.

Account Related Operation

In order to proceed with any account-related operations, the client is required to log into their account and retrieve customer details. Subsequently, they should utilize the obtained token to establish a reconnection with the event mesh. It is important to note that all subsequent account-related operations will rely on this permanent connection.

Infrastructure Design

Most server-side systems adopt a stateless microservice design, which operates within a Kubernetes infrastructure to support auto-scaling capabilities. To ensure efficient distribution of incoming requests, it is essential to configure a software load balancer in front of these systems.

Security Control Design

Infrastructure Security Control

Security controls are dependent on network infrastructure, firewall technology, and event mesh infrastructure, all of which must meet the security compliance requirements of your organization. The diagram below presents suggested security controls for different areas.

MQTT Security Control — OAuth

Establish the MQTT Security Connection

The MQTT connection can be secured through OAuth 2.0 authentication, which offers enhanced security measures. An MQTT security connection can be established by:

• Step 1: The client establishes a connection with the CIAM to obtain the customer JWT.
• Step 2: The CIAM system responds to the client’s request by providing a valid JWT, which encompasses essential information such as the unique account key, scope, and expiration time.
• Step 3: The client utilizes the newly received JWT to establish a connection with the event mesh.
• Step 4: The event mesh validates the JWT to ensure its authenticity and integrity, and then
  • Stores the expiration time of the JWT in its cache, and terminates the connection once the timer reaches the expiration time.
  • Applies the publish and subscription rule on the Access Control List (ACL) profile, taking into account the account’s unique key and scope.

All message exchanges rely on this persistent connection, and the ACL profile is checked to ensure that the account’s unique key matches the unique key associated with each published topic.

Security Control on Publish and Subscribe Events

The token provided by the client upon establishing the connection allows the event mesh to perform a cross-check between the account’s unique key on the JWT and the topic name. The backend consumer further validates the account’s unique key against the topic and message payload. More detailed information regarding this process will be provided during the “event flow design” session.

MQTT Configuration on the Event Mesh

To enable the event mesh, it is necessary to configure it since it is not enabled by default. This involves setting up the event mesh to create both the OAuth Provider and the Authorization Groups. To illustrate the configuration process using MQTT, let us take PubSub+ as an example.

The following diagram shows the major steps required to enable MQTT on PubSub+. Detailed steps are provided in the appendix.

Event Flow Design

The event flow design should adhere to the principles of Event-Driven Architecture, specifically utilizing the Publish/Subscribe pattern to implement the Request/Reply mechanism.

A well-defined naming standard for event addressing, or message topic, should include two parts: a mandatory and an optional part. The mandatory part should include at least four fields: information domain, information subdomain, message type, and schema version. [Refer to “The Importance of Event Governance in EDA”]

Events might pass through various systems during its lifetime where it could mean different things to different systems or applications, but the topic that represents the event message must remain constant. From a topic naming perspective, the event’s data component values (the mandatory part, and the event specific optional part) should be solely based on the definition of the original purpose of where the event originated from.

The event topic name needs to align with the microservice design and MQTT security controls to ensure that only logged-in customers can publish or subscribe to events on relevant topics. This is achieved by incorporating the account’s unique key and verifying that the events originate from the same device app or browser used for authentication.

On the article titled “The Importance of Event Governance in EDA”, we utilized an EFT service process flow to illustrate how a topic name change can occur within a single event flow, and now we enhance the topic name structure to fulfil the MQTT requirement above.

EFT Process Flow

The user initiates an EFT withdraw event on the mobile client, and subsequently, this request event is transmitted to the backend service through the event mesh for further processing. Ultimately, the outcome of the backend process needs to be forwarded back to the respective customer device.

Original Design:

arrangement/txn/evt/v01/eft/{component}/{status}

Enhanced Design for MQTT

We will insert additional information ({account’s unique key} and {device id} between “eft” and “{component}”. The ACL profile restricts client access to the topic based on the proper arrangement of the {account’s unique key}

arrangement/txn/evt/v01/eft/{account’s unique key}/{device id}/{component}/ {status}

Distributed Tracing

Due to the event mesh handling a high volume of events, often reaching tens of thousands per second, it is crucial to enable distributed tracing across the entire path. This allows for tracing the flow of both REST messages and MQTT events.

It is necessary to have a tool which can quickly debug problems, optimize complex architecture environments for performance and cost, and enable proof-of-delivery and audit. This solution’s distributed tracing is built on the open standard, OpenTelemetry, which fulfils all these requirements. [The detail of OpenTelemetry please refer to The Importance of Event Governance in EDA]

The trigger point (mobile app) generates a 16-byte trace ID and then propagates it throughout the entire flow via different protocols (e.g. REST, Messaging, etc.) Each operation in the flow generates a separate span ID and writes the span ID, along with the input/output time and trace ID, to the application performance monitoring (APM) solution.

MQTT does not support OpenTelemetry natively, and some event mesh do not support direct message. So, we need to modify the application code to add trace ID to the message payload or message header. The downstream system exposes the trace ID from the message, and then adds to the span attribute. This case will generate two separated trace IDs, and the APM tool will link them together as a single trace flow.

You can refer to the article titled “Building an OpenTelemetry Distributed Tracing Solution” to gain a comprehensive understanding of OpenTelemetry, and its implementation in JMS applications.

Summary

This general design caters to the needs of many organizations, fulfilling their requirements for performance, reliability, scalability, latency, and security. However, the actual implementation of this design is significantly more complex. It involves various types of mobile clients with diverse message size requirements due to display limitations and device resource constraints. Additionally, the same service must be processed through different backend systems and integrated with internal or external systems such as banking services. The network infrastructure is divided into segments, including the DMZ and internal network, with multiple firewalls deployed for different purposes within the internal network. Depending on your organization’s architecture and infrastructure design, additional technologies such GraphQL, AMQP, APIM, and others should be incorporated into the solution.

Appendix A: PubSub+ MQTT Configuration Procedure

We will use Solace as an example to demonstrate how to enable MQTT. You can configure other event buses by following a similar procedure.

Step 1: Enable MQTT on the PubSub+ Broker Level

Step 2: Enable MQTT on the VPN Level

Step 3: Enable OAuth Authentication on the VPN Level

Step 4: Create a new OAuth Provider

1. Create a new OAuth provider.
   
2. Fill in the claim source and claim name.
   
3. Enable and fill in the claim source and claim name on the authorization group.
   
4. Fill in the URI and refresh interval on the JWKS.
   

Step 5: Create a new Authorization Group

1. Create a new Authorization Group.
   
2. Add the corresponding topic name to the “Publish Topic ACL Profile”. For example:
   
3. Add the corresponding topic name to the “Subscribe Topic ACL Profile”. For example: