The Case for Event APIs and Unified Event & API Management

APIs have changed the IT world, and they’re becoming more important and widely adopted as we move towards an API economy where APIs are increasingly the distribution channel for products and services. However, as new use cases emerge, traditional synchronous APIs based on REST, GraphQL, etc., are no longer sufficient. In this article, I will outline the need for event APIs that complement synchronous APIs, and the capabilities developers need to find, understand, and use them.

I will start with an example from my own experience with retail customers and describe what a good event API and developer experience looks like. Then, I will outline the capabilities required to make it happen.

Example: Enabling Click and Collect Purchasing in Retail

Imagine you’re a developer responsible for fixing the pain described in this social media post.  Working for a home improvement retailer, you’re tasked with building a new app or online shop that lets customers order supplies via click-and-collect, also commonly called buy online, pick up in store or “BOPIS.” Customers want to know that the products are available and when they will be ready for pickup in store to finish a job.

A lot of the data and functionality is easily obtainable – you can access catalo, stock, order and similar REST APIs through your company’s developer portal. In addition, presenting current in-store stock levels while the user is browsing or viewing product details is crucial.

Once an order is placed, the customer also needs to be informed on the status – when it is estimated to be ready for pickup? When is it actually ready?

You could use existing synchronous RESTful APIs to poll for stock levels or the estimated time for pickup, but what if there was a better way? A way that doesn’t introduce the risk of stale data, that doesn’t put load on the synchronous APIs by constantly polling for stock levels, or require complex caching and rate limiting to reduce that load?

I have good news, there is such a way: event APIs let you access and receive events such as real-time stock levels from the user’s chosen store and others in its vicinity, and updates on the order status once an order is placed. Such “events” are automatically delivered to your application when they happen instead of when the information is requested or scheduled polling. You can use web protocols such as WebSocket, Server Sent Events or Webhooks, – or subscribe to relevant stock updates using protocols such as MQTT or AMQP.

There’s a myriad of ways these same events could be used across a retail organization, from stock pickers who need order and stock level notifications to central inventory management applications and data analytics engines and dashboards.

What Makes a Good Event API?

Before you can consider solutions to access and receive events, you must understand what event APIs are, how they differ from RESTful APIs, and how they can differ from each other.

Well Defined and Documented

Events are discrete in technical terms whereas a REST resource bundles several operations. This means REST APIs are quite simple to understand and hence ubiquitous.

If you don’t have access to an OpenAPI or RAML API specification (which you’d normally want for documentation and code generation) there are well-defined basic conventions that make it relatively easy to figure out how a resource works.

Once you know a resource URL (e.g. /orders) and a means to authenticate (e.g. an API key) you can obtain a list of orders using “GET /orders”. The response will help to understand what an order document looks like. You can proceed to obtain a single order appending its id to the resource URL “GET /orders/123” or attempt to create an order using “POST /orders”. HTTP as a protocol enables this behaviour as it defines a Request-Reply exchange pattern and methods such as GET, POST, DELETE that the caller can use to communicate intent.

Event protocols typically define the event exchange pattern – send or receive an event, but there is no standardized way to convey the intent or behavior. If you represented a RESTful order resource as events multiple events are required to represent its CRUD operations. Additionally, you may also choose to generate different types of “orderUpdated” events – there could be an “orderStateChangeEvent” or more fine granular events such as “orderProcessed”, “orderShipped”.

There isn’t a widely accepted convention (implicit or explicit) for how to identify events as relating to the same resource.

Consequently, providing documentation for an event API as a collection of related events becomes crucial. Besides the event exchange direction – send, receive – each event needs to be documented with its specific channel – a topic, queue, tag, resource URL or similar – and the payload of the event. AsyncAPI is the ideal way to achieve this as it is modelled on OpenAPI and caters to the idiosyncrasies of events as described above.

Puts You in Control of the Events You Consume

An event API should put the developer and their application in control over the events it receives. Just like a REST resource allows you to specify exactly which order you are interested in; an event API must allow you to specify which exact order’s events you want to receive and the type of order events.

Or, for the example in the introduction, the application must be able to receive stock updates for a set of relevant products from stores close to the user.

Some protocols used in event-driven architecture (EDA) lend themselves nicely to express/filter the events you want to receive like a resource URL, such as  MQTT, Solace’s Solace Message Format (SMF) or HTTP itself.

The AsyncAPI specification enables you to document such resource channels including parameters similarly to path parameters used in REST APIs. Here’s an example:

Instead of using a coarse-grained channel such as products_stocklevel, the channel above lets the client control exactly which events to consume. The different parameters in the channel are color coded as a developer may want to set some parameters statically (such as the region if developing an application targeted only at the UK) or dynamically (such as the store id or the productid depending on where the user is located and what they are interested in.)

There may also be topics for specific update events (color coded in blue) allowing clients to subscribe to specific event types such as:
products/stocklevel/v1/outofstock/{region}/{productid}/{store} or
products/stocklevel/v1/backinstock/{region}/{productid}/{store}

Easy to Connect To

An event API, along with the events that it defines, specifies a contract or interface. When an event API is made available to developers to use it needs to be considered where the event API is provided. Similarly to a REST API endpoint event APIs should be available on either a single standardized endpoint or a set of geographically distributed endpoints where a client could choose the best option for its location. The underlying network topology should be invisible to the client –a failover to a hot standby or a redirect to a specific broker instance should be handled internally. From an operational perspective this also has the advantage of reducing the network attack surface.

Discovering & Accessing Events & APIs: The Ideal API Portal Experience

Getting hold of events still often presents the same challenge that traditional middleware integration prior to the adoption of REST APIs and API Management experienced – it isn’t easy to find and get access to resources quickly.

As a developer it would be very helpful to find relevant related events – combined into event APIs – that help you solve the challenge at hand, learn about and try these APIs interactively, register for access to the API and instantly connect their application(s).

You’d probably also appreciate that you can find and gain access to event APIs in the same developer portal where you usually go to and use them in a similar manner to other (RESTful or other synchronous) APIs.

The key to successfully providing self-service access to events is to create an experience that is as familiar to developers as possible.

Enabling Self-Service Access to Driven APIs

Developers have grown accustomed to a simple, self-service onboarding process when using APIs. As an API provider how do you enable this kind of experience for event APIs. The API developer portal needs to be able to present event APIs, allow developers to request access to APIs, and obtain connection information near instantaneously.

It is also helpful to present all relevant APIs for specific use cases or scenarios regardless of type. The retail example in the introduction outlined the need for a developer to have access to synchronous APIs (for catalog, order management) and asynchronous APIs (e.g. stock updates, order updates) to implement their use case. If the developer portal presents these APIs together as a bundle of APIs called an API product, it’s easy for the developer to identify the required set of APIs.

When related event and REST APIs are presented as a single API or API product, a consistent design is helpful to developers. This could include using the terminology, similar structure of resource paths and event topics as well as reusing schemas to establish a common object model.

There’s a lot to be carried out behind the scenes from API modelling and publishing, access request approvals and revocations, to configuring the API entry point with the resources and access controls required to enable the application to connect.

Documenting, Managing and Governing Event APIs

There is a lot of back-office functionality required to run an event API program alongside or integrated within an existing API program. The API Team needs to discover or import, model and document events and event APIs, package event APIs and make these available to developer portals.

Looking at API specifications from an API Management perspective – where typically the API is the most granular object   – it may seem surprising that decomposing imported event APIs into events or discovering discrete events is key to creating developer friendly event APIs.

However, as applications using event APIs are typically decoupled by event brokers there is no need for a single “provider” to implement the same API as the “client”. Once you deal with discrete events, you can decompose event APIs of “providers” – such as a system of record – and reassemble them into whatever “client” APIs you need. As long as the events are available or routable to the gateway or broker or event mesh that all applications connect to, you don’t need to create an API Proxy or façade that mediates to various provider APIs as you would in the synchronous/REST world. You are merely packaging events and advertising them as event APIs. In essence “You can have as many APIs as you like” – event APIs are essentially documentation and configuration templates: new APIs can be composed as needed.

Designing and documenting APIs is only the start of the journey – once developers are using APIs, you need to carefully manage the evolution and lifecycle of an event API. Essentially all the lessons learned from synchronous APIs can be applied.

Event APIs should provide a stable contract which requires minor versions to be backwards compatible and therefore only allows non-breaking changes. This enables developers to update to newer minor versions of the API when they need to use newer features or when their development and deployment schedule allows. On the other hand, API providers can provide and maintain the latest version only.

Once a major release is required the lifecycle of the previous major release needs to be managed carefully as to limit impact on developers and their applications – which means that both the new and the previous major version must be supported for a transition period. Also, the state of the API must be communicated to the developers – moving to deprecated, retired and eventually to deletion – while also preventing developers to sign up for deprecated APIs.

From an operations point of view, tracking users of deprecated and retired APIs also becomes important to gauge the impact of retiring an API – which may render applications inoperable – and therefore whose migration to a newer version we must facilitate.

Configuring an Event Broker or Gateway for Application Access

The API Team also needs to manage access requests by developers. Typically access requests require some sort of approval – for example as it grants access to production environments or requests from partners or customers need to be validated.

Once access to an event API has been granted the broker(s) or event gateway(s) need to be configured with client specific resources and access controls.

While registering access to a REST API mainly involves issuing credentials and potentially associating scopes with the credentials, event APIs often require the configuration of client specific resources – such as queues and quotas for guaranteed messaging, configuration for Webhook invocation or guaranteed messaging quotas for MQTT clients.

Conclusion

I’ve outlined here the various components and capabilities needed to offer event APIs:

• Why offering events is important to most API Programs
• Event API Design, Documentation, Versioning and Life cycle Management
• Event API and API – publishing, product management and curation
• Self-service access through an API developer portal
• Enabling access and provisioning resources for applications on the event (data) plane

A lot of these are API management concerns that many organizations today already address for synchronous APIs, with either a full API management solution or a combination of tools and gateways. Rarely are organizations building this infrastructure from scratch.

Plugging event APIs into these tools and platforms is essential to provide a one stop shop for APIs to developers, however some aspects of managing events and event APIs go beyond the typical scope of (RESTful) API Management and therefore require additional tooling and platforms – typically found in an event portal. For example, the ability to manage a discrete events’ life cycle and the composition of events into event APIs is a capability that event modelling and governance must add to the stack.

We will explore design and governance as well as runtime architecture approaches using existing API Management solutions, extensions to these as well as EDA capabilities in a subsequent blog – so stay tuned.