Architectural Patterns for Event APIs and Unified Event & API Management

In the first blog post of this series, The Case for Event APIs and Unified Event & API Management, I outlined the various components and capabilities needed to offer event APIs and:

• Why offering events is important to most API Programs
• Event API design, documentation, versioning and lifecycle management
• Event API and API products – publication, product management and curation
• Self-service access through an API developer portal
• Enabling access and provisioning resources for applications on the event (data) plane

A lot of these are API management concerns that many organisations today already address for synchronous APIs. Plugging event APIs into these tools and platforms is essential to provide a one stop shop to developers to find all APIs they need. Thus, a customer will require additional tooling and platforms – an event portal.

In this blog post I will outline:

• How event APIs fit into a layered API architecture, and which eventing protocols and paradigms are appropriate.
• What an event API lifecycle looks like and how it relates to and differs from the established API management lifecycle.
• The capabilities required to support the event API lifecycle.
• How the concept of event API products enables surfacing APIs to developers in a form relevant to their use cases and business context.
• Different approaches to providing a unified developer experience.
• Runtime or “data plane” architectures for event APIs.

Event APIs and API Architecture Layers

As with any system or enterprise architecture an API-centric architecture is commonly associated with a layered approach.

Organizations, vendors and architects in the API space propose different names and numbers of layers, however there are a lot of commonalities.

Let’s use the simplest, three-layer architecture with experience, process and system APIs:

• System APIs provide information and operations on an underlying system of record as it is held in that system without any business specific context.
• Process APIs are contextualized for a specific business purpose, potentially combined from multiple underlying System APIs
• Experience APIs are highly contextualized, providing targeted information and operations for a specific client type.

Applying these layers to event APIs, the control consumers have over the events they consume and therefore fine granular event filtering becomes a crucial aspect. The contextualization and specificity of the event stream must increase from system APIs up to experience APIs.

Let’s consider a stock level event feed from a system of record such as an inventory management or ERP. If you wanted to expose it as a system API, you can supply the events on coarse-grained channels as the consumers are generally interested in the full event stream, e.g. for data replication purposes.

Similarly, for process APIs the consumers would be interested in a large proportion of events but maybe with some filtering based on region or line of business which again leads to broad (not fine-grained) channels. If there are multiple systems providing stock level event feeds the process API layer could also combine these streams into a single event API.

Finally, as an experience API layer, event feed consumers must have great control to receive contextualized, relevant events – for example a mobile application wants to consume stock-levels for a particular product in a particular store close to the user’s location. Therefore, events must be consumable on fine grained channels with filtering capabilities.

For this reason, the event broker must process events on a fine grained topic hierarchy and be able to route events directly to consumers as well as aggregate multiple fine grained subscriptions into a single destination such as a queue or external endpoint e.g. a Webhook. Event or messaging protocols used by clients to connect to brokers and create subscriptions to event streams should allow fine grained subscriptions with filtering restricting the volume of events consumed from an event API.

The Event API Lifecycle and API Lifecycle Management

When it comes to managing the evented API lifecycle you can draw similar parallels as in the section above – the evented API lifecycle generally should follow tried and tested API management lifecycle models.

The API lifecycle encompasses a sequence of phases essential for the effective design, development, deployment, and utilization of APIs.

An established API lifecycle forms a crucial component of robust API governance strategies supporting collaboration and enabling organizations to maximize the value of their API portfolio.

API providers and clients follow separate, related lifecycles. These are connected by the lifecycle of the API contract and service level agreement.

API providers publish their stable contract for use by clients. As these API contracts are exposed, their lifecycle must be carefully managed due to the potentially large number of clients.

Event API management should follow this lifecycle closely to provide a developer experience that is consistent with other types of APIs as well as integrating into existing API governance strategies.

As discussed in the previous article, event API management requires managing the lifecycle of discrete events and associated schemas as the elementary, reusable building blocks of event APIs.

What are the Ingredients for Event API Lifecycle Management?

Several aspects play into managing the lifecycle of event APIs and their components:

• Managing dependencies between different objects
• Managing versions of a component
• Managing the state of a component

Managing Dependencies

As depicted in the diagram above you must be able to create event specifications and associate them with data schemas.

These in turn are composed into event APIs creating further dependencies between objects.

For example, changes in a schema have a ripple effect across events and event APIs – therefore you need to manage versions of objects, so changes create a new version and dependencies on the original version are not impacted. The same applies to events and event APIs. To provide a stable event API version while allowing iterations through the provider lifecycle you need to track object state.

Managing versions of an Event API component

Each component of the API should be versioned:

• Schemas
• Events
• Event APIs

I suggest using the SemVer (semantic versioning) approach that defines a convention using three main components in a version number: major, minor and patch.

Multiple versions may be active (or released) at the same time. For example, there may be a major version v1 and v2 of a “stock level update” event referenced by v1 and v2 of a stock level API respectively.

For APIs that provide stability to users and developers a major version change may introduce breaking changes. However, minor version changes should always be backwards compatible. The patch component may either be used to signal insignificant changes – e.g. typos and amendments to descriptions or similar elements that do not impact use of the API or may be unused by always incrementing the minor version and keeping patch at “0”.

Often the major version is also embedded in the events’ channel (or topic) to clearly communicate breaking change to developers – for example the topic root orders/beer/updated/v1 contains the major version “v1”, if there was a major version upgrade it would change to orders/beer/updated/v2.

By the way – the topic root above follows our topic architecture best practices – a great help to designing event APIs.

Managing the state of an Event API component

As mentioned above, multiple versions of an event, schema or event API may exist and even be in use at the same time.

However, there must be a lifecycle state associated with these component versions to signal when they are under design, development and test and not yet ready for release. Given the effort involved in maintaining multiple active versions there also must be states to communicate to developers that a version should no longer be used or are no longer available for use.

The graphic below depicts a suitable object state lifecycle – the “draft” state covers the provider lifecycle from definition to deployment. “Released” corresponds to the distribution phase while “deprecated” and “retired” allows for the graceful retirement of APIs giving developers ample opportunity to migrate to a more recent version.

Event API Products surface APIs to Developers

Once a stable API contract is available and the API provider side is deployed it’s time to make event APIs available to developers.

What are Event API Products?

Event API products add a layer for packaging and marketing event APIs to developers. These are typically exposed in developer portals such as API Marketplaces so developers can find and use relevant event APIs.

While an event API can be regarded as defining the contract, the event API product enables us to package multiple APIs relevant for a specific use case or target developer profile.

It also adds connection information and applies policies – which may differ for various audiences.

You can think of event APIs as the services a mobile network provides – voice, data, text – and event API products as bundles of services with specific plans – like a mobile operator’s “unlimited data only” plan or “unlimited voice, 2GB data” plan.

So, there may be an “Omnichannel APIs” event API product that combines a “Stock Level API” and “Order Status API” for developers that is targeted at partner developers which means it’s going to be available on an external facing server. You may want to provide a better service level to our premium partners, in which case you could create a standard plan that provides “at most once” event delivery and a premium plan which provide “at least once” (guaranteed delivery) with a retention policy and allocated storage on the server.

It is also possible to use event API products as a mechanism to restrict permissions for API clients.

For example, you may want to limit stock level updates that specific API clients can receive to a region or country. Similarly to how a mobile provider’s plan may bar international calls.

In essence, event API products provide a mechanism to expose event APIs in developer portals and define policies on event APIs such as:

• Create a service level agreement such as resource allocation and data retention.
• Associate API contracts with physical endpoints where API clients can connect.
• Set permissions and restrictions for a specific class of API clients.

How do Event API Products support the API Exposure lifecycle?

As all other API components, event API products require versioning and object state to support the API lifecycle – although in a slightly different way.

As event API products form the stable, public contract for API clients they need to be locked down preventing any change upon “release”.

For this reason, the object state previously described should be amended with a “published” state which corresponds to the “Distribute” phase of the API Exposure lifecycle.

What is a good versioning strategy for Event API Products?

Event API product versioning also needs to be approached with stability in mind:

• Minor versions must not contain any breaking changes. Specifically, no elements can be removed from the API contracts. It’s possible to mark obsolete elements as deprecated and/or document that elements no longer are semantically supported although they are still formally declared (possibly even as required elements).
• Event API product versions should continually be updated with minor versions as new minor versions of underlying API contracts are released. This helps to ensure new clients are granted access to the latest event APIs avoiding dependencies on older versions.
• At the same time the previous version should be deprecated to prevent new access requests by API clients.
• A good strategy is also to create a new event API product on major version change to start with a “new slate” for new access requests and allow for graceful sun-setting and retirement of the previous version. As an example – for major version 1 of an “OmniChannel event API” you would create “OmniChannel Event API v1” and subsequently for v2 “OmniChannel Event API v2”.

Event Portals for Event API Management

Event platforms evolved from a broker-based infrastructure to add the capabilities described in the preceding sections. These go beyond capabilities of synchronous APIM when it comes to API composition and de-composition as well as inventorying existing events and auditing event brokers.

There are other aspects to this evolution such as end-to-end event tracing or event driven integration. These also add to the tooling for API management e.g. to event-enable existing systems and therefore create new event APIs or as a hook to gather API analytics.

Adding Event APIs to API Programs

Working with our customers and prospects it has become very apparent that, commonly, an API Developer Portal should provide information about and access to all kind of API assets – synchronous and asynchronous.

There are two main drivers for adding event APIs to existing API programs:

• Extending existing, web-protocol focused API programme with event APIs.
• Leveraging investment in EDA by providing easy access to more developers via event APIs, thereby increasing innovation and reducing time to market for new applications.

The first scenario is often driven by providing access to developers outside the organisation and, as it is meant to augment existing APIs, favours web protocols such as WebSocket, Webhooks or Server Sent Events.

However, event APIs targeted at external (outside the organisation) developers are not exclusively based on web protocols. In the aerospace industry AMQP is widely adopted for communications and similarly IoT related use cases are better addressed with the MQTT protocol.

The second scenario is often driven by the need to provide developers with easier access to events from existing event broker infrastructure. Providing events as APIs lowers the barrier to entry for these developers and fosters re-use of existing resources. It also lowers the middleware teams’ workload and eliminates the need for individually designed access controls and resources for applications.
This scenario is primarily focused on improving productivity of integration teams or enabling other developers in the organisation to make use of events by providing self-service access. As described above there are industries that also are looking to provide access to their events to external (to the organisation) partners and developers on non-web event protocols.

Different API Management approaches to providing a unified developer experience.

As a result of the two different drivers for adding event APIs to a unified developer experience, there are two API management approaches:

• Event Native API Management: vendors are adding event API capabilities to existing REST based platforms extending their management and runtime components to support web event protocols such as Server Sent Events or WebSocket.
• API Marketplace (or Federated API Management): given that more and more organisations utilize multiple API management solutions this approach introduces a unified management layer across API management solutions.

Both require the platform to support an event API specification standard such as AsyncAPI in addition to RESTful, GraphQL specifications. This requires import support for existing AsyncAPI specifications and capabilities to design event APIs as well as to manage the lifecycle of the API, to publish to a developer portal.

Event Native API Management

Event Native APIM adds event APIs as first-class citizen into an existing synchronous platform. event APIs appear side by side with other types in back-office management and developer portals.

Developers can request access to event APIs in the Developer Portal and the appropriate API Gateway(s) are configured to allow access to the data plane at runtime.

Event Portal APIM capabilities are required for event API design and life cycle management as the requirements for specification design – managing discrete events, de-composition and composition of APIs – go beyond editors offered by APIM.

This approach typically emerges when there is a need to provide “web friendly” event APIs using web protocols. HTTP based API Gateways are extended to support these new, additional protocols which is quite a natural evolution. Thus, the API Gateway continues to be the data pane for all APIs allowing to apply policies such as payload validation or traffic shaping.

Consequently, there is limited to no support for other event protocols. Existing event brokers are treated as back-end services and any application (client) specific elements such as queues are created by the APIM.

Universal API Marketplaces (or Federated API Management)

In contrast the API Marketplace approach acts as a management layer on top of existing APIM solutions.

Many organisations have multiple API management platforms – internal facing, external facing, departmental or side loaded with container infrastructure. These solutions may be on-premises, SaaS or commodity services of cloud providers.

The API marketplace provides a means to connect to all these solutions and manage the exposure of all APIs in a single place and in a consistent manner – some even as API products that combine APIs from multiple underlying solutions.

API marketplaces are also adding AsyncAPI support so they can be extended to integrate an event management platform similarly to how synchronous API management are integrated.

The back-office API management of event APIs is covered by the event portal’s APIM capabilities.

The data plane remains a responsibility of the Event Platform –event brokers become the gateways for API access. Consequently, event APIs can be accessed using web protocols (REST, WebSocket …) as well as event protocols (AMQP, MQTT, JMS …). Policies – service level agreement, permissions, protocols and endpoints – are configured in the Event Portal and enforced on the event broker(s).

Summary

In the blog post, I discussed how API architecture concepts apply to Evented APIs, underscoring the significance of context and specificity in event APIs. I noted the increasing importance of controlling event consumption from system to experience APIs.

Examining lifecycle management of event APIs, I drew parallels to conventional API management lifecycle models. Adhering to established API lifecycle phases is pivotal for effective design, development, deployment, and utilization, fostering collaboration and maximizing API value. Furthermore, I emphasized the integration of event API management into existing API governance strategies for ensuring a consistent developer experience.

I also outlined essential elements for managing the lifecycle of event APIs and their components, with a focus on managing dependencies, versions, and component states.

Introducing event API products as a means of packaging and marketing event APIs to developers, I highlighted their exposure in developer portals like API marketplaces, bundling multiple APIs for specific use cases or target developer profiles.

Event portals emerged as critical tools for API management, evolving from broker-based infrastructure to incorporate advanced capabilities such as API composition, de-composition, and event-driven integration.

Exploring the integration of event APIs into existing API programs, we identified two main drivers: extending web-protocol focused API programs with event APIs and leveraging investments in EDA. The first scenario aimed to augment existing APIs by providing access to developers outside the organization, while the second scenario focused on improving productivity by simplifying access to events from existing event broker infrastructure.

I discussed two API management approaches for providing a unified developer experience with event APIs: event-native API management and API marketplace (or federated API management). In event-native API management, vendors enhanced existing REST-based platforms to support event API capabilities, integrating them into existing management and runtime components. Conversely, API marketplace acted as a management layer across multiple existing API management platforms, ensuring consistent API exposure.