Mainframe Integration: Use an Event Mesh to Extract Real-Time Events From Systems of Record

Mainframe technology is still considered a powerhouse when it comes to Systems of Record (SoR) and bulk data processing like statistics and transaction processing, but as peripheral systems are moving away from mainframe, real-time integration is becoming more important.

Hybrid cloud, microservices, reducing high capital investment, the need for cloud scale, etc. are accelerating the migration of systems out of the mainframe. CIOs and CTOs are realizing that incremental modernization will need a coexistence strategy with mainframe. Systems like statement generation are being moved out, but still require real-time integration with mainframe.

The Challenges of Mainframe SoR, SoE, and SoI Integration

Organizations continue to run a lot of critical workloads, especially Systems of Record on z/OS-based mainframe technology. When it comes to high-speed transaction processing, mainframes simply have no competition in terms of speed, the volume of transactions they can handle, and cost-effectiveness. That’s why banks still lean on mainframes for their core operations. Many customer interactions, such as credit card and ATM transactions, are carried out through high volume, real-time, online transaction processing (OLTP).

Engagement between companies and their customers continues to become a major component of the customer experience. While data about customers is useful, data about their engagement with a company adds a whole new layer. Organizations invest in Systems of Engagement (SoE) to manage this – to facilitate the customer journey across multiple touch points and providing a seamless experience. SoE workloads are generally highly parallel, with each individual interaction being of low value.

Systems of Engagement must interact in real-time with Systems of Record (SoR), which act as the source of truth and as custodians of critical business data. SoR are mostly transactional, highly stateful, and demand absolute consistency and transactional integrity, regardless of the value of an individual transaction. The state of an airline seat, for example, must be exact and must show consistency for every querying entity.

Systems of Intelligence (SoI) complete the picture by performance tracking and predictions based on artificial intelligence (AI). Organizations generally want to avoid converting the SoR where possible, but still need to be able to extract real-time events from the SoR for building the inference engines and to develop analytics and artificial intelligence models. Machine learning is often used to train and improve the models.

Architecturally, collection and aggregation interfaces have always been a problem in software design. In the case of SoE and SoR, one must interface front-end SoE (a highly variable and unpredictable event stream) to back-end SoR (originally architected for a more deterministic and serialized environment). Tight coupling between the two layers nullifies the benefits of nimbleness and agility that SoE offers.

An Event Mesh Facilitates Integration

An event mesh facilitates the integration between mainframe SoR, SoE, and SoI – creating a highly scalable and distributed architecture. The loosely coupled approach supported by the event mesh solves the ‘impedance mismatch’ across the layers.

An event mesh is a configurable and dynamic infrastructure layer for distributing events among decoupled applications, cloud services and devices. It enables event communications to be governed, flexible, reliable, and fast. An event mesh is created and enabled through a network of interconnected event brokers.

In other words, an event mesh is an architecture layer that allows events from one application to be dynamically routed and received by any other application no matter where these applications are deployed (no cloud, private cloud, public cloud). This layer is composed of a network of event brokers.

The generic capabilities of an event mesh include:

• A network of interconnected event brokers that can be deployed in any cloud, PaaS, or non-cloud environment.
• Dynamic distribution of events so that event consumers can receive events from any event producer, no matter where the producer and consumer are attached to the mesh, without the need for configuration of event routing

z/OS Mainframe Integration Options

z/OS provides a few options mainframe technology integration with external systems. In the section below, some of the commonly used patterns are listed. These include:

• CICS SOAP/JSON WebServices
• ESB/Adapter
• z/OS Connect
• z/OS with Event Mesh

CICS SOAP/JSON WebServices

CICS supports integration of applications using web services and offers two approaches: rapid deployment using an opinionated approach with least amount of programming or a flexible model using code to customize the web service application. Either way, application programs running in CICS can participate in a heterogeneous web services environment as service requesters, service providers, or both. This model also supports CICS as a JSON server, with the limitation of only supporting POST method.

CICS support for SOAP Webservices conforms to open standards, including SOAP 1.1 & 1.2, HTTP 1.1, WSDL 1.1 and 2.0 and JSON. These capabilities have been available since CICS v3.1 and have been continuously enhanced.

Support of open protocols allows CICS WebServices to be seamlessly connected to the event mesh without a need for a third-party adapter.

ESB/Adapter

Traditional enterprise service bus (ESB) technology will integrate with z/OS by leveraging an adapter. At a very minimum, apart from transport level integration, the adapter is required to do character-set translation as well as convert from mainframe COBOL message structures. Mainframe are big-endian systems and use EBCDIC character sets. This contrasts with little-endian Intel-based systems running ASCII and UTF-8. There is also a need to do message transformation.

The message structures in COMMAREA used by CICS programs are in COBOL COPYBOOK formats, and this must be converted to an open message format like XML or JSON. This implies that the adapter is tightly coupled with the z/OS program and any change in copybook needs to be done simultaneously on z/OS and the ESB adapter.

The transport between the Adapter and the z/OS is either an MQ Channel or raw TCP/IP.

In this approach, mainframe still uses a proprietary connector, and open protocol is available from the ESB/Adapter. In this model, the connectivity to the event mesh leverages the adapter.

While all leading integration products have these adapters, modern enterprises are now moving towards leveraging the native WebServices/JSON support available natively on the mainframes. This further reduces the reliance on ESBs and accelerates the journey to microservices.

z/OS Connect

z/OS Connect is IBM’s premiere technology for implementing JSON Services and APIs in CICS. z/OS Connect is the entry level low-cost option, and z/OS Connect EE is a separate IBM product and offers a richer RESTful API support.

z/OS Connect EE provides a RESTful API interface to z/OS with a security model, the ability to map the request to the backend data requirements, and data transformation.

Support of open protocol allows z/OS Connect to be seamlessly connected to the event mesh.

z/OS with Event Mesh

Based on the above mainframe integration approaches, CICS programs on z/OS can be connected to the event mesh using REST. Since the patterns use open protocols, there is no need for additional connector frameworks; Solace can natively connect to it. The three options are shown in a single picture below. Note that while the ESB/Adapter approach is being considered, availability of WebServices should allow you to integrate without ESB as well.

The Security Layer for Mainframe Integration

Any conversation about mainframe integration with the mesh will arrive at the topic of security. Here we will provide with a few basics about the security layer that is enforced in Solace, apart from the additional controls in the z layer.

Encryption

Solace makes use of TLS 1.2 to provide transport level security between Solace PubSub+ Event Broker and the mainframe.

Authentication

Solace API can be used to use a client certificate authentication by providing valid X509v3 certificates. It also supports basic authentication by authenticating with a username and a password. These credentials can be validated against Internal user repository, corporate LDAP server or an external RADIUS server. While OAuth (typically useful for connecting third party applications ) is not supported, consuming data APIs require system-to-system authentication, and not end-user authentication.

Authorization

You can use the ACL profile and topic-to-queue mapping to define which API is invoked by any of the clients (using either RESTful or messaging APIs) and which will be authorized and dispatched to the mainframe. The administrator can also be used to define a set of IP addresses/subnets from where connection is allowed or disallowed.

Solace supports wildcards while granting access, greatly simplifying the administration. Instead of defining every API separately, the taxonomy can be used to allow permissions at a broad subtree level.

IBM’s REST implementation requires several custom HTTP headers, and Solace PubSub+ will forward the headers sent by the microservices so that the CICS program can function correctly.

A Solace PubSub+ Event Broker acts as the gateway to connect to z/OS, allowing the synchronous REST request/reply mechanism to be event-enabled and to reach across the mesh.

Summary

In summary, Solace eases mainframe integration by bringing it onto an event mesh. By doing so, the mainframe SoR and the distributed SoE and SoI layers can become totally integrated into an end-to-end, real-time, event-enabled architecture using open standards and simple, widely understood protocols.

In turn, this means that the visualization of events and the tracking (lineage) of the data that make up the event is fully transparent and more available for audit, analysis, and sharing with other systems. A further benefit is that the development and management of end-to-end events is made simpler through tools such as the Solace PubSub+ Event Portal.