Amazon Kinesis Data Streams

This connector provides an integration of Amazon Kinesis Data Streams to Solace technology and vice versa, so Amazon Kinesis clients become seamless event mesh consumers and producers. With the help of this connector, Amazon Kinesis Data Streams and its producers and consumers become part of an event mesh powered by Solace.

Amazon Kinesis makes it easy to collect, process, and analyze real-time, streaming data so you can get timely insights and react quickly to new information. Amazon Kinesis offers key capabilities to cost-effectively process streaming data at any scale, along with the flexibility to choose the tools that best suit the requirements of your application.

This 2-way integration is configured via “workflows” in your connector instance. A workflow is a source-to-processing-to-target data pipeline configured within the connector runtime. Each workflow (you can have up to 20 defined per connector instance), defines a source (e.g. PubSub+ queue), any header processing necessary for the target, and a target destination. When active, events will stream across these workflows.

The connector is available (below) as:

• A runnable package based on a Java JAR file including a start script
• A container image suitable for running in a container runtime such as Docker or Podman

The PubSub+ Connector is a “self-contained connector” from Solace. All self-contained connectors share a common architecture and provide a number of enterprise services to the connectors such as:

• A local management server accessible over HTTP(s) and JMX exposing endpoints for:
  • Health check
  • Metrics monitoring
  • Log file access
  • Workflow adminstration (start & stop workflows)
• A common set of configuration options for:
  • logging – log levels, log file size, archive and rollover rules, appenders to export to other log services
  • security setup for management endpoints – authentication and authorization to the endpoints, TLS for HTTPS endpoints
• Various runtime deployment options:
  • Standalone
  • Active_Standby – for redundancy (you can have more than 1 standby instance)
  • Active_Active – for horizontal scaling (where the source of data will support multiple active consumers such as a non-exclusive queue in PubSub+)