Integrating the Remote Edges of Retail Operations; Why Kafka isn’t the Entire Answer

One of the biggest challenges modern retailers face is getting products from increasingly distributed suppliers into the hands of consumers via diverse in-person, online and mobile channels. The consumer-facing side of that equation is what’s called omni-channel customer experience, and it can’t happen without integrating diverse applications and devices across those far-reaching supply chains.

Many have taken to doing so with Apache Kafka, but there are a few problems with doing so. As background, Apache Kafka was designed for streaming and logging massive amounts of logs, not for streaming and managing time-critical operational and transactional events and information like those that make up retail operations.

There are several problems associated with applying Kafka to geographically distributed retail ecosystems:

• Kafka deployments are inherently complex in terms of both architecture and data path, and they get worse as you scale to geographically distributed locations.
• Kafka isn’t IoT friendly, so you need to add dedicated MQTT brokers to the mix.
• Kafka clusters can get massive and require increasing amounts of governance to keep storage under control.
• Kafka does not support sophisticated event streaming features like topic wildcards, dynamic inter-cluster routing, multiple protocols, sophisticated disaster recovery, or non-persistent messaging.

4-Part Series: Why You Need to Look Beyond Kafka for Operational Use CasesA series that identifies the ways in which PubSub+ is a better fit for operational use cases that require the flexible, robust and secure distribution of events across cloud, on-premises and IoT environments.Read Now

How PubSub+ Complements Kafka to Help Retailers Integrate Remote Systems

To become truly real-time, retailers need a solution that connects all kinds of applications and devices, including those already sharing information with Kafka.

The best way to do so is with event-driven architecture, specifically an event mesh that enables the automatic routing of real-time information from wherever it’s generated to wherever it needs to be, anywhere in the world, even during periods of peak traffic or system failures.

Solace PubSub+ Platform is a sophisticated event streaming and event management platform that lets you build and manage an event mesh that that integrates applications and devices across globally distributed supply chains, distribution networks, and both online and offline retail channels. As part of that, PubSub+ simplifies your Kafka deployment by connecting Kafka clusters in such a way that data/events can be streamed between them.

Before I give some customer examples of retailers using PubSub+ to integrate their operation from supplier to point of sale, I’ll introduce to capabilities that are key to PubSub+’s ability to help you link remote systems: internet of things (IoT) connectivity and wide area network (WAN) routing.

Connecting Devices at the Edge

IoT is revolutionizing the way all kinds of companies do business, and the retail space is no exception. The ability to collect data from sensors in warehouses, on vehicles and in stores can give retailers an accurate view of where exactly inventory is, what’s in transit, what’s selling, and more. That up-to-date understanding of all inventory can help them optimize the efficiency of their distribution network, hold suppliers accountable to service level agreements, and ensure that product is making its way to whatever channels and markets require it.

One of the ways Solace helps retailers track the movement of products across their supply chain, distribution network and stores is with support for an IoT-friendly protocol called MQTT. Known as the standard for IoT communications, MQTT is a lightweight publish/subscribe messaging transport that is ideal for connecting remote devices, often at edges of their globally distributed ecosystem of suppliers and stores, with a small code footprint and minimal network bandwidth.

Last year one of my colleague’s wrote an article called Real-Time Data Streaming IoT that breaks down the benefits of enabling data streaming between IT and operational systems as part of what some call Industry 4.0. If you’re already familiar with MQTT, you can learn how we support it in our documentation.

Here’s how information routed via MQTT and HTTP can make its way from sensors through edge devices to cloud services and on-premises applications:

Sophisticated Hierarchical Topics

Kafka utilizes a flat topic structure that doesn’t give you the flexibility you need. Kafka’s flat topic structure stores events across multiple partitions, which requires publishers and subscribers to be very specific about the topics they’re writing to or reading from. As your system scales, things can get out of hand. To get around that, you need to build lots of message handling and filtration into your applications, or create a dedicated filtering app that manages topics and streams. Either way you’re adding unnecessary overhead and making your applications and/or system as a whole more complex, rigid, and prone to failure.

PubSub+ supports topic hierarchy and wildcard filtering to give you the flexibility in application design that you need from a modern event-driven solution. It can make a difference that’s a like the difference between looking for information among thousands of files, or in a well organized folder structure. And when it comes to the distribution of data across geographically dispersed applications and devices, the difference is even bigger. With single and multi-level wildcards, your consuming applications can filter down from thousands of potential topics to receive only the data they need, which can save you money on WAN bandwidth, which I’ll get into below, and cloud egress charges.

To learn more watch this video:

Efficient Long Distance Data Sharing

As far as computer networks have come, it’s still a fact that distributing large volumes of information over long distances introduces some issues. For example, most systems distribute all information everywhere, regardless of which systems need it, even though wide area networks feature limited, and more expensive, bandwidth. In addition, they can lose information in transit, causing inconsistent information across your system.

PubSub+ lets you efficiently stream data across long-distance links with limited bandwidth in two ways. First, it can automatically compress data for transmission, and decompress it on the other end. Second, PubSub+ only sends information over network links when at least one system on the other side has subscribed to it via any of the sophisticated topics described above.

And to eliminate the risk of lost information, PubSub+ can buffer/replay data so disconnected and slow consumers get messages as soon as they can, and in the event of network failures or slowdowns information is dynamically rerouted via an alternate route.

You can learn more about multi-node routing in our documentation here.

Examples of Retailers Improving Cross-Organization Information Flow with PubSub+ Platform

The Schwarz Group

As one of the top retailers in the world, the Schwarz Group’s goal is to meet customer expectations and to act efficiently and sustainably in the face of growing demand in the market. Schwarz Group knew that to needed to re-architect their messaging and integration layer to support rapid innovation and growth, and to meet customer expectations for a more integrated shopping experience. That’s why they replaced their legacy messaging middleware with PubSub+ Platform, which is helping them accelerate time to market for new services, and improve agility so they can meet fast-changing customer needs at any time, even in times of seasonal bursts. Learn more here.

Luxury Good Maker

This European conglomerate produces and sells the world’s most luxurious jewelry, watches, leather goods, pens, firearms, clothing and accessories through its ownership of over a dozen maisons that make and sell products under a variety of brands. They wanted to modernize their IT architecture to facilitate real-time data distribution across a global network of maisons and associated retail outlets. The company used PubSub+ to build an event mesh that links applications running on-premises and in Amazon, Google and Alibaba clouds. This event mesh will facilitate the flow of information about events and transactions occurring across thousands of factories and stores, both physical and virtual. Learn more here.

Les Mousquetaires

Thanks to Solace’s technology, Les Mousquetaires has deployed over 450 new event workflows into production in just two years, successfully implementing EDA for all of the use cases they envisioned with very fast time-to-market. They have, for example, implemented new systems that have or will use AI to improve order accuracy, automatically disseminate product updates across their organization in real-time, and let them dynamically change produce prices across stores whenever they want. Learn more here.

The Architect's Guide to Real-Time RetailThe 6 steps necessary to architect a scalable, real-time, event-based ecosystem for retailers expanding their global, multichannel footprint.Download Now