In my role I have the privilege of talking with all kinds of IT practitioners and executives representing businesses around the world, and have picked up on a theme that transcends industry, company size and technology stack. It’s not that the COVID pandemic changed everything, no newsflash there, but that it did so in a way that’s changed the way consumers expect to have experiences with both digital and physical products.
In addition, the pandemic left in its wake an enormous amount of technical debt that businesses are struggling with. Take for example, grocery stores – before COVID most businesses focused solely on in-store, non-digital experience. COVID forced business to become online retailers overnight, integrating their storefronts with a variety of hybrid delivery, pickup and return services. Restaurants, hospitality and healthcare all experienced similar transformations of experience, making the need for real-time event driven architecture and streaming an imperative for IT groups in all industries.
Fast forward to today, and you can see that businesses that made the necessary digital experience changes but are now faced with two new problems at the same time:
The challenge faced is that we must repair and heal the fallout that came from the pandemic, all while doing less as companies cut IT budgets while expecting the kind of innovation they’ve seen in recent years.
Remember how I mentioned that the use of real-time event-driven architecture and streaming with Kafka and Pulsar blossomed with the transformation in customer experience? It turns out that has created a new opportunity for IT and businesses alike. Most of the events and data streams that exist today are used to solve specific real-time use cases. I have found most of these interactions are real-time application to application integrations. Thus, the need was real-time, asynchronous processing between two applications. An example of this would be an online order needing to be fulfilled within a grocery store, an order being delivered etc. But the power of event-driven architecture and streaming is that it natively was built for one-to-many integrations, and you essentially get that at runtime for free!
So what’s the problem and what’s the opportunity you may ask? Here are the problems, each paired within an opportunity you can seize with Event Portal for Kafka!
If you don’t know about it, you cannot reuse it. So, when all the architects were developing these real-time data streams, and generally getting things done, they did not spend time documenting what they were doing let alone exposing information about the event streams. They were under crazy time crunches and expected to deliver trailblazing business capabilities. Sure, maybe it got documented somewhere in a confluent page, draw.io diagram for a technical review, but not in any sort of social aspect, where the business and IT could discover event streams and reuse them in new, novel ways.
The parallel universe to all this is API Management which we all know and love. Before this, the enterprise was littered with RESTful APIs that were unknown to the general Business and IT Populus. The solution was to use a standard like OpenAPI to describe these services, import them into a API Management Solution that has a developer portal, and expose the APIs to internal and external parties.
Today, event management is starting to enter the fold. We now have AsyncAPI specs that enable architects and developers to document applications, import them into an Event Management Solution that integrates with developer portals and expose event streams to internal and external parties. But this time, it’s all about real-time data, vs APIs that query entity state. This is important as Gartner and others believe data is the most valuable at the moment it was produced. Thus opportunity #1 is the exposure of the most valuable data in your business. It can even create new value streams for your business especially if you think about providing a third-party marketplace for your event streams.
Using a Streaming Platform is far from free as you know. Most think exclusively about licensing and operationalization costs, but what about developer costs? Your developers are very highly skilled and costly resources that cannot be idle or non-productive. Consider the activities required today to create a new event stream from existing stream data coming from another team in the business that uses Kafka:
What these activities display is that each application and the process therein is bespoke and suffers from a lack of specialized tooling and process automation.
Again, taking lessons from the API Management world, imagine if we could make the experience of building event streaming applications as easy as writing RESTful Services or RESTful clients.
This would save countless hours of wasted developer time, while also providing tangible benefits to documentation, consistency, and time to market.
Indefinable Costs such as security and data breaches, failures/downtime of applications and project delays are nearly impossible predict. These costs can become huge depending on the business and the criticality of the data and use case. Consider the following two problems:
All these challenges are inviting regulatory fines, lawsuits, and customer lack of trust. The challenge is that all of this takes time and effort especially when you don’t have a concise view of what’s going on AND automation that ensures least access.
These scenarios are real, and they happen. I have the battle scars to prove it. The cost of these problems can easily mount as you continue to enhance and evolve your applications in response to new and changing business requirements. Can you continue to afford delayed time to value simply because you cannot easily lifecycle and change manage your real-time event streams and applications?
With the proper event management strategy, these challenges can be overcome while enhancing time to market and agility. This strategy entails the following key steps:
By doing these steps it will reduce your risk of outages and failures (and associated costly troubleshooting and hot bug fixing) while increasing he speed in which you deliver value to your customers. Remember, each day a business capability is running in production will add to your top and bottom line.
If you want to seize upon any of the aforementioned opportunities, you will have to decide if you want to implement the tooling yourself our purchase an event management tool. I am pretty sure your business is not to create the worlds best event management Tool, but rather to provide the best digital experiences for your customers. To that end, I would suggest you check out Solace PubSub+ Event Portal which enables all the opportunities presented, with minimal capital investment. The gains in productivity as well as the ability to provide new, real-time digital experiences will enable you to do more with less.
As Solace’s Field CTO, Jonathan helps companies understand how they can capitalize on the use of event-driven architecture to make the most of their microservices, and deploy event-driven applications into platform-as-a-services (PaaS) environments running in cloud and on-prem environments. He is an expert at architecting large-scale, mission critical enterprise systems, with over a decade of experience designing, building and managing them in domains such as air traffic management (FAA), satellite ground systems (GOES-R), and healthcare.
Based on that experience with the practical application of EDA and messaging technologies, and some painful lessons learned along the way, Jonathan conceived and has helped spearhead Solace’s efforts to create powerful new tools that help companies more easily manage enterprise-scale event-driven systems, including the company’s new event management product: PubSub+ Event Portal.
Jonathan is highly regarded as a speaker on the subject of event-driven architecture, having given presentations as part of SpringOne, Kafka Summit, and API Specs conferences. Jonathan holds a BS Computer Science, Florida State University, and in his spare time he enjoys spending time with his family and skiing the world-class slopes of Utah where he lives.
Subscribe to Our Blog
Get the latest trends, solutions, and insights into the event-driven future every week.
Thanks for subscribing.