Why Developers Need an Event Portal; Creating Applications that Disseminate Real-Time Data

As developers, we are asked to create software that solves hard problems in a way that’s scalable, secure, and can be modified as needs evolve – all with deadlines expressed in terms of how long ago they wish they’d had this. Does “We need this yesterday!” ring a bell? Things change so quickly these days, in business and the world at large, that the applications we deliver need to help people achieve positive outcomes very quickly, based on the latest information. That’s where event-driven architecture comes in. But event-driven architecture presents developers with some interesting challenges throughout the development process. These challenges include:

1. How to capture the design and collaborate with the architects
2. How to implement the design in a simple and consistent way
3. How to understand the flow of information from an operational perspective in order to alert and solve problems

Letting applications produce, consume, and act on information that’s sent to and fro based on user actions and changes in condition (i.e. “events”) instead of periodic request/reply polling, makes it possible to identify and act on opportunities as they emerge, in real-time, instead of looking at them in the rear-view mirror.

I’d like to explain the three challenges above in terms of how they’re addressed today, and how they can be dealt with now, thanks to our new product PubSub+ Event Portal.

I find that using an example is the best way to expose challenges and solutions, so I’ll frame my explanation in terms of a project some colleagues and I took on last week. Simply put, we used PubSub+ Event Portal to create event streams related to the COVID-19 pandemic so those looking to report on, track, or help with the pandemic could do so with access to real-time information.

Creating an Application to Provide Real-Time COVID-19 Data

Lots of developers out there are looking to do their part by helping government agencies, journalists, and – most importantly – scientists and public health authorities process and analyze data related to the COVID-19 pandemic. We decided to help them do so by making it easier to overcome the first challenge each of them faces – how to access the data and understand what it looks like.

Historically, architects have used tools like Confluence and PowerPoint, etc. to describe and document the design of events, applications, and data flow we’re being asked to implement. This can be challenging for a variety of reasons, leaving us asking questions like:

• Where do I connect?
• What are the payload schemas so I can serialize/deserialize the data?
• Where do I send the output of my newly created application/microservice?
• If I’m using an event generated outside of my team, who do I contact for more info?
• What do I do if the documentation gets stale?

API Management solutions like Mulesoft provide a great experience for synchronous RESTful interactions, but there isn’t an equivalent for event-driven or streaming interactions. Well, there wasn’t, anyway. We at Solace felt your pain and have come up with a solution that overcomes some of the challenges described above so developers like us can get the answers we need. Our product PubSub+ Event Portal is the first solution that lets you collaboratively create and manage events and event-driven applications.

Collaborative Design and Documentation

Getting back to the COVID-19 project, our first step was to document the design. Our Architect colleague (Jonathan Schabowsky, who wrote about his experiences here), kicked off the project by identifying the data source we would access (Johns Hopkins CSSE) and the applications we would need to acquire the raw data, look for changes, and publish smaller, more fine-grained events that represent changes to regions or cases so the community can consume just the state changes they‘re interested in.

They design the system they envisioned with Solace PubSub+ Event Portal so we, as developers, could understand and make comments. For example, we had some interesting discussions around how the topic hierarchy would work and how to apply best practices. In addition, we collectively decided to split up the logic into different Spring Boot microservices so they were more purpose-specific and manageable. All these changes were decided upon in Slack meetings, and updates were made in the event portal. It was awesome to be able to collaborate, make updates, and be in sync before we even started to write code.

The architecture was kept up to date in the event portal as we implemented it so we could easily provide the COVID-19 data feeds to the technical community. Collaboration and design documentation: Check!

Simple, Consistent Development

Moving beyond collaboration and documentation, one of the biggest pains I experience as a developer advocate at Solace is taking the design, topic hierarchy, asynchronous API (JMS, Solace, Spring Cloud Stream, Paho, etc.) and consistently developing applications.

These problems don’t come up with REST because OpenAPI provides a machine-readable API specification that lets code generators create a client (API consumer) or server (API provider) that adheres to the spec. This ensures consistent code, conformance to the design, and a simple development experience. Once again, the event-driven world hasn’t had anything like this… until now!

AsyncAPI is a machine-readable specification for event-driven applications that also provides code generators. We at Solace recently contributed a Spring Cloud Stream code generator for AsyncAPI and PubSub+ Event Portal supports the exportation of AsyncAPI files that represent the asynchronous application interfaces.

Going back to the usage, our architect designed the event-driven interactions within our event portal and assigned us different microservices to implement. Once we exported the AsyncAPI file for each application, we ran the code generator, reviewed the data models, made small tweaks, and added the business logic (like splitting the data by cases or regions) we had running.

Consistent and scalable microservices within minutes –a huge step forward for developers!

What’s going on with my application?

It’s not all sunshine and roses though, and as the philosophy of DevOps has become the norm we as developers increasingly own the applications throughout their lifecycle. Not only do we design and implement applications, but we also have a hand in deploying, monitoring, and managing them. This includes troubleshooting and resolving problems when the inevitable rainy day occurs. Although we try to avoid bugs, they always seem to show up eventually. The challenge is that when problems pop up we need to be able to efficiently identify the cause and push out a fix.

This has been a problem with event driven architecture for ages. The decoupling of applications, while it provides many advantages, makes it hard to trace the flow of events through the system. All we hear is, “Where did my message go?” and we have to start tracking and tracing to figure out where the “hose” got kinked – application failed, data parsing issue, what have you.

So, it’s after hours and you’re hunting down that pesky design document you last reviewed 6 months ago… #badmoment. You find it, realize it has not been updated, and wonder if it’s even up to date: #worsemoment! The point is, we can now use PubSub+ Event Portal to understand the implementation, know where to look for problems, and diagnose them without the stress and anxiety of having to “reverse engineer” the implementation.

Conclusion

I get that sometimes it feels like you could add another tool to your development environment every day, but if you’re developing event-driven applications development, Solace PubSub+ Event Portal is a no brainer. Seeing as how it helps with documentation (which isn’t personally my favorite part of being a developer), automates code generation, and lets you see what’s going on in the production environment, the benefits for developers like you and me are enormous. Take it for a spin on your next event-driven application project and see for yourself! I’m confident you’ll be glad you did.

From everyone at Solace, we would like to thank everyone out there working to alert and inform the public during this pandemic. You are truly our heroes!

If there is a project or event-driven application you have in mind, we encourage you to try PubSub+ Event Portal FREE for 30 days. We’d love to see what you can do with this toolset and are eager to receive any feedback you may have on our product.