How a Sidecar Proxy Can Rule your Event Mesh

As any parent knows, you need to give children the freedom to grow and learn but must also put in place boundaries to keep them safe. The same applies to an event mesh, and a sidecar proxy can be the responsible adult that establishes and enforces firm but fair rules for everyone using your event mesh. It sits alongside your application, intercepts data flowing to and from it, and ensure that microservices are following the rules.

But why does an event mesh need rules? Why not have a dynamic, free-spirited “playground on a weekend”-style event mesh? The answer lies with cross-cutting concerns and their effect on your ability to get stuff done quickly while not getting sued.

“Gonna Get Sued” Enterprise Policies for Event Mesh

As architects decompose applications into microservices, the responsibility for developing business functionality gets assigned to multiple teams. That freedom can feel like a breath of fresh air to developers used to working in lockstep with the rest of the monolith, much like kids getting the first glimpse of the playground.

But there are still important policies that need to be in place, the first category of which I’ll call “gonna get sued” enterprise policies. These enterprise policies are required by law and frequently carry stiff financial penalties if not enforced.

• Access control policies that govern which people or programs can produce or consume what information.
• Jurisdictional agreements (like GDPR) that regulate the transmission of information across geographies and political boundaries, i.e. states, countries and regions.
• Confidentiality policies that define which information about a person can be distributed where. For example, HIPAA’s policies around personally identifiable information (PII) and protected health information (PHI).
• Encryption rules that set expectations around securing information as it moves through and potentially beyond an enterprise. Although modern event brokers encrypt the wireline connection between brokers and clients with TLS, they don’t usually encrypt an event’s payload so it can be visible while sitting on a queue, or in application logs.

Some event brokers try to address these requirements using access control lists (ACLs) and well-structured topic taxonomies, but legal requirements don’t always apply to an entire event, so enforcing policies at the event level can mean throwing away valuable information. While a person’s name and other identifying information may not be able to move between geographies, non-identified data could move freely, and be extremely valuable when aggregated. If a hotel contracts with a housekeeping service, the service shouldn’t see the name of the guest checking out but should be notified which room is ready for cleaning.

 “Not Gonna Get Sued” Enterprise Policies

Other enterprise policies govern things that are less likely to land you in court but can affect a developer’s ability to get stuff done quickly and focus on business problems, not boilerplate code.

• Data Quality: Perhaps the most important aspect of data policy is schema validation to ensure that producers and consumer can trust the information available on the event mesh. Traditional message brokers let producers send freeform payloads, regardless of what downstream consumers expect, so . In turn, every consumers need to implement error handling to deal with faulty data.
• Metadata: One of the challenges of event-driven systems is that information arrives without a standard description of who, what, where, when and how the event was produced. For consumers, that context is crucial for understanding how to effectively process an event, especially if it comes from outside an organization’s boundaries. Standards like CloudEvents help consumers to “fill in the blanks” but can’t help if they aren’t consistently present.
• Observability: The need to observe events has grown with the complexity of interactions between events. Sometimes a single business outcome requires dozens of microservices. Enforcing the inclusion of observability information, like an OpenTelemetry trace id, for example, can give you great insight into how things are working, especially when things break down.
• Topics: Enterprises spend enormous energy defining a topic taxonomy that captures the nuances of their business. can make or break a system. Mandating that microservices follow topic taxonomy policies at runtime prevents messages from being misrouted.

Cross Cutting Concerns and Why Event Brokers Struggle with Them

All these policies and standards need to be enforced consistently across environments, programming languages, geographies, departments and divisions, cloud providers and even the personal preferences and habits of individual developers.

That introduces a challenge that architects fancier than me call “cross-cutting concerns,”, aspects of an enterprise system that affect multiple components and which require external dependencies. An example of a cross cutting concern would be logging. If you’re developing an application that deals with inventory, aggregation of inventory data is the core concern, while logging the information to a database for archival or an authentication system to control who can access it would be cross-cutting concerns.  Addressing cross-cutting concerns in each individual application can lead to duplication of code, excessive dependencies between applications, or both.

Event-driven architecture struggles with cross-cutting concerns. The evolution of EDA from legacy message brokers shoulders much of the blame: back in the day, message brokers ignored the payload of messages and didn’t validate much else. Instead, a message broker’s mission was to route messages as fast as possible based on metadata, namely a topic that described what the message was about. If you could get access to the message broker, you could send and receive pretty much anything.

With the advent of event-driven data mesh, flow architectures and microservices using modern event brokers for connectivity, architects need to address the “What me worry?” aspect of event brokers. So how to do it?

The Wrong Way to Address Cross-Cutting Concerns

One way you can try to address crosscutting concerns is by writing down policies and trying to enforce those policies manually. Case in point: code reviews. Code reviews are a nightmare for both the developer and the reviewer because: they’re highly subjective, time consuming and inconsistent. Only slightly better are automated static code reviews, where scripts try to enforce rules embedded in application code during commits.

EDA experts have pointed out the need for true automation. For example, Zhamak Dehghani, who came up with the idea for and coined the term “dash mesh” says, “Data Mesh governance heavily relies on the automation of governance concerns for a consistent, connected and trustworthy experience using the domains’ data products.”

Hop in the Sidecar

The best strategy for cross-cutting application concerns is to not try to address them as part of the application. Putting enterprise policies outside the control of the domain teams during development allows for consistency across domains during deployment, and ensures that policies are enforced and can’t be bypassed at runtime.

Previously, creating an architectural pattern like this required deep knowledge of underlying infrastructure, perhaps writing custom exits for your IBM MQ Message broker. However, the ubiquity of Kubernetes as an application deployment strategy makes it much easier.

In a nutshell:

• A Kubernetes pod holds one to many containers (usually Docker containers).
• One of those can be a sidecar, which through configuration, is the only container allowed to talk with the event broker.
• In turn, actual implementation Services (like Service A and Service B) are only allowed to communicate with the sidecar.
• This sequence makes the sidecar “non-bypassable”. Events entered in the event mesh must comply with enterprise policies.

Figure 1: Sidecar Diagram courtesy of CodeOpinion

(If you find all of that interesting, go watch Derek Comartin’s brilliant explanation of all things sidecar.)

What does the Sidecar do? Well, basically, anything you can imagine! But in this scenario, it oversees enforcement of all the gonna-get-sued and not-gonna-get-sued policies that I described earlier. From its prime position just outside the event mesh, a sidecar proxy can reject events that don’t conform with policy–or perhaps modify them so that they do.

Even better, because a sidecar is a separate process, it’s isolated but still physically close to the service, so communication between them is fast. That isolation means policies can’t be changed by an application team, and the implementation of the sidecar also can be independent of the development language of the main application logic.

Conclusion

Event brokers aren’t always mature enough to support every use case and associated cross-cutting concerns. To address those gaps, a sidecar proxy acts as a non-bypassable extension of the event broker, allowing for much greater flexibility and agility. Architects would be wise to gather lessons learned from the synchronous world and products like Istio, and think about ways to integrate them into their event-driven systems.