Migrating Risk and Post-Trading Analytics to the Cloud

As a solutions architect at Solace, I work with a lot of capital markets customers, including large investment banks and leading hedge funds. Historically, these firms have been at the forefront of cutting-edge technology, focused on maximizing performance and resilience. For several decades, they have invested in large scale infrastructure for heavy workloads, especially real-time jobs.

For example, most large buy-side and sell-side firms have invested over the years in building grid compute infrastructure to handle computing-heavy jobs. One such job is risk analytics, where the firm performs compute jobs to gauge their exposure. While they used to be batch jobs, they eventually migrated to being daily jobs that would be executed at the end of the day. In the last few years, several firms have started migrating to more real-time jobs so they can maintain a more accurate and up-to-date representation of their risk.

One way to achieve that is with the help of the elastic compute provide by cloud providers such as AWS, GCP, and Azure. Instead of having a large on-prem grid compute infrastructure that only gets used certain times of the day, capital markets firms are starting to leverage elastic compute available via managed Kubernetes services such as EKS (AWS), GKE (GCP), and AKS (Azure).

With these managed services, firms don’t have to manage their own server farm and waste resources while they sit idle most of the time. Instead, they are charged for whenever resources are used via pay-as-you-go model. Additionally, firms are able to easily scale horizontally without worrying about procuring and deploying additional servers themselves. With such flexibility, capital markets firms have found it easy to migrate their risk and post-trade calculations to the cloud.

Just because these firms can migrate their calculations to the cloud, that doesn’t mean that the risk app is already on the cloud. In fact, many large firms still have most of their critical applications on-prem which means they require a flexible, performant, and reliable solution to bridge their on-prem environment to their cloud environment. That’s where we come in!

A typical deployment for a risk application to leverage elastic compute on cloud looks something like this:

The risk application is on-prem and sends requests to the cloud-based calc engine deployed on AKS, EKS, or GKE. However, the requests are important messages that need to be sent from an on-prem application to a cloud-native service. We do not want to lose the request/message due to any network issues and want to be able to handle large burst of requests. Similarly, we do not want to lose any replies/results that need to be sent back to the risk application once the computation job is finished.

To achieve that, several investment banks and hedge funds rely on Solace PubSub+ Event Broker’s guaranteed messaging quality of service. Solace brokers help applications send requests and receive responses without the risk of losing any messages, so risk applications can easily publish calculation requests to Solace and know the messages will end up in a queue for the calc engine to consume and process.

The architecture for such a deployment would look something like this:

Why not use a cloud native messaging service?

Many ask why they should use Solace when they could use one of the cloud-native messaging service that every cloud provider has, such as Google Cloud Platform’s Pub/Sub and AWS’s Simple Queueing Service.

There are two main reasons: the need to connect systems across hybrid cloud systems, and multi-cloud systems.

The Need for Hybrid Cloud

Most companies have a prominent on-premises infrastructure, so even as they embrace the cloud wherever possible, for the foreseeable future they will adopt a hybrid cloud approach that sees them integrating systems across both cloud and on-premises environments. As our CTO Shawn McAllister likes to say: “Hybrid cloud is an end-state; enterprises need to be thinking beyond transition.”

With hybrid cloud, it’s easier and simpler to rely on one messaging layer that can be easily deployed on-prem and on cloud. Additionally, it needs to be able to create an Event Mesh between the on-prem datacenter and cloud region so that on-prem applications can easily share data with cloud applications and vice versa. A cloud-native messaging solution cannot be extended to on-prem and that’s why Solace excels in hybrid-cloud usecases. Here is a sample deployment:

Last year my colleague Hari wrote a nice blog post about how Solace helps with the implementation of hybrid cloud systems, so if you’re interested, check that out.

The Need for Multi-Cloud Integration

Secondly, more and more capital markets firm are starting to adopt multi-cloud as their long-term strategy. With AI becoming more prominent and accessible, and each cloud provider having their own offering, firms are opening up to multi-cloud approach.

For example, a firm might use AWS for S3 and EKS, GCP for BigQuery and Azure for its AI models. Tomorrow, they might use AWS for its AI models and GCP for storage. By choosing Solace, with its cloud-agnostic approach and the ability to route messages to and from any cloud provider, firms let their teams cherry-pick the most suitable service for each use case.

For example, a common deployment is to store post-trade data in BigQuery and BigTable for further analytics for use cases such as Research and Transaction Cost Analysis.

In this diagram, data from on-prem application(s) is sent to a local broker and then forwarded to AWS for risk metrics to be computed. Once computed, those results can be sent back to the Risk application.

Additionally, they can also be easily sent to GCP and stored in BigQuery for further analysis via DataFlow (GCP’s managed service for easy integration). This fabric that has been constructed between the three environments to easily distribute real-time data is called an event mesh and is a crucial part of any hybrid-cloud or multi-cloud infrastructure in a modern data stack.

Conclusion

Wrapping up, more and more capital markets firms are migrating their usual computation-intensive jobs to cloud so they can leverage elastic and scalable compute offered by managed services such as AKS, EKS, and GKE. Additionally, these firms are starting to consider multi-cloud as an option to provide their teams with the flexibility of choosing the most suitable offering for their use case. To achieve both use cases, these firms are using Solace to build an event mesh that lets them send calc requests and computed results from one environment to another environment with zero message loss.