New Micro-Integration for Snowflake Enables Real-time Data Ingestion with Snowpipe Streaming

Solace is excited to announce the general availability of a new self-contained micro-integration which enables real-time streaming from Solace Event Broker into Snowflake. This new capability both simplifies the process of data ingestion and reduces the latency of time-to-availability in the database.

What are Self-Contained Micro-Integrations?

Self-contained micro-integrations  are a relatively new and growing set of micro-integrations being offered by Solace. Rather than live within the message broker, these micro-integration are pre-built, packaged applications which are configuration driven with a collection of common features such as:

• Standard management and maintenance endpoints accessible over HTTP(s) and JMX
  • Health and performance metrics monitoring
  • Workflow administration and log access
• Common configuration options
  • Solace connectivity and options
  • Management endpoint control
• Built-in high-availability semantics
  • Standalone operation
  • Active/Standby
  • Active/Active*

This new family of micro-integrations, which are built on the Java Spring Cloud Stream framework, provides a perfect runtime for integrating the Snowflake Ingest SDK which powers Snowpipe Streaming.

Snowflake and the Snowpipe Streaming API

As Snowflake has grown beyond its roots as a data warehouse into a full-featured data cloud platform, the need for real-time data ingestion became self-apparent. Organizations which traditionally have relied upon file-based and batch-based database loading needed a simpler and faster approach to loading their source tables to feed downstream systems.

The Snowpipe Streaming API was created to address these concerns, and by using the Snowflake Ingest SDK, it is now possible to directly load data into Snowflake tables from any Java application with typical latencies in the range of seconds instead of minutes.

Using the New Snowflake Micro-Integration

Solace Micro-Integration for Snowflake is available as a container image or Java binary which can be run in any compatible host environment. Through a configuration file or Spring Cloud Config server, connection settings for the Solace broker and Snowflake instance are provided and workflows are specified.

A workflow is defined as mapping from a Solace queue to a Snowflake table, and each micro-integration instance can contain up to 20 such definitions. As messages are received from the broker they are inserted in-order into the destination table as rows in real-time.

A typical end-to-end architecture is pictured below:

As shown, the Solace broker and Snowflake micro-integration provide an on-ramp for events from a variety of sources into a real-time data platform provided by Snowflake. Publishers can generate events with dynamic, hierarchical metadata, and allow downstream consumers, workflows, and tables to define specific event types to filter and subscribe to. Once the selected data has been streamed into Snowflake, it is further processed, enriched, and merged with static reference data to provide complete models for dashboard, monitoring, and triggering additional events.

Use Cases and Benefits

The immediate benefit of Solace Micro-Integration for Snowflake is the ability to convert real-time events into real-time data, which are available for analysis and processing within seconds after they occur. Even for use-cases where low-latency is not required, by leveraging Snowpipe Streaming, the micro-integration has the additional benefit of reduced complexity and cost when compared to traditional file-based ingestion.

Some common uses for event streaming into Snowflake cloud are:

• Change Data Capture (CDC) and Replication: Generate data change events from a traditional database and stream them into Snowflake for near real-time synchronization of data stores.
• Real-Time Security Monitoring: Feed real time activities and transactions into Snowflake for query-driven fraud detection and alerting.
• Manufacturing and IoT Analytics: Process streaming sensor data to generate real-time views of production performance and fault monitoring.
• Advanced Retail Marketing: Analyze consumer behavior as it occurs to provide relevant recommendations and experiences by merging their present and historical interests.

Impact

The Solace and Snowflake teams are excited to see the use cases that are enabled by combining these two best in class technologies.

Snowpipe Streaming was built from the ground up to enable fast and simple ingestion of streaming data directly into Snowflake. The Solace Micro-Integration for Snowflake makes it simple to reliably bring event message data at scale with low cost, all within the desired low latency requirements. With this new integration, our joint customers are able to unlock more value by joining, transforming, and analyzing their Solace event stream data with the rest of their data in Snowflake’s data cloud.”

– Xin Huang, product manager, Snowpipe Streaming

The Solace Micro-Integration for Snowflake offers a transformative solution for organizations seeking to harness the power of real-time data. By seamlessly integrating Solace event streams with Snowflake’s data cloud, businesses gain unparalleled agility and insight into their operations. This integration unlocks the potential for instant data availability, enabling timely decision-making and actionable insights. By leveraging this integration, businesses can stay ahead in today’s data-driven landscape, driving innovation and competitive advantage.”

– Andrew MacKenzie, principal product manager, Solace

Try it Yourself!

Solace Micro-Integration for Snowflake is available in our Integration Hub, and we look forward to hearing what you think. We are already working on making this micro-integration even better with performance updates and additional schema handling modes and are excited to see where it goes!