New Connector for Snowflake Enables Real-time Data Ingestion with Snowpipe Streaming

Solace is excited to announce the general availability of a new self-contained connector which enables real-time streaming from PubSub+ Event Broker into Snowflake. This new capability both simplifies the process of data ingestion and reduces the latency of time-to-availability in the database.

What are Self-Contained Connectors?

Self-contained connectors are a relatively new and growing set of connectors being offered by Solace. Rather than live within the message broker, these connectors are pre-built, packaged applications which are configuration driven with a collection of common features such as:

• Standard management and maintenance endpoints accessible over HTTP(s) and JMX
  • Health and performance metrics monitoring
  • Workflow administration and log access
• Common configuration options
  • Solace connectivity and options
  • Management endpoint control
• Built-in high-availability semantics
  • Standalone operation
  • Active/Standby
  • Active/Active*

This new family of connectors, which are built on the Java Spring Cloud Stream framework, provides a perfect runtime for integrating the Snowflake Ingest SDK which powers Snowpipe Streaming.

Snowflake and the Snowpipe Streaming API

As Snowflake has grown beyond its roots as a data warehouse into a full-featured data cloud platform, the need for real-time data ingestion became self-apparent. Organizations which traditionally have relied upon file-based and batch-based database loading needed a simpler and faster approach to loading their source tables to feed downstream systems.

The Snowpipe Streaming API was created to address these concerns, and by using the Snowflake Ingest SDK, it is now possible to directly load data into Snowflake tables from any Java application with typical latencies in the range of seconds instead of minutes.

Using the New Snowflake Connector

PubSub+ Connector for Snowflake is available as a container image or Java binary which can be run in any compatible host environment. Through a configuration file or Spring Cloud Config server, connection settings for the Solace broker and Snowflake instance are provided and workflows are specified.

A workflow is defined as mapping from a Solace queue to a Snowflake table, and each connector instance can contain up to 20 such definitions. As messages are received from the broker they are inserted in-order into the destination table as rows in real-time.

A typical end-to-end architecture is pictured below:

As shown, the Solace broker and Snowflake connector provide an on-ramp for events from a variety of sources into a real-time data platform provided by Snowflake. Publishers can generate events with dynamic, hierarchical metadata, and allow downstream consumers, workflows, and tables to define specific event types to filter and subscribe to. Once the selected data has been streamed into Snowflake, it is further processed, enriched, and merged with static reference data to provide complete models for dashboard, monitoring, and triggering additional events.

Use Cases and Benefits

The immediate benefit of PubSub+ Connector for Snowflake is the ability to convert real-time events into real-time data, which are available for analysis and processing within seconds after they occur. Even for use-cases where low-latency is not required, by leveraging Snowpipe Streaming, the connector has the additional benefit of reduced complexity and cost when compared to traditional file-based ingestion.

Some common uses for event streaming into Snowflake cloud are:

• Change Data Capture (CDC) and Replication: Generate data change events from a traditional database and stream them into Snowflake for near real-time synchronization of data stores.
• Real-Time Security Monitoring: Feed real time activities and transactions into Snowflake for query-driven fraud detection and alerting.
• Manufacturing and IoT Analytics: Process streaming sensor data to generate real-time views of production performance and fault monitoring.
• Advanced Retail Marketing: Analyze consumer behavior as it occurs to provide relevant recommendations and experiences by merging their present and historical interests.

Impact

The Solace and Snowflake teams are excited to see the use cases that are enabled by combining these two best in class technologies.

Snowpipe Streaming was built from the ground up to enable fast and simple ingestion of streaming data directly into Snowflake. The PubSub+ Connector for Snowflake makes it simple to reliably bring event message data at scale with low cost, all within the desired low latency requirements. With this new integration, our joint customers are able to unlock more value by joining, transforming, and analyzing their Solace event stream data with the rest of their data in Snowflake’s data cloud.”

– Xin Huang, product manager, Snowpipe Streaming

The PubSub+ Connector for Snowflake offers a transformative solution for organizations seeking to harness the power of real-time data. By seamlessly integrating Solace event streams with Snowflake’s data cloud, businesses gain unparalleled agility and insight into their operations. This integration unlocks the potential for instant data availability, enabling timely decision-making and actionable insights. By leveraging this integration, businesses can stay ahead in today’s data-driven landscape, driving innovation and competitive advantage.”

– Andrew MacKenzie, principal product manager, Solace

Try it Yourself!

PubSub+ Connector for Snowflake is available in our Integration Hub, and we look forward to hearing what you think. We are already working on making this connector even better with performance updates and additional schema handling modes and are excited to see where it goes!