Technical Look at Using Solace as a Channel for Apache Flume
In a previous blog I explained what the Solace channel for Apache Flume was and how it could be used to streamline data workflow from enterprise message bus to big data infrastructure and vice-versa. In this blog I will describe how the Solace channel works and what are the technical advantages compared to memory or file channels.
Solace Flume channel receiving Flume Events from Source.
The Flume Source interacts with the Flume channel with 4 methods calls:
createTransaction()
, doPut(Event)
and finally doCommit()
or doRollback()
.
The Flume Channel uses the Solace message router session based transactions to implement the channel transactions. For more information on Solace session based transactions see Using Local Transactions
If the channel receives doPut(Flume.Event)
it takes the Event and transposes it into a Solace message. This is done by taking the Event header name value pairs and placing them into a Solace message header map, then taking the Events body and writing it as a byte buffer into the body of the Solace message. … Read the rest