PubSub+ Message Handling Features: Dead Message Queues

In the first blog in this series, I explained the concept of message expiry, or message-TTL. Here I’ll look at dead message queues (DMQs), which assumes you have background knowledge on Solace endpoints and Solace persistent messaging.

What is a Dead Message Queue?

A dead message queue, sometimes referred to as a dead letter queue, is an endpoint to store messages that have lost delivery eligibility in their original endpoint, and stores messages for special processing or delayed consumption.

You might be wondering how a message can lose delivery eligibility, especially when it is originally published as persistent. There are two ways messages can be deemed ineligible for delivery:

1.  Message Expiry: The first post in this series goes into more detail on message expiry, but as a summary it is when a message is published to an endpoint with a timer, or time-to-live. When this timer expires, the message is removed from the original endpoint and can be deleted or can be moved to a DMQ.
2. Poison Message: The second reason a message can lose delivery eligibility is if it is redelivered from an endpoint to a consumer too many times. This scenario could occur when a consumer is unable to process a message and crashes before sending an application acknowledgement. When that application reconnects to the endpoint, it will receive the same message and would be stuck in a loop. In this case, the message is referred to as a poison message. To ensure an application does not get stuck in this loop, an endpoint can be set to redeliver a message a maximum number of times with the setting ‘Maximum Redelivery Count’. Once exceeded, the message will be removed from the endpoint or can be moved to a DMQ.
   

How do I use a Dead Message Queue with Solace PubSub+ Event Broker?

A DMQ on Solace PubSub+ Event Broker is configured as a normal queue and can be created in the same manner on the Queue page of PubSub+ Broker Manager.

Once created, the DMQ is configured on the original endpoint in the ‘DMQ Name’ field.

Note that messages will not be moved into a DMQ unless they are published with the flag ‘DMQ-Eligible’. Without this set, the messages will be deleted from the original queue when they lose delivery eligibility.

That is an important point that bears repeating: Messages must be DMQ-Eligible!

 If you want messages that expire to go to a DMQ, you need:

• Queue with ‘Respect-TTL’ enabled, a DMQ configured, and optionally a ‘Maximum-TTL’ specified
• Published messages can have a ‘Message-TTL’ and must be ‘DMQ-Eligible’

If you want messages that reach the maximum redelivery count to go to a DMQ, you need:

• Queue with ‘Maximum Redelivery Count’ set (Redeliver Forever disabled), and a DMQ configured
• Published messages must be ‘DMQ-Eligible’

When a message moves from an endpoint to its DMQ, the below highlighted statistics will reflect this.

Why would I use a DMQ?

A DMQ is useful for special processing when messages are unable to be consumed on their original endpoint. This could be for expedited processing of orders, or the DMQ consumer may process the message in a different way than the original consumer.

Take for example an order management system that must process orders in a timely manner. For orders that are not processed in the desired amount of time, they could be expired to a dead message queue for expedited processing.

Another good example is error handling: a message that can’t be consumed from the original queue is moved to a DMQ to be inspected and processed.

Conclusion

I hope I’ve helped you feel more comfortable with what DMQs are and how to use them. If you found this post useful, you might want to check out the message properties section of our docs, and our PubSub+ for Developers page for more information. If you have any questions about queues and topic endpoints, try posting them to the Solace Developer Community!

How to disable TTL (Time to live) using solace-pubsubplus Python libraryLearn how to publish messages with TTL turned off.Read the Community Post

Leah Robert

I have worked at Solace as a Customer Support Engineer, and as a Global Training Professional. I help customers to migrate from a siloed, monolithic architecture to an Event Driven Architecture. Solace PubSub+ enables the movement of data in an asynchronous manner, allowing distributed applications to seamlessly connect over any protocol.

See all posts written by Leah Robert