Idempotency in message handler
89 views
Skip to first unread message
james...@eccountancy.net
Sep 5, 2017, 12:05:20 PM9/5/17
to Particular Software
What is the best way to handle idempotency (and de-duplication) in my message handler?
In my example, I have a Payments endpoint that subscribes to OrderAccepted. It calls a BillCustomer() operation on a PaymentProvider API. It then publishes CustomerBilled as a consequence.
Does the idempotency include the published output event?
In this scenario my handler operation is not idempotent. If I call BillCustomer() again, the customer will be billed twice. So in order to de-duplicate that side effect I will not call the API again if I receive the OrderAccepted event for a second time. But if I do receive OrderAccepted twice, should I publish the CustomerBilled event again? Or should I silently swallow the input message the second time?
If I am genuinely republishing because downstream systems were broken and now fixed, silently swallowing the input and not publishing an output may cause choreography to stop.
Conversely, republishing the output message could cause floods of connected handlers to be reinvoked (even detect duplication and do nothing).
I am struggling to find opinions on this, so would be grateful for your thoughts.
Szymon Pobiega
Sep 5, 2017, 12:16:05 PM9/5/17
to particula...@googlegroups.com
Hi James
In general you handler cannot guarantee the idempotency of the whole solution if the BillCustomer() API is itself not idempotent. The call to the payment provider API can fail either before the cusomer is billed or after and the handler won't be able to tell what has happened. So you in case of BillCustomer() error you can either retry (risking double billing) or not (e.g. mark the order as requiring manual intervention) and risk not billing the customer at all.
In most cases when dealing with external APIs a client-generated ID is provided for ensuring end-to-end idempotency. Your handler would generate such an ID in a deterministic manner (i.e. derive it from the message ID or order ID) and pass it to the BillCustomer() API. This way you shall be able to safely retry BillCustomer() in case of failure and the payment provider will have to deduplicate the requests based on the ID provided.
Does this answer your question?
Szymon
--
You received this message because you are subscribed to the Google Groups "Particular Software" group.
To unsubscribe from this group and stop receiving emails from it, send an email to particularsoftware+unsub...@googlegroups.com.
To post to this group, send email to particularsoftware@googlegroups.com.
Visit this group at https://groups.google.com/group/particularsoftware.
To view this discussion on the web visit https://groups.google.com/d/msgid/particularsoftware/c0163c93-c035-4315-b806-fdb228a49501%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Have you tried NServiceBus V6 yet?
Learn how to scale applications with microservices and NServiceBus 6.
Join Udi Dahan's next Advanced Distributed Systems Design Course:
September 2017 Los Angeles, CA, USA
November 2017 Sydney, Australia
Andreas Ohlund
Sep 6, 2017, 2:35:21 AM9/6/17
to particula...@googlegroups.com
To add to what Szymon said in general you want to avoid mixing transactional and non transactional work on the same message pipeline. Ie try to split off the call to the payment provider by sending a new"BillCustomer" message. That way you are in full control over the recoverability settings for that message. Eg. if there is no way to make the call idempotent (like Szymon suggests) you can turn off all retries and send the message over to the error queue after just a single processing attempt to prevent double billing.
With this approach you could use a saga to monitor the status of the billing operation using timeouts. Eg. have a saga started when OrderAccepted arrives. Send the BillCustomer message and then set a timeout to deal with the case where you would not receive a "BillingSucessful" response. When you do receive the response the saga can then publish the OrderBilled event and complete the saga.
I gave a talk a few years back on this topic that might be of help:
Does this make any sense?
Cheers,
Andreas
On Tue, Sep 5, 2017 at 6:16 PM Szymon Pobiega <szymon....@particular.net> wrote:
Hi JamesIn general you handler cannot guarantee the idempotency of the whole solution if the BillCustomer() API is itself not idempotent. The call to the payment provider API can fail either before the cusomer is billed or after and the handler won't be able to tell what has happened. So you in case of BillCustomer() error you can either retry (risking double billing) or not (e.g. mark the order as requiring manual intervention) and risk not billing the customer at all.In most cases when dealing with external APIs a client-generated ID is provided for ensuring end-to-end idempotency. Your handler would generate such an ID in a deterministic manner (i.e. derive it from the message ID or order ID) and pass it to the BillCustomer() API. This way you shall be able to safely retry BillCustomer() in case of failure and the payment provider will have to deduplicate the requests based on the ID provided.Does this answer your question?Szymon
2017-09-05 18:04 GMT+02:00 <james...@eccountancy.net>:
What is the best way to handle idempotency (and de-duplication) in my message handler?In my example, I have a Payments endpoint that subscribes to OrderAccepted. It calls a BillCustomer() operation on a PaymentProvider API. It then publishes CustomerBilled as a consequence.Does the idempotency include the published output event?In this scenario my handler operation is not idempotent. If I call BillCustomer() again, the customer will be billed twice. So in order to de-duplicate that side effect I will not call the API again if I receive the OrderAccepted event for a second time. But if I do receive OrderAccepted twice, should I publish the CustomerBilled event again? Or should I silently swallow the input message the second time?If I am genuinely republishing because downstream systems were broken and now fixed, silently swallowing the input and not publishing an output may cause choreography to stop.Conversely, republishing the output message could cause floods of connected handlers to be reinvoked (even detect duplication and do nothing).I am struggling to find opinions on this, so would be grateful for your thoughts.
--
You received this message because you are subscribed to the Google Groups "Particular Software" group.
To unsubscribe from this group and stop receiving emails from it, send an email to particularsoftw...@googlegroups.com.
To post to this group, send email to particula...@googlegroups.com.
Visit this group at https://groups.google.com/group/particularsoftware.
To view this discussion on the web visit https://groups.google.com/d/msgid/particularsoftware/c0163c93-c035-4315-b806-fdb228a49501%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--Have you tried NServiceBus V6 yet?
Learn how to scale applications with microservices and NServiceBus 6.
Join Udi Dahan's next Advanced Distributed Systems Design Course:
September 2017 Los Angeles, CA, USA
November 2017 Sydney, Australia
--
You received this message because you are subscribed to the Google Groups "Particular Software" group.
To unsubscribe from this group and stop receiving emails from it, send an email to particularsoftw...@googlegroups.com.
To post to this group, send email to particula...@googlegroups.com.
Visit this group at https://groups.google.com/group/particularsoftware.
To view this discussion on the web visit https://groups.google.com/d/msgid/particularsoftware/CAE87YxouRd-itdByib_ZEBpBP538euBD%2Bv2%3DcMjEpr0gsA02vw%40mail.gmail.com.
james...@eccountancy.net
Sep 9, 2017, 6:46:09 AM9/9/17
to particula...@googlegroups.com
Thank you Szymon and Andreas for responding.
Can I rephrase the question slightly differently, using Andreas' slides as a reference point?
https://skillsmatter.com/skillscasts/2295-reliable-integrations-nservicebus
At 20:58 the slide deck (very quickly) shows a BookShipment command being sent to the service that integrates with Fedex.
It the booking was succesful, a ShipmentBooked event is published.
If we re-sent the BookShipment command, even in a scenerio where the previous call to Fedex was succesful, would we expect the ShipmentBooked event to be re-published?
The service handler may know that the previous call to Fedex succeeded, and if Fedex was not idempotent, it would not repeat the operation. It might be costly too. In that scenario it would de-duplicate that call to Fedex.
But would it republish ShipmentBooked?
FYI, in our scenario it is slightly different. Rather than command the Fedex integration service, effectively orchestrating, our Fedex integration would subscribe to OrderSettled, or a similar event. But the principle remains. If the Fedex service is re-triggered, it may or may not call Fedex depending upon Fedex's idempotency guarantee. But as a service, does it republish it's output?
In other words, Do we include the "output" message in our definition of idempotency when the integration service is seen from the outside? Does it's definition of idempotency include both ensuring 1. One delivery is booked with Fedex and 2. A ShipmentBooked event is published?
Or just the first action?
I'd appreciate your thoughts...
Can I rephrase the question slightly differently, using Andreas' slides as a reference point?
https://skillsmatter.com/skillscasts/2295-reliable-integrations-nservicebus
At 20:58 the slide deck (very quickly) shows a BookShipment command being sent to the service that integrates with Fedex.
It the booking was succesful, a ShipmentBooked event is published.
If we re-sent the BookShipment command, even in a scenerio where the previous call to Fedex was succesful, would we expect the ShipmentBooked event to be re-published?
The service handler may know that the previous call to Fedex succeeded, and if Fedex was not idempotent, it would not repeat the operation. It might be costly too. In that scenario it would de-duplicate that call to Fedex.
But would it republish ShipmentBooked?
FYI, in our scenario it is slightly different. Rather than command the Fedex integration service, effectively orchestrating, our Fedex integration would subscribe to OrderSettled, or a similar event. But the principle remains. If the Fedex service is re-triggered, it may or may not call Fedex depending upon Fedex's idempotency guarantee. But as a service, does it republish it's output?
In other words, Do we include the "output" message in our definition of idempotency when the integration service is seen from the outside? Does it's definition of idempotency include both ensuring 1. One delivery is booked with Fedex and 2. A ShipmentBooked event is published?
Or just the first action?
I'd appreciate your thoughts...
Szymon Pobiega
Sep 10, 2017, 11:33:10 PM9/10/17
to particula...@googlegroups.com
Hi James
The mathematical definition of idempotence talks about results of function application. In our world I prefer to use the term "side effects": if the handler is idempotent than the side effects of invoking it multiple times are the same as if it was invoked once. When we talk about re-publishing or re-sending outgoing messages the side effects depend both on the sender and the receiver.
Assuming the receiver of the event can't deduplicate messages, if you re-publish than for that receiver the side effects of your handler invocation will be different so the combo is not idempotent.
Assuming the receiver of the event deduplicates based on ID and you re-publish with different ID then the combo is still not idempotent because the receiver will treat the re-published event as a new one
Assuming the receiver of the event deduplicates based on message content/hash and you republish, the combo is idemponent
The moral is that the side effects of a handler depend on the things it sends messages to.
Now let me give you an example of NServiceBus's Outbox implementation. It stores the outgoing messages in the database and uses the same row for deduplication. It tries to minimize the risk of re-publishing messages by having a "IsDispatched" flag in that row. When a pair of duplicate messages arrives, if the first one succeeds then the second one see the messages has been dispatched and does not dispatch them again. That said, it can happen that there is a failure right before marking the row as dispatched and in that case the messages will be sent again. In general, no matter which solution you use, it is advisable to try to minimize the chances of generating downstream duplicates (re-publishing) but in certain failure modes you can't reliably check if you already sent so you need to err on the side of safety are send again.
Hope it helps,
Szymon
2017-09-09 12:46 GMT+02:00 <james...@eccountancy.net>:
Thank you Szymon and Andreas for responding.
Can I rephrase the question slightly differently, using Andreas' slides as a reference point?
https://skillsmatter.com/skillscasts/2295-reliable-integrations-nservicebus
At 20:58 the slide deck (very quickly) shows a BookShipment command being sent to the service that integrates with Fedex.
It the booking was succesful, a ShipmentBooked event is published.
If we re-sent the BookShipment command, even in a scenerio where the previous execution was succesful, would we expect the ShipmentBooked event to be re-published?
The service handler may know that the previous call to Fedex succeeded, and if Fedex was not idempotent, it would not repeat the operation. It might be costly too. In that scenario it would de-duplicate that call to Fedex.
But would it republish ShipmentBooked?
FYI, in our scenario it is slightly different. Rather than command the Fedex integration service, effectively orchestrating, our Fedex integration would subscribe to OrderSettled, or a similar event. But the principle remains. If the Fedex service is re-triggered, it may or may not call Fedex depending upon Fedex's idempotency guarantee. But as a service, does it republish it's output?
In other words, Do we include the "output" message in our definition of idempotency when the integration service is seen from the outside? Does it's definition of idempotency include both ensuring 1. One delivery is booked with Fedex and 2. A ShipmentBooked event is published?
Or just the first action?
I'd appreciate your thoughts...
On Wednesday, 6 September 2017 07:35:21 UTC+1, andreas.ohlund wrote:
To add to what Szymon said in general you want to avoid mixing transactional and non transactional work on the same message pipeline. Ie try to split off the call to the payment provider by sending a new"BillCustomer" message. That way you are in full control over the recoverability settings for that message. Eg. if there is no way to make the call idempotent (like Szymon suggests) you can turn off all retries and send the message over to the error queue after just a single processing attempt to prevent double billing.With this approach you could use a saga to monitor the status of the billing operation using timeouts. Eg. have a saga started when OrderAccepted arrives. Send the BillCustomer message and then set a timeout to deal with the case where you would not receive a "BillingSucessful" response. When you do receive the response the saga can then publish the OrderBilled event and complete the saga.I gave a talk a few years back on this topic that might be of help:Does this make any sense?Cheers,Andreas
On Tue, Sep 5, 2017 at 6:16 PM Szymon Pobiega <szymon....@particular.net> wrote:
Hi JamesIn general you handler cannot guarantee the idempotency of the whole solution if the BillCustomer() API is itself not idempotent. The call to the payment provider API can fail either before the cusomer is billed or after and the handler won't be able to tell what has happened. So you in case of BillCustomer() error you can either retry (risking double billing) or not (e.g. mark the order as requiring manual intervention) and risk not billing the customer at all.In most cases when dealing with external APIs a client-generated ID is provided for ensuring end-to-end idempotency. Your handler would generate such an ID in a deterministic manner (i.e. derive it from the message ID or order ID) and pass it to the BillCustomer() API. This way you shall be able to safely retry BillCustomer() in case of failure and the payment provider will have to deduplicate the requests based on the ID provided.Does this answer your question?Szymon
2017-09-05 18:04 GMT+02:00 <james...@eccountancy.net>:
What is the best way to handle idempotency (and de-duplication) in my message handler?In my example, I have a Payments endpoint that subscribes to OrderAccepted. It calls a BillCustomer() operation on a PaymentProvider API. It then publishes CustomerBilled as a consequence.Does the idempotency include the published output event?In this scenario my handler operation is not idempotent. If I call BillCustomer() again, the customer will be billed twice. So in order to de-duplicate that side effect I will not call the API again if I receive the OrderAccepted event for a second time. But if I do receive OrderAccepted twice, should I publish the CustomerBilled event again? Or should I silently swallow the input message the second time?If I am genuinely republishing because downstream systems were broken and now fixed, silently swallowing the input and not publishing an output may cause choreography to stop.Conversely, republishing the output message could cause floods of connected handlers to be reinvoked (even detect duplication and do nothing).I am struggling to find opinions on this, so would be grateful for your thoughts.
--
You received this message because you are subscribed to the Google Groups "Particular Software" group.
To unsubscribe from this group and stop receiving emails from it, send an email to particularsoftware+unsubscribe@googlegroups.com.
To post to this group, send email to particula...@googlegroups.com.
Visit this group at https://groups.google.com/group/particularsoftware.
To view this discussion on the web visit https://groups.google.com/d/msgid/particularsoftware/c0163c93-c035-4315-b806-fdb228a49501%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
----Have you tried NServiceBus V6 yet?
Learn how to scale applications with microservices and NServiceBus 6.
Join Udi Dahan's next Advanced Distributed Systems Design Course:
September 2017 Los Angeles, CA, USA
November 2017 Sydney, Australia
You received this message because you are subscribed to the Google Groups "Particular Software" group.
To unsubscribe from this group and stop receiving emails from it, send an email to particularsoftware+unsubscribe@googlegroups.com.
To post to this group, send email to particula...@googlegroups.com.
Visit this group at https://groups.google.com/group/particularsoftware.
To view this discussion on the web visit https://groups.google.com/d/msgid/particularsoftware/CAE87YxouRd-itdByib_ZEBpBP538euBD%2Bv2%3DcMjEpr0gsA02vw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "Particular Software" group.
To unsubscribe from this group and stop receiving emails from it, send an email to particularsoftware+unsub...@googlegroups.com.
To post to this group, send email to particularsoftware@googlegroups.com.
Visit this group at https://groups.google.com/group/particularsoftware.
To view this discussion on the web visit https://groups.google.com/d/msgid/particularsoftware/67e8663e-993d-4064-bf78-eb9362a51b4c%40googlegroups.com.
Udi Dahan
Sep 11, 2017, 9:49:01 AM9/11/17
to Particular Software
If you can know that the previous invocation was handled successfully, then the answer is no, don't publish again.
- Udi
Reply all
Reply to author
Forward
0 new messages