Prevent logging of first and second-level retries unless both fail

50 views
Skip to first unread message

Neil Barnwell

unread,
Sep 13, 2017, 7:01:08 AM9/13/17
to particula...@googlegroups.com
Product name: NSB
Version: v5

I'm relying on first and second-level retries to solve messaging ordering issues (particular when two instances of an endpoint share a single queue). In my case, a "OrderAllocatedToWarehouse" event may be picked up before the actual "OrderPlaced" event. I allow it to fail when looking up the order, and let NServiceBus take over from there.

The trouble is, the logging output is INSANE. Our logging (to textfiles) outputs the details of the exception, including the stack trace 5 * 5 times for each message. I have log files for just a few hours of this normal, deliberate behaviour, that are 4Gb in size, full of stuff I don't care about because I expect it. If a message fails entirely, then I'd want THAT in the log, but not all the "business as usual" retries.

Is there some way I can configure FLR/SLR so they don't log exceptions?

Cheers,

Neil.

Andreas Ohlund

unread,
Sep 13, 2017, 8:37:47 AM9/13/17
to Particular Software
Hi Neal!

There is no way to turn the logging off but depending on the logging library you use you should be able to filter out the logs for FLR and SLR using something similar to


Does this help?

Cheers,

Andreas

On Wed, Sep 13, 2017 at 1:01 PM Neil Barnwell <neilba...@gmail.com> wrote:
Product name: NSB
Version: v5

I'm relying on first and second-level retries to solve messaging ordering issues (particular when two endpoints share a queue). In my case, a "OrderAllocatedToWarehouse" event may be picked up before the actual "OrderPlaced" event. I allow it to fail when looking up the order, and let NServiceBus take over from there.

The trouble is, the logging output is INSANE. Our logging (to textfiles) outputs the details of the exception, including the stack trace 5 * 5 times for each message. I have log files for just a few hours of this normal, deliberate behaviour, that are 4Gb in size, full of stuff I don't care about because I expect it. If a message fails entirely, then I'd want THAT in the log, but not all the "business as usual" retries.

Is there some way I can configure FLR/SLR so they don't log exceptions?

Cheers,

Neil.

--
You received this message because you are subscribed to the Google Groups "Particular Software" group.
To unsubscribe from this group and stop receiving emails from it, send an email to particularsoftw...@googlegroups.com.
To post to this group, send email to particula...@googlegroups.com.
Visit this group at https://groups.google.com/group/particularsoftware.
To view this discussion on the web visit https://groups.google.com/d/msgid/particularsoftware/a3f7ad3c-f0ee-4a08-854e-fdb05122c9d6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Neil Barnwell

unread,
Sep 13, 2017, 8:49:59 AM9/13/17
to Particular Software
It doesn't really help, unfortunately. There's nothing to filter on, really.

Maybe it's worth thinking about as a feature? FLR/SLR are really BAU, so you only really need log output if it genuinely fails to process a message (i.e. forwards it to the error queue). Obviously this is separate from any metrics about how much activity FLR/SLR is doing, as it'd be useful to see that there is lots of retrying going on. One could always have the intermediate failures logged on a different log level or something.

For now I'm trying to come up with a way to solve my message ordering problem with sagas. This is more complex, not to mention the impact of saga persistence etc, but I don't see a way around it.



On Wednesday, 13 September 2017 13:37:47 UTC+1, andreas.ohlund wrote:
Hi Neal!

There is no way to turn the logging off but depending on the logging library you use you should be able to filter out the logs for FLR and SLR using something similar to


Does this help?

Cheers,

Andreas

On Wed, Sep 13, 2017 at 1:01 PM Neil Barnwell <neilba...@gmail.com> wrote:
Product name: NSB
Version: v5

I'm relying on first and second-level retries to solve messaging ordering issues (particular when two endpoints share a queue). In my case, a "OrderAllocatedToWarehouse" event may be picked up before the actual "OrderPlaced" event. I allow it to fail when looking up the order, and let NServiceBus take over from there.

The trouble is, the logging output is INSANE. Our logging (to textfiles) outputs the details of the exception, including the stack trace 5 * 5 times for each message. I have log files for just a few hours of this normal, deliberate behaviour, that are 4Gb in size, full of stuff I don't care about because I expect it. If a message fails entirely, then I'd want THAT in the log, but not all the "business as usual" retries.

Is there some way I can configure FLR/SLR so they don't log exceptions?

Cheers,

Neil.

--
You received this message because you are subscribed to the Google Groups "Particular Software" group.
To unsubscribe from this group and stop receiving emails from it, send an email to particularsoftware+unsub...@googlegroups.com.

Andreas Ohlund

unread,
Sep 13, 2017, 9:15:24 AM9/13/17
to Particular Software

>Maybe it's worth thinking about as a feature?

As of v6 we are logging FLR as INFO, SLR as WARN and moving to error as ERROR


Is that what you are after?

Cheers,

Andreas

On Wed, Sep 13, 2017 at 2:50 PM Neil Barnwell <neilba...@gmail.com> wrote:
It doesn't really help, unfortunately. There's nothing to filter on, really.

Maybe it's worth thinking about as a feature? FLR/SLR are really BAU, so you only really need log output if it genuinely fails to process a message (i.e. forwards it to the error queue). Obviously this is separate from any metrics about how much activity FLR/SLR is doing, as it'd be useful to see that there is lots of retrying going on. One could always have the intermediate failures logged on a different log level or something.

For now I'm trying to come up with a way to solve my message ordering problem with sagas. This is more complex, not to mention the impact of saga persistence etc, but I don't see a way around it.



On Wednesday, 13 September 2017 13:37:47 UTC+1, andreas.ohlund wrote:
Hi Neal!

There is no way to turn the logging off but depending on the logging library you use you should be able to filter out the logs for FLR and SLR using something similar to


Does this help?

Cheers,

Andreas

On Wed, Sep 13, 2017 at 1:01 PM Neil Barnwell <neilba...@gmail.com> wrote:
Product name: NSB
Version: v5

I'm relying on first and second-level retries to solve messaging ordering issues (particular when two endpoints share a queue). In my case, a "OrderAllocatedToWarehouse" event may be picked up before the actual "OrderPlaced" event. I allow it to fail when looking up the order, and let NServiceBus take over from there.

The trouble is, the logging output is INSANE. Our logging (to textfiles) outputs the details of the exception, including the stack trace 5 * 5 times for each message. I have log files for just a few hours of this normal, deliberate behaviour, that are 4Gb in size, full of stuff I don't care about because I expect it. If a message fails entirely, then I'd want THAT in the log, but not all the "business as usual" retries.

Is there some way I can configure FLR/SLR so they don't log exceptions?

Cheers,

Neil.

--
You received this message because you are subscribed to the Google Groups "Particular Software" group.
To unsubscribe from this group and stop receiving emails from it, send an email to particularsoftw...@googlegroups.com.
To post to this group, send email to particula...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Particular Software" group.
To unsubscribe from this group and stop receiving emails from it, send an email to particularsoftw...@googlegroups.com.

To post to this group, send email to particula...@googlegroups.com.
Visit this group at https://groups.google.com/group/particularsoftware.
To view this discussion on the web visit https://groups.google.com/d/msgid/particularsoftware/0d60fddc-18ca-4c7f-808a-d23548beb977%40googlegroups.com.

Neil Barnwell

unread,
Sep 13, 2017, 11:59:02 AM9/13/17
to Particular Software
I'm not 100% sure how it's hooked together; there's a bit of an abstraction in there so that the logging works the same way as the rest of our stuff. Maybe I'll roll my sleeves up and see if there's something I can do.

As for the v6 comment - if the actual full exception detail is hidden unless it's moved to the error queue, that'll likely do. If I had that now, I could "just" let the retrying take over entirely. Ho hum.
To unsubscribe from this group and stop receiving emails from it, send an email to particularsoftware+unsub...@googlegroups.com.
To post to this group, send email to particula...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Particular Software" group.
To unsubscribe from this group and stop receiving emails from it, send an email to particularsoftware+unsub...@googlegroups.com.

Andreas Ohlund

unread,
Sep 14, 2017, 2:12:08 AM9/14/17
to Particular Software
>As for the v6 comment - if the actual full exception detail is hidden unless it's moved to the error queue, that'll likely do. If I had that now, I could "just" let the retrying take over entirely. Ho hum.

What log level are you running with? you shouldn't see the FLR unless you have INFO turned on? (not recommended in the long run)

Pruning callstacks for the sake of less text in the FLR and SLR logs feels off to me since that would make things harder to debug?

To unsubscribe from this group and stop receiving emails from it, send an email to particularsoftw...@googlegroups.com.
To post to this group, send email to particula...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Particular Software" group.
To unsubscribe from this group and stop receiving emails from it, send an email to particularsoftw...@googlegroups.com.

To post to this group, send email to particula...@googlegroups.com.
Visit this group at https://groups.google.com/group/particularsoftware.

--
You received this message because you are subscribed to the Google Groups "Particular Software" group.
To unsubscribe from this group and stop receiving emails from it, send an email to particularsoftw...@googlegroups.com.

To post to this group, send email to particula...@googlegroups.com.
Visit this group at https://groups.google.com/group/particularsoftware.
To view this discussion on the web visit https://groups.google.com/d/msgid/particularsoftware/dc74a667-ce01-4dd9-aaaf-7cbc17fb8278%40googlegroups.com.

Neil Barnwell

unread,
Sep 14, 2017, 3:40:37 AM9/14/17
to particula...@googlegroups.com
We log at info level because not all problems are characterised by exceptions. It's helpful to see what the "story" of the app was when something happened that was unexpected, but did not necessarily raise an actual exception. That said, reducing the logging to "warn" or even "error" temporarily could help with my problem logfiles while I implement my changes.

I'm considering putting something together around Bus.Defer() and (for now, until it goes away) Bus.HandleCurrentMessageLater() that I can invoke deliberately. Perhaps HCML 5 times, then Defer 5 times with increasing timeouts, and then perhaps start failing and therefore let FLR/SLR take over from that point. This is basically all around messages being handled out of order.


Neil Barnwell.
Mobile: 07900 221457
Twitter: @neilbarnwell
e-mail: mai...@neilbarnwell.co.uk
web: http://www.neilbarnwell.co.uk

To unsubscribe from this group and stop receiving emails from it, send an email to particularsoftware+unsub...@googlegroups.com.
To post to this group, send email to particula...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Particular Software" group.
To unsubscribe from this group and stop receiving emails from it, send an email to particularsoftware+unsub...@googlegroups.com.

To post to this group, send email to particula...@googlegroups.com.
Visit this group at https://groups.google.com/group/particularsoftware.

--
You received this message because you are subscribed to the Google Groups "Particular Software" group.
To unsubscribe from this group and stop receiving emails from it, send an email to particularsoftware+unsub...@googlegroups.com.
To post to this group, send email to particularsoftware@googlegroups.com.

Visit this group at https://groups.google.com/group/particularsoftware.
To view this discussion on the web visit https://groups.google.com/d/msgid/particularsoftware/dc74a667-ce01-4dd9-aaaf-7cbc17fb8278%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Particular Software" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/particularsoftware/LG16obVRmP8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to particularsoftware+unsub...@googlegroups.com.
To post to this group, send email to particularsoftware@googlegroups.com.

Visit this group at https://groups.google.com/group/particularsoftware.
To view this discussion on the web visit https://groups.google.com/d/msgid/particularsoftware/CA%2BFW%2Begu-rjvrKn6CE2dFWsE6M32VTLQZ0BZnF73euScf5qfFg%40mail.gmail.com.
Reply all
Reply to author
Forward
0 new messages