Sudden increase in in-transit and request expiring

[shadei...@gmail.com]

Hi,

I find myself in a dilemma trying to find out which participant is causing an issue.

I had my in-transit numbers between 2 to 10 and then suddenly it increased to 20 and kept climbing up. Looking through the logs repeatedly, I don't see any elapsed time for my transactions beyond 1000ms to 2000ms, which is a good time for the transactions I am processing. 

Unfortunately, I noticed the request expired and unhandled increased but can't see exactly which participant is impacting the system negatively.

I went from this in transit:

     in-transit=31/34, head=1881724, tail=1881690, paused=0, outstanding=0, active-sessions=800/1200, tps=9, peak=955, avg=3.11, elapsed=82ms
     in-transit=33/37, head=1881727, tail=1881690, paused=0, outstanding=0, active-sessions=800/1200, tps=9, peak=955, avg=3.11, elapsed=356ms
     in-transit=32/37, head=1881727, tail=1881690, paused=0, outstanding=0, active-sessions=800/1200, tps=9, peak=955, avg=3.11, elapsed=430ms
     in-transit=32/38, head=1881728, tail=1881690, paused=0, outstanding=0, active-sessions=800/1200, tps=9, peak=955, avg=3.11, elapsed=429ms

to what I have below within less than 3 minutes:

     in-transit=9/267, head=1882005, tail=1881738, paused=0, outstanding=0, active-sessions=800/1200, tps=9, peak=955, avg=3.11, elapsed=626ms
     in-transit=8/267, head=1882005, tail=1881738, paused=0, outstanding=0, active-sessions=800/1200, tps=9, peak=955, avg=3.11, elapsed=83ms
     in-transit=7/267, head=1882005, tail=1881738, paused=0, outstanding=0, active-sessions=800/1200, tps=9, peak=955, avg=3.11, elapsed=567ms

and in another 2 minutes, I got to the below:

     in-transit=11/1467, head=1883205, tail=1881738, paused=0, outstanding=0, active-sessions=800/1200, tps=34, peak=955, avg=3.11, elapsed=332ms
     in-transit=13/1470, head=1883208, tail=1881738, paused=0, outstanding=0, active-sessions=800/1200, tps=34, peak=955, avg=3.11, elapsed=100ms
     in-transit=12/1470, head=1883208, tail=1881738, paused=0, outstanding=0, active-sessions=800/1200, tps=34, peak=955, avg=3.11, elapsed=274ms

Can you advise what else to look for or what additional logging maybe required to help identify what the issue is? 

Regards,

Yetunde

[Alejandro Revilla]

Look at the SystemMonitor output in the log for the transaction 1881690. It will show you which participant is running, and that’s the participant that needs review because it might be lacking a proper timeout.

--

@apr

--
--
jPOS is licensed under AGPL - free for community usage for your open-source project. Licenses are also available for commercial usage. Please support jPOS, contact: sa...@jpos.org
---
You received this message because you are subscribed to the Google Groups "jPOS Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jpos-users+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jpos-users/b1b413eb-f91e-40ed-93e2-ad6737ccf6d2n%40googlegroups.com.

[shadei...@gmail.com]

Hello Alejandro,

Thanks so much for that pointer. That transaction completed before the System monitor logs dropped and so I didn't see the participant that held onto it.

But it gave me an idea of what to check for. I got another transaction that showed in the logs within 4 minutes as the tail over 8000 times and I noticed the in-transit numbers growing into the thousands by that point.

I was able to discover which participant was the problem because it showed in the system monitor logs.

Thanks for your support.

Regards,

Yetunde

[Alejandro Revilla]

BTW, this is something related to your problem that is coming soon (actually already available in jPOS 3.0.0-SNAPSHOT).

https://jpos.org/blog/2022/05/tm-timeout-and-max-time

--

@apr

To view this discussion on the web visit https://groups.google.com/d/msgid/jpos-users/ae7bec0e-0f17-4157-b920-634b84055c7an%40googlegroups.com.

[shadei...@gmail.com]

Great news Alejandro.

I can already see participants that I can use it for.

Great job on all the updates to make our lives easier.

Regards,

Yetunde

[Alejandro Revilla]

Great to know!

We have developed these tools because we faced the same challenges you are encountering now. The beauty of open source and a established user base is that the problems you experience today have likely already been solved by someone else in the past. This means there's often a ready-made fix available :)

--

@apr

To view this discussion on the web visit https://groups.google.com/d/msgid/jpos-users/e4b50751-5900-4fd9-9363-35115579292cn%40googlegroups.com.