Intermittent 504s when browser is accessing ajaxBuildQueue and ajaxExecutors endpoints

[Prasad Velidi]

Hello,

Recently we have been seeing 504 errors on the ELB status logs (AWS) for Jenkins. When investigated, we came to know that, to keep the page on browser in sync with latest information, the browser keeps on polling data using ajaxBuildQueue and ajaxExecutors endpoints. When the logs for these requests and their responses are observed, we saw that intermittently 504 errors are popping up. This tripped our thresholds for 5xx errors. We want to know how to eradicate the 504s as it might also impacts performance if we start to see more of these errors.

When observed, we found 26 of the 504s on the logs over a period of 15 mins. While many of the requests land on ELB one after the other, only one out of 10 subsequent requests to the mentioned endpoints produced the 504s.

 

Not sure if this is related to performance tuning where in we tune the thresholds in configuration somewhere or is it something to do with jenkins itself. Any suggestions are highly appreciated.

Regards,

Prasad Velidi

[James Nord]

This could be a number of things without any data to correlate

1. Garbage collection performing stop the world collection (Google 'Sam van oort jenkins garbage collection 

cloudbees')

2. Not enough http handlers in jenkins and someone is making http requests that are not quick so you exhaust the thread pool (look at' java - jar jenkins.war - - help' also look in a support bundle for slow requests)

3. Jenkins being starved of underlying resources (cpu,  memory paging disk io..)

There's probably plenty of others as well but this are possibly the biggest 3 off the top of my head. 

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/2593f9a0-89ab-420e-9016-2c83dede6da1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Prasad Velidi]

Thank you for your valuable inputs. You have covered more things than what i had in mind. Appreciate your help.