JIRA search REST API returns HTTP error 504 after intensive usage

Raimonds Simanovskis

unread,

Sep 18, 2013, 4:52:04 AM9/18/13

to atlassian-...@googlegroups.com

In eazyBI add-on I am using JIRA search API /rest/api/latest/search with jql parameter to get all issues of selected projects and use startAt and maxResults (by default 500) parameters to paginate through results.

To make download of all issues faster I do these requests in parallel.

In recent month I started to experience issues for some customers which have larger amount of issues (tens of thousands) that I start to get HTTP error 504 response with body

<!DOCTYPE html>

<head>

<title>Atlassian - Down for maintenance</title>

</head>

<img src="/maintenance/images/DownforMaintenance_CharlieZap.gif" width="960" height="625" alt="We're down for maintenance. Be

right back!">

</body>

</html>

I try to retry

Are there any limitations added to JIRA OnDemand REST API usage which start to give HTTP error 504 when this limit is exceeded?

Now I manage to avoid exceptions when I decreased number of maximum parallel search API requests from 10 to 5 but would like to know what would be the best option how to dynamically change number of max parallel REST API requests to avoid hitting HTTP 504 errors.

Kind regards,

Raimonds Simanovskis

unread,

Dec 17, 2013, 12:49:19 PM12/17/13

to atlassian-...@googlegroups.com

Now one JIRA OnDemand customer is getting HTTP 504 every time after intensive JIRA search REST API usage.

I wanted to clarify is there any usage limitations at the JIRA OnDemand side for REST API usage?

How should I proceed with this issue? Should I report it to support.atlassian.com?

Kind regards,

Raimonds

Matt Quail

unread,

Dec 17, 2013, 5:37:20 PM12/17/13

to atlassian-...@googlegroups.com

Raimonds,

Those 504 "electric Charlie" errors are caused when the request takes too long, and the edge proxy times out the request. (There is are various ngnix/apache proxies in front of the actual JIRA instance). The timeout is 90s.

Why is JIRA timing out? It may be that the JQL request is just taking too long, but otherwise JIRA is healthy. That could "naturally" happen on some instances with complicated permissions schemes. It would also happen if the instance was being restarted. But I would be surprised if it would happen very often with maxResults=500. If you were doing a lot of requests in parallel, then maybe that extra load is causing each request to take longer.

The other reason is the JIRA instance is not healthy. Raising a support.atlassian.com request for these situations will allow us to collect information for those specific instances and maybe track down the problem further. Get the customer to tell the support stuff their instance, the JQL that is running, and the user the JQL is running as. Also the time that the errors occurred.

In either case: this is probably a good opportunity to "embrace failure". Getting a 504 might be rare, but it is normal. If you are executing in parallel and get a failure, you should probably "back off" your parallelism and maybe introduce some delay between requests. In fact a "ramp up" mechanism is probably a good idea. Start by making sequential requests, and if there are lots of results to get, gradually introduce parallelism, but on error or slowness back off. Effectively: you are introducing parallelism to increase throughput. Therefore: measure your throughput (number of issues fetched per minute) and adaptively decrease parallelism if the throughput is actually going down.

=Matt

--
You received this message because you are subscribed to the Google Groups "Atlassian Connect Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to atlassian-connec...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Raimonds Simanovskis

unread,

Dec 18, 2013, 9:03:30 AM12/18/13

to atlassian-...@googlegroups.com

Matt,

Thanks for your comments and suggestions.

I was already retrying these failed requests (initially with constant 5 seconds sleep time but later with exponentially increasing sleep time) but once I started to get these errors I continued to get them.

But finally I managed to get requests working by reducing maxResults=500 to maxResults=300 and now I didn't get any HTTP 504 during importing all JIRA issues. So it seems that preparing of JSON output for 500 issues took longer time than front-end web server gateway timeout. As the default maximum maxResults value is 1000 then it would be good to either investigate why maxResults=500 result in timeout or just increase front-end web server gateway timeout.

In addition I reported this issue for support.atlassian.com as well.

Kind regards,

Raimonds

To unsubscribe from this group and stop receiving emails from it, send an email to atlassian-connect-dev+unsub...@googlegroups.com.

Reply all

Reply to author

Forward