http 599 timeout error when using http.query runner

[Paul Azad]

Hi

When we are using the runner.http.query that is triggered by a reactor, we are constantly seeing an error 599. The reactor is sending 6 queries - 1 after another with grains data. We can use Advanced REST Client and push much more data (ie constantly press SEND - 5-60 times in a row) and the end point never gives us any errors.

When using ARC - i am seeing the response time is between 80ms and 130ms - yet when the runner is sending the data, the last 1 or 2 constantly show a time out.

Researching, it seems to be caused by tornado, and i have seen people mention to adjust the connect_timeout &  request_timeout in tornado - but not show how in salt.

I did come across a setting http_request_timeout in this page https://github.com/saltstack/salt/issues/32400 , but adding this to /etc/salt/master has not helped.

When i run the salt master in debug mode, i noticed the error 599:

[DEBUG   ] Sending event: tag = salt/run/20180124182653281912/ret; data = {'fun_args': ['https://<URL>', OrderedDict([('method', 'POST'), ('data', OrderedDict([('id', '<minion>'), ('grains', '{ "virtual": "VMware"}')]))])], 'jid': '20180124182653281912', 'return': {'status': 599, 'error': 'HTTP 599: Timeout'}, 'success': True, '_stamp': '2018-01-24T07:27:13.320753', 'user': 'Reactor', 'fun': 'runner.http.query'}

[DEBUG   ] LazyLoaded nested.output

error:

    HTTP 599: Timeout

status:

    599

The a short time later i saw this message appear:

[INFO    ] Runner completed:20180124182653281912

I am not to fussed about the message, as long as it works.

Would the fact i get the 'runner completed' message come through confirm its working?

Thanks