salt-api (CherryPy) timing out with 504 error message
Paul Azad
Hi
I have looked at previous posts regarding the time out issue, and the issue isnt resolved with expire_responses. I have tried this on multiple versions of salt from 2016.11 to 2019.2, and further more according to CherryPy's corresponding variable timeout_monitor, this was removed last year, so from 2018.3.0 the expire_responses variable is being ignored anyway?
(https://github.com/cherrypy/cherrypy/blob/c17be0f9c42bbce95b37688dd548e47114f4a136/CHANGES.rst#v1200).
The issue is that when sending a command that takes more then 53 seconds, it always times out. I am pretty sure the issue is with CherryPy timing out as if i run the exact command from the command line its fine:
salt 'paul-in*' cmd.run 'timeout 75 > NULL & dir'
minion:
Volume in drive C is OS
Volume Serial Number is 48A4-087B
Directory of C:\WINDOWS\system32\config\systemprofile
25/03/2019 08:11 AM <DIR> .
25/03/2019 08:11 AM <DIR> ..
12/04/2018 10:38 AM <DIR> AppData
26/03/2019 06:13 AM 355 NULL
1 File(s) 355 bytes
3 Dir(s) 932,373,143,552 bytes free
real 1m22.292s
user 0m1.824s
sys 0m0.172s
I have tested it on 2017.7.2 - which is the last version that should of supported this variabled according to https://docs.saltstack.com/en/latest/ref/netapi/all/salt.netapi.rest_cherrypy.html, but i had the same issue as i did on 2016.11.8 and also now on 2019.2.0.
So when i send a command with the timeout value, if the timeout value was less then 53, and the command would take more then this time, i would see an appropriate response.
time curl -ksi https://SALT_SERVER -H "Accept: application/x-yaml" -H "X-Auth-Token: $TOKEN" -d client='local' -d tgt=minion -d fun=cmd.run -d arg="timeout 20 > NULL & dir" -d arg=timeout=3
HTTP/1.1 200 OK
Server: nginx
Date: Sun, 24 Mar 2019 23:11:30 GMT
Content-Type: application/x-yaml
Content-Length: 118
Connection: keep-alive
Access-Control-Expose-Headers: GET, POST
Cache-Control: private
Vary: Accept-Encoding
Allow: GET, HEAD, POST
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
Set-Cookie: session_id=e6e5a3f653eb66c84acdd26a529a10f3e41565ea; expires=Mon, 25 Mar 2019 09:11:21 GMT; Path=/
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
return:
- minion: 'chcp 437 > nul & timeout 20 > NULL : Timed
out after 3 seconds'
real 0m9.090s
user 0m0.016s
sys 0m0.004s
If i send a command that takes more then 60 seconds, and the timeout command to be more then 60 seconds then the timeout command seems to be ignored:
curl -ksi https://SALT_SERVER -H "Accept: application/x-yaml" -H "X-Auth-Token: $TOKEN" -d client='local' -d tgt=minion -d fun=cmd.run -d arg="timeout 75 > NULL & dir" -d arg=timeout=90
HTTP/1.1 504 Gateway Time-out
Server: nginx
Date: Mon, 25 Mar 2019 19:20:34 GMT
Content-Type: text/html
Content-Length: 176
Connection: keep-alive
<html>
<head><title>504 Gateway Time-out</title></head>
<body bgcolor="white">
<center><h1>504 Gateway Time-out</h1></center>
<hr><center>nginx</center>
</body>
</html>
real 1m0.133s
user 0m0.012s
sys 0m0.008s
You can see that the real time is pretty much exactly 60 seconds- so the timeout looks to be pretty accurate and hard from the webserver side, as you would expect that if it was a salt command issue the time would be a little more then 60 seconds
Another run:
HTTP/1.1 504 Gateway Time-out
Server: nginx
Date: Mon, 25 Mar 2019 19:22:24 GMT
Content-Type: text/html
Content-Length: 176
Connection: keep-alive
<html>
<head><title>504 Gateway Time-out</title></head>
<body bgcolor="white">
<center><h1>504 Gateway Time-out</h1></center>
<hr><center>nginx</center>
</body>
</html>
real 1m0.131s
user 0m0.020s
sys 0m0.004s
Again:
HTTP/1.1 504 Gateway Time-out
Server: nginx
Date: Mon, 25 Mar 2019 19:23:44 GMT
Content-Type: text/html
Content-Length: 176
Connection: keep-alive
<html>
<head><title>504 Gateway Time-out</title></head>
<body bgcolor="white">
<center><h1>504 Gateway Time-out</h1></center>
<hr><center>nginx</center>
</body>
</html>
real 1m0.126s
user 0m0.012s
sys 0m0.008s
Again:
HTTP/1.1 504 Gateway Time-out
Server: nginx
Date: Mon, 25 Mar 2019 19:25:11 GMT
Content-Type: text/html
Content-Length: 176
Connection: keep-alive
<html>
<head><title>504 Gateway Time-out</title></head>
<body bgcolor="white">
<center><h1>504 Gateway Time-out</h1></center>
<hr><center>nginx</center>
</body>
</html>
real 1m0.106s
user 0m0.012s
sys 0m0.008s
Here if i run it with even 55 seconds - it fails:
time curl -ksi https://SALT_SERVER:1000 -H "Accept: application/x-yaml" -H "X-Auth-Token: $TOKEN" -d client='local' -d tgt=minion -d fun=cmd.run -d arg="timeout 75 > NULL & dir" -d arg=timeout=55
HTTP/1.1 504 Gateway Time-out
Server: nginx
Date: Mon, 25 Mar 2019 19:25:11 GMT
Content-Type: text/html
Content-Length: 176
Connection: keep-alive
<html>
<head><title>504 Gateway Time-out</title></head>
<body bgcolor="white">
<center><h1>504 Gateway Time-out</h1></center>
<hr><center>nginx</center>
</body>
</html>
real 1m0.106s
user 0m0.012s
sys 0m0.008s
But on the final run i set the time out to 53 seconds - and it timed out at the salt command level:
time curl -ksi https://SALT_SERVER:1000 -H "Accept: application/x-yaml" -H "X-Auth-Token: $TOKEN" -d client='local' -d tgt=minion -d fun=cmd.run -d arg="timeout 65 > NULL & dir" -d arg=timeout=53
HTTP/1.1 200 OK
Server: nginx
Date: Mon, 25 Mar 2019 19:30:11 GMT
Content-Type: application/x-yaml
Content-Length: 119
Connection: keep-alive
Access-Control-Expose-Headers: GET, POST
Cache-Control: private
Vary: Accept-Encoding
Allow: GET, HEAD, POST
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
Set-Cookie: session_id=efa39f76d6aa85e8d7857ddab605bf2a031c5ead; expires=Tue, 26 Mar 2019 05:29:12 GMT; Path=/
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
return:
- paul-insprion3576.thissolution.local: 'chcp 437 > nul & timeout 65 > NULL : Timed out after 53 seconds'
real 0m59.836s
user 0m0.012s
sys 0m0.016s
Here is the API part of my master config:
rest_cherrypy:
port: 8000
debug: True
disable_ssl: True
expire_responses: False
rest-timeout: 7200
timeout: 7200
rest_timeout: 7200
Paul
Found the issue - it wasn't CherryPy causing the time-out, it was nginx. We had to use nginx as enabling ssl directly on the salt-api in docker doesnt work.