https webhook failing

[James Smith]

I've got an http webhook endpoint configured and working, and I'm trying to configure an https webhook endpoint but am meeting with difficulty. Both http and https endpoints are the same server, an nginx frontend proxying to gunicorn/Flask.

I can POST data to my https endpoint (using curl) from my scalr server with no issues, but any scalr-triggered notifications are failing.

From python-dbqueue.log: 

[01/Nov/2016 21:05:31][       dbqueue_event][ 16163]      ERROR Unable to process webhook: 1B700AFC1B9B443DA4CDAE8520DC69FC, reason: <type 'exceptions.TypeError'> __init__() got an unexpected keyword argument 'server_hostname'

In the webhook event history:

Nov 1, 2016 21:05:31

Farm Cleanup

HostDown

1

Failed

None

Info tooltip has: Attempt #1 failed. Cause: Internal error

I've got debug logging enabled on my webhook endpoint, and am not seeing any access attempts (in nginx/access.log), any errors (in nginx/errors.log), or any indication that the notification payload has reached my endpoint (the first thing my endpoint does on entry is print to the logger "Data received" or something similar)

tcpdump output showing syn/ack and fin/ack but no push between the scalr server (10.100.252.28) and the relay (10.2.209.162):

21:25:48.118617 IP 10.100.252.28.56594 > 10.2.209.162.https: Flags [S], seq 1803375911, win 26883, options [mss 1379,sackOK,TS val 3392812082 ecr 0,nop,wscale 7], length 0

21:25:48.118666 IP 10.2.209.162.https > 10.100.252.28.56594: Flags [S.], seq 2422394964, ack 1803375912, win 26844, options [mss 8960,sackOK,TS val 2308329776 ecr 3392812082,nop,wscale 7], length 0

21:25:48.118752 IP 10.100.252.28.56595 > 10.2.209.162.https: Flags [S], seq 4126344781, win 26883, options [mss 1379,sackOK,TS val 3392812082 ecr 0,nop,wscale 7], length 0

21:25:48.118762 IP 10.2.209.162.https > 10.100.252.28.56595: Flags [S.], seq 3138004224, ack 4126344782, win 26844, options [mss 8960,sackOK,TS val 2308329776 ecr 3392812082,nop,wscale 7], length 0

21:25:48.130091 IP 10.100.252.28.56594 > 10.2.209.162.https: Flags [.], ack 1, win 211, options [nop,nop,TS val 3392812094 ecr 2308329776], length 0

21:25:48.130116 IP 10.100.252.28.56595 > 10.2.209.162.https: Flags [.], ack 1, win 211, options [nop,nop,TS val 3392812094 ecr 2308329776], length 0

21:25:48.138606 IP 10.100.252.28.56594 > 10.2.209.162.https: Flags [F.], seq 1, ack 1, win 211, options [nop,nop,TS val 3392812102 ecr 2308329776], length 0

21:25:48.138652 IP 10.2.209.162.https > 10.100.252.28.56594: Flags [F.], seq 1, ack 2, win 210, options [nop,nop,TS val 2308329796 ecr 3392812102], length 0

21:25:48.147263 IP 10.100.252.28.56595 > 10.2.209.162.https: Flags [F.], seq 1, ack 1, win 211, options [nop,nop,TS val 3392812110 ecr 2308329776], length 0

21:25:48.147300 IP 10.2.209.162.https > 10.100.252.28.56595: Flags [F.], seq 1, ack 2, win 210, options [nop,nop,TS val 2308329805 ecr 3392812110], length 0

21:25:48.149932 IP 10.100.252.28.56594 > 10.2.209.162.https: Flags [.], ack 2, win 211, options [nop,nop,TS val 3392812113 ecr 2308329796], length 0

21:25:48.158661 IP 10.100.252.28.56595 > 10.2.209.162.https: Flags [.], ack 2, win 211, options [nop,nop,TS val 3392812122 ecr 2308329805], length 0

Does anyone have any guidance here? The http webhook is working and I can continue using it for now, but I really would rather have a secure endpoint - there are credentials stored in global variables that I need to access on my webhook endpoint, so I can't just toggle "Don't expose private global variables".

[Marc O'Brien]

Hi James,

/opt/scalr-server/var/logs will be useful as a first step for investigations here.

Many thanks,
Wm. Marc O'Brien
Scalr Technical Support

[James Smith]

Hi Marc,

As I noted above, /opt/scalr-server/var/logs/service/python-dbqueue.log has an error "unable to process webhook: got an unexpected keyword argument 'server_hostname'" - and /opt/scalr-server/var/logs/supervisor/service-dbqueue.err has the same.

There's nothing in any of the other logs that references my endpoint server.

[James Smith]

The "unexpected keyword argument 'server_hostname'" error was resolved by an update to gevent (see my other thread https://groups.google.com/d/msg/scalr-discuss/dnxla5wwUME/tl5w5zRPCgAJ).

However, once the initial error was resolved I started getting an SSLError. My scalr server and the webhook endpoint are using certs generated by my internal CA (a FreeIPA server), and the CA cert had been added to the system trust on the scalr server. I ended up adding a line to dbqueue-event.py:

self.https_session = requests.Session()

self.https_session.mount('https://', helper.HttpsAdapter())

+ self.https_session.verify = <path to FreeIPA CA cert>

Now https webhooks are working as expected, even if this isn't the ideal solution (I would prefer to add the ca cert to whatever custom ca bundle scalr is using for verifications)