health check for reverse proxy with https members

[Nam Van]

Hi all,

I am struggling to make the config working for a reverse proxy with members serving with https. All suggestions are welcome and appreciated.

In my set up I have a few servers serving the same service but not all of them are online at the same time. Hence I got a health check service running as well to detect the offline members.

My set up was working fine until I added the health check bit. The health check is disabling all members due to SSL handshake issue. The I've been adding a few directives but only can access the service via the reverse proxy for a minute before the health check kicks in and marked all members off.

My config is something like this
<VirtualHost *:443>
ServerName <masked-out>

SSLProxyEngine on
ProxyRequests off
SSLProxyVerify none
SSLProxyCheckPeerCN off
SSLProxyCheckPeerName off
SSLProxyCheckPeerExpire off

SSLCertificateFile <masked-out>.crt
SSLCertificateKeyFile <masked-out>.key
SSLCertificateChainFile <masked-out>.crt

CustomLog <masked-out>.log combined
ErrorLog <masked-out>.log


ProxyHCExpr api_good {hc('body') =~ /Healthy/}

ProxyHCTemplate hc_dev_ams_sec hcinterval=45 hcpasses=1 hcfails=1 hcmethod=GET hcexpr=api_good hcuri=/sec/health

<proxy balancer://api_dev_ams_sec>
BalancerMember https://brgvdhasf16:4301 hctemplate=hc_dev_ams_sec
BalancerMember https://brgvdhasf17:4301 hctemplate=hc_dev_ams_sec
</proxy>

ProxyPreserveHost on
ProxyPass /sec balancer://api_dev_ams_sec/sec
ProxyPassReverse /sec balancer://api_dev_ams_sec/sec
</VirtualHost>

I could see this in the log

[Thu Jun 09 17:01:51.899492 2022] [proxy_hcheck:debug] [pid 61220:tid 812] mod_proxy_hcheck.c(480): AH03248: Creating hc worker 21c546c7cc0 for https://brgvdhasf16:4301
[Thu Jun 09 17:01:51.899492 2022] [proxy:debug] [pid 61220:tid 812] proxy_util.c(2123): AH00925: initializing worker 21c546c7cc0 shared
[Thu Jun 09 17:01:51.899492 2022] [proxy:debug] [pid 61220:tid 812] proxy_util.c(2183): AH00927: initializing worker 21c546c7cc0 local
[Thu Jun 09 17:01:51.899492 2022] [proxy:debug] [pid 61220:tid 812] proxy_util.c(2214): AH00930: initialized pool in child 61220 for (brgvdhasf16) min=0 max=64 smax=64
[Thu Jun 09 17:01:51.899492 2022] [proxy_hcheck:debug] [pid 61220:tid 812] mod_proxy_hcheck.c(894): AH03256: Health checking https://brgvdhasf16:4301
[Thu Jun 09 17:01:51.899492 2022] [proxy:debug] [pid 61220:tid 812] proxy_util.c(2532): AH00942: HCOH: has acquired connection for (brgvdhasf16)
[Thu Jun 09 17:01:51.900491 2022] [proxy:debug] [pid 61220:tid 812] proxy_util.c(3277): AH02824: HCOH: connection established with 10.130.55.158:4301 (brgvdhasf16)
[Thu Jun 09 17:01:51.900491 2022] [proxy:debug] [pid 61220:tid 812] proxy_util.c(3463): AH00962: HCOH: connection complete to 10.130.55.158:4301 (brgvdhasf16)
[Thu Jun 09 17:01:51.900491 2022] [ssl:info] [pid 61220:tid 812] [remote 10.130.55.158:4301] AH01964: Connection to child 0 established (server ####<masked-out>#######:443)
[Thu Jun 09 17:01:51.901490 2022] [ssl:info] [pid 61220:tid 812] [remote 10.130.55.158:4301] AH02003: SSL Proxy connect failed
[Thu Jun 09 17:01:51.901490 2022] [ssl:info] [pid 61220:tid 812] [remote 10.130.55.158:4301] AH01998: Connection closed to child 0 with abortive shutdown (server ####<masked-out>#######:443)
[Thu Jun 09 17:01:51.901490 2022] [ssl:info] [pid 61220:tid 812] [remote 10.130.55.158:4301] AH01997: SSL handshake failed: sending 502
[Thu Jun 09 17:01:51.901490 2022] [proxy:debug] [pid 61220:tid 812] proxy_util.c(2547): AH00943: HCOH: has released connection for (brgvdhasf16)
[Thu Jun 09 17:01:51.901490 2022] [proxy_hcheck:debug] [pid 61220:tid 812] mod_proxy_hcheck.c(589): AH03251: Health check GET Status (1) for 21c546c7cc0.
[Thu Jun 09 17:01:51.901490 2022] [proxy_hcheck:info] [pid 61220:tid 812] AH03303: Health check DISABLING https://brgvdhasf16:4301
[Thu Jun 09 17:01:51.901490 2022] [proxy_hcheck:debug] [pid 61220:tid 812] mod_proxy_hcheck.c(480): AH03248: Creating hc worker 21c546c2940 for https://brgvdhasf17:4301
[Thu Jun 09 17:01:51.901490 2022] [proxy:debug] [pid 61220:tid 812] proxy_util.c(2123): AH00925: initializing worker 21c546c2940 shared
[Thu Jun 09 17:01:51.901490 2022] [proxy:debug] [pid 61220:tid 812] proxy_util.c(2183): AH00927: initializing worker 21c546c2940 local
[Thu Jun 09 17:01:51.901490 2022] [proxy:debug] [pid 61220:tid 812] proxy_util.c(2214): AH00930: initialized pool in child 61220 for (brgvdhasf17) min=0 max=64 smax=64
[Thu Jun 09 17:01:51.901490 2022] [proxy_hcheck:debug] [pid 61220:tid 812] mod_proxy_hcheck.c(894): AH03256: Health checking https://brgvdhasf17:4301
[Thu Jun 09 17:01:51.901490 2022] [proxy:debug] [pid 61220:tid 812] proxy_util.c(2532): AH00942: HCOH: has acquired connection for (brgvdhasf17)
[Thu Jun 09 17:01:53.934330 2022] [proxy:error] [pid 61220:tid 812] (OS 10061)No connection could be made because the target machine actively refused it. : AH00957: HCOH: attempt to connect to 10.130.55.160:4301 (brgvdhasf17) failed
[Thu Jun 09 17:01:53.934330 2022] [proxy:debug] [pid 61220:tid 812] proxy_util.c(2547): AH00943: HCOH: has released connection for (brgvdhasf17)
[Thu Jun 09 17:01:53.934330 2022] [proxy_hcheck:debug] [pid 61220:tid 812] mod_proxy_hcheck.c(589): AH03251: Health check GET Status (-1) for 21c546c2940.
[Thu Jun 09 17:01:53.934330 2022] [proxy_hcheck:info] [pid 61220:tid 812] AH03303: Health check DISABLING https://brgvdhasf17:4301