Blacbox exporter soap check 500

Yusuf Dönmez

unread,

Jul 14, 2020, 5:17:29 AM7/14/20

to Prometheus Users

Hi,

I need to check if some of our client soap api server is live.

in my prometheus.yml

- job_name : 'clients'

metrics_path: /probe

scrape_interval: 30s

file_sd_configs:

- files:

- '/etc/prometheus/targets/*.yml'

relabel_configs:

- source_labels: [__address__]

target_label: __param_target

- source_labels: [__param_target]

target_label: instance

- target_label: __address__

replacement: X.X.202.179:9115 # The blackbox exporter's real hostname:port.

my blackbox.yml:

modules:

http_2xx:

prober: http

timeout: 5s

http:

valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]

valid_status_codes: [200,201] # Defaults to 2xx

method: GET

headers:

Host:

Accept-Language: en-US

Origin: example.com

no_follow_redirects: false

fail_if_ssl: false

fail_if_not_ssl: false

fail_if_body_matches_regexp:

- "Could not connect to database"

fail_if_body_not_matches_regexp:

- "Download the latest version here"

fail_if_header_matches: # Verifies that no cookies are set

- header: Set-Cookie

allow_missing: true

regexp: '.*'

fail_if_header_not_matches:

- header: Access-Control-Allow-Origin

regexp: '(\*|example\.com)'

tls_config:

insecure_skip_verify: false

preferred_ip_protocol: "ip4" # defaults to "ip6"

ip_protocol_fallback: false # no fallback to "ip6"

magento1_x:

prober: http

timeout: 10s

http:

method: POST

headers:

SOAPAction: "urn:Mage_Api_Model_Server_V2_HandlerAction"

Content-Type: text/xml; charset="utf-8"

body: '<Envelope xmlns="http://schemas.xmlsoap.org/soap/envelope/">

<Body>

<login>

</login>

</Body>

</Envelope>'

preferred_ip_protocol: "ip4"

in /targets/ directory

http_2xx.yml:( I can get 200 for below)

- labels:

module: http_2xx

targets:

- 'https://xxxxxxx.com'

- 'https://www.xxxxxx.com'

magento1_x.yml(I cant get 200 for below ones)

- labels:

module: magento1_x

targets:

- 'https://www._client_host_.com/api/v2_soap/'

with curl I got 200

curl -v --location --request POST 'https://www._client_host_.com/api/v2_soap/' --header 'SOAPAction: "urn:Mage_Api_Model_Server_V2_HandlerAction"' --header 'Content-Type: text/xml; charset="utf-8"' --data-raw '<Envelope xmlns="http://schemas.xmlsoap.org/soap/envelope/">

<Body>

<login>

</login>

</Body>

</Envelope>'

Note: Unnecessary use of -X or --request, POST is already inferred.

* Trying 192.240.171.50:443...

* TCP_NODELAY set

* Connected to www._client_host_.com (192.240.171.50) port 443 (#0)

* ALPN, offering h2

* ALPN, offering http/1.1

* successfully set certificate verify locations:

* CAfile: /etc/ssl/certs/ca-certificates.crt

CApath: /etc/ssl/certs

* TLSv1.3 (OUT), TLS handshake, Client hello (1):

* TLSv1.3 (IN), TLS handshake, Server hello (2):

* TLSv1.2 (IN), TLS handshake, Certificate (11):

* TLSv1.2 (IN), TLS handshake, Server key exchange (12):

* TLSv1.2 (IN), TLS handshake, Server finished (14):

* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):

* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):

* TLSv1.2 (OUT), TLS handshake, Finished (20):

* TLSv1.2 (IN), TLS handshake, Finished (20):

* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384

* ALPN, server did not agree to a protocol

* Server certificate:

* subject: CN=_client_host_.com

* start date: May 23 23:10:21 2020 GMT

* expire date: Aug 21 23:10:21 2020 GMT

* subjectAltName: host "www._client_host_.com" matched cert's "www._client_host_.com"

* issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3

* SSL certificate verify ok.

> POST /api/v2_soap/ HTTP/1.1

> Host: www._client_host_.com

> User-Agent: curl/7.68.0

> Accept: */*

> SOAPAction: "urn:Mage_Api_Model_Server_V2_HandlerAction"

> Content-Type: text/xml; charset="utf-8"

> Content-Length: 192

>

* upload completely sent off: 192 out of 192 bytes

* Mark bundle as not supporting multiuse

< HTTP/1.1 200 OK

< Date: Tue, 14 Jul 2020 08:57:43 GMT

< Server: Apache

< Set-Cookie: PHPSESSID=glb8rlg296d2ssdki0q2qjh1v5; expires=Tue, 14-Jul-2020 09:57:43 GMT; Max-Age=3600; path=/; domain=_client_host_.com; secure; HttpOnly

< Expires: Thu, 19 Nov 1981 08:52:00 GMT

< Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0

< Pragma: no-cache

< Set-Cookie: CUSTOMER=deleted; expires=Thu, 01-Jan-1970 00:00:01 GMT; Max-Age=0; path=/; domain=_client_host_.com; secure; httponly

< Set-Cookie: CUSTOMER_INFO=deleted; expires=Thu, 01-Jan-1970 00:00:01 GMT; Max-Age=0; path=/; domain=_client_host_.com; secure; httponly

< Set-Cookie: CUSTOMER_AUTH=deleted; expires=Thu, 01-Jan-1970 00:00:01 GMT; Max-Age=0; path=/; domain=_client_host_.com; secure; httponly

< Content-Length: 321

< Vary: Accept-Encoding,User-Agent

< Content-Type: text/xml; charset=UTF-8

<

<?xml version="1.0" encoding="UTF-8"?>

<SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/"><SOAP-ENV:Body><SOAP-ENV:Fault><faultcode>6</faultcode><faultstring>Required parameter is missing, for more details see "exception.log".</faultstring></SOAP-ENV:Fault></SOAP-ENV:Body></SOAP-ENV:Envelope>

* Connection #0 to host www._client_host_.com left intact

but in http://52.59.202.179:9115/probe?target=https%3A%2F%2Fwww._client_host_.com%2Fapi%2Fv2_soap%2F

probe_dns_lookup_time_seconds 0.000907718

probe_duration_seconds 0.715782868

probe_failed_due_to_regex 0

probe_http_content_length 269

probe_http_duration_seconds{phase="connect"} 0.107355228

probe_http_duration_seconds{phase="processing"} 0.387106167

probe_http_duration_seconds{phase="resolve"} 0.000907718

probe_http_duration_seconds{phase="tls"} 0.32711412

probe_http_duration_seconds{phase="transfer"} 0.000230468

probe_http_redirects 0

probe_http_ssl 1

probe_http_status_code 500

probe_http_uncompressed_body_length 269

probe_http_version 1.1

probe_ip_addr_hash 2.37858288e+08

probe_ip_protocol 4

probe_ssl_earliest_cert_expiry 1.598051421e+09

probe_ssl_last_chain_expiry_timestamp_seconds 1.598051421e+09

probe_success 0

probe_tls_version_info{version="TLS 1.2"} 1

I dont understand what is wrong with blackbox.yml I cant get 200 from there.

Christian Hoffmann

unread,

Jul 14, 2020, 1:14:48 PM7/14/20

to Yusuf Dönmez, Prometheus Users

Hi,

On 7/14/20 11:17 AM, Yusuf Dönmez wrote:
[...]

>       - source_labels: [__address__]
>         target_label: __param_target
>       - source_labels: [__param_target]
>         target_label: instance
>       - target_label: __address__
>         replacement:

[...]

> - labels:
>     module: magento1_x
>   targets:
>   - 'https://www._client_host_.com/api/v2_soap/'

The only thing I notice immediately is that you seem to set a label
which is identical to your blackbox_exporter module (module=magento1_x),
but I suspect that it is not passed to blackbox_exporter. Can you
confirm via the targets on the Prometheus Web UI? I suspect there is no
module=magento1_x.

If you don't need the module as a label in your metrics, I simply
suggest to change it to __param_module: magento1_x so that it is used
internally only.

If you do need it, I suggest adding a relabeling rule which copies the
value of the module label into the __param_module special label.

If this is not the case, I could only suspect some problem with quoting
or some other headers. A tcpdump would help, but only if you are able to
run this without TLS or are able to intercept it some how (mitmproxy or
Wireshark + TLS secret capturing via LD_PRELOAD or something). You could
also hack up blackbox_exporter to output the relevant headers if possible.

As far as I can see, the existing debug methods do not cover it.

I guess you've already tried what's listed here:

https://www.robustperception.io/debugging-blackbox-exporter-failures

Kind regards,
Christian

Yusuf Dönmez

unread,

Jul 15, 2020, 5:20:46 AM7/15/20

to Prometheus Users

Thanks Chritian,

You re right. I missed.

I followed that https://www.sethryder.com/blackbox-exporter-accessing-multiple-modules-and-targets-with-a-single-job/

but I missed:

- source_labels: [module]

target_label: __param_module

in my prometheus.yml

- job_name : 'clients'

metrics_path: /probe

scrape_interval: 30s

file_sd_configs:

- files:

- '/etc/prometheus/targets/*.yml'

relabel_configs:

- source_labels: [__address__]

target_label: __param_target

- source_labels: [module]

target_label: __param_module

- source_labels: [__param_target]

target_label: instance

- target_label: __address__

replacement: xx.xx.202.179:9115 # The blackbox exporter's real hostname:port.

Thanks again.

saipradeep bojja

unread,

May 13, 2022, 2:28:43 AM5/13/22

to Prometheus Users

Hi Christian,

I have configured the magento module able to get the metrics for Soap url's with status code 200.

But when when ever i get a response "200". There is also an error is generated in an error log.

Below is the error log when i get "200" if i get "500" response i wont get any error.

[date and time] [core:eorror] [pid 9731:tid 140637200057483] (104) Connection reset by peer: [client <ip>:38464] AH000574: ap_content_length_filter:apr_bucket_read() failed

How to resolve this am not getting any clue actually. Atleast how to bypass the error.log? would be helpfull

saipradeep bojja

unread,

May 18, 2022, 1:11:32 PM5/18/22

to Prometheus Users

Hi any update on the above message

Reply all

Reply to author

Forward