curl api tests with prometheus

[Nikos Kalenzis]

Hi,

New prometheus user here, still learning.

I have a task of monitoring an api with prometheus. For that we have 10 curls tests prepared. I managed to configure them to run on Zabbix with no problems, but being new in prometheus i'm having trouble with it. 

I have to run the tests against a google compute engine instance group (every instance) and look for return code 200 and a matching regex for every test. I managed to configure gcp authentication and got the instances as scrape targets using <gce_sd_config>, but i have no idea on how to configure the tests. Did a lot of reading and realized that blackbox_exporter should do the trick here. Got this as one of the modules in blackbox.yml:

modules:

  http_get:

    prober: http

    timeout: 5s

    http:

      method: GET

      valid_status_codes: [200]

      fail_if_not_matches_regexp:

      - "location_id"i

Now i have no idea how to configure prometheus.yml to run this module on all instances discovered by gce_sd_config using the URL's for the tests (i also have post and put tests). This was nicely done in zabbix using web scenarios and a variable for the IP of teh instance and the url was: http://{ip.address]/restofurl with post data and headers given separately. Any help would be deeply appreciated.

[Ben Kochie]

One thing you might want to consider before you worry about blackbox testing is getting your whitebox metrics in order.  Blackbox probes are of course the only thing that monitoring systems like Zabbix support.  But Prometheus can do much better here and give you full whitebox instrumentation.

I would recommend getting whitebox metrics setup to monitor all of your API servers first, and deal with the blackbox stuff later.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/5abbd0ec-342c-4307-b198-55a22ee82acb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Nikos Kalenzis]

Hi Ben, 

Thank you for the reply. 

I know i need to get whitebox monitoring going too... Like i said, i managed to get the info for the vm's from gcp. It kept looking for the /metrics path on them. This is a clients api and i'm not sure if i can change anything. I tried running node_exporter on them and got the /metrics on 9100, but i don't think that's the way to do it. How do i get the proper metrics for prometheus to scrape?

thank you,

Nikos

[Ben Kochie]

If you're unable to change the code, one option is to implement a log-based metrics generator with something like mtail[0] or grok_exporter[1].

You can parse the application servers logs and use that to create whitebox metrics.

If the code is capable of push metrics, something like statsd, you can use the statsd_exporter[2].

[0]: https://github.com/google/mtail

[1]: https://github.com/fstab/grok_exporter

[2]: https://github.com/prometheus/statsd_exporter

[Nikos Kalenzis]

Thank you again. I'll give this a try and see how it goes.

kind regards,

Nikos

[Nikos Kalenzis]

Hi All,

Me again. We are going to use Elastic stack to take care of the logging. So i will need to use prometheus to monitor the health of the instances (machine metrics) and those API tests i mentioned (like i said, i was planning on using blackbox_exporter for the HTTP tests). 

I apologize in advance if i'm not making any sense, i'm still trying to figure things out.

As i mentioned before i managed to get the instances as scrape targets, all showing as down (no metrics path). My questions are:

 - do i just run node_exporter on each of the instances to provide the machine metrics in the /metrics path, then add the gce_sd_configs in a new job?

 - how do i make a job that does a HTTP check (http://{ip of instance}/rest of URL with GET, POST and PUT parameters for each) on all discovered instances using blackbox_exporter?

[Nikos Kalenzis]

Hi All,

I still have this issue and cannot seem to be able to resolve it. Any help would be deeply appreciated.

I'm using blackbox exporter to run HTTP checks for some of my services and it works fine when running against the URL. But i need to run the tests using this format: (http://{ip of instance}/rest of URL with GET, POST and PUT parameters) and i need to run it agains a Google compute instance group. I have managed to discover the instances using gce_sd_configs. Now i need a way to run the http test against all of them.

thanks.

Nikos