Unable to push metrics to Prometheus Pushgateway

[Mahendra M]

Hi All,

I'm facing below exception while migrating to prometheus. Could anyone pls help here?

Exception:

218004:31:17.818 [PushGatewayFactory.PushGatewayTaskScheduler-1] ERROR c.e.r.b.p.adaptor.PushGatewayFactory - Unable to push metrics to Prometheus Pushgateway
2181java.io.IOException: Response code from http://ingress.sherlock-service.qa.ebay.com/api/pushgateway/v1/metrics/job/testjob was 400
2182 at io.prometheus.client.exporter.PushGateway.doRequest(PushGateway.java:304)
2183 at io.prometheus.client.exporter.PushGateway.pushAdd(PushGateway.java:178)
2184 at com.ebay.raptor.batch.pushgateway.adaptor.PushGatewayFactory.push(PushGatewayFactory.java:106)
2185 at com.ebay.raptor.batch.pushgateway.adaptor.PushGatewayFactory.access$100(PushGatewayFactory.java:21)
2186 at com.ebay.raptor.batch.pushgateway.adaptor.PushGatewayFactory$2.run(PushGatewayFactory.java:77)
2187 at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
2188 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
2189 at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
2190 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
2191 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
2192 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2193 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2194 at java.lang.Thread.run(Thread.java:750)
219504:31:

[Bjoern Rabenstein]

On 13.10.23 05:12, Mahendra M wrote:
>
> I'm facing below exception while migrating to prometheus. Could anyone pls
> help here?
>
> Exception:
>
> 218004:31:17.818 [PushGatewayFactory.PushGatewayTaskScheduler-1] ERROR
> c.e.r.b.p.adaptor.PushGatewayFactory - Unable to push metrics to Prometheus
> Pushgateway
> 2181java.io.IOException: Response code from
> http://ingress.sherlock-service.qa.ebay.com/api/pushgateway/v1/metrics/job/testjob
> was 400

So your HTTP endpoint is returning a 400 status code. I actually doubt
that it comes from the Pushgateway itself. Looks more like something
from some proxy or HTTP routing layer in between.

Is the URL correct?

You could use the curl command to push something to that URL manually
and get a more detailed dump of the response. See
https://github.com/prometheus/pushgateway/blob/master/README.md for
curl usage with the Pushgateway.

--
Björn Rabenstein
[PGP-ID] 0x851C3DA17D748D03
[email] bjo...@rabenste.in

[Mahendra M]

Thanks for the reply.

Yes url is proper. Is there specific way of writing metrics to push gateway?

[Brian Candler]

> Is there specific way of writing metrics to push gateway?

Yes, but you are getting an error. Something isn't right, and it's up to you to debug the problem.

Very likely the body of the HTTP error response tells you what's wrong. If it's indeed http:// rather than https:// then at worst you can use tcpdump to inspect it:

tcpdump -i eth0 -nn -s0 -A host ingress.sherlock-service.qa.ebay.com

(Adjust as necessary, e.g. if the interface is not called eth0)