Prometheus CrateDB Adapter - Context Deadline exceeded

[ryan woods]

Hey guys,

So I'm currently running a 3 node CrateDB cluster for Prometheus long term storage and using the CrateDB adapter provided by crateio. I've ran into an issue where Prometheus is attempting to POST data to the remote_write (crate adapter) but it's unable to. Logs are below from CrateDB adapter

time="2020-04-17T08:46:31Z" level=error msg="Failed to POST inserts to Crate." err="context deadline exceeded" source="server.go:332"

time="2020-04-17T08:46:31Z" level=error msg="Failed to POST inserts to Crate." err="context deadline exceeded" source="server.go:332"

time="2020-04-17T08:46:31Z" level=error msg="Failed to POST inserts to Crate." err="context deadline exceeded" source="server.go:332"

time="2020-04-17T08:46:31Z" level=error msg="Failed to POST inserts to Crate." err="context deadline exceeded" source="server.go:332"

time="2020-04-17T08:46:31Z" level=error msg="Failed to POST inserts to Crate." err="context deadline exceeded" source="server.go:332"

time="2020-04-17T08:46:31Z" level=error msg="Failed to POST inserts to Crate." err="context deadline exceeded" source="server.go:332"

Although these were collected from the cratedb adapter I'm thinkig this is a prometheus issue/misconfiguration by myself. Any help would be great :) i've tried increasing the scrape_interval and it doesn't seem to help.

[Julius Volz]

"context deadline exceeded" means there's a timeout between the CrateDB adapter and CrateDB itself. The "POST" is actually misleading, as the adapter has been changed from using HTTP in its original version to using the Postgres protocol via https://github.com/jackc/pgx.

A timeout could either mean there's a network / reachability problem, or that the CrateDB is too overloaded to process write requests in time (I couldn't quickly figure out what the timeout value would be here, but I guess it wouldn't be tiny). Have you tried sending tiny data (like just a handful of series) first, to see if it works at all?

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/b7ba1543-076d-4514-8d19-6dec69eee20c%40googlegroups.com.

[Julius Volz]

FWIW, submitted a PR to remove the mention of "POST": https://github.com/crate/crate_adapter/pull/34

[ryan woods]

Okay cool I'll start looking into CrateDB, although it's showing the CPU usage at roughly 10% the load averages for 1m,5m,15m are 8.5,8,9.

To unsubscribe from this group and stop receiving emails from it, send an email to promethe...@googlegroups.com.

[Brian Candler]

> although it's showing the CPU usage at roughly 10% the load averages for 1m,5m,15m are 8.5,8,9.

The high load average is suggesting it's I/O bound.  Look in "ps auxwww" output for processes in state D.

Also look at "iostat -5 x": ignore the first output which is the average since system boot, look at what it shows every 5 seconds thereafter.  Look in particular for the %util figure in the last column: if that's near 100% for any drive then it's saturated on I/O.