Prometheus and AWS timestream

[Vinod Sai Kumar]

Hi There ,

We have integrated AWS timestream to prometheus via the remote write & read settings. We have the below setup,

1. Prometheus is a pod in EKS cluster

2. We have prometheus-timestream-adapter as well deployed in a pod

3. Prometheus remote read & write is configured to this adapter.

4. AWS timestream has a table setup with 24hrs memory & 7 days magnetic retention

All connectivity, setup etc, looks good apart from the prometheus seems to be sending old record sets. Hence we get this below message in the adapter and it fails to store the data in timestream

{"level":"warn","ts":1610368736.6555033,"caller":"prometheus-timestream-adapter/timestream.go:131","msg":"Error sending samples to remote storage","err":"RejectedRecordsException: One or more records have been rejected. See RejectedRecords for details.\n{\n  RespMetadata: {\n    StatusCode: 419,\n    RequestID: \"AHHJ7P7BBH4TIUEGMKOPO42FQQ\"\n  },\n  Message_: \"One or more records have been rejected. See RejectedRecords for details.\",\n  RejectedRecords: [\n    {\n      Reason: \"The record timestamp is outside the time range [2021-01-10T12:33:56.651Z, 2021-01-11T13:13:56.651Z) of the memory store.\",\n      RecordIndex: 0\n    }

Looking forward for some suggestions ...

Regards,

Vinod.

[Vinod Sai Kumar]

Further analysis...

The records recieved from prometheus by the adapter proves that the records are older than 24 hours

Does prometheus return oldest metrics as well as part of the remote writes ?

For ex: One of the records recieved from prometheus

"MeasureName":"fluentd_output_status_num_errors","MeasureValue":"162","MeasureValueType":"DOUBLE","Time":"1610130823809","TimeUnit":"MILLISECONDS"}]}

The time converts to - Fri Jan 08 2021 18:33:43 Where as we have the retention on AWS timestream set to 24 hrs

Any suggestions will be helpful.

Regards,

Vinod.

[Vinod Sai Kumar]

Further .....

We created a temp table in AWS timestream and increased the retention to 7 days, this allowed the data to be written in the remote storage. What we notice now is the records are always for the same time period that is in our case its between

[2021-01-08 18:33:44.280000000 & 2021-01-08 18:35:14.390000000]Are we missing any configuration somemwhere ?

Regards,

Vinod