component=tsdb msg="compaction failed" err="persist head block: mkdir /prometheus/aaaa.tmp: permission denied"

Bharanidharan M

unread,

Apr 15, 2021, 9:17:36 AM4/15/21

to Prometheus Users

Hi,

I got this error (mentioned in subject) for a period of 2 weeks in our Kubernetes cluster in Prometheus, this caused all our scraping rules to fail and grafana was not showing any data. After 2 weeks, everything started to work again.

I am trying to understand what could have happened whether I need to increase disk size? It is already 10gi or whether i have to increase memory.. currently it is 2 gi limit and 1 gi request limit.

please help.

thanks in advance.

Regards,

Bharani

Bharanidharan M

unread,

Apr 16, 2021, 6:09:35 AM4/16/21

to Prometheus Users

Hi experts,

Could you please help me with this issue, i am trying to understand what would have caused this issue.. for two weeks of time there is no data logged in prometheus by its scraping rules.

it started working again after 2 weeks, not sure what went wrong. am i right in assuming that it could have been memory , disk issue and just need to increase it to avoid such issues again?

[prometheus-users] component=tsdb msg="compaction failed" err="persist head block: mkdir /prometheus/aaaa.tmp: permission denied"

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/7d21f32e-2e69-41c0-9ccd-710c616a500cn%40googlegroups.com.

Stuart Clark

unread,

Apr 16, 2021, 6:27:18 AM4/16/21

to Bharanidharan M, Prometheus Users

On 2021-04-16 11:08, Bharanidharan M wrote:
> Hi experts,
>
> Could you please help me with this issue, i am trying to understand
> what would have caused this issue.. for two weeks of time there is no
> data logged in prometheus by its scraping rules.
> it started working again after 2 weeks, not sure what went wrong. am i
> right in assuming that it could have been memory , disk issue and just
> need to increase it to avoid such issues again?
>
> [prometheus-users] component=tsdb msg="compaction failed" err="persist
> head block: mkdir /prometheus/aaaa.tmp: permission denied"
>

That error suggests the user/group Prometheus is running as does not
have the correct permissions for the /prometheus directory.

--
Stuart Clark

Bharanidharan M

unread,

Apr 16, 2021, 6:35:46 AM4/16/21

to Stuart Clark, Prometheus Users

thanks a lot Stuart for the reply.

We have not done any change to the environment and it started working again by itself. That is something puzzling and i am not sure how to time travel to check the logs in prometheus, we have configured 30 days retention. i doubt it would be stlll there, or is there a way to get the logs at kubernetes level ?