Wal directory

Timur Shteinberg

unread,

Mar 10, 2021, 10:09:49 AM3/10/21

to Prometheus Users

Help i have prometheus 2.12 version in azure cluster

after update cluster node pool i have this error :

err="opening storage failed: repair corrupted WAL: cannot handle error: open WAL segment: 2202: open /prometheus/wal/00002202: no such file or directory"

and pod crushes

Matthias Rampke

unread,

Mar 12, 2021, 8:51:33 AM3/12/21

to Timur Shteinberg, Prometheus Users

Try deleting /prometheus/wal altogether. You will lose some data but at this point the gap from Prometheus not starting is worse.

Generally, Prometheus 2.12 is quite old, consider upgrading to the latest version – there have been lots of WAL related improvements since then.

/MR

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/d7541b4f-ed07-4e29-8fa8-8b77802a4bc5n%40googlegroups.com.

Julien Pivotto

unread,

Mar 12, 2021, 8:53:49 AM3/12/21

to Matthias Rampke, Timur Shteinberg, Prometheus Users

On 12 Mar 13:51, Matthias Rampke wrote:
> Try deleting /prometheus/wal altogether. You will lose some data but at
> this point the gap from Prometheus not starting is worse.
>
> Generally, Prometheus 2.12 is quite old, consider upgrading to the latest
> version – there have been lots of WAL related improvements since then.

It might also indicate that you are using an unsupported file system,
such as NFS. You should use a posix-compliant non-nfs filesystem.

>
> /MR
>
> On Wed, Mar 10, 2021 at 3:09 PM Timur Shteinberg <ti...@a-heads.eu> wrote:
>
> > Help i have prometheus 2.12 version in azure cluster
> > after update cluster node pool i have this error :
> > err="opening storage failed: repair corrupted WAL: cannot handle error:
> > open WAL segment: 2202: open /prometheus/wal/00002202: no such file or
> > directory"
> > and pod crushes
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Prometheus Users" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to prometheus-use...@googlegroups.com.
> > To view this discussion on the web visit
> > https://groups.google.com/d/msgid/prometheus-users/d7541b4f-ed07-4e29-8fa8-8b77802a4bc5n%40googlegroups.com

> > <https://groups.google.com/d/msgid/prometheus-users/d7541b4f-ed07-4e29-8fa8-8b77802a4bc5n%40googlegroups.com?utm_medium=email&utm_source=footer>
> > .

> >
>
> --
> You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAMV%3D_gaacerQvocd%2ByP%3DyadxrEfV4hQgFtcng3oVSnE35CDUSA%40mail.gmail.com.

--
Julien Pivotto
@roidelapluie

Timur Shteinberg

unread,

Mar 13, 2021, 3:10:13 AM3/13/21

to Julien Pivotto, Matthias Rampke, Prometheus Users

Thank you

I can't delete prometheus/wal, pod just restart after 5 seconds because this error , I tried to mount another folder but error still comes back