Storage Retention 15d for Prometheus

273 views
Skip to first unread message

Bhupendra kumar

unread,
Jul 13, 2020, 5:01:09 AM7/13/20
to Prometheus Users
Hi All,

I want to know how to change Storage Retention in prometheus, And I am already tried with this command ( prometheus --storage.tsdb.retention=20d ) but still not working.
So please give me a solution for this so that prometheus working fine, And also when i need to restart prometheus server then prometheus very slowly start so to do any change is required for this.

Prometheus version.

[root@server prometheus]# prometheus --version
prometheus, version 2.17.1 (branch: HEAD, revision: ae041f97cfc6f43494bed65ec4ea4e3a0cf2ac69)
  build user:       root@806b02dfe114
  build date:       20200326-16:18:19
  go version:       go1.13.9
 

Prometheus.JPG

Julius Volz

unread,
Jul 13, 2020, 6:58:01 AM7/13/20
to Bhupendra kumar, Prometheus Users
Hi,

The flag "--storage.tsdb.retention=20d" should definitely work (although "--storage.tsdb.retention.time=20d" is preferred nowadays). If the /status page still says "15d", are you sure you restarted the right Prometheus server?

Long startup times come from write-ahead-log (WAL) replay. There have been improvements to the WAL size recently, so maybe also try the latest Prometheus version to get startup times lowered.

Cheers,
Julius

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/e336cb8c-2789-47f9-8e20-e4828ab123d8o%40googlegroups.com.


--
Julius Volz
PromLabs - promlabs.com

Bhupendra kumar

unread,
Jul 13, 2020, 9:18:31 AM7/13/20
to Prometheus Users
Yes, I restart the Prometheus service still showing me 15d. Please check the snapshot.
Prometheus.JPG

Julien Pivotto

unread,
Jul 13, 2020, 9:24:37 AM7/13/20
to Bhupendra kumar, Prometheus Users
Can we get the full log?

Thanks
> --
> You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/725c76cb-758b-4ba0-8826-b4c14d262f70o%40googlegroups.com.



--
Julien Pivotto
@roidelapluie

Bhupendra kumar

unread,
Jul 13, 2020, 9:34:50 AM7/13/20
to Prometheus Users
Yes


On Monday, 13 July 2020 18:54:37 UTC+5:30, Julien Pivotto wrote:
Can we get the full log?

Thanks


On 13 Jul 06:18, Bhupendra kumar wrote:
> Yes, I restart the Prometheus service still showing me 15d. Please check
> the snapshot.
>
> On Monday, 13 July 2020 14:31:09 UTC+5:30, Bhupendra kumar wrote:
> >
> > Hi All,
> >
> > I want to know how to change Storage Retention in prometheus, And I am
> > already tried with this command ( prometheus --storage.tsdb.retention=20d )
> > but still not working.
> > So please give me a solution for this so that prometheus working fine, And
> > also when i need to restart prometheus server then prometheus very slowly
> > start so to do any change is required for this.
> >
> > Prometheus version.
> >
> > [root@server prometheus]# prometheus --version
> > prometheus, version 2.17.1 (branch: HEAD, revision:
> > ae041f97cfc6f43494bed65ec4ea4e3a0cf2ac69)
> >   build user:       root@806b02dfe114
> >   build date:       20200326-16:18:19
> >   go version:       go1.13.9
> >  
> >
> >
>
> --
> You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to promethe...@googlegroups.com.
Prometheus.docx

Julien Pivotto

unread,
Jul 13, 2020, 9:47:22 AM7/13/20
to Bhupendra kumar, Prometheus Users
On 13 Jul 06:34, Bhupendra kumar wrote:
> Yes


Mmmh can you use --log.level=debug

and look for the line:

level=debug ts=2020-07-13T13:46:04.886Z caller=main.go:702 msg="TSDB options" MinBlockDuration=2h
MaxBlockDuration=36h MaxBytes=0B NoLockfile=false RetentionDuration=15d
WALSegmentSize=0B AllowOverlappingBlocks=false WALCompression=true
> > an email to promethe...@googlegroups.com <javascript:>.
> > > To view this discussion on the web visit
> > https://groups.google.com/d/msgid/prometheus-users/725c76cb-758b-4ba0-8826-b4c14d262f70o%40googlegroups.com.
> >
> >
> >
> >
> > --
> > Julien Pivotto
> > @roidelapluie
> >
>
> --
> You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/b98c126e-af21-48e5-a5f0-45e5cde9002bo%40googlegroups.com.



--
Julien Pivotto
@roidelapluie

Bhupendra kumar

unread,
Jul 13, 2020, 9:58:18 AM7/13/20
to Prometheus Users
Yes pls check.
> To unsubscribe from this group and stop receiving emails from it, send an email to promethe...@googlegroups.com.
Prometheus.JPG

Christian Hoffmann

unread,
Jul 13, 2020, 11:12:31 AM7/13/20
to Bhupendra kumar, Prometheus Users
On 7/13/20 3:58 PM, Bhupendra kumar wrote:
> Yes pls check.
The last line looks like the restart was not successful. This might mean
that Prometheus is still running with an older cmdline. Can you check
the process list?

ps aux | grep prometheus or something? I suspect it might still say 15d.

If this is the case, I suggest debuggin the restart problem. Looks like
you are using systemd. What unit are you using? Has Prometheus always
been started using the systemd unit?

Does it work reliably if you kill Prometheus once and start/restart it
using systemd?

Kind regards,
Christian
Reply all
Reply to author
Forward
0 new messages