Prometheus data storage above retension size

40 views
Skip to first unread message

João Rua

unread,
May 22, 2020, 12:10:51 PM5/22/20
to Prometheus Users
Hello

i have a prometheus on version 2.14.0 with option "--storage.tsdb.retention.size=350GB" but the data storage at this moment is on 400GB.

400G    /data/prometheus/
796M    /data/prometheus/wal/

I have try the upgrade to the version 2.18.1 but can't resolve this problem, can someone help me?

Thanks

Screenshot from 2020-05-22 17-06-17.png




Stuart Clark

unread,
May 22, 2020, 12:24:59 PM5/22/20
to João Rua, Prometheus Users
Is it staying at that size or varying?

Are there temporarily files due to failed compaction?

Do you have any backup snapshots in that directory?

During compaction you can have various temporary files, but they should go once completed.
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

João Rua

unread,
May 22, 2020, 12:57:49 PM5/22/20
to Prometheus Users


sexta-feira, 22 de Maio de 2020 às 17:24:59 UTC+1, Stuart Clark escreveu:
Is it staying at that size or varying?
 
It still decreases the free free you can see in the attachment the free space in the last 2 days.

Note: data partition is only for prometheus.
 
Are there temporarily files due to failed compaction?

I think not, how i can check that?


Do you have any backup snapshots in that directory?

I have these files in the prometheus 

total 644
drwxr-xr-x    3 65534 65534   4096 Jul  3  2019 01DEVZMSDQC5FKQ722E2280YEK
drwxr-xr-x    3 65534 65534   4096 Jul  6  2019 01DF388SXDD0TH237YA13SWVR5
drwxr-xr-x    3 65534 65534   4096 Jul 26  2019 01DGQCWR2XMDB8TBBME6F082J7
drwxr-xr-x    3 65534 65534   4096 Aug 15  2019 01DJBHE3PDY1VY16Y0G4NGHQK4
drwxr-xr-x    3 65534 65534   4096 Sep  5  2019 01DKZNZFH5PGZDBCJG9SH7F6D8
drwxr-xr-x    3 65534 65534   4096 Sep 25  2019 01DNKTHAG66XWSX4D1J408AA1G
drwxr-xr-x    3 65534 65534   4096 Oct 15  2019 01DQ7Z32CS4QRAWPWPHEPHVKV3
drwxr-xr-x    3 65534 65534   4096 Nov  4  2019 01DRW3RPVS0K1PM5ECJJ5S1MFY
drwxr-xr-x    3 65534 65534   4096 Nov 25 03:05 01DTG810E871YV8ERF1NPMZ6PQ
drwxr-xr-x    3 65534 65534   4096 Dec 15 09:00 01DW4CJKXHC6A2YRXNBTWRW3MK
drwxr-xr-x    3 65534 65534   4096 Jan  4 15:00 01DXRH45Q6FPHZ3HK61YQ31PE6
drwxr-xr-x    3 65534 65534   4096 Jan 24 21:00 01DZCNNRE9F3F6CAWHSSCHC7C7
drwxr-xr-x    3 65534 65534   4096 Feb 14 03:00 01E10T7A3QHF6FFRZGD47PTSXB
drwxr-xr-x    3 65534 65534   4096 Mar  5 09:00 01E2MYRVWN7645VMYDD9Y28Q23
drwxr-xr-x    3 65534 65534   4096 Mar 25 15:00 01E493AD91RS9JTAF2JQF7H5CQ
drwxr-xr-x    3 65534 65534   4096 Apr 14 21:00 01E5X7VZKGDFERBC6JX1YSSK66
drwxr-xr-x    3 65534 65534   4096 May  5 03:00 01E7HCDH81FW6YH2S76HECBS78
drwxr-xr-x    3 65534 65534   4096 May 11 21:00 01E82RKAPYE4JKTKK88WV113G3
drwxr-xr-x    3 65534 65534   4096 May 18 15:00 01E8M4S64MZM1W224JGTS14GG0
drwxr-xr-x    3 65534 65534   4096 May 20 21:01 01E8SY7KF2HGP9PT9HBSQ4BRG0
drwxr-xr-x    3 65534 65534   4096 May 21 15:01 01E8VW0K2Y3T95264KMFH31JKQ
drwxr-xr-x    3 65534 65534   4096 May 22 09:01 01E8XSTEWXDEBQ3QPTAAX7G51B
drwxr-xr-x    3 65534 65534   4096 May 22 15:00 01E8YECDXJ7YM06JEKHTG5RQKC
drwxr-xr-x    3 65534 65534   4096 May 22 15:01 01E8YEDC1PC0SNA9Q2VEWA0HDG
-rw-r--r--    1 65534 65534      0 Apr 30  2019 lock
-rw-r--r--    1 65534 65534  20001 May 22 15:12 queries.active
drwxr-xr-x 8821 65534 65534 540672 May 22 16:30 snapshots
drwxr-xr-x    3 65534 65534  12288 May 22 15:15 wal
 

During compaction you can have various temporary files, but they should go once completed.

Thanks for the help.
Screenshot from 2020-05-22 17-52-02.png

Julien Pivotto

unread,
May 22, 2020, 1:09:10 PM5/22/20
to João Rua, Prometheus Users
Can you also provide the listing of the wal directory?

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/8f06d694-d394-477a-8d05-f9fcbd2f6e63%40googlegroups.com.

João Rua

unread,
May 22, 2020, 2:35:46 PM5/22/20
to Prometheus Users


sexta-feira, 22 de Maio de 2020 às 18:09:10 UTC+1, Julien Pivotto escreveu:
Can you also provide the listing of the wal directory?

 
total 362900
-rw-r--r-- 1 65534 65534   1540096 May 22 15:12 00022551
-rw-r--r-- 1 65534 65534    327680 May 22 15:15 00022552
-rw-r--r-- 1 65534 65534 175865856 May 22 15:15 00022553
-rw-r--r-- 1 65534 65534 126615552 May 22 17:00 00022554
-rw-r--r-- 1 65534 65534  67243222 May 22 18:35 00022555
drwxr-xr-x 2 65534 65534      4096 May 22 17:00 checkpoint.00022550

To unsubscribe from this group and stop receiving emails from it, send an email to promethe...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages