Re: How to Clean Up .run / .index Files

[Konstantin Osipov]

* Mai Abdel-Qader <mai.abd...@gmail.com> [19/01/09 15:28]:

Please set box.checkpoint_count to 1 and run box.snapshot().

>
> I noticed today that .run and .index files on my instance are consuming 30
> G of disk, however I my queue and tuples are empty.
>
> I checked the .run / .index files, and noticed that I have many of these
> files, and some of them are old files since months.
>
> Shouldn't these files cleaned up with related .snap files which I already
> only keep the latest 6 of them?
>
> wal_mode = "write";
> wal_max_size = 256 * 1024 * 1024;
>
> checkpoint_interval = 60 * 60; -- one hour
> checkpoint_count = 6;
>
> Can you please help me how to clean them up? And why I have these files
> since months?
>
> Thanks a lot,
> Mai
>
> --
> You received this message because you are subscribed to the Google Groups "Tarantool discussion group (English)" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to tarantool+...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.


--
Konstantin Osipov, Moscow, Russia, +7 903 626 22 32
http://tarantool.io - www.twitter.com/kostja_osipov

[Konstantin Osipov]

* Mai Abdel-Qader <mai.abd...@gmail.com> [19/01/09 16:20]:
> I forget to mention that I am using version 1.7

1.7 is very old if you're using vinyl engine. Please upgrade to
1.10.

[Mai Abdel-Qader]

Thanks a lot, It is worked fine.

However, the size of  .run and .index files are consuming big size of disk, it is only reduced to 29 G although the tuples and queue are empty now.

I insert into the tuples data, and read it between 9 AM to 3 PM, and then clean the read data.

what might cause the huge consuming of data?

[Konstantin Osipov]

* Mai Abdel-Qader <mai.abd...@gmail.com> [19/01/10 10:22]:

> Thanks a lot, It is worked fine.
>
> However, the size of .run and .index files are consuming big size of disk,
> it is only reduced to 29 G although the tuples and queue are empty now.
>
> I insert into the tuples data, and read it between 9 AM to 3 PM, and then
> clean the read data.
>
> what might cause the huge consuming of data?

Vinyl garbage collection in 1.7 is very primitive, temporary files
are only removed after snapshot. Please upgrade.

[Mai Abdel-Qader]

Thanks a lot. I will.

On Wednesday, 9 January 2019 17:09:57 UTC+3, Konstantin Osipov wrote:

[Mai Abdel-Qader]

We upgraded it 1.10 but we are still facing same issue.

Can you please help me if there something further can be done here?

[Konstantin Osipov]

* Mai Abdel-Qader <mai.abd...@gmail.com> [19/05/07 14:43]:

>
> We upgraded it 1.10 but we are still facing same issue.
>
> Can you please help me if there something further can be done here?

try issuing index:compact() for each index on the space in
question, including the primary key.

[Mai Abdel-Qader]

I did compact on indices. But this doesn't help at all. The size has not been decreased.

Can you please help me if there is something else I can do? Currently, I clean up the files manually which isn't a practical solution.

On Tuesday, 7 May 2019 19:24:46 UTC+3, Konstantin Osipov wrote:

* Mai Abdel-Qader <mai.ab...@gmail.com> [19/05/07 14:43]:

[Mai Abdel-Qader]

I just tried it again, and it reduce the size.

But shouldn't should be an automatically way to do this instead of keep doing it manually?

[Konstantin Osipov]

* Mai Abdel-Qader <mai.abd...@gmail.com> [19/05/10 07:41]:

> I just tried it again, and it reduce the size.
>
> But shouldn't should be an automatically way to do this instead of keep
> doing it manually?

It is done automatically, unless you switched the checkpoint daemon off.

Please check your checkpoint_count and checkpoint_interval. Over
checkpoint_count * checkpoint_interval, the garbage is removed by
the checkpoint daemon.

[Mai Abdel-Qader]

These are my values, shouldn't this mean to clean the old files each 1 hour?

checkpoint_interval = 60 * 60; -- one hour

checkpoint_count = 1;

"unless you switched the checkpoint daemon off. "

From where can I check if this switched off?

On Friday, 10 May 2019 07:58:01 UTC+3, Konstantin Osipov wrote:

* Mai Abdel-Qader <mai.ab...@gmail.com> [19/05/10 07:41]:

[Konstantin Osipov]

* Mai Abdel-Qader <mai.abd...@gmail.com> [19/05/13 00:04]:

>
> These are my values, shouldn't this mean to clean the old files each 1 hour?
>
> checkpoint_interval = 60 * 60; -- one hour
> checkpoint_count = 1;
>
> "unless you switched the checkpoint daemon off. "
>
> From where can I check if this switched off?

If you have the settings above, it's on.

[Mai Abdel-Qader]

I noticed that the size will reduce significantly if I did compact for the indices, and then do box.snapshot(). One of them without the others will not effect the size.

Does that lead to any conclusion?

On Monday, 13 May 2019 00:12:35 UTC+3, Konstantin Osipov wrote:

* Mai Abdel-Qader <mai.ab...@gmail.com> [19/05/13 00:04]:

[Konstantin Osipov]

* Mai Abdel-Qader <mai.abd...@gmail.com> [19/05/13 10:44]:

>
> I noticed that the size will reduce significantly if I did compact for the
> indices, and then do box.snapshot(). One of them without the others will
> not effect the size.
>
> Does that lead to any conclusion?

index:compact() forces a major compaction. A major compaction
creates a compact representation of index data on disk.
box.snapshot() creates a checkpoint, which allows to delete old
files as long as they are not used by the checkpoint. The old
files compacted by index:compact() are indeed not used in the new
checkpoint - the new checkpoint references the new files already.
This is how and why the old files are deleted.