Using space

[Vadim Vorontsov]

I'm using Reiserfs (initialized with mkreiserfs -f -b 4096 -s 513 as recommended in the documentation).

S3backer blocksize is 256k.

When I mount s3 file, s3backer shows "s3backer: found 32085 non-zero blocks".

My S3 provider says there are 32086 objects in my S3 storage with 6.16 Gb used.

But "upper" filesystem has only 3.18 Gb used.

What's can be the issue?

Too little blocksize?

[Archie Cobbs]

This sounds entirely plausible. ReiserFS is using a 4k block size, while s3backer is using a block size 64 times larger, yet you are seeing only a 2x blowup.

In any case, this is really an upper layer filesystem question, not an s3backer question. Not saying it's wrong to ask about it here, just that you might find more expertise elsewhere, e.g., you could ask some filesystem experts how "compactly" various filesystems tend to arrange their blocks on disk, etc.

-Archie

[Vadim Vorontsov]

Ok.

Now I know it's due to ReiserFS tuning.

Thank you!

среда, 28 апреля 2021 г. в 21:33:28 UTC+7, Archie Cobbs:

[Vadim Vorontsov]

Excuse me but I reread s3backer Wiki page where it's recommended to choose even 1Mb block size.

Whilst ReiserFS max block size is 8kb.

So that size increasing is normal?

среда, 28 апреля 2021 г. в 21:33:28 UTC+7, Archie Cobbs:

This sounds entirely plausible. ReiserFS is using a 4k block size, while s3backer is using a block size 64 times larger, yet you are seeing only a 2x blowup.

[Archie Cobbs]

The best choice depends on lots of factors, so it's hard to give absolute advice.

Some of the factors include: your I/O usage patterns, whether you are doing local caching, how big that cache is, upper filesystem type, network bandwidth, how you have s3backer caching configured, etc.

To take a specific example, if you have a big, local cache that's configured for delayed write-back, then it should absorb a lot of the "noise" created by the smaller upper filesystem as it reads and writes tiny 4k blocks. When it finally does come time to write back the data, it does so more efficiently using the larger chunks.

But I'm certainly not an expert... and the true answer can only be found by doing real measurements. Maybe setting up such a performance profiling analysis would be a good project for an undergraduate CS student :)

-Archie