Change metadata block size

[Antoine Colombier]

Hi all,

I have recently started a filesystem migration to a new S3QL filesystem using the S3 backend and I went with the default setting for metadata block size.

Unfortunately, the FS is will be storing about 23 millions of files and the database is growing fast (already 28268 block of 64k, and I've barely uploaded a third of the total data)

Since the copy process is due to take 10+ days (already 3 days in), I was wondering if I could change the metadata block after creation, since (if I understand correctly) this is only used to segment the SQLite DB....?

Thanks

[Nikolaus Rath]

In principle it's possible. In practice, no one has written the code to
do this.


Best,
-Nikolaus

[Antoine Colombier]

Hi Nikolaus and thanks for your reply.

I have started looking at the code and as I though I understood how to implement this, I came across upgrade function in src/s3ql/adm.py. Am I correct in thinking that adding some flag allowing to shortcut the following condition would be enough?

elif local_params['revision'] >= CURRENT_FS_REV:

print('File system already at most-recent revision')

return

If yes, is there any risk or potential side effect on upgrading a filesystem again I should be aware of and potentially guard against? At first glance, this function looks idempotent but I'm not yet familiar with all the helper it relies on.

Best,

Antoine

[Nikolaus Rath]

Hi Antoine,

This would work if there were no existing metadata objects in the backend. If you create additional objects with a different blocksize, you'll probably run into trouble. Removing all the existing ones might work, but may be a bit risky if something goes wrong...

Best,

-Nikolaus

--

You received this message because you are subscribed to the Google Groups "s3ql" group.

To unsubscribe from this group and stop receiving emails from it, send an email to s3ql+uns...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/s3ql/f41db2f5-05d6-4b88-8dda-fd061932b349n%40googlegroups.com.