clone_fs exception / old metadata files

[Shannon Dealy]

Hello Nikolaus,

I am trying to copy a large S3QL file system from the "local" backend to the
"s3c" backend using "clone_fs.py". I made a small test file system to try
it out which worked fine, but with the real file system it crashes immediately
with the following exception:

s3ql.backends.common.CorruptedObjectError: Invalid object header:
b'\x80\x02}q\x00(X\x0b\x00'

The file name it was attempting to process was:

s3ql_metadata_bak_5_pre21

I assume from the "_pre21" suffix in the file name and the 2014 date stamp
that this is an old metadata file backup made while upgrading to version 21 of
the file system. If my assumption is correct, then presumably the exception
was caused by the fact that this file is from an old/incompatible version of
the file system. Looking at the top level directory for the S3QL file system,
there are a number of other files with not only the "_pre21" suffix, but with
names that include "_pre2.13_metadata_" as well as some with "#" in the name
such as these:

s3ql_pre2.13_metadata
s3ql_metadata_bak_2#23037-140201956722432

This leads me to the following questions:

1 - Am I correct in assuming that all of these files can/should be deleted?
Presumably this would fix my clone_fs problem.

2 - Should there be (or is there already) some way of listing and/or cleaning
up cruft files in the S3QL file system metadata directory.

3 - Assuming I am correct and all of the above files should be discarded,
perhaps some documentation should be added to the clone_fs program about
this issue (not sure if this affects other programs). Ideally it would
actually recognize that these files can be skipped, but I know that more
work is the last thing you need.

Thanks for all the work you do on S3QL.

Best regards,

Shannon C. Dealy | DeaTech Research Inc.
de...@deatech.com | Biotechnology Development Services
Telephone USA: +1 541-929-4089 | USA and the Netherlands
Netherlands: +31 85 208 5570 | www.deatech.com

[Nikolaus Rath]

On Aug 01 2019, Shannon Dealy <de...@deatech.com> wrote:
> Hello Nikolaus,
>
> I am trying to copy a large S3QL file system from the "local" backend to the "s3c" backend
> using "clone_fs.py". I made a small test file system to try it out which worked fine, but
> with the real file system it crashes immediately
> with the following exception:
>
> s3ql.backends.common.CorruptedObjectError: Invalid object header:
> b'\x80\x02}q\x00(X\x0b\x00'
>
> The file name it was attempting to process was:
>
> s3ql_metadata_bak_5_pre21
>
> I assume from the "_pre21" suffix in the file name and the 2014 date stamp that this is an
> old metadata file backup made while upgrading to version 21 of the file system. If my
> assumption is correct, then presumably the exception
> was caused by the fact that this file is from an old/incompatible version of the file
> system.

That's right.

> Looking at the top level directory for the S3QL file system, there are a number of
> other files with not only the "_pre21" suffix, but with names that include
> "_pre2.13_metadata_" as well as some with "#" in the name such as these:
>
> s3ql_pre2.13_metadata
> s3ql_metadata_bak_2#23037-140201956722432
>
> This leads me to the following questions:
>
> 1 - Am I correct in assuming that all of these files can/should be deleted?
> Presumably this would fix my clone_fs problem.

Yes.

> 2 - Should there be (or is there already) some way of listing and/or cleaning
> up cruft files in the S3QL file system metadata directory.

There is no tool, since there should be almost none of these files.

The files with '#' in there should not exist unless mount.s3ql is
hard-crashed (SIGKILL or power cycle). The _pre files were only created
during one upgrade process.

> 3 - Assuming I am correct and all of the above files should be discarded,
> perhaps some documentation should be added to the clone_fs program about
> this issue (not sure if this affects other programs). Ideally it would
> actually recognize that these files can be skipped, but I know that more
> work is the last thing you need.

Yeah, it'd be a good idea for clone_fs to just skip over the files. Pull
requests welcome :-).


Best,
-Nikolaus

--
GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

»Time flies like an arrow, fruit flies like a Banana.«