fhgfs-fsck backward is really slow
Christian Iseli
I'm performing some tests of fhgfs, and one of the test involves running fhgfs-fsck
I have 1 metadata server and 2 storage servers, each storage server has 4 XFS filesystems of 3.7 TB each, used for storage.
I launched "fhgfs-fsck mode=fullcheck"
forward mode took roughly 2 hours, which I suppose is ok (the filesystem was almost full)
backward mode started last Monday (Feb 20 2012) and is still running (Sunday Feb 26).
Is this expected ? How can I know what percentage is done and what remains ?
I also had a look at fragmentation at the XFS level. The frag command of xfs_db reports roughly 2% fragmentation in each filesystem, which I suppose is not bad.
Do you recommend running xfs_fsr at all ?
I could not find detailed documentation of fhgfs-fsck anywhere...
Cheers,
Christian
Christian Mohrbacher
> Hi,
>
> I'm performing some tests of fhgfs, and one of the test involves running
> fhgfs-fsck
>
> I have 1 metadata server and 2 storage servers, each storage server has 4
> XFS filesystems of 3.7 TB each, used for storage.
>
> I launched "fhgfs-fsck mode=fullcheck"
>
> forward mode took roughly 2 hours, which I suppose is ok (the filesystem
> was almost full)
>
> backward mode started last Monday (Feb 20 2012) and is still running
> (Sunday Feb 26).
> Is this expected ? How can I know what percentage is done and what remains
> ?
The runtime of fsck strongly depends on the number of files in your file system. For a runtime calculation that's far more important than the size of the file system and the files. If you have a lot of files a long runtime is possible, because checking the integrity of the file system in backward mode is very communication intensive and time-consuming at the moment.
>
> I could not find detailed documentation of fhgfs-fsck anywhere...
>
A detailed documentation of fsck is work in progress and can be found in the wiki in the next two week, at the latest.
Regards,
Christian
--
=======================================================
Christian Mohrbacher
Competence Center High Performance Computing
Institut fuer Techno- und Wirtschaftsmathematik (ITWM)
Fraunhofer-Platz 1
D-67663 Kaiserslautern
=======================================================
Tel.: (49) 631 31600 4425
Fax: (49) 631 31600 1099
E-Mail: christian....@itwm.fraunhofer.de
Internet: http://www.itwm.fraunhofer.de
=======================================================
Christian Iseli
On Sunday, February 26, 2012 6:04:52 PM UTC+1, Christian Mohrbacher wrote:
The runtime of fsck strongly depends on the number of files in your file system. For a runtime calculation that's far more important than the size of the file system and the files. If you have a lot of files a long runtime is possible, because checking the integrity of the file system in backward mode is very communication intensive and time-consuming at the moment.
Thanks for the explanations.
Any chance that a percent of completion could be displayed and updated while running ?
Christian Mohrbacher
> Any chance that a percent of completion could be displayed and updated
> while running ?
>
Well, in the current version, a percentage indicator would be hard to
implement, as we are not fully aware of the number of chunks in the file
system at runtime (we could guess the number of chunks according to the
number of files in the FhGFS, but even then, we would not know about
orphaned chunks, defective metadata entries, etc.). For the next major
release of FhGFS we are implementing the fsck with a different approach,
which has some improvements. We will try to have some more detailed
statistics at runtime in that version.
Regards,
Christian
--
=====================================================
| Christian Mohrbacher |
| Competence Center for High Performance Computing |
| Institut fuer Techno- und |
| Wirtschaftsmathematik (ITWM) |
| Fraunhofer-Platz 1 |
| |
| D-67663 Kaiserslautern |
=====================================================
| Tel: (49) 631 31600 4425 |
Christian Iseli
Thanks for the quick reply.
On Monday, March 5, 2012 10:45:18 AM UTC+1, Christian Mohrbacher wrote:
Well, in the current version, a percentage indicator would be hard to
implement, as we are not fully aware of the number of chunks in the file
system at runtime (we could guess the number of chunks according to the
number of files in the FhGFS, but even then, we would not know about
orphaned chunks, defective metadata entries, etc.).
I see. In the current tests I'm running, I have:
--------------------------------------------------------------------
Started FhGFS fsck in forward check mode [Tue Feb 28 11:27:01 2012]
Output will be written to /var/log/fhgfs-fsck.log
--------------------------------------------------------------------
Step 1 : Checking reachability of nodes : OK!
Step 2 : Checking metadata for file and directory links :
1255360 file and directory links checked.
--------------------------------------------------------------------
Forward check mode finished
--------------------------------------------------------------------
The reverse check is still running, and currently displays:
Checked 105289750 metadata entries and 1459334 chunks-
Do you have any wild guesstimate whether I'm 10%, half-way, or 90% through ?
For the next major
release of FhGFS we are implementing the fsck with a different approach,
which has some improvements. We will try to have some more detailed
statistics at runtime in that version.
Thanks,
Christian
Christian Mohrbacher
as already discussed in our ticket system, the strange behaviour in your tests was due to a bug in the software. This issue is fixed in the new release 2011.04-r15, which was uploaded just a few minutes ago.
Regards,
Christian
Christian Iseli
I have performed the upgrade, and now fsck goes to completion. The command and output is pasted below.
There are 179 messages in the humanReadableErrorLog output file:
[root@fhgfs04 /tmp/fhgfs_fsck ] wc -l humanReadableErrorLog
179 humanReadableErrorLog
[root@fhgfs04 /tmp/fhgfs_fsck ] head -2 humanReadableErrorLog
6FC0-4F57A8AF-fhgfs.vital-it.ch(fhgfs.vital-it.ch) [Dentry not found for existing Inode]
C6CD-4F57F159-fhgfs.vital-it.ch(fhgfs.vital-it.ch) [Dentry not found for existing Inode]
I suppose this is due to the filesystem being live during the fsck ?
Cheers,
Christian
[root@fhgfs04 ~ ] fhgfs-fsck mode=fullcheck readOnly=true
--------------------------------------------------------------------
Started FhGFS fsck in forward check mode [Wed Mar 7 16:41:51 2012]
Output will be written to /var/log/fhgfs-fsck.log
--------------------------------------------------------------------
Step 1 : Checking reachability of nodes : OK!
Step 2 : Checking metadata for file and directory links :
--------------------------------------------------------------------
Forward check mode finished
--------------------------------------------------------------------
Started FhGFS fsck in backward check mode [Wed Mar 7 16:49:54 2012]
Output will be written to /var/log/fhgfs-fsck.log
--------------------------------------------------------------------
Step 1 : Checking reachability of nodes : OK!
Checked 1297227 metadata entries and 1462025 chunks.
--------------------------------------------------------------------
Pass 1 finished
Starting second pass [Wed Mar 7 16:49:54 2012]
A user readable error log will be written to /tmp/fhgfs_fsck/humanReadableErrorLog.recheck
--------------------------------------------------------------------
Checked 179 metadata entries and 0 chunks.
--------------------------------------------------------------------
Pass 2 finished
--------------------------------------------------------------------
--------------------------------------------------------------------
Backward check mode finished
--------------------------------------------------------------------
Christian Mohrbacher
>
> I have performed the upgrade, and now fsck goes to completion. The command
> and output is pasted below.
>
Good to hear.
> There are 179 messages in the humanReadableErrorLog output file:
>
> I suppose this is due to the filesystem being live during the fsck ?
Yes, this can be due to the fact that you are using the FS. These "false positives" should be sorted out in the second pass, which is performed by the fsck. So, this is ok.
Regards,
Christian
>
> Cheers,
> Christian
>
>
--
=======================================================
Christian Mohrbacher
Competence Center High Performance Computing
Institut fuer Techno- und Wirtschaftsmathematik (ITWM)
Fraunhofer-Platz 1
D-67663 Kaiserslautern
=======================================================
Tel.: (49) 631 31600 4425
Fax: (49) 631 31600 1099
E-Mail: christian....@itwm.fraunhofer.de
Internet: http://www.itwm.fraunhofer.de
=======================================================