xtfs_remove_osd

[eckhar...@uni-due.de]

Hi,
I have a test installation (xtfs1.4) with 1 DIR, 1 MRC on host cs3 and 9 OSD slaves.
A file system testfs with read/write replication and quorum (3 replicas; see below) has been created.

When I try to clean out one of the OSD nodes, I obtain an error more often than not (sometimes it works, usually it doesn't; see below),
although I have plenty of space on each one of the OSD nodes. What seems to happen is that more and more replicas are allocated
on other nodes (some partial, some full) but none is erased on the OSD to be cleaned.

Can somebody give me a hint what I'm doing wrong?

Thanks a lot
Eckhard


-------------------------------
FILE SYSTEM PROPERTIES:
XtreemFS URL         pbrpc://cs3.theochem.uni-due.de:32638/testfs
Owner                root
Group                root
Type                 volume
Free/Used Space      4 TB / 36 kB
Num. Files/Dirs      23 / 6
Access Control p.    POSIX (permissions & ACLs)
OSD Selection p.     1000,3003
Replica Selection p. 3003
Default Striping p.  STRIPING_POLICY_RAID0 / 1 / 128kB
Default Repl. p.     WqRq with 3 replicas
Snapshots enabled    no
Selectable OSDs      0e245182-6729-4f4b-bf16-d93aca5c8230 (192.168.13.31:32640)
                     2efe4aa3-d473-4629-918b-dd752ae91aa8 (132.252.87.122:32640)
                     3deadbe6-cc4a-4f52-ade8-0382ae300f49 (132.252.87.121:32640)
                     7cd0f69d-30fd-44f7-8e44-7b81cc27f33e (192.168.13.37:32640)
                     8f311ef7-c7e7-4b7a-9793-b80b184c2d0c (192.168.13.36:32640)
                     dae55360-7a54-4dfb-a909-f21855ffb1f8 (192.168.13.33:32640)
                     e36f39c6-1424-43a4-973f-4c780ede2113 (192.168.13.35:32640)
                     ede0105f-7fd0-445c-872d-1cfa1b5605b1 (192.168.13.34:32640)
                     fdf9ba95-b1d2-454e-b21a-b111b301835c (192.168.13.32:32640)

-------------------------------------------
CLEANOUT:
root@cs3:/imports/testfs# xtfs_remove_osd -s uuid:7cd0f69d-30fd-44f7-8e44-7b81cc27f33e
[ E | OSDDrain             | main            |   1 | Feb 08 11:41:59 ] Failed to remove original replicas
ERROR: An error accured during the OSD drain process. See logging outputfor details. It is NOT save to shutdown the OSD.
[ E | OSDDrain             | main            |   1 | Feb 08 11:41:59 ] From the following files the changes to the originalreplica couldn't be reverted:
 96208164-e81f-43d1-8d2b-5d181f6c6145:14
[ E | OSDDrain             | main            |   1 | Feb 08 11:41:59 ] From following files the newly created replicas couldn't be removed:
 96208164-e81f-43d1-8d2b-5d181f6c6145:14
 96208164-e81f-43d1-8d2b-5d181f6c6145:14
....

[Michael Berlin]

Dear Eckhard,

You spotted a black spot in the current implementation. Unfortunately,
it is currently not possible to use xtfs_remove_osd with R/W replicated
files. However, the tool currently does not ignore R/W replicated files
and therefore it did not succeed. I'm sorry for the inconvenience.

So far, this hasn't been documented. Therefore, I created a bug report
for this issue: http://code.google.com/p/xtreemfs/issues/detail?id=273

We are currently fixing issues in XtreemFS regarding the addition and
removal of R/W replicated files. After that, it will be fairly easy to
adapt the xtfs_remove_osd and add support for R/W replicated files. Feel
free to star the issue to get notified when its status changes.

Best regards,
Michael

> --
>
> ---
> You received this message because you are subscribed to the Google
> Groups "XtreemFS" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to xtreemfs+u...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[eckhar...@uni-due.de]

Dear Michael,
thanks a lot for the clarification. I'll wait what for things to happen.
Eckhard

[xfu...@gmail.com]

Is there any workaround?

Will the following work?

1. Change all things to RO replica

2. xtfs_remove_osd

3. Convert them back to RW replica

Thanks

Michael Berlin於 2013年2月8日星期五UTC+8下午10時08分24秒寫道：

[Michael Berlin]

Hi,

No, the MRC will prevent you from switching between a RO and RW update
policy with the following error:

"Currently, it is not possible to change from a read-only to a
read/write replication policy or vise versa."

There are good reasons for doing so, at least in case of WqRq files:
Since only a majority of replicas have to be updated, it is possible
that a replicated chunk of the file is no longer available on the
majority of replicas - and then inconsistencies at file access may be
possible.

To prevent these problems, the addition or removal of replicas has to
ensure that the data is replicated to the majority of old and new
replicas. Johannes is currently working on this. After he finished the
implementation in his branch, he'll merge it back to the master and the
functionality will be part of a future XtreemFS release.

Best regards,
Michael

On 05/12/2013 02:55 AM, xfu...@gmail.com wrote:
> Is there any workaround?
>
> Will the following work?
>
> 1. Change all things to RO replica
> 2. xtfs_remove_osd
> 3. Convert them back to RW replica
>
> Thanks
>

> Michael Berlin嚙踝蕭 2013嚙羯2嚙踝蕭8嚙踝蕭P嚙踝蕭嚙踝蕭UTC+8嚙磊嚙踝蕭10嚙踝蕭08嚙踝蕭24嚙踝蕭g嚙瘩嚙瘦

>
> Dear Eckhard,
>
> You spotted a black spot in the current implementation. Unfortunately,
> it is currently not possible to use xtfs_remove_osd with R/W replicated
> files. However, the tool currently does not ignore R/W replicated files
> and therefore it did not succeed. I'm sorry for the inconvenience.
>
> So far, this hasn't been documented. Therefore, I created a bug report
> for this issue:
> http://code.google.com/p/xtreemfs/issues/detail?id=273
> <http://code.google.com/p/xtreemfs/issues/detail?id=273>
>
> We are currently fixing issues in XtreemFS regarding the addition and
> removal of R/W replicated files. After that, it will be fairly easy to
> adapt the xtfs_remove_osd and add support for R/W replicated files.
> Feel
> free to star the issue to get notified when its status changes.
>
> Best regards,
> Michael
>

> On 02/08/2013 12:58 PM, eckhar...@uni-due.de <javascript:> wrote:
> > Hi,
> > I have a test installation (xtfs1.4) with 1 DIR, 1 MRC on host cs3 and 9
> > OSD slaves.
> > A file system testfs with read/write replication and quorum (3 replicas;
> > see below) has been created.
> >
> > When I try to clean out one of the OSD nodes, I obtain an error more
> > often than not (sometimes it works, usually it doesn't; see below),
> > although I have plenty of space on each one of the OSD nodes. What seems
> > to happen is that more and more replicas are allocated
> > on other nodes (some partial, some full) but none is erased on the OSD
> > to be cleaned.
> >
> > Can somebody give me a hint what I'm doing wrong?
> >
> > Thanks a lot
> > Eckhard
> >
> >
> > -------------------------------
> > FILE SYSTEM PROPERTIES:
> > XtreemFS URL pbrpc://cs3.theochem.uni-due.de:32638/testfs

> <http://cs3.theochem.uni-due.de:32638/testfs>

> > Owner root
> > Group root
> > Type volume
> > Free/Used Space 4 TB / 36 kB
> > Num. Files/Dirs 23 / 6
> > Access Control p. POSIX (permissions & ACLs)
> > OSD Selection p. 1000,3003
> > Replica Selection p. 3003
> > Default Striping p. STRIPING_POLICY_RAID0 / 1 / 128kB
> > Default Repl. p. WqRq with 3 replicas
> > Snapshots enabled no
> > Selectable OSDs 0e245182-6729-4f4b-bf16-d93aca5c8230

> > (192.168.13.31:32640 <http://192.168.13.31:32640>)
> > 2efe4aa3-d473-4629-918b-dd752ae91aa8
> > (132.252.87.122:32640 <http://132.252.87.122:32640>)
> > 3deadbe6-cc4a-4f52-ade8-0382ae300f49
> > (132.252.87.121:32640 <http://132.252.87.121:32640>)
> > 7cd0f69d-30fd-44f7-8e44-7b81cc27f33e
> > (192.168.13.37:32640 <http://192.168.13.37:32640>)
> > 8f311ef7-c7e7-4b7a-9793-b80b184c2d0c
> > (192.168.13.36:32640 <http://192.168.13.36:32640>)
> > dae55360-7a54-4dfb-a909-f21855ffb1f8
> > (192.168.13.33:32640 <http://192.168.13.33:32640>)
> > e36f39c6-1424-43a4-973f-4c780ede2113
> > (192.168.13.35:32640 <http://192.168.13.35:32640>)
> > ede0105f-7fd0-445c-872d-1cfa1b5605b1
> > (192.168.13.34:32640 <http://192.168.13.34:32640>)
> > fdf9ba95-b1d2-454e-b21a-b111b301835c
> > (192.168.13.32:32640 <http://192.168.13.32:32640>)

> >
> > -------------------------------------------
> > CLEANOUT:
> > root@cs3:/imports/testfs# xtfs_remove_osd -s
> > uuid:7cd0f69d-30fd-44f7-8e44-7b81cc27f33e
> > [ E | OSDDrain | main | 1 | Feb 08 11:41:59 ]
> > Failed to remove original replicas
> > ERROR: An error accured during the OSD drain process. See logging
> > outputfor details. It is NOT save to shutdown the OSD.
> > [ E | OSDDrain | main | 1 | Feb 08 11:41:59 ]
> > From the following files the changes to the originalreplica couldn't be
> > reverted:
> > 96208164-e81f-43d1-8d2b-5d181f6c6145:14
> > [ E | OSDDrain | main | 1 | Feb 08 11:41:59 ]
> > From following files the newly created replicas couldn't be removed:
> > 96208164-e81f-43d1-8d2b-5d181f6c6145:14
> > 96208164-e81f-43d1-8d2b-5d181f6c6145:14
> > ....
> >
> >
> > --
> >
> > ---
> > You received this message because you are subscribed to the Google
> > Groups "XtreemFS" group.
> > To unsubscribe from this group and stop receiving emails from it, send

> > an email toxtreemfs+u...@googlegroups.com <javascript:>.
> > For more options, visithttps://groups.google.com/groups/opt_out
> <https://groups.google.com/groups/opt_out>.