OSD Drain and Log

rberengu...@gmail.com

unread,

May 7, 2018, 3:30:29 AM5/7/18

to XtreemFS

Hello,

I have a basic structure with XtreemFS. A server with DIR, MCR and OSD, and another server with other OSD. All works fine, I have information stripped throw both OSD, but now I want to drain my first OSD, and I get this :

xtfs_remove_osd -d -dir localhost:32638 uuid:334094f6-7d98-409e-bdec-ac07c87c8904

[ W | OSDDrain             | main            |   1 | May 07 09:23:20 ] java.io.IOException: sending RPC failed: request timed out
...                                           org.xtreemfs.foundation.pbrpc.client.RPCResponse.get(RPCResponse.java:69)
...                                           org.xtreemfs.osd.drain.OSDDrain.getFileListOfOSD(OSDDrain.java:290)
...                                           org.xtreemfs.osd.drain.OSDDrain.drain(OSDDrain.java:146)
...                                           org.xtreemfs.utils.xtfs_remove_osd.drainOSD(xtfs_remove_osd.java:295)
...                                           org.xtreemfs.utils.xtfs_remove_osd.main(xtfs_remove_osd.java:189)
[ E | OSDDrain             | main            |   1 | May 07 09:23:20 ] Failed to get filelist from OSD
ERROR: An error accured during the OSD drain process. See logging outputfor details. It is NOT save to shutdown the OSD.
[ D | OSDDrain             | main            |   1 | May 07 09:23:20 ] org.xtreemfs.osd.drain.OSDDrainException: sending RPC failed: request timed out
...                                           org.xtreemfs.osd.drain.OSDDrain.getFileListOfOSD(OSDDrain.java:295)
...                                           org.xtreemfs.osd.drain.OSDDrain.drain(OSDDrain.java:146)
...                                           org.xtreemfs.utils.xtfs_remove_osd.drainOSD(xtfs_remove_osd.java:295)
...                                           org.xtreemfs.utils.xtfs_remove_osd.main(xtfs_remove_osd.java:189)

Both servers are Ubuntu, could be a problem with JAVA version? xtreemfs-tools asks for 1.6 version, actually I've 1.8. Or its a problem with MCR or the OSD, I can't find any log about "Failed to get filelist from OSD", where I should look?

Thanks in advance,

Robert Schmidtke

unread,

May 8, 2018, 2:55:00 AM5/8/18

to XtreemFS

Hi,

this should not be a problem with Java. First thing that comes to mind is, try using pbrpc://localhost:32638 as the DIR uri.

Do you happen to have the MRC log output?

Do you have SSL configured? If so, you need to supply additional arguments to xtfs_remove_osd.

Are you sure the machine that you're running the xtfs_remove_osd command from has the MRC running at localhost, port 32638?

Hope this helps.

Cheers

Robert

rberengu...@gmail.com

unread,

May 8, 2018, 5:00:45 AM5/8/18

to XtreemFS

Hi Robert,

MRC doesn't says anithing about OSDDrain, even doesn't change after running OSDDrain. There's no SSL. Everything is configured in a local network :

MRC - 192.168.1.7:32636
DIR - 192.168.1.7:32638
OSD1 - 192.168.1.7:32640
OSD2 - 192.168.1.5:32640

A very basic configuration. It doesn't matter if I use localhost or the ip, I get the same result. If I don't set the port I only get this :

root@xtreemfs:/etc/xos/xtreemfs# xtfs_remove_osd -d -dir pbrpc://localhost uuid:334094f6-7d98-409e-bdec-ac07c87c8904
[ I | TimeSync             | TSync Thr       |   8 | May 08 10:57:42 ] Thread TSync Thr started
[ I | TimeSync             | TSync Thr       |   8 | May 08 10:57:42 ] TimeSync is running using the local clock
[ I | RPCNIOSocketClient   | xtfs_remove_... |   9 | May 08 10:57:42 ] Thread xtfs_remove_osd (dir) started

But nothing happens.

Thanks,

Robert Schmidtke

unread,

May 8, 2018, 5:31:36 AM5/8/18

to XtreemFS

Hi,

so it appears the OSD is not reachable by the tool ... Is it the same error for both of your OSDs? I.e. have you tried using both UUIDs?

Have you tried running the tool on the other machine (also with both UUIDs)?

Is the installation otherwise functional, i.e. can you store and retrieve files to/from the OSDs without any issue?

Leaving out the port in the DIR uri seems to cause an error, yes, thanks for pointing that out.

Cheers

Robert

rberengu...@gmail.com

unread,

May 8, 2018, 6:23:23 AM5/8/18

to XtreemFS

...                                           org.xtreemfs.foundation.pbrpc.client.RPCResponse.get(RPCResponse.java:69)
...                                           org.xtreemfs.osd.drain.OSDDrain.getFileListOfOSD(OSDDrain.java:290)
...                                           org.xtreemfs.osd.drain.OSDDrain.drain(OSDDrain.java:146)
...                                           org.xtreemfs.utils.xtfs_remove_osd.drainOSD(xtfs_remove_osd.java:295)
...                                           org.xtreemfs.utils.xtfs_remove_osd.main(xtfs_remove_osd.java:189)

[ E | OSDDrain | main | 1 | May 08 10:51:34 ] Failed to get filelist from OSD

ERROR: An error accured during the OSD drain process. See logging outputfor details. It is NOT save to shutdown the OSD.

[ D | OSDDrain | main | 1 | May 08 10:51:34 ] org.xtreemfs.osd.drain.OSDDrainException: sending RPC failed: request timed out

...                                           org.xtreemfs.osd.drain.OSDDrain.getFileListOfOSD(OSDDrain.java:295)
...                                           org.xtreemfs.osd.drain.OSDDrain.drain(OSDDrain.java:146)
...                                           org.xtreemfs.utils.xtfs_remove_osd.drainOSD(xtfs_remove_osd.java:295)
...                                           org.xtreemfs.utils.xtfs_remove_osd.main(xtfs_remove_osd.java:189)

Robert Schmidtke

unread,

May 8, 2018, 6:31:18 AM5/8/18

to XtreemFS

Quick question: which version of XtreemFS are you using?

rberengu...@gmail.com

unread,

May 8, 2018, 7:25:32 AM5/8/18

to XtreemFS

Hi,

I think the last one 1.5.1. Installed on Ubuntu Server 16.04 throught repo http://download.opensuse.org/repositories/home:/xtreemfs/xUbuntu_16.04/

Robert Schmidtke

unread,

May 8, 2018, 7:57:45 AM5/8/18

to XtreemFS

Hi,

could you try the latest 1.6.0.6 packages available at: http://download.opensuse.org/repositories/home:/xtreemfs:/unstable/ ?

There have been some bug fixed in between that might resolve your issue (I cannot guarantee it though).