mogilefs fuse Input/output error

[李帅]

hi~all

i use the mogilefs fuse,mount domain to dir:

mount-mogilefs --tracker test.mogilefs.idc.cn --domain domain1 /mnt/tmp

# ls /mnt/tmp

ls: reading directory /mnt/tmp: Input/output error 

what can i debug it?

[李帅]

i use:

mount-mogilefs --log log.txt  --mountopt allow_other --tracker test.mogilefs.idc.cn --domain test /mnt/tmp  --verbose --verbose --verbose --verbose

log:

FUSE library version: 2.8.3

nullpath_ok: 0

unique: 1, opcode: INIT (26), nodeid: 0, insize: 56

INIT: 7.13

flags=0x0000007b

max_readahead=0x00020000

   INIT: 7.12

   flags=0x00000011

   max_readahead=0x00020000

   max_write=0x00020000

   unique: 1, success, outsize: 40

getattr /

fuse_getattr("/")

   unique: 2, success, outsize: 120

unique: 3, opcode: OPENDIR (27), nodeid: 1, insize: 48

   unique: 3, success, outsize: 32

unique: 4, opcode: READDIR (28), nodeid: 1, insize: 80

getdir[0]

fuse_getdir("/")

   unique: 4, error: -5 (Input/output error), outsize: 16

unique: 5, opcode: RELEASEDIR (29), nodeid: 1, insize: 64

   unique: 5, success, outsize: 16

在 2013年10月8日星期二UTC+8下午6时05分29秒，李帅写道：

[Daniel Frett]

something in the MogileFS::Client::Fuse::FilePaths::_listDir function is failing before it even attempts to issue the request to MogileFS.

I've pushed a patch up to github that will actually log the error encountered instead of just suppressing it.

try running the latest version of the code from here: https://github.com/frett/MogileFS-Fuse and let me know what error is being thrown, it should output the actual error to the log file now.

-Daniel

[李帅]

oh~

it's work,i can mount fuse,can ls,but can't touch file,can't cp file to fuse

the info:

------------------------------------

unique: 94, opcode: GETATTR (3), nodeid: 1, insize: 56

getattr /

[2013-10-11 19:07:16] fuse_getattr("/")

   unique: 94, success, outsize: 120

unique: 95, opcode: LOOKUP (1), nodeid: 1, insize: 46

LOOKUP /hosts

getattr /hosts

[2013-10-11 19:07:16] fuse_getattr("/hosts")

SOCK: cached = Sock_10.208.30.155:7001, REQ: plugin_filepaths_list_directory arg1=/&argcount=1&domain=domain1

$VAR1 = undef;

RESPONSE: ERR failure Operation+failed

$VAR1 = undef;

LASTERR: failure Operation+failed

$VAR1 = undef;

   unique: 95, error: -2 (No such file or directory), outsize: 16

unique: 96, opcode: LOOKUP (1), nodeid: 1, insize: 46

LOOKUP /hosts

getattr /hosts

[2013-10-11 19:07:16] fuse_getattr("/hosts")

   unique: 96, error: -2 (No such file or directory), outsize: 16

unique: 97, opcode: MKNOD (8), nodeid: 1, insize: 62

mknod /hosts 0100644 0x0 umask=0022

[2013-10-11 19:07:16] fuse_mknod("/hosts", "33188", "0")

SOCK: cached = Sock_10.208.30.155:7001, REQ: get_paths key=/hosts&domain=domain1&noverify=0

$VAR1 = undef;

RESPONSE: OK path2=http://10.208.30.155:7500/dev101/0/000/059/0000059410.fid&path1=http://10.208.30.158:7500/dev401/0/000/059/0000059410.fid&paths=2

$VAR1 = undef;

RETURN_VARS: 

$VAR1 = {

          'path2' => 'http://10.208.30.155:7500/dev101/0/000/059/0000059410.fid',

          'path1' => 'http://10.208.30.158:7500/dev401/0/000/059/0000059410.fid',

          'paths' => '2'

        };

getattr /hosts

[2013-10-11 19:07:16] fuse_getattr("/hosts")

SOCK: cached = Sock_10.208.30.155:7001, REQ: plugin_filepaths_list_directory arg1=/&argcount=1&domain=domain1

$VAR1 = undef;

RESPONSE: ERR failure Operation+failed

$VAR1 = undef;

LASTERR: failure Operation+failed

$VAR1 = undef;

   unique: 97, error: -2 (No such file or directory), outsize: 16

--------------------------------------------------

and why i can't put my file to default class?

在 2013年10月10日星期四UTC+8下午12时59分35秒，Daniel Frett写道：

[Daniel Frett]

it looks like you don't have the FilePaths plugin[1] installed & enabled for your MogileFS domain. Currently the FUSE client only works with domains that have the FilePaths plugin enabled [2].

The FilePaths plugin changes how the MogileFS tracker internally stores keys to support quicker access of what files exist in a directory. Changing this setting will render all existing files in the domain unaccessible.

-Daniel

[1] http://search.cpan.org/~hachi/MogileFS-Plugin-FilePaths-0.02/
[2] https://github.com/frett/MogileFS-Fuse/blob/master/README#L21

[李帅]

yes~i find the Filepaths have problem

i find three FilePaths:

http://search.cpan.org/~hachi/MogileFS-Plugin-FilePaths-0.02/

http://search.cpan.org/~hachi/MogileFS-Client-FilePaths-0.02/

and 

cpanm  MogileFS::Client::Fuse::FilePaths

what Filepaths i must install?

在 2013年10月11日星期五UTC+8下午10时31分01秒，Daniel Frett写道：

[Daniel Frett]

MogileFS::Plugin::FilePaths needs to be installed and configured on your tracker.

configuration involves adding the following line to /etc/mogilefs/mogilefsd.conf and restarting it
plugins = FilePaths

after that you need to make sure the tables are created, which can be done by executing this command:

perl -mMogileFS::Server `which mogdbsetup`  --plugins=FilePaths --verbose --yes --dbuser=**** --dbpass=***

then to enable filepaths for your domain you need to execute the following command:

perl -MMogileFS::Client -e'MogileFS::Client->new("domain" => "domain1", "hosts" => ["test.mogilefs.idc.cn"])->filepaths_enable()'

if you want to disable filepaths later, you can use this command:

perl -MMogileFS::Client -e'MogileFS::Client->new("domain" => "domain1", "hosts" => ["test.mogilefs.idc.cn"])->filepaths_disable()'

-Daniel

[李帅]

en~

THX for your reply~

i install MogileFS::Plugin::FilePaths

when i run mount-mogilefs,it print:

Can't locate MogileFS/Client/Fuse/FilePaths.pm in @INC

then i install :

# cpanm MogileFS::Client::Fuse::FilePaths

 then i can run mount-mogilefs,but can't touch file.

display:

--------------------------------------------

SOCK: cached = Sock_10.208.30.155:7001, REQ: get_paths domain=domain1&noverify=0&key=/ls

$VAR1 = undef;

RESPONSE: ERR plugin_aborted Action+aborted+by+plugin

$VAR1 = undef;

LASTERR: plugin_aborted Action+aborted+by+plugin

$VAR1 = undef;

SOCK: cached = Sock_10.208.30.155:7001, REQ: create_open domain=domain1&fid=0&multi_dest=0&key=/ls

$VAR1 = undef;

RESPONSE: OK devid=203&fid=59430&path=http://10.208.30.156:7500/dev203/0/000/059/0000059430.fid

$VAR1 = undef;

RETURN_VARS: 

$VAR1 = {

          'fid' => '59430',

          'devid' => '203',

          'path' => 'http://10.208.30.156:7500/dev203/0/000/059/0000059430.fid'

        };

SOCK: cached = Sock_10.208.30.155:7001, REQ: create_close devid=203&plugin.meta.key0=mtime&path=http://10.208.30.156:7500/dev203/0/000/059/0000059430.fid&size=0&key=/ls&plugin.meta.value0=1381734547&domain=domain1&plugin.meta.keys=1&fid=59430

$VAR1 = undef;

RESPONSE: ERR failure Operation+failed

$VAR1 = undef;

LASTERR: failure Operation+failed

$VAR1 = undef;

[2013-10-14 15:09:07] Error flushing file: /ls

   unique: 11, error: -5 (Input/output error), outsize: 16

unique: 12, opcode: LOOKUP (1), nodeid: 1, insize: 43

LOOKUP /ls

getattr /ls

[2013-10-14 15:09:07] fuse_getattr("/ls")

MogileFS object: [MogileFS::Client::FilePaths=HASH(0x7f464c542ea8)]

$VAR1 = bless( {

                 'readonly' => 0,

                 'domain' => 'domain1',

                 'backend' => bless( {

                                       'hosts' => [

                                                    '10.208.30.155:7001'

                                                  ],

                                       'host_dead' => {},

                                       'timeout' => 3

                                     }, 'MogileFS::Backend' )

               }, 'MogileFS::Client::FilePaths' );

SOCK: Sock_10.208.30.155:7001, REQ: plugin_filepaths_list_directory argcount=1&arg1=/&domain=domain1

$VAR1 = undef;

RESPONSE: ERR failure Operation+failed

$VAR1 = undef;

LASTERR: failure Operation+failed

$VAR1 = undef;

   unique: 12, error: -2 (No such file or directory), outsize: 16

----------------------------------------------

在 2013年10月14日星期一UTC+8下午1时42分46秒，Daniel Frett写道：

[李帅]

now i put a file to fuse ,then show mysql info:

mysql> select * from file;

+-----+------+---------------+--------+---------+----------+

| fid | dmid | dkey          | length | classid | devcount |

+-----+------+---------------+--------+---------+----------+

|  27 |    1 | fid:27        |      0 |       0 |        2 |

|  30 |    1 | fid:30        |      0 |       0 |        2 |

|  36 |    1 | fid:36        |      0 |       0 |        2 |

|  37 |    1 | fid:37        |      0 |       0 |        2 |

|  38 |    1 | fid:38        |  26505 |       0 |        2 |

|  39 |    1 | fid:39        |  26505 |       0 |        2 |

|  40 |    2 | fid:40        |      0 |       0 |        2 |

+-----+------+---------------+--------+---------+----------+

use mogtool put file too~

在 2013年10月14日星期一UTC+8下午3时23分23秒，李帅写道：

[Daniel Frett]

when the FilePaths plugin is active it splits the path into the path segments and stores the segments in: plugin_filepaths_paths

The difference in how it stores keys is why earlier I mentioned that "Changing this setting will render all existing files in the domain unaccessible."

now that you do have FilePaths enabled, MogileFS::Client::Fuse should work with that domain using the mount command you originally listed.

-Daniel

[Daniel Frett]

looking at this log it appears that something about the plugin isn't working correctly, I'll have to look at the code later to see if I can figure out what is wrong.

What version of MogileFS::Plugin::FilePaths are you using? is it the version on CPAN [1], or a development build from github [2]?

[1] http://search.cpan.org/~hachi/MogileFS-Plugin-FilePaths-0.02/

[2] https://github.com/hachi/MogileFS-Plugin-FilePaths

both should work fine, but it would help debugging knowing what code you are actually running :)

-Daniel

[Daniel Frett]

I walked through the code & log you sent and it looks like there is an error when it is closing a file after writing it (see inline comments). My next step would be checking the MogileFS tracker server error log [1] to see if it is outputing the error there.

I would guess that it might have something to do with the HTTP server you are using for your storage nodes. Not all HTTP servers operate the same way. For example, the current code utilizes partial PUT requests to store files. When I originally wrote this library and tested it out, partial PUT was not supported by Perlbal or nginx, but it was supported by cmogstored and lighttpd.

-Daniel

[1] https://code.google.com/p/mogilefs/wiki/Troubleshooting

On Monday, October 14, 2013 3:23:23 AM UTC-4, 李帅 wrote:

SOCK: cached = Sock_10.208.30.155:7001, REQ: get_paths domain=domain1&noverify=0&key=/ls

$VAR1 = undef;

RESPONSE: ERR plugin_aborted Action+aborted+by+plugin

$VAR1 = undef;

LASTERR: plugin_aborted Action+aborted+by+plugin

$VAR1 = undef;

This is expected when a path doesn't exist

 

SOCK: cached = Sock_10.208.30.155:7001, REQ: create_open domain=domain1&fid=0&multi_dest=0&key=/ls

$VAR1 = undef;

RESPONSE: OK devid=203&fid=59430&path=http://10.208.30.156:7500/dev203/0/000/059/0000059430.fid

$VAR1 = undef;

RETURN_VARS: 

$VAR1 = {

          'fid' => '59430',

          'devid' => '203',

          'path' => 'http://10.208.30.156:7500/dev203/0/000/059/0000059430.fid'

        };

successfully created a temporary location to write the file to

 

SOCK: cached = Sock_10.208.30.155:7001, REQ: create_close devid=203&plugin.meta.key0=mtime&path=http://10.208.30.156:7500/dev203/0/000/059/0000059430.fid&size=0&key=/ls&plugin.meta.value0=1381734547&domain=domain1&plugin.meta.keys=1&fid=59430

$VAR1 = undef;

RESPONSE: ERR failure Operation+failed

$VAR1 = undef;

LASTERR: failure Operation+failed

$VAR1 = undef;

This is the unexpected error, it had problems closing the temporary file it just created. More info should be in the MogileFS tracker error log

 

[2013-10-14 15:09:07] Error flushing file: /ls

   unique: 11, error: -5 (Input/output error), outsize: 16

unique: 12, opcode: LOOKUP (1), nodeid: 1, insize: 43

LOOKUP /ls

getattr /ls

[2013-10-14 15:09:07] fuse_getattr("/ls")

MogileFS object: [MogileFS::Client::FilePaths=HASH(0x7f464c542ea8)]

$VAR1 = bless( {

                 'readonly' => 0,

                 'domain' => 'domain1',

                 'backend' => bless( {

                                       'hosts' => [

                                                    '10.208.30.155:7001'

                                                  ],

                                       'host_dead' => {},

                                       'timeout' => 3

                                     }, 'MogileFS::Backend' )

               }, 'MogileFS::Client::FilePaths' );

SOCK: Sock_10.208.30.155:7001, REQ: plugin_filepaths_list_directory argcount=1&arg1=/&domain=domain1

$VAR1 = undef;

RESPONSE: ERR failure Operation+failed

$VAR1 = undef;

LASTERR: failure Operation+failed

$VAR1 = undef;

This is an odd error, I'm not sure why this failed, there will probably be info on this in the log file as well

[李帅]

i see the tracker log:

-------------------------------------

[Tue Oct 15 17:46:18 2013] Job job_master has only 0, wants 1, making 1.

Error: Undefined subroutine &Mgd::get_dbh called at /usr/local/share/perl5/MogileFS/Plugin/MetaData.pm line 33.

[Tue Oct 15 17:46:27 2013] [queryworker(21683)] Error running command 'plugin_filepaths_list_directory': Undefined subroutine &Mgd::get_dbh called at /usr/local/share/perl5/MogileFS/Plugin/MetaData.pm line 33.

---------------------------------------

在 2013年10月15日星期二UTC+8上午6时43分14秒，Daniel Frett写道：

[Daniel Frett]

it looks like the MetaData plugin that FilePaths utilizes is causing the problem,

try installing this branch[1] of MogileFS::Plugin::MetaData which updates database interaction to follow the current pattern in MogileFS::Server

[1] https://github.com/frett/MogileFS-Plugin-MetaData/tree/storeUpdate

-Daniel

[李帅]

Daniel,what os version you use?

Mogilefs version?

i use rhel 6.2

cpanm mogilefs~

i want replace my os version and mogilefs verion to old version ^_^

在 2013年10月15日星期二UTC+8下午5时48分23秒，李帅写道：

[李帅]

yeah～

Daniel

it's work!!!!

THX! very much!!!

very very much!^_^

在 2013年10月15日星期二UTC+8下午10时09分55秒，Daniel Frett写道：

[李帅]

Deniel,

i have a new problem

i can cp a small file to fuse,

but i can't cp a 300k jpg file to fuse.

it print:

-------------------------------------------

[2013-10-17 10:50:06] fuse_write("/1381978206767656200", "4096 bytes", "405504", "MogileFS::Client::Fuse::File::Generated0=HASH(0x7faa549739b0)")

[2013-10-17 10:50:06] fuse_write("/1381978206767656200", "4096 bytes", "409600", "MogileFS::Client::Fuse::File::Generated0=HASH(0x7faa4c9224a0)")

[2013-10-17 10:50:06] fuse_write("/1381978206767656200", "4096 bytes", "413696", "MogileFS::Client::Fuse::File::Generated0=HASH(0x7faa54971ca0)")

[2013-10-17 10:50:06] fuse_write("/1381978206767656200", "4096 bytes", "417792", "MogileFS::Client::Fuse::File::Generated0=HASH(0x7faa4c935348)")

[2013-10-17 10:50:06] fuse_write("/1381978206767656200", "3736 bytes", "421888", "MogileFS::Client::Fuse::File::Generated0=HASH(0x7faa54932508)")

[2013-10-17 10:50:06] fuse_flush("/1381978206767656200", "MogileFS::Client::Fuse::File::Generated0=HASH(0x7faa4c9224a0)")

SOCK: cached = Sock_10.208.30.155:7001, REQ: create_close devid=402&plugin.meta.key0=mtime&path=http://10.208.30.158:7500/dev402/0/000/059/0000059535.fid&size=425624&key=/1381978206767656200&plugin.meta.value0=1381978206&domain=domain3&plugin.meta.keys=1&fid=59535

$VAR1 = undef;

RESPONSE: ERR size_mismatch Expected:+425624%3B+actual:+7832%3B+path:+http://10.208.30.158:7500/dev402/0/000/059/0000059535.fid

$VAR1 = undef;

LASTERR: size_mismatch Expected:+425624%3B+actual:+7832%3B+path:+http://10.208.30.158:7500/dev402/0/000/059/0000059535.fid

$VAR1 = undef;

[2013-10-17 10:50:06] Error flushing file: /1381978206767656200

---------------------------------

the file size have some problem?

在 2013年10月15日星期二UTC+8下午10时09分55秒，Daniel Frett写道：

[李帅]

i stat the file:

stat lalala.ha 

  File: `lalala.ha'

  Size: 425624          Blocks: 832        IO Block: 4096   regular file

Device: 802h/2050d      Inode: 2374054     Links: 1

file size mismatch

在 2013年10月17日星期四UTC+8上午10时57分00秒，李帅写道：

[李帅]

i use cmostroed replace mogstroed,

it's ok

fuse & mogstroed can't support large file cp?

在 2013年10月17日星期四UTC+8上午11时12分22秒，李帅写道：

[Daniel Frett]

This is an issue I've run into before and mentioned briefly in a previous post. The HTTP server that is running for your storage nodes does not support partial PUTs for the WebDAV interface.

I know lighttpd and cmogstored both support partial PUTs, but Perlbal (default server with mogstored) and nginx do not.

Partial PUTs are not an official part of the http spec, but I haven't been able to find an alternative (i.e. HTTP PATCH) that is actually implemented.

Here are several things that factor into the need to store files in chunks:

1. Fuse writes files in 4K blocks with no indication of how large the file will be in total

2. multi-threaded mode would require some C code to share open sockets between multiple perl interpreters (how perl implements threads)

3. Standard HTTP PUT commands write the entire file in one request

4. HTTP PUT commands require the total size of the data being written before any data is sent

5. buffering the entire file in memory/on disk would potentially be prohibitive with large files (I store 16GB files on my home cluster regularly)

6. writing & rewriting files for every block received to work around PUT limitations would impose an O(n^2) performance penalty for storing a file

-Daniel

[李帅]

en~

Daniel,what version you used?

i want replace my version to your wesion test it.

i used:

os:redhat el6.2

perl v5.10.1

MogileFS-Server-2.70

MogileFS-Utils-2.28

MogileFS-Client-FilePaths-0.02

MogileFS-Plugin-FilePaths-0.03_01

MogileFS-Plugin-MetaData-0.01

and 

https://github.com/frett/MogileFS-Fuse

在 2013年10月17日星期四UTC+8上午11时49分00秒，Daniel Frett写道：

[Daniel Frett]

I currently run 2 separate clusters, 1 at home and 1 at work. The version of linux and perl varies between the 2 locations (and even servers within the clusters)

I'm running regular releases of these:

Fedora Core 11 - 19 (varies by server)

perl 5.12.4 - 5.16.3 (varies by server)

MogileFS::Server 2.70

cmogstored 1.2.0 - 1.3.1 (varies by server)

I'm running custom versions of these with various patches

Fuse [1]

- addresses a memory leak when using threaded mounts

MogileFS::Plugin::MetaData [2]

- updated db interaction for current paradigm

- adds internal method to get meta-data in bulk

MogileFS::Plugin::FilePaths [3]

- refactors internal objects to reduce db load

- adds support for an rmDir command

- utilizes new bulk meta-data support

MogileFS::Client::Fuse [4][5] (I merge the branches together before building)

- adds support for the rmDir command

At home I also utilize a private branch for MogileFS::Client::Fuse that adds preliminary checksum support, it currently only supports non-threaded mounts (and I'm not happy with the current state of the code) so I haven't made it available publicly or merged it into master.

the majority of these changes have been pushed back to the main projects, but have not been merged in yet.

[1] https://github.com/frett/perl-fuse/tree/threads

[2] https://github.com/frett/MogileFS-Plugin-MetaData/tree/bulkMeta

[3] https://github.com/frett/MogileFS-Plugin-FilePaths/tree/remainingPatches

[4] https://github.com/frett/MogileFS-Fuse/tree/master

[5] https://github.com/frett/MogileFS-Fuse/tree/rmDir

-Daniel

[李帅]

Daniel,

i cp a file to fuse:

# cp /etc/hosts /mnt/tmp/`cat /dev/urandom | head -1 | md5sum | head -c 16`

the log display:

---------------------------------

[2013-10-21 15:34:54] Starting up MogileFS::Client::Fuse

[2013-10-21 15:34:59] fuse_getattr("/923515d995f16bdd")

[2013-10-21 15:35:01] fuse_getattr("/923515d995f16bdd")

[2013-10-21 15:35:03] fuse_getattr("/923515d995f16bdd")

[2013-10-21 15:35:05] fuse_mknod("/923515d995f16bdd", "33188", "0")

[2013-10-21 15:35:05] fuse_getattr("/923515d995f16bdd")

[2013-10-21 15:35:07] fuse_open("/923515d995f16bdd", "32769", "HASH(0x7f50b0dd5548)")

[2013-10-21 15:35:07] fuse_getxattr("/923515d995f16bdd", "security.capability")

[2013-10-21 15:35:07] fuse_write("/923515d995f16bdd", "176 bytes", "0", "MogileFS::Client::Fuse::File::Generated0=HASH(0x7f50b0d94aa8)")

[2013-10-21 15:35:07] fuse_flush("/923515d995f16bdd", "MogileFS::Client::Fuse::File::Generated0=HASH(0x7f50b0669ae8)")

[2013-10-21 15:35:07] fuse_release("/923515d995f16bdd", "32769", "MogileFS::Client::Fuse::File::Generated0=HASH(0x7f50b0d94ac0)")

--------------------------------------

why it need 4 fuse_getattr("/ce6d24e4e3b01258")

and i find fuse_getattr use _list_Dir function

it return all file list~

the cp time is too slow~

在 2013年10月18日星期五UTC+8下午10时04分27秒，Daniel Frett写道：

[Daniel Frett]

those are OS calls that we are handling, MogileFS::Client::Fuse doesn't make any decisions about when and how frequently to trigger one of it's callbacks.

With that said, getattr is appearing to take longer than it should. (the actual writing of data performed normally)

There is a short (2 secs) in-memory cache of the _listDir response to improve performance when doing an 'ls'. The requests you showed are unfortunately falling outside of that threshold.

about how many files do you have in the root directory?

which versions of the FilePaths and MetaData plugins are you using? (the versions I had linked in the my previous post have several patches that reduce the number of db queries and would improve performance of _listDir)

-Daniel

[李帅]

i use you list version~

my root dir have 2 thousand file

use multi level dir less file have good performance?

在 2013年10月21日星期一UTC+8下午8时34分05秒，Daniel Frett写道：

[Daniel Frett]

currently performance for directories with a large amount of files is poor. It is something I know about and am working to improve [1][2]. The patches in the branches that you are using eliminate a lot of extra queries, but there is still lots of room for improvement.

Right now the only way to improve performance for getdir or getattr calls is to reduce the number of files in the directory.

-Daniel

[1] https://github.com/frett/MogileFS-Fuse/issues/12

[2] https://github.com/frett/MogileFS-Fuse/issues/13