Issue 256 in s3fs: S3FS throws input/output error when trying to access files (cp, ls, etc.)
s3...@googlecode.com
Owner: ----
Labels: Type-Defect Priority-Medium
New issue 256 by matthew....@gmail.com: S3FS throws input/output error when
trying to access files (cp, ls, etc.)
http://code.google.com/p/s3fs/issues/detail?id=256
Detailed description of observed behavior:
I mount an S3 volume using s3fs. The directories display properly, but when
I try to do "ls -all" I simply get a input/output error with no further
information.
What steps will reproduce the problem - please be very specific and
detailed. (if the developers cannot reproduce the issue, then it is
unlikely a fix will be found)?
I have seen that the problem is intermittent. If I mount one bucket, it
works okay and I can list directories. A moment later, the same bucket
throws the input/output error.
===================================================================
The following information is very important in order to help us to help
you. Omission of the following details may delay your support request or
receive no attention at all.
===================================================================
Version of s3fs being used (s3fs --version):
s3fs --version
Amazon Simple Storage Service File System 1.61
Version of fuse being used (pkg-config --modversion fuse):
2.8.4
System information (uname -a):
uname -a
Linux ip-10-0-5-200 2.6.18-274.12.1.el5xen #1 SMP Tue Nov 29 14:56:22 EST
2011 i686 i686 i386 GNU/Linux
Distro (cat /etc/issue):
CentOS release 5.7 (Final)
Kernel \r on an \m
s3fs command line used (if applicable):
/etc/fstab entry (if applicable):
s3fs syslog messages (grep s3fs /var/log/syslog):
sudo grep s3fs /var/log/messages
Feb 15 16:56:33 ip-10-0-5-200 s3fs: init $Rev: 367 $
Feb 16 16:36:30 ip-10-0-5-200 s3fs: init $Rev: 367 $
Feb 17 17:54:44 ip-10-0-5-200 s3fs: init $Rev: 367 $
Feb 19 03:48:45 ip-10-0-5-200 s3fs: init $Rev: 367 $
Feb 19 03:51:11 ip-10-0-5-200 s3fs: init $Rev: 367 $
Feb 19 04:01:28 ip-10-0-5-200 s3fs: init $Rev: 367 $
Feb 19 04:05:32 ip-10-0-5-200 s3fs: init $Rev: 367 $
Feb 19 04:12:01 ip-10-0-5-200 s3fs: init $Rev: 367 $
Feb 19 04:12:04 ip-10-0-5-200 s3fs: ###curlCode: 47 msg: number of
redirects hit maximum amount
Feb 19 04:12:29 ip-10-0-5-200 s3fs: init $Rev: 367 $
Feb 19 04:12:31 ip-10-0-5-200 s3fs: ###curlCode: 47 msg: number of
redirects hit maximum amount
Feb 19 04:19:07 ip-10-0-5-200 s3fs: init $Rev: 367 $
Feb 19 04:22:41 ip-10-0-5-200 s3fs: init $Rev: 367 $
Feb 19 04:33:55 ip-10-0-5-200 s3fs: init $Rev: 367 $
Feb 19 04:36:58 ip-10-0-5-200 s3fs: init $Rev: 367 $
Feb 19 04:46:22 ip-10-0-5-200 s3fs: init $Rev: 367 $
Feb 19 04:47:48 ip-10-0-5-200 s3fs: init $Rev: 367 $
Feb 19 04:50:23 ip-10-0-5-200 s3fs: init $Rev: 367 $
Feb 19 04:56:45 ip-10-0-5-200 s3fs: init $Rev: 367 $
Feb 19 05:03:39 ip-10-0-5-200 s3fs: init $Rev: 367 $
s3...@googlecode.com
Comment #1 on issue 256 by matthew....@gmail.com: S3FS throws input/output
http://code.google.com/p/s3fs/issues/detail?id=256
[releaseuser@ip-10-0-5-200 ~]# sudo umount /s3mount
[releaseuser@ip-10-0-5-200 ~]# s3fs
NAME_OF_MY_BUCKET_HIGHLY_SPECIFIC_DUE_TO_GLOBAL_CONTEXT /s3mount
[releaseuser@ip-10-0-5-200 ~]# cd s3/mount
-bash: cd: s3/mount: No such file or directory
[releaseuser@ip-10-0-5-200 ~]# cd /s3mount
[releaseuser@ip-10-0-5-200 s3mount]# cd 0/0/0
[releaseuser@ip-10-0-5-200 0]# ls
ls: reading directory .: Input/output error
[releaseuser@ip-10-0-5-200 0]#
s3...@googlecode.com
Comment #2 on issue 256 by matthew....@gmail.com: S3FS throws input/output
http://code.google.com/p/s3fs/issues/detail?id=256
Here you can see the very ugly nature of this issue. The ls commands are
only a couple seconds apart and no changes are being made to the bucket
between ls commands.
releaseuser@PortalReleaseServer2
/s3mount/HASHED_PUF_BACKUP/portal-uploaded-files/5/5/5: ls
ls: reading directory .: Input/output error
4ef2e47daee76_f2a67fb0812b4a4568745ad47413468a_905f03e8e6fc6dfd894e9b10ba79987b_thumb
4ef3260b007d3_12828781b3993ea697ecdee0b8cfc6cb_e3ae22f42c3fdc55c6c6a772c045c77c_thumb
4f346ed47760f_bb61fb8a61f73f98ca96581199da7e08_784e28bb8090bfa81f80b60c0p
4f3bf71c072ab_cf26ef8ad13623e44d84095b7ed0d477_91dd02ea744049764c6df9fbb3a3c9a8_thumb
releaseuser@PortalReleaseServer2
/s3mount/HASHED_PUF_BACKUP/portal-uploaded-files/5/5/5: ls
ls: reading directory .: Input/output error
releaseuser@PortalReleaseServer2
/s3mount/HASHED_PUF_BACKUP/portal-uploaded-files/5/5/5: ls
ls: reading directory .: Input/output error
releaseuser@PortalReleaseServer2
/s3mount/HASHED_PUF_BACKUP/portal-uploaded-files/5/5/5: ls
ls: reading directory .: Input/output error
releaseuser@PortalReleaseServer2
/s3mount/HASHED_PUF_BACKUP/portal-uploaded-files/5/5/5: ls
ls: reading directory .: Input/output error
4f346ed47760f_bb61fb8a61f73f98ca96581199da7e08_784e28bb8090bfa81f80b60c0p
4f3bf71c072ab_cf26ef8ad13623e44d84095b7ed0d477_91dd02ea744049764c6df9fbb3a3c9a8_thumb
releaseuser@PortalReleaseServer2
/s3mount/HASHED_PUF_BACKUP/portal-uploaded-files/5/5/5: ls
ls: reading directory .: Input/output error
releaseuser@PortalReleaseServer2
/s3mount/HASHED_PUF_BACKUP/portal-uploaded-files/5/5/5:
s3...@googlecode.com
Comment #3 on issue 256 by matthew....@gmail.com: S3FS throws input/output
http://code.google.com/p/s3fs/issues/detail?id=256
On my other server, I connected to the same bucket. Running the ls command
a few times, with a couple seconds in between, I get different results
although nothing with the actual bucket is changing with time!
username@MyServer /s3mount/HASHED_PUF_BACKUP/uploaded-files/5/5/5: ls
ls: reading directory .: Input/output error
username@MyServer /s3mount/HASHED_PUF_BACKUP/uploaded-files/5/5/5: ls
ls: reading directory .: Input/output error
username@MyServer /s3mount/HASHED_PUF_BACKUP/uploaded-files/5/5/5: ls
ls: reading directory .: Input/output error
username@MyServer /s3mount/HASHED_PUF_BACKUP/uploaded-files/5/5/5: ls
ls: reading directory .: Input/output error
4eef757b5b038_c70807a0f4a24cd5eef03d7358c376a9_8164940c353183685f9211630p
4f346ed47760f_bb61fb8a61f73f98ca96581199da7e08_784e28bb8090bfa81f80b60c0p
4ef2e47daee76_f2a67fb0812b4a4568745ad47413468a_905f03e8e6fc6dfd894e9b10ba79987b_thumb
4f3bf71c072ab_cf26ef8ad13623e44d84095b7ed0d477_91dd02ea744049764c6df9fbb3a3c9a8_thumb
4ef3260b007d3_12828781b3993ea697ecdee0b8cfc6cb_e3ae22f42c3fdc55c6c6a772c045c77c_thumb
username@MyServer /s3mount/HASHED_PUF_BACKUP/uploaded-files/5/5/5: ls
ls: reading directory .: Input/output error
4f346ed47760f_bb61fb8a61f73f98ca96581199da7e08_784e28bb8090bfa81f80b60c0p
4f3bf71c072ab_cf26ef8ad13623e44d84095b7ed0d477_91dd02ea744049764c6df9fbb3a3c9a8_thumb
username@MyServer /s3mount/HASHED_PUF_BACKUP/uploaded-files/5/5/5:
s3...@googlecode.com
Comment #4 on issue 256 by matthew....@gmail.com: S3FS throws input/output
http://code.google.com/p/s3fs/issues/detail?id=256
On my other server, I connected to the same bucket. Running the ls command
a few times, with a couple seconds in between, I get different results
although nothing with the actual bucket is changing with time! The problem
is clearly intermittent with the results of requests to list files in a
directory changing with time.
s3...@googlecode.com
Comment #5 on issue 256 by matthew....@gmail.com: S3FS throws input/output
http://code.google.com/p/s3fs/issues/detail?id=256
My 2 servers are configured identically so hard to determine whether it is
on AWS side of things or whether it is s3fs.
s3...@googlecode.com
Comment #6 on issue 256 by steve.ne...@gmail.com: S3FS throws input/output
I get the same error message if I have over 1200 or so objects in a
directory. I can access files by name but listing the directory contents
fails reliably over 1400 files in a directory with version 1.68 and fuse
2.8.4 on ubuntu 12.04
--
You received this message because this project is configured to send all
issue notifications to this address.
You may adjust your notification preferences at:
https://code.google.com/hosting/settings
s3...@googlecode.com
Comment #7 on issue 256 by ggta...@gmail.com: S3FS throws input/output
Hi all
Hello, steve
Your problem is same as Issue 235.
It fixed by r430.
Please use revision 430(not packaged yet), and if you have more problem,
please post new issue.
Hello matthew
Do you have this problem yet?
If you use v1.61, please upgrade s3fs as same as steve(r432 now).
Thanks in advance for your help.
s3...@googlecode.com
Comment #8 on issue 256 by pjais...@sfchronicle.com: S3FS throws
http://code.google.com/p/s3fs/issues/detail?id=256
message while copying large file to S3 mount:
[ec2-user@ip-10-0-10-196 oradump]$ cp DRAW.out
/mnt/sfc-database-backups/exports/
cp: writing `/mnt/sfc-database-backups/exports/DRAW.out': No space left on
device
cp: closing `/mnt/sfc-database-backups/exports/DRAW.out': Input/output error
s3...@googlecode.com
Comment #9 on issue 256 by ggta...@gmail.com: S3FS throws input/output
Hi,
So you got error "No space left on device"(28), I think it means there are
no space for writing to file.
s3fs opens/writes/reads local file for it's operation.
When you specifies use_cache option, s3fs uses temporally files in
specified directory.
If you does not specify it, s3fs uses system temporally files.
For this issue, do you copy big file?
And do you have enough of disk space for copying objects?
Please let me know whichever this issue depends on about disk space or not.
Thanks in advance for your help.
s3...@googlecode.com
Comment #10 on issue 256 by pjais...@sfchronicle.com: S3FS throws
http://code.google.com/p/s3fs/issues/detail?id=256
linux got mounted on /root after the reboot. Once I added these mounts on
ephemeral device it worked like a charm.
Thanks so much and appreciate your prompt response to my question.
s3...@googlecode.com
Status: Fixed
Comment #11 on issue 256 by ggta...@gmail.com: S3FS throws input/output
Hi,
Can I close this issue?
And I upgraded s3fs as v1.71, please use latest version.
I closed this issue, but if you need more please let me know or post new
issue.
Thanks in advance.