Issue 381 in s3fs: folder showing up as a file instead of folder on unix system

[s3...@googlecode.com]

Status: New
Owner: ----
Labels: Type-Support Priority-Medium

New issue 381 by hbakke...@gmail.com: folder showing up as a file instead
of folder on unix system
http://code.google.com/p/s3fs/issues/detail?id=381

Detailed description of support request:

I have an s3 bucket, lets just say it is called (hb-s3-bucket) that has a
folder (filepicker) with images in there. In the s3 console and S3Fox
Organizer the folder shows up as a folder. But when I mount the bucket,
the folder shows up as a file. Other folders in the bucket show up as
folders, not sure why it is only this particular one.

I even tried mounting on a clean server, but still shows up as file not a
directory. This was working just fine until people started reporting 500
errors on my site. That is how I found that the folder was somehow changed
to a file not a directory.

I did a support chat with amazon and they said that the folder in the
bucket was a key and they could see objects in there, so thought it was
just an s3fs issue.


===================================================================
The following information is very important in order to help us to help
you. Omission of the following details may delay your support request or
receive no attention at all.
===================================================================
Version of s3fs being used (s3fs --version): 1.73

Version of fuse being used (pkg-config --modversion fuse): 2.9.2

System information (uname -a): Linux ip-xx-xxx-xx-xxx
3.4.57-48.42.amzn1.x86_64 #1 SMP Mon Aug 12 21:43:36 UTC 2013 x86_64 x86_64
x86_64 GNU/Linux

Distro (cat /etc/issue): Amazon Linux AMI release 2013.03

s3fs command line used (if applicable):
s3fs hb-s3-bucket /media/hb-s3-bucket -o
rw,allow_other,uid=48,gid=48,umask=0002,use_cache=/s3_cache/hb-s3-bucket,default_acl=public-read,dev,suid

/etc/fstab entry (if applicable):

s3fs syslog messages (grep s3fs /var/log/syslog):

Oct 24 13:31:21 s3fs: file locked(/hb-s3-bucket-
/s3_cache/.hb-s3-bucket.stat/filepicker)
Oct 24 13:31:21 s3fs: file unlocked(/filepicker)
Oct 24 13:31:21 s3fs: Body Text:
Oct 24 13:31:21 s3fs: could not download. start(0), size(16384), errno(-2)
Oct 24 13:31:21 s3fs: failed to read file(/filepicker). result=-5


--
You received this message because this project is configured to send all
issue notifications to this address.
You may adjust your notification preferences at:
https://code.google.com/hosting/settings

[s3...@googlecode.com]

Comment #1 on issue 381 by hbakke...@gmail.com: folder showing up as a file

instead of folder on unix system
http://code.google.com/p/s3fs/issues/detail?id=381

When I ran with debug, the logs say this(I changed the bucket name in this
ticket from the actual bucket name.)

Oct 24 13:31:21 s3fs: file locked(/filepicker-

[s3...@googlecode.com]

Comment #2 on issue 381 by ggta...@gmail.com: folder showing up as a file

instead of folder on unix system
http://code.google.com/p/s3fs/issues/detail?id=381

Hi,

Which console or application did make the folder(filepicker)?
I think that s3fs con not understand your folder as directory.

if you can, please let me know about your folder status.
* s3cmd ls s3://hb-s3-bucket
Which does the filepicker folder show "file object" or "DIR" object?
* s3cmd info s3://hb-s3-bucket/filepicker
Is the stats showed?
* s3cmd ls s3://hb-s3-bucket/filepicker/
Is the stats showed?

s3fs can understand the object as directory by below cases.
* the directory object exists and its mime type is "octet-stream"
or "x-directory"
* the directory object does not exists, but the object exists under object
path.
* etc

So I want to know whichever your directory object exists and what
status(attributes) does the object have.

Thanks in advance for your help.

[s3...@googlecode.com]

Comment #3 on issue 381 by psir...@gmail.com: folder showing up as a file

instead of folder on unix system

https://code.google.com/p/s3fs/issues/detail?id=381

I am seeing the same thing. Here is an answer to your question from my POV.

Note that in this case, this is where Amazon is creating CloudTrail logs in
DIR/SUBDIR1/NNNN where NNNNNNNNNNNN is our AWS account ID. I can view the
directories in s3 browser and from the amazon console but s3fs just shows
(Though it does allow me to descend into SUBDIR1)

# ls -l
total 1
---------- 1 root root 0 Feb 25 2014 NNNNNNNNNNNN



# s3cmd info s3://DIR/
s3://DIR/ (bucket):
Location: any
ACL: PPPP: READ
ACL: PPPP: WRITE
ACL: PPPP: READ_ACP
ACL: PPPP: WRITE_ACP


# s3cmd info s3://DIR/SUBDIR1/

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
An unexpected error has occurred.
Please report the following lines to:
s3tool...@lists.sourceforge.net
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Problem: ParseError: no element found: line 1, column 0
S3cmd: 1.0.0

Traceback (most recent call last):
File "/usr/bin/s3cmd", line 2006, in <module>
main()
File "/usr/bin/s3cmd", line 1950, in main
cmd_func(args)
File "/usr/bin/s3cmd", line 631, in cmd_info
info = s3.object_info(uri)
File "/usr/share/s3cmd/S3/S3.py", line 324, in object_info
response = self.send_request(request)
File "/usr/share/s3cmd/S3/S3.py", line 511, in send_request
raise S3Error(response)
File "/usr/share/s3cmd/S3/Exceptions.py", line 48, in __init__
tree = getTreeFromXml(response["data"])
File "/usr/share/s3cmd/S3/Utils.py", line 66, in getTreeFromXml
tree = ET.fromstring(xml)
File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1302, in XML
return parser.close()
File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1655, in close
self._raiseerror(v)
File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1507, in
_raiseerror
raise err
ParseError: no element found: line 1, column 0

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
An unexpected error has occurred.
Please report the above lines to:
s3tool...@lists.sourceforge.net
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!


# s3cmd info s3://DIR/SUBDIR1/NNNNNNNNNNNN/
s3://DIR/SUBDIR1/NNNNNNNNNNNN/ (object):
File size: 0
Last mod: Tue, 25 Feb 2014 14:25:28 GMT
MIME type: text/plain
MD5 sum: d41d8cd98f00b204e9800998ecf8427e
ACL: aws_cloudtrail_us-east-1: FULL_CONTROL
ACL: PPPP: FULL_CONTROL


# s3cmd ls s3://DIR/SUBDIR1/NNNNNNNNNNNN/
DIR s3://DIR/SUBDIR1/NNNNNNNNNNNN/CloudTrail/
2014-02-25 14:25 0 s3://DIR/SUBDIR1/NNNNNNNNNNNN/


And I can continue to traverse subdirs with s3cmd. Just for completeness...


# s3fs --version
Amazon Simple Storage Service File System 1.74

[s3...@googlecode.com]

Comment #4 on issue 381 by psir...@gmail.com: folder showing up as a file

instead of folder on unix system
https://code.google.com/p/s3fs/issues/detail?id=381

So it's looking like the issue is that there is not s3fs metadata on the
files/directories in question.

It looks very similar to this issue
https://code.google.com/p/s3fs/issues/detail?id=73
However, that is marked as fixed and this is still an issue.

It seems like the answer would be to have an option for a default mode if
one is not obtained from the metadata. e.g. -o mode=0500. I would suggest
that the default remain as 0000

As well as the permissions, I note that the directory was not detected as a
directory. I'm not sure what the deal is there.