Issue 259 in s3fs: hadoop pig, s3fs, 0 kb files
s3...@googlecode.com
Owner: ----
Labels: Type-Defect Priority-Medium
New issue 259 by Ia...@affectv.co.uk: hadoop pig, s3fs, 0 kb files
http://code.google.com/p/s3fs/issues/detail?id=259
s3fs creates 0 kb files in s3 for each folder in the filesystem. These
files have the same name as the folders. This works fine until I try to run
a pig-script. Pig gets confused and can't access directories if the files
are not removed. Removing the files lets pig run however makes folders
invisible to the server from where im copying files to s3.
Is there a solution for this problem?
Reproduce: create a folder with files through s3fs and try to access files
in it through pig
===================================================================
The following information is very important in order to help us to help
you. Omission of the following details may delay your support request or
receive no attention at all.
===================================================================
Version of s3fs being used (s3fs --version): 1.61
Version of fuse being used (pkg-config --modversion fuse): 2.8.6
Distro (cat /etc/issue): amazon linux
s3...@googlecode.com
Comment #1 on issue 259 by nielsen....@gmail.com: hadoop pig, s3fs, 0 kb
files
http://code.google.com/p/s3fs/issues/detail?id=259
I'm also very keen to hear a solution to this. Accessing my files in
folders through s3fox also breaks because of the problem above.
s3...@googlecode.com
Comment #2 on issue 259 by Ia...@affectv.co.uk: hadoop pig, s3fs, 0 kb files
http://code.google.com/p/s3fs/issues/detail?id=259
I was unable to find a solution, so now I have a daily python script using
boto that downloads and uploads files to s3 directly without using s3fs. So
no 0 kb files are generated. I am also using boto to launch pig jobs.
If you by chance delete the 0 kb file I think you can see it again by
creating a new folder with the same name and then delete it so you have a
new 0 kb file or something
like that
s3...@googlecode.com
Comment #3 on issue 259 by nielsen....@gmail.com: hadoop pig, s3fs, 0 kb
files
http://code.google.com/p/s3fs/issues/detail?id=259
Thanks for that - I'll look into a scripted solution instead as well then.
s3...@googlecode.com
Status: Fixed
Comment #4 on issue 259 by ggta...@gmail.com: hadoop pig, s3fs, 0 kb files
http://code.google.com/p/s3fs/issues/detail?id=259
Hi
Please use latest version(now v1.69) for this issue.
Latest version supports a directory which is made by other S3 clients.
It solves your issue.
I closed this issue, but if you have a problem yet, please post new issue.
Thanks in advance.
--
You received this message because this project is configured to send all
issue notifications to this address.
You may adjust your notification preferences at:
https://code.google.com/hosting/settings