Issue 180 in s3fs: When using FTP to send to an s3fs backed directory, files are appended to instead of overwritten

[s3...@googlecode.com]

Status: New
Owner: ----
Labels: Type-Defect Priority-Medium

New issue 180 by asifrahm...@gmail.com: When using FTP to send to an s3fs
backed directory, files are appended to instead of overwritten
http://code.google.com/p/s3fs/issues/detail?id=180

Detailed description of observed behavior:

When sending a file using FTP to an s3fs backed directory, if the filename
already exists in that directory, the data being uploaded is appended to
the end of the file rather than overwriting the existing file. When
uploading the file to a non-s3fs-backed directory on the same FTP server
using the same credentials, the file is properly overwritten. The FTP
server is VSFTPd 2.2.2.

===================================================================
The following information is very important in order to help us to help
you. Omission of the following details may delay your support request or
receive no attention at all.
===================================================================
Version of s3fs being used (s3fs --version): 1.40

Version of fuse being used (pkg-config --modversion fuse): 2.8.5

System information (uname -a): Linux domU-12-31-39-09-9E-58 2.6.32-309-ec2
#18-Ubuntu SMP Mon Oct 18 21:00:20 UTC 2010 i686 GNU/Linux

Distro (cat /etc/issue): Ubuntu 10.04.2 LTS \n \l

s3fs command line used (if applicable): s3fs -o allow_other
newscred-ftp-movies.ch uploads

/etc/fstab entry (if applicable):

s3fs syslog messages (grep s3fs /var/log/syslog):

[s3...@googlecode.com]

Comment #1 on issue 180 by moore...@suncup.net: When using FTP to send to

an s3fs backed directory, files are appended to instead of overwritten
http://code.google.com/p/s3fs/issues/detail?id=180

Please run sf3 in foreground mode (-f option) and upload a small file where
that file all ready exists. Then please share the output seen on in the
terminal.

I would like to see what commands FUSE is giving s3fs.

[s3...@googlecode.com]

Comment #2 on issue 180 by moore...@suncup.net: When using FTP to send to

an s3fs backed directory, files are appended to instead of overwritten
http://code.google.com/p/s3fs/issues/detail?id=180

I'll try to take a look at this one soon.

[s3...@googlecode.com]

Updates:
Status: CannotReproduce

Comment #3 on issue 180 by ben.lema...@gmail.com: When using FTP to send to

an s3fs backed directory, files are appended to instead of overwritten
http://code.google.com/p/s3fs/issues/detail?id=180

Just tested against s3fs 1.61, unable to reproduce. Let us know if you're
still having issues. This is most likely a problem with the ftp server in
use.

[s3...@googlecode.com]

Comment #4 on issue 180 by dimonz...@gmail.com: When using FTP to send to

an s3fs backed directory, files are appended to instead of overwritten
http://code.google.com/p/s3fs/issues/detail?id=180

Hi. Just reproduced this issue.
I'm using vsftpd as ftp server and ftp folder pointed to s3fs mounted folder

Version of s3fs being used (s3fs --version): 1.61

Version of fuse being used (pkg-config --modversion fuse): 2.8.5

System information (uname -a): Linux localhost.localdomain
2.6.32-131.21.1.el6.x86_64 #1 SMP Tue Nov 22 19:48:09 GMT 2011 x86_64
x86_64 x86_64 GNU/Linux
Distro (cat /etc/issue): CentOS release 6.1 (Final)
s3fs command line used (if applicable): s3fs -o allow_other ...

s3fs logs (uploading text file test.txt with single word test):
list_bucket [path=/testuser/test]
s3fs_getattr[path=/testuser/test]
stat cache hit [path=/testuser/test] [hit count=14]
s3fs_getattr[path=/testuser/test/test.txt]
s3fs_create[path=/testuser/test/test.txt][mode=33188][flags=35905]
create_file_object[path=/testuser/test/test.txt][mode=33188]
get_local_fd[path=/testuser/test/test.txt]
downloading[path=/testuser/test/test.txt][fd=5]
s3fs_getattr[path=/testuser/test/test.txt]
add_stat_cache_entry[path=/testuser/test/test.txt]
truncate[path=/testuser/test/test.txt][size=0]
calling get_headers [path=/testuser/test/test.txt]
put_local_fd[path=/testuser/test/test.txt][fd=6]
put_local_fd_small_file[path=/testuser/test/test.txt][fd=6]
uploading[path=/testuser/test/test.txt][fd=6][size=0]
s3fs_getattr[path=/testuser/test/test.txt]
stat cache hit [path=/testuser/test/test.txt] [hit count=0]
s3fs_write[path=/testuser/test/test.txt]
s3fs_flush[path=/testuser/test/test.txt][fd=5]
calling get_headers [path=/testuser/test/test.txt]
put_local_fd[path=/testuser/test/test.txt][fd=5]
put_local_fd_small_file[path=/testuser/test/test.txt][fd=5]
uploading[path=/testuser/test/test.txt][fd=5][size=4]
s3fs_release[path=/testuser/test/test.txt][fd=5]
delete_stat_cache_entry[path=/testuser/test/test.txt]
s3fs_getattr[path=/testuser/test]
stat cache hit [path=/testuser/test] [hit count=15]
readdir[path=/testuser/test]
list_bucket [path=/testuser/test]
add_stat_cache_entry[path=/testuser/test/test.txt]
s3fs_getattr[path=/testuser/test]
stat cache hit [path=/testuser/test] [hit count=16]
s3fs_getattr[path=/testuser/test/test.txt]
stat cache hit [path=/testuser/test/test.txt] [hit count=0]

modifying file test.txt. now it contains (test test)
s3fs logs (uploading with overwriting text file test.txt):

s3fs_getattr[path=/testuser]
stat cache hit [path=/testuser] [hit count=6]
access[path=/testuser]
access[path=/testuser]
s3fs_getattr[path=/testuser/.message]
s3fs_getattr[path=/testuser/test]
stat cache hit [path=/testuser/test] [hit count=17]
readdir[path=/testuser/test]
list_bucket [path=/testuser/test]
stat cache hit [path=/testuser/test/test.txt] [hit count=1]
s3fs_getattr[path=/testuser/test/test.txt]
stat cache hit [path=/testuser/test/test.txt] [hit count=2]
s3fs_getattr[path=/testuser/test]
stat cache hit [path=/testuser/test] [hit count=18]
s3fs_getattr[path=/testuser/test/test.txt]
stat cache hit [path=/testuser/test/test.txt] [hit count=3]
s3fs_open[path=/testuser/test/test.txt][flags=35841]
get_local_fd[path=/testuser/test/test.txt]
downloading[path=/testuser/test/test.txt][fd=5]
truncate[path=/testuser/test/test.txt][size=0]
calling get_headers [path=/testuser/test/test.txt]
put_local_fd[path=/testuser/test/test.txt][fd=6]
put_local_fd_small_file[path=/testuser/test/test.txt][fd=6]
uploading[path=/testuser/test/test.txt][fd=6][size=0]
s3fs_getattr[path=/testuser/test/test.txt]
stat cache hit [path=/testuser/test/test.txt] [hit count=4]
s3fs_write[path=/testuser/test/test.txt]
s3fs_flush[path=/testuser/test/test.txt][fd=5]
calling get_headers [path=/testuser/test/test.txt]
put_local_fd[path=/testuser/test/test.txt][fd=5]
put_local_fd_small_file[path=/testuser/test/test.txt][fd=5]
uploading[path=/testuser/test/test.txt][fd=5][size=13]
s3fs_release[path=/testuser/test/test.txt][fd=5]
delete_stat_cache_entry[path=/testuser/test/test.txt]
s3fs_getattr[path=/testuser/test]
stat cache hit [path=/testuser/test] [hit count=19]
readdir[path=/testuser/test]
list_bucket [path=/testuser/test]
add_stat_cache_entry[path=/testuser/test/test.txt]
s3fs_getattr[path=/testuser/test]
stat cache hit [path=/testuser/test] [hit count=20]
s3fs_getattr[path=/testuser/test/test.txt]
stat cache hit [path=/testuser/test/test.txt] [hit count=0]

now file content on amazon s3 is "testtest test"

[s3...@googlecode.com]

Comment #5 on issue 180 by dimonz...@gmail.com: When using FTP to send to

an s3fs backed directory, files are appended to instead of overwritten
http://code.google.com/p/s3fs/issues/detail?id=180

I've find out why this problem happens. It appears because of specific
implementation of s3fs_open and behavior of s3fs_get_local_fd with cache
option turned off. When cache is turned off s3fs_get_local_fd creates new
temp file and downloads data from s3 to it each time, but s3fs_open stores
own file descriptor in fuse_file_info which not being updated by truncate
operation.

So, this bug is not reproducing with cache option set.

Also, s3fs_truncate implementation works wrong if size parameter != 0. It
always truncates file to 0 length instead truncate to specified size.

[s3...@googlecode.com]

Comment #6 on issue 180 by p...@pbptech.com: When using FTP to send to an

s3fs backed directory, files are appended to instead of overwritten
http://code.google.com/p/s3fs/issues/detail?id=180

TH problem seems to be the FTP Server in use for transfering files to the
S3FS. It works fine with SFTP, SSH, proftpd and some others. But not vsftpd.

Does anyone know why, or if there is a setting in vsftpd config i need to
set to fix this?

[s3...@googlecode.com]

Comment #7 on issue 180 by jazriel...@jewelry.com: When using FTP to send

to an s3fs backed directory, files are appended to instead of overwritten
http://code.google.com/p/s3fs/issues/detail?id=180

I'm seeing the same issue with vsftpd. Very strange. Would love to get an
update from the devs on this bug!

--
You received this message because this project is configured to send all
issue notifications to this address.
You may adjust your notification preferences at:
https://code.google.com/hosting/settings

[s3...@googlecode.com]

Comment #8 on issue 180 by ggta...@gmail.com: When using FTP to send to an

s3fs backed directory, files are appended to instead of overwritten
http://code.google.com/p/s3fs/issues/detail?id=180

Hi, all

Please check s3fs after r465 for this issue.
Because this problem seems to be caused by s3fs_open() and s3fs's cache
logic.
Latest version(revision) is changed about these logic.
Maybe your problem is solved on new one.

Thanks in advance for your help.