Issue 274 in s3fs: S3FS consuming 11.1G of RAM

[s3...@googlecode.com]

Status: New
Owner: ----
Labels: Type-Defect Priority-Medium

New issue 274 by ad...@onair.tv: S3FS consuming 11.1G of RAM
http://code.google.com/p/s3fs/issues/detail?id=274

Detailed description of observed behavior:

32748 root 20 0 11.1G 10.8G 2036 S 0.0 73.6 1h29:25 s3fs ddccs-files
/var/www/sites/default/files -o
rw,noatime,allow_other,uid=33,gid=3,use_cache=/tmp,default_acl=public-read

What steps will reproduce the problem - please be very specific and
detailed. (if the developers cannot reproduce the issue, then it is
unlikely a fix will be found)?

S3FS version 1.61
AWS EC2 instance with S3 bucket
15GB RAM
4 EC2 units
Ubuntu 11.10

===================================================================
The following information is very important in order to help us to help
you. Omission of the following details may delay your support request or
receive no attention at all.
===================================================================
Version of s3fs being used (s3fs --version):

Version of fuse being used (pkg-config --modversion fuse):

System information (uname -a):

Distro (cat /etc/issue):

s3fs command line used (if applicable):

/etc/fstab entry (if applicable):

s3fs syslog messages (grep s3fs /var/log/syslog):

[s3...@googlecode.com]

Comment #1 on issue 274 by sklia...@gmail.com: S3FS consuming 11.1G of RAM
http://code.google.com/p/s3fs/issues/detail?id=274

I see it consumed 90 minutes of CPU time. Can you tell me how long the s3fs
been running?
How many queries there are on average?

We consider to use s3fs in production and try to get feedback from others
who used s3fs seriously.

[s3...@googlecode.com]

Comment #2 on issue 274 by vkviatko...@gmail.com: S3FS consuming 11.1G of
RAM
http://code.google.com/p/s3fs/issues/detail?id=274

Have the same issue. When listing S3 bucket with big amount of files (>
400000), s3fs starts consuming memory up to its limit and then got killed
by kernel.
Here is the output from grep s3fs /var/log/messages:

May 28 08:12:32 devc1ffappex1 kernel: s3fs invoked oom-killer:
gfp_mask=0x201da, order=0, oom_adj=0
May 28 08:12:32 devc1ffappex1 kernel: s3fs cpuset=/ mems_allowed=0
May 28 08:12:32 devc1ffappex1 kernel: Pid: 22007, comm: s3fs Not tainted
2.6.32-131.17.1.el6.x86_64 #1
May 28 08:12:32 devc1ffappex1 kernel: Out of memory: kill process 22005
(s3fs) score 292656 or a child
May 28 08:12:32 devc1ffappex1 kernel: Killed process 22005 (s3fs)
vsz:14047500kB, anon-rss:7381776kB, file-rss:68kB
May 28 08:13:46 devc1ffappex1 s3fs: init $Rev: 424 $
May 28 08:24:32 devc1ffappex1 s3fs: init $Rev: 424 $

What is bad - s3fs crashes so needs to be restarted. Is it possible to
track such cases and just list first 10000 for example (configurable
parameter would be great)?

--
You received this message because this project is configured to send all
issue notifications to this address.
You may adjust your notification preferences at:
https://code.google.com/hosting/settings

[s3...@googlecode.com]

Comment #3 on issue 274 by ggta...@gmail.com: S3FS consuming 11.1G of RAM
http://code.google.com/p/s3fs/issues/detail?id=274

Hi

s3fs v1.61 can not clear curl handle when s3fs gets curl error(example:
Couldn't connect to server).
And if there are many object(over 400000) in directory, old s3fs makes each
head request into one multi-request(curl_multi).
I think that maybe almost request in multi request are failed by
CURLE_COULDNT_CONNECT.
Then memory leaks is occurred and memory usage grownup.

I think that "90 minutes of CPU time" is because it is taken by request
when there are many objects in directory.

Revision 432 fixes about multi request, I hope it helps this issue.
Please use new revision(not packaged now), and let me know about this case.

Regards,

[s3...@googlecode.com]

Comment #4 on issue 274 by vkviatko...@gmail.com: S3FS consuming 11.1G of
RAM
http://code.google.com/p/s3fs/issues/detail?id=274

Forgot to mention - I have this issue on the latest 1.69 version. My CPU
usage was not high (11-20%). Ok, we'll try this out, can wait for packaged
version.
Thanks!

[s3...@googlecode.com]

Updates:
Status: Fixed

Comment #5 on issue 274 by ggta...@gmail.com: S3FS consuming 11.1G of RAM
http://code.google.com/p/s3fs/issues/detail?id=274

OK, I closed this issue.

Thanks