Issue 324217 in chromium: inotify_init() failed: Too many open files

[chro...@googlecode.com]

Status: Assigned
Owner: euge...@chromium.org
Labels: Type-Bug Pri-2 OS-Chrome

New issue 324217 by euge...@chromium.org: inotify_init() failed: Too many
open files
http://code.google.com/p/chromium/issues/detail?id=324217

unit_tests hang on chromeos/valgrind bot with "Too many open files" in
file_path_watcher_linux.cc.

http://build.chromium.org/p/chromium.memory.fyi/builders/Chromium%20OS%20%28valgrind%29%286%29/builds/22795/steps/memory%20test%3A%20unit/logs/stdio

...

[----------] 2 tests from FileManagerFileWatcherTest
[ RUN ] FileManagerFileWatcherTest.AddAndRemoveOneExtensionId
[ OK ] FileManagerFileWatcherTest.AddAndRemoveOneExtensionId (17 ms)
[ RUN ] FileManagerFileWatcherTest.WatchLocalFile
[5031:5031:1127/085944:2741428393220:ERROR:file_path_watcher_linux.cc(221)]
inotify_init() failed: Too many open files

command timed out: 1200 seconds without output, attempting to kill
process killed by signal 9


This started happening in this build:
http://build.chromium.org/p/chromium.memory.fyi/builders/Chromium%20OS%20%28valgrind%29%286%29/builds/22795

All new commits in that build look unrelated.
This could be a file descriptor leak somewhere else.

This new change looks like it opens files, and this test is executed on the
failing bot:
http://src.chromium.org/viewvc/chrome?view=rev&revision=237579


--
You received this message because this project is configured to send all
issue notifications to this address.
You may adjust your notification preferences at:
https://code.google.com/hosting/settings

[chro...@googlecode.com]

Updates:
Cc: hash...@chromium.org

Comment #1 on issue 324217 by euge...@chromium.org: inotify_init() failed:

Too many open files
http://code.google.com/p/chromium/issues/detail?id=324217

I see the hanging process. There are just ~27 open files with the limit of
65K (ulimit -a). There are 2 "unusual" descriptors:

9 -> anon_inode:[eventfd]
12 -> anon_inode:[eventpoll]

[chro...@googlecode.com]

Comment #2 on issue 324217 by euge...@chromium.org: inotify_init() failed:

Too many open files
http://code.google.com/p/chromium/issues/detail?id=324217

In fact, the error message is misleading. "Too many open files" actually
refers to:

EMFILE The user limit on the total number of inotify instances has been
reached.

[chro...@googlecode.com]

Updates:
Status: Fixed

Comment #3 on issue 324217 by euge...@chromium.org: inotify_init() failed:

Too many open files
http://code.google.com/p/chromium/issues/detail?id=324217

This is no longer happening after the bot was restarted.

[chro...@googlecode.com]

Updates:
Status: Available
Cc: euge...@chromium.org

Comment #5 on issue 324217 by gli...@chromium.org: inotify_init() failed:

Too many open files
http://code.google.com/p/chromium/issues/detail?id=324217

This is happening again:
http://build.chromium.org/p/chromium.memory.fyi/builders/Chromium%20OS%20%28valgrind%29%285%29/builds/24183/steps/memory%20test%3A%20unit/logs/stdio

[ RUN ] FileManagerFileWatcherTest.WatchLocalFile
[19789:19789:1210/035427:3846349045242:ERROR:file_path_watcher_linux.cc(221)]

inotify_init()
failed: Too many open files

[chro...@googlecode.com]

Updates:
Owner: ---
Cc: mnis...@chromium.org bre...@chromium.org

Comment #6 on issue 324217 by gli...@chromium.org: inotify_init() failed:

Too many open files
http://code.google.com/p/chromium/issues/detail?id=324217

(No comment was entered for this change.)

[chro...@googlecode.com]

Comment #7 on issue 324217 by bugdro...@chromium.org: inotify_init()

failed: Too many open files

http://code.google.com/p/chromium/issues/detail?id=324217#c7

------------------------------------------------------------------------
r239780 | gli...@chromium.org | 2013-12-10T14:32:15.202202Z

Changed paths:
M
http://src.chromium.org/viewvc/chrome/trunk/tools/build/masters/master.chromium.memory.fyi/master.cfg?r1=239780&r2=239779&pathrev=239780

Enable auto-rebooting of the bots on the Memory FYI waterfall.

BUG=324217
TBR=timu...@chromium.org

Review URL: https://codereview.chromium.org/110783003
------------------------------------------------------------------------

[chro...@googlecode.com]

Updates:
Cc: mar...@chromium.org vad...@chromium.org

Comment #9 on issue 324217 by vad...@chromium.org: inotify_init() failed:
Too many open files
https://code.google.com/p/chromium/issues/detail?id=324217

In this case it happened on Swarming bot (swarm222-c4):
https://chromium-swarm.appspot.com/user/task/14976f962d62f00 (how I found
this: open full log, search for "Too many open file", scroll up until Shard
header (ASCII art box...) Shard 0 in that case.)

This bot failed lot of tasks due to this error. I thought bots reboot after
task failures...

M-A is it still so?

[chro...@googlecode.com]

Comment #10 on issue 324217 by mar...@chromium.org: inotify_init() failed:

Too many open files
https://code.google.com/p/chromium/issues/detail?id=324217

Yes bots reboot after failure. Unless there's a catastrophic failure where
they can't even reboot. This normally only happen on Windows.

[chro...@googlecode.com]

Comment #11 on issue 324217 by mar...@chromium.org: inotify_init() failed:

Too many open files
https://code.google.com/p/chromium/issues/detail?id=324217

Yes there was bug introduced in the past days, which I fixed moments ago.
It seems to have killed ~half (23k) of the tasks in the past 24 hours.

[chro...@googlecode.com]

Comment #12 on issue 324217 by mar...@chromium.org: inotify_init() failed:

Too many open files
https://code.google.com/p/chromium/issues/detail?id=324217

Err, I misread; 915 failures out of 26429 tasks; 25532 success so it's not
*that* bad.