Issue 13752 in dart: polymer build: OS Error: Too many open files

[da...@googlecode.com]

Status: New
Owner: ----
CC: jmes...@google.com, sig...@google.com, rnys...@google.com
Labels: Type-Defect Priority-Unassigned Library-PolymerBuild OpSys-OSX

New issue 13752 by kev...@j832.com: polymer build: OS Error: Too many open
files
http://code.google.com/p/dart/issues/detail?id=13752

project: https://github.com/GoogleChrome/notational_velocity.dart

Running build via hop:

dart tool/hop_runner.dart build

Result:

Exception thrown by task
FileException: Cannot open file, path =
/Users/kevin/.pub-cache/hosted/pub.dartlang.org/analyzer_experimental-0.7.6+4/lib/src/generated/resolver.dart
(OS Error: Too many open files, errno = 24)
#0 _File.open.<anonymous closure> (file_impl.dart:324)
#1 _Future._propagateToListeners.<anonymous closure>
(dart:async/future_impl.dart:449)
#3 _rootRun (dart:async/zone.dart:510)
#4 _ZoneDelegate.run (dart:async/zone.dart:323)
#5 _CustomizedZone.run (dart:async/zone.dart:468)
#6 _Future._propagateToListeners (dart:async/future_impl.dart:441)
#7 _Future._complete (dart:async/future_impl.dart:298)
#8 _Future._asyncComplete.<anonymous closure>
(dart:async/future_impl.dart:354)
#9 _asyncRunCallback (dart:async/event_loop.dart:18)
#10 _createTimer.<anonymous closure>
(dart:async-patch/timer_patch.dart:11)
#11 _Timer._createTimerHandler._handleTimeout (timer_impl.dart:151)
#12 _Timer._createTimerHandler.<anonymous closure> (timer_impl.dart:168)
#13 _ReceivePortImpl._handleMessage
(dart:isolate-patch/isolate_patch.dart:113)

--
You received this message because this project is configured to send all
issue notifications to this address.
You may adjust your notification preferences at:
https://code.google.com/hosting/settings

[da...@googlecode.com]

Comment #1 on issue 13752 by kev...@j832.com: polymer build: OS Error: Too

many open files
http://code.google.com/p/dart/issues/detail?id=13752

0.7.6.4 r28108

All libraries are updated

[da...@googlecode.com]

Updates:
Status: Triaged
Labels: Area-Polymer

Comment #2 on issue 13752 by seth...@google.com: polymer build: OS Error:

Too many open files
http://code.google.com/p/dart/issues/detail?id=13752

(No comment was entered for this change.)

[da...@googlecode.com]

Comment #3 on issue 13752 by sig...@google.com: polymer build: OS Error:

Too many open files
http://code.google.com/p/dart/issues/detail?id=13752

Not sure what's going on here.

FWIW, we explicitly exclude all files in 'polymer' and its dependent
packages when using barback so that barback doesn't have to load any of
these files, but we copy each file by hand at the end of the build step. My
current hunch is that we are coping too many files in parallel and
the 'dart:io' library is failing on us.

Maybe switching to a sync copy instead of an async copy will fix the issue?
I hope to take a closer look later today.

[da...@googlecode.com]

Comment #4 on issue 13752 by devon...@google.com: polymer build: OS

Error: Too many open files
http://code.google.com/p/dart/issues/detail?id=13752

As a point of info, I am running into this as well. This is for deploying
polymer in a chrome app, but I see this when trying to deploy the editor's
sample polymer app as well. Strangely, it fails on the command-line:

dart build.dart --deploy web/foofoo.html

but works with the deploy command run from the editor's context menu.

[da...@googlecode.com]

Comment #5 on issue 13752 by kev...@j832.com: polymer build: OS Error: Too

many open files
http://code.google.com/p/dart/issues/detail?id=13752

Work around:

It seems the shell has a different limit on mac.

Type 'ulimit -a' -- my default I was getting 256

I just changed it to my 'ulimit -n 512'

The process now completes.

Random, annoying, but at least there's a work-around

[da...@googlecode.com]

Comment #6 on issue 13752 by sig...@google.com: polymer build: OS Error:

Too many open files
http://code.google.com/p/dart/issues/detail?id=13752

I don't have a Mac at hand, can you tell me if this works for you:

change this method:
https://github.com/dart-lang/bleeding_edge/blob/master/dart/pkg/polymer/lib/src/build/runner.dart#L305

and use synchronous read/writes instead of async ones? Something among
these lines:

new File(outpath).writeBytesSync(new File(inpath).readBytesSync());

[da...@googlecode.com]

Comment #7 on issue 13752 by d...@google.com: polymer build: OS Error: Too

many open files
http://code.google.com/p/dart/issues/detail?id=13752

Just a heads up to let you know that the workaround didn't work for me. It
certainly requires a higher limit in my case. Though, I'll just wait for
the fix.

[da...@googlecode.com]

Comment #8 on issue 13752 by sig...@google.com: polymer build: OS Error:

Too many open files
http://code.google.com/p/dart/issues/detail?id=13752

Ok - I managed to test it out locally (setting my limit in linux to 256).
I'll send out a fix shortly.

[da...@googlecode.com]

Comment #9 on issue 13752 by jmes...@google.com: polymer build: OS Error:

Too many open files
http://code.google.com/p/dart/issues/detail?id=13752

fyi -- Bob seemed to think this could be worked around in a general way in
barback.

[da...@googlecode.com]

Comment #10 on issue 13752 by sig...@google.com: polymer build: OS Error:

Too many open files
http://code.google.com/p/dart/issues/detail?id=13752

Good to know. copyFile is affected by the number of files in packages that
we don't process with barback, so I just sent out this CL to fix that part:

https://codereview.chromium.org/26071003/

[da...@googlecode.com]

Updates:
Status: Started

Comment #11 on issue 13752 by kev...@j832.com: polymer build: OS Error: Too

many open files
http://code.google.com/p/dart/issues/detail?id=13752

I tried Siggi's CL without luck.

Here's an alternate attempt: https://codereview.chromium.org/26268002

I replaced the usage of Future.wait w/ a Future.forEach

This ensures that we don't get overlapping file.open for every asset passed
in.

Speed also seems fine.

[da...@googlecode.com]

Updates:
Status: Fixed

Comment #12 on issue 13752 by kev...@j832.com: polymer build: OS Error: Too

many open files
http://code.google.com/p/dart/issues/detail?id=13752

Addressed in r28314, although we may want a more general solution

[da...@googlecode.com]

Comment #13 on issue 13752 by sig...@google.com: polymer build: OS Error:

Too many open files
http://code.google.com/p/dart/issues/detail?id=13752

Thanks Kevin!

I tried my patch setting the ulimit -n 256 and worked ok. But I think I
tried it with a smaller example, so I'm not surprised you also had to fix
how we read/write assets on the other code path.

[da...@googlecode.com]

Comment #14 on issue 13752 by d...@google.com: polymer build: OS Error: Too

many open files
http://code.google.com/p/dart/issues/detail?id=13752

How can I use the latest?
I tried to update with pub (polymer 0.8.1) and it did not fix the issue.

[da...@googlecode.com]

Comment #15 on issue 13752 by sig...@google.com: polymer build: OS Error:

Too many open files
http://code.google.com/p/dart/issues/detail?id=13752

Since Kevin submitted it this morning, this wont be available in pub until
the next release.

If there were no other changes, I can try to coordinate releasing a hotfix
if you need it though.

[da...@googlecode.com]

Comment #16 on issue 13752 by dinh.vie...@gmail.com: polymer build: OS

Error: Too many open files
http://code.google.com/p/dart/issues/detail?id=13752

Just an update about it. With the patch of Kevin, I still had to raise the
limit of opened file to 512 to have it working.
I think we should re-open the issue then.

[da...@googlecode.com]

Updates:
Status: Triaged

Comment #17 on issue 13752 by sig...@google.com: polymer build: OS Error:

Too many open files
http://code.google.com/p/dart/issues/detail?id=13752

sigh, you might be hitting the issue earlier then. Does your project have a
lot of html files by any chance? I wonder if in your case we are hitting
the problem while barback itself is running.

[da...@googlecode.com]

Comment #18 on issue 13752 by kev...@j832.com: polymer build: OS Error: Too

many open files
http://code.google.com/p/dart/issues/detail?id=13752

Perhaps we need a throttledFutureForEach(iterable, Future exec(e))

Expose it publicly from barback

It's basically a hybrid of Future.wait and .forEach

It allows a maximum number of parallel ops (100?) for all callers.

Then barback and folks using barback can have some level of coordination on
the number of open files.

This also eliminates all of the List<Future> variables sprinkled around.

[da...@googlecode.com]

Comment #19 on issue 13752 by d...@google.com: polymer build: OS Error: Too

many open files
http://code.google.com/p/dart/issues/detail?id=13752

Maybe we should not try to parallelize disk operations. Anyway it's blocked
by I/O.

[da...@googlecode.com]

Comment #21 on issue 13752 by jmes...@google.com: polymer build: OS

Error: Too many open files
http://code.google.com/p/dart/issues/detail?id=13752

@kevin, that only works if there aren't other async operations going
elsewhere.

I think trying to throttle the async ops is the wrong approach. Better
would be something that throttles # of open files.

Honestly, we might be better off just setting the ulimit higher, rather
than messing up our code (and slowing it down on other platforms) just to
deal with a quirk of Mac OS

[da...@googlecode.com]

Comment #22 on issue 13752 by kev...@j832.com: polymer build: OS Error: Too

many open files
http://code.google.com/p/dart/issues/detail?id=13752

I *strongly* disagree. There will be polymer projects with dozens of
components and dozens of packages with hundreds or thousands of assets.

They will be built on Mac.

One cannot set a ulimit higher than 512 without sudo.

This needs to be fixed generally without requiring bash tricks for the user.

[da...@googlecode.com]

Comment #23 on issue 13752 by d...@google.com: polymer build: OS Error: Too

many open files
http://code.google.com/p/dart/issues/detail?id=13752

kevin@: +1

[da...@googlecode.com]

Comment #24 on issue 13752 by jmes...@google.com: polymer build: OS

Error: Too many open files
http://code.google.com/p/dart/issues/detail?id=13752

I don't get why it's so onerous. When I searched around for Mac file
limits, there were plenty of hits of folks hitting this problem and
suggesting solutions, including ways to fix it permanently.

[da...@googlecode.com]

Comment #25 on issue 13752 by jmes...@google.com: polymer build: OS

Error: Too many open files
http://code.google.com/p/dart/issues/detail?id=13752

here's another way of looking at it: if you fix it once on your machine,
you're done :). If you don't, you need to follow up with every Dart
library/app ever, and try and convince the developers to switch all their
code to sync file IO. That seems hard.

Unless we can fix it at the library level, which of course is better :)

[da...@googlecode.com]

Comment #26 on issue 13752 by d...@google.com: polymer build: OS Error: Too

many open files
http://code.google.com/p/dart/issues/detail?id=13752

I think I'd prefer if it would "just work" by default.

[da...@googlecode.com]

Updates:
Labels: -Library-PolymerBuild -Area-Polymer Area-IO

Comment #27 on issue 13752 by jmes...@google.com: polymer build: OS

Error: Too many open files
http://code.google.com/p/dart/issues/detail?id=13752

here's info on the node.js solution:
http://stackoverflow.com/questions/8965606/node-and-error-emfile-too-many-open-files,
https://github.com/isaacs/node-graceful-fs

if thes maybe we need "package:io" to fix the issue if we cannot
change "dart:io"

@dvh @kevmoo -- I feel your pain... I'm typing this on a Mac laptop :). I
just don't like that this bug is Area=Polymer. We're just an application
that happens to open files. Any Dart code can hit this and the "fix" taken
is not scalable (as evidenced by the fact that the bug was reopened after
several patches). Also, it makes other platforms worse.

It's frustrating to me because we were using dart:io exactly as prescribed
(no doubt if we'd tried to use sync IO, someone would've yelled about it in
code review). And here we are with our code throwing exceptions on Mac.

I'm assigning this to Area-IO. Hopefully we can come up with a general
solution (either dart:io or something like package:io), and not keep
hacking Polymer's build code.

[da...@googlecode.com]

Comment #28 on issue 13752 by jmes...@google.com: polymer build: OS

Error: Too many open files
http://code.google.com/p/dart/issues/detail?id=13752

oops, typo, ignore "if thes" ... I must've tried to backspace it and got
one letter instead of selecting both words :)

[da...@googlecode.com]

Comment #29 on issue 13752 by dgr...@google.com: polymer build: OS Error:

Too many open files
http://code.google.com/p/dart/issues/detail?id=13752

Perhaps a package along these lines?
https://github.com/isaacs/node-graceful-fs

[da...@googlecode.com]

Comment #30 on issue 13752 by kev...@j832.com: polymer build: OS Error: Too

many open files
http://code.google.com/p/dart/issues/detail?id=13752

Here's my solution https://codereview.chromium.org/26273003

The method is forEachPooledFuture - looks like Future.forEach, but with a
hard limit of 10 ops running at the same time.

barback tests pass
poylmer tests pass
my app builds without any console tricks

[da...@googlecode.com]

Comment #31 on issue 13752 by devon...@google.com: polymer build: OS

Error: Too many open files
http://code.google.com/p/dart/issues/detail?id=13752

@Kevin, thanks for working that solution up into a patch!

@John, I hope we can land this. This solves a real problem users are having
building polymer apps. I'm understand what you're saying about pushing a
solution down to dart:io. But I think from a practical perspective, tools
built on top of those AIPs need to be aware of certain limitations in their
usage.

Perhaps a longer term goal would be to push forEachPooledFuture into
dart:async?

[da...@googlecode.com]

Comment #32 on issue 13752 by rnys...@google.com: polymer build: OS Error:

Too many open files
http://code.google.com/p/dart/issues/detail?id=13752

Kevin's solution is nice, but I really don't think punting to the user is
the right answer here. People have to know when to use this pooled future
method instead of Future.wait() or Future.forEach(). I also agree with John
that making this an async thing obscures the fact that it's an IO issue.

I'd rather have a wrapper around opening a file, not a wrapper around
working with futures. For Barback, that's probably what I'll end up doing.
(Barback already has an abstraction layer wrapping IO, so this is
relatively easy for us.)

But I think we'd benefit the entire Dart community if we solved it at the
IO layer. Either in dart:io, or by making a package that wraps Dart IO that
we encourage users to use. See:
https://code.google.com/p/dart/issues/detail?id=13882

[da...@googlecode.com]

Comment #33 on issue 13752 by jmes...@google.com: polymer build: OS

Error: Too many open files
http://code.google.com/p/dart/issues/detail?id=13752

forEachPooledFuture is not the right solution. It's attacking the problem
at the wrong level. The issue is not that we need to cap the number of
async ops in general (a job it fails at anyway, since other futures might
be running). We need something that specifically attacks open file, like
graceful-fs linked earlier.

[da...@googlecode.com]

Comment #34 on issue 13752 by jmes...@google.com: polymer build: OS

Error: Too many open files
http://code.google.com/p/dart/issues/detail?id=13752

Oops, comment race condition :). Agree 100% with Bob's message.

[da...@googlecode.com]

Comment #35 on issue 13752 by kev...@j832.com: polymer build: OS Error: Too

many open files
http://code.google.com/p/dart/issues/detail?id=13752

I'd love for IO to offer a solution.

In the mean time:
- Future.wait - blows up on Mac for at least two projects
- Future.forEach - works great, but it's a bit slower
- Kevin's hack - offers some of the benefits of Future.wait, without
blowing up

Short term: can we move to Future.forEach or some manifestation of my hack?

[da...@googlecode.com]

Updates:
Cc: ajoh...@google.com sgj...@google.com

Comment #36 on issue 13752 by seth...@google.com: polymer build: OS Error:

Too many open files
http://code.google.com/p/dart/issues/detail?id=13752

Hi guys, I noticed that no one from the IO team is on this thread. Might I
suggest opening a discussion with the folks I just CC'ed and in the
meantime offer a solution for people who need to launch Polymer apps today
(with the understanding and expectation we'll replace it with something
better.)

[da...@googlecode.com]

Comment #37 on issue 13752 by ag...@google.com: polymer build: OS Error: Too

many open files
http://code.google.com/p/dart/issues/detail?id=13752

Ultra quick input from me since I'm travelling.

This is something the user of dart:io has to deal with. There is no way
that dart:io can know which async operations can safely be delayed. If
dart:io starts guessing it will end up dead-locking applications. Only the
app developer knows which of the async io operations can wait and therefore
he needs to throttle his IO.

Providing a small package to help with that could be good but the
programmer does need to worry about throttling when using async io
programming. That is the name of the game and I don't see any way the
dart:io library can correctly throttle for the user.

[da...@googlecode.com]

Comment #38 on issue 13752 by ajoh...@google.com: polymer build: OS Error:

Too many open files
http://code.google.com/p/dart/issues/detail?id=13752

Hi,

The IO team is OOO these days, so I'll just throw in some quick comments.

@kev...@j832.com: I think your solution in
https://codereview.chromium.org/26273003 is very elegant (didn't read the
code very closely, but thinking about the overall approach), and IMO a good
fix for now.

Note that the file-descriptor limit is shared between _all_
file-descriptors. Before we get to much into designing a new File package
that will 'queue' up async requests when seeing EMFILE or similar, note
that using up all file-descriptors will also lead to lock-down of
ServerSockets as well (not able to accept new connections), etc.

I hope this can help moving polymer forward for now.

Cheers,

- Anders

[da...@googlecode.com]

Comment #39 on issue 13752 by d...@google.com: polymer build: OS Error: Too

many open files
http://code.google.com/p/dart/issues/detail?id=13752

Here's my thoughts about it:
- I don't care about speed since it's a "compile phase".
- If people care about speed, we could add a flag (for example "-j 8" to
had 8 parallel operations).

I think that serializing it would be fine for me as a first step (using
queues or blocking operations).
It sounds like a dead simple (not trying to be smart) solution to that
problem.