[jruby-user] [jruby-1.7.0.preview2] WebBrick: ERROR RuntimeError: can't add a new key into hash during iteration

[Mark Mandel]

I've got a REST webservice I've built with JRuby 1.7 p2, Sinatra and Mizuno.

Intermittently, and only under some load, I see the following error from it:

[2012-08-19 21:13:50] ERROR RuntimeError: can't add a new key into hash during iteration

org/jruby/RubyHash.java:905:in `[]='

/opt/jruby/active/lib/ruby/1.9/webrick/utils.rb:205:in `register'

/opt/jruby/active/lib/ruby/1.9/webrick/utils.rb:161:in `register'

org/jruby/ext/thread/Mutex.java:149:in `synchronize'

/opt/jruby/active/lib/ruby/1.9/webrick/utils.rb:160:in `register'

/opt/jruby/active/lib/ruby/1.9/webrick/utils.rb:232:in `timeout'

/opt/jruby/active/lib/ruby/1.9/webrick/httprequest.rb:398:in `_read_data'

/opt/jruby/active/lib/ruby/1.9/webrick/httprequest.rb:409:in `read_line'

/opt/jruby/active/lib/ruby/1.9/webrick/httprequest.rb:314:in `read_header'

/opt/jruby/active/lib/ruby/1.9/webrick/httprequest.rb:92:in `parse'

/opt/jruby/active/lib/ruby/1.9/webrick/httpserver.rb:81:in `run'

/opt/jruby/active/lib/ruby/1.9/webrick/server.rb:191:in `start_thread'

NoMethodError: undefined method `<' for nil:NilClass

  initialize at /opt/jruby/active/lib/ruby/1.9/webrick/utils.rb:181

        each at org/jruby/RubyArray.java:1612

  initialize at /opt/jruby/active/lib/ruby/1.9/webrick/utils.rb:179

        each at org/jruby/RubyHash.java:1192

  initialize at /opt/jruby/active/lib/ruby/1.9/webrick/utils.rb:178

It seems to be coming from inside of webrick, so I'm at a loss as to what I could be doing that would be causing this?

The upstart command for Mizuno is the following:

exec su -c 'cd /home/foo/current; ./bin/mizuno >> output.log 2>&1' -l deploy

Any help would be appreciated in working out what is causing the error.

Thanks!

Mark

--
E: mark....@gmail.com
T: http://www.twitter.com/neurotic
W: www.compoundtheory.com

2 Devs from Down Under Podcast

http://www.2ddu.com/

[Keith Bennett]

Mark -

I've had issues like this when modifying a hash that I am already iterating over.  So instead of:

my_hash.each_pair { ... do something that potentially modifies the hash ... }

keys = my_hash.keys
keys.each { ... do something that potentially modifies the hash ... }

(or my_hash.keys.each {}, I think would work too.)

Don't know if this is your issue, but I hope it helps.

Are you deleting a key inside the loop?  If so, I don't think the enumerable on which the loop depends would know.  It would try to perform an iteration on that key, that would now be missing.

If you're just adding keys, and you're iterating over the hash to which you're adding them, you might instead want to create a temporary hash for the additions within the loop, and then merge the two hashes after the loop.

- Keith

--
Keith R. Bennett
http://about.me/keithrbennett

[Mark Mandel]

Thanks for the advice.

The only issue is, it's inside JRuby's Webrick implementation that the error is occurring, which is not something I have control over:

https://github.com/jruby/jruby/blob/master/lib/ruby/1.9/webrick/utils.rb#L205

It seems to only happen once until all the values get populated, but the error seems to be while this loop is running:

https://github.com/jruby/jruby/blob/master/lib/ruby/1.9/webrick/utils.rb#L181

Seems like a bug in the webrick implementation, no?

@timeout_info.each {} is iterating, while @timeout_info[thread] ||= Array.new is adding new values to the hash.

The error stops happening once the thread pool has filled, and the hash doesn't get iterated over anymore.  And it happens intermittently, just because the while loop could be sleeping during that time.

I think you are right though, I could monkey patch this to use @timeout_info.keys.each, and see if that resolves the issue, and if so, push it back.

Thanks for your help!

Mark

[Keith Bennett]

Mark -

My opinion is that even if the error only manifests itself intermittently, the issue is a conceptual one (not to modify the hash you're iterating over), and it's best to avoid it.

Cheers,

Keith

--
Keith R. Bennett
http://about.me/keithrbennett

[Mark Mandel]

Sorry, I'm confused - are you saying that I shouldn't be using a web framework with a Webrick layer then?

Mark

On Mon, Aug 20, 2012 at 9:50 PM, Keith Bennett <keithr...@gmail.com> wrote:

My opinion is that even if the error only manifests itself intermittently, the issue is a conceptual one (not to modify the hash you're iterating over), and it's best to avoid it.

[Keith Bennett]

No, not at all.  I'm just saying that it's better not to iterate over the hash, but instead get its keys, and then loop on them.

I may be misunderstanding your code, but I thought you were using hash.each_pair or something like that.

- Keith

--
Keith R. Bennett
http://about.me/keithrbennett

[Mark Mandel]

I think were we are getting our wires crossed is over one specific point: This is not my code. This is code that is implement in JRuby's implementation of Webrick.

I have no control over that code whatsoever, unless I monkey patch it or similar.

(if you look at the source links, you can see it all goes back to the official JRuby GitHub repository).

Either way, what you were saying is making a lot of sense, and I may be able to patch this code and push it back to JRuby (here's hoping I don't break anything).

Mark

[kristian]

I guess first step is to monkey patch and see if it helps :)

that code comes from ruby stdlib - not sure where the right place for
patching JRuby problems with this though.

I just needed added my two cents to this thread :)
-Kristian

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email

[Mark Mandel]

Thanks for your help.

Actual crux of my issue - my config.ru was totally not correct, and I was actually just running Webrick, rather than Mizuno.

* facepalm *

Live and learn.

Mark

[Brian Walsh]

I'm having the same problem with:

jruby 1.7.0 

rails (3.2.8)

Is there a specific change to enable testing with webrick?

-b

[Charles Oliver Nutter]

After reading through this thread, I'm not clear whether there's an
issue to be fixed here. Seems like the original reporter's issue was
due to a bad config.ru. And if there's actually anything to be fixed,
it sounds like it's something in WEBrick.

At this point, if there's an issue, please file it and we'll try to fix it.

- Charlie

[Brian Walsh]

Agreed.  I just tested with trinidad and did not have this problem.

Thank you

[Mark Mandel]

The upshot was, for my original post:

Don't use webrick in anything in but development (which is well documented). It tends to fall over under load.

I had a bad config.ru, and also very little knowledge of J/Ruby, and din't realise this, and thought I was running Mizuno (which is fine under load), when in fact all I was running was Webrick.

Mark