rack-contrib reorg
Ryan Tomayko
around a bit to prevent potential name clashes with rack core. Right now
all of the source files for rack-contrib are named as 'rack/foo.rb' and
attach themselves to the Rack module as Rack::Foo.
At the very least, we need to move the files from 'rack/' into a
different directory to prevent require clashes and rubygems weirdness. e.g.,
lib/rack/foo.rb -> lib/rack/contrib/foo.rb
What I'm not sure about is whether we also want to move all of the
objects into a separate module once loaded. e.g.,
Rack::Foo -> Rack::Contrib::Foo
For some reason, I'd like to leave the classes under the Rack module but
I'm not sure that this accomplishes our goals. Would you guys mind
weighing in here with what makes sense to you so I can take the
temperature a bit?
Also, let me know if you have any local changes that should go in before
the big move takes place so that we can plan accordingly.
Thanks,
Ryan
Daniel Roethlisberger
> What I'm not sure about is whether we also want to move all of the
> objects into a separate module once loaded. e.g.,
>
> Rack::Foo -> Rack::Contrib::Foo
Personally, I think it would make sense to be able to move stuff
from contrib to rack proper (and back) without breaking existing
code.
--
Daniel Roethlisberger
http://daniel.roe.ch/
Ryan Tomayko
> Ryan Tomayko <r...@tomayko.com> 2009-01-12:
>> What I'm not sure about is whether we also want to move all of the
>> objects into a separate module once loaded. e.g.,
>>
>> Rack::Foo -> Rack::Contrib::Foo
>
> Personally, I think it would make sense to be able to move stuff
> from contrib to rack proper (and back) without breaking existing
> code.
Right. I think that's what makes me want to leave it under Rack as well.
rack/contrib.rb adds autoloads under the Rack module for all
rack-contrib components so it's a nice little compatibility story: you
require 'rack/contrib' and use the autoloads to bring in the other
modules. Later, when things get moved from rack-contrib into rack (or
vise-versa), your code should continue to work as long as you don't
require 'rack/contrib/foo' directly.
Is that too much magic? I can't see a lot of benefit in forcing an
explicit project reference here.
Thanks,
Ryan
Nicolás Sanguinetti
No, I think that's the right approach.
-foca
Yehuda Katz
Ryan Tomayko
> What about the autoload threadsafety issues?
We talked about this on IRC the other day and were unable to come up
with a real scenario where it would cause an actual issue. None of the
rack-contrib classes would be first referenced within the context of a
request. They should all be first referenced from the "main thread"
since they're required to construct the application.
We should probably discuss this more broadly, though. Both rack and
rack-contrib use autoload for everything. I'm not familiar enough with
the threading issues to say strongly whether that should change but my
impression is that 1.) an actual problem here would be extremely rare,
and 2.) it's fairly simple for people that do run into an issue to work
around it by using explicit requires in a main thread or synchronized block.
Thanks,
Ryan
Michael Fellinger
I haven't actually been able to reproduce the problem while using
Rack, but ran into it in another place.
Maybe doing something like this could fix it?
http://paste.rubyists.com/pastes/view/15346
^ manveru
Joshua Peek
> What about the autoload threadsafety issues?
Not a real issue since middleware classes are referenced when your
application is initialized (before any requests threads are spawned).
Though it could be a problem if your application is dynamically
rebuilding itself on the fly ;)
+1 for keeping the autoload approach.
--
Joshua Peek
Ryan Tomayko
I went ahead and pushed the change that moved the source files:
http://github.com/rack/rack-contrib/commit/c55a655
This will break code that required a rack-contrib module directly.
We can deal with the autoload issue separately. manveru was going to try
to get a test case together that illustrated the problem in Rack (with
Rack::Utils being referenced within the request context). I don't think
rack-contrib will be impacted but we may want to use the same
conventions in both places for parity.
Thanks,
Ryan
James Tucker
What about the autoload threadsafety issues?
On Mon, Jan 12, 2009 at 4:24 PM, Nicolás Sanguinetti <god...@gmail.com> wrote:
On Mon, Jan 12, 2009 at 9:46 PM, Ryan Tomayko <r...@tomayko.com> wrote:
>
> Is that too much magic? I can't see a lot of benefit in forcing an
> explicit project reference here.
No, I think that's the right approach.
Matt Todd
Yehuda Katz
James Tucker
On 13 Jan 2009, at 16:55, Yehuda Katz wrote:
> The difference between require and autoload is that require is
> triggered by the "user" on purpose, and is almost always triggered
> at boot-time.
A quick grep for spaces in front of require in very commonly used ruby
projects can reveal interesting contradictions.
> Autoload is triggered by the *interpreter*, and it's impossible to
> mutex the require that is caused by autoload, even if you would want
> to.
Mutexes are not a solution to thread safety. They are just a tool.
Yes, you can't protect autoload, but if you were that worried about a
mutex, you could re-write it on the ruby side. That wasn't accepted
because of the deadlock potentials. Those exist in require, too.
> The important thing is to make SURE that nothing is being autoloaded
> that will be hit the first time at runtime.
Nope.
The important thing is to make sure that code running in potentially
multiple threads does not raise race conditions.
This is one of those areas where you get common statements not holding
to the true state of the world, such as "survival of the fittest",
which is wrong, and should be in fact "unsurvival of the unfittest".
Yes - this does mean you could autoload twice, but you can require
twice with late requires, too.
This remains ok until assignment is not atomic. When that happens,
we've got a whole other world of crap, and again, require doesn't help
unless you eliminate all of what I mentioned at the top.
> I suppose the preload solution works, but we still need to know
> which autoloads won't be triggered until runtime, and make sure to
> include them in the preloader.
No, you *need* to add unsafe loads to the preloader, and that's it. If
you want to be 'extensively safe', don't use threads. I'm dead serious
about that.
> This situation is more complicated than it looks at first glance,
> and asking all users of Rack to understand how it works so that they
> can make the appropriate decision is unfair burden-shifting onto
> many other users of Rack.
So you're suggesting users run around with burning scissors and their
eyes shut? (wrt Threading without understanding Threading impacts).
Like I explained, this problem is in no way unique to autoload.
The critical thing, is to not write runtime code in classes that are
to be late loaded or late evaluated, or, pre-run that code. That rule
needs to be understood by ruby developers who write in a threaded
environment, and has nothing to do (in reality) with the specific
loading mechanism. As I've already said, hiding issues by taking out
the obvious hits, doesn't make them magically disappear.
Yehuda Katz
A quick grep for spaces in front of require in very commonly used ruby projects can reveal interesting contradictions.
On 13 Jan 2009, at 16:55, Yehuda Katz wrote:
The difference between require and autoload is that require is triggered by the "user" on purpose, and is almost always triggered at boot-time.
Mutexes are not a solution to thread safety. They are just a tool. Yes, you can't protect autoload, but if you were that worried about a mutex, you could re-write it on the ruby side. That wasn't accepted because of the deadlock potentials. Those exist in require, too.
Autoload is triggered by the *interpreter*, and it's impossible to mutex the require that is caused by autoload, even if you would want to.
Nope.
The important thing is to make SURE that nothing is being autoloaded that will be hit the first time at runtime.
The important thing is to make sure that code running in potentially multiple threads does not raise race conditions.
This is one of those areas where you get common statements not holding to the true state of the world, such as "survival of the fittest", which is wrong, and should be in fact "unsurvival of the unfittest".
Yes - this does mean you could autoload twice, but you can require twice with late requires, too.
This remains ok until assignment is not atomic. When that happens, we've got a whole other world of crap, and again, require doesn't help unless you eliminate all of what I mentioned at the top.
No, you *need* to add unsafe loads to the preloader, and that's it. If you want to be 'extensively safe', don't use threads. I'm dead serious about that.
I suppose the preload solution works, but we still need to know which autoloads won't be triggered until runtime, and make sure to include them in the preloader.
So you're suggesting users run around with burning scissors and their eyes shut? (wrt Threading without understanding Threading impacts).
This situation is more complicated than it looks at first glance, and asking all users of Rack to understand how it works so that they can make the appropriate decision is unfair burden-shifting onto many other users of Rack.
Like I explained, this problem is in no way unique to autoload.
The critical thing, is to not write runtime code in classes that are to be late loaded or late evaluated, or, pre-run that code. That rule needs to be understood by ruby developers who write in a threaded environment, and has nothing to do (in reality) with the specific loading mechanism. As I've already said, hiding issues by taking out the obvious hits, doesn't make them magically disappear.
See above.
James Tucker
On 13 Jan 2009, at 18:28, Yehuda Katz wrote:
> The underlying problem with autoload is that the current behavior is
> to remove the "autoload needed" flag from the constant before the
> file is required, which means that another thread trying to trigger
> the same constant before the file has been loaded produces a
> NameError. This is definitely unexpected and is not something that
> users of Rack are likely to be able to figure out at first glance.
Agreed.
Charles Oliver Nutter
> > A quick grep for spaces in front of require in very commonly used ruby
> > projects can reveal interesting contradictions.
concurrent requires will block. I think there's still some question
about the correct way to handle it, though. All 1.8 versions will
either proceed with execution as if the require were complete or re-
require the file in parallel. Both are issues.
> > Mutexes are not a solution to thread safety. They are just a tool. Yes, you
> > can't protect autoload, but if you were that worried about a mutex, you
> > could re-write it on the ruby side. That wasn't accepted because of the
> > deadlock potentials. Those exist in require, too.
state that changes the behavior of other threads. It is not, by any
definition of the word, threadsafe...and it can't easily be made
threadsafe.
> > Yes - this does mean you could autoload twice, but you can require twice
> > with late requires, too.
undefined constant. The latter case results in either an error (const
missing) or the lookup returns an incorrect result from a higher
scope. Both cases are irreparably broken, which means autoloads must
never be used in a situation where there may be threads concurrently
hitting the autoloaded constant in question.
Yes, you can require users to explicitly load all libraries they will
need if they know they'll be running multi-threaded. Unfortunately
it's not always easy to know that you're running multithreaded, since
some libraries spin up threads or execute tasks in parallel. Even net/
* libraries use timeout.rb which spins up a thread. So in general any
application might use multiple threads since Ruby supports multiple
threads and library authors use them. And because of that, you also
expect users to know about the threading characteristics of all
libraries they're running. That sounds unreasonable, right? Expecting
them to pre-load libraries assumes they know the threading
characteristics of Rack.
There is one way you can get around this...define your own mutexed
service loader that lazily requires in the files. Essentially,
implement your own autoloader that handles threading concerns
correctly. Something like this:
def get_service
CONST ||= load_service 'something_big'
end
def load_service(service)
begin
mutex.lock
require service
ensure
mutex.unlock
end
end
Obviously the trick here is ensuring there aren't a whole bunch of
load_service mutexes, since that can lead to a deadlock. Also, the
method + const is slower than just a const lookup, but not by a lot
(and by a whole lot less in JRuby than in MRI). So it needs to be
fleshed out, but I don't think we're going to see C Ruby fix autoload
any time soon.
- Charlie
Pratik
--
Cheers!
- Pratik
http://m.onkey.org
James Tucker
On 13 Jan 2009, at 18:50, Charles Oliver Nutter wrote:
>
>>> Yes - this does mean you could autoload twice, but you can require
>>> twice
>>> with late requires, too.
>
> Incorrect. Autoload may fire twice, or the second thread may see an
> undefined constant. The latter case results in either an error (const
> missing) or the lookup returns an incorrect result from a higher
> scope. Both cases are irreparably broken, which means autoloads must
> never be used in a situation where there may be threads concurrently
> hitting the autoloaded constant in question.
class A
def foo
end
include MakeABlankSlate
end
Thread.new { require 'a' }
Thread.new { require '../a/a.rb' }
Is A#foo defined or not, for how long, when and why?
Mixing load and run times in a reflective environment can be
dangerous, with or without immediately broken paradigms.
Yehuda Katz
Charles Oliver Nutter
> > Incorrect. Autoload may fire twice, or the second thread may see an
> > undefined constant. The latter case results in either an error (const
> > missing) or the lookup returns an incorrect result from a higher
> > scope. Both cases are irreparably broken, which means autoloads must
> > never be used in a situation where there may be threads concurrently
> > hitting the autoloaded constant in question.
>
> class A
> def foo
> end
> include MakeABlankSlate
> end
>
> Thread.new { require 'a' }
> Thread.new { require '../a/a.rb' }
>
> Is A#foo defined or not, for how long, when and why?
>
> Mixing load and run times in a reflective environment can be
> dangerous, with or without immediately broken paradigms.
two different names in two different requires. An autoload is always
the same name in the same position.
Under 1.9 the above code would never see A#foo because the concurrent
requires would block. I'm not sure if this is specified behavior yet.
Under 1.8, the above could might see A#foo defined or might not, and
there's no way to predict that.
But again, totally different case from autoload.
- Charlie