[ruby-core:33322] [Ruby 1.9-Feature#4085][Open] Refinements and nested methods
Shugo Maeda
http://redmine.ruby-lang.org/issues/show/4085
Author: Shugo Maeda
Status: Open, Priority: Normal
Category: core, Target version: 1.9.3
As I said at RubyConf 2010, I'd like to propose a new features called
"Refinements."
Refinements are similar to Classboxes. However, Refinements doesn't
support local rebinding as mentioned later. In this sense,
Refinements might be more similar to selector namespaces, but I'm not
sure because I have never seen any implementation of selector
namespaces.
In Refinements, a Ruby module is used as a namespace (or classbox) for
class extensions. Such class extensions are called refinements. For
example, the following module refines Fixnum.
module MathN
refine Fixnum do
def /(other) quo(other) end
end
end
Module#refine(klass) takes one argument, which is a class to be
extended. Module#refine also takes a block, where additional or
overriding methods of klass can be defined. In this example, MathN
refines Fixnum so that 1 / 2 returns a rational number (1/2) instead
of an integer 0.
This refinement can be enabled by the method using.
class Foo
using MathN
def foo
p 1 / 2
end
end
f = Foo.new
f.foo #=> (1/2)
p 1 / 2
In this example, the refinement in MathN is enabled in the definition
of Foo. The effective scope of the refinement is the innermost class,
module, or method where using is called; however the refinement is not
enabled before the call of using. If there is no such class, module,
or method, then the effective scope is the file where using is called.
Note that refinements are pseudo-lexically scoped. For example,
foo.baz prints not "FooExt#bar" but "Foo#bar" in the following code:
class Foo
def bar
puts "Foo#bar"
end
def baz
bar
end
end
module FooExt
refine Foo do
def bar
puts "FooExt#bar"
end
end
end
module Quux
using FooExt
foo = Foo.new
foo.bar # => FooExt#bar
foo.baz # => Foo#bar
end
Refinements are also enabled in reopened definitions of classes using
refinements and definitions of their subclasses, so they are
*pseudo*-lexically scoped.
class Foo
using MathN
end
class Foo
# MathN is enabled in a reopened definition.
p 1 / 2 #=> (1/2)
end
class Bar < Foo
# MathN is enabled in a subclass definition.
p 1 / 2 #=> (1/2)
end
If a module or class is using refinements, they are enabled in
module_eval, class_eval, and instance_eval if the receiver is the
class or module, or an instance of the class.
module A
using MathN
end
class B
using MathN
end
MathN.module_eval do
p 1 / 2 #=> (1/2)
end
A.module_eval do
p 1 / 2 #=> (1/2)
end
B.class_eval do
p 1 / 2 #=> (1/2)
end
B.new.instance_eval do
p 1 / 2 #=> (1/2)
end
Besides refinements, I'd like to propose new behavior of nested methods.
Currently, the scope of a nested method is not closed in the outer method.
def foo
def bar
puts "bar"
end
bar
end
foo #=> bar
bar #=> bar
In Ruby, there are no functions, but only methods. So there are no
right places where nested methods are defined. However, if
refinements are introduced, a refinement enabled only in the outer
method would be the right place. For example, the above code is
almost equivalent to the following code:
def foo
klass = self.class
m = Module.new {
refine klass do
def bar
puts "bar"
end
end
}
using m
bar
end
foo #=> bar
bar #=> NoMethodError
The attached patch is based on SVN trunk r29837.
----------------------------------------
http://redmine.ruby-lang.org
Elise Huard
+1 definitely
----------------------------------------
http://redmine.ruby-lang.org/issues/show/4085
----------------------------------------
http://redmine.ruby-lang.org
Konstantin Haase
+1, can't wait to see this merged into trunk.
Etienne Vallette d'Osia
+1 for refinements.
What about the problems you shown during your talk ?
For nested function, I suppose it is better to transform them into lambdas,
but the concept is nice.
Javier Cicchelli
+1 This is a neat solution. I like it ;)
Yukihiro Matsumoto
In message "Re: [ruby-core:33322] [Ruby 1.9-Feature#4085][Open] Refinements and nested methods"
on Wed, 24 Nov 2010 22:12:24 +0900, Shugo Maeda <red...@ruby-lang.org> writes:
|Feature #4085: Refinements and nested methods
|http://redmine.ruby-lang.org/issues/show/4085
My +1.
However I'd like to see Ko1's consent before checking in. Last time I
talked with him about this issue, he wanted to consider this for a
week. That was last Friday. So we need to wait for a few more days.
matz.
Shugo Maeda
2010/11/24 Etienne Vallette d'Osia <red...@ruby-lang.org>:
> +1 for refinements.
Thank you.
> What about the problems you shown during your talk ?
No progress yet. So details may be changed after it is merged into trunk.
> For nested function, I suppose it is better to transform them into lambdas,
I think it's a (maybe too) big change because it means that real
functions are introduced into Ruby.
I like the fact that there are no functions (except lambda) in Ruby.
Even def at the top-level defines not a function but a method. It's
consistent as an object-oriented language.
Anyway, if my proposal about nested methods is not acceptable, I
withdraw it and propose only refinements.
--
Shugo Maeda
Shugo Maeda
2010/11/24 Yukihiro Matsumoto <ma...@ruby-lang.org>:
> |Feature #4085: Refinements and nested methods
> |http://redmine.ruby-lang.org/issues/show/4085
>
>
> My +1.
Thank you.
> However I'd like to see Ko1's consent before checking in. Last time I
> talked with him about this issue, he wanted to consider this for a
> week. That was last Friday. So we need to wait for a few more days.
I met ko1 last Thursday, and he agreed with my proposal at that time.
I hope he has not changed his mind.
I know that he doesn't like my implementation, where I have added a
new rb_control_frame_t member, which is equivalent to
ruby_frame->last_class in Ruby 1.8.
If he can implement refinements without the new member, it's OK for me:)
--
Shugo Maeda
Dave Thomas
module Quux
using FooExt
foo = Foo.new
foo.bar # => FooExt#bar
foo.baz # => Foo#bar
end
This behavior makes me nervous—I can see arguments for it, but at the same time I can see in leading to problems, particularly in well structured classes where a large set of behaviors is defined in terms of one method. I'm sure the syntax below is wrong, but look at the spirit of it.
module DoubleEach
refine Array do
def each
super do |val|
yield 2*val
end
end
end
end
using DoubleEach
[ 1, 2, 3 ].each {|v| p v } #=> 2, 4, 6
[ 1, 2, 3 ].min #=> 1
That would be surprising to me, as I'd expect the behavior of all the Enumerable methods, which depend on each, to change if I change the behavior of each.
Yusuke ENDOH
2010/11/24 Shugo Maeda <red...@ruby-lang.org>:
> As I said at RubyConf 2010, I'd like to propose a new features called
> "Refinements."
Basically I agree with your proposal, but I think that some
discussion is needed.
- Your patch is too big. Could you separate it to some parts?
It is hard (for me) to review it.
- Your patch adds `klass' to stack frame, which may cause
big performance degradation. We should check benchmark
result (though I'm not so concerned).
- API design. Why did you select:
module MathN
refine(Fixnum) do
def /(other) quo(other) end
end
end
using MathN
and not select others, such as:
module MathN
refine(Fixnum)
def /(other) quo(other) end
end
using MathN
or:
MathN = refine(Fixnum) do
def /(other) quo(other) end
end
using MathN
IMO, it will be more natural to provide this feature as new
constract with new syntax, instead of Module's methods.
- Is it intended to reject refining module methods?
module ComplexExt
refine(Math) do
def sqrt(x)
(x >= 0 ? 1 : Complex::I) * super(x.abs)
end
end
end
#=> wrong argument type Module (expected Class) (TypeError)
--
Yusuke Endoh <ma...@tsg.ne.jp>
Urabe Shyouhei
> - Your patch is too big. Could you separate it to some parts?
> It is hard (for me) to review it.
Shugo, push it to github.
> - Your patch adds `klass' to stack frame, which may cause
> big performance degradation. We should check benchmark
> result (though I'm not so concerned).
According to Shugo's presentation performance impact is up to 3% or so, which
is why Ko1 is not against it. It is almost no slowdown.
Shugo Maeda
Thanks for you comment.
2010/11/25 Dave Thomas <red...@ruby-lang.org>:
> module Quux
> using FooExt
>
> foo = Foo.new
> foo.bar # => FooExt#bar
> foo.baz # => Foo#bar
> end
>
> This behavior makes me nervous—I can see arguments for it, but at the same time I can see in leading to problems, particularly in well structured classes where a large set of behaviors is defined in terms of one method. I'm sure the syntax below is wrong, but look at the spirit of it.
>
> module DoubleEach
> refine Array do
> def each
> super do |val|
> yield 2*val
> end
> end
> end
> end
>
> using DoubleEach
>
> [ 1, 2, 3 ].each {|v| p v } #=> 2, 4, 6
>
> [ 1, 2, 3 ].min #=> 1
The syntax right and currently [1,2,3].min returns 2.
However, It's a bug (methods implemented by C behave wrong),
and I think it should return 1 as you showed above.
> That would be surprising to me, as I'd expect the behavior of all the Enumerable methods, which depend on each, to change if I change the behavior of each.
However, some other person might expect the original behavior of each.
For example,
class Foo
using DoubleEach
def bar
SomeOtherModule.do_something # DoubleEach may break this method
end
end
If SomeOtherModule.do_something is implemented by someone else, and
Array#each is used in do_something, DoubleEach breaks the code.
If you'd like to change the behavior of callees, you can use monky
patching or singleton methods instead of refinements.
--
Shugo Maeda
Shugo Maeda
2010/11/25 Yusuke ENDOH <ma...@tsg.ne.jp>:
>> As I said at RubyConf 2010, I'd like to propose a new features called
>> "Refinements."
>
>
> Basically I agree with your proposal, but I think that some
> discussion is needed.
>
> - Your patch is too big. Could you separate it to some parts?
> It is hard (for me) to review it.
Shyouhei suggested github, but my github account has no enough disk
space, so I put the git format-patch result at the following site:
http://shugo.net/tmp/refinement-r29837-20101124.zip (ZIP archive)
http://shugo.net/tmp/refinement-r29837-20101124/ (Expanded directory)
However, I'm afraid that there are too many patches, some of which
have been already reverted.
> - Your patch adds `klass' to stack frame, which may cause
> big performance degradation. We should check benchmark
> result (though I'm not so concerned).
I have run "make benchmark" 5 times, and it shows that the modified
version is slower than average 2.5% than the original version.
The benchmark result is available at:
http://shugo.net/tmp/refinement-benchmark-20101107.csv
Additional tests are welcome.
> - API design. Why did you select:
>
> module MathN
> refine(Fixnum) do
> def /(other) quo(other) end
> end
> end
> using MathN
>
> and not select others, such as:
>
> module MathN
> refine(Fixnum)
> def /(other) quo(other) end
> end
> using MathN
>
> or:
>
> MathN = refine(Fixnum) do
> def /(other) quo(other) end
> end
> using MathN
Because I'd like to allow multiple refinements in one module, for
example, in the following case:
module MyXmlFormat
refine Integer do
def to_xml; ...; end
end
refine String do
def to_xml; ...; end
end
refine Hash do
def to_xml; ...; end
end
...
end
In this case, I prefer the above syntax to your proposals.
> IMO, it will be more natural to provide this feature as new
> constract with new syntax, instead of Module's methods.
First, I consider it, but I wouldn't like to introduce new keywords.
> - Is it intended to reject refining module methods?
>
> module ComplexExt
> refine(Math) do
> def sqrt(x)
> (x >= 0 ? 1 : Complex::I) * super(x.abs)
> end
> end
> end
> #=> wrong argument type Module (expected Class) (TypeError)
It's limitation of the current implementation.
However, I don't think it's critical, because If you'd like to refine
a module, you can refine a class which includes the module instead.
--
Shugo Maeda
Urabe Shyouhei
>> - Your patch is too big. Could you separate it to some parts?
>> It is hard (for me) to review it.
>
> Shyouhei suggested github, but my github account has no enough disk
> space, so I put the git format-patch result at the following site:
I've just uploaded in place of you. Here it is:
Shugo Maeda
2010/11/25 Urabe Shyouhei <shyo...@ruby-lang.org>:
>> Shyouhei suggested github, but my github account has no enough disk
>> space, so I put the git format-patch result at the following site:
>
> I've just uploaded in place of you. Here it is:
>
> https://github.com/shyouhei/ruby/compare/shugo%2Frefinement
Thank you.
--
Shugo Maeda
Haase, Konstantin
>> IMO, it will be more natural to provide this feature as new
>> constract with new syntax, instead of Module's methods.
>
> First, I consider it, but I wouldn't like to introduce new keywords.
Plus that way it would be possible to stub the refine method and write code that does work on earlier ruby versions. Imagine RSpec switching to refinements in order to avoid global namespace pollution. It would be rather easy to implement a Module#refine that does in fact pollute the global namespace as a fall back. If new syntax is added, this wouldn't be possible.
Konstantin
Yusuke ENDOH
Thank you for your reply!
2010/11/25 Shugo Maeda <sh...@ruby-lang.org>:
> However, I'm afraid that there are too many patches, some of which
> have been already reverted.
Please fair them :-)
>> - Your patch adds `klass' to stack frame, which may cause
>> big performance degradation. We should check benchmark
>> result (though I'm not so concerned).
>
> I have run "make benchmark" 5 times, and it shows that the modified
> version is slower than average 2.5% than the original version.
Sounds good!
>> - API design.
> Because I'd like to allow multiple refinements in one module, for
> example, in the following case:
>
> module MyXmlFormat
> refine Integer do
> def to_xml; ...; end
> end
>
> refine String do
> def to_xml; ...; end
> end
>
> refine Hash do
> def to_xml; ...; end
> end
> ...
> end
After my short trial of this feature, I'd like an API to do
`refine' and `using' at once without explicit module, such as:
using_refine(Fixnum) do
def /(other) quo(other) end
end
(equivalent to)
using(Module.new do
refine(Fixnum) do
def /(other) quo(other) end
end
end)
Note that we cannot define `using_define' by ourselves because
of lexical scope limitation:
def using_refine(klass, &blk)
using(Module.new { refine(klass, &blk) })
end
using_refine(Fixnum) do
def /(other) quo(other) end
end
p 1 / 2 #=> 0
I guess that this `using_refine' is useful itself (though its
name is arguable). In addition, it allows us to write
MyXmlFormat as follows:
module MyXmlFormat
using_refine(Integer) do
def to_xml; ...; end
end
using_refine(String) do
def to_xml; ...; end
end
using_refine(Hash) do
def to_xml; ...; end
end
end
>> IMO, it will be more natural to provide this feature as new
>> constract with new syntax, instead of Module's methods.
>
> First, I consider it, but I wouldn't like to introduce new keywords.
I thought so, but I think that this feature deserves new
keywords because it is big evolution of Ruby's OO paradigm
(involving semantics change).
However, we should discuss this topic (new keyword) towards
2.0. Module's methods are not bad, as a part of reflection
features (such as Module#define_method for `def' keyword).
>> - Is it intended to reject refining module methods?
> It's limitation of the current implementation.
If so, it may be good to raise a NotImplementedError.
> However, I don't think it's critical, because If you'd like to refine
> a module, you can refine a class which includes the module instead.
Do you mean:
include Math
$toplevel = self
module ComplexExt
def sqrt(x)
(x >= 0 ? 1 : Complex::I) * super(x.abs)
end
refine(class << Math; self; end) { include ComplexExt }
refine(class << $toplevel; self; end) { include ComplexExt }
end
using ComplexExt
p sqrt(-4)
?
--
Yusuke Endoh <ma...@tsg.ne.jp>
Shugo Maeda
2010/11/25 Yusuke ENDOH <ma...@tsg.ne.jp>:
>> However, I'm afraid that there are too many patches, some of which
>> have been already reverted.
>
> Please fair them :-)
I have tried "git rebase -i", but these commits are hard to reorder
because they are related to each other. So I have to separate the
whole patch manually.
How about to separate them into the following three patches?
1. changes of control frames and method lookup
2. Refinements support
3. nested methods support
> After my short trial of this feature, I'd like an API to do
> `refine' and `using' at once without explicit module, such as:
>
> using_refine(Fixnum) do
> def /(other) quo(other) end
> end
>
> (equivalent to)
>
> using(Module.new do
> refine(Fixnum) do
> def /(other) quo(other) end
> end
> end)
I forgot to mention that refinements are enabled in class or module
definitions where Module#refine is called.
class Foo
refine Fixnum do
def /(other) quo(other) end
end
p 1 / 2 #=> (1/2)
end
Do we need Kernel#refine?
> I guess that this `using_refine' is useful itself (though its
> name is arguable). In addition, it allows us to write
> MyXmlFormat as follows:
>
> module MyXmlFormat
> using_refine(Integer) do
> def to_xml; ...; end
> end
>
> using_refine(String) do
> def to_xml; ...; end
> end
>
> using_refine(Hash) do
> def to_xml; ...; end
> end
> end
I guess Module#refine works the same as using_refine in this case.
>>> IMO, it will be more natural to provide this feature as new
>>> constract with new syntax, instead of Module's methods.
>>
>> First, I consider it, but I wouldn't like to introduce new keywords.
>
> I thought so, but I think that this feature deserves new
> keywords because it is big evolution of Ruby's OO paradigm
> (involving semantics change).
> However, we should discuss this topic (new keyword) towards
> 2.0. Module's methods are not bad, as a part of reflection
> features (such as Module#define_method for `def' keyword).
I don't think we need new keywords even if it is a big change,
because some essential features such as module inclusion
have no keyword in Ruby.
>>> - Is it intended to reject refining module methods?
>> It's limitation of the current implementation.
>
> If so, it may be good to raise a NotImplementedError.
Agreed.
>> However, I don't think it's critical, because If you'd like to refine
>> a module, you can refine a class which includes the module instead.
>
> Do you mean:
I guess it's unlikely that you need refine module functions in
real-world applications.
Why don't you simply override module functions?
include Math
def sqrt(x)
(x >= 0 ? 1 : Complex::I) * super(x.abs)
end
p sqrt(-4)
In the case of mix-in modules, you can refine concrete classes which
include mix-in modules.
class Foo
include BarMixin
end
module Baz
refine Foo do # instead of refine(BarMixin)
def method_in_bar_mixin
...
end
end
end
--
Shugo Maeda
Shugo Maeda
> As I said at RubyConf 2010, I'd like to propose a new features called
> "Refinements."
FYI, the slides of my talk are available at:
http://www.slideshare.net/ShugoMaeda/rc2010-refinements
or:
http://shugo.net/tmp/RubyConf2010.pdf
--
Shugo Maeda
SASADA Koichi
> version is slower than average 2.5% than the original version.
>
> The benchmark result is available at:
>
> http://shugo.net/tmp/refinement-benchmark-20101107.csv
My benchmark result is here:
http://www.atdot.net/fp_store/f.132gcl/file.graph.png
(Y-axis is a speed-up ratio (up is better))
on ruby 1.9.3dev (2010-11-25 trunk 29925) [x86_64-linux]
raw data:
http://www.atdot.net/sp/view/402gcl
I want to note that the average score using YARV's benchmark is not good
idea :)
--
// SASADA Koichi at atdot dot net
Yusuke ENDOH
2010/11/25 SASADA Koichi <k...@atdot.net>:
> (2010/11/25 13:55), Shugo Maeda wrote:
>> I have run "make benchmark" 5 times, and it shows that the modified
>> version is slower than average 2.5% than the original version.
>>
>> The benchmark result is available at:
>>
>> http://shugo.net/tmp/refinement-benchmark-20101107.csv
>
> My benchmark result is here:
> http://www.atdot.net/fp_store/f.132gcl/file.graph.png
> (Y-axis is a speed-up ratio (up is better))
I also conducted retest.
Up to 24.9% performance degradation occured on my environment.
(become slower)
24.9%: vm2_super
24.5%: loop_for
21.9%: vm1_ivar
20.9%: vm2_zsuper
20.7%: vm1_swap
15.7%: vm2_poly_method
15.0%: app_tak
12.7%: vm3_thread_mutex
11.6%: vm2_mutex
11.3%: so_nsieve
....
-1.6%: so_partial_sums
-2.6%: vm2_array
-3.0%: so_random
-4.0%: so_nbody
-4.3%: app_tarai
-5.0%: so_binary_trees
-5.1%: vm1_const
-7.1%: vm1_ensure
-7.5%: vm1_simplereturn
-7.8%: so_array
(become faster)
raw data:
http://www.atdot.net/sp/view/gu5gcl
csv:
program,original,refinement,time ratio (refinement / original)
app_answer,0.0971895694732666,0.10370779037475586,1.0670670827828104
app_erb,0.7143167495727539,0.724087905883789,1.0136790244900171
app_factorial,0.3495138168334961,0.35308108329772947,1.0102063675094504
app_fib,1.3584553718566894,1.383923053741455,1.0187475292986308
app_mandelbrot,0.31960186958312986,0.34449019432067873,1.0778729009627253
app_pentomino,31.527720880508422,33.666850090026855,1.0678491546415878
app_raise,0.8428834438323974,0.8716522693634033,1.0341314398111803
app_strconcat,0.5739905834197998,0.5915801525115967,1.0306443513184473
app_tak,1.676273727416992,1.9272963523864746,1.149750378392131
app_tarai,1.4925862312316895,1.427922534942627,0.9566767434028238
app_uri,1.4658241271972656,1.4655370235443115,0.9998041349929865
io_file_create,0.42700395584106443,0.44702463150024413,1.0468863938736708
io_file_read,0.39592618942260743,0.40063791275024413,1.0119005093714764
io_file_write,0.2036591053009033,0.20306005477905273,0.9970585625377982
loop_for,2.656430149078369,3.306622934341431,1.2447618603819273
loop_generator,0.8340113639831543,0.8517292499542236,1.0212441781206079
loop_times,2.4262061595916746,2.4653957366943358,1.0161526162761274
loop_whileloop,1.10532808303833,1.1343581676483154,1.026263771865985
loop_whileloop2,0.22827577590942383,0.23306760787963868,1.020991416855883
so_ackermann,1.4401548862457276,1.5601153373718262,1.083296909430914
so_array,2.876900815963745,2.6525750160217285,0.9220251881131089
so_binary_trees,0.7443643569946289,0.7073556423187256,0.950281452452498
so_concatenate,0.6735883235931397,0.6702359199523926,0.9950230674681766
so_count_words,0.33946728706359863,0.3388693332672119,0.99823855252281
so_exception,1.7248088836669921,1.778290367126465,1.0310071938786456
so_fannkuch,28.669377708435057,30.272033739089967,1.0559013190643263
so_fasta,3.9797119140625,3.990163469314575,1.0026262090014968
so_k_nucleotide,2.4159430503845214,2.584695339202881,1.0698494481446907
so_lists,0.5479694843292237,0.5965001106262207,1.0885644687977543
so_mandelbrot,9.189719247817994,9.212599325180054,1.00248974715604
so_matrix,0.7333712100982666,0.7263244152069092,0.9903912305332886
so_meteor_contest,9.425535678863525,9.882782745361329,1.0485115204139714
so_nbody,6.576598167419434,6.311960506439209,0.9597607069424975
so_nested_loop,2.17801456451416,2.1627822875976563,0.9930063475402418
so_nsieve,4.845090913772583,5.39297251701355,1.1130797363747225
so_nsieve_bits,5.009188795089722,5.168913412094116,1.0318863240213594
so_object,1.4976745128631592,1.5413443088531493,1.029158402319677
so_partial_sums,8.703983974456786,8.56481056213379,0.9840103781519562
so_pidigits,2.3049992084503175,2.294805479049683,0.9955775562250678
so_random,0.5043463706970215,0.48915743827819824,0.9698839264019453
so_reverse_complement,2.5664955139160157,2.577986478805542,1.0044772978667682
so_sieve,0.1414027690887451,0.15396690368652344,1.0888535258449785
so_spectralnorm,5.634414100646973,5.971523380279541,1.0598304053643943
vm1_block,4.044720840454102,4.3206627368927,1.06822272965756
vm1_const,2.2964776039123533,2.180025339126587,0.9492909207617028
vm1_ensure,1.3428500652313233,1.247154951095581,0.9287373053675523
vm1_ivar,2.4599918842315676,2.9994019985198976,1.2192731275846573
vm1_ivar_set,2.4699801445007323,2.4521598339080812,0.9927852413581838
vm1_length,2.1807806491851807,2.199760341644287,1.0087031643765723
vm1_neq,1.9046989917755126,1.9770766735076903,1.0379995380082125
vm1_not,1.5559280872344972,1.6081242561340332,1.0335466461000196
vm1_rescue,1.2895635128021241,1.297832202911377,1.0064120068745475
vm1_simplereturn,3.474906158447266,3.21527214050293,0.9252831569816115
vm1_swap,1.6546783447265625,1.9964598178863526,1.2065546299370165
vm2_array,1.432089138031006,1.3952114582061768,0.9742490332162336
vm2_case,0.46059327125549315,0.4548778057098389,0.9875910789359233
vm2_eval,28.152580356597902,29.735985946655273,1.0562437108783984
vm2_method,3.3265981674194336,3.469304418563843,1.0428985540069338
vm2_mutex,2.5152658462524413,2.8074283599853516,1.1161557193519764
vm2_poly_method,4.341252231597901,5.024772596359253,1.157447742793274
vm2_poly_method_ov,0.6244135856628418,0.6301047325134277,1.009114386652149
vm2_proc,1.251944398880005,1.3809085845947267,1.1030111128178643
vm2_regexp,2.2819092750549315,2.2643832683563234,0.9923195865452688
vm2_send,0.7383904933929444,0.8028009414672852,1.0872308739761414
vm2_super,1.1586995601654053,1.4471320629119873,1.2489277744313703
vm2_unif1,0.6489530563354492,0.6893436431884765,1.0622396126479587
vm2_zsuper,1.2690173149108888,1.5337296962738036,1.2085963510919495
vm3_gc,1.2378501892089844,1.2501022815704346,1.0098978797824312
vm3_thread_create_join,2.7109290599822997,2.7120433807373048,1.0004110475524624
vm3_thread_mutex,237.79514360427856,268.030443572998,1.1271485174610414
--
Yusuke Endoh <ma...@tsg.ne.jp>
Yusuke ENDOH
2010/11/25 Shugo Maeda <sh...@ruby-lang.org>:
> How about to separate them into the following three patches?
>
> 1. changes of control frames and method lookup
> 2. Refinements support
> 3. nested methods support
Seems good.
> I forgot to mention that refinements are enabled in class or module
> definitions where Module#refine is called.
>
> class Foo
> refine Fixnum do
> def /(other) quo(other) end
> end
>
> p 1 / 2 #=> (1/2)
> end
Aha!
> Do we need Kernel#refine?
I want it.
>> I guess that this `using_refine' is useful itself (though its
>> name is arguable). In addition, it allows us to write
>> MyXmlFormat as follows:
>>
>> module MyXmlFormat
>> using_refine(Integer) do
>> def to_xml; ...; end
>> end
>>
>> using_refine(String) do
>> def to_xml; ...; end
>> end
>>
>> using_refine(Hash) do
>> def to_xml; ...; end
>> end
>> end
>
> I guess Module#refine works the same as using_refine in this case.
Hmm. Then, when no block is given to Module#refine, how about
adding an implicit block that includes the outer module?
module FooExt
refine(Foo)
...
end
(equivalent to)
module FooExt
refine(Foo) { include FooExt }
...
end
> I don't think we need new keywords even if it is a big change,
> because some essential features such as module inclusion
> have no keyword in Ruby.
Indeed. But conventionally, essential features that involve
code block (such as class/module definition, method definition
and control statements) have their special keywords.
> I guess it's unlikely that you need refine module functions in
> real-world applications.
I thought that one of motivating examples of this feature is
mathn.rb. In fact, mathn.rb changes Math module to complex-
aware one.
--
Yusuke Endoh <ma...@tsg.ne.jp>
Yusuke ENDOH
Are these intended?
(1)
class Foo
end
module FooExtCore
def foo
bar
end
def bar
end
end
module FooExt
refine(Foo) { include FooExtCore }
end
using(FooExt)
Foo.new.foo
#=> undefined local variable or method `bar'
I can guess what's going on, but this will be a FAQ...
(2)
class Foo
end
module FooExt
end
using FooExt
module FooExt
refine(Foo) { include FooExt }
def foo
end
end
Foo.new.foo
#=> undefined method `foo'
Note that I don't want to attack this proposal.
I'm really for this and I hope to help improve it.
Please don't misunderstand me :-)
--
Yusuke Endoh <ma...@tsg.ne.jp>
Magnus Holm
1. Could you give an example of how it would behave if it had local
rebinding as in classbox?
2. I don't like the idea of having both #using and #include. I don't
want to think about which one to use; I just want to use the module! I
think this is getting even more messy than with the new mixin-feature.
Even right *now* people only use #include (and def included(mod);
mod.extend ClassMethods; end) because That's How Modules Work™.
3. Is this expected behaviour?
module Ext
refine(Object) do
def to_json; "something"; end
end
end
Fixnum.send(:using, Ext)
"Hello".to_json # => Works because #using worked at the file scope
This makes it impossible to add refinements within a included-callback.
---
Other than that: This is definitely a contender to sliced bread :-)
// Magnus Holm
Haase, Konstantin
>> Do we need Kernel#refine?
>
> I want it.
What would be the difference between Kernel#refine and Module#prepend?
I mean, what point do local refinements have in a global scope?
Konstantin
Haase, Konstantin
On Nov 25, 2010, at 22:06 , Magnus Holm wrote:
> 2. I don't like the idea of having both #using and #include. I don't
> want to think about which one to use; I just want to use the module! I
> think this is getting even more messy than with the new mixin-feature.
> Even right *now* people only use #include (and def included(mod);
> mod.extend ClassMethods; end) because That's How Modules Work™.
But what if you want to use refinements only in the module included, not the module it is included into?
This would be impossible. One could always do a `def self.included(klass) klass.send(:using, self) end` (in case your point #3 will be fixed).
Konstantin
Shugo Maeda
2010/11/25 SASADA Koichi <k...@atdot.net>:
>> I have run "make benchmark" 5 times, and it shows that the modified
>> version is slower than average 2.5% than the original version.
>>
>> The benchmark result is available at:
>>
>> http://shugo.net/tmp/refinement-benchmark-20101107.csv
>
> My benchmark result is here:
> http://www.atdot.net/fp_store/f.132gcl/file.graph.png
> (Y-axis is a speed-up ratio (up is better))
>
> on ruby 1.9.3dev (2010-11-25 trunk 29925) [x86_64-linux]
>
> raw data:
> http://www.atdot.net/sp/view/402gcl
>
>
> I want to note that the average score using YARV's benchmark is not good
> idea :)
I agree.
Please remember that I'm a daimyo programmer.
I'd like you to improve the performance:)
--
Shugo Maeda
Shugo Maeda
2010/11/26 Yusuke ENDOH <ma...@tsg.ne.jp>:
>> How about to separate them into the following three patches?
>>
>> 1. changes of control frames and method lookup
>> 2. Refinements support
>> 3. nested methods support
>
> Seems good.
I'll make the patches.
>> Do we need Kernel#refine?
>
> I want it.
I agree with you. What do you think of it, Matz?
>> I guess Module#refine works the same as using_refine in this case.
>
> Hmm. Then, when no block is given to Module#refine, how about
> adding an implicit block that includes the outer module?
>
> module FooExt
> refine(Foo)
> ...
> end
>
> (equivalent to)
>
> module FooExt
> refine(Foo) { include FooExt }
> ...
> end
Could you tell me why you need this feature?
>> I don't think we need new keywords even if it is a big change,
>> because some essential features such as module inclusion
>> have no keyword in Ruby.
>
> Indeed. But conventionally, essential features that involve
> code block (such as class/module definition, method definition
> and control statements) have their special keywords.
I guess that most of these constructs have reasons why they need
keywords and special syntax. For example, class, module, and method
definitions take undefined identifiers (alias doesn't invoke code
block, but it's a keyword for the same reason), and some control
statements take expressions evaluated lazily. However, there is no
such reason for Refinements.
If refine is a keyword, there is one good thing. We don't need "do"
after class names.
refine Fixnum
...
end
I'm not sure it's an enough reason to introduce a new keyword.
What do you think of it, Matz?
>> I guess it's unlikely that you need refine module functions in
>> real-world applications.
>
> I thought that one of motivating examples of this feature is
> mathn.rb. In fact, mathn.rb changes Math module to complex-
> aware one.
If think it's better to provide complex-aware features as module functions
of a new module such as MathN or MathN::Math. Users can include it
instead of Math.
--
Shugo Maeda
Shugo Maeda
2010/11/26 Yusuke ENDOH <ma...@tsg.ne.jp>:
> Are these intended?
>
> (1)
>
> class Foo
> end
> module FooExtCore
> def foo
> bar
> end
> def bar
> end
> end
> module FooExt
> refine(Foo) { include FooExtCore }
> end
> using(FooExt)
> Foo.new.foo
> #=> undefined local variable or method `bar'
>
> I can guess what's going on, but this will be a FAQ...
This behavior is intended, and you need define foo and bar inside the
block passed to refine.
I don't know any way to make the above code work without local rebinding.
> (2)
>
> class Foo
> end
> module FooExt
> end
> using FooExt
> module FooExt
> refine(Foo) { include FooExt }
> def foo
> end
> end
> Foo.new.foo
> #=> undefined method `foo'
I'd like to allow the above code if possible, but it's hard for me.
> Note that I don't want to attack this proposal.
> I'm really for this and I hope to help improve it.
> Please don't misunderstand me :-)
I know it. Thanks for your helpful comments.
--
Shugo Maeda
Jos Backus
Shugo Maeda
2010/11/26 Magnus Holm <jud...@gmail.com>:
> Woah, this is very nice stuff! Some comments/questions:
>
> 1. Could you give an example of how it would behave if it had local
> rebinding as in classbox?
If it had local rebinding, the following code would print "Quux::Foo#bar".
class Foo
def bar
puts "Foo#bar"
end
def baz
bar
end
end
module Quux
refine Foo do
def bar
puts "Quux::Foo#bar"
end
end
end
using Quux
foo = Foo.new
foo.baz
> 2. I don't like the idea of having both #using and #include. I don't
> want to think about which one to use; I just want to use the module! I
> think this is getting even more messy than with the new mixin-feature.
> Even right *now* people only use #include (and def included(mod);
> mod.extend ClassMethods; end) because That's How Modules Work™.
Matz doesn't like it, but I think it's worth considering.
However, it's a problem that we have main#include, where main is self
at the top-level, but not Kernel#include now.
> 3. Is this expected behaviour?
>
> module Ext
> refine(Object) do
> def to_json; "something"; end
> end
> end
>
> Fixnum.send(:using, Ext)
> "Hello".to_json # => Works because #using worked at the file scope
It is intended.
> This makes it impossible to add refinements within a included-callback.
If a binding is passed to include, it may be possible as follows:
module Foo
refine XXX do ... end
def do_something
end
...
def self.included(klass, binding)
eval("using Foo", binding)
end
end
class Bar
include Foo
# both do_something and refinements are available
end
It's necessary to check the arity of included for backward compatibility.
Currenty, you can use a used-callback instead.
module Foo
refine XXX do ... end
def do_something
end
...
def self.used(klass)
klass.send(:include, self)
end
end
class Bar
using Foo
end
--
Shugo Maeda
Shugo Maeda
2010/11/26 Haase, Konstantin <Konstant...@student.hpi.uni-potsdam.de>:
>>> Do we need Kernel#refine?
>>
>> I want it.
>
> What would be the difference between Kernel#refine and Module#prepend?
> I mean, what point do local refinements have in a global scope?
Kernel#refine doesn't refine classes in a global scope, but in a file
or method scope.
If you call Kernel#refine at the top-level, the refinements are
enabled only in that file.
If you call Kernel#refine in a method, the refinements are enabled
only in the method.
--
Shugo Maeda
Shugo Maeda
2010/11/26 Jos Backus <j...@catnook.com>:
> Regarding naming: how about using the name `use' instead of `using'? This
> would be in line with other names such as `include', `extend', etc.
Some other people said the same, but I think `using' is better than
`use' because `use' is already used in Rack.
> Thanks for working on this, it looks like a cool and useful feature.
Thank you.
--
Shugo Maeda
Jos Backus
2010/11/26 Jos Backus <j...@catnook.com>:
> Regarding naming: how about using the name `use' instead of `using'? This
> would be in line with other names such as `include', `extend', etc.
Some other people said the same, but I think `using' is better than
`use' because `use' is already used in Rack.
Jos Backus
jos at catnook.com
James Edward Gray II
> The only other possibly appropriate verb I can think of at the moment is `introduce'.
That feels too close to include.
James Edward Gray II
Yusuke ENDOH
2010/11/26 Shugo Maeda <sh...@ruby-lang.org>:
>>> I guess Module#refine works the same as using_refine in this case.
>>
>> Hmm. Then, when no block is given to Module#refine, how about
>> adding an implicit block that includes the outer module?
>>
>> module FooExt
>> refine(Foo)
>> ...
>> end
>>
>> (equivalent to)
>>
>> module FooExt
>> refine(Foo) { include FooExt }
>> ...
>> end
>
> Could you tell me why you need this feature?
Because it requires less indentation, I thought.
But I found out that it has a problem of [ruby-core:33386] (1)...
>>> I don't think we need new keywords even if it is a big change,
>>> because some essential features such as module inclusion
>>> have no keyword in Ruby.
>>
>> Indeed. But conventionally, essential features that involve
>> code block (such as class/module definition, method definition
>> and control statements) have their special keywords.
>
> I guess that most of these constructs have reasons why they need
> keywords and special syntax.
I don't think so. "class Foo; end" can be written as "Foo =
Class.new { }" (though there are indeed subtle differences between
them).
> If refine is a keyword, there is one good thing. We don't need "do"
> after class names.
>
> refine Fixnum
> ...
> end
The API design that "def" statements are put in a Ruby's block,
is slightly weird (for me). I guess that there is no precedent of
such a style in Ruby's embedded featues, except meta programming
(such as Class.new and class_eval).
From now on, does Ruby encourage such a style in casual use?
--
Yusuke Endoh <ma...@tsg.ne.jp>
Shugo Maeda
2010/11/26 Yusuke ENDOH <ma...@tsg.ne.jp>:
>>> module FooExt
>>> refine(Foo)
>>> ...
>>> end
>>>
>>> (equivalent to)
>>>
>>> module FooExt
>>> refine(Foo) { include FooExt }
>>> ...
>>> end
>>
>> Could you tell me why you need this feature?
>
> Because it requires less indentation, I thought.
I see. refine without blocks looks confusing for me because it works
different from refine with a block.
> But I found out that it has a problem of [ruby-core:33386] (1)...
I think the above code should work because the refinement of Foo is
enabled in FooExt.
It may be a bug.
>>>> I don't think we need new keywords even if it is a big change,
>>>> because some essential features such as module inclusion
>>>> have no keyword in Ruby.
>>>
>>> Indeed. But conventionally, essential features that involve
>>> code block (such as class/module definition, method definition
>>> and control statements) have their special keywords.
>>
>> I guess that most of these constructs have reasons why they need
>> keywords and special syntax.
>
> I don't think so. "class Foo; end" can be written as "Foo =
> Class.new { }" (though there are indeed subtle differences between
> them).
"refine Foo do end" is different from "Foo = Class.new {}" because
"refine Foo do end" looks good, but "Foo = Class.new {}" doesn't.
I think how it looks is more important than whether it uses keywords or not.
>> If refine is a keyword, there is one good thing. We don't need "do"
>> after class names.
>>
>> refine Fixnum
>> ...
>> end
>
> The API design that "def" statements are put in a Ruby's block,
> is slightly weird (for me). I guess that there is no precedent of
> such a style in Ruby's embedded featues, except meta programming
> (such as Class.new and class_eval).
> From now on, does Ruby encourage such a style in casual use?
I think Module#refine is a meta programming feature like class_eval,
and most application programmers need not use it directly.
And, "refine Foo do end" looks not so bad, so I think the new keyword
refine has more cons than pros.
--
Shugo Maeda
Shugo Maeda
File control_frame_change-r29944-20101127.diff added
File refinements-r29944-20101127.diff added
File nested_methods-r29944-20101127.diff added
Hi,
> How about to separate them into the following three patches?
>
> 1. changes of control frames and method lookup
> 2. Refinements support
> 3. nested methods support
I have attached these three patches.
Please check them.
The following code works now:
class Foo; end
module FooExt
refine(Foo) { include FooExt }
def foo
puts "foo"
end
def bar
puts "bar"
foo
end
end
using FooExt
f = Foo.new
f.foo
f.bar
----------------------------------------
http://redmine.ruby-lang.org/issues/show/4085
----------------------------------------
http://redmine.ruby-lang.org
Yukihiro Matsumoto
In message "Re: [ruby-core:33393] Re: [Ruby 1.9-Feature#4085][Open] Refinements and nested methods"
on Fri, 26 Nov 2010 11:02:28 +0900, Shugo Maeda <sh...@ruby-lang.org> writes:
|>> Do we need Kernel#refine?
|>
|> I want it.
|
|I agree with you. What do you think of it, Matz?
I don't see strong requirement, but I accept.
matz.
Yukihiro Matsumoto
In message "Re: [ruby-core:33396] Re: [Ruby 1.9-Feature#4085][Open] Refinements and nested methods"
on Fri, 26 Nov 2010 11:24:43 +0900, Shugo Maeda <sh...@ruby-lang.org> writes:
|Matz doesn't like it, but I think it's worth considering.
|However, it's a problem that we have main#include, where main is self
|at the top-level, but not Kernel#include now.
If we introduce local rebinding, I think we have to rename it to
"classbox" from "refinement". Besides that, we sill have lot of
issues about implementation, performance, etc. But I believe, Shugo,
you are the key person. I am the one who approve.
matz.
Chauk-Mean Proum
>> Regarding naming: how about using the name `use' instead of `using'? This
>> would be in line with other names such as `include', `extend', etc.
>Some other people said the same, but I think `using' is better than
>`use' because `use' is already used in Rack.
IMHO, it would be a shame to have such a new core feature with a "strange" naming.
I also prefer use instead of using.
May be the core team can request Rack developers to rename their use method to e.g. rack_use.
I guess that this new feature will be available only for ruby-1.9.3+.
So this leaves time for Rack developers and users to migrate their code base.
There are already some "strange" namings for some methods (http://redmine.ruby-lang.org/issues/show/4065),
so please take care of consistent naming for new methods.
Having said that, the feature looks very interesting.
Haase, Konstantin
On Nov 27, 2010, at 09:35 , Chauk-Mean Proum wrote:
> Issue #4085 has been updated by Chauk-Mean Proum.
>
>
>>> Regarding naming: how about using the name `use' instead of `using'? This
>>> would be in line with other names such as `include', `extend', etc.
>
>> Some other people said the same, but I think `using' is better than
>> `use' because `use' is already used in Rack.
>
> IMHO, it would be a shame to have such a new core feature with a "strange" naming.
> I also prefer use instead of using.
> May be the core team can request Rack developers to rename their use method to e.g. rack_use.
> I guess that this new feature will be available only for ruby-1.9.3+.
> So this leaves time for Rack developers and users to migrate their code base.
This would be about any Ruby web application/framework/library out there. Not only Rack implements use, but Rails and Sinatra (and probably others) do so, too, in order to behave like Rack.
Konstantin
Haase, Konstantin
So, is local rebinding still on the table? From my point of view local rebinding is far more intuitive and there are some use cases I am not sure I could solve without local rebinding.
Konstantin
Loren Segal
On 11/27/2010 3:35 AM, Chauk-Mean Proum wrote:
> IMHO, it would be a shame to have such a new core feature with a "strange" naming.
> I also prefer use instead of using.
> May be the core team can request Rack developers to rename their use method to e.g. rack_use.
> I guess that this new feature will be available only for ruby-1.9.3+.
> So this leaves time for Rack developers and users to migrate their code base.
name because it's "strange". "Migrating" a codebase doesn't work when
you need to support older Ruby versions, it only makes things messier
for no good reason. FWIW, since you brought it up, your linked issue has
the same problem of completely breaking backwards compat. for nothing
but vanity, and I wouldn't be fond of that either.
Personally I see no problem with `using`. I'm not sure what the fuss is
about.
- Loren
Chauk-Mean Proum
11/27/2010 09:46 AM - Loren Segal wrote :
>> I also prefer use instead of using.
>> May be the core team can request Rack developers to rename their use method to e.g. rack_use.
>> I guess that this new feature will be available only for ruby-1.9.3+.
>> So this leaves time for Rack developers and users to migrate their code base.
>This is really bad release engineering. You don't just change a method name because it's "strange".
Yes, I know that changes that break backward compatibility should be avoided.
>"Migrating" a codebase doesn't work when you need to support older Ruby versions,
In this case, the impact is on Rack and Web frameworks that rely on Rack (and I agree that there is lot as pointed by Konstantin). But there is no impact on older Ruby versions as refine/using is a new feature.
It is also just a proposal.
> FWIW, since you brought it up, your linked issue has the same problem of completely
> breaking backwards compat. for nothing but vanity, and I wouldn't be fond of that either.
Vanity ?
It seems that I was not alone to prefer use to using.
Regarding the proposal for renaming append_features, it was only to make the relationship with include more clear.
If other people think that this is not worth breaking the compatibility, then that's fine.
Please leave other people express their opinion, a different opinion does not mean vanity.
Chauk-Mean.
Yukihiro Matsumoto
In message "Re: [ruby-core:33421] [Ruby 1.9-Feature#4085] Refinements and nested methods"
on Sat, 27 Nov 2010 17:35:43 +0900, Chauk-Mean Proum <red...@ruby-lang.org> writes:
|>Some other people said the same, but I think `using' is better than
|>`use' because `use' is already used in Rack.
|
|IMHO, it would be a shame to have such a new core feature with a "strange" naming.
|I also prefer use instead of using.
For specifying namespace, "using" is used in many languages, C++, C#
etc. On the other hand, as far as I know, no language use the keyword
"use" for namespaces, but for other purposes. For example, Perl6's
"use" is to mix-in traits, just like Ruby's "include".
So, from language designers' view, "using" is not that strange.
matz.
Chauk-Mean Proum
Hi Matz,
>For specifying namespace, "using" is used in many languages, C++, C#
>etc. On the other hand, as far as I know, no language use the keyword
>"use" for namespaces, but for other purposes. For example, Perl6's
>"use" is to mix-in traits, just like Ruby's "include".
>
>So, from language designers' view, "using" is not that strange.
'using' is not strange in itself.
It's just that it is not "in line with other names such as `include', `extend', etc." as other people said.
You're Ruby's grand master, the decision is definitively yours.
Magnus Holm
Thanks a lot. Could you explain why included modules are rebound though?
class CharArray
include Enumerable
def initialize(str)
@array = str.unpack("C*") # Unpacks to integers
end
def each(&blk)
@array.each(&blk)
end
def map
res = []
each { |val| res << yield(val) }
res
end
end
test = CharArray.new("Hello World")
test.each { |x| p x } # Prints a list of integers (expected)
module CharArrayStr
refine CharArray do
def each
super { |c| yield c.chr }
end
end
end
using CharArrayStr
test.each { |x| p x } # Prints a list of strings (expected)
test.map { |x| p x } # Prints a list of integers (exptected;
no local rebinding)
test.select { |x| p x } # Prints a list of strings?!
Yusuke ENDOH
Sorry for late reply.
2010/11/26 Shugo Maeda <sh...@ruby-lang.org>:
>>
>> Because it requires less indentation, I thought.
>
> I see. refine without blocks looks confusing for me because it works
> different from refine with a block.
Maybe another name is needed?
I think that the short name `refine' is more appropriate to the non-
block style than the block style because I believe the non-block
style is more suitable for casual use. It requires less indentation,
and it complies with traditional style (like Module#include).
A longer (more "meta-programming-like") name would be appropriate to
the block style, such as Module#refine_class, #refine_class_eval,
#class_eval_with_refinement.
>>> I guess that most of these constructs have reasons why they need
>>> keywords and special syntax.
>>
>> I don't think so. "class Foo; end" can be written as "Foo =
>> Class.new { }" (though there are indeed subtle differences between
>> them).
>
> "refine Foo do end" is different from "Foo = Class.new {}" because
> "refine Foo do end" looks good, but "Foo = Class.new {}" doesn't.
> I think how it looks is more important than whether it uses keywords or not.
"refine Foo do def ... end end" looks not so good to me.
>>> If refine is a keyword, there is one good thing. We don't need "do"
>>> after class names.
>>>
>>> refine Fixnum
>>> ...
>>> end
>>
>> The API design that "def" statements are put in a Ruby's block,
>> is slightly weird (for me). I guess that there is no precedent of
>> such a style in Ruby's embedded featues, except meta programming
>> (such as Class.new and class_eval).
>> From now on, does Ruby encourage such a style in casual use?
>
> I think Module#refine is a meta programming feature like class_eval,
> and most application programmers need not use it directly.
I guess you think so because we are not used to the feature yet.
If it is really just a meta-programming feature, the name should be
more "meta-programming-like".
The non-block style has a precedent (Module#include), so, if it is
adopted, I agree that any new keyword is not needed. Otherwise, I
prefer a new keyword to a new weird (to me) coding style.
--
Yusuke Endoh <ma...@tsg.ne.jp>
Charles Oliver Nutter
I'm being clear about my concerns, so they can be addressed in a
meaningful way.
SUMMARY:
* "using" not being a keyword requires all calls to check for
refinements all the time, globally degrading performance.
* instance_eval propagating refinements requires all block invocations
to localize what refinements they use for invocation on every
activation.
* shared, mutable structures can't be used to store the active
refinement, due to concurrency issues
* there are very likely many more complexities than illustrated here
that result from the combination of runtime-mutable lexical scoping
structures, concurrency, and method caching.
On Wed, Nov 24, 2010 at 7:12 AM, Shugo Maeda <red...@ruby-lang.org> wrote:
> If a module or class is using refinements, they are enabled in
> module_eval, class_eval, and instance_eval if the receiver is the
> class or module, or an instance of the class.
I am surprised nobody else has questioned this behavior. I believe it
is a problem.
Currently, blocks handle method dispatch like any other scope...i.e.
they look only at the "self" object's class (for fcall/vcall) or the
target object's class (for normal call). A typical caching structure
to optimize this then has an entry that stores previously-seen
method(s) and invalidates based on some trivial guard. In 1.9, this is
a global serial number. In JRuby, it's a class hierarchy-based guard.
The global serial number approach in 1.9 means that any change that
flip that serial number cause all caches everywhere to invalidate.
Normally this only happens on method definition or module inclusion,
which is why defining methods or including modules at runtime is
strongly discouraged for performance reasons.
Refinements make method lookup more complicated, since now any scope
where a refinement can no longer use the simple "target class" lookup
and cache-validation logic. Because refinements are enabled at
runtime, after parse, this also means that all calls everywhere must
constantly check if a refinement is enabled. This is performance hit
#1.
If "using" were a keyword, we could know at parse time that some calls
must check for refinements and other calls do not need to, localizing
the performance hit to only scopes where refinements are active. I
would strongly encourage "using" be made into a keyword.
Without "using" being a keyword, we can still avoid a global
performance hit by pretending it's a keyword and proactively changing
how scopes are parsed in the presence of "using" in a containing
scope. This is likely what we would do in JRuby, forcing all
class-body calls named "using" to "damage" performance in child
scopes. We would also disallow or strongly discourage aliasing of
"using", since it would be impossible at parse time to make a proper
decision. We already do this for methods like "eval", which force a
method body to be completely deoptimized.
The instance_eval case basically makes it impossible to avoid the
performance hit for method calls within a block, since at any time a
previously-captured block could be instance_eval'ed against a receiver
class with refinements enabled. So all blocks everywhere would have to
constantly check for refinements, forever. One possible suggestion to
get around this would be to make all method calls in blocks check a
global serial number, as in CRuby. At best, this is still an
additional check in implementations that don't use a global serial
number to invalidate method caches. At worst, it's still infeasible.
Recall that previously, refinements were largely lexical and morely
static. In other words, even though refinements would not be applied
at parse time, they would be applied at method-definition time and
unchangeable from then on. The instance_eval case throws this out
completely. The same block could be instance_eval'ed against two
different refinements at the same time. Since the current logic stores
the active refinement in the cache, and the cache is shared across all
invocations (including concurrent invocations), we now have a case
where mid-call, the static in-memory code/caches for a block would
have to switch to a different refinement. Obviously this is
intractable, since we wanted refinements in the first place for their
isolation characteristics. In order to avoid this, all blocks
everywhere would need to *never* cache method lookups, and always do a
full slow-path lookup on their thread-local structures.
Even if an implementation isn't actually concurrent, things are still
intractable, since any invocation of instance_eval against a
refinement would have to force a global serial number change (at
minimum) to force caches to be invalidated. This means that any use
anywhere of instance_eval against a refinement would cause all
block-borne method calls to flush and recache their next invocation.
And if that's not bad enough, even on a non-concurrent implementation
the context-switch boundaries are finer-grained than individual
calls...so any shared mutable data structures indicating what
refinement to use would *still* require slow-path lookup every time in
order to isolate one refinement-targeted instance_eval's effects from
others.
And even if we don't consider concurrency, there's the issue of the
*same* block being used in the *same* call stack for *different*
refinements. Any call you make downstream from a given block could
cause that block's static in-memory structures to be modified.
It might be possible to reduce the slow-path logic to checking the
call frame for *every single call* to see if a refinement is active,
and then if the caller knows that a refinement is active call frames
would have to have this bit set. But the caller is not responsible for
the call frame construction, so all calls everywhere would have to
check the caller's frames to see if a refinement is active. Accessing
the caller's frame means every call needs to do additional pointer
dereferences and checks for every Ruby method call.
And one last case that's a problem: author's intent. If I write a
block of code that does "1 + 1", I intend for that code to do normal
Fixnum#+, and I intend for the result to be 2. It should not be
possible for a caller to change the intent of my code just because I
passed it in a bock. This has been my argument against using blocks as
bindings, and it's another argument against instance_eval being able
to force refinments into "someone else's code".
I could continue to try to theorize about possible implementations,
but they all lead toward runtime-alterable refinements being a
devlishly complicated thing to implement and potentially impossible to
optimize. I could be wrong, especially if my understanding about the
feature is flawed.
Now, some positive reinforcement for "using" being a keyword and
instance_eval not propagating refinements.
If "using" were a keyword, calls within the related lexical scopes
would become "refinement-aware calls", localizing performance impact
to only those calls. They would need additional cache guards,
potentially with global impact, but at least normal code would work
exactly as it does today. Block bodies would be no exception; unless a
"using" were active at parse time, all calls could be taken at face
value. This would also preclude instance_eval of a block propagating
refinements, since the parse-time nature of "using" would mean a block
is what it is, forever. Your code can't modify the intent of my code,
and only calls where a parent scope at parse time contains the "using"
keyword would know anything about refinements.
This is, in fact, how I implemented "selector namespaces" over a year
ago in JRuby, as an example.
http://groups.google.com/group/ruby-core-google/msg/6f45dcb363e75267
I can try to come up with a concrete example of the problems with the
current proposal and implementation, but the concurrency cases would
be difficult to show.
- Charlie
Jedediah Smith
It would be great to finally have scoped monkey patching for Ruby, but I am finding some problems with this approach.
There are many ways for refinements to leak out of the lexical scope that uses them:
1. Inheritance (which is used in many APIs e.g. ActiveRecord::Base)
2. DSLs that use instance_eval
3. Re-opening a class
4. When a class that uses refinements is itself refined (or does it not work that way?)
Classes that are to be used in any of these ways can't use refinements internally, I guess. For 1 and 4, that could be any class at all. So we still have to be paranoid about using refinements in library code, which kind of defeats the purpose.
Also, if "using" and "include" both operate on modules, the difference between them is going to be confusing, particularly for those unfamiliar with the history of the language. It seems arbitrary to overload modules with a use unrelated to their existing purpose. And having to group refinements in modules seems overly complicated for many cases. I would like to be able to do something like this:
refinement StringExt < String
def pig_latin
"#{self[1..-1]}#{self[0]}ay"
end
end
...
using StringExt
"woot".pig_latin # ==> "ootway"
But there does need to be a way to group refinements, and there should probably be a way to combine groups as well. Perhaps this:
refinement CoreExt # empty refinement bound to constant CoreExt
using ArrayExt # import other refinement(s)
refinement < String # anonymous refinement of String
...
end
end
This keeps refinements and modules separate and makes their usage clear: modules are "included" dynamically, refinements are "used" lexically. Mix them up and you get a TypeError. It also keeps definition of refinements separate from their use, and lets you use an anonymous refinement inside a module without exporting it. Refinements can still be nested members of modules. The above syntax may not be quite right, but I think the semantics are: refinements are composable sets of lexically scoped monkey patches.
Shugo Maeda
2010/11/27 Haase, Konstantin <Konstant...@student.hpi.uni-potsdam.de>:
> So, is local rebinding still on the table? From my point of view local rebinding is far more intuitive and there are some use cases I am not sure I could solve without local rebinding.
Could you tell me one of the use cases?
--
Shugo Maeda
Shugo Maeda
2010/11/27 Yukihiro Matsumoto <ma...@ruby-lang.org>:
> |Matz doesn't like it, but I think it's worth considering.
> |However, it's a problem that we have main#include, where main is self
> |at the top-level, but not Kernel#include now.
>
> If we introduce local rebinding, I think we have to rename it to
> "classbox" from "refinement". Besides that, we sill have lot of
> issues about implementation, performance, etc. But I believe, Shugo,
> you are the key person. I am the one who approve.
Thanks for your encouragement.
I guess it's hard to get impeccable conclusion.
I'll express my preferences, but I would like you to make the final decision.
--
Shugo Maeda
Shugo Maeda
2010/11/28 Magnus Holm <jud...@gmail.com>:
> Thanks a lot. Could you explain why included modules are rebound though?
It was a bug. Please try the following three patches instead of
refinement-r29837-20101124.diff.
control_frame_change-r29944-20101127.diff
refinements-r29944-20101127.diff
nested_methods-r29944-20101127.diff
They are available at:
http://redmine.ruby-lang.org/issues/show/4085
--
Shugo Maeda
Haase, Konstantin
> 2010/11/27 Haase, Konstantin <Konstant...@student.hpi.uni-potsdam.de>:
>> So, is local rebinding still on the table? From my point of view local rebinding is far more intuitive and there are some use cases I am not sure I could solve without local rebinding.
>
> Could you tell me one of the use cases?
When writing an asynchronous Rack application, you cannot use Rack::Lint, since you return a status code of -1 to signal your Rack handler that you will respond to the incoming request later. One option would be to completely disable Rack::Lint, which might not be what you want and is a bit painful, as it is hardcoded to be used in development mode in Rack. One could monkey-patch Rack::Lint directly, but it would be even better if it only excepts -1 for your application:
module AsyncRack
refine Rack::Lint do
def check_status(status)
super unless status == -1
end
end
end
using AsyncRack
The complete async rack library (https://github.com/rkh/async-rack) could be implemented that way. Evil hacks for replacing constants are necessary at the moment, though it would also be solvable - modulo having the changes only locally instead of globally - by the proposed Module#prepend.
I think in general there are two rather distinct use case groups: Those where I don't know and don't want to have to care about the internals of the class that's being refined and those where I do and explicitly want to reach in deep to change a single internal (say in Rails I want to change how class names are mapped to files only in one initializer). If I don't have local rebinding, I would still have to care about the original implementation in order to figure out what methods are calling the method I want to change. In the Enumerable example I would have to override about every method, not only each, in order to change the behavior consistently.
Konstantin
Shugo Maeda
2010/11/30 Yusuke ENDOH <ma...@tsg.ne.jp>:
>>> Because it requires less indentation, I thought.
>>
>> I see. refine without blocks looks confusing for me because it works
>> different from refine with a block.
>
> Maybe another name is needed?
>
> I think that the short name `refine' is more appropriate to the non-
> block style than the block style because I believe the non-block
> style is more suitable for casual use. It requires less indentation,
> and it complies with traditional style (like Module#include).
It doesn't make sense because Module#include is a very different
feature from refine.
My proposal is to use modules as namespaces for refinements. So
indentation is a necessary evil. Otherwise, we need syntax like Java
packages and one file for each package.
> A longer (more "meta-programming-like") name would be appropriate to
> the block style, such as Module#refine_class, #refine_class_eval,
> #class_eval_with_refinement.
Meta-programming means programming on programs, so the non-block style
is also a meta-programming feature. Why should only the block style
be named more "meta-programming-like"?
>>> I guess that most of these constructs have reasons why they need
>>>> keywords and special syntax.
>>>
>>> I don't think so. "class Foo; end" can be written as "Foo =
>>> Class.new { }" (though there are indeed subtle differences between
>>> them).
>>
>> "refine Foo do end" is different from "Foo = Class.new {}" because
>> "refine Foo do end" looks good, but "Foo = Class.new {}" doesn't.
>> I think how it looks is more important than whether it uses keywords or not.
>
> "refine Foo do def ... end end" looks not so good to me.
Could you tell me why
refine Foo
def bar; end
end
is good but
refine Foo do
def bar; end
end
is not so good?
They look similar for me. The latter has "do", but it seems a good
word in this context.
# I'm not sure because I'm not a good English writer.
>>> The API design that "def" statements are put in a Ruby's block,
>>> is slightly weird (for me). I guess that there is no precedent of
>>> such a style in Ruby's embedded featues, except meta programming
>>> (such as Class.new and class_eval).
>>> From now on, does Ruby encourage such a style in casual use?
>>
>> I think Module#refine is a meta programming feature like class_eval,
>> and most application programmers need not use it directly.
>
> I guess you think so because we are not used to the feature yet.
> If it is really just a meta-programming feature, the name should be
> more "meta-programming-like".
I don't know why meta-programming features should have long names.
In Ruby, meta-programming is encouraged, and meta-programming features
sometimes have a short name such as eval, but rarely have a keyword.
> The non-block style has a precedent (Module#include), so, if it is
> adopted, I agree that any new keyword is not needed. Otherwise, I
> prefer a new keyword to a new weird (to me) coding style.
Are precedents so important for innovations?
--
Shugo Maeda
elise huard
I was wondering if you could also address Charlie Nutter's concerns, I
think he makes some valid points there.
Regards,
Elise
Loren Segal
On 12/2/2010 7:55 AM, Shugo Maeda wrote:
>
> Could you tell me why
>
> refine Foo
> def bar; end
> end
>
> is good but
>
> refine Foo do
> def bar; end
> end
>
> is not so good?
The "refine do ... end" block implies a method call which therefore
becomes easily overridable at run time. This means the compiler cannot
statically compute refinements unless it just assumed "refine" was never
overridden. If refinements are truly meant to be lexically scoped, this
should be reflected in the compiler's handling of them. Charles Nutter's
post illustrates why this might matter.
- Loren
Yusuke ENDOH
2010/12/2 Shugo Maeda <sh...@ruby-lang.org>:
> My proposal is to use modules as namespaces for refinements. So
> indentation is a necessary evil. Otherwise, we need syntax like Java
> packages and one file for each package.
I'm not against using modules as namespaces and refinement scope,
but I don't like to see the same module being used for refinement
and traditional use at the same time.
Modules for mix-in, modules for collection of helper methods, and
modules for refinement should be separated, at least, in casual
use.
We can clearly see that the following module FooExt is only for
refinement. Thus I like this style.
module FooExt
refine Foo
def ... end
end
>> A longer (more "meta-programming-like") name would be appropriate to
>> the block style, such as Module#refine_class, #refine_class_eval,
>> #class_eval_with_refinement.
>
> Meta-programming means programming on programs, so the non-block style
> is also a meta-programming feature. Why should only the block style
> be named more "meta-programming-like"?
I can call it "not for casual use" instead of "meta-programming-
like."
> Could you tell me why
>
> refine Foo
> def bar; end
> end
>
> is good but
>
> refine Foo do
> def bar; end
> end
>
> is not so good?
Because block includes method definition. A block looks to me
"dynamic" behavior, while method definition (using `def' keyword)
looks "static" behavior.
Of course I know that both are also evaluated dynamically in
Ruby, but I don't think that Ruby encourages such a style so much.
> I don't know why meta-programming features should have long names.
> In Ruby, meta-programming is encouraged, and meta-programming features
> sometimes have a short name such as eval, but rarely have a keyword.
Ah, we found the perception gap between I and you.
I do NOT think that meta-programming is so encouraged even in Ruby.
It is like "a trick," and should be used only when it is appropriate.
If it was really so encouraged, Ruby would not provide many
syntax, such as `def' and `class.'
And I guess that "eval" just came from Perl.
>> The non-block style has a precedent (Module#include), so, if it is
>> adopted, I agree that any new keyword is not needed. Otherwise, I
>> prefer a new keyword to a new weird (to me) coding style.
>
> Are precedents so important for innovations?
I like this feature as new OO paradigm, but syntax is another
matter.
I also like traditional syntax design principle --including
"syntax-like convention" such as Kernel#require and #include--
and I want this feature also to respect it.
--
Yusuke Endoh <ma...@tsg.ne.jp>
Yusuke ENDOH
2010/11/30 Charles Oliver Nutter <hea...@headius.com>:
> The global serial number approach in 1.9 means that any change that
> flip that serial number cause all caches everywhere to invalidate.
> Normally this only happens on method definition or module inclusion,
> which is why defining methods or including modules at runtime is
> strongly discouraged for performance reasons.
Sorry I'm not sure that I could follow your argument about performance,
so I may miss your point.
I guess that casual users will execute all refinements immediately after
program is started, like class definition and method definition.
Thus, the global serial number approach will work well for refinement
in main use cases, I think.
In the sense, nested function by using refinements may be a problem.
> And one last case that's a problem: author's intent. If I write a
> block of code that does "1 + 1", I intend for that code to do normal
> Fixnum#+, and I intend for the result to be 2. It should not be
> possible for a caller to change the intent of my code just because I
> passed it in a bock. This has been my argument against using blocks as
> bindings, and it's another argument against instance_eval being able
> to force refinments into "someone else's code".
This is not a problem, but rather improvement. There is already open
class which so often breaks your intent. Refinements may also break
your intent, but it is less often and more controllable than open class.
> Now, some positive reinforcement for "using" being a keyword and
> instance_eval not propagating refinements.
I'm not against your proposal, but I wonder if it does not make sense
because we can still write: eval("using FooExt")
To address your concern, `using' keyword should have a block:
using FooExt
# FooExt enabled
end
# FooExt disabled
I don't like this syntax because of more indentation, though.
> I can try to come up with a concrete example of the problems with the
> current proposal and implementation, but the concurrency cases would
> be difficult to show.
I might find serious concurrency problem of Shugo's patch, though I'm
not sure that this is what you mean. Indeed, we may have to give up
propagating refinements via block.
class C
def test; p :test; end
end
module FooExt
refine(C) { def test; p :foo; end }
end
module BarExt
refine(C) { def test; p :bar; end }
end
f = proc do
sleep 1
C.new.test
end
FooExt.class_eval(&f) #=> :foo (expected)
BarExt.class_eval(&f) #=> :bar (expected)
[ Thread.new { FooExt.class_eval(&f) }, #=> :foo (expected)
Thread.new { BarExt.class_eval(&f) } #=> :foo (not expected)
].each {|t| t.join }
--
Yusuke Endoh <ma...@tsg.ne.jp>
Shugo Maeda
2010/11/30 Charles Oliver Nutter <hea...@headius.com>:
> * "using" not being a keyword requires all calls to check for
> refinements all the time, globally degrading performance.
This means that you should check a flag or something in StaticScope
for every method invocation, and you cannot accept the overhead,
right?
What do you think of refine? Should it also be a keyword?
How about refinements activation in reopened definitions and
refinement inheritance?
Can all they be problems?
> * instance_eval propagating refinements requires all block invocations
> to localize what refinements they use for invocation on every
> activation.
> * shared, mutable structures can't be used to store the active
> refinement, due to concurrency issues
> * there are very likely many more complexities than illustrated here
> that result from the combination of runtime-mutable lexical scoping
> structures, concurrency, and method caching.
As Yusuke showed in [ruby-core:33535], the current implementation has
this problem.
The current implementation checks only the (singleton) class of the
receiver and the VM version. It should also check the refinements in
cref to avoid this problem,
but it causes more overhead.
I might withdraw the proposal of refinement propagation for blocks
given to instance_eval,
but what do you think of instance_eval without blocks?
x.instance_eval("...")
And, how about to introduce a new method (e.g.,
Module#refinement_eval) which copies a given block and make the copy
refinement aware?
I think blocks are useful to implement DSLs like RSpec.
it "should ..." do
foo = Foo.new
foo.should == ... # Object#should is available only in the block.
end
--
Shugo Maed
Shugo Maeda
2010/11/30 Jedediah Smith <red...@ruby-lang.org>:
> There are many ways for refinements to leak out of the lexical scope that uses them:
That is why I called it *pseudo*-lexical.
> 1. Inheritance (which is used in many APIs e.g. ActiveRecord::Base)
I guess this needs more discussion.
We may need something like `private using' which does not affect subclasses.
> 2. DSLs that use instance_eval
> 3. Re-opening a class
I guess they are more controllable than inheritance. But they may
have other problems as pointed out by Charles.
> 4. When a class that uses refinements is itself refined (or does it not work that way?)
I guess it doesn't work that way because we have no local rebinding.
> Also, if "using" and "include" both operate on modules, the difference between them is going to be confusing, particularly for those unfamiliar with the history of the language. It seems arbitrary to overload modules with a use unrelated to their existing purpose. And having to group refinements in modules seems overly complicated for many cases. I would like to be able to do something like this:
I don't think it's so confusing. It complies with the Large Class Principle.
--
Shugo Maeda
Shugo Maeda
2010/12/2 Yusuke ENDOH <ma...@tsg.ne.jp>:
>> My proposal is to use modules as namespaces for refinements. So
>> indentation is a necessary evil. Otherwise, we need syntax like Java
>> packages and one file for each package.
>
> I'm not against using modules as namespaces and refinement scope,
> but I don't like to see the same module being used for refinement
> and traditional use at the same time.
> Modules for mix-in, modules for collection of helper methods, and
> modules for refinement should be separated, at least, in casual
> use.
I agree with you. But I don't think Ruby should enforce it.
> We can clearly see that the following module FooExt is only for
> refinement. Thus I like this style.
>
> module FooExt
> refine Foo
> def ... end
> end
I don't like this style as the primary way because it can be used for
only one class.
A namespace should be able to have multiple names.
>>> A longer (more "meta-programming-like") name would be appropriate to
>>> the block style, such as Module#refine_class, #refine_class_eval,
>>> #class_eval_with_refinement.
>>
>> Meta-programming means programming on programs, so the non-block style
>> is also a meta-programming feature. Why should only the block style
>> be named more "meta-programming-like"?
>
> I can call it "not for casual use" instead of "meta-programming-
> like."
I see.
>> Could you tell me why
>>
>> refine Foo
>> def bar; end
>> end
>>
>> is good but
>>
>> refine Foo do
>> def bar; end
>> end
>>
>> is not so good?
>
> Because block includes method definition. A block looks to me
> "dynamic" behavior, while method definition (using `def' keyword)
> looks "static" behavior.
> Of course I know that both are also evaluated dynamically in
> Ruby, but I don't think that Ruby encourages such a style so much.
method definitions don't look static for me, but nari said the same....
My brain might be too optimized for Ruby.
>> I don't know why meta-programming features should have long names.
>> In Ruby, meta-programming is encouraged, and meta-programming features
>> sometimes have a short name such as eval, but rarely have a keyword.
>
> Ah, we found the perception gap between I and you.
> I do NOT think that meta-programming is so encouraged even in Ruby.
> It is like "a trick," and should be used only when it is appropriate.
> If it was really so encouraged, Ruby would not provide many
> syntax, such as `def' and `class.'
>
> And I guess that "eval" just came from Perl.
I guess it's a problem of the term meta-programming, so let's not use
the word. What I want to say is that I think use frequency of refine
is not so high to require a new keyword, but enough high to have a
nice method name.
To tell the truth, I can accept new keywords refine and using, but I'm
afraid it makes Refinements a Ruby 2.0 feature.
--
Shugo Maeda
Shugo Maeda
2010/12/2 Haase, Konstantin <Konstant...@student.hpi.uni-potsdam.de>:
> I think in general there are two rather distinct use case groups: Those where I don't know and don't want to have to care about the internals of the class that's being refined and those where I do and explicitly want to reach in deep to change a single internal (say in Rails I want to change how class names are mapped to files only in one initializer). If I don't have local rebinding, I would still have to care about the original implementation in order to figure out what methods are calling the method I want to change. In the Enumerable example I would have to override about every method, not only each, in order to change the behavior consistently.
I admit that local rebinding is useful in some cases.
However, Ruby already has dynamic way to extend classes such as monkey
patching and singleton methods, so I guess refinements should be more
(pseudo) static.
--
Shugo Maeda
Shugo Maeda
2010/12/2 Loren Segal <lse...@soen.ca>:
Even if using is a keyword, the compiler cannot compute refinements because
refinements are defined at run-time.
The compiler can only know whether refinements are active or not.
However, it may be important for localizing the performance hit to
only scopes where refinements are active, as Charles says.
If the overhead of this check is enough small to apply for every
method invocation, there is no need to make using a keyword.
--
Shugo Maeda
Yusuke ENDOH
2010/12/3 Shugo Maeda <sh...@ruby-lang.org>:
> 2010/12/2 Yusuke ENDOH <ma...@tsg.ne.jp>:
>>> My proposal is to use modules as namespaces for refinements. So
>>> indentation is a necessary evil. Otherwise, we need syntax like Java
>>> packages and one file for each package.
>>
>> I'm not against using modules as namespaces and refinement scope,
>> but I don't like to see the same module being used for refinement
>> and traditional use at the same time.
>> Modules for mix-in, modules for collection of helper methods, and
>> modules for refinement should be separated, at least, in casual
>> use.
>
> I agree with you. But I don't think Ruby should enforce it.
Yes. In fact, I'm ok even if the block style API is *also* included.
However, Ruby can (and should) "navigate" user to encouraged style,
by making encouraged API more useful, such as using shorter method
name.
>> We can clearly see that the following module FooExt is only for
>> refinement. Thus I like this style.
>>
>> module FooExt
>> refine Foo
>> def ... end
>> end
>
> I don't like this style as the primary way because it can be used for
> only one class.
We can use the block style API for such a use case:
module FooBarBazExt
refine_eval Foo do
def ext; end
end
refine_eval Bar do
def ext; end
end
refine_eval Baz do
def ext; end
end
end
However, it would be better to separate modules, especially when the
code grows:
module FooExt
refine Foo
def ext1; end
def ext2; end
def ext3; end
...
end
module BarExt
refine Bar
...
end
module BaZExt
refine Baz
...
end
module FooBarBazExt
using FooExt # or include FooExt
using BarExt
using BazExt
end
... Oops! This does not work as excepted. I had believed that this
would work... Why don't you allow this?
>> Because block includes method definition. A block looks to me
>> "dynamic" behavior, while method definition (using `def' keyword)
>> looks "static" behavior.
>> Of course I know that both are also evaluated dynamically in
>> Ruby, but I don't think that Ruby encourages such a style so much.
>
> method definitions don't look static for me, but nari said the same....
> My brain might be too optimized for Ruby.
Many men, many (pseudo) Ruby. My (and nari's) Ruby could be wrong.
Only matz has the true Ruby. We should ask his opinion.
> To tell the truth, I can accept new keywords refine and using, but I'm
> afraid it makes Refinements a Ruby 2.0 feature.
Your suggestion is really well-conceived, but still needs discussion.
If it is included in 1.9.x once, we cannot change the interface because
of compatibility. Thus, we should take cautious steps to include the
feature.
As a first step, how about including only the mechanism and its
*undocumented* C API, and publishing a refinement gem for Ruby API?
Then, trunk developers can easily examine the feature by using the gem.
We can also examine the actual performance. If we find any problem or
came up with a better idea, we can change the API without care of
compatibility (because it is just an undocumented API and an untrustful
gem!), or even remove and forget it at worst.
# Of course, this is a topic after the discussion is closed.
--
Yusuke Endoh <ma...@tsg.ne.jp>
Shugo Maeda
2010/12/3 Yusuke ENDOH <ma...@tsg.ne.jp>:
>>> I'm not against using modules as namespaces and refinement scope,
>>> but I don't like to see the same module being used for refinement
>>> and traditional use at the same time.
>>> Modules for mix-in, modules for collection of helper methods, and
>>> modules for refinement should be separated, at least, in casual
>>> use.
>>
>> I agree with you. But I don't think Ruby should enforce it.
>
> Yes. In fact, I'm ok even if the block style API is *also* included.
> However, Ruby can (and should) "navigate" user to encouraged style,
> by making encouraged API more useful, such as using shorter method
> name.
Hmm, I prefer the keyword refine to the method refine without blocks....
>> I don't like this style as the primary way because it can be used for
>> only one class.
>
> We can use the block style API for such a use case:
>
> module FooBarBazExt
> refine_eval Foo do
> def ext; end
> end
> refine_eval Bar do
> def ext; end
> end
> refine_eval Baz do
> def ext; end
> end
> end
refine_eval doesn't look a good name. I don't think the above code
should be so discouraged.
> However, it would be better to separate modules, especially when the
> code grows:
>
> module FooExt
> refine Foo
> def ext1; end
> def ext2; end
> def ext3; end
> ...
> end
> module BarExt
> refine Bar
> ...
> end
> module BaZExt
> refine Baz
> ...
> end
>
> module FooBarBazExt
> using FooExt # or include FooExt
> using BarExt
> using BazExt
> end
>
>
> ... Oops! This does not work as excepted. I had believed that this
> would work... Why don't you allow this?
Currently refine doesn't work without blocks, but do you mean that?
>> To tell the truth, I can accept new keywords refine and using, but I'm
>> afraid it makes Refinements a Ruby 2.0 feature.
>
> Your suggestion is really well-conceived, but still needs discussion.
> If it is included in 1.9.x once, we cannot change the interface because
> of compatibility. Thus, we should take cautious steps to include the
> feature.
>
> As a first step, how about including only the mechanism and its
> *undocumented* C API, and publishing a refinement gem for Ruby API?
> Then, trunk developers can easily examine the feature by using the gem.
> We can also examine the actual performance. If we find any problem or
> came up with a better idea, we can change the API without care of
> compatibility (because it is just an undocumented API and an untrustful
> gem!), or even remove and forget it at worst.
I think it's hard to add new keywords by a gem. Have you abandoned
keywords? I prefer to the keyword refine to the method refine without
blocks suggested by you.
Isn't it enough to introduce refinements as an experimental feature,
at least in trunk?
--
Shugo Maeda
Yusuke ENDOH
2010/12/3 Shugo Maeda <sh...@ruby-lang.org>:
>> ... Oops! This does not work as excepted. I had believed that this
>> would work... Why don't you allow this?
>
> Currently refine doesn't work without blocks, but do you mean that?
Ah sorry.
module FooExt
refine(Fixnum) do
def /(other); quo(other); end
end
end
module BarExt
include FooExt # or using FooExt
end
using BarExt
p 1 / 2 #=> actual: 0, expected: (1/2)
> I think it's hard to add new keywords by a gem. Have you abandoned
> keywords? I prefer to the keyword refine to the method refine without
> blocks suggested by you.
I first said in [ruby-core:33375]:
> However, we should discuss this topic (new keyword) towards
> 2.0. Module's methods are not bad, as a part of reflection
> features (such as Module#define_method for `def' keyword).
Needless to say, we must not add any new keywords to 1.9.x, especially
normal simple word like "refine." I'm ok to include this feature in
1.9.x, and now I believe that gem is a good idea as the first step.
However, I received many negative comments to this approach (gem),
from nars*, kosak*, ko*, nakad*, shyouhe*. They seem to think that
it is better to import your patch "as is".
> Isn't it enough to introduce refinements as an experimental feature,
> at least in trunk?
I don't like to include a feature called "experimental", not because
it is not complete yet, but because it becomes "de facto standard."
It would be good if there are not only "document" but also "mechanism"
to inform users that the feature is experimental, such as warning or
a new method like "RubyVM.enable_experimental_features".
Note that this is just my opinion, and that I seem to be in the
minority ;-)
I hope that 1.9.x would be stable, but many other committers seem to
hope to include new feature in 1.9.x aggressively.
--
Yusuke Endoh <ma...@tsg.ne.jp>
Charles Oliver Nutter
On Thu, Dec 2, 2010 at 11:05 PM, Shugo Maeda <sh...@ruby-lang.org> wrote:
> 2010/11/30 Charles Oliver Nutter <hea...@headius.com>:
>> * "using" not being a keyword requires all calls to check for
>> refinements all the time, globally degrading performance.
>
> This means that you should check a flag or something in StaticScope
> for every method invocation, and you cannot accept the overhead,
> right?
Every little bit matters. In experimental optimizations, JRuby is able
to reduce a dynamic call to two memory indirections + compare + static
dispatch directly to jitted code. When the dispatch path is this fast,
adding multiple additional layers of memory indirection and comparison
to support rarely-changing refinements can definitely show up. I also
have not attempted to implement an optimized version of refinement
dispatch, and I worry that there will be additional
performance-impacting issues.
> What do you think of refine? Should it also be a keyword?
> How about refinements activation in reopened definitions and
> refinement inheritance?
> Can all they be problems?
refine as keyword: I don't think so, since it, like many other
meta-programming methods, applies its changes only once to the
class/module that surrounds it and the block it has been given. It's
very "magical", but I'm not sure that's enough of a reason to make it
a keyword.
refinement activation in reopened definition: Reopened definitions are
hierarchies of new lexical scopes. If a refinement is active for those
scopes it will only affect them and no other scopes. Obviously you
should not be able to apply a new refinement to scopes that have
already been closed without refinements in place. That is my #1
concern with this proposal...what is static (as in StaticScope in
JRuby) should remain static. I don't believe "refine" or refinements
applied in reopened classes violate that rule.
>> * there are very likely many more complexities than illustrated here
>> that result from the combination of runtime-mutable lexical scoping
>> structures, concurrency, and method caching.
>
> As Yusuke showed in [ruby-core:33535], the current implementation has
> this problem.
> The current implementation checks only the (singleton) class of the
> receiver and the VM version. It should also check the refinements in
> cref to avoid this problem,
> but it causes more overhead.
More overhead is always bad if it affects performance globally :)
> I might withdraw the proposal of refinement propagation for blocks
> given to instance_eval,
> but what do you think of instance_eval without blocks?
>
> x.instance_eval("...")
This form is "safe" since the string needs to be parsed and evaluated
(and provided with a new scope) each time. I worry a bit about the
inconsistency of having the "" form propagate refinements but the {}
form not propagating refinements. Perhaps this is a big like the
duality of constant lookup inside instance_eval?
> And, how about to introduce a new method (e.g.,
> Module#refinement_eval) which copies a given block and make the copy
> refinement aware?
> I think blocks are useful to implement DSLs like RSpec.
>
> it "should ..." do
> foo = Foo.new
> foo.should == ... # Object#should is available only in the block.
> end
Yes, I agree this is a useful and difficult case to address. One
possible solution would be making the modification of the block
explicit and permanent:
p = proc { bar }
p.refine! BarModifyingExt
# p from now on has BarModifyingExt's refinements applied to it
This still can suffer from concurrency problems, if the "p" proc is
stored somewhere and in use when Proc#refine is called, but I think it
addresses the problem of having blocks flip back and forth between
refined and unrefined. It's not perfect wrt concurrency, but it's
(somewhat) better.
I'm not sure this addresses the problems with performance, though. All
method calls in all blocks everywhere would still need to check if
they've been turned into a Proc and "refined", and that would be a
larger impact than simply checking a global flag.
The bottom line is that anything which makes static parse output now
mutable will always be a concurrency problem, and probably always be a
performance hit. All Ruby implementations currently depend on certain
immutable truths about parser output, and this proposal violates many
of those truths.
To be honest, I think it should be a using directive at the file's
toplevel, so subsequently-parsed blocks know they're going to be
refined. I believe more and more that "using" needs to be a keyword;
having yet another "magic" method that can alter parse/execution
behavior beyond its scope scares me.
require 'rspec'
# RSpec::Refinements would define all methods currently
# defined against Object or Kernel, and the refine here
# would simply apply them to this file.
# TODO: using affects current scope or only child scopes?
# The latter would be best, since the current scope is already
# "active" and should not be modified.
using RSpec::Refinements
# does this scope have refinements active?
# describe is from refinement(?) or a real method
describe 'blah' do
# this scope has refinements active
# "it" is either from refinement or a real method
it 'blah' do
# this scope has refinements active
# "should" is from refinement
x.should == y
end
end
- Charlie
Shugo Maeda
2010/12/3 Yusuke ENDOH <ma...@tsg.ne.jp>:
>>> ... Oops! This does not work as excepted. I had believed that this
>>> would work... Why don't you allow this?
>>
>> Currently refine doesn't work without blocks, but do you mean that?
>
> Ah sorry.
>
> module FooExt
> refine(Fixnum) do
> def /(other); quo(other); end
> end
> end
> module BarExt
> include FooExt # or using FooExt
> end
> using BarExt
> p 1 / 2 #=> actual: 0, expected: (1/2)
I forgot to support it. Please apply the attached patch. I have also
added Kernel#refine.
>> I think it's hard to add new keywords by a gem. Have you abandoned
>> keywords? I prefer to the keyword refine to the method refine without
>> blocks suggested by you.
>
> I first said in [ruby-core:33375]:
>
>> However, we should discuss this topic (new keyword) towards
>> 2.0. Module's methods are not bad, as a part of reflection
>> features (such as Module#define_method for `def' keyword).
>
> Needless to say, we must not add any new keywords to 1.9.x, especially
> normal simple word like "refine." I'm ok to include this feature in
> 1.9.x, and now I believe that gem is a good idea as the first step.
>
> However, I received many negative comments to this approach (gem),
> from nars*, kosak*, ko*, nakad*, shyouhe*. They seem to think that
> it is better to import your patch "as is".
I also wouldn't like to see such a language core feature in a gem.
>> Isn't it enough to introduce refinements as an experimental feature,
>> at least in trunk?
>
> I don't like to include a feature called "experimental", not because
> it is not complete yet, but because it becomes "de facto standard."
> It would be good if there are not only "document" but also "mechanism"
> to inform users that the feature is experimental, such as warning or
> a new method like "RubyVM.enable_experimental_features".
As far as I know, the current status of trunk is unstable, and if new
features break backward compatibility, Yugui will create the branch
ruby_1_9, won't you, Yugui? So, we don't need the above mechanism
until new experimental features are decided to include to 1.9.x.
> Note that this is just my opinion, and that I seem to be in the
> minority ;-)
> I hope that 1.9.x would be stable, but many other committers seem to
> hope to include new feature in 1.9.x aggressively.
I also hope that 1.9.x would be stable, but I'd like to develop
aggressively in trunk. I don't think the current design and
implementation of refinements are mature, but I can't make them mature
on my own, so I'd like to have help from other committers.
--
Shugo Maeda
Charles Oliver Nutter
On Thu, Dec 2, 2010 at 9:42 PM, Yusuke ENDOH <ma...@tsg.ne.jp> wrote:
> 2010/11/30 Charles Oliver Nutter <hea...@headius.com>:
>> The global serial number approach in 1.9 means that any change that
>> flip that serial number cause all caches everywhere to invalidate.
>> Normally this only happens on method definition or module inclusion,
>> which is why defining methods or including modules at runtime is
>> strongly discouraged for performance reasons.
>
> Sorry I'm not sure that I could follow your argument about performance,
> so I may miss your point.
>
> I guess that casual users will execute all refinements immediately after
> program is started, like class definition and method definition.
> Thus, the global serial number approach will work well for refinement
> in main use cases, I think.
> In the sense, nested function by using refinements may be a problem.
For the main cases, I see it working this way:
* Parser sees "using" in a scope
* Child scopes will now be parsed as though they are refined
* VCALL, FCALL, CALL in child scopes are instead RVCALL, RFCALL, RCALL
** Likely other calls must be replaced/wrapped too, e.g. +=, []=,
[]+=, and so on. (messy!)
* Refined call versions check refinement first before checking
metaclass for target method
Maybe this helps make it clear why the refinement state of a given
scope must never be mutable...we want to be able to limit the extra
refinement checks to calls where refinements are active, leaving
unrefined calls as they are today.
>> And one last case that's a problem: author's intent. If I write a
>> block of code that does "1 + 1", I intend for that code to do normal
>> Fixnum#+, and I intend for the result to be 2. It should not be
>> possible for a caller to change the intent of my code just because I
>> passed it in a bock. This has been my argument against using blocks as
>> bindings, and it's another argument against instance_eval being able
>> to force refinments into "someone else's code".
>
> This is not a problem, but rather improvement. There is already open
> class which so often breaks your intent. Refinements may also break
> your intent, but it is less often and more controllable than open class.
Open classes break my intent globally and usually at boot time.
Refinement propagation into blocks is much more likely to break my
intent at some arbitrary time in the future, since refinements are
much more lazily applied than open class modifications. Lazy breakage
is always worse than eager breakage.
>> Now, some positive reinforcement for "using" being a keyword and
>> instance_eval not propagating refinements.
>
> I'm not against your proposal, but I wonder if it does not make sense
> because we can still write: eval("using FooExt")
eval is not a concern, because it always creates a new scope. If a
"using" is active, those scopes (or at least their child scopes) would
be statically marked as "refined", and my requirements are still met.
> To address your concern, `using' keyword should have a block:
>
> using FooExt
> # FooExt enabled
> end
> # FooExt disabled
>
> I don't like this syntax because of more indentation, though.
I mention this at the bottom of my reply to Shugo...I'm uncomfortable
with "using" applying to the current scope and not just to child
scopes encountered after it. Once code begins executing against a
given scope, that scope's static state (like whether a refinement is
active) should never change. Given that "using" occurs only at
toplevel or in class/module bodies (i.e., during file load/require),
there's not as many concurrency concerns here. There are just a lot of
messy issues with "using" applying to the current scope:
* Does it affect calls that came before it?
* Do multiple "using" directives combine or overwrite each other?
* When pre-parsing or pre-compiling (AOT stuff), you would have to
walk the whole file looking for "using" to know how to emit compiled
results.
What we're really talking about with refinements is a way to lexically
say "make calls in these scopes do an extra indirection during
lookup." Unrefined calls should never have to perform this additional
indirection.
>> I can try to come up with a concrete example of the problems with the
>> current proposal and implementation, but the concurrency cases would
>> be difficult to show.
>
> I might find serious concurrency problem of Shugo's patch, though I'm
> not sure that this is what you mean. Indeed, we may have to give up
> propagating refinements via block.
>
> class C
> def test; p :test; end
> end
> module FooExt
> refine(C) { def test; p :foo; end }
> end
> module BarExt
> refine(C) { def test; p :bar; end }
> end
> f = proc do
> sleep 1
> C.new.test
> end
> FooExt.class_eval(&f) #=> :foo (expected)
> BarExt.class_eval(&f) #=> :bar (expected)
> [ Thread.new { FooExt.class_eval(&f) }, #=> :foo (expected)
> Thread.new { BarExt.class_eval(&f) } #=> :foo (not expected)
> ].each {|t| t.join }
Thank you, this helps illustrate the problems with modifying a block's
(formerly) static state. We really must not modify an
already-parsed/compiled scope with new refinements.
- Charlie
Shugo Maeda
2010/12/3 Yusuke ENDOH <ma...@tsg.ne.jp>:
> I might find serious concurrency problem of Shugo's patch, though I'm
> not sure that this is what you mean. Indeed, we may have to give up
> propagating refinements via block.
I have fixed it with additional overhead:)
Please try the attached patch.
--
Shugo Maeda
Shugo Maeda
2010/12/4 Charles Oliver Nutter <hea...@headius.com>:
>>> * "using" not being a keyword requires all calls to check for
>>> refinements all the time, globally degrading performance.
>>
>> This means that you should check a flag or something in StaticScope
>> for every method invocation, and you cannot accept the overhead,
>> right?
>
> Every little bit matters. In experimental optimizations, JRuby is able
> to reduce a dynamic call to two memory indirections + compare + static
> dispatch directly to jitted code. When the dispatch path is this fast,
> adding multiple additional layers of memory indirection and comparison
> to support rarely-changing refinements can definitely show up. I also
> have not attempted to implement an optimized version of refinement
> dispatch, and I worry that there will be additional
> performance-impacting issues.
I see.
>> What do you think of refine? Should it also be a keyword?
>> How about refinements activation in reopened definitions and
>> refinement inheritance?
>> Can all they be problems?
>
> refine as keyword: I don't think so, since it, like many other
> meta-programming methods, applies its changes only once to the
> class/module that surrounds it and the block it has been given. It's
> very "magical", but I'm not sure that's enough of a reason to make it
> a keyword.
I'm also not sure.
> refinement activation in reopened definition: Reopened definitions are
> hierarchies of new lexical scopes. If a refinement is active for those
> scopes it will only affect them and no other scopes. Obviously you
> should not be able to apply a new refinement to scopes that have
> already been closed without refinements in place. That is my #1
> concern with this proposal...what is static (as in StaticScope in
> JRuby) should remain static. I don't believe "refine" or refinements
> applied in reopened classes violate that rule.
I see.
>>> * there are very likely many more complexities than illustrated here
>>> that result from the combination of runtime-mutable lexical scoping
>>> structures, concurrency, and method caching.
>>
>> As Yusuke showed in [ruby-core:33535], the current implementation has
>> this problem.
>> The current implementation checks only the (singleton) class of the
>> receiver and the VM version. It should also check the refinements in
>> cref to avoid this problem,
>> but it causes more overhead.
>
> More overhead is always bad if it affects performance globally :)
I thought you might say that, but I did it ;)
>> I might withdraw the proposal of refinement propagation for blocks
>> given to instance_eval,
>> but what do you think of instance_eval without blocks?
>>
>> x.instance_eval("...")
>
> This form is "safe" since the string needs to be parsed and evaluated
> (and provided with a new scope) each time. I worry a bit about the
> inconsistency of having the "" form propagate refinements but the {}
> form not propagating refinements. Perhaps this is a big like the
> duality of constant lookup inside instance_eval?
I think the inconsistency is not a problem because the "" form and the
{} form are already inconsistent as to constant lookup as you say.
>> And, how about to introduce a new method (e.g.,
>> Module#refinement_eval) which copies a given block and make the copy
>> refinement aware?
>> I think blocks are useful to implement DSLs like RSpec.
>>
>> it "should ..." do
>> foo = Foo.new
>> foo.should == ... # Object#should is available only in the block.
>> end
>
> Yes, I agree this is a useful and difficult case to address. One
> possible solution would be making the modification of the block
> explicit and permanent:
>
> p = proc { bar }
> p.refine! BarModifyingExt
> # p from now on has BarModifyingExt's refinements applied to it
It's an interesting idea.
> To be honest, I think it should be a using directive at the file's
> toplevel, so subsequently-parsed blocks know they're going to be
> refined. I believe more and more that "using" needs to be a keyword;
> having yet another "magic" method that can alter parse/execution
> behavior beyond its scope scares me.
I understand your worry. But refinements are still immature in other
aspects than performance, so I'd like to experiment on refinements
before improving performance. Methods are easier than keywords to do
it.
--
Shugo Maeda
Yukihiro Matsumoto
In message "Re: [ruby-core:33568] Re: [Ruby 1.9-Feature#4085][Open] Refinements and nested methods"
on Sat, 4 Dec 2010 21:48:00 +0900, Charles Oliver Nutter <hea...@headius.com> writes:
|For the main cases, I see it working this way:
|
|* Parser sees "using" in a scope
|* Child scopes will now be parsed as though they are refined
|* VCALL, FCALL, CALL in child scopes are instead RVCALL, RFCALL, RCALL
I see this optimization nice for refinement, but it's impossible to
implement local rebinding (a la classbox) that some wanted in this
thread this way.
matz.
Charles Oliver Nutter
On Sat, Dec 4, 2010 at 11:56 AM, Yukihiro Matsumoto <ma...@ruby-lang.org> wrote:
> |For the main cases, I see it working this way:
> |
> |* Parser sees "using" in a scope
> |* Child scopes will now be parsed as though they are refined
> |* VCALL, FCALL, CALL in child scopes are instead RVCALL, RFCALL, RCALL
>
> I see this optimization nice for refinement, but it's impossible to
> implement local rebinding (a la classbox) that some wanted in this
> thread this way.
Yes, and that is a primary reason why I would not include local
rebinding if I were implementing refinements. I don't see how it's
possible to do without adding overhead to every call (to constantly
check if there's a refinement active) and forcing the global cache to
be flushed repeatedly (since a refinement can be applied from any
scope in any thread at any time). In essence, with local rebinding,
all method calls would have to:
* Ping on every call to see if a refinement is active, since it's not
possible to narrow the refinement to only locally-rebound methods.
* If a refinement is active, check to see if any active refinements
affect them. This would at minimum require inspecting the refinement
to see if it includes methods for the target object's class.
* Either only cache refined methods within that one activation or
don't cache methods at all, since other calls shouldn't see refined
methods in the global cache.
* Do all this in a way that is thread-safe, does not modify any
globally-visible state, and only impacts the activation where
refinements are active.
So at a minimum, all calls in the system would have to constantly ping
*additional* some global state. Because any refinement anywhere would
modify that state, any time any thread does a local rebinding, all
calls would have to check to see if they are affected. And refined
code would require at least adding extra checks to all method cache
validations (to ensure what was cached was not from a refinement not
active in the current activation) or require that refined code never
cache (or only cache within that activation, which would only be
useful if those call sites are encountered many times in a single
activation).
For a real-world example of how local rebinding causes problems, look
at Groovy. Groovy allows local rebinding down-thread; i.e. if a
"category" is active, all invocations deeper in that thread must check
the thread-local category to see if they are affected by it. As a
result:
* All calls everywhere have to constantly check thread-local state to
see if a category is active
* Calls down-stream from a category have to check whether the category
affects them
* Calls down-stream from a category that are affected by it have to do
a slow-path uncached method lookup every time
The end result of Categories in Groovy are:
* All performance is affected by them, and fixing it is difficult
* Users don't use them anymore, since the performance of calls
down-stream from the category are much slower than regular calls
* The Groovy team generally regrets that they were added, due to the
long-term effects
They are considered (by most folks I've talked to) to be a mistake,
but now there's no going back. It's interesting to note that the
Groovy approach at least only affects one thread, and there are no
concurrency issues; only the categorized thread sees the change, and
since it's explicitly in thread-local state, there's no impact across
threads. But as in Refinements, the presence of Categories as a
feature means all calls have to be slower. And even worse, as Groovy
has moved forward with more and more optimizations, two things have
made it difficult: open classes and categories. The latter could have
been avoided.
A Refinements implementation that adds overhead to all calls will
damage Ruby (or at least, Ruby's chances of being a high-performance
language) forever.
- Charlie
Yukihiro Matsumoto
I am neutral about local rebinding. It is useful sometimes (as
someone has pointed out in this thread). At the same time, I agree
with you that performance impact is a huge negative factor against
local rebinding. We need more input. Dave, what do you think?
Anyone else?
matz.
In message "Re: [ruby-core:33578] Re: [Ruby 1.9-Feature#4085][Open] Refinements and nested methods"
James Tucker
On Dec 4, 2010, at 5:37 PM, Yukihiro Matsumoto wrote:
> Hi,
>
> I am neutral about local rebinding. It is useful sometimes (as
> someone has pointed out in this thread). At the same time, I agree
> with you that performance impact is a huge negative factor against
> local rebinding. We need more input. Dave, what do you think?
> Anyone else?
>
> matz.
I don't think it's just performance, it's debugging too.
If something fails due to a data error introduced mid-stack due to an unexpected refinement then you have to walk up the stack checking to see what refinements are used. This means I can't use irb with a loaded environment and reflection anymore. I have to use a debugger and walk up frames reflecting on the current state of the world.
At best you can add the refinement to the backtrace (I have no idea if this is done already? - sorry haven't been following in detail). This will only help if the errored refinement method is actually in the stack, if it isn't you have to walk around odd places in the program / libraries being sure that you're clear what refinements are where. I'm struggling to see how debugging this is any easier than debugging monkey patching?
Yehuda Katz
Yehuda Katz
Architect | Strobe
(ph) 718.877.1325
Issue #4085 has been updated by Dave Thomas.
This behavior makes me nervous—I can see arguments for it, but at the same time I can see in leading to problems, particularly in well structured classes where a large set of behaviors is defined in terms of one method. I'm sure the syntax below is wrong, but look at the spirit of it.
module Quux
using FooExt
foo = Foo.new
foo.bar # => FooExt#bar
foo.baz # => Foo#bar
end
module DoubleEach
refine Array do
def each
super do |val|
yield 2*val
end
end
end
end
using DoubleEach
[ 1, 2, 3 ].each {|v| p v } #=> 2, 4, 6
[ 1, 2, 3 ].min #=> 1
That would be surprising to me, as I'd expect the behavior of all the Enumerable methods, which depend on each, to change if I change the behavior of each.
----------------------------------------
----------------------------------------
Shugo Maeda
2010/12/5 Yukihiro Matsumoto <ma...@ruby-lang.org>:
> I am neutral about local rebinding. It is useful sometimes (as
> someone has pointed out in this thread). At the same time, I agree
> with you that performance impact is a huge negative factor against
> local rebinding. We need more input. Dave, what do you think?
> Anyone else?
I believe we should not support local rebinding.
I guess local rebinding is not designed for refinements of classes
which have many clients like built-in classes. With local rebinding,
when we refine a built-in class such as String, we have to take care
carefully not to break existing code. This means that, it's
ridiculous to use the following refinement with local rebinding.
module MathN
refine Fixnum do
def /(other) quo(other) end
end
end
My intention is to bring modularity into monkey patching, not to bring
extensibility. We already have enough extensibility in Ruby.
--
Shugo Maeda
Yehuda Katz
Charles Oliver Nutter
> I think that, for this same reason, `using` should normally not apply
> outside of the immediate lexical scope. I actually believed that this was
> the default behavior, and explained why I thought it was a good idea on my
> blog (http://yehudakatz.com/2010/11/30/ruby-2-0-refinements-in-practice/).
> I can understand the utility of offering the inherited style when desired,
> but I don't think it should be the default. By default, the principle of
> this feature requires that refinements modify zero code outside its lexical
> scope.
> Perhaps a special form of using that would also affect subclasses (for
> example, for the case where Rails wants to expose ActiveSupport into its
> subclasses):
> class ActionController::Base
> using ActiveSupport, inherited: true
> end
Im not sure inheritance is possible to support at all, since classes
don't know about their child classes (officially) and reopening a
class could not easily walk its children and make refinements apply to
them after the fact. Perhaps I missed the discussion about this
feature.
"using" should be applied similar to
"public/private/protected/module_function". In a sense, it's a
"refined" flag on the current scope that specifies that subsequent
scope creations should be "refined" in some way. There's no room for
leaking "private" and friends to definitions outside the lexical
scope, and the same should apply to "using".
By a similar token, "using" isn't impossible to apply after the fact
(i.e. adding a refinement "later" to some scope), but it is
*infeasible*. Because ideally we'd want to be able to flip a bit on
all calls about to be affected by the refinement, we'd need to
guarantee across all implementations that call sites are mutable at
runtime, where they aren't mutable now (or at least aren't
consistently and equivalently mutable). And just as mutating global
state would be bad (like flipping methods "private" in one thread
while they're being used in a another), so would having "using" change
the way a particular call site does its invocation after that site has
been constructed and released into the wild.
I suggest we drop the "pseudo" from "pseduolexical" in the Refinements
specification, and see where that takes us. Anything "pseudo" lexical
is dynamic scoping, which fails multiple performance, concurrency, and
obviousness tests for me.
- Charlie
Charles Oliver Nutter
>> Note that this is just my opinion, and that I seem to be in the
>> minority ;-)
>> I hope that 1.9.x would be stable, but many other committers seem to
>> hope to include new feature in 1.9.x aggressively.
>
> I also hope that 1.9.x would be stable, but I'd like to develop
> aggressively in trunk. I don't think the current design and
> implementation of refinements are mature, but I can't make them mature
> on my own, so I'd like to have help from other committers.
This is what topic branches are for :) I don't believe refinements
should land on trunk, since it's not yet clear whether they should be
included at all. Landing them now, making multiple additional commits
to them, and propagating their changes throughout other
subsystems...all will make it harder to roll back Refinements if it is
decided they shouldn't get into standard Ruby.
I think everyone knows how to check out an alternative branch or
repository :) If they would like to assist you (and you would like
them to assist you) they can do it that way. It shouldn't be done on
mainline, in my opinion, until it has solidified a bit more and it's
certain to be included in an upcoming "standard" Ruby release.
- Charlie
Urabe Shyouhei
> On Sat, Dec 4, 2010 at 6:32 AM, Shugo Maeda <sh...@ruby-lang.org> wrote:
>>> Note that this is just my opinion, and that I seem to be in the
>>> minority ;-)
>>> I hope that 1.9.x would be stable, but many other committers seem to
>>> hope to include new feature in 1.9.x aggressively.
>>
>> I also hope that 1.9.x would be stable, but I'd like to develop
>> aggressively in trunk. I don't think the current design and
>> implementation of refinements are mature, but I can't make them mature
>> on my own, so I'd like to have help from other committers.
>
> This is what topic branches are for :)
No. That idea is not a SVN way. A trunk in SVN is an actively developed edge
branch. In contrast a master branch in Git is a merged collection of topic
branches, thus they are far less active than SVN trunk. So in SVN,
developments goes on the trunk.
> I don't believe refinements
> should land on trunk, since it's not yet clear whether they should be
> included at all.
I think it's all what you tell. You just don't like it, do you?
> Landing them now, making multiple additional commits
> to them, and propagating their changes throughout other
> subsystems...all will make it harder to roll back Refinements if it is
> decided they shouldn't get into standard Ruby.
You say "now" ... Do you believe it gets easier to roll back as time goes? I
don't think so. If it's a matter of time I can agree with you but...
> I think everyone knows how to check out an alternative branch or
> repository :) If they would like to assist you (and you would like
> them to assist you) they can do it that way.
Life is much easier if everyone assisted me testing our ruby_1_8_7 branch
every day. Did you know I'm planning a patchlevel release this month? Have
you compiled it and run tests? So far all who actually assisted me was Luis
Lavena only.
No, I'm not blaming you. It's all as usual. Assistance won't happen until
someone gets some trouble.
> It shouldn't be done on
> mainline, in my opinion, until it has solidified a bit more and it's
> certain to be included in an upcoming "standard" Ruby release.
Define your "a bit more". Otherwise you can reject a new feature as many time
as you want with this exact phrase.
Loren Segal
On 12/6/2010 7:13 AM, Charles Oliver Nutter wrote:
> I suggest we drop the "pseudo" from "pseduolexical" in the Refinements
> specification, and see where that takes us. Anything "pseudo" lexical
> is dynamic scoping, which fails multiple performance, concurrency, and
> obviousness tests for me.
I would have to agree with Yehuda and Charlie on this. Refinements make
far more sense and are far easier to deal with when they are purely
lexical, both on a performance and a conceptual level. Although I'm not
nearly as qualified as Charlie to comment on the technical / performance
issues, it seems as though the entire point of refinements is to allow
*scoped* "monkey patching", and this can only work if there is
absolutely no scope leaking. Supporting inheritance gives way too much
wiggle room to unintentionally leak your scope that it effectively
limits how the feature can be used in real world situations.
I'd say that focusing on a pure lexical scoping for now would make it
far easier to get to a point where the implementation is "correct" and
can be commented on by more people in the community. Then, *if* the
performance / technical issues can be resolved, Yehuda's suggestion of
supporting opt-in inheritance-scoping through a hash argument would
still be on the table-- although a discussion should be had on whether
inheritance support really worth implementing after people are able to
test if a lexical-only implementation is sufficient.
- Loren
mathew
To be honest, I think it should be a using directive at the file's
toplevel, so subsequently-parsed blocks know they're going to be
refined.
Yehuda Katz
Shugo Maeda
2010/12/6 Yehuda Katz <wyc...@gmail.com>:
> I think that, for this same reason, `using` should normally not apply
> outside of the immediate lexical scope. I actually believed that this was
> the default behavior, and explained why I thought it was a good idea on my
> blog (http://yehudakatz.com/2010/11/30/ruby-2-0-refinements-in-practice/).
I agree with you. I added inheritance support for frameworks such as
Rails, but it was wrong at least as the default behavior.
--
Shugo Maeda
Shugo Maeda
2010/12/6 Charles Oliver Nutter <hea...@headius.com>:
>> I also hope that 1.9.x would be stable, but I'd like to develop
>> aggressively in trunk. I don't think the current design and
>> implementation of refinements are mature, but I can't make them mature
>> on my own, so I'd like to have help from other committers.
>
> This is what topic branches are for :) I don't believe refinements
> should land on trunk, since it's not yet clear whether they should be
> included at all. Landing them now, making multiple additional commits
> to them, and propagating their changes throughout other
> subsystems...all will make it harder to roll back Refinements if it is
> decided they shouldn't get into standard Ruby.
What do `other subsystems' mean? libraries?
I think standard libraries should not use refinements until they are
stated as an official feature.
> I think everyone knows how to check out an alternative branch or
> repository :) If they would like to assist you (and you would like
> them to assist you) they can do it that way. It shouldn't be done on
> mainline, in my opinion, until it has solidified a bit more and it's
> certain to be included in an upcoming "standard" Ruby release.
As Shyouhei says, I doubt they can do it that way.
--
Shugo Maeda
Charles Oliver Nutter
> (2010/12/06 21:17), Charles Oliver Nutter wrote:
>> This is what topic branches are for :)
>
> No. That idea is not a SVN way. A trunk in SVN is an actively developed edge
> branch. In contrast a master branch in Git is a merged collection of topic
> branches, thus they are far less active than SVN trunk. So in SVN,
> developments goes on the trunk.
My opinion, from dealing with SVN-based projects in the past:
Experimental development should never go on SVN trunk. Trunk should
only contain features that are intended to eventually get into a
release. I'm not sure Refinements qualifies (yet) as an official path
for Ruby, and so it should stay off trunk until that happens.
The alternative would be to let everyone doing any experiment
whatsoever commit to trunk, and then have to sort out what gets into
future releases and what does not. That does not make sense to me.
>> I don't believe refinements
>> should land on trunk, since it's not yet clear whether they should be
>> included at all.
>
> I think it's all what you tell. You just don't like it, do you?
I like Refinements, once they are "refined" to deal with the issues
brought up in this thread :) I have no objection to them going to MRI
trunk if they have reached the point where they're stable and
acceptable enough to officially be part of Ruby's future.
>> Landing them now, making multiple additional commits
>> to them, and propagating their changes throughout other
>> subsystems...all will make it harder to roll back Refinements if it is
>> decided they shouldn't get into standard Ruby.
>
> You say "now" ... Do you believe it gets easier to roll back as time goes? I
> don't think so. If it's a matter of time I can agree with you but...
It gets harder to roll back as time goes on. Maintaining experimental
features on a separate branch until they're blessed to become part of
Ruby's future would ensure they don't have to be reverted much later.
>> I think everyone knows how to check out an alternative branch or
>> repository :) If they would like to assist you (and you would like
>> them to assist you) they can do it that way.
>
> Life is much easier if everyone assisted me testing our ruby_1_8_7 branch
> every day. Did you know I'm planning a patchlevel release this month? Have
> you compiled it and run tests? So far all who actually assisted me was Luis
> Lavena only.
>
> No, I'm not blaming you. It's all as usual. Assistance won't happen until
> someone gets some trouble.
If I were more familiar with MRI codebase, I might try to help
"refine" Refinements, but unfortunately the best I can do is explain
my opinions and offer suggestions for improvement here on the list. I
can also try to implement the "refined" portions of Refinements in
JRuby.
As for others who *are* familiar with MRI...I don't see why they'd be
any more likely to assist in "refining" refinements if it were on
trunk versus on a branch. If they're interested in the feature,
they'll contribute wherever it is, no?
>> It shouldn't be done on
>> mainline, in my opinion, until it has solidified a bit more and it's
>> certain to be included in an upcoming "standard" Ruby release.
>
> Define your "a bit more". Otherwise you can reject a new feature as many time
> as you want with this exact phrase.
A bit more = once Matz says it should go on trunk (ideally, after
other implementers, ruby-core folks, and key community members agree
it's ready to go to trunk).
No offense is intended by either my commentary or my opinions. I just
would like to see Refinements land *after* these issues have been
sorted out. If there's more I can do to help that happen, please let
me know.
- Charlie
Shugo Maeda
2010/12/7 Charles Oliver Nutter <hea...@headius.com>:
>>> It shouldn't be done on
>>> mainline, in my opinion, until it has solidified a bit more and it's
>>> certain to be included in an upcoming "standard" Ruby release.
>>
>> Define your "a bit more". Otherwise you can reject a new feature as many time
>> as you want with this exact phrase.
>
> A bit more = once Matz says it should go on trunk (ideally, after
> other implementers, ruby-core folks, and key community members agree
> it's ready to go to trunk).
I don't mean to merge refinements into trunk without Matz's permission.
I guess he's still wondering whether we need local rebinding.
--
Shugo Maeda
Urabe Shyouhei
> On Mon, Dec 6, 2010 at 8:48 AM, Urabe Shyouhei <shyo...@ruby-lang.org> wrote:
>> (2010/12/06 21:17), Charles Oliver Nutter wrote:
>>> This is what topic branches are for :)
>>
>> No. That idea is not a SVN way. A trunk in SVN is an actively developed edge
>> branch. In contrast a master branch in Git is a merged collection of topic
>> branches, thus they are far less active than SVN trunk. So in SVN,
>> developments goes on the trunk.
>
> My opinion, from dealing with SVN-based projects in the past:
>
> Experimental development should never go on SVN trunk. Trunk should
> only contain features that are intended to eventually get into a
> release. I'm not sure Refinements qualifies (yet) as an official path
> for Ruby, and so it should stay off trunk until that happens.
>
> The alternative would be to let everyone doing any experiment
> whatsoever commit to trunk, and then have to sort out what gets into
> future releases and what does not. That does not make sense to me.
I think this is the whole point of this issue; what is a trunk. At least
until now, trunk has been what I explained. And you say it should not.
>>> I don't believe refinements
>>> should land on trunk, since it's not yet clear whether they should be
>>> included at all.
>>
>> I think it's all what you tell. You just don't like it, do you?
>
> I like Refinements, once they are "refined" to deal with the issues
> brought up in this thread :) I have no objection to them going to MRI
> trunk if they have reached the point where they're stable and
> acceptable enough to officially be part of Ruby's future.
I see. Sorry for the unnecessary attack.
>>> Landing them now, making multiple additional commits
>>> to them, and propagating their changes throughout other
>>> subsystems...all will make it harder to roll back Refinements if it is
>>> decided they shouldn't get into standard Ruby.
>>
>> You say "now" ... Do you believe it gets easier to roll back as time goes? I
>> don't think so. If it's a matter of time I can agree with you but...
>
> It gets harder to roll back as time goes on. Maintaining experimental
> features on a separate branch until they're blessed to become part of
> Ruby's future would ensure they don't have to be reverted much later.
So we have a consensus here. The difference is I'm expecting a revert so I'd
see it happen sooner to reduce the risk, while you don't want it at all.
Yukihiro Matsumoto
In message "Re: [ruby-core:33610] Re: [Ruby 1.9-Feature#4085][Open] Refinements and nested methods"
on Tue, 7 Dec 2010 16:08:28 +0900, Shugo Maeda <sh...@ruby-lang.org> writes:
|I don't mean to merge refinements into trunk without Matz's permission.
|I guess he's still wondering whether we need local rebinding.
I do not object local rebinding (yet). In fact, I am slightly
negative against it. I am waiting for someone to try persuade me for
local rebinding, despite its potential performance penalty. No one
has ever tried yet.
matz.
Haase, Konstantin
Konstantin
Charles Oliver Nutter
<Konstant...@student.hpi.uni-potsdam.de> wrote:
> Since I explained one use case I'd have for local rebinding: I think not having local rebinding is mostly what we want, local rebinding would mostly cause unwanted side effects and would be a pure horror to debug.
So to clarify (for me) what you are saying: you had a use case for
local rebinding, but you don't think it's worth the side effects, yes?
In other words, a +1 reverted to a 0 or -1?
- Charlie
Haase, Konstantin
Yes. Even ignoring the performance impact, the benefits of local rebinding don't justify the harm it could do on an architectural level.
The only option would be introducing a switch (as proposed for inheritance), but that would get us back to a global performance hit, would it?
One other thing:
At the moment, refinements are not only activated by using, but by refine, too. Wouldn't that mean on one hand, that refine would have to be a keyword too, and on the other hand, that you cannot import refinements at all, if those are purely lexically scoped?
Konstantin