[ruby-core:37585] [Ruby 1.9 - Feature #4085] Refinements and nested methods

[Hiroshi Nakamura]

Issue #4085 has been updated by Hiroshi Nakamura.

Target version changed from 1.9.3 to 1.9.x

----------------------------------------
Feature #4085: Refinements and nested methods
http://redmine.ruby-lang.org/issues/4085

Author: Shugo Maeda
Status: Open
Priority: Normal
Assignee:
Category: core
Target version: 1.9.x

=begin
As I said at RubyConf 2010, I'd like to propose a new features called
"Refinements."

Refinements are similar to Classboxes. However, Refinements doesn't
support local rebinding as mentioned later. In this sense,
Refinements might be more similar to selector namespaces, but I'm not
sure because I have never seen any implementation of selector
namespaces.

In Refinements, a Ruby module is used as a namespace (or classbox) for
class extensions. Such class extensions are called refinements. For
example, the following module refines Fixnum.

module MathN
refine Fixnum do
def /(other) quo(other) end
end
end

Module#refine(klass) takes one argument, which is a class to be
extended. Module#refine also takes a block, where additional or
overriding methods of klass can be defined. In this example, MathN
refines Fixnum so that 1 / 2 returns a rational number (1/2) instead
of an integer 0.

This refinement can be enabled by the method using.

class Foo
using MathN

def foo
p 1 / 2
end
end

f = Foo.new
f.foo #=> (1/2)
p 1 / 2

In this example, the refinement in MathN is enabled in the definition
of Foo. The effective scope of the refinement is the innermost class,
module, or method where using is called; however the refinement is not
enabled before the call of using. If there is no such class, module,
or method, then the effective scope is the file where using is called.
Note that refinements are pseudo-lexically scoped. For example,
foo.baz prints not "FooExt#bar" but "Foo#bar" in the following code:

class Foo
def bar
puts "Foo#bar"
end

def baz
bar
end
end

module FooExt
refine Foo do
def bar
puts "FooExt#bar"
end
end
end

module Quux
using FooExt

foo = Foo.new
foo.bar # => FooExt#bar
foo.baz # => Foo#bar
end

Refinements are also enabled in reopened definitions of classes using
refinements and definitions of their subclasses, so they are
*pseudo*-lexically scoped.

class Foo
using MathN
end

class Foo
# MathN is enabled in a reopened definition.
p 1 / 2 #=> (1/2)
end

class Bar < Foo
# MathN is enabled in a subclass definition.
p 1 / 2 #=> (1/2)
end

If a module or class is using refinements, they are enabled in
module_eval, class_eval, and instance_eval if the receiver is the
class or module, or an instance of the class.

module A
using MathN
end
class B
using MathN
end
MathN.module_eval do
p 1 / 2 #=> (1/2)
end
A.module_eval do
p 1 / 2 #=> (1/2)
end
B.class_eval do
p 1 / 2 #=> (1/2)
end
B.new.instance_eval do
p 1 / 2 #=> (1/2)
end

Besides refinements, I'd like to propose new behavior of nested methods.
Currently, the scope of a nested method is not closed in the outer method.

def foo
def bar
puts "bar"
end
bar
end
foo #=> bar
bar #=> bar

In Ruby, there are no functions, but only methods. So there are no
right places where nested methods are defined. However, if
refinements are introduced, a refinement enabled only in the outer
method would be the right place. For example, the above code is
almost equivalent to the following code:

def foo
klass = self.class
m = Module.new {
refine klass do
def bar
puts "bar"
end
end
}
using m
bar
end
foo #=> bar
bar #=> NoMethodError

The attached patch is based on SVN trunk r29837.
=end

--
http://redmine.ruby-lang.org

[nahi]

Issue #4085 has been updated by nahi.

Assignee set to shugo

----------------------------------------
Feature #4085: Refinements and nested methods

https://bugs.ruby-lang.org/issues/4085#change-24747

Author: shugo
Status: Open
Priority: Normal
Assignee: shugo
Category: core
Target version: 2.0.0

--
http://bugs.ruby-lang.org/

[jaffa62 (jaffa wify)]

Issue #4085 has been updated by jaffa62 (jaffa wify).


Data movement is typically from one place in memory to another. Sometimes it involves moving data between memory and registers which enable high speed data access in the CPU. Thanks.
Regards,
http://www.personalstatementwriters.com/services/

----------------------------------------
Feature #4085: Refinements and nested methods

https://bugs.ruby-lang.org/issues/4085#change-27846

Author: shugo (Shugo Maeda)
Status: Assigned
Priority: Normal
Assignee: shugo (Shugo Maeda)

[shugo (Shugo Maeda)]

Issue #4085 has been updated by shugo (Shugo Maeda).

Assignee changed from shugo (Shugo Maeda) to ko1 (Koichi Sasada)

Matz said the only problem is performance before, so I assign this issue to ko1.

----------------------------------------
Feature #4085: Refinements and nested methods

https://bugs.ruby-lang.org/issues/4085#change-28285

Author: shugo (Shugo Maeda)
Status: Assigned
Priority: Normal

Assignee: ko1 (Koichi Sasada)

[shugo (Shugo Maeda)]

Issue #4085 has been updated by shugo (Shugo Maeda).

shugo (Shugo Maeda) wrote:
> Matz said the only problem is performance before, so I assign this issue to ko1.

I've committed refinements in r36596 with Matz's permission. However,
it's just an experimental feature, and may be reverted before the release of 2.0.
Please try it, and give us feedback.

----------------------------------------
Feature #4085: Refinements and nested methods

https://bugs.ruby-lang.org/issues/4085#change-28594

[trans (Thomas Sawyer)]

Issue #4085 has been updated by trans (Thomas Sawyer).


=begin
One question I have is how this would work with something like Facets, including its ability to cherry pick methods.

So lets say we have `facets/object/foo` and `facets/object/bar`.

# facets/object/foo.rb
module Facets
refine Object
def foo
...
end
end
end

# lib/facets/object/bar.rb
module Facets
refine Object
def bar
...
end
end
end

So now one could include "Facets" in their application's namespace?

require 'facets/object/foo'
require 'facets/object/bar'

module MyApp
include Facets
end

Would that work?
=end

----------------------------------------
Feature #4085: Refinements and nested methods

https://bugs.ruby-lang.org/issues/4085#change-28788

[Eregon (Benoit Daloze)]

Issue #4085 has been updated by Eregon (Benoit Daloze).


trans (Thomas Sawyer) wrote:
> =begin

> module MyApp
> include Facets
> end
>
> Would that work?
> =end

Yes, except you need to use `using` and not `include` in `module MyApp`.

----------------------------------------
Feature #4085: Refinements and nested methods

https://bugs.ruby-lang.org/issues/4085#change-28792

[ko1 (Koichi Sasada)]

Issue #4085 has been updated by ko1 (Koichi Sasada).

Assignee changed from ko1 (Koichi Sasada) to shugo (Shugo Maeda)

Any problem now?

----------------------------------------
Feature #4085: Refinements and nested methods

https://bugs.ruby-lang.org/issues/4085#change-29656

Author: shugo (Shugo Maeda)
Status: Assigned
Priority: Normal

Assignee: shugo (Shugo Maeda)

[shugo (Shugo Maeda)]

Issue #4085 has been updated by shugo (Shugo Maeda).

Assignee changed from shugo (Shugo Maeda) to matz (Yukihiro Matsumoto)

ko1 (Koichi Sasada) wrote:
> Any problem now?

I like the current behavior of Refinements, so please give me feedback if you don't like it.

Some objections to Refinements are summarized below:

* Refinements are too complex.
* Refinements should support local rebinding.
* using should be a keyword.
* refine should be a keyword.

Matz should decide whether Refinements should be included in Ruby 2.0, but I'm not sure whether he has tried Refinements or not.

----------------------------------------
Feature #4085: Refinements and nested methods

https://bugs.ruby-lang.org/issues/4085#change-31846

Author: shugo (Shugo Maeda)
Status: Assigned
Priority: Normal

Assignee: matz (Yukihiro Matsumoto)

[headius (Charles Nutter)]

Issue #4085 has been updated by headius (Charles Nutter).


I would like to request time to implement refinements as they stand today in JRuby.

We have not had much time to look into how refinements will affect the way we optimize Ruby code, structure class hierarchies, and so on. The impact could be severe, or it could be minimal. I am worried because I don't know, and because up until today (Matz's keynote) I thought they were still questionable for 2.0.0.

So...we will implement refinements in JRuby (possibly on a branch) as soon as possible...within the next month for sure. I hope if we run into issues with the currently-specified behavior there will be time to make changes.

----------------------------------------
Feature #4085: Refinements and nested methods

https://bugs.ruby-lang.org/issues/4085#change-32195

[shugo (Shugo Maeda)]

Issue #4085 has been updated by shugo (Shugo Maeda).

headius (Charles Nutter) wrote:
> I would like to request time to implement refinements as they stand today in JRuby.
>
> We have not had much time to look into how refinements will affect the way we optimize Ruby code, structure class hierarchies, and so on. The impact could be severe, or it could be minimal. I am worried because I don't know, and because up until today (Matz's keynote) I thought they were still questionable for 2.0.0.
>
> So...we will implement refinements in JRuby (possibly on a branch) as soon as possible...within the next month for sure. I hope if we run into issues with the currently-specified behavior there will be time to make changes.

Thanks for your consideration of Refinements.
Please inform me if the current behavior of Refinements is hard to implement in JRuby.

----------------------------------------
Feature #4085: Refinements and nested methods

https://bugs.ruby-lang.org/issues/4085#change-32203

[headius (Charles Nutter)]

Issue #4085 has been updated by headius (Charles Nutter).


Ok, early notes from writing up specs and starting to explore an implementation for JRuby.

* Refinements in modules and class hierarchies does not seem like a problem to me yet.
* Refinements are "used" in temporal order...methods added before "using" won't see refinements, refinements added after "using" won't be applied. I think this is a good thing, since it allows us to have a one-shot flag for refinements on methods at definition time.
* I notice 2.0.0 does not appear to have a performance impact from refinements in a simple case. No idea how refinements interact with cache invalidation yet.

Now, the bad:

* Months ago when the original proposal came out, I expressed my concern about refinements applying to module_eval and friends. I still strongly object to this behavior.

The problems I enumerated then still apply. Because module/class_eval can take any arbitrary block, including blocks of code from distant locations that do not know anything about used refinements, all calls in all blocks would need to check for refinements every time. This is the simple performance/optimization concern. MRI's invalidation mechanisms are simple enough that it may not affect MRI right now...but if invalidation mechanisms like those in JRuby are ever desired in MRI, I don't think this feature can exist.

It is also a GROSS security issue on the scale of block-as-binding. If a library can force code within a block to make completely different calls than the author intended, you're going to cause terrible confusion. Say I write some code in a block and pass it to a badly-written or malicious receiver. That receiver module_eval's my block in a refined scope, changing one or more of the calls I wrote. Suddenly my code, tested in my environment, can behave completely differently just by being passed as a block. If there are errors, they'll be errors reporting my code calling some other method I never intended.

Look at this code and tell me what method it calls:

def my_code(string)
your_code { string.to_i(16) }
end

The answer is that you can't tell me what method it is calling because the your_code method can basically force it to call anything.

This is actually *worse* than monkey patching, because after the call to your_code has completed, I have no evidence that a refinement was active. I need to go spelunking in your_code to figure out why my code didn't behave the way it should.

This is dynamic application of refinements, which has been hotly debated and which I *thought* was supposed to be removed. I assume it has been left in because it is required to apply refinements "magically" to all-block code like rspec. I do not see this as an excuse to introduce such an unpredictable feature.

Thoughts?

I am adding rubyspecs and continuing my research.

----------------------------------------
Feature #4085: Refinements and nested methods

https://bugs.ruby-lang.org/issues/4085#change-32244

[headius (Charles Nutter)]

Issue #4085 has been updated by headius (Charles Nutter).

Adding a couple additional positive notes:

* For now, I don't see a reason for using and refine to be keywords.

using and refine basically just add some additional state to modules and methods.

For refine:

* add to the module a mapping from refined class/module to the refinements that should be applied
* mark the containing module as having refinements active

For using:

* copy to the target module all currently active module => refinement mappings.
* tag methods defined after the using call to indicate that they should use refinement lookup

I think this is all doable without major alteration of most VMs caching, jitting, optimization mechanisms.

module_eval and class_eval are a different issue. There's no indication ahead of time that calls within those scopes need to use refinements, so you have to check constantly and enlist in refinement lookup unconditionally.

----------------------------------------
Feature #4085: Refinements and nested methods

https://bugs.ruby-lang.org/issues/4085#change-32245

[shugo (Shugo Maeda)]

Issue #4085 has been updated by shugo (Shugo Maeda).


headius (Charles Nutter) wrote:

> Ok, early notes from writing up specs and starting to explore an implementation for JRuby.

Thanks.

> * Refinements in modules and class hierarchies does not seem like a problem to me yet.
> * Refinements are "used" in temporal order...methods added before "using" won't see refinements, refinements added after "using" won't be applied. I think this is a good thing, since it allows us to have a one-shot flag for refinements on methods at definition time.

The current behavior is mainly for an implementation reason, but Matz and ko1 seem not to like it:(

> * Months ago when the original proposal came out, I expressed my concern about refinements applying to module_eval and friends. I still strongly object to this behavior.

I also wonder whether module_eval with blocks should be affected by refinements or not, but I think module_eval with strings (e.g., M.module_eval("C.new.foo")) has no problem, right?

> This is dynamic application of refinements, which has been hotly debated and which I *thought* was supposed to be removed. I assume it has been left in because it is required to apply refinements "magically" to all-block code like rspec. I do not see this as an excuse to introduce such an unpredictable feature.

instance_eval and module_eval themselves have the same problem because they change self "magically".
At first, I thought they are evil, but they are popular now.
I'd like to ask Matz's opinion.

----------------------------------------
Feature #4085: Refinements and nested methods

https://bugs.ruby-lang.org/issues/4085#change-32261

[Charles Oliver Nutter]

On Fri, Nov 2, 2012 at 7:12 PM, shugo (Shugo Maeda)
<red...@ruby-lang.org> wrote:

> headius (Charles Nutter) wrote:
> > * Refinements in modules and class hierarchies does not seem like a problem to me yet.
> > * Refinements are "used" in temporal order...methods added before "using" won't see refinements, refinements added after "using" won't be applied. I think this is a good thing, since it allows us to have a one-shot flag for refinements on methods at definition time.
>
> The current behavior is mainly for an implementation reason, but Matz and ko1 seem not to like it:(

I commented on ko1's bug. I see using a bit like visibility changes,
only affecting methods defined later on.

I understand the implementation reason as well...in order to limit the
damage of refinements, your impl flags methods as having refinements
active. For optimization purposes, that means we know at definition
time whether we need to handle refinements at all.

> > * Months ago when the original proposal came out, I expressed my concern about refinements applying to module_eval and friends. I still strongly object to this behavior.
>
> I also wonder whether module_eval with blocks should be affected by refinements or not, but I think module_eval with strings (e.g., M.module_eval("C.new.foo")) has no problem, right?

String eval would not be a problem, that is correct. We would be able
to see at eval time that the target module has refinements active.

> > This is dynamic application of refinements, which has been hotly debated and which I *thought* was supposed to be removed. I assume it has been left in because it is required to apply refinements "magically" to all-block code like rspec. I do not see this as an excuse to introduce such an unpredictable feature.
>
> instance_eval and module_eval themselves have the same problem because they change self "magically".
> At first, I thought they are evil, but they are popular now.
> I'd like to ask Matz's opinion.

Yes, module_eval, class_eval, and instance_eval are all problematic
because of the self changing, but module_eval and class_eval are
especially bad if they force refinements on code that doesn't know
about them.

I am starting to see some intractable problems with refinements, unfortunately.

In order to avoid having every call in the system check for
refinements, they are applied in evaluation order. However, this means
that the load order of scripts can now completely change which methods
get called. For example...

a.rb:

class Foo
def go(obj)
obj.something
end
end

b.rb:

class Foo
using Baz # refines the "something" call
end

If the files are loaded in the order a, b, no refinements are applied
to the something call. If b is loaded before a, refinements are
applied to the something call. No other features in Ruby are so
sensitive to load order (other than those that introspect classes and
methods, obviously).

The alternative is to have refinements not be applied temporally.
However this means every call in the system needs to check for
refinements every time. Given the complexity of method lookup in Ruby
today, adding refinements to that process seems like a terrible idea.

In order to cache a method call in the presence of refinements, we
need to track all of the following:

1. whether any refinements are active
2. whether there are refinements that affect the class of the target
of the method call
3. whether refinements that affect the target class redefine the target method

If any of these change at any time, we need to invalidate the cache.
This is on top of all the information we need to do to cache methods
normally.

At this point I would not vote for refinements to be included in Ruby
2.0. I feel like there are far too many edge cases and implementation
concerns.

I will continue my investigation.

[Charles Oliver Nutter]

More thoughts...

I could get behind refinements if using were a keyword (so we could
tell at parse time refinements would be active) and purely lexical.
The following features of the current implementation would have to be
removed:

* refinements cascading down class hierarchies
* refinements affecting code in module_eval'ed blocks

If using became a keyword and purely lexical, the following examples
would be fine:

rails_thing.rb:

class Foo
using ActiveRecord::SomeExt
...
end

rspec_thing.rb:

using RSpec

describe 'Refinements' do
it 'should be purely lexical' do
...

If refinements can affect code outside the lexical scope where they
are activated, I believe it will be confusing, potentially dangerous,
very hard to debug, and potentially difficult or impossible to
implement without slowing all of Ruby down.

[headius (Charles Nutter)]

Issue #4085 has been updated by headius (Charles Nutter).


Discussed refinements a bit with Matz + ko1 plus a_matsuda and others at lunch...here's summary.

The using method currently touches cref, making it largely scoped like constants. If refinement lookup at a call site proceeds the same way, it would simplify things in my head quite a bit.

Benefits to having refinement lookup follow cref:

* No frame field is necessary
* Easier-to-understand behavior, since the lookup would mimic constant lookup (lexical then hierarchical)

There are down sides:

* module_eval cases would not be possible, since like constant lookup refinements cannot be injected into a scope after the fact
* It is not purely lexical, as I had hoped. However, I recognize that having to "using" in every scope where you want refinements to be active would be painful.
* It may need a global invalidation guard, like constants. It may not, though, since refinements are applied only once, at definition time; so after caching once we may have everything we need for future invalidation.

One concern of mine was avoiding refinement checks at all call sites. With cref-based "using" there's no way to determine at parse time which methods might need refinements. We can determine it at definition time, but the AST is already parsed at that point, so we'd have to rewrite it to have "refined calls" instead of normal calls, or something similar. This also impacts JRuby's ahead-of-time compilation support...we do not have the luxury of running the target code for precompilation, so we can't see if there are refinements active.

For normal runtime jitted code, rewriting the AST or adding a "refined" flag to call nodes may be feasible. Calls can then use a refined call site only in the cases where refinements are active.

For AOT mode, I think the best I can do is to have a flag on the call sites to indicate if refinements are active. If there's no refinements for the first call, there will be no refinements for any future calls, and we can turn that logic off. It would still require that boolean check every time...I don't see a way to eliminate that. (Side note: on invokedynamic, that boolean check would end up free, so it may not be a huge deal long term).

I will attempt to prototype pure cref-based refinement lookup over the next week.

----------------------------------------
Feature #4085: Refinements and nested methods

https://bugs.ruby-lang.org/issues/4085#change-32309

[SASADA Koichi]

I have another implementation idea, using "prepend".
I also try it next week.
(I can't explain this idea in English, so I will write code).

// SASADA Koichi at atdot dot net

[Akira Matsuda]

Playing with refinements after talking with Charlie, I found
refinements are not very much attractive feature without module_eval.
I think I do understand the down sides Charlie described.
And I'm sure people will abuse this refinements + module_eval
technique if they are allowed to, because it just looks cool.

So, I guess the point is whether to allow users to write code like this or not;

module A
refine Fixnum do
def +(o); self * o; end
end
end

class C
def foo(&b)
Module.new { using A }.module_eval &b
end
end

C.new.foo { p 2 + 3 }
#=> 6

My personal opinion is, still, as a library / framework author, I'm
sort of anticipating to see such future that people are abusing this.
People may find good practices to control the power somehow, then it
might definitely push the DSL culture up to the absolute next level,
which must be totally exciting.

--
Akira Matsuda<ron...@dio.jp>

[Yukihiro Matsumoto]

Hi,

In message "Re: [ruby-core:48828] Re: [ruby-trunk - Feature #4085] Refinements and nested methods"

on Sun, 4 Nov 2012 08:42:11 +0900, Akira Matsuda <ron...@dio.jp> writes:
|
|Playing with refinements after talking with Charlie, I found
|refinements are not very much attractive feature without module_eval.
|I think I do understand the down sides Charlie described.
|And I'm sure people will abuse this refinements + module_eval
|technique if they are allowed to, because it just looks cool.

I agree. Even though there's chance to make program behavior
unpredictable, extending behavior in block with refinements would
enhance the possibility of expressiveness. ActiveRecord example is
one of them. As far as I understand, Charlie came up with an idea to
resolve refinements at the first method invocation. If the idea is a
right one, we can allow such cases.

But I am not sure allowing refinement application in blocks should be
done by module_eval or not. Generating anonymous module for every
block execution sound inefficient.

matz.

[shugo (Shugo Maeda)]

Issue #4085 has been updated by shugo (Shugo Maeda).


headius (Charles Nutter) wrote:

> Discussed refinements a bit with Matz + ko1 plus a_matsuda and others at lunch...here's summary.

Thanks for the summary.

The current implementation is cref-based because I came up with the idea inspired by pseudo lexical scope of constants. However, I admit that the current implementation has some issues, and don't stick to it.

I'd like to see your implementation and ko1's prepend-based implementation.

----------------------------------------

Feature #4085: Refinements and nested methods

https://bugs.ruby-lang.org/issues/4085#change-32354

http://bugs.ruby-lang.org/

[shugo (Shugo Maeda)]

Issue #4085 has been updated by shugo (Shugo Maeda).

matz wrote:
> But I am not sure allowing refinement application in blocks should be
> done by module_eval or not. Generating anonymous module for every
> block execution sound inefficient.

Such an anonymous module may be reused in a_matsuda's cases, but method caching may be a worse problem. In the current implementation, using invalidates the global method cache and module_eval invokes using internally, which means that the global method cache is invalidated for each call of module_eval.

----------------------------------------
Feature #4085: Refinements and nested methods

https://bugs.ruby-lang.org/issues/4085#change-32357

[Charles Oliver Nutter]

On Sun, Nov 4, 2012 at 8:50 PM, shugo (Shugo Maeda)
<red...@ruby-lang.org> wrote:
> Such an anonymous module may be reused in a_matsuda's cases, but method caching may be a worse problem. In the current implementation, using invalidates the global method cache and module_eval invokes using internally, which means that the global method cache is invalidated for each call of module_eval.

Unless an implementation can be found that doesn't do this, I don't
see how the module_eval feature can be supported. It seems totally
unreasonable to me that the global method cache would be invalidated
in this way.

[headius (Charles Nutter)]

Issue #4085 has been updated by headius (Charles Nutter).

shugo (Shugo Maeda) wrote:
> matz wrote:
> > But I am not sure allowing refinement application in blocks should be
> > done by module_eval or not. Generating anonymous module for every
> > block execution sound inefficient.
>

> Such an anonymous module may be reused in a_matsuda's cases, but method caching may be a worse problem. In the current implementation, using invalidates the global method cache and module_eval invokes using internally, which means that the global method cache is invalidated for each call of module_eval.

To me, this means the module_eval feature absolutely cannot be supported unless a new implementation can be found. No normal feature of Ruby should force full cache invalidation globally.

----------------------------------------
Feature #4085: Refinements and nested methods

https://bugs.ruby-lang.org/issues/4085#change-32365

[matz (Yukihiro Matsumoto)]

Issue #4085 has been updated by matz (Yukihiro Matsumoto).


I agree with Charles here. Avoiding global cache invalidation is priority one.
At the same time, I want a way to enable refinement to given block.

Matz.

----------------------------------------
Feature #4085: Refinements and nested methods

https://bugs.ruby-lang.org/issues/4085#change-32366

[headius (Charles Nutter)]

Issue #4085 has been updated by headius (Charles Nutter).


I currently don't see any way to support enabling refinements in blocks without a global, unavoidable impact to all call sites.

If refinements are applied lexically, then a block suddenly having refinements active does not make sense, since the code in the block is not lexically surrounded by a scope with active refinements.

If refinements are applied based on the cref module, then a block still can't see them because cref for the block is determined lexically. This was explicitly done to allow constant lookup to always proceed based on lexical scopes.

Formally, the clearest way for refinements to work is for you to explicitly opt into them everywhere you want to use them. A couple thoughts about different forms:

1. using as a scope-opening keyword like "class" or "module":

using RSpec
describe "Some Spec" do
it "does some RSpec things" do # "it" comes from refinements
'foo'.should == 'foo' # "should" comes from refinements
end
end
end

This is clear to the user and the parser that all calls downstream from the "using" must consider refinements. They would see the refinements based on cref (or something similar) because of the explicit nesting of scopes.

2. "using" as a pseudo-keyword, affecting lexical scopes and code following "using"

using RSpec

describe "Some Spec" do
it "does some RSpec things" do # "it" comes from refinements
'foo'.should == 'foo' # "should" comes from refinements
end
end

In this case, the parser could still treat all calls following "using" as needing refinements. This is less ideal, because it force us to treat all "using" calls everywhere in the system as potentially triggering refined call sites. It also makes it impossible to detect if "using" is ever aliased or sent. If such features are expected, we'd be back to treating all calls as refined all the time.

Honestly, I really believe that allowing refinements to affect scopes that have no "using" anywhere in sight is going to be really confusing, and it has the added pain of making efficient call site caching hard or impossible.

----------------------------------------
Feature #4085: Refinements and nested methods

https://bugs.ruby-lang.org/issues/4085#change-32371

[shugo (Shugo Maeda)]

Issue #4085 has been updated by shugo (Shugo Maeda).

matz (Yukihiro Matsumoto) wrote:
> I agree with Charles here. Avoiding global cache invalidation is priority one.
> At the same time, I want a way to enable refinement to given block.
>
> Matz.

At first, I expected using is called only at an early stage of a program execution.
However, this expectation doesn't hold true if module_eval is affected by refinements.

I guess invalidation of the global method cache can be avoided if a refinement table (nd_refinements) and global cache entries have a "refinement version". It's enough to increment the refinement version when the refinement table is changed.

----------------------------------------
Feature #4085: Refinements and nested methods

https://bugs.ruby-lang.org/issues/4085#change-32374

[shugo (Shugo Maeda)]

Issue #4085 has been updated by shugo (Shugo Maeda).

headius (Charles Nutter) wrote:
> I currently don't see any way to support enabling refinements in blocks without a global, unavoidable impact to all call sites.
>
> If refinements are applied lexically, then a block suddenly having refinements active does not make sense, since the code in the block is not lexically surrounded by a scope with active refinements.
>
> If refinements are applied based on the cref module, then a block still can't see them because cref for the block is determined lexically. This was explicitly done to allow constant lookup to always proceed based on lexical scopes.

In Ruby 1.9, a new cref node with the special flag (NODE_FL_CREF_PUSHED_BY_EVAL) is pushed by module_eval. The cref node is used to determine where a method is defined by def, and is skipped for a constant lookup.

Is it hard to implement JRuby's cref equivalent (StaticScope?) in the same way?

----------------------------------------
Feature #4085: Refinements and nested methods

https://bugs.ruby-lang.org/issues/4085#change-32375

[SASADA Koichi]

(2012/11/04 5:57), SASADA Koichi wrote:
> I have another implementation idea, using "prepend".

Nobu taught me that this approach is not work.

[Aaron Patterson]

On Mon, Nov 05, 2012 at 02:17:49PM +0900, headius (Charles Nutter) wrote:
>
> Issue #4085 has been updated by headius (Charles Nutter).
>
>
> I currently don't see any way to support enabling refinements in blocks without a global, unavoidable impact to all call sites.
>
> If refinements are applied lexically, then a block suddenly having refinements active does not make sense, since the code in the block is not lexically surrounded by a scope with active refinements.
>
> If refinements are applied based on the cref module, then a block still can't see them because cref for the block is determined lexically. This was explicitly done to allow constant lookup to always proceed based on lexical scopes.
>
> Formally, the clearest way for refinements to work is for you to explicitly opt into them everywhere you want to use them. A couple thoughts about different forms:
>
> 1. using as a scope-opening keyword like "class" or "module":
>
> using RSpec
> describe "Some Spec" do
> it "does some RSpec things" do # "it" comes from refinements
> 'foo'.should == 'foo' # "should" comes from refinements
> end
> end
> end
>
> This is clear to the user and the parser that all calls downstream from the "using" must consider refinements. They would see the refinements based on cref (or something similar) because of the explicit nesting of scopes.
>
> 2. "using" as a pseudo-keyword, affecting lexical scopes and code following "using"
>
> using RSpec
>
> describe "Some Spec" do
> it "does some RSpec things" do # "it" comes from refinements
> 'foo'.should == 'foo' # "should" comes from refinements
> end
> end
>
> In this case, the parser could still treat all calls following "using" as needing refinements. This is less ideal, because it force us to treat all "using" calls everywhere in the system as potentially triggering refined call sites. It also makes it impossible to detect if "using" is ever aliased or sent. If such features are expected, we'd be back to treating all calls as refined all the time.
>
> Honestly, I really believe that allowing refinements to affect scopes that have no "using" anywhere in sight is going to be really confusing, and it has the added pain of making efficient call site caching hard or impossible.

If I'm understanding you correctly, the thing I don't like about
explicitly requiring people to add "using" means that we can't remove
the monkey patches from Rails without breaking everything.

For example:

class MyModel < ActiveRecord::Base
def foo
# constantize comes from Rails monkey patches on String
"somestring".constantize
end
end

Ideally, in the Rails source code, I would just add "using StringExt" to
the ActiveRecord::Base class, and the existing user code would Just
Work. If existing Rails users are required to add the "using" word to
all of their code, then there is no way we (the rails team) can remove
the monkey patches and remain backwards compatible.

Being able to remove monkey patches and remain backwards compatible is
the number one most important thing I want from refinements.

--
Aaron Patterson
http://tenderlovemaking.com/

[shugo (Shugo Maeda)]

Issue #4085 has been updated by shugo (Shugo Maeda).

Assignee changed from matz (Yukihiro Matsumoto) to shugo (Shugo Maeda)

shugo (Shugo Maeda) wrote:
> At first, I expected using is called only at an early stage of a program execution.
> However, this expectation doesn't hold true if module_eval is affected by refinements.
>
> I guess invalidation of the global method cache can be avoided if a refinement table (nd_refinements) and global cache entries have a "refinement version". It's enough to increment the refinement version when the refinement table is changed.

It seems that singleton method definitions and Kernel#extend also invalidate the entire global method cache.
Is it intended?

The current implementation of module_eval doesn't invalidate the global method cache if no refinement is used in the receiver module. Do you want module_eval not to invalidate the global method cache even if refinements are used, Matz?

I wonder which of singleton method definitions and module_eval with refinements are often used in real-world applications.

----------------------------------------
Feature #4085: Refinements and nested methods

https://bugs.ruby-lang.org/issues/4085#change-32530

Author: shugo (Shugo Maeda)
Status: Assigned
Priority: Normal

Assignee: shugo (Shugo Maeda)

[Rodrigo Rosenfeld Rosas]

I agree with Charles here. I'd love to see Rails moving to a more
explicit approach instead of monkey patching core classes.

It is hard for someone new to Rails to understand what is the Ruby
behavior and what has been changed by ActiveSupport. Once you force
users to include the "using" whenever they're using some AS extension it
would become clear to understand where some methods came from.

I think Rails should change its mindset with regards to all "magic" it
has been providing from the beginning and become closer to Ruby.

It already started doing so by deprecating dynamic finders in AR, which
was a good start. I'd love to see more "magic" to disappear from Rails
as code is easier to understand once it is more explicit about where all
magic is coming from. Rails 4 could be a great opportunity for a new
paradigm like this. So, I'd also suggest you to take the chance to
change the behavior of constantize to be the same as const_get as you
have requested for feedback on Rails-core mailing list.

We should work to reduce the gap between Rails and Ruby and try to make
Rails just a web framework, keeping Ruby classes unpatched.

Cheers,
Rodrigo.

[headius (Charles Nutter)]

Issue #4085 has been updated by headius (Charles Nutter).

Well, I have some bad news.

I have spent some time trying to find a reasonable way to implement refinements in JRuby, and without reducing the feature set it's simply not possible to do without global (and sometimes terrible) impact to general application performance.

At this point I'm largely of the opinion that pure-lexical, syntactical refinements are the way to go. And there are several reasons for it:

* Dynamic refinements have far-reaching impact, both peformance-wise and "understandability" of the code. It's both hard (or impossible) to optimize and hard (or impossible) to reason about.
* Syntactic/lexical refinements are nearly self-explanatory in code. Currently, when you look at a piece of code with method calls (and constant lookup, as well), you can reason about it in terms of the enclosing classes/modules and basically know what it will call. Monkeypatching can obscure the eventual method called, but every piece of code everywhere will still call the same method. With dynamic refinements, it's impossible to look at a piece of code and the classes involved and know what method will be called, because it can change arbitrarily at runtime.
* The current refinements are poorly specified and have only been explored in a very limited way. This is of *great* concern to me. There are a huge number of potential edge cases, ranging from abuse of the module_eval feature to reopening existing refinements (which essentially just makes monkeypatching harder to discover), and implementation-wise there has not been enough exploration of how it will affect current and future performance of Ruby. We need more time to define what refinements should do in terms of what users want and what's reasonable from a language-design perspective.

I would support the idea of pure lexical/syntactic refinements of the following form, but I don't see a way to support the features of the current refinements implementation reasonably on any VM.

module Foo
class Quux
using Bar, Baz
def blah
'abc'.efg # refined
end
end

def blah2
'asdf'.qwer # unrefined
end
end
end

----------------------------------------
Feature #4085: Refinements and nested methods

https://bugs.ruby-lang.org/issues/4085#change-32829

[headius (Charles Nutter)]

Issue #4085 has been updated by headius (Charles Nutter).

Replies to recent comments:

Aaron: I totally understand the use case, and I support it. Unfortunately I don't feel like the use case and the current feature set have been aligned properly without major impact to unrefined code. There's a certain realism we need here...the feature itself may be useful, but drastically altering the way Ruby does method lookup is a very blunt way to add it. Refinements as they are currently implemented in trunk add complexity well beyond simply localizing monkey-patches, and I would argue that that complexity is a bigger risk than monkey patches ever were. This is of course ignoring the fact that it may be impossible to implement refinements efficiently given the current feature-set.

Rodrigo: You agree with me, so there's not a lot to say :-) I too would like to see Rails move away from magically patching classes. For example, the following code is really no worse then the monkey-patched version:

class Foo
include CamelizeString
def bar(str)
camelize(str)
end
end

versus

class Foo
def bar(str)
str.camelize
end
end

We're talking about a difference of a single character here, but the non-monkeypatched code is actually clearer (you know to look in CamelizeString for the camelize logic) and less invasive (only code that calls CamelizeString#camelize will hit that code). This may be a bit of a paradigm shift for Rails, but I think that's exactly what's needed to make monkey-patching GO AWAY rather than trying to find cute ways to localize monkeypatching.

----------------------------------------
Feature #4085: Refinements and nested methods

https://bugs.ruby-lang.org/issues/4085#change-32830

[shugo (Shugo Maeda)]

Issue #4085 has been updated by shugo (Shugo Maeda).


headius (Charles Nutter) wrote:

> Well, I have some bad news.
>
> I have spent some time trying to find a reasonable way to implement refinements in JRuby, and without reducing the feature set it's simply not possible to do without global (and sometimes terrible) impact to general application performance.

ko1 came up with a new idea to implement refinements without impact to applications which don't use refinements.
The basic idea is:

* When a method is defined in a refinement (a module given as self in a block of Module#refine), add a method entry with the special type such as VM_METHOD_TYPE_REFINED to the class to be refined. If the original class has a method with the same name, the original method is linked to the new method entry with VM_METHOD_TYPE_REFINED.
* If and only if a method entry with VM_METHOD_TYPE_REFINED is found in a method invocation, search methods of refinements. However, even if a method is found in refinements, don't store the method of refinements in an inline method cache. Instead, store the method entry with VM_METHOD_TYPE_REFINED. (So it may be better to have a cache dedicated to refinements) If no method is found in refinements, use the original method entry linked to the method entry with VM_METHOD_TYPE_REFINED.

What do you think of this idea?

----------------------------------------
Feature #4085: Refinements and nested methods

https://bugs.ruby-lang.org/issues/4085#change-32836

[jballanc (Joshua Ballanco)]

Issue #4085 has been updated by jballanc (Joshua Ballanco).


Perhaps I am missing something, but for the case that Aaron points out, it seems to me that refinements are an overly complicated solution. I think a much better solution would be to make literal construction use current scope when assigning class. For example:

-----

class Foo
String = ::String.dup
class String
def upcase
self.downcase
end
end

def say
String.new("Hello!").upcase
end
end

class Bar < Foo
def shout
String.new("HELLO!").upcase
end
end

"test".upcase #=> "TEST" (Foo's monkey patching doesn't leak)

Foo.new.say #=> hello!

Bar.new.shout #=> hello! (monkey patch is inherited)
Bar.new.shout.class #=> Foo::String (class name tells us exactly where to look for patched method definitions)

-----

The only thing that would need to change to make this work would be for litteral construction to inspect current scope (similar to simply typing an unqualified "String.new"). I suppose this would be a backwards incompatible change, but it seems to me like an easier solution to implement and work with...

----------------------------------------
Feature #4085: Refinements and nested methods

https://bugs.ruby-lang.org/issues/4085#change-32842

[jballanc (Joshua Ballanco)]

Issue #4085 has been updated by jballanc (Joshua Ballanco).

Ok, I think I finally figured out what it is about refinements that makes me so uncomfortable... Refinements violate PLOS in a very bad way, I think, because the behavior of code no longer depends only on the objects and statements in the code, but also where that code is located. For example, consider the following:

-----

module RefString
refine String do
def camelize
"I'm a camel!"
end
end
end

class ARBase
using RefString
end

class DoIt < ARBase
def handle(a_string)
puts a_string.camelize
# ... other stuff
end
end

-----

Now, consider the very common case where we would refactor this code by extracting a portion of the method into a helper class:

-----

class DoIt
def handle(a_string)
Helper.new.help_me(a_string)
# ... other stuff
end
end

class Helper
def help_me(a_string)
puts a_string.camelize
end
end

-----

This, of course, breaks...and I worry that this will have a chilling effect on the willingness of people to refactor their code (something we all don't do enough of as is). I think any benefit that library authors gain in not monkey-patching core classes would be outweighed by the reduced ability to do simple refactorings.

----------------------------------------
Feature #4085: Refinements and nested methods

https://bugs.ruby-lang.org/issues/4085#change-32850

[rosenfeld (Rodrigo Rosenfeld Rosas)]

Issue #4085 has been updated by rosenfeld (Rodrigo Rosenfeld Rosas).


Charles, although I agree that the example you gave reads just as good as the monkey patched version, sometimes I can't find a better approach to some DSL created by monkey patches. This is why I'd like to have something like refinements, but with a much more strict scope. Consider the usage below from ActiveSupport, which is included by Rails:

expire_at = 2.days.from_now

This is very useful and readable but it requires monkey patching Numeric. But it is not clear by looking at this code where "days" was defined. So, I'd prefer to have something more explicit like:

using TimeManipulation do
expire_at = 2.days.from_now
end

Groovy implements something like this:

http://groovy.codehaus.org/api/groovy/time/TimeCategory.html

----------------------------------------
Feature #4085: Refinements and nested methods

https://bugs.ruby-lang.org/issues/4085#change-32854

[headius (Charles Nutter)]

Issue #4085 has been updated by headius (Charles Nutter).


shugo: I believe I understand the implementation. There are a lot of open questions for it, however:

* If the refined class is reopened and the method redefined, does the VM_METHOD_TYPE_REFINED flag get lost?

* How are refinements looked up? In JRuby, we need to know ahead of time if a method body will have refinements active, or we'll have to have them available all the time...with a perf cost. Moving the trigger to a flag on the target method means we can't do that.

* If a refinement is not present the first time, do we never check again?

* If we do check again, doesn't it mean all call sites would need invalidation for potential refinements on every call?

This implementation doesn't really save us anything because it still requires that all call sites check for refined methods, and we can't tell until call time that they'll need to look refinements up. Ultimately, we have to cripple all code just the same as we did before.

Of course it also doesn't address the fact usability, debuggability issues I mentioned either.

----------------------------------------
Feature #4085: Refinements and nested methods

https://bugs.ruby-lang.org/issues/4085#change-32870

[headius (Charles Nutter)]

Issue #4085 has been updated by headius (Charles Nutter).

jballanc: That is exactly the sort of problem I'm worried about. It is *impossible* to look at the DoIt class in your example and know what methods it will call. Even with monkey-patching, you would have the consistency of all callers seeing the same code. Not so with refinements; the location of the code can now change what methods are called. Moving code can change what methods are called. Changing your class hierarchy can change what methods are called. Passing a block of code to a new method can change what methods are called.

I feel more and more like refinements will make monkey-patching *more* confusing and patched code *more* difficult to understand.

rosenfeld: Groovy does indeed have categories (their equivalent of refinements, but thread-local and down-stack), and they have been a real pain point for them improving Groovy. Because they need to be checked all the time, they impact call performance. They've found various tricks to improve that, but they mostly have been trying to move away from them. And I know this from talking to the implementers themselves...they regret ever adding categories for exactly the reasons I have enumerated: they make optimization harder, they make code more difficult to follow, they don't solve as many problems as they create.

----------------------------------------
Feature #4085: Refinements and nested methods

https://bugs.ruby-lang.org/issues/4085#change-32871

[headius (Charles Nutter)]

Issue #4085 has been updated by headius (Charles Nutter).

shugo: I may have a possible compromise that fixes some of the technical issues.

Currently, refinements have to be looked up via cref, basically (there's oddities for module_eval case, but it's similar to cref). If instead refinements are located *solely* based on the caller object, we can implement refinements using ko1's method flag trick.

The logic would work like this:

* Any method that gets refined gets flagged, as you described earlier.
* A call site that encounters a refined method at lookup time will do the search for refinements by looking at the calling object.
* Lookup will check the object's class and superclasses searching for the refined version of the method.
* If a refined method is found in the caller's hierarchy, it is cached at the call site, but instead of guarding only on the target class it also guards based on the calling class. Modifications to either invalidate the site.
* If a refined method is not found in the caller's hierarchy, caching proceeds as normal.

The edges here are whether refinements added to a caller's hierarchy later should get picked up. If that was a requirement, it would require all call sites cache based on caller as well, regardless of whether they see refinements the first time or not. The alternative would be like ko1 suggested, not caching refinements at the call site at all and only caching the target method as a trigger to look in a second cache that's invalidated globally when modifications come in.

If refinements added after the first call should not be seen, then it's more like the current "temporal" application of refinements.

This implementation would add no overhead to unrefined call logic (other than the initial check) because it's similar to visibility checking. In order to check visibility, we already have to pass caller to the call site anyway.

I'll note again that this does not make any of the added complexity of refinements (from a user/programmer perspective) go away...it just might make it easier to implement without impacting unrefined calls. It also doesn't address the concerns about future modifications to the refined class and whether they can overwrite previously-refined methods.

----------------------------------------
Feature #4085: Refinements and nested methods

https://bugs.ruby-lang.org/issues/4085#change-32872

[rosenfeld (Rodrigo Rosenfeld Rosas)]

Issue #4085 has been updated by rosenfeld (Rodrigo Rosenfeld Rosas).

Charles, I'm not sure if I completely understand your concerns, but I think that from a code readability point of view refinements should be local to the current block or it would become just as difficult to read code as the current monkey-patch approach. See this example:

def a
using TimeConversion do
2.days.ago # OK
b # see below
end
end

def b
2.days.ago # should raise a NoMethodError exception in my opinion
end

If the behavior for refinements were like described above, would it still impact on performance optimizations?

I didn't test something like this in Groovy, but I'm assuming "b" wouldn't raise any exceptions in a situation like above, right? Maybe that is why it is hard to improve performance there...

----------------------------------------
Feature #4085: Refinements and nested methods

https://bugs.ruby-lang.org/issues/4085#change-32876

[rosenfeld (Rodrigo Rosenfeld Rosas)]

Issue #4085 has been updated by rosenfeld (Rodrigo Rosenfeld Rosas).

Just tested it to confirm:

a = {1.hour.ago}; use(groovy.time.TimeCategory){a()}

If "a()" throwed an exception in Groovy for the example above would it be easier for them to optimize the interpreter?

----------------------------------------
Feature #4085: Refinements and nested methods

https://bugs.ruby-lang.org/issues/4085#change-32877

[headius (Charles Nutter)]

Issue #4085 has been updated by headius (Charles Nutter).


Talking more with folks online about refinements, there's a lot of confusion about what they solve.

So, in one sense, refinements are to localize monkey-patching. But they don't actually localize it much better since they can apply at a distance to blocks (module_eval feature), and classes down-hierarchy.

Previously, all code determined what methods to call based solely on the target object's class hierarchy. Even with monkeypatches in place, we still have to look solely at the target class to determine what's being called.

With refinements, every piece of code everywhere in the system could potentially see refinements active whether there's a "using" clause near them or not. Blocks could be forced to call different methods at any time, normal code could see a superclass add a refinement and change all future calls. Refinements may prevent monkeypatches from affecting the entire runtime, but don't make it any easier to determine what methods will actually be called.

They also don't solve the monkeypatching problem in any way. Monkeypatches have been used for a few reasons:

* Adding methods to existing types, for DSL or fluent API purposes.

Refinements would limit the visibility of those methods, somewhat, but you can't tell without digging around both the target class hierarchy and the calling class hierarchy what methods will really be called.

* Replacing existing methods with "better" versions

Refinements would again limit the visibility of those changes, but ultimately result in some code calling one method and some code calling another, with no easy way to determine the code that will be called ahead of time.

It may be possible to address the technical issues of optimizing call sites with and without refinements, but I really don't feel like refinements are solving as many problems as they're going to create. I lament a future where I can't look at a piece of code and determine the methods it's calling solely based on the types it is calling against. It's going to be harder -- not easier -- to reason about code with refinements in play.

----------------------------------------
Feature #4085: Refinements and nested methods

https://bugs.ruby-lang.org/issues/4085#change-32878

[headius (Charles Nutter)]

Issue #4085 has been updated by headius (Charles Nutter).

rosenfeld: Yes, I am arguing that same case. I believe refinements should only be active for code that appears within a refined context. My example from earlier:

class Foo < SomeParent
def bar(str)
str.upcase # unrefined
end

using StringRefinement
def baz(str)
str.camlize # refined
end
end
end

There are implementation reasons why this is simpler, but the more important reasons are readability, understandability of the code. You know exactly whose methods will be called in both cases -- String's in the bar() body and StringRefinement's or String's in the baz() body -- and there's no question whether refinements are active for a given call. Compare that to the following code:

class Foo < SomeParent
def bar(str)
str.upcase
end

def baz(str)
str.camelize
end
end

What methods are being called? Where are they coming from? You can't know, since you need more information that the type of object that str is. You need to know whether Foo has previously had refinements applied, whether SomeParent previously had refinements applied, whether its parents previously had refinements applied...you need to know what those refinements are and whether they affect String methods...and you need to know whether any of the methods you are calling have been refined.

EVERY PIECE OF CODE in a given system now forces users to understand BOTH the target class being called AND the hierarchy of code surrounding the call. That's not simpler, it's more complicated...and it affects the readability of ALL CODE.

And then there's this:

class Foo < SomeParent
def baz(str)
ary.map {|name| str.camelize + name}
end
end

In this case, you have to check even more places for refinements to know what methods will be called:

* Foo may have been previously refined. You must look for all reopenings of Foo to know what will be called.
* SomeParent or its parents may have been previously refined. You must look for all reopenings of SomeParent and its parents.
* The map method may force refinements on the block. you must look for all implementations of map() that might be called here to see if they force refinements into the block.

This is supposed to be simpler?

----------------------------------------
Feature #4085: Refinements and nested methods

https://bugs.ruby-lang.org/issues/4085#change-32879

[rosenfeld (Rodrigo Rosenfeld Rosas)]

Issue #4085 has been updated by rosenfeld (Rodrigo Rosenfeld Rosas).


It reminds me C++. So powerful that you can have a hard time trying to understand what is happening in a given piece of code since you have so many operators and each of them can be overloaded. Very powerful but once you're relying on some third-party library using those powerful features instead of using them yourself you can find yourself in trouble trying to understand what is going on with your code.

----------------------------------------
Feature #4085: Refinements and nested methods

https://bugs.ruby-lang.org/issues/4085#change-32880

[brainopia (Ravil Bayramgalin)]

Issue #4085 has been updated by brainopia (Ravil Bayramgalin).

> Ideally, in the Rails source code, I would just add "using StringExt" to
> the ActiveRecord::Base class, and the existing user code would Just
> Work. If existing Rails users are required to add the "using" word to
> all of their code, then there is no way we (the rails team) can remove
> the monkey patches and remain backwards compatible.

Aaron, if refinements were not leaking to inherited classes (as Yehuda nicely described why) then you still would have the ability to manually propagate them if needed, eg:

class ActiveRecord::Base
using ActiveRecord::Refinements

def self.inherited(klass)
klass.module_eval { using ActiveRecord::Refinements }
end
end

----------------------------------------
Feature #4085: Refinements and nested methods

https://bugs.ruby-lang.org/issues/4085#change-32896

[Aaron Patterson]

On Wed, Nov 14, 2012 at 05:38:12AM +0900, headius (Charles Nutter) wrote:

[snip]

> And then there's this:
>
> class Foo < SomeParent
> def baz(str)
> ary.map {|name| str.camelize + name}
> end
> end
>
> In this case, you have to check even more places for refinements to know what methods will be called:
>
> * Foo may have been previously refined. You must look for all reopenings of Foo to know what will be called.
> * SomeParent or its parents may have been previously refined. You must look for all reopenings of SomeParent and its parents.
> * The map method may force refinements on the block. you must look for all implementations of map() that might be called here to see if they force refinements into the block.
>
> This is supposed to be simpler?

I have to agree. It seems like this would be *much* more difficult to
debug than if `camelize` is just monkey patched on to String.

This code:

class Foo < SomeParent
def baz(str)
cached = str.camelize
ary.map {|name| cached + name}
end
end

Could have a completely different meaning than this code:

class Foo < SomeParent

def baz(str)
ary.map {|name| str.camelize + name}
end
end

That seems extremely bad.

[Trans]

Perhaps refinements should be scoped per-gem, rather than any arbitrary "using" delimitation. Seems to me, that is generally the level at which we care about them.

Would that simplify implementation and comprehensibility of usage to something more manageable?

--
Sorry, says the barman, we don't serve neutrinos. A neutrino walks into a bar.

Trans <tran...@gmail.com>
7r4n5.com      http://7r4n5.com

[The 8472]

> So, in one sense, refinements are to localize monkey-patching. But they don't actually localize it much better since they can apply at a distance to blocks (module_eval feature), and classes down-hierarchy.
> Previously, all code determined what methods to call based solely on the target object's class hierarchy. Even with monkeypatches in place, we still have to look solely at the target class to determine what's being called.
> With refinements, every piece of code everywhere in the system could potentially see refinements active whether there's a "using" clause near them or not. Blocks could be forced to call different methods at any time, normal code could see a superclass add a refinement and change all future calls. Refinements may prevent monkeypatches from affecting the entire runtime, but don't make it any easier to determine what methods will actually be called.

I think I might have a solution to this:

1. refinements should only go through the local module hierarchy, not
the class hierarchy because all contributors to a module namespace
should be familiar with the conventions and refinements used within that
module.

If for example ActiveRecord wants to add .constantize their Strings and
use that feature throughout their project it should be fine, as they
should be putting everything under the ::ActiveRecord module anyway.

If someone has his own rails application he can use the refinement again
in his own application module or - if he isn't using any module - apply
it to all of his classes.

2. instance_eval/module_eval/class_eval should NOT apply the refinements
of the target. Instead they should use the same refinements as the
context they were defined in.

To still allow for fancy DSLs there should be a way to explicitly rebind
the context of the Proc to a different refinement context. Basically
bind its lookup to a different module.

A Proc passed to the DSL would first have to be re-bound to the DSL's
own module and then eval'd. As long as nobody else rebinds the Proc
again this shouldn't invalidate any cache as the Proc was never called
in its old context before. What is even better is that someone else
could extend the DSL under a different module but the DSL's extensions
would still get applied to the block, as expected by the caller.



3. there should also be an way to remove refinements from a module and
all its submodules. This makes it possible to apply a refinement to
::Object and then prune it out of some foreign namespaces where it turns
out to cause trouble.

Consider it "optimistic refining".



1 and 2 mean that every Callsite and Proc can only have one refinement
context at any given time.
Rule 3 makes monkeypatching safer as we can still apply it globally as
we already do and simply undo it in places where the monkey bites us.

While this may seem less flexible you also have to consider that these
single-scope refinements can be used together with Module.prepend

This allows you to move an aspect of code to a different module inside
your application, apply refinements only to that module (and its
submodules), write your refined code inside that module and then prepend
that module with the refined behavior to the target class which you do
not want to "pollute" with the refinements.

[shugo (Shugo Maeda)]

Issue #4085 has been updated by shugo (Shugo Maeda).

Assignee changed from shugo (Shugo Maeda) to matz (Yukihiro Matsumoto)

Thanks for your feedback, Charles and others.
I understand your worries.

The feature set of Ruby 2.0 has already been frozen, so it's impossible to introduce a completely different feature in Ruby 2.0. So we have only the following options:

1. introduce the whole features of Refinements currently implemented
2. introduce some of the features of Refinements (= drop some features)
3. remove all features of Refinements

I think optional features of Refinements are as follows:

A. refinement inheritance in class hierarchies
B. refinement activation for reopened module definitions
C. refinement activation for the string version of module_eval/instance_eval
D. refinement activation for the block version of module_eval/instance_eval

I've asked Matz to decide whether Refinements should be included in Ruby 2.0, and if so, which of these features should be included.

My own take is as follows:

* I'm not sure A is good or not. It's useful in some cases, but it may be confusing because include doesn't inherit refinements, but class inheritance does. So it's OK to remove A from Ruby 2.0.
* I want C and D for internal DSLs, but D might be difficult to implement in VMs other than CRuby. So it's OK to remove D from Ruby 2.0.
FYI, I'm implementing it without performance overhead when refinements are not used.
http://shugo.net/tmp/refinement_fix_1119.diff
In this implementation, refined methods are stored in neither an inline method cache nor the global method cache,
so there's no need to invalidate cache for module_eval. I hope D will be introduced in the future.
* From the perspective of consistency, C and D depend on B. So if C or D is included in Ruby 2.0, B should also be included.

And, I explain some things to clarify my intention.

I have used the word "lexical" to describe Refinements, but by the word I've meant just that Refinements doesn't support local rebinding. For example, in the following code, FooExt doesn't affect Bar#call_foo even if it's called from Baz, which is a module using FooExt.

class Foo

end
module FooExt
refine Foo do

def foo
puts "foo"
end
end
end
class Bar
def call_foo(f)
f.foo
end
end
module Baz
using FooExt
f = Foo.new
f.foo # => foo
Bar.new.call_foo(f) # => NoMethodError
end

I think it's the most important feature of Refinements. Without it, it's hard to avoid conflicts among multiple refinements.

Some people seem to suspect that code using refinements is difficult to debug, but reflection APIs may be useful to debug such code.

module M
refine Fixnum do
def foo; puts "foo" end
end
end
using M
p 123.method(:foo).owner #=> #<refinement:Fixnum@M>

I admit that Refinements are complex, but it's because issues to address by Refinements are themselves complex. And, I think Refinements should not be over-used. Application programmers should not use Refinements. Refinements are for library/framework programmers. Besides, even if you're a library or framework programmer, consider other features such as subclassing before Refinements.

----------------------------------------
Feature #4085: Refinements and nested methods

https://bugs.ruby-lang.org/issues/4085#change-33146

Author: shugo (Shugo Maeda)
Status: Assigned
Priority: Normal

Assignee: matz (Yukihiro Matsumoto)

[trans (Thomas Sawyer)]

Issue #4085 has been updated by trans (Thomas Sawyer).


=begin
So...

class Foo
end
module FooExt
refine Foo do
def foo
puts "foo"
end
end
end

module Kernel
def safe_call(f, m)
if f.respond_to? m
return f.send(:m)
end
nil
end
end
class Object
def safe_send(m)
if respond_to? m
return f.send(:m)
end
nil

end
end
module Baz
using FooExt
f = Foo.new

f.respond_to? :foo # => true
safe_call(f, :foo) # => nil
f.safe_send(:foo) # => ?
end
=end

And how much worse if:

=begin
class Object
def safe_send(m)
safe_call(self, m)
end
end
=end

----------------------------------------
Feature #4085: Refinements and nested methods

https://bugs.ruby-lang.org/issues/4085#change-33156

[shugo (Shugo Maeda)]

Issue #4085 has been updated by shugo (Shugo Maeda).

shugo (Shugo Maeda) wrote:
> Application programmers should not use Refinements.

I meant that application programmers should not use Module#refine.
It's OK to use Kernel#using and Module#using.

----------------------------------------
Feature #4085: Refinements and nested methods

https://bugs.ruby-lang.org/issues/4085#change-33166

[shugo (Shugo Maeda)]

Issue #4085 has been updated by shugo (Shugo Maeda).

trans (Thomas Sawyer) wrote:
> f.safe_send(:foo) # => ?

This should return nil. Otherwise, refinements can break code which doesn't expect the refined behavior.

> And how much worse if:
>

> class Object
> def safe_send(m)
> safe_call(self, m)
> end
> end

I don't understand why it's worse.

----------------------------------------
Feature #4085: Refinements and nested methods

https://bugs.ruby-lang.org/issues/4085#change-33168

[rosenfeld (Rodrigo Rosenfeld Rosas)]

Issue #4085 has been updated by rosenfeld (Rodrigo Rosenfeld Rosas).

shugo (Shugo Maeda) wrote:
> Some people seem to suspect that code using refinements is difficult to debug, but reflection APIs may be useful to debug such code.

I don't think the big problem is debugging, but readability. This is my own definition of readability: the longer it takes for one to understand some code, the less readable it is. In that sense, refinements in libraries seem even less readable than refinements in application code.

Consider someone new to Ruby reading a Rails code like "some_string.camelize". He could look for camelize in String class RDoc and he wouldn't find it there. He would be forced to use the reflection API to understand where is that method coming from. This reduces readability as the developer now needs an extra step (time) to figure out where that method is coming from. Looking at the API is quick. Debugging or using the reflection API is not that quick. That is the same problem I have with Monkey Patches. They reduce readability. It is even worse when it overrides some built-in method and change its behavior.

That is the same reason I don't add extensions of my own to core classes (like aliasing each_with_object to a shorter name, etc). It makes it harder for others in the team to read the application code.

I'm pretty sure that once refinements are included in final 2.0.0 people will start using it just because they want to use the new fancy feature and not because they really feel its need. This is what probably happened when DHH first introduced dynamic finders in ActiveRecord in my opinion. I believe he found it fantastic the method_missing feature and decided to use it just because he could. There was no real need for that. And people keep saying that it is ok to use monkey patches and method_missing at will and that Ruby even encourages that practice or otherwise those features wouldn't exist. People build some culture over what is Ruby best practices and what is not. I remember that someone from the Rails core team judging my patch some years ago stating that I shouldn't use "is_a?" because the Ruby way is to use duck-typing. Even so I really wanted to test for the specific class instead of just asking if the object responds to some method. It was much more readable to use
"is_a?" in that context.

I'm afraid that once refinements are possible in Ruby they will be immediately abused just because they are fancy and it will take years for libraries to start to avoid them because they are no longer fancy (like what happened to ActiveRecord removing the dynamic finders just now, years after it has been introduced to AR).

I know this is hard to balance. People moved from other languages to Ruby because Ruby was a more powerful language. But that is not the single reason. It was also more readable. But a feature like this one has its tradeoffs. It makes Ruby more powerful, by reducing conflicts that might be created by conflicting versions of monkey patches. But at the cost of both performance (which I think is the least problem here) and readability.

The worst part for me is that even if I opt out for using refinements myself, I'll still have to live with it and create tons of checks when debugging code and trying to understand others' (libraries/frameworks/applications) code. I realize this issue is not introduced by this feature as monkey patches have the same effect. But the problem this feature introduces is that it becomes even harder to read code once refinements are possible.

For instance, with monkey patches, if I just make sure I load/require all files in the same order, I could write a code using the reflection API to understand where a method is coming from. But now this won't suffice anymore because the same method could be defined elsewhere when inside another context. If that other context is not easily triggered/reproducible than it gets even hard to try to understand why that special condition is not working under production, for instance. Imagine yourself trying to fix a bug caused by some race-condition that you can't always replicate. Usually some code inspection is all that you get left to work on. But once refinements are possible, inspecting some code becomes so much more difficult to perform :(

Then, the real question is: what is most important? Powerful or readability/performance? We can't get both with this feature.

----------------------------------------
Feature #4085: Refinements and nested methods

https://bugs.ruby-lang.org/issues/4085#change-33185

[shugo (Shugo Maeda)]

Issue #4085 has been updated by shugo (Shugo Maeda).

rosenfeld (Rodrigo Rosenfeld Rosas) wrote:
> Then, the real question is: what is most important? Powerful or readability/performance? We can't get both with this feature.

I proposed Refinements for power. Refinements are more powerful than monkey patching in the sense that incompatible class extensions can be mixed in a single program without conflicts.

But it doesn't mean I don't care readability, just mean I prioritize power over readability in this case.

Without module_eval and refinement inheritance in class hierarchies, refinements don't decrease readability so much, because using is used in a file you're reading, except when a module is reopened, in which case you have to pay a price for monkey patching.

In terms of module_eval, expected use cases are internal DSLs. They might look magical, but users don't need to understand the whole stuff in the underlying implementation of a DSL.

I don't see any enough reason to justify refinement inheritance in class hierarchies. That's why I withdrew it in [ruby-core:49631]#141.

Finally, as to performance, it's important that there's no overhead when refinements are not used. I'm working to achieve it, and believe it's possible with ko1's help at least in CRuby.

----------------------------------------
Feature #4085: Refinements and nested methods

https://bugs.ruby-lang.org/issues/4085#change-33285

[rosenfeld (Rodrigo Rosenfeld Rosas)]

Issue #4085 has been updated by rosenfeld (Rodrigo Rosenfeld Rosas).


shugo (Shugo Maeda) wrote:

> ...Without module_eval and refinement inheritance in class hierarchies, refinements don't decrease readability so much, because using is used in a file you're reading, except when a module is reopened, in which case you have to pay a price for monkey patching.

Since this introduces no more harm then monkey patching and since monkey patching isn't going anywhere, I don't object refinements for the implementation strategy you just described if I understand it correctly.

> In terms of module_eval, expected use cases are internal DSLs. They might look magical, but users don't need to understand the whole stuff in the underlying implementation of a DSL.

This is true until something goes weird and you don't know if there is a bug in the DSL implementation and you have to dig on that to understand what is happening. That is the problem with black boxes. Once they are no longer black boxes to you (you need to dig its source-code) it helps when it doesn't use much magic (it gets more readable).

I'd like to state that I'm not against DSLs. I'm all for them. But I wouldn't mind if I had to write my specs like this:

RSpec.describe "Some use case" do |spec|
spec.example "some example" do |dsl|
dsl.expect(something).to happen
end
end

Or even something like:

class << RSpec::Description("Some use case") # RSpec::Description would return an anonymous class, which we're extending below
pending "some example" # pending is defined in RSpec::Description() returned class
end

I just don't think that lots of powerful features would justify some readability loss. I know this is very subjective as lots of developers would find the current DSL for RSpec better than the ones above, but for me I prefer explicit things over implicit ones. Just because it is much easier for me to understand code written this way when bad things happen. Black boxes are painful to work with when things goes wrong in unexpected ways.

----------------------------------------
Feature #4085: Refinements and nested methods

https://bugs.ruby-lang.org/issues/4085#change-33339

[Trans]

> This should return nil.  Otherwise, refinements can break code which doesn't expect the refined behavior.

Ok. Makes sense. 

> I don't understand why it's worse.

Only worse in that case if you had not said `nil`.

Since you did say `nil`, then what about:

    class S

      def foo

        "foo"      

      end

      def bar

        "bar"      

      end

      def foobar

        foo + bar

      end

    end

    module R
      refine S do
        def bar
          "bar!"
        end
      end
    end

    class C < S

      using R

      alias :foobar! :foobar

      def foobar?

        foo + bar

      end

      def brainfart1

        c = C.new

        c.foobar! == c.foobar?

      end

      def brainfart2

        c = C.new

        s = S.new

        c.foobar! == s.foobar

      end

      def brainfart3

        c = C.new

        s = S.new

        c.bar == s.bar

      end

    end

    C.new.brainfart1  #=> ?

    C.new.brainfart2  #=> ?

    C.new.brainfart3  #=> ?

[trans (Thomas Sawyer)]

Issue #4085 has been updated by trans (Thomas Sawyer).


=begin

=end

----------------------------------------
Feature #4085: Refinements and nested methods

https://bugs.ruby-lang.org/issues/4085#change-33344

[The 8472]

On 20.11.2012 03:45, shugo (Shugo Maeda) wrote:
> I think optional features of Refinements are as follows:
>
> A. refinement inheritance in class hierarchies

I generally think that class/module inheritance is the wrong propagation
strategy for refinements. If you see refinements as monkey patches they
are only necessary to get *your own code* working as desired. When you
provide an abstract class/module in a library that application code can
inherit from, then the application itself may not need the refinements
you use to make the internals of your class tick.

In other words, module/class inheritance is about inheriting desired
*behavior*. Monkey patches may not be desired behavior, they are the
dirty internal mechanics that should be hidden from subclasses.

There is an axis orthogonal to behavior. It's responsibility. This
second axis is generally associated with the module namespaces. E.g. I
expect the rails maintainers to be responsible for the ::ActiveRecord
namespace and be aware of what their own code is doing. Their own
refinements should not pose a problem to them. But they might be for me.

If they "need" a String.camelize in their code then they should be able
to add it without polluting my code. If I consider it useful I can still
include their refinements into my code.

Therefore I think that class inheritance should be removed. And if it
gets replaced in the future then it should be with submodule based
inheritance.

The other issue i have with inheritance is that there is no opt-out.

This is the very same issue we're trying to fix! If some piece of code
monkey-patches Object then the whole application is hit by that
modification. Refinements are supposed to prevent this. But what happens
if i want to use a module that applies refinements? Then I would get hit
by those refinements too.

If we want inheritance then we need some way to opt-out of refinements.
Consider someone applying "using" to Object itself early on during the
application loading process. We would be in the same mess we are in now.
Actually. It would be worse, some methods might see the refinements and
others don't, depending on their definition time.


For now people can use Module.extended/.included if they really want to
add refinement inheritance themselves.

> B. refinement activation for reopened module definitions
> C. refinement activation for the string version of module_eval/instance_eval
> D. refinement activation for the block version of module_eval/instance_eval

I don't feel strongly about those, but if the module_eval performance
really has such a big issue as headius asserts then it might be better
to postpone it until a solution has been found.

Probably the safest approach for now would be to use the source
refinement scope (which is quasi-static) for module_eval by default and
add a way to use the target scope (or an explicit scope) later on as
needed. If there is any performance impact it would restricted to the
target-scoped procs.

I think some clever optimizations should be able to eliminate cases
where procs flow through consistent code paths, i.e. are always
evaluated against the same target refinement scope as usually is the
case with DSLs or class-level configurations.

> I have used the word "lexical" to describe Refinements, but by the word I've meant just that Refinements doesn't support local rebinding. For example, in the following code, FooExt doesn't affect Bar#call_foo even if it's called from Baz, which is a module using FooExt.
>
> class Foo
> end
> module FooExt
> refine Foo do
> def foo
> puts "foo"
> end
> end
> end
> class Bar
> def call_foo(f)
> f.foo
> end
> end
> module Baz
> using FooExt
> f = Foo.new
> f.foo # => foo
> Bar.new.call_foo(f) # => NoMethodError
> end
>
> I think it's the most important feature of Refinements. Without it, it's hard to avoid conflicts among multiple refinements.

What about cases like

module SomeExt
refine String do
def bar
end
end
end

class Foo
using SomeExt

def self.test1
"".tap(&:bar)
end

def self.test2
"".tap{|f| f.bar}
end
end


String.bar is only visible inside Foo, but in test1 the Proc is created
in .to_proc of Symbol, i.e. on a different stack frame, which shouldn't
be able to see bar due to the scoping. Which leads to counter-intuitive
results.

[shugo (Shugo Maeda)]

Issue #4085 has been updated by shugo (Shugo Maeda).


trans (Thomas Sawyer) wrote:

> Since you did say `nil`, then what about:

(snip)
> C.new.brainfart1 #=> ?

false. Because bar in C#foobar? is refined by R, but bar in C#foobar!, which is an alias of S#foobar, is not refined.

> C.new.brainfart2 #=> ?

true. Because both bar in C#foobar! and bar in S#foobar are not refined.

> C.new.brainfart3 #=> ?

true. Because both c.bar and s.bar calls bar refined by R.

Refinements are available in trunk, so could you try it yourself?
If the behavior is unexpected, please report it with a reason why you think it's wrong.

----------------------------------------
Feature #4085: Refinements and nested methods

https://bugs.ruby-lang.org/issues/4085#change-33379

end

module FooExt
refine Foo do

[shugo (Shugo Maeda)]

Issue #4085 has been updated by shugo (Shugo Maeda).

The8472 (Aaron G) wrote:
> Therefore I think that class inheritance should be removed. And if it
> gets replaced in the future then it should be with submodule based
> inheritance.

I'll remove it if permission granted by Matz.

> For now people can use Module.extended/.included if they really want to
> add refinement inheritance themselves.

Currently this wouldn't work because you cannot get the caller context in these hooks.

> > B. refinement activation for reopened module definitions
> > C. refinement activation for the string version of module_eval/instance_eval
> > D. refinement activation for the block version of module_eval/instance_eval
>
> I don't feel strongly about those, but if the module_eval performance
> really has such a big issue as headius asserts then it might be better
> to postpone it until a solution has been found.

A solution has been found at least in CRuby.
I'm waiting for ko1's review.

> Probably the safest approach for now would be to use the source
> refinement scope (which is quasi-static) for module_eval by default and
> add a way to use the target scope (or an explicit scope) later on as
> needed. If there is any performance impact it would restricted to the
> target-scoped procs.

Do you mean that a new option of module_eval should be introduced?
For example,

Foo.module_eval { # use refinements in the current context }
Foo.module_eval(using_refinements: true) { # use refinements in the receiver }

> What about cases like
>
> module SomeExt
> refine String do
> def bar
> end
> end
> end
>
> class Foo
> using SomeExt
>
> def self.test1
> "".tap(&:bar)
> end
>
> def self.test2
> "".tap{|f| f.bar}
> end
> end
>
>
> String.bar is only visible inside Foo, but in test1 the Proc is created
> in .to_proc of Symbol, i.e. on a different stack frame, which shouldn't
> be able to see bar due to the scoping. Which leads to counter-intuitive
> results.

Originally, String#bar was not visible in the Proc created by Symbol#to_proc.
But I've changed it because Matz asked to do. I think the current behavior
is not consistent, but useful.

If Symbol#to_proc were written in Ruby, it would be impossible, but
Symbol#to_proc is written in C. There are some such special methods.
For example, Module.nesting returns the module nesting information in the
caller context. Module#using also affects the caller context.

----------------------------------------
Feature #4085: Refinements and nested methods

https://bugs.ruby-lang.org/issues/4085#change-33380

[duerst (Martin Dürst)]

Issue #4085 has been updated by duerst (Martin Dürst).


My personal opinion is that at this point in time, there are too many uncertainties surrounding refinements, and so it is too early to include them. I would therefore support "3. remove all features of Refinements", of course putting them back again in trunk once 2.0.0 is forked from trunk, or into a separate branch, so that further work on them can continue.

I also have my doubts about whether they are really that much of a needed. ActiveRecord and friends are often cited. But I wonder what's the problem with ActiveRecord adding #camelize to String. It's a method that's often necessary in metaprogramming, so it might even be added to Ruby core. And str.camelize is much more natural than camelize(str). Good monkey patching is just extending the functionality of base (and other) classes in natural ways. Of course there's bad monkey patching, but there will be bad refinements, too.

So as for other proposals, I think we should work out the use case much more clearly.

(disclaimer: I had a use case, but discussing it with Shugo a bit over a week ago, he told me that this would have been covered by classboxes, but not by refinements)

----------------------------------------
Feature #4085: Refinements and nested methods

https://bugs.ruby-lang.org/issues/4085#change-33385

[trans (Thomas Sawyer)]

Issue #4085 has been updated by trans (Thomas Sawyer).


@shugo Thanks you for answering. I'm still trying to fully understand refinements so it helps to hear some explanation. That the alias is not refined seems maybe a little confusing, since alias effectively copies the method to the current context. Or do I misunderstand the alias spec? But I can see why it is probably best that way regardless.

Refinements scare me a bit. I am not sure what it is exactly, and maybe its just a matter of getting used to them. I guess my biggest question is what happens if Refinements start getting used a lot. Is it possible that this solution to the monkey-patching issue could become worse then the original problem?

----------------------------------------
Feature #4085: Refinements and nested methods

https://bugs.ruby-lang.org/issues/4085#change-33395

[brixen (Brian Ford)]

Issue #4085 has been updated by brixen (Brian Ford).


Please remove Refinements completely from Ruby 2.0 features. They are not well defined, well understood, or well justified.

There are many arguments both for and against. However, the most compelling argument against is that we cannot accurately say what consequences they will have in practice. There is a ton of speculation. That is all.

A feature this far-reaching should not be blindly dumped into the language, especially when the semantics of it are not even fully developed less than three months before the planned release date.

Thanks,
Brian

----------------------------------------
Feature #4085: Refinements and nested methods

https://bugs.ruby-lang.org/issues/4085#change-33412

[headius (Charles Nutter)]

Issue #4085 has been updated by headius (Charles Nutter).


I would also vote to remove refinements from 2.0 features, perhaps for reinclusion in 2.1. The various points made about lack of clear specification, lack of time to experiment, lack of clarity on the extent of damage/risk, and lack of time for other implementers to fully implement and explore the feature all point toward this being too much, too late in 2.0's dev cycle...and I believe it is too close to 2.0's release to shove the features in.

That said, I will say I appreciate the reduction in scope. Having refinements only searched via lexical enclosures makes the feature much simpler to implement and much easier to understand in real code. To help the process of fleshing out the feature, I have implemented a large part of the reduced refinements feature for JRuby on the refinements branch: https://github.com/jruby/jruby/tree/refinements

Here are my notes on the current JRuby implementation:

* Only the interpreter is supported for now. Compiler support will require a rework of how we access the scope.

* Much of the determination of whether refinements are active can be static. I set a flag in StaticScope after any call to "using" that flags subsequent calls as refined. This narrows refinement impact to calls following "using".

* I have no strong preference as to whether "refine" additions after a "using" call should be expressed; if they are to be expressed, it means holding a reference to the refinement holder module in the cref; if they are not, it means copying them at the moment of the "using" call.

* Reflective methods (method, instance_method, etc) will have to be special-cased in the refined call site, so that the refinements are looked up. We do not want to have to propagate refinements through to the default implementation of those methods.

* My current implementation caches refinements at the call site based on a global token, incremented whenever a refinement change happens. This allows refined calls to be equivalent performance to unrefined calls in most cases, but it ignores all other invalidation mechanisms (hierarchy changes, etc).

* I am not currently searching modules included into modules for refinements.

* super is not implemented; it is not clear to me how to implement it simply/efficiently in JRuby.

---

I believe we can implement the lexical-only version with reasonable efficiency. My implementation still follows your basic structure, putting refinements on cref and using anonymous modules to hold them. As mentioned in the notes above, I do not have the special behavior for reflection or super implemented; several of the missing features will be more complicated to complete in JRuby due to the way we optimize code. I will also need to rework the way we handle scoping in JRuby's compiler to efficiently access the cref scope without requiring a full frame, but so far I believe everything is doable.

I still vote to delay adding refinements until after 2.1. It feels very rushed now.

----------------------------------------
Feature #4085: Refinements and nested methods

https://bugs.ruby-lang.org/issues/4085#change-33413

[headius (Charles Nutter)]

Issue #4085 has been updated by headius (Charles Nutter).

Sorry, I meant "I still vote to delay adding refinements until after 2.0."

----------------------------------------
Feature #4085: Refinements and nested methods

https://bugs.ruby-lang.org/issues/4085#change-33414

[headius (Charles Nutter)]

Issue #4085 has been updated by headius (Charles Nutter).

Here is a trivial benchmark of refined versus unrefined calls in JRuby and MRI trunk. Note that this is only running in JRuby's interpreter, but performance is roughly equivalent between refined and unrefined. I will note again that refined call sites must be globally invalidated right now; this may or may not be a problem for real-world applications.

module X
refine Object do
def blah
end
end
end

class String
def blah2
end
end

10.times {
Benchmark.bmbm {|bm|
bm.report('unrefined') {
str = 'foo'
1_000_000.times {
str.blah2
}
}

bm.report('refined') {
using X
str = 'foo'
1_000_000.times {
str.blah
}
}
}
}

Best results for JRuby and MRI trunk:

JRuby:

Rehearsal ---------------------------------------------
unrefined 0.100000 0.000000 0.100000 ( 0.095000)
refined 0.100000 0.000000 0.100000 ( 0.099000)
------------------------------------ total: 0.200000sec

user system total real
unrefined 0.100000 0.000000 0.100000 ( 0.094000)
refined 0.100000 0.000000 0.100000 ( 0.098000)

MRI:

Rehearsal ---------------------------------------------
unrefined 0.070000 0.000000 0.070000 ( 0.076026)
refined 0.080000 0.000000 0.080000 ( 0.076520)
------------------------------------ total: 0.150000sec

user system total real
unrefined 0.080000 0.000000 0.080000 ( 0.078869)
refined 0.080000 0.000000 0.080000 ( 0.077133)

JRuby compiled performance should be significantly better, but this at least shows there's not a great cost at the refined call sites.

----------------------------------------
Feature #4085: Refinements and nested methods

https://bugs.ruby-lang.org/issues/4085#change-33415

[The 8472]

On 21.11.2012 07:15, shugo (Shugo Maeda) wrote:
> I'll remove it if permission granted by Matz.
>
>> For now people can use Module.extended/.included if they really want to
>> add refinement inheritance themselves.
>
> Currently this wouldn't work because you cannot get the caller context in these hooks.

What about the following?

module RefinementInheritor
def extended(base)
base.send(:using, FooExt)
# or
base.module_eval "using FooExt"
end
end

>> Probably the safest approach for now would be to use the source
>> refinement scope (which is quasi-static) for module_eval by default and
>> add a way to use the target scope (or an explicit scope) later on as
>> needed. If there is any performance impact it would restricted to the
>> target-scoped procs.
>
> Do you mean that a new option of module_eval should be introduced?
> For example,
>
> Foo.module_eval { # use refinements in the current context }
> Foo.module_eval(using_refinements: true) { # use refinements in the receiver }

I was thinking about

Proc.new.rebind_refinements(TargetClass)

since this would only allow a single scope per proc at any given time
which might make optimizations easier. But maybe your way would work too.

> Originally, String#bar was not visible in the Proc created by Symbol#to_proc.
> But I've changed it because Matz asked to do. I think the current behavior
> is not consistent, but useful.
>
> If Symbol#to_proc were written in Ruby, it would be impossible, but
> Symbol#to_proc is written in C. There are some such special methods.
> For example, Module.nesting returns the module nesting information in the
> caller context. Module#using also affects the caller context.

So we need to special-case .to_proc. What happens when I
alias-method-chain to_proc? Would it use the wrong scope? Would things
break?

.__send__ and .method obviously suffer from the same issues of shifting
stack frames and aliasing. Other metaprogramming things might be
expected to do the "right thing(tm)" too and thus would have to rely on
stack inspection which is an absolute minefield in Ruby.

Should the anonymous refinement module be mutable? E.g. by adding some
respond_to_missing to it?

Has anyone even defined how metaprogramming should work with refinements?

[shugo (Shugo Maeda)]

Issue #4085 has been updated by shugo (Shugo Maeda).


The8472 (Aaron G) wrote:

> On 21.11.2012 07:15, shugo (Shugo Maeda) wrote:
> > I'll remove it if permission granted by Matz.
> >
> >> For now people can use Module.extended/.included if they really want to
> >> add refinement inheritance themselves.
> >
> > Currently this wouldn't work because you cannot get the caller context in these hooks.
>
> What about the following?
>
> module RefinementInheritor
> def extended(base)
> base.send(:using, FooExt)
> # or
> base.module_eval "using FooExt"
> end
> end

The above code cannot activate refinements in the caller context.
That is, a NoMethodError is raised in the following example:

module FooExt
refine String do

def foo
puts "foo"
end
end
end

module RefinementInheritor
def self.included(mod)
mod.send(:using, FooExt)
# or
mod.module_eval "using FooExt"
end
end

module Foo
include RefinementInheritor

p "abc".foo #=> NoMethodError
end

> I was thinking about
>
> Proc.new.rebind_refinements(TargetClass)
>
> since this would only allow a single scope per proc at any given time
> which might make optimizations easier. But maybe your way would work too.

What happens if the receiver of rebind_refinements has already been called before rebind_refinements?

p = Proc.new { ... }
p.call
p.rebind_refinements(TargetClass)
p.call

> > Originally, String#bar was not visible in the Proc created by Symbol#to_proc.
> > But I've changed it because Matz asked to do. I think the current behavior
> > is not consistent, but useful.
> >
> > If Symbol#to_proc were written in Ruby, it would be impossible, but
> > Symbol#to_proc is written in C. There are some such special methods.
> > For example, Module.nesting returns the module nesting information in the
> > caller context. Module#using also affects the caller context.
>
> So we need to special-case .to_proc. What happens when I
> alias-method-chain to_proc? Would it use the wrong scope? Would things
> break?

Do you mean to redefine Symbol#to_proc yourself? If so, it's impossible to close refinements in the caller context of Symbol#to_proc into the created Proc.

> .__send__ and .method obviously suffer from the same issues of shifting
> stack frames and aliasing. Other metaprogramming things might be
> expected to do the "right thing(tm)" too and thus would have to rely on
> stack inspection which is an absolute minefield in Ruby.

__send__ and .method work the same as Symbol#to_proc in the current implementation.

> Should the anonymous refinement module be mutable? E.g. by adding some
> respond_to_missing to it?

Currently respond_to_missing doesn't work in a refinement module.

module FooExt
refine String do
def respond_to_missing?(mid, include_all)
mid == :foo
end

def method_missing(mid, *args)
if mid == :foo
puts "foo!"
else
super
end
end
end
end

using FooExt

if "abc".respond_to?(:foo) #=> false
"abc".foo
end

I guess respond_to? need to be fixed to make the above code work.

> Has anyone even defined how metaprogramming should work with refinements?

I think, in principle, metaprogramming APIs related to method dispatching should use refinements in the caller context.

----------------------------------------
Feature #4085: Refinements and nested methods

https://bugs.ruby-lang.org/issues/4085#change-33464

[shugo (Shugo Maeda)]

Issue #4085 has been updated by shugo (Shugo Maeda).

headius (Charles Nutter) wrote:
> I would also vote to remove refinements from 2.0 features, perhaps for reinclusion in 2.1. The various points made about lack of clear specification, lack of time to experiment, lack of clarity on the extent of damage/risk, and lack of time for other implementers to fully implement and explore the feature all point toward this being too much, too late in 2.0's dev cycle...and I believe it is too close to 2.0's release to shove the features in.

It may be better to remove Refinements completely than to introduce Refinements partially.
I defer all decisions to Matz.

> That said, I will say I appreciate the reduction in scope. Having refinements only searched via lexical enclosures makes the feature much simpler to implement and much easier to understand in real code. To help the process of fleshing out the feature, I have implemented a large part of the reduced refinements feature for JRuby on the refinements branch: https://github.com/jruby/jruby/tree/refinements

Thank you. I've tried it and it works well as a lexical version of Refinements.

----------------------------------------
Feature #4085: Refinements and nested methods

https://bugs.ruby-lang.org/issues/4085#change-33465

[jballanc (Joshua Ballanco)]

Issue #4085 has been updated by jballanc (Joshua Ballanco).

shugo (Shugo Maeda) wrote:
> headius (Charles Nutter) wrote:
> > I would also vote to remove refinements from 2.0 features, perhaps for reinclusion in 2.1. The various points made about lack of clear specification, lack of time to experiment, lack of clarity on the extent of damage/risk, and lack of time for other implementers to fully implement and explore the feature all point toward this being too much, too late in 2.0's dev cycle...and I believe it is too close to 2.0's release to shove the features in.
>
> It may be better to remove Refinements completely than to introduce Refinements partially.
> I defer all decisions to Matz.

I wonder if refinements could be included as a "hidden" feature, similar to tail-call optimization? I suspect that even if refinements were included in Ruby 2.0 it would be at least a year or two before library authors (presumably the true audience for this feature) could really make use of refinements, as they will likely want to continue supporting Ruby 1.9 in the near term. I think leaving them in but "off" by default would allow library authors to begin experimenting with refinements, find all of the various corner cases, and then there could be a well defined roadmap for resolving the remaining issues and having refinements enabled by default.

----------------------------------------
Feature #4085: Refinements and nested methods

https://bugs.ruby-lang.org/issues/4085#change-33630

[headius (Charles Nutter)]

Issue #4085 has been updated by headius (Charles Nutter).


Escaping the "should we or shouldn't we" question for a bit, I thought of an alternative implementation, building off ko1's idea.

ko1's suggestion, as I understand it, was to add a flag to the method table (or method entry) of a refined class/method as a trigger for the call site to search refinements. While writing my blog post, I started to type the sentence "the methods defined in the refinement do not actually go on the class in question", and then I realized: why not?

Currently, Ruby implementations structure the method table as a simple map from names to method bodies. If instead the method table was a map from names to collections of methods, we could use that to choose the appropriate method for a given context.

So, for code like this:

module X
refine String do
def upcase; downcase; end
end
end

String's method table would contain an entry like this:

{:upcase => {
:default => <builtin upcase>,
X => <upcase patched>}
}

Method lookup would then proceed as normal in all situations. The result of lookup would be a table mapping refinements to methods with a default entry if the method is defined directly on String.

After lookup, call sites would know there's potentially refinements active for the given method. The calling scope (or parent scopes) would have references to individual refinements, and if there were an entry for one of them it would be used.

This still requires access to the caller scope, of course, to understand what refinements are active. However, because refinement changes would invalidate the String class directly (since they actually modify the method table), the method (refined or otherwise) could be cached as normal. The caller's scope never changes (statically determined at compile time), so it does not participate in invalidation.

This also works for refinements added to classes after the initial run through. If we cache the default downcase method from String, and then the refinement is updated to add downcase, we would see that as an invalidation event for String's method table. Future calls would then re-cache and pick up the change.

This also feels a bit more OO-friendly to me. Rather than storing patches on separate structures sprinkled around memory, we store the patches directly on the refined class, only using the module containing the refinements as a key. The methods *do* live on String, but depending on the *namespace* they're looked up from we *select* the appropriate implementation. It's basically just double-dispatch at that point, with the selector being the calling context.

It also makes available an interesting possibility for #method and friends: return all methods. So...

using X

String.instance_methods(:upcase) # => {:default => <builtin upcase>, X => <new upcase>}

Note that this is "instance_methods", plural, to avoid breaking instance_method and to make it explicit that we're asking for all implementations of a given method. This allows accessing the original method even if refinements are active, and still also allows searching for the refined method active in the current scope.

I admit I am a bit reluctant to suggest this, because I still have concerns about the feature itself. But it would be possible for call sites to only need a reference to their calling scope (determined at parse time) to implement dynamic refinements without severe impact to normal code. Dynamic refinements, as in module_eval, would work by simply invalidating the call sites they contain. This could be done actively, walking all call sites and resetting them. This could also be done by invalidating the classes refined. An example in pseudo-code:

def X.module_eval_refined(&block)
unless block.using? self
refinements.each_key {|cls| cls.touch } # invalidate all refined classes
block.using(self)
end
module_eval &block
end

This is obviously not a thread-safe mechanism. An alternative that invalidates the block's call sites (this would require more work and be more expensive at invalidation time, but less globally-damaging):

def X.module_eval_refined(&block)
unless block.using? self
block.invalidate
block.using(self)
end
module_eval &block
end

Proc#using would either mutate the block's already-present scope (permanently adding the refinement) or duplicate the block and its scope and tweak it (more expensive, of course).

---

In any case, I would really like more time for this dialog to continue. If we push refinements into Ruby in their current form, we're not giving adequate time to flesh out the edge cases. If we push a partial implementation now, we may be making a future implementation harder and we would not be protecting ourselves from mistakes. I want to work with you to find a definition and implementation of refinements that meets requirements without punishing future Rubyists.

I also must apologize for not joining the dialog sooner. This bug was filed in 2010, and the current refinements implementation was pushed to master a few months ago. We should have started discussing a long time ago.

----------------------------------------
Feature #4085: Refinements and nested methods

https://bugs.ruby-lang.org/issues/4085#change-33640

[The 8472]

On 22.11.2012 17:40, headius (Charles Nutter) wrote:

> After lookup, call sites would know there's potentially refinements active for the given method. The calling scope (or parent scopes) would have references to individual refinements, and if there were an entry for one of them it would be used.
>
> This still requires access to the caller scope, of course, to understand what refinements are active. However, because refinement changes would invalidate the String class directly (since they actually modify the method table), the method (refined or otherwise) could be cached as normal. The caller's scope never changes (statically determined at compile time), so it does not participate in invalidation.
>
> This also works for refinements added to classes after the initial run through. If we cache the default downcase method from String, and then the refinement is updated to add downcase, we would see that as an invalidation event for String's method table. Future calls would then re-cache and pick up the change.
>
> This also feels a bit more OO-friendly to me. Rather than storing patches on separate structures sprinkled around memory, we store the patches directly on the refined class, only using the module containing the refinements as a key. The methods *do* live on String, but depending on the *namespace* they're looked up from we *select* the appropriate implementation. It's basically just double-dispatch at that point, with the selector being the calling context.

This (together with Module.prepend) is reminding me a bit of AspectJ's
pointcuts. Which in turn leads me to think that we are missing something
here:
We don't know and cannot know in advance which kind of scopes the
developer will need to apply his patches.

We have many different ideas flying around how to determine the scope of
the refinement.

a) Local only? Maybe even constrained inside a block?

Good for builder DSLs or the like where you basically want to extend
core objects to make it look more like written sentences than code.

b) Class inheritance?

E.g. If you want to provide some nice class configuration syntax

c) Module namespace?

If you like to use some convenience methods throughout your project
without creating conflicts with extensions that libraries might use in
their own code

d) Stack-down X frames

Black Magic: Patch some behavior inside a single method by wrapping it

e) Thread local

More black magic: Fix some broken interaction between library code.
Stub any kind of method out temporarily without breaking other things in
multi-threaded environments.



So what I am saying is that we don't just need a way to define
refinement namespaces. We also need to let the programmer define where
and when those namespaces get applied. And we need the common cases to
be fast. The madness-driven ones (d and e) can be slow, but can only be
allowed to be slow at those callsites that are affected, not globally.

So I would suggest not providing *any* inheritance at all. Refinements
scopes must be activated in every single module (or possibly even
method) that they should be applied to. They shouldn't even apply to
methods overriding another method when the super-method is
refinement-scoped. If you want to apply them in many places at once you
can do so via metaprogramming.

module FooExt
refine String do

def downcase
upcase
end
end
end



case a)

class ClassA
def bar; end

# apply to all methods when no block is passed
using(FooExt)

def baz; end

# both .bar and .baz are refined now.
# we can apply them retroactively!
# this is important for monkey-patching
end

class ClassB
def foo
"x".downcase => "x"
using_refinements(FooExt) do
"x".downcase => "X"
end
end
end

class ClassC
def bar
"x".downcase
end
def baz
"x".downcase
end
end

o = ClassC.new

o.bar # => "x"
o.send(:using, FooExt, :on => :bar)
o.bar # => "X"
o.baz # => "x"

# acts as instance_eval with refinements
Object.new.using_refinements(FooExt) do
"x".downcase # => "X"
end


case b)

class MyModel < ActiveRecord::Base; end
class SubModel < MyModel; end
class OtherModel < ActiveRecord::Base; end

ActiceRecord::Base.descendants.each do |c|
c.send(:using, FooExt)
end

case c)

# assume this traverses the constants downwards
MyApplicationNamespace.recursive_submodules.each do |mod|
mod.send(:using, FooExt)
end

# only modify callsites for a single method
Bar.instance_method(:test).using(FooExt)

case d)
case e)

# applies refinement to callsites in method :bar in MyClass
# but only if the guard condition is true
# otherwise the unrefined method is used
MyClass.send(:using, FooExt, :on => :bar, :if => lambda do
Thread[:use_string_patches?]
end)

# applies FooExt a dynamic refinement scope
# to *all* String.downcase callsites throughout the application
FooExt.send(:use_everywhere) do
Thread[:use_string_patches?] && caller[1]["<stack match here>"]
end



I think this should demonstrate the power of letting the programmer
decide how refinement scopes are determined instead of having the
language dictate a fixed lookup strategy.

Cases d) and e) are just for demonstration and don't have to be taken
seriously!


But the metaprogramming issues with __send__, respond_to? and
Symbol.to_proc would still remain.

[mame (Yusuke Endoh)]

Issue #4085 has been updated by mame (Yusuke Endoh).


To be honest, I do not follow this discussion at all. I just heard from ko1 that there are still room to discuss this feature.

I propose: will we release the refinement as an "experimental" feature? It is enough for 2.0 to clarify:

* how usage is "defined" (we will ensure the compatibility), and
* how usage is "undefined" (the behavior may change in future)

By doing so, we will be able to improve (or refine) the feature based on actual experiment in 2.0.0.

(I'll come back to this ticket to understand the discussion on a deeper level, after I finished other tasks.
I will be very very happy to create a summary of the discussion.)

--
Yusuke Endoh <ma...@tsg.ne.jp>

----------------------------------------
Feature #4085: Refinements and nested methods

https://bugs.ruby-lang.org/issues/4085#change-33786

[mame (Yusuke Endoh)]

Issue #4085 has been updated by mame (Yusuke Endoh).

P.S. This is just my current impression, but we have no option to remove the whole feature from 2.0.0.

--
Yusuke Endoh <ma...@tsg.ne.jp>

----------------------------------------
Feature #4085: Refinements and nested methods

https://bugs.ruby-lang.org/issues/4085#change-33787

[duerst (Martin Dürst)]

Issue #4085 has been updated by duerst (Martin Dürst).

mame (Yusuke Endoh) wrote:

> I propose: will we release the refinement as an "experimental" feature? It is enough for 2.0 to clarify:
>
> * how usage is "defined" (we will ensure the compatibility), and
> * how usage is "undefined" (the behavior may change in future)
>
> By doing so, we will be able to improve (or refine) the feature based on actual experiment in 2.0.0.

If this means that every time (or the first time) refinements are used in a program, a warning is issued, e.g. "Use of refinements is currently experimental, and implementation may change in future versions of Ruby!", then this would be okay with me. If there is no such warning, I'm affraid that we won't be able to make changes even if we want.

----------------------------------------
Feature #4085: Refinements and nested methods

https://bugs.ruby-lang.org/issues/4085#change-33794

[mame (Yusuke Endoh)]

Issue #4085 has been updated by mame (Yusuke Endoh).

duerst (Martin Dürst) wrote:
> If this means that every time (or the first time) refinements are used in a program, a warning is issued, e.g. "Use of refinements is currently experimental, and implementation may change in future versions of Ruby!", then this would be okay with me. If there is no such warning, I'm affraid that we won't be able to make changes even if we want.

Ideally, only "undefined" behavior should be warned.
If they are automatically indistinguishable,,, it is unfortunate.

Anyway I should study this long long discussion...

--
Yusuke Endoh <ma...@tsg.ne.jp>

----------------------------------------
Feature #4085: Refinements and nested methods

https://bugs.ruby-lang.org/issues/4085#change-33795

[dbussink (Dirkjan Bussink)]

Issue #4085 has been updated by dbussink (Dirkjan Bussink).


Here are few points I'd like to make as to why I think the feature should not be part of 2.0. First of all, there is currently not a single idea / design that describes what refinements are. In this issue different approaches and ideas are explored. This is of course a good thing, but it also means that we haven't come to a conclusion.

This means that if refinements were to be part of Ruby 2.0, they would be an implementation that hasn't been fleshed out, is still being debated and can change in the future. I think doing this would be a great disservice to the Ruby community. This is because we would release a feature that people cannot depend on. It may change in the future, break people's code and cause heavy debates on whether we can actually change it in the future.

As a reference, originally in 1.9, Hash being insertion ordered was defined as an implementation detail of CRuby. The position of CRuby in the community however, means that this also turned into spec since people depend on it. This was basically a decision made for us by the community, neither JRuby or Rubinius could say, for us it's not insertion ordered. This means that if some things are released, what is and what is not defined behavior is not always just up to the implementers. The Ruby community will come to expect certain behavior of certain features and depend on it.

If Ruby 2.0 is released without a properly defined way of how refinements work and that they will be supported in that way well into the future, we very well may end up with an implementation that no one really likes but has to be supported indefinitely.

Ruby is not an experimental language anymore, so asking of the community to accept that this all can change in the future is not the right thing to do. What is part of Ruby the language is something that we should be able to stand behind and say of "we will support this and provide a stable platform for you for the future". If that can't be said of a feature, it should not be added to a released Ruby version.

Even if removing the feature for now is a hard decision, I feel it is the correct. We should not fall into a sunk cost fallacy, where the amount of development on it results in feeling like it could not be removed.

Please realize that nowhere in the argument I've discussed my personal preferences on the feature itself. The thing is that I don't think that should be even necessary at this point, because of all the unclarity about the feature and how it should work.

All in all, I think the Ruby community would be done a great disservice by adding a feature that is still being debated so short before a 2.0 release, does not have clear semantics (looking at the alternatives discussed here) and therefore isn't mature enough to be put out there in the wild.

----------------------------------------
Feature #4085: Refinements and nested methods

https://bugs.ruby-lang.org/issues/4085#change-33803

[trans (Thomas Sawyer)]

Issue #4085 has been updated by trans (Thomas Sawyer).


I was reading this discussion (http://branch.com/b/rubyists-which-would-you-rather-have-in-ruby-2-0-0) about refinements, and read the last post by Magnus Holm (@judofyr) which said:

> I think refinements are useful for some specific tasks; String#camelize is *not* one of them.
> I see no real gain of using `"foo".camelize` over `StringUtils.camelize("foo")`.
>
> No, I think refinements should be used for DSLs. As @wycats mentioned: It's the difference
> between DataMapper.not(:age => 12) and `:age.not => 12`. Or `5.days.ago` instead
> of `DateUtilts.now.subtract(:days => 5)`.
>
> And in that context I think that *lexical* refinements makes a whole lot more sense than
> dynamic refinements; both in terms of debugging and in terms of sanity.

For a while I too was thinking that same thing. But now it seems to me that perhaps this kind of DSL is just bad API design --in particular the whole idea of adding a bunch of methods to Symbol class.

For example, something like `:age.not => 12` could be written as a block DSL instead as `{ age != 12 }`. That's so much sexier anyway.

As for something like `5.days.ago`, I agree with Steve Klabnik who calls such cases more of a "social" problem. It is something that's common enough that it would be better if Ruby had a concept of Duration built-in and some convenient conversion methods on Numeric. At worse there could be single entry point like `5.calendar.days.ago` or maybe `5.th(:day).ago`. Certainly we don't need something as course as `DateUtilts.now.subtract(:days => 5)`, but we could still do something reasonable without a bunch of refinements.

So I guess my point is that refinements are not really even addressing a "20%" problem, it seems more like a "1%" problem. And as cool as they seem, I am not so sure that's enough.

Again, I'll mention that I am a bit afraid of what might happen if refinements become popular. Imagine a library with dozens of various refinements on all the core classes. Might we start seeing DSLs like: `:batter.count(1.ball && 2.strikes)`.

----------------------------------------
Feature #4085: Refinements and nested methods

https://bugs.ruby-lang.org/issues/4085#change-33871

[shugo (Shugo Maeda)]

Issue #4085 has been updated by shugo (Shugo Maeda).


headius (Charles Nutter) wrote:

> ko1's suggestion, as I understand it, was to add a flag to the method table (or method entry) of a refined class/method as a trigger for the call site to search refinements. While writing my blog post, I started to type the sentence "the methods defined in the refinement do not actually go on the class in question", and then I realized: why not?
>
> Currently, Ruby implementations structure the method table as a simple map from names to method bodies. If instead the method table was a map from names to collections of methods, we could use that to choose the appropriate method for a given context.
>
> So, for code like this:
>
> module X
> refine String do
> def upcase; downcase; end
> end
> end
>
> String's method table would contain an entry like this:
>
> {:upcase => {
> :default => <builtin upcase>,
> X => <upcase patched>}
> }
>
> Method lookup would then proceed as normal in all situations. The result of lookup would be a table mapping refinements to methods with a default entry if the method is defined directly on String.

It's an interesting idea.
How does method lookup work if multiple modules are used by using?

module Z
using X
using Y
"foo".upcase
end

Are both X and Y used as a key of the table in the reverse order they are used by using?

And, how does super work?

> I admit I am a bit reluctant to suggest this, because I still have concerns about the feature itself. But it would be possible for call sites to only need a reference to their calling scope (determined at parse time) to implement dynamic refinements without severe impact to normal code. Dynamic refinements, as in module_eval, would work by simply invalidating the call sites they contain.

FYI, in my new implementation (http://shugo.net/tmp/refinement_fix_1119.diff), refined methods are not stored in inline cache, so there's no need to invalidate inline cache for module_eval.
Instead, refined method invocations are slower than the implementation in the trunk HEAD.

> In any case, I would really like more time for this dialog to continue. If we push refinements into Ruby in their current form, we're not giving adequate time to flesh out the edge cases. If we push a partial implementation now, we may be making a future implementation harder and we would not be protecting ourselves from mistakes. I want to work with you to find a definition and implementation of refinements that meets requirements without punishing future Rubyists.

I'm starting to think it's not good to rush to introduce Refinements as an official feature.
What do you think of introducing Refinements as an experimental feature as Endoh-san suggested?
I don't know what does Endoh-san mean by "an experimental feature", but it may require an explicit compile option or a runtime option (e.g., require "refinements" like continuation) to enable it. If Refinements are enabled, warning should be shown not to make the current Refinements de-facto standard.

> I also must apologize for not joining the dialog sooner. This bug was filed in 2010, and the current refinements implementation was pushed to master a few months ago. We should have started discussing a long time ago.

I should also have warned people to think of Refinements sooner. Anyway, thanks for your comments.

----------------------------------------
Feature #4085: Refinements and nested methods

https://bugs.ruby-lang.org/issues/4085#change-33946

[headius (Charles Nutter)]

Issue #4085 has been updated by headius (Charles Nutter).

shugo (Shugo Maeda) wrote:
> headius (Charles Nutter) wrote:

> > {:upcase => {
> > :default => <builtin upcase>,
> > X => <upcase patched>}
> > }
> >
> > Method lookup would then proceed as normal in all situations. The result of lookup would be a table mapping refinements to methods with a default entry if the method is defined directly on String.
>
> It's an interesting idea.
> How does method lookup work if multiple modules are used by using?
>
> module Z
> using X
> using Y
> "foo".upcase
> end
>
> Are both X and Y used as a key of the table in the reverse order they are used by using?

I would think they stack like module includes, so at lookup time we'd see refined methods on String, look in calling scope in reverse order, and use the first refinement we encounter as the key.

> And, how does super work?

Well, I'm still questioning how super should work in general. Refinements are not actually modifying class hierarchy, so the current behavior of super calling the old method seems like magic to me.

You have it implemented currently as though used refinements are "virtually" in the hierarchy beneath the class, similar to prepend. So stacked refinements fire "super" in reverse order:

irb(main):001:0> module X; refine(String) { def upcase; puts 'first'; super; end }; end
=> #<Module:0x007ffbf30a4898>
irb(main):002:0> module Y; refine(String) { def upcase; puts 'second'; super; end }; end
=> #<Module:0x007ffbf3082ae0>
irb(main):003:0> using X
=> main
irb(main):004:0> using Y
=> main
irb(main):005:0> 'foo'.upcase
second
first
=> "FOO"

But there are some oddities when refinements are mixed into multiple elements of the hierarchy:

irb(main):001:0> module A; refine(Numeric) { def blah; puts 'A'; super; end }; end
=> #<Module:0x007ffeeb930d90>
irb(main):002:0> module B; refine(Integer) { def blah; puts 'B'; super; end }; end
=> #<Module:0x007ffeeb90f578>
irb(main):003:0> module C; refine(Fixnum) { def blah; puts 'C'; super; end }; end
=> #<Module:0x007ffeeb8f1870>
irb(main):004:0> using A; using B; using C
=> main
irb(main):005:0> 1.blah
C
NoMethodError: super: no superclass method `blah' for 1:Fixnum
from (irb):3:in `blah'
from (irb):6
from /usr/local/bin/irb-2.0.0:12:in `<main>'

Obviously refined super is not simulating the full hierarchy here. It would be difficult to do so, but I feel like you either need to support super in refinements consistently or not at all.

If I add modules to the same locations in the hierarchy, the supers fire "properly". So basically, refined "super" is only working for one level up from the refinement itself.

irb(main):007:0> module A2; def blah; puts 'A2'; super; end; end
=> nil
irb(main):008:0> module B2; def blah; puts 'B2'; super; end; end
=> nil
irb(main):009:0> module C2; def blah; puts 'C2'; super; end; end
=> nil
irb(main):010:0> class Fixnum; prepend C2; end
=> Fixnum
irb(main):011:0> class Integer; prepend B2; end
=> Integer
irb(main):012:0> class Numeric; prepend A2; end
=> Numeric
irb(main):013:0> 1.blah
C
C2
C
B2
B
A2
A
NoMethodError: super: no superclass method `blah' for 1:Fixnum
from (irb):1:in `blah'
from (irb):7:in `blah'
from (irb):2:in `blah'
from (irb):8:in `blah'
from (irb):3:in `blah'
from (irb):9:in `blah'
from (irb):3:in `blah'
from (irb):13
from /usr/local/bin/irb-2.0.0:12:in `<main>'

The double call of C's blah here is unexpected as well.

Another example showing that refinements don't honor refined hierarchies for "super":

irb(main):026:0> class Foo
irb(main):027:1> def blah; puts 'in Foo'; end
irb(main):028:1> end
=> nil
irb(main):029:0> class Bar < Foo
irb(main):030:1> def blah; puts 'in Bar'; super; end
irb(main):031:1> end
=> nil
irb(main):032:0> module Baz
irb(main):033:1> refine Foo do
irb(main):034:2* def blah; puts 'in Baz'; super; end
irb(main):035:2> end
irb(main):036:1> end
=> #<Module:0x007ffeeb05f978>
irb(main):037:0> using Baz
=> main
irb(main):038:0> Bar.new.blah
in Bar
in Foo
=> nil

Again, inconsistent behavior, but I'm not sure which specification is correct.

FWIW, there's something similar to refinements in the form of extension methods in C# and defender methods in Java 8. It would be worth researching how those features handle super. In Java 8, the defender methods live only on interfaces and are somewhat "virtually" in the hierarchy, so there's a lot of oddities surrounding the process of selecting the proper super method.

> > I admit I am a bit reluctant to suggest this, because I still have concerns about the feature itself. But it would be possible for call sites to only need a reference to their calling scope (determined at parse time) to implement dynamic refinements without severe impact to normal code. Dynamic refinements, as in module_eval, would work by simply invalidating the call sites they contain.
>
> FYI, in my new implementation (http://shugo.net/tmp/refinement_fix_1119.diff), refined methods are not stored in inline cache, so there's no need to invalidate inline cache for module_eval.
> Instead, refined method invocations are slower than the implementation in the trunk HEAD.

I considered this possibility, but are you willing to accept that large parts of Rails code will have slower overall performance because they want to use refinements? I will revise my earlier refinements requirements: refined calls should exhibit exactly the same performance characteristics as regular calls. I believe if refinements go in, many many libraries will want to start using them. We should not force Ruby perf to take a major step backward just by introducing a new and potentially popular feature that has implementation problems.

> > In any case, I would really like more time for this dialog to continue. If we push refinements into Ruby in their current form, we're not giving adequate time to flesh out the edge cases. If we push a partial implementation now, we may be making a future implementation harder and we would not be protecting ourselves from mistakes. I want to work with you to find a definition and implementation of refinements that meets requirements without punishing future Rubyists.
>
> I'm starting to think it's not good to rush to introduce Refinements as an official feature.
> What do you think of introducing Refinements as an experimental feature as Endoh-san suggested?
> I don't know what does Endoh-san mean by "an experimental feature", but it may require an explicit compile option or a runtime option (e.g., require "refinements" like continuation) to enable it. If Refinements are enabled, warning should be shown not to make the current Refinements de-facto standard.

I do not have any objection to refinements being included as an experimental feature.

If it's a compile-time feature, I'm not sure I see the value in having it in 2.0.0 at all; people could download source and build that.

If it's a flag or require, I assume you'd have to enable it to turn on parse/compile-time flagging of refined methods/calls, correct? I think that's easy enough in JRuby too.

----------------------------------------
Feature #4085: Refinements and nested methods

https://bugs.ruby-lang.org/issues/4085#change-33968

[shugo (Shugo Maeda)]

Issue #4085 has been updated by shugo (Shugo Maeda).


headius (Charles Nutter) wrote:

> I would think they stack like module includes, so at lookup time we'd see refined methods on String, look in calling scope in reverse order, and use the first refinement we encounter as the key.

I see.

> > And, how does super work?
>
> Well, I'm still questioning how super should work in general. Refinements are not actually modifying class hierarchy, so the current behavior of super calling the old method seems like magic to me.

It may seem magical, but is intended for use like aspect oriented programming.

> But there are some oddities when refinements are mixed into multiple elements of the hierarchy:
>
> irb(main):001:0> module A; refine(Numeric) { def blah; puts 'A'; super; end }; end
> => #<Module:0x007ffeeb930d90>
> irb(main):002:0> module B; refine(Integer) { def blah; puts 'B'; super; end }; end
> => #<Module:0x007ffeeb90f578>
> irb(main):003:0> module C; refine(Fixnum) { def blah; puts 'C'; super; end }; end
> => #<Module:0x007ffeeb8f1870>
> irb(main):004:0> using A; using B; using C
> => main
> irb(main):005:0> 1.blah
> C
> NoMethodError: super: no superclass method `blah' for 1:Fixnum
> from (irb):3:in `blah'
> from (irb):6
> from /usr/local/bin/irb-2.0.0:12:in `<main>'
>
> Obviously refined super is not simulating the full hierarchy here. It would be difficult to do so, but I feel like you either need to support super in refinements consistently or not at all.

I admit that the above example looks odd. I'd like to fix it if possible.

> The double call of C's blah here is unexpected as well.

It looks odd too.

> Another example showing that refinements don't honor refined hierarchies for "super":
>
> irb(main):026:0> class Foo
> irb(main):027:1> def blah; puts 'in Foo'; end
> irb(main):028:1> end
> => nil
> irb(main):029:0> class Bar < Foo
> irb(main):030:1> def blah; puts 'in Bar'; super; end
> irb(main):031:1> end
> => nil
> irb(main):032:0> module Baz
> irb(main):033:1> refine Foo do
> irb(main):034:2* def blah; puts 'in Baz'; super; end
> irb(main):035:2> end
> irb(main):036:1> end
> => #<Module:0x007ffeeb05f978>
> irb(main):037:0> using Baz
> => main
> irb(main):038:0> Bar.new.blah
> in Bar
> in Foo
> => nil
>
> Again, inconsistent behavior, but I'm not sure which specification is correct.

I and Matz discussed this behavior before, and concluded that Baz's blah should not be called in this case, because Bar's blah is outside the scope where Baz is activated and Refinements do not support local rebinding.
However, I admit that the behavior looks odd.

> FWIW, there's something similar to refinements in the form of extension methods in C# and defender methods in Java 8. It would be worth researching how those features handle super. In Java 8, the defender methods live only on interfaces and are somewhat "virtually" in the hierarchy, so there's a lot of oddities surrounding the process of selecting the proper super method.

I'll check them. Thank you.

> > > I admit I am a bit reluctant to suggest this, because I still have concerns about the feature itself. But it would be possible for call sites to only need a reference to their calling scope (determined at parse time) to implement dynamic refinements without severe impact to normal code. Dynamic refinements, as in module_eval, would work by simply invalidating the call sites they contain.
> >
> > FYI, in my new implementation (http://shugo.net/tmp/refinement_fix_1119.diff), refined methods are not stored in inline cache, so there's no need to invalidate inline cache for module_eval.
> > Instead, refined method invocations are slower than the implementation in the trunk HEAD.
>
> I considered this possibility, but are you willing to accept that large parts of Rails code will have slower overall performance because they want to use refinements? I will revise my earlier refinements requirements: refined calls should exhibit exactly the same performance characteristics as regular calls. I believe if refinements go in, many many libraries will want to start using them. We should not force Ruby perf to take a major step backward just by introducing a new and potentially popular feature that has implementation problems.

I think the performance of refined calls can be improved by new cache dedicated for refinements, but it might still be a little slower than normal calls.

> I do not have any objection to refinements being included as an experimental feature.
>
> If it's a compile-time feature, I'm not sure I see the value in having it in 2.0.0 at all; people could download source and build that.

It may not be worth having such a feature. If people can build it themselves, they can use SVN trunk.

> If it's a flag or require, I assume you'd have to enable it to turn on parse/compile-time flagging of refined methods/calls, correct? I think that's easy enough in JRuby too.

I meant to provide refinements.so, which just publishes Module#refine, Module#using, etc... in Ruby level, like continuation.so.

----------------------------------------
Feature #4085: Refinements and nested methods

https://bugs.ruby-lang.org/issues/4085#change-34020

[headius (Charles Nutter)]

Issue #4085 has been updated by headius (Charles Nutter).


shugo (Shugo Maeda) wrote:
> headius (Charles Nutter) wrote:

> > Well, I'm still questioning how super should work in general. Refinements are not actually modifying class hierarchy, so the current behavior of super calling the old method seems like magic to me.
>
> It may seem magical, but is intended for use like aspect oriented programming.

The interaction between super chains and refinements bothers me.

A new idea...

Because we'd like refinements to act like they live in the class hierarchy...let's just make them live in the class hierarchy.

So...we start out with String < Object. Do a refinement:

module X
refine String do

def blah; end
end
end

Now the String hierarchy looks like this: RefinedByX < String < Object

It's rather prepend-like. Lookup proceeds as normal for a given string by getting the metaclass (now RefinedByX). However when refined intermediate classes are encountered, the calling scope is queried to see if it has activated that refinement.

CallSite pseudo-code:

metaclass = obj.metaclass
while metaclass != null
if metaclass.refinement?
unless caller_scope.include_refinement? metaclass
# skip refinement
metaclass = metaclass.superclass
next
end
end

method = metaclass.search_method method_name
return method if method
metaclass = metaclass.superclass
end

Super logic than can operate as it should, using the refined scope to do the subsequent super lookup, which it finds in the hierarchy in the normal way. Caching also proceeds largely the same, based on the target object's hierarchy only; the only difference is whether refined elements in the hierarchy are included in the search.

Not sure about other edge cases for this, but it's another way to look at it, and I think this starts to move refinements more directly toward being structured in the same way as module inclusion rather than based on the more magical concept of "current frame overlay modules".

----------------------------------------
Feature #4085: Refinements and nested methods

https://bugs.ruby-lang.org/issues/4085#change-34056

[matz (Yukihiro Matsumoto)]

Issue #4085 has been updated by matz (Yukihiro Matsumoto).


Since there still remain undefined corner case behavior in refinements, and the time is running out, I decided not to introduce full refinement for Ruby 2.0. The limited Ruby 2.0 refinement spec will be:

* refinements are file scope
* only top-level "using" is available
* no module scope refinement
* no refinement inheritance
* module_eval do not introduce refinement (even for string args)

In addition, Module#include should add refinements to included modules, e.g.

module R1

refine String do
def bar

p :bar
end
end
end

module R2
include R1

refine String do
def foo

p :foo
end
end
end

using R2
"".foo
"".bar

module R1
refine String do
def bar; p :bar end
end
end

module R2
include R1
refine String do
def foo; p :foo end
end
end

using R2
"".foo
"".bar # does not work now

You can treat top-level "using" as soft-keyword, as long as it does not change the behavior (but performance).

Matz.

----------------------------------------
Feature #4085: Refinements and nested methods

https://bugs.ruby-lang.org/issues/4085#change-34117

[shugo (Shugo Maeda)]

Issue #4085 has been updated by shugo (Shugo Maeda).

matz (Yukihiro Matsumoto) wrote:
> Since there still remain undefined corner case behavior in refinements, and the time is running out, I decided not to introduce full refinement for Ruby 2.0. The limited Ruby 2.0 refinement spec will be:

I don't understand what do you mean by these constraints. Let me ask some questions.

> * refinements are file scope
> * only top-level "using" is available
> * no module scope refinement

Do these constraints just mean that main.using is available, but Module#using is not?
How should the following code behave?

module R

refine String do
def foo; p :foo; end
end

"".foo # (a)
end
"".foo # (b)

Currently, (a) prints :foo, and (b) raises a NoMethodError.
And, how about the following example, where a nested module is defined?

module R

refine String do
def foo; p :foo; end
end

module M
"".foo
end
"".foo
end
"".foo

If the behavior in the new spec is the same as the current implementation,
I don't know well why Module#using should be removed.

> * no refinement inheritance
> * module_eval do not introduce refinement (even for string args)

I understand these.

> In addition, Module#include should add refinements to included modules, e.g.

This is very different from the current feature, so we need a discussion about it.
What does "add refinements" mean here?
There are two aspects about refinement addition. They are defined in modules by Module#refine, and activated in certain scopes by using.
Does "to add refinements" mean to define (or inherit indirectly) refinements in modules, or to activate refinements in modules, or both of them?

For example, how should the following code behave?

module R1
refine String do
def bar
p :bar
end
end
end

module R2
include R1
refine String do
def foo
p :foo
end
end

"".foo
"".bar
end

Finally, how super in refinements should behave in the new spec?

----------------------------------------
Feature #4085: Refinements and nested methods

https://bugs.ruby-lang.org/issues/4085#change-34121

[dbussink (Dirkjan Bussink)]

Issue #4085 has been updated by dbussink (Dirkjan Bussink).


The last two comments here again confirm my view that we should not add refinements to Ruby 2.0. Apparently it is not even clear to the people developing Ruby itself what the feature means. If there is ambiguity and questions like this have to asked, how can we add this with a straight face?

I feel we should not treat Ruby as an experimental place where we can try stuff like this that affects the whole language. Please consider removing refinements for 2.0, since this discussions keeps confirming that the ideas are not clear and not fleshed out. Adding half done features at the language level mean a serious disservice to the Ruby community and does not clearly communicate the path to the future.

----------------------------------------
Feature #4085: Refinements and nested methods

https://bugs.ruby-lang.org/issues/4085#change-34131

[Gibheer (Stefan Radomski)]

Issue #4085 has been updated by Gibheer (Stefan Radomski).


Hello,

I followed the discussion of refinements not as close as most others commenting here, but the state I saw till now was a bit disturbing. As I understand it, there are some definition, implementation and performance problems in MRI or other ruby implementations, which have to be considered.

When I understood Matz correct, he wants to add refinements in a different way in MRI 2.0. As some library maintainers already said, that they want to use refinements because it helps them, I think that is not a helpful step. Refinements will change afterwards in behavior and cause incompatibilities for future versions of refinements and ruby implementations.

As you already put so much effort into it, it would be bad to see it split and cut down to a state, which does not represent the original idea behind it. Take a step back and don't include it in 2.0 but instead use the time to polish it, write specifications and tests. With that you can give developers something to fully understand the trade-offs/impact of refinements and the use-case for the feature. This can also help developers working on the various ruby implementations.

thank you

----------------------------------------
Feature #4085: Refinements and nested methods

https://bugs.ruby-lang.org/issues/4085#change-34134

[trans (Thomas Sawyer)]

Issue #4085 has been updated by trans (Thomas Sawyer).


=begin
> refinements are file scope

That's a very interesting and significant recalibration of refinements. On one hand it certainly simplifies the whole scoping issue. On the other, would it mean we would have to reiterate `using` for every file, i.e. we couldn't gain a `using` via require?

> cat foo.rb
require 'facets'
using Facets

> cat bar.rb
require 'foo.rb'
# Are we using Facets ?

I suppose the answer would have to be "no", otherwise anyone who required it would have to be using it too, which would defeat the point whole of refinements.

Unfortunately it is a little annoying to have to put `using` at the top of every file. Is it possible to determine the gem a file comes from? Is it conceivable to have something like:

__GEM__ #=> 'foo-1.2.1'

If so then `using` could be tied per-gem rather then just per file.
=end

----------------------------------------
Feature #4085: Refinements and nested methods

https://bugs.ruby-lang.org/issues/4085#change-34136

[matz (Yukihiro Matsumoto)]

Issue #4085 has been updated by matz (Yukihiro Matsumoto).


@trans I am thinking of combination of require and using, but I don't want to put half baked idea into Ruby 2.0.
So I leave it to the future.

Matz.

----------------------------------------
Feature #4085: Refinements and nested methods

https://bugs.ruby-lang.org/issues/4085#change-34137

[rosenfeld (Rodrigo Rosenfeld Rosas)]

Issue #4085 has been updated by rosenfeld (Rodrigo Rosenfeld Rosas).


@matz, I like the suggestion of "using" being file scoped, although I don't understand why you want it to be applied to top-level object only... Would you mind in explaining why you think this is more interesting or would simplify things?

----------------------------------------
Feature #4085: Refinements and nested methods

https://bugs.ruby-lang.org/issues/4085#change-34140

[brainopia (Ravil Bayramgalin)]

Issue #4085 has been updated by brainopia (Ravil Bayramgalin).


Yeah, "using" at file scope will mean a bit of boilerplate to specify it across several files, but since there are no leaking abstractions I think new proposal answers all concerns about debugging and readability.

And in future releases refinements can be easily improved to incorporate other features while staying backward compatible. Well done!

----------------------------------------
Feature #4085: Refinements and nested methods

https://bugs.ruby-lang.org/issues/4085#change-34141

[steveklabnik (Steve Klabnik)]

Issue #4085 has been updated by steveklabnik (Steve Klabnik).


Wasn't a 'feature freeze' declared on October 24th? Why are major additions to the language being modified post-freeze, so close to the deadline for an actual release?

----------------------------------------
Feature #4085: Refinements and nested methods

https://bugs.ruby-lang.org/issues/4085#change-34144

[steveklabnik (Steve Klabnik)]

Issue #4085 has been updated by steveklabnik (Steve Klabnik).


Wasn't a 'feature freeze' declared on October 24th? Why are major additions to the language being modified post-freeze, so close to the deadline for an actual release?
----------------------------------------
Feature #4085: Refinements and nested methods

https://bugs.ruby-lang.org/issues/4085#change-34145

[Yukihiro Matsumoto]

In message "Re: [ruby-core:50299] [ruby-trunk - Feature #4085] Refinements and nested methods"

on Thu, 29 Nov 2012 15:02:03 +0900, "shugo (Shugo Maeda)" <red...@ruby-lang.org> writes:

|> * refinements are file scope
|> * only top-level "using" is available
|> * no module scope refinement
|
|Do these constraints just mean that main.using is available, but Module#using is not?

Yes, only main.using should be available. No Module#using (for 2.0).

|How should the following code behave?
|
|module R
| refine String do
| def foo; p :foo; end
| end
| "".foo # (a)
|end
|"".foo # (b)
|
|Currently, (a) prints :foo, and (b) raises a NoMethodError.

Refinements will be available only from:

* the scope where refinements are added by calling "using"
* or inside of refine blocks

Inside of refine blocks (not whole module scope) might be
controversial, but I think it's OK to restrict refinements there. As
a result, both (a) and (b) raise NoMethodError. But "".foo can be called
from within the refine block.

|And, how about the following example, where a nested module is defined?
|
|module R
| refine String do
| def foo; p :foo; end
| end
|
| module M
| "".foo
| end
| "".foo
|end
|"".foo

Every "".foo in the above example should raise NoMethodError, because
they are outside of refine blocks. I admit I've been less careful
about nested refinement modules. For nested refinement modules, it
should behave as following:

>module R
> refine String do
> def foo; p :foo; end
> end
>
> module M

> refine Array do
> "".foo # => OK
> end
> end
>end
>
>using R::M
> "".foo # => NG

|> In addition, Module#include should add refinements to included modules, e.g.
|
|This is very different from the current feature, so we need a discussion about it.
|What does "add refinements" mean here?
|There are two aspects about refinement addition. They are defined in modules by Module#refine, and activated in certain scopes by using.
|Does "to add refinements" mean to define (or inherit indirectly) refinements in modules, or to activate refinements in modules, or both of them?

I meant included module will provide refinement of combination of including
module(s) and the module itself.

|For example, how should the following code behave?
|
| module R1
| refine String do
| def bar
| p :bar
| end
| end
| end
|
| module R2
| include R1
| refine String do
| def foo
| p :foo
| end
| end
| "".foo
| "".bar
| end

Since all calls of "".foo and "".bar are outside of refine blocks,
they should raise NoMethodError. But R2 should provide refinement to
add method #bar and #foo to String class.

|Finally, how super in refinements should behave in the new spec?

Refinements should come before normal methods, so super in the normal
method will not see a refined method, and super in the refined method
will see a normal method (or other refined method if refinements are
stacked).

The whole point is separation of defining refinements (for library
developers) and using refinements (for library users).

Any more questions?

matz.

[myronmarston (Myron Marston)]

Issue #4085 has been updated by myronmarston (Myron Marston).


I find refinements to be a _very_ interesting idea, but, as with many others, it concerns me that there aren't rubyspecs behind it, there's still undefined edge cases, the main language maintainers are still discussing how it should work, and the existing implementation has a perf impact.

In the past, I've heard RSpec used as a main example of a library that could really benefit from refinements. As of yesterday, I'm the lead maintainer of RSpec, and I'm doubtful whether or not we'll ever use them. Given the fact that we need to maintain support for 1.8.7, 1.9.2 and 1.9.3 for as long as its common for gem authors to support those versions (since we'd like RSpec to be useable to test all the commonly supported rubies), we won't be in a position to use refinements for many years. In the meantime, we've already started significantly reducing the number of monkey patches RSpec does to Object in the 2.x releases, and I'm hoping to reduce this even more for 3.0.

Really, I think a feature like refinements needs to see some *actual production usage* before being included in the language. It needs a long vetting period. I think a key part of rails' success is the fact that it was extracted from a production app, rather than being built on its own first.

I haven't read this whole thread (it's huge, and it's hard to find the time to read every comment), so I have no idea if this has been discussed or not...but here's an idea I just had: For 2.0, extract refinements out of the core language into a c-extension gem. Ask the community to give refinements a shot. Encourage actual production usage. Learn from real-world usage, and clarify the spec before considering putting them back in the core language.

Of course, this idea depends upon it being doable to extract refinements into a gem; given the affect they have at the core language level, that may not be doable. But it would be great to have some way for refinements to be released so people could start trying them *without* putting them directly in MRI -- that gives the language designers and maintainers time to learn from real world usage and refine refinements (sorry for the bad pun).

Myron
----------------------------------------

Feature #4085: Refinements and nested methods

https://bugs.ruby-lang.org/issues/4085#change-34153

[trans (Thomas Sawyer)]

Issue #4085 has been updated by trans (Thomas Sawyer).


@headius Your idea of having refinements in class hierarchy is very interesting. It is remarkably similar to Cuts. If you are not familiar with cuts see http://rubyworks.github.com/cuts/rcr.html (yea an RCR, remember those!). In fact, I think it means that refinements could be seen as a subset of cuts. In the case of refinements, the methods apply only if the refinement is used in the given scope. This suggests an important piece of functionality from the original cuts concept could use, namely, a condition that applies to the cut's application. If we added that, then refinements could be implement via cuts with:

cut RefinementX < String
def self.applies?(scope, method=nil)
scope.include_refinement? self
end

def blah; end
end

----------------------------------------
Feature #4085: Refinements and nested methods

https://bugs.ruby-lang.org/issues/4085#change-34154

[rosenfeld (Rodrigo Rosenfeld Rosas)]

Issue #4085 has been updated by rosenfeld (Rodrigo Rosenfeld Rosas).


Myron, congrats for the lead maintainer position in RSpec. You're really doing a great job there.

With regards to your suggestion, I think I've read another comment suggesting something like you want but most probably more doable for the time being. Instead of extracting the code to an external gem (which I would guess it won't be done) it would be disabled instead by default (using or refine would raise NoMethodError). Then it could be enabled for experimental usage with either some external c-extension gem that would expose it or through some ruby flag.

I guess this should be much more simpler to implement and would have the same effect you desire, right? I also vote for such approach and delay refinements a bit more until its specs are refined.

----------------------------------------
Feature #4085: Refinements and nested methods

https://bugs.ruby-lang.org/issues/4085#change-34155

[myronmarston (Myron Marston)]

Issue #4085 has been updated by myronmarston (Myron Marston).


@rosenfeld -- yeah, that sounds like it would work fine, as long as there wasn't a perf impact just by having the code there in MRI w/ it turned off.

----------------------------------------
Feature #4085: Refinements and nested methods

https://bugs.ruby-lang.org/issues/4085#change-34157

[Eric Hodel]

On Nov 29, 2012, at 4:56, "steveklabnik (Steve Klabnik)" <st...@steveklabnik.com> wrote:

> Wasn't a 'feature freeze' declared on October 24th? Why are major additions to the language being modified post-freeze, so close to the deadline for an actual release?

Matz decided which features would be in Ruby 2.0.0 on October 24. Between then and the code freeze coming in December we have been completing those accepted features, but not accepting new features.