[ruby-core:67346] Future of test suites for Ruby

[Charles Oliver Nutter]

I'll try to be brief so we can discuss all this. tl;dr: RubySpec is
valuable, MRI tests are valuable, we need to better utilize both of
them.

As you may have heard, Brian Shirai officially ended the RubySpec
project last week:
http://rubini.us/2014/12/31/matz-s-ruby-developers-don-t-use-rubyspec/

I believe this is a good opportunity for us to address concerns about
how Ruby behavior is being tested over time.

I think we can agree on the following facts:

* ruby-core devs *have* been using RubySpec on and off, and at least
some of us see value in continuing to do so. naruse already maintains
a fork at github/naruse/rubyspec.

* MRI's suite is not nearly as bad as the post claims. I have done
some coverage measurements that show it doing at least as well as
RubySpec for core classes, and significantly better for standard
libraries.

* MRI's tests are often hard to read, too large, undercommented.
There's a lot of room for improvement, but this is obviously an
organic codebase that has had many contributors.

* Ugly tests are harder for users (and contributors) to grok. Ugly
tests are less attractive targets for contribution.

* RubySpec represents a valuable set of tests for Ruby behavior. There
will be a large overlap with MRI's tests, but RubySpec does things
that MRI's suite does not (and vice versa).

* RubySpec is not gone; Rubinius will still need a suite to verify
Ruby compatibility. RubySpec lives in the Rubinius repository now.

There's a number of paths forward for us right now.

* Do nothing. Run the last public RubySpec dump periodically to ensure
we don't regress, as has been done with naruse's fork. New tests
continue to go into MRI suite.

This is the status quo, and I don't think anyone benefits from it
other than folks who dislike change.

* Begin an effort to bring missing tests from RubySpec into MRI's
suite and to clean up and improve MRI's suite. Abandon RubySpec at
some point.

I think *some* effort should be made to improve MRI's suite in the
short term, since we obviously aren't going to throw it away. I don't
like the idea of spending a lot of time porting tests one way or the
other...at least until we've picked a winner. I don't like the idea of
abandoning RubySpec, since it has many valuable features.

* Take over RubySpec as a fork and begin to maintain it as a secondary
suite for MRI. Encourage contributors and committers to write specs
rather than tests.

I think this is possible. The barrier to contributing to RubySpec is
much lower if we're maintaining a fork of it, and I think that
eliminates many obstacles. I also feel like the future needs to be in
either RubySpec or a much cleaner version of MRI's suite, and there's
less work to start using the former now.

...

So that's what I have for now. Give me your thoughts...anything goes.

[Thomas E Enebo]

Adding to this list:

* I have little doubt MRI developers would prefer committing to a single repository (MRI), rather than two repositories.  As a JRuby developer, I am happy contributing to what MRI developers prefer.  I am resolved that I will be contributing to two repositories regardless of what is decided.

* I would like to have commit access to whatever MRI developers prefer.   I am guessing that is MRI's repository.  If it is MRIs current test suite (which is my current assumption), then I would like to help breaking up the granularity of the tests more and possibly help to address some of the technical critiques Brian brought up in his blog article last week.

* We need to make an extra simple process for non-MRI developers to submit tests.  Up to this point, I tell people: "Add a test and submit to MRI repo or add a spec to rubyspec".  At this point I want to tell them one thing and I want it to be easy and unambiguous.

So main point here is I am willing to do what MRI devs want to do...

-Tom

--

blog: http://blog.enebo.com       twitter: tom_enebo
mail: tom....@gmail.com

[Tanaka Akira]

2015-01-06 7:18 GMT+09:00 Charles Oliver Nutter <hea...@headius.com>:
> I'll try to be brief so we can discuss all this. tl;dr: RubySpec is
> valuable, MRI tests are valuable, we need to better utilize both of
> them.

I like a smaller test framework.
mspec is larger than our minitest4-based one.
Ruby implementation is sometimes broken.
Larger test framework tends to cause a problem (such as SEGV) in the
test framework itself.
It is difficult to invesitigate the problem for larger test framework.
I'm tired for this situtation until Ruby 1.8 which use old test/unit
(ancestor of test-unit gem).
I'm happy in this aspect since Ruby 1.9.

RubySpec uses "should" style which is not recommended by RSpec now.
http://myronmars.to/n/dev-blog/2012/06/rspecs-new-expectation-syntax
I guess people familiar with "should" syntax will decrease.

Changing a tool is not enough for readabiliy.
Who wrote hard-to-read tests in a tool will still write hard-to-read
tests in another tool.
Some kind of feedback for hard-to-read tests should be considered to
address readability.
It is a long way to go, regardless of a tool.
If such feedback is not designed, changing a tool causes more trouble
than benefits.
--
Tanaka Akira

[Eric Wong]

Charles Oliver Nutter <hea...@headius.com> wrote:

> * Begin an effort to bring missing tests from RubySpec into MRI's
> suite and to clean up and improve MRI's suite. Abandon RubySpec at
> some point.

This is ideal (but also a lot of effort). Other Ruby implementations
ought to be able to use MRI's test suite easily.

> I don't like the idea of abandoning RubySpec, since it has many
> valuable features.

Valuable features outside of specs themselves? Can you elaborate?

Like akr, I prefer a smaller test library with fewer surprises to ease
debugging. One recent example was Bug #10685 where you and Chris both
had trouble reproducing the test as a standalone case.

> * Take over RubySpec as a fork and begin to maintain it as a secondary
> suite for MRI. Encourage contributors and committers to write specs
> rather than tests.
>
> I think this is possible. The barrier to contributing to RubySpec is
> much lower if we're maintaining a fork of it, and I think that
> eliminates many obstacles. I also feel like the future needs to be in
> either RubySpec or a much cleaner version of MRI's suite, and there's
> less work to start using the former now.

I wouldn't mind this, except there's a major barrier to some folks
(OK, perhaps I'm the only one :<):

Contributing to RubySpec requires an account and accepting a
Terms-of-Service on a proprietary service provider.

We use Ruby freely without accepting any ToS from for-profit
company; so I hope we may all contribute without jumping through
that hoop as well.

[Marc-Andre Lafortune]

Hi,

On Mon, Jan 5, 2015 at 4:18 PM, Charles Oliver Nutter
<hea...@headius.com> wrote:
> * Take over RubySpec as a fork and begin to maintain it as a secondary
suite for MRI. Encourage contributors and committers to write specs
rather than tests.

I really like this approach. I would be glad to help with maintaining this.

On Mon, Jan 5, 2015 at 5:25 PM, Tanaka Akira <a...@fsij.org> wrote:
> RubySpec uses "should" style which is not recommended by RSpec now

This is true. I believe it would be easy to modify the existing
codebase to use "expect" style instead and I volunteer to do this.

> Larger test framework tends to cause a problem (such as SEGV) in the test framework itself.

I'd say that a pure Ruby test framework that creates more SEGV is a
good thing. I want to know of any SEGV in pure Ruby, whatever the
means to get it!

I can't remember encountering actual unwanted problems with mspec.

> Changing a tool is not enough for readability.

This is true in theory (and it is possible to write obscure code in
Ruby and very clear code in C). Still, in practice, the tools and
conventions make a big impact (like the average Ruby code vs the
average C code).

[Tanaka Akira]

2015-01-06 7:18 GMT+09:00 Charles Oliver Nutter <hea...@headius.com>:

> I'll try to be brief so we can discuss all this. tl;dr: RubySpec is
> valuable, MRI tests are valuable, we need to better utilize both of
> them.

Other points.

It is difficult to distinguish between specification and non-specification.
If the distinction is required before writing a test, it discourage
writing a test.
Also, non-specification behavior can be de facto standard eventually and
finaly be a specification.
I think the distinction should be easily changed after a test is written.

It is not appropriate to block a release arbitrary from outside of project.
If a specification is changed too fast or changed just before a release,
it is nearly impossible to release a project without test failures.
So what tests to be applied should be choosen by each project.

Specification is version dependent.
We change specification to improve language and library.
However the improved specification is not applicable for older implementations.
RubySpec is interesting here but even RubySpec doesn't support too old Ruby.
RubySpec drops Ruby 1.9.x at 2013-10-21 (commit
0e4e64cfd431f6f7aec0171af80065154b377561)
while Ruby 1.9.3 is still supported.
(The end of security fix for Ruby 1.9.3 is 2015-02-23. It is still alive.)
https://www.ruby-lang.org/en/news/2014/01/10/ruby-1-9-3-will-end-on-2015/
This may be similar to above: what tests to be applied should be
choosen by each version.

I feel "excludes" feature recently added by [Feature #10682] is a good approach.
https://bugs.ruby-lang.org/issues/10682
It makes easy to choose what tests to be applied.

Of course, tests for mature behavior (such as behaviors specified in
ISO spec.) can
be extracted to choose them easily.
--
Tanaka Akira

["Martin J. Dürst"]

On 2015/01/06 07:18, Charles Oliver Nutter wrote:
> I'll try to be brief so we can discuss all this. tl;dr: RubySpec is
> valuable, MRI tests are valuable, we need to better utilize both of
> them.

I agree. I think that one problem with test frameworks in general is
that they are too easy to write, and people have too many opinions about
them, so there are too many variations.

Ruby always has had an inclusive rather than prescriptive culture (e.g.
when compared to Python), so the fact that there are two test suites may
not be that bad. Also, mixing and matching things, and combining stuff
from different sources, is something that is very frequent in Japanese
culture.

> As you may have heard, Brian Shirai officially ended the RubySpec
> project last week:
> http://rubini.us/2014/12/31/matz-s-ruby-developers-don-t-use-rubyspec/

I have read it. As usual, Brian is very good at stating his view :-(.

> I believe this is a good opportunity for us to address concerns about
> how Ruby behavior is being tested over time.
>
> I think we can agree on the following facts:
>
> * ruby-core devs *have* been using RubySpec on and off, and at least
> some of us see value in continuing to do so. naruse already maintains
> a fork at github/naruse/rubyspec.

As far as I have heard from Yui, he has good reasons why it's forked.

I have tried to run it recently, on Cygwin with cruby trunk. My first
comment would be that I got some 'F's, and after about 5 lines of
....F.... output, it got stuck (not even responding to Ctrl-C). I had no
idea how far it got, or what went wrong, so I abandoned it for the
moment. But this is just my personal experience.


I think contributions to both MRI's test suite and to RubySpec are very
welcome. We are getting lots of small but valuable contributions to the
documentation from what I understand are people who use this as a way to
study Ruby. Maybe something similar could work for tests; it would be
somewhat more difficult than working on documentation but may have a
similar educational effect and likewise produce very valuable improvements.

Regards, Martin.

[Charles Oliver Nutter]

On Mon, Jan 5, 2015 at 5:25 PM, Tanaka Akira <a...@fsij.org> wrote:

> I like a smaller test framework.
> mspec is larger than our minitest4-based one.

That is indeed true, and mspec may be more of a barrier to running the
tests than minitest (e.g. for new implementations). However, there's a
lot of helper logic in MRI's tests that also make it hard for new
impls to run the tests.

That could be fixed more easily than simplifying mspec, I would guess.

> Ruby implementation is sometimes broken.
> Larger test framework tends to cause a problem (such as SEGV) in the
> test framework itself.
> It is difficult to invesitigate the problem for larger test framework.
> I'm tired for this situtation until Ruby 1.8 which use old test/unit
> (ancestor of test-unit gem).
> I'm happy in this aspect since Ruby 1.9.

It would be helpful if the entire suite used only minitest4 but it
appears some suites use test/unit. It would also have been nice for
minitest to stay compatible :-( Is there interest in making the test
suite minitest5 compatible, since we have removed minitest from
stdlib?

> RubySpec uses "should" style which is not recommended by RSpec now.
> http://myronmars.to/n/dev-blog/2012/06/rspecs-new-expectation-syntax
> I guess people familiar with "should" syntax will decrease.

Good point. I have no preference for one over the other.

> Changing a tool is not enough for readabiliy.
> Who wrote hard-to-read tests in a tool will still write hard-to-read
> tests in another tool.

Very good point. Some of these things are hard to test.

> Some kind of feedback for hard-to-read tests should be considered to
> address readability.
> It is a long way to go, regardless of a tool.
> If such feedback is not designed, changing a tool causes more trouble
> than benefits.

This is actually very insightful. Tom and I were chatting about this
yesterday, and it seems like one of the first steps would be adopting
an official coding standard for tests. That would cover new tests and
give the community a clear path toward fixing the existing tests.

Is there an official ruby-core style guide for Ruby code?

- Charlie

[Charles Oliver Nutter]

On Tue, Jan 6, 2015 at 1:50 AM, "Martin J. Dürst"
<due...@it.aoyama.ac.jp> wrote:
> Ruby always has had an inclusive rather than prescriptive culture (e.g. when
> compared to Python), so the fact that there are two test suites may not be
> that bad. Also, mixing and matching things, and combining stuff from
> different sources, is something that is very frequent in Japanese culture.

I'm getting the impression that half the folks like RubySpec better
and half the folks like the MRI suite better, and both have fair
reasons.

At the moment there's a tremendous amount of inertia behind the MRI
suite and far more tests go there than to RubySpec over time. New
features, modified features, and bugs all have tests added to MRI's
suite first...probably because it's "there" in the repository.

That is, of course, why JRuby considers MRI its primary compatibility suite.

I have run coverage reports for both projects on JRuby.

MRI: http://headius.github.io/mri-coverage/

RubySpec: http://headius.github.io/rubyspec/-coverage/

Note that the aggregate numbers are not useful because RubySpec no
longer includes any tests for the standard library (they were
transplanted into Rubinius's "rubysl" projects) and JRuby does not run
all tests/specs from either suite.

Individual classes and methods do have useful numbers though. For
example, RubyString (which implements String) is 85% covered by MRI's
suite and 79% covered by RubySpec.

RubyString under MRI suite:
http://headius.github.io/mri-coverage/org.jruby/RubyString.html#dn-a
Highlighted source:
http://headius.github.io/mri-coverage/org.jruby/RubyString.java.html

RubyString under RubySpec:
http://headius.github.io/rubyspec-coverage/org.jruby/RubyString.html#dn-a
Highlighted source:
http://headius.github.io/rubyspec-coverage/org.jruby/RubyString.java.html

Other cases have slightly more coverage running RubySpec.

Overall, coverage of the core classes seems roughly the same between
MRI's suite and RubySpec, but if either suite is to be the "winner" we
need reports like this to ensure we maintain test coverage.

> I think contributions to both MRI's test suite and to RubySpec are very
> welcome. We are getting lots of small but valuable contributions to the
> documentation from what I understand are people who use this as a way to
> study Ruby. Maybe something similar could work for tests; it would be
> somewhat more difficult than working on documentation but may have a similar
> educational effect and likewise produce very valuable improvements.

Perhaps the biggest benefit of RubySpec is that all specs are small
and contain an English specdoc string describing what they test.
Compare that to MRI where you may have one method called
"test_function" that tests 100 different behaviors and you are left
reading the code to figure it out.

The specdoc requirement is not necessarily a *good* think, though.
Today I was fixing some bugs in JRuby and found that the related
RubySpec's English description described the exact *opposite*
behavior. English descriptions, just like comments, must be
maintained. And if all specs must have a specdoc string, then
non-English speakers are required to compose formal specdoc English
just to write a test.

That said, if we broke up the large test methods in MRI's suite and
added one-liner comments indicating what they do (perhaps with rdoc
formatting?) it would go a long way toward better readability.

I'm taking notes on what an "MRI test suite coding guide" might look like.

- Charlie

[Andrew Vit]

On 15-01-05 18:37, Marc-Andre Lafortune wrote:
> On Mon, Jan 5, 2015 at 5:25 PM, Tanaka Akira <a...@fsij.org> wrote:
>> RubySpec uses "should" style which is not recommended by RSpec now
>
> This is true. I believe it would be easy to modify the existing
> codebase to use "expect" style instead and I volunteer to do this.

Whatever the decision will be, there is a tool for this:
http://yujinakayama.me/transpec/

Andrew Vit

[Marc-Andre Lafortune]

Hi,

On Fri, Jan 9, 2015 at 7:09 PM, Andrew Vit <and...@avit.ca> wrote:
>
> Whatever the decision will be, there is a tool for this:
> http://yujinakayama.me/transpec/

Nice, I didn't know about it. Should make that transition trivial.

[Benoit Daloze]

Hello,

I think we all agree both test suites are useful.

The big question remains of what do we do with the existing RubySpec tests.

I do not think it is reasonable to migrate one suite entirely over the other as this is too much work and there are various reasons to prefer one over the other in different contexts.

But, I would like to have the possibility to still contribute to RubySpec in a way that any implementation may use it
and to merge efforts coming from different sources to improve RubySpec.

- The first step is to bring nurse/rubyspec up to date with rubyspec/rubyspec archive's branch.

I am willing to do that if no one has an objection against it. Any help is welcome of course! (just email me)

- The second step is really to choose a canonical RubySpec repository, to avoid "death by too much forks".

This repository should only contain RubySpec tests for practical reasons.

We should allow many specs contributors to take part in merging changes and maintaining specs.
I think this was a fatal flaw of rubyspec/rubyspec in that too few people had the large burden of merging and maintaining the specs.

The main existing repository I see today is nurse/rubyspec.
I am thinking the process could be similar to handling pull requests on ruby/ruby in that some contributors would provide feedback and merge them.
The CI is very useful in this regard to ensure MRI is not broken inadvertently.

- The third step is to decide what to do about new specs which are not contributed to the canonical repository directly.

This is worth another discussion and I think it is wiser to first achieve the two first steps before discussing this in more details.

I have been contributing to MRI, JRuby, RubySpec, and more recently on a new Ruby implementation.
I really think we should not let RubySpec rot, as it is a valuable test suite.
I spoke to people from each main Ruby implementation and they all wish for a bright future for Ruby tests and specs, so let's do this!

What do you think?

[Benoit Daloze]

On 17 February 2015 at 13:32, Benoit Daloze <ereg...@gmail.com> wrote:

- The second step is really to choose a canonical RubySpec repository, to avoid "death by too much forks".

This repository should only contain RubySpec tests for practical reasons.

We should allow many specs contributors to take part in merging changes and maintaining specs.
I think this was a fatal flaw of rubyspec/rubyspec in that too few people had the large burden of merging and maintaining the specs.

The main existing repository I see today is nurse/rubyspec.
I am thinking the process could be similar to handling pull requests on ruby/ruby in that some contributors would provide feedback and merge them.
The CI is very useful in this regard to ensure MRI is not broken inadvertently.

I think it would make sense in that case to move nurse/rubyspec to ruby/rubyspec for clarity.

[SHIBATA Hiroshi]

Hi, Benoit.

> - The first step is to bring nurse/rubyspec up to date with
> rubyspec/rubyspec archive's branch.
> I am willing to do that if no one has an objection against it. Any help is
> welcome of course! (just email me)

I will help to this work. 1 month ago, I investigate rubyspec's archive branch.
I found following issues:

* rubyspec removed to support Ruby 1.9.3
* but MRI 1.9.3 is ended at Feb. 23, it's not problem now.
* some spec is failing on 2.3.0

> - The second step is really to choose a canonical RubySpec repository, to
> avoid "death by too much forks".
> This repository should only contain RubySpec tests for practical reasons.
> We should allow many specs contributors to take part in merging changes and
> maintaining specs.
> I think this was a fatal flaw of rubyspec/rubyspec in that too few people
> had the large burden of merging and maintaining the specs.
>
> The main existing repository I see today is nurse/rubyspec.
> I am thinking the process could be similar to handling pull requests on
> ruby/ruby in that some contributors would provide feedback and merge them.
> The CI is very useful in this regard to ensure MRI is not broken
> inadvertently.

> I think it would make sense in that case to move nurse/rubyspec to ruby/rubyspec for clarity.

I agreed with your suggestion. I will coordinate with nurse. Please
wait few days.
(PS. I think it's better to move nurse/mspec to ruby/mspec)

> - The third step is to decide what to do about new specs which are not
> contributed to the canonical repository directly.
> This is worth another discussion and I think it is wiser to first achieve
> the two first steps before discussing this in more details.

this topic is epic for me. we need to finish 1 and 2 steps at first.

--
SHIBATA Hiroshi shibata...@gmail.com
http://www.hsbt.org/

[Anthony Crumley]

Beniot,

I have been working on reconciling RubySpec with the 2.x MRI versions over the last month.  https://github.com/anthonycrumley/rubyspec/commits/master

My intention is to:

1) Get RubySpec updated to run with all the 2.x versions of MRI.

2) Get the nurse/rubyspec updates since the fork added.

3) Get the updated RubySpecs into the MRI CI.

4) Hopefully find that repo a home at either rubyspec/rubyspec or ruby/rubyspec.

I agree with you that the RubySpec tests are very valuable and would LOVE to work with you on them.

Anthony

[Anthony Crumley]

FYI...

When I started working through the 2.x issues the results on 2.3 were...

1801 files, 18020 examples, 153894 expectations, 36 failures, 19 errors

Now the results are...

1801 files, 18024 examples, 153918 expectations, 16 failures, 12 errors

[Anthony Crumley]

Beniot and SHIBATA,

The reason I started with rubyspec/rubyspec rather than nurse/rubyspec was, as SHIBATA mentioned, the 1.9.3 specs had already been removed.  I felt like 1.9.3 support would be dropped before the tests were up to date so that would save the work of removing them.  Also, it seemed like it would be easier to cherry pick new commits from nurse/rubyspec than then the other way around.

SHIBATA, I am EXCITED to be able to work with you on RubySpec also.  :)

Anthony

[Anthony Crumley]

FYI...

This patch for a Vagrant setup was created to make it easier for non-MRI developers to work on RubySpec.  Setting up an environment to compile the most recent version of MRI can be intimidating and time consuming.  With a Vagrant setup, developers that are capable of writing specifications can get up and running in less than an hour without having to worry about causing problems on their host machine.

https://bugs.ruby-lang.org/issues/10757

Anthony

[Benoit Daloze]

Hi!

On 17 February 2015 at 14:15, SHIBATA Hiroshi <shibata...@gmail.com> wrote:

Hi, Benoit.

> - The first step is to bring nurse/rubyspec up to date with
> rubyspec/rubyspec archive's branch.
> I am willing to do that if no one has an objection against it. Any help is
> welcome of course! (just email me)

I will help to this work. 1 month ago, I investigate rubyspec's archive branch.
I found following issues:

 * rubyspec removed to support Ruby 1.9.3
  * but MRI 1.9.3 is ended at Feb. 23, it's not problem now.
 * some spec is failing on 2.3.0

Great, I think maintaining many different versions is hard so caring only about >= 2.0 should make things a bit simpler.

We need to have a look at these failures before merging.
Anthony Crumley already looked at some recently, let's coordinate our efforts.

 

> - The second step is really to choose a canonical RubySpec repository, to
> avoid "death by too much forks".
> This repository should only contain RubySpec tests for practical reasons.
> We should allow many specs contributors to take part in merging changes and
> maintaining specs.
> I think this was a fatal flaw of rubyspec/rubyspec in that too few people
> had the large burden of merging and maintaining the specs.
>
> The main existing repository I see today is nurse/rubyspec.
> I am thinking the process could be similar to handling pull requests on
> ruby/ruby in that some contributors would provide feedback and merge them.
> The CI is very useful in this regard to ensure MRI is not broken
> inadvertently.
> I think it would make sense in that case to move nurse/rubyspec to ruby/rubyspec for clarity.

I agreed with your suggestion. I will coordinate with nurse. Please
wait few days.
(PS. I think it's better to move nurse/mspec to ruby/mspec)

Thanks!

Yes, of course mspec should follow as well.

[Benoit Daloze]

Anthony,

Thank you for what you already did!

I am not sure about how to merge concretely yet, but cherry-picking 152 commits might prove quite problematic.

I would rather merge to keep both historic as intact as possible and add commits after to fix the various issues, errors and failures.

Let's coordinate when we have ruby/rubyspec, either in an issue or IRC.