[ruby-core:39260] RubySpec vs CRuby's test/...
Marc-Andre Lafortune
would pass RubySpec, in addition to `make test` and `make test-all`
which runs all `test/*` files.
RubySpec is a project to write an executable specification for the
Ruby Programming Language. As such, any test that is not
implementation specific has its place in RubySpec.
I feel that the Ruby community would benefit most if all new
(implementation independent) tests were written in RubySpec instead of
`test/*` for the following reasons:
- tests improve all other implementations that use RubySpec (rubinius,
JRuby, etc...) in addition to CRuby
- tests are usually more readable, as the style of RubySpec encourages
a more fine grained approach and written explanations
- RubySpec makes for a good central reference point for users, again
because it is better structured as well as typically more exhaustive
and applies to all implementations.
- as RubySpec aims to be as complete as possible, it encourages
thinking about more cases (e.g. what happens if a block is passed),
which makes for better new features.
Moreover, I see no disadvantage in tests being contributed to RubySpec
instead of `test/*`.
In [ruby-core:39206], Yui Naruse says that he would gladly contribute
to RubySpec for new tests if other committers can be persuaded. I'd
note that he already contributes a lot through modifications to
existing RubySpecs.
In [ruby-core:39209], Urabe Shyouhei asks:
> "So what's the test going to achieve? What's that test for?" said my
> teacher another day. A test has a perspective. No test can be suitable
> for everything.
>
> It seems to me RubySpec is meant to be a written behavioral description.
> Which is not a bad thing of course, but I think Yui needs a regression
> test instead. It does test CRuby's implementation but doesn't help him.
I believe that most tests that are implementation independent are
checks for general cases or boundary conditions and are specs of the
language. These make perfect regression tests and they fit very well
with RubySpec.
So the questions to be answered are:
- am I missing advantages to CRuby's `test/*`?
- what are the reservations against adding tests to RubySpec?
- is there anything that should be done to make it easier for CRuby's
committers to add tests to RubySpec?
Thanks,
Marc-André
Yukihiro Matsumoto
In message "Re: [ruby-core:39260] RubySpec vs CRuby's test/..."
on Mon, 5 Sep 2011 03:54:32 +0900, Marc-Andre Lafortune <ruby-core-m...@marc-andre.ca> writes:
|So the questions to be answered are:
|- am I missing advantages to CRuby's `test/*`?
* Since test/* is a test suites only for CRuby, it can contain CRuby
(or platform) specific tests.
* Core CRuby developers have used test/* for long time, so they are
more familiar to test/*
|- what are the reservations against adding tests to RubySpec?
|- is there anything that should be done to make it easier for CRuby's
|committers to add tests to RubySpec?
Developers are often lazy and ignorant. Simple documentation for
RubySpec contributors may work. If it already exists, advertising on
it (or referencing it from w.r.o) may work.
matz.
Urabe Shyouhei
> So the questions to be answered are:
> - am I missing advantages to CRuby's `test/*`?
> - what are the reservations against adding tests to RubySpec?
> - is there anything that should be done to make it easier for CRuby's
> committers to add tests to RubySpec?
Hey, RubySpec and test/ are not hostile each other. No "vs" please.
They have different goals. Please stop preventing us from adding
tests into test/.
Marc-Andre Lafortune
On Sun, Sep 4, 2011 at 10:10 PM, Urabe Shyouhei <shyo...@ruby-lang.org> wrote:
> (09/05/2011 03:54 AM), Marc-Andre Lafortune wrote:
>> So the questions to be answered are:
>> - am I missing advantages to CRuby's `test/*`?
>> - what are the reservations against adding tests to RubySpec?
>> - is there anything that should be done to make it easier for CRuby's
>> committers to add tests to RubySpec?
>
> Hey, RubySpec and test/ are not hostile each other. No "vs" please.
Hey, there actually is a "vs" since a test can be committed to either
RubySpec or test/ but I doubt any committer will ever take the time to
do both (as we all have better things to do). Developers must make a
choice as to where to add new tests.
> They have different goals.
They have distinct but similar goals and the majority of tests can be in either.
> Please stop preventing us from adding tests into test/.
Please stop accusing me of something I am not doing.
Please realize that the reverse is true. I have been told repeatedly
to add tests to "test/". I prefer adding them to RubySpec and want to
continue to do so. I am against the notion that "You must add a test
to test-all when you change CRuby's behavior" [ruby-core:39169].
I would like that we all agree on the following: to change CRuby, one
must add a test, either in RubySpec or in test/. If it is
implementation specific, it belongs in test/, otherwise it is
acceptable in either RubySpec or test/. I also believe it would be
preferable if it was added to RubySpec, but I will not fight over that
notion.
I apologize in advance if this post sounds confrontational; I simply
tried to mirror your post. I normally avoid telling people what to do
(especially with the imperative) or using expressions that can sound
like I'm annoyed (Hey, ...). I understand english is not everyone's
native language (it is not mine either) but I would still recommend to
everybody of being careful about the way a message can be interpreted.
Thanks,
Marc-André
Marc-Andre Lafortune
On Sun, Sep 4, 2011 at 4:02 PM, Yukihiro Matsumoto <ma...@ruby-lang.org> wrote:
> Hi,
>
> In message "Re: [ruby-core:39260] RubySpec vs CRuby's test/..."
> on Mon, 5 Sep 2011 03:54:32 +0900, Marc-Andre Lafortune <ruby-core-m...@marc-andre.ca> writes:
>
> |So the questions to be answered are:
> |- am I missing advantages to CRuby's `test/*`?
>
> * Since test/* is a test suites only for CRuby, it can contain CRuby
> (or platform) specific tests.
Indeed. I'd note that RubySpec can handle platform or implementation
specific tests, but I'm really only interested in implementation
independent tests.
> * Core CRuby developers have used test/* for long time, so they are
> more familiar to test/*
>
> Developers are often lazy and ignorant. Simple documentation for
> RubySpec contributors may work. If it already exists, advertising on
> it (or referencing it from w.r.o) may work.
Indeed, I should have pointed out the historic aspec of test/*. It
looks like you feel that, today and for the developers that can, many
tests would ideally be added to RubySpec, is this what you mean?
Also, I'm sorry, I'm not sure what you mean by "w.r.o."? I'll not that
the {Developer|Committer}HowTo wikis gives information on how to
update and run the RubySpec and that http://rubyspec.org has extensive
documentation.
Thanks,
Marc-André
NARUSE, Yui
> Also, I'm sorry, I'm not sure what you mean by "w.r.o."? I'll not that
> the {Developer|Committer}HowTo wikis gives information on how to
> update and run the RubySpec and that http://rubyspec.org has extensive
> documentation.
w.r.o is www.ruby-lang.org.
--
NARUSE, Yui <nar...@airemix.jp>
Yukihiro Matsumoto
In message "Re: [ruby-core:39271] Re: RubySpec vs CRuby's test/..."
on Mon, 5 Sep 2011 13:48:46 +0900, Marc-Andre Lafortune <ruby-core-m...@marc-andre.ca> writes:
|> * Core CRuby developers have used test/* for long time, so they are
|> more familiar to test/*
|>
|> Developers are often lazy and ignorant. Simple documentation for
|> RubySpec contributors may work. If it already exists, advertising on
|> it (or referencing it from w.r.o) may work.
|
|Indeed, I should have pointed out the historic aspec of test/*. It
|looks like you feel that, today and for the developers that can, many
|tests would ideally be added to RubySpec, is this what you mean?
Yes, but I don't want to make rubyspec full of implementation
dependent corner case messes, like test/*. Maybe the strategy should
be extracting implementation independent specification from messy
test/*.
|Also, I'm sorry, I'm not sure what you mean by "w.r.o."? I'll not that
|the {Developer|Committer}HowTo wikis gives information on how to
|update and run the RubySpec and that http://rubyspec.org has extensive
|documentation.
w.r.o is www.ruby-lang.org, sorry for using uncommon abbreviation.
Updating test/* is very easy for core developers to update, comparing
to making a patch to rubyspec and sending pull request. Considering
extracting strategy above, this situation may not be bad.
matz.
Urabe Shyouhei
(09/05/2011 01:10 PM), Marc-Andre Lafortune wrote:
> On Sun, Sep 4, 2011 at 10:10 PM, Urabe Shyouhei <shyo...@ruby-lang.org> wrote:
>> (09/05/2011 03:54 AM), Marc-Andre Lafortune wrote:
>>> So the questions to be answered are:
>>> - am I missing advantages to CRuby's `test/*`?
>>> - what are the reservations against adding tests to RubySpec?
>>> - is there anything that should be done to make it easier for CRuby's
>>> committers to add tests to RubySpec?
>>
>> Hey, RubySpec and test/ are not hostile each other. No "vs" please.
>
> Hey, there actually is a "vs" since a test can be committed to either
> RubySpec or test/ but I doubt any committer will ever take the time to
> do both (as we all have better things to do). Developers must make a
> choice as to where to add new tests.
So this is a structural problem. The problem is not someone's fault, but
anyone suffer for a same problem that adding tests for both projects is
cumbersome. Enumerating advantages / disadvantages over them are not
effective to solve the situation it seems.
NARUSE, Yui
> Hi,
>
> On Sun, Sep 4, 2011 at 10:10 PM, Urabe Shyouhei <shyo...@ruby-lang.org> wrote:
>> (09/05/2011 03:54 AM), Marc-Andre Lafortune wrote:
>>> So the questions to be answered are:
>>> - am I missing advantages to CRuby's `test/*`?
>>> - what are the reservations against adding tests to RubySpec?
>>> - is there anything that should be done to make it easier for CRuby's
>>> committers to add tests to RubySpec?
>>
>> Hey, RubySpec and test/ are not hostile each other. No "vs" please.
>
> Hey, there actually is a "vs" since a test can be committed to either
> RubySpec or test/ but I doubt any committer will ever take the time to
> do both (as we all have better things to do). Developers must make a
> choice as to where to add new tests.
It means we must run test-all and RubySpec before each commit.
>> Please stop preventing us from adding tests into test/.
>
> Please stop accusing me of something I am not doing.
"us" includes you; you don't add tests.
> Please realize that the reverse is true. I have been told repeatedly
> to add tests to "test/". I prefer adding them to RubySpec and want to
> continue to do so. I am against the notion that "You must add a test
> to test-all when you change CRuby's behavior" [ruby-core:39169].
You can add tests both RubySpec and test/.
What you are doing is, you ignore our historical policy and breaks
test-all's completeness.
> I would like that we all agree on the following: to change CRuby, one
> must add a test, either in RubySpec or in test/. If it is
> implementation specific, it belongs in test/, otherwise it is
> acceptable in either RubySpec or test/. I also believe it would be
> preferable if it was added to RubySpec, but I will not fight over that
> notion.
I disagree.
I want test-all to be complete.
> I apologize in advance if this post sounds confrontational; I simply
> tried to mirror your post. I normally avoid telling people what to do
> (especially with the imperative) or using expressions that can sound
> like I'm annoyed (Hey, ...). I understand english is not everyone's
> native language (it is not mine either) but I would still recommend to
> everybody of being careful about the way a message can be interpreted.
Of course suggestion is welcome.
What I only want to complain is that you stopped to add test before
you get consensus.
--
NARUSE, Yui <nar...@airemix.jp>
NARUSE, Yui
> So the questions to be answered are:
> - am I missing advantages to CRuby's `test/*`?
Ah, I must say one big difference:
CRuby test-all is regression test, so if a new failure or error
occurs, someone must breaks CRuby.
In other words, regression is just happened.
# under this policy, a test which fails on CRuby won't be added
RubySpec doesn't have such policy so even if failure happens, I can't
know who adds the failure.
--
NARUSE, Yui <nar...@airemix.jp>
Charles Oliver Nutter
JRuby runs more tests/suites than any other Ruby implementation. We
run the old Rubicon tests...MRI's tests...RubySpec...ruby_test...old
minirunit tests from JRuby's past...even BFTS until a year ago or so.
We are very sensitive to the issues of dealing with many test suites.
RubySpec is by far the easiest and cleanest to:
* use
* modify
* filter for things we don't pass yet (mspec's "tags" are
beautiful...Brian Ford did an amazing job putting mspec together)
* understand
No other test suite even comes close to the ease of use and utility of
RubySpec today. Because of that, we have actively (but slowly) been
trying to transition to using only RubySpec for all our testing, and
as Ruby behavioral bugs are reported we add them to RubySpec.
There are many reasons why tests that live solely in MRI's repository
are problematic for other implementations:
* Impl-specific tests, as Matz mentioned
* Test harnesses specific to MRI's build
* Inability to easily filter specific tests to get a green partial run
* Nobody but MRI committers can contribute easily (!!!!!!!!!)
A perfect example of this is the Psych library. JRuby implements Psych
atop SnakeYAML. We reuse all the Psych .rb files as-is and only
reimplement what would be in C. And we run the Psych tests, of which
we have a copy in the JRuby repository.
But all the issues above affect us. Some of Psych's tests have minor
impl-specific details. They're test/unit based, so we need a test/unit
harness to run them. There's no way to filter out any impl-specific,
unsupported, or undefined behavior tests. And there's no way for us to
add new tests other than submitting pull requests and praying.
The result of these issues is that JRuby and Rubinius simply do not
contribute tests to MRI, and run MRI's tests less and less (Rubinius
doesn't run *any* as far as I know). It means that JRuby and Rubinius
contributors must do the test-writing work of all the MRI contributors
*again* to keep RubySpec up to date. And it means that MRI is always
drifting away from RubySpec, making a moving target even more
difficult to chase for JRuby and Rubinius.
It is my sincere belief that if ruby-core cares about having a rich
Ruby implementation ecosystem, more effort should be put into making
RubySpec the standard suite of tests, with MRI's tests reserved for
impl-specific tests (similar to how I see JRuby going over time).
Conversely, I worry that the failure of ruby-core to adopt RubySpec
indicates a complete disregard for other Ruby implementations and the
contributions they make to the Ruby community. This makes me very sad,
and I hope it's not true.
I understand the desire for ruby-core to maintain the status quo, and
I sympathize wrt the language barrier involved in contributing to
RubySpec. But maintaining Ruby behavioral test suites in the current
ruby-core way makes life much more difficult for alternative
implementations...and we have enough challenges already.
- Charlie
Marc-Andre Lafortune
> It means we must run test-all and RubySpec before each commit.
On my 5 year old machine, running all RubySpec takes 42 seconds.
That's less than `make test` and 14 times less than `make test-all`
which takes more than 10 minutes on my machine.
If someone decides he should `make test-all`, then yes, he should also
`make test-rubyspec`. This will take him 6% more time than just
running `test-all`. Is that a cost even worth mentioning?
>> Please realize that the reverse is true. I have been told repeatedly
>> to add tests to "test/". I prefer adding them to RubySpec and want to
>> continue to do so. I am against the notion that "You must add a test
>> to test-all when you change CRuby's behavior" [ruby-core:39169].
>
> You can add tests both RubySpec and test/.
> What you are doing is, you ignore our historical policy and breaks
> test-all's completeness.
>
>> I would like that we all agree on the following: to change CRuby, one
>> must add a test, either in RubySpec or in test/. If it is
>> implementation specific, it belongs in test/, otherwise it is
>> acceptable in either RubySpec or test/. I also believe it would be
>> preferable if it was added to RubySpec, but I will not fight over that
>> notion.
>
> I disagree.
> I want test-all to be complete.
Why do you want it to be complete?
Or why not just include `make RubySpec` in `make test-all`?
> Of course suggestion is welcome.
> What I only want to complain is that you stopped to add test before
> you get consensus.
Indeed, I've added tests to RubySpec instead of test/* as soon as it
was decided that CRuby would pass RubySpec. That was two years ago...
> Ah, I must say one big difference:
> CRuby test-all is regression test, so if a new failure or error
> occurs, someone must breaks CRuby.
> In other words, regression is just happened.
> # under this policy, a test which fails on CRuby won't be added
>
> RubySpec doesn't have such policy so even if failure happens, I can't
> know who adds the failure.
I'm not sure I follow. RubySpec that fail on trunk are guarded as such
by `ruby_bug`. That refers to issue #, so it should be easy to know
what it is about. If a different behavior is desired, this can
definitely be discussed.
Urabe Shyouhei
> On Mon, Sep 5, 2011 at 3:08 AM, NARUSE, Yui <nar...@airemix.jp> wrote:
>> It means we must run test-all and RubySpec before each commit.
>
> On my 5 year old machine, running all RubySpec takes 42 seconds.
> That's less than `make test` and 14 times less than `make test-all`
> which takes more than 10 minutes on my machine.
>
> If someone decides he should `make test-all`, then yes, he should also
> `make test-rubyspec`. This will take him 6% more time than just
> running `test-all`. Is that a cost even worth mentioning?
Why you can't add your test to test/* using that 42 seconds?
NARUSE, Yui
> I'll jump in with some context from the JRuby perspective.
>
> JRuby runs more tests/suites than any other Ruby implementation. We
> run the old Rubicon tests...MRI's tests...RubySpec...ruby_test...old
> minirunit tests from JRuby's past...even BFTS until a year ago or so.
> We are very sensitive to the issues of dealing with many test suites.
>
> RubySpec is by far the easiest and cleanest to:
>
> * use
> * modify
> * filter for things we don't pass yet (mspec's "tags" are
> beautiful...Brian Ford did an amazing job putting mspec together)
> * understand
>
> No other test suite even comes close to the ease of use and utility of
> RubySpec today. Because of that, we have actively (but slowly) been
> trying to transition to using only RubySpec for all our testing, and
> as Ruby behavioral bugs are reported we add them to RubySpec.
I don't think RubySpec (RSpec syntax) is beautiful.
I want to write assertion with declarative.
For example I can't find which is the expectation 4 or 5.
Fixnum#div coerces self and the given argument to Floats and returns
self divided by other as Fixnum FAILED
Expected 4
to equal 5
> There are many reasons why tests that live solely in MRI's repository
> are problematic for other implementations:
>
> * Impl-specific tests, as Matz mentioned
> * Test harnesses specific to MRI's build
> * Inability to easily filter specific tests to get a green partial run
> * Nobody but MRI committers can contribute easily (!!!!!!!!!)
>
> A perfect example of this is the Psych library. JRuby implements Psych
> atop SnakeYAML. We reuse all the Psych .rb files as-is and only
> reimplement what would be in C. And we run the Psych tests, of which
> we have a copy in the JRuby repository.
>
> But all the issues above affect us. Some of Psych's tests have minor
> impl-specific details. They're test/unit based, so we need a test/unit
> harness to run them. There's no way to filter out any impl-specific,
> unsupported, or undefined behavior tests. And there's no way for us to
> add new tests other than submitting pull requests and praying.
What you want to say seems MRI's test-all doesn't have implementation guards.
But minitest has `skip` and we use it for platform dependent tests.
People can request test-all to add such skips but I didn't see such request.
--
NARUSE, Yui <nar...@airemix.jp>
"Martin J. Dürst"
On 2011/09/07 13:33, Charles Oliver Nutter wrote:
> I'll jump in with some context from the JRuby perspective.
> * Nobody but MRI committers can contribute easily (!!!!!!!!!)
> And there's no way for us to
> add new tests other than submitting pull requests and praying.
On the other hand, somebody (I think it was Shyouhei) said that MRI
committers had to submit pull requests when they wanted to add tests to
RubySpec.
So this seems to go both ways, or am I getting something wrong?
Regards, Martin.
Charles Oliver Nutter
<due...@it.aoyama.ac.jp> wrote:
> On the other hand, somebody (I think it was Shyouhei) said that MRI
> committers had to submit pull requests when they wanted to add tests to
> RubySpec.
>
> So this seems to go both ways, or am I getting something wrong?
RubySpec has an open commit policy. Submit one pull request, and you
can get commit access (and even that restriction may be loosened for
known ruby-core hackers).
- Charlie
Charles Oliver Nutter
> I don't think RubySpec (RSpec syntax) is beautiful.
> I want to write assertion with declarative.
>
> For example I can't find which is the expectation 4 or 5.
> Fixnum#div coerces self and the given argument to Floats and returns
> self divided by other as Fixnum FAILED
> Expected 4
> to equal 5
I think this is part of the language barrier I sympathize with. It's a
very terse way to say "I expected 5, but the code returned 4". I agree
it might be a bit clearer to say "Expected 5 but got 4", but this is
mostly a language issue; as written, it is correct, if confusing for
you (and others).
I'm starting to feel like the RubySpec resistance boils down to "it
has too much English in it."
> What you want to say seems MRI's test-all doesn't have implementation guards.
>
> But minitest has `skip` and we use it for platform dependent tests.
> People can request test-all to add such skips but I didn't see such request.
If test/unit could do something similar to mspec's guards, I would be
happy with that. Working on RubySpec has taught me that we really need
per-test guards, as in individual test methods. RubySpec accomplishes
this by splitting specs up into isolated examples for each aspect of
behavior, rather than test/* which often tests many things in a given
method.
However, I don't see any easy path toward accomplishing this. The
current tests are structured with the expectation that an
implementation must pass them en masse, and there's no way to break
that up.
For example...there are many test files for core classes that include
SAFE level checks. JRuby will never pass those, since we have opted to
leave SAFE unimplemented and fall back on JVM security. There's no
easy way to filter out just the SAFE tests, which means we either have
to comment them out and keep a local copy (that gets outdated) or
build a complicated harness that can tell test/unit to skip certain
test methods. That's what I get out of the box with RubySpec, so even
though JRuby doesn't pass everything, we can maintain at some level
and avoid regressing as we move forward.
I propose a challenge: Show me how to get the current test/* tests to
run on JRuby *green* without modifying JRuby. In other words, show me
how to run MRI's tests on JRuby while filtering anything that doesn't
currently pass. If that's not possible, how can JRuby make use of the
test/* suite without already passing everything? And if JRuby can't
make use of test/*, test/* is the wrong way to test Ruby.
- Charlie
Yukihiro Matsumoto
In message "Re: [ruby-core:39365] Re: RubySpec vs CRuby's test/..."
on Fri, 9 Sep 2011 00:29:54 +0900, Charles Oliver Nutter <hea...@headius.com> writes:
|I propose a challenge: Show me how to get the current test/* tests to
|run on JRuby *green* without modifying JRuby. In other words, show me
|how to run MRI's tests on JRuby while filtering anything that doesn't
|currently pass. If that's not possible, how can JRuby make use of the
|test/* suite without already passing everything? And if JRuby can't
|make use of test/*, test/* is the wrong way to test Ruby.
test/* is no way to test Ruby, it's test suites for MRI. You know the
difference.
matz.
Michael Klishin
> test/* is no way to test Ruby, it's test suites for MRI. You know the
> difference.
Matz,
We all know the difference. The question is, as Ruby the language evolves, why would CRuby developers not use RubySpec to specify *language* behavior (which is what a lot of test/* tests currently do)? Right now there is a fair share of
duplication of effort going on and it is disappointing. What testing framework produces what output is a red herring: that can be easily changed in a week.
I think not using RubySpec as the primary test suite for every Ruby implementation (including MRI) is damaging to the long term Ruby competitiveness/relevance. Fragmentation is a terrible thing for developers. This does not go hand in hand with the (stated) goal of Ruby to make developers happy.
MK
NARUSE, Yui
>> Ah, I must say one big difference:
>> CRuby test-all is regression test, so if a new failure or error
>> occurs, someone must breaks CRuby.
>> In other words, regression is just happened.
>> # under this policy, a test which fails on CRuby won't be added
>>
>> RubySpec doesn't have such policy so even if failure happens, I can't
>> know who adds the failure.
>
> I'm not sure I follow. RubySpec that fail on trunk are guarded as such
> by `ruby_bug`. That refers to issue #, so it should be easy to know
> what it is about. If a different behavior is desired, this can
> definitely be discussed.
If trunk are guarded as a bug, the test is useless.
--
NARUSE, Yui <nar...@airemix.jp>
NARUSE, Yui
> Yukihiro Matsumoto:
>
>> test/* is no way to test Ruby, it's test suites for MRI. You know the
>> difference.
>
> Matz,
>
> We all know the difference. The question is, as Ruby the language evolves, why would CRuby developers not use RubySpec to specify *language* behavior (which is what a lot of test/* tests currently do)? Right now there is a fair share of
> duplication of effort going on and it is disappointing. What testing framework produces what output is a red herring: that can be easily changed in a week.
Why you said test-all is duplicated though test-all exists before RubySpec?
> I think not using RubySpec as the primary test suite for every Ruby implementation (including MRI) is damaging to the long term Ruby competitiveness/relevance. Fragmentation is a terrible thing for developers. This does not go hand in hand with the (stated) goal of Ruby to make developers happy.
The spec we believe is not the implementation or tests or doc.
It is abstract one and exists over them.
On this point, our direction is different from other implementations.
Anyway why you want us to blame RubySpec?
I don't like RubySpec because of its idea, Windows Support, RSpec like syntax, and so on.
Moreover why you want us to work?
On beneficiary-pays principle, you should use your hands or you should show our benefit.
Only to request is not good thing on OSS projects.
--
NARUSE, Yui <nar...@airemix.jp>
Michael Klishin
> If trunk are guarded as a bug, the test is useless.
Again, this is a red herring and has nothing to do with what the original poster is inquiring about: why MRI developers
do not use RubySpec as the primary test suite for Ruby the language, while every other implementation does (in addition to implementation-specific tests).
Test suite output differences and remarks about guards in the trunk are all driving attention away from that.
MK
Luis Lavena
>
> Anyway why you want us to blame RubySpec?
> I don't like RubySpec because of its idea, Windows Support, RSpec like syntax, and so on.
> Moreover why you want us to work?
MSpec works on Windows, I use it, RubySpec too:
platform_is_not :windows do
end
platform_is :windows do
end
I'm using RubySpec File.expand_path specs because there aren't tests
(except for 5 in test/sample.rb) that explain expand_path behavior.
--
Luis Lavena
AREA 17
-
Perfection in design is achieved not when there is nothing more to add,
but rather when there is nothing more to take away.
Antoine de Saint-Exupéry
Michael Klishin
> I don't like RubySpec because of its idea
Can you elaborate? I can understand other arguments but the idea of RubySpec is the most noble one and people who work on RubySpec keep finding edge cases and bugs in MRI that Ruby developers cannot learn about in any other source.
If this knowledge is ignored *just* because RubySpec uses RSpec-like DSL syntax, how will the Ruby community avoid fragmentation?
MK
Chris White
Masaya TARUI
Does RubySpec work fine on your Windows?
On my environment, there are too many failures and errors.
3456 files, 16823 examples, 220283 expectations, 99 failures, 608 errors
( ruby 1.9.4dev (2011-09-06 trunk 33199) [i386-mswin32_100] )
It seems not to consider portability.
For Instance, temporary directory is assumed to be / tmp .
I think that RubySpec is good compatibility check tool on Linux,
but it has to solve portability issues for using as test tool for development.
it looks like to be hard work.
2011/9/9 Luis Lavena <luisl...@gmail.com>:
> On Thu, Sep 8, 2011 at 4:19 PM, NARUSE, Yui <nar...@airemix.jp> wrote:
>>
>> Anyway why you want us to blame RubySpec?
>> I don't like RubySpec because of its idea, Windows Support, RSpec like syntax, and so on.
>> Moreover why you want us to work?
>
> MSpec works on Windows, I use it, RubySpec too:
>
> platform_is_not :windows do
> end
>
> platform_is :windows do
> end
>
> I'm using RubySpec File.expand_path specs because there aren't tests
> (except for 5 in test/sample.rb) that explain expand_path behavior.
Best regards,
--
Masaya TARUI
No Tool,No Life.
Luis Lavena
> Hello Luis,
>
> Does RubySpec work fine on your Windows?
>
Yes, I have failures due missing guards (platform_is / platform_is_not)
> On my environment, there are too many failures and errors.
>
> 3456 files, 16823 examples, 220283 expectations, 99 failures, 608 errors
> ( ruby 1.9.4dev (2011-09-06 trunk 33199) [i386-mswin32_100] )
>
Most of the times is due missing guards and sometimes is missing lot
of Windows specific code.
Please see my rant at the bottom [1]
> It seems not to consider portability.
> For Instance, temporary directory is assumed to be / tmp .
>
Hmn, sounds hardcoded, for comparison, file/expand_path_spec.rb:
before :each do
platform_is :windows do
@base = `cd`.chomp.tr '\\', '/'
@tmpdir = "c:/tmp"
@rootdir = "c:/"
end
platform_is_not :windows do
@base = Dir.pwd
@tmpdir = "/tmp"
@rootdir = "/"
end
end
Which is used later on the specs.
If that is hardcoded somewhere, is wrong and needs to be corrected. I
will happily fix that if you open an issue here:
https://github.com/rubyspec/rubyspec/issues
> I think that RubySpec is good compatibility check tool on Linux,
> but it has to solve portability issues for using as test tool for development.
> it looks like to be hard work.
>
If you're used to a tool, getting full to speed and covering those
guards inside other tool can be time challenging and hard work, sure.
Now, my rant [1]
I'm trying to see if I can improve Ruby performance on Windows, and is
really hard to find ANY documentation on the behaviors of certain
methods.
I was expecting see some test for File.expand_path inside
test/test_file.rb but there is none, except a few in test/sample.rb,
not to mention the lack of result taint testing or encoding.
So, under Windows, neither test/* or RubySpec solves the problem.
However, there are *some* specs in RubySpec that did help me more than
4 regex validations that I found in test/sample.rb
But honestly, nobody from core really "care" to improve this on
Windows and the few that use Ruby on Windows don't have the time to
dig into Ruby C code and extract what is the expected behavior of Ruby
to add to RubySpec.
Charles Oliver Nutter
I tend to agree about ruby_bug. I think MRI should have to maintain a
set of "tags" just like every other implementation, rather than
getting special treatment from RubySpec.
That said...this is just another way of turning certain tests off when
they're known to fail, so you can maintain the working behaviors in a
green suite.
- Charlie
Charles Oliver Nutter
> Hello Luis,
>
> Does RubySpec work fine on your Windows?
>
> On my environment, there are too many failures and errors.
>
> 3456 files, 16823 examples, 220283 expectations, 99 failures, 608 errors
> ( ruby 1.9.4dev (2011-09-06 trunk 33199) [i386-mswin32_100] )
Several RubySpec committers worked very hard to get RubySpec green on
Windows against 1.9.2. Since then, 1.9.x has changed, more specs have
been added, and nobody has done an updated pass over RubySpec to
ensure it works.
If it were used more by ruby-core and kept green on Windows against
MRI trunk, this would not happen.
- Charlie
Charles Oliver Nutter
> But honestly, nobody from core really "care" to improve this on
> Windows and the few that use Ruby on Windows don't have the time to
> dig into Ruby C code and extract what is the expected behavior of Ruby
> to add to RubySpec.
This may be the biggest problem.
Matz: Is it official policy that all feature changes in Ruby must be
accompanied by a test or spec? If not, why not? If so, why aren't they
going into a non-MRI-specific test suite?
Shouldn't ruby-core help maintain an implementation-agnostic Ruby
behavioral test suite? If so, why isn't that happening, and why
couldn't it be RubySpec? If not, why not?
- Charlie
Charles Oliver Nutter
<hea...@headius.com> wrote:
> Matz: Is it official policy that all feature changes in Ruby must be
> accompanied by a test or spec? If not, why not? If so, why aren't they
> going into a non-MRI-specific test suite?
>
> Shouldn't ruby-core help maintain an implementation-agnostic Ruby
> behavioral test suite? If so, why isn't that happening, and why
> couldn't it be RubySpec? If not, why not?
I'm going to try to dial back the rhetoric a bit and lay out the
facts. Jump in where I go astray.
THE NEED:
* There must be a test suite all Ruby implementations can use to
confirm that they are compatible with "Ruby" the
language/core/libraries
* This suite must not be specific to any one implementation, and
should only include tests for features considered "standard" Ruby
(i.e. not implementation-specific).
* This test suite would ideally support running a subset of tests if
some are known to fail. This is necessary to maintain a watermark on
partial or in-progress implementations.
* All Ruby features should have a complete set of tests in this suite.
* All Ruby feature changes should be accompanied by a set of tests or
changes to tests in this suite.
How do we get there?
- Charlie
SASADA Koichi
> THE NEED:
Thank you for summary of needs. Following is problems I feel (and
discussed on IRC).
Problems:
(1) Most of MRI developers (is it true?) can't change RubySpec. Also
most of MRI developers don't know RubySpec rules.
(2) We can add code modification and test cases in one commit. If we
can't do it, we need commit commit modifications and RubySpec isolation.
(3) It is difficult to judge test is an independent or not.
Additionally, we don't need to insert "guard" about version or
implementation on our "test/*". It seems tough work to insert correct
"guard".
(4) Current RubySpec is not portable (especially Windows)
[ruby-core:39379].
(5) As you know, trunk is not unstable. For example, some modifications
are revertd soon. Should we add such features as "Specifications"? (I
know that there is an opinion that the experimental code should have a
spec).
(6) test/* contains many many corner cases depend on MRI implementation
to increase code coverage (Thanks Endo-san). It is independent as Ruby
language, but dependent on MRI. Where should we write such tests?
I think this problem is "who pay efforts on it?".
One solution is continuing current style (using test/*) and someone
migrate tests to RubySpec if needed. But we need "someone".
Other solution is MRI committer join to maintain RubySpec. MRI
developer (including me) should change development style and several
overhead to do (I think inserting correct "guard" is difficult).
Brian Ford told me that we can have our own RubySpec repository in MRI
repository and migrating updating specs from MRI's RubySpec repository
to the RubySpec repository is easy. It is also one solution. However,
we need "someone" who can migrate. It solves (1), (2) and maybe (3).
Regards,
Koichi
--
// SASADA Koichi at atdot dot net
Luis Lavena
> (2011/09/08 15:28), Charles Oliver Nutter wrote:
>> THE NEED:
>
>
> Thank you for summary of needs. Following is problems I feel (and
> discussed on IRC).
>
Hello Koichi-san,
> Problems:
>
> (1) Most of MRI developers (is it true?) can't change RubySpec. Also
> most of MRI developers don't know RubySpec rules.
>
Commit bit can be given to the ones that ask, they just need a GitHub account.
RubySpec rules are clearly detailed here:
Getting Started:
http://rubyspec.org/getting_started/
How RubySpec is organized:
http://rubyspec.org/organization/
The Style Guide:
http://rubyspec.org/style_guide/
Guards for platforms, Ruby versions, bugs and implementation compliance:
http://rubyspec.org/guards/
The matchers availables (expectations):
http://rubyspec.org/matchers/
> (2) We can add code modification and test cases in one commit. If we
> can't do it, we need commit commit modifications and RubySpec isolation.
>
I believe that tests that fix bugs in MRI can be done in a single commit.
Changes in MRI that do change the behavior of Ruby itself need to be
reflected in RubySpec.
If not, how other implementations will match?
> (3) It is difficult to judge test is an independent or not.
> Additionally, we don't need to insert "guard" about version or
> implementation on our "test/*". It seems tough work to insert correct
> "guard".
>
> (4) Current RubySpec is not portable (especially Windows)
> [ruby-core:39379].
>
RubySpec is portable, see [ruby-core:39382]
> (5) As you know, trunk is not unstable. For example, some modifications
> are revertd soon. Should we add such features as "Specifications"? (I
> know that there is an opinion that the experimental code should have a
> spec).
>
When a feature gets stable/solid, documentation of it in RubySpec
should be done.
> (6) test/* contains many many corner cases depend on MRI implementation
> to increase code coverage (Thanks Endo-san). It is independent as Ruby
> language, but dependent on MRI. Where should we write such tests?
>
corner cases for MRI, things that fix MRI specific bug or behavior are
for MRI test/* only.
Things that affect and define Ruby language should be added to RubySpec
>
> I think this problem is "who pay efforts on it?".
>
We all pay, MRI and other implementation developers pay the price.
There is no consolidated place of reference to look into the language
specification.
> One solution is continuing current style (using test/*) and someone
> migrate tests to RubySpec if needed. But we need "someone".
>
> Other solution is MRI committer join to maintain RubySpec. MRI
> developer (including me) should change development style and several
> overhead to do (I think inserting correct "guard" is difficult).
>
> Brian Ford told me that we can have our own RubySpec repository in MRI
> repository and migrating updating specs from MRI's RubySpec repository
> to the RubySpec repository is easy. It is also one solution. However,
> we need "someone" who can migrate. It solves (1), (2) and maybe (3).
>
In my personal opinion, the developer who change the behavior of Ruby
should be the one adding or correcting the proper specs in RubySpec
(either the MRI mirror or the common one)
NARUSE, Yui
If you think so, that's can be why we don't use RubySpec as primary.
--
NARUSE, Yui <nar...@airemix.jp>
NARUSE, Yui
Other implementation's goal is released ruby.
But we are making spec; such specs are described with intension and use case.
On this view, I don't like naming/treating tests as specs.
--
NARUSE, Yui <nar...@airemix.jp>
NARUSE, Yui
What you are saying in this message is, "MRI committers please use RubySpec!
please fix RubySpec on Windows!". You don't think this is odd?
--
NARUSE, Yui <nar...@airemix.jp>
Yukihiro Matsumoto
In message "Re: [ruby-core:39386] Re: RubySpec vs CRuby's test/..."
on Fri, 9 Sep 2011 07:07:28 +0900, Charles Oliver Nutter <hea...@headius.com> writes:
|Matz: Is it official policy that all feature changes in Ruby must be
|accompanied by a test or spec? If not, why not?
In general, all feature changes should be accompanied by tests.
|If so, why aren't they
|going into a non-MRI-specific test suite?
Because some feature changes are platform specific.
|Shouldn't ruby-core help maintain an implementation-agnostic Ruby
|behavioral test suite? If so, why isn't that happening, and why
|couldn't it be RubySpec? If not, why not?
The answer depends on the definition of ruby-core. If ruby-core
refers the core CRuby committers, I don't want make their burden
heavier. If ruby-core means Ruby developer community as a whole, I
agree they should help maintain an implementation-agnostic Ruby
behavioral test suite. The reason behind the current problem is
simply lack of contributors. We need more contributors on both core
and spec.
matz.
NARUSE, Yui
>> But honestly, nobody from core really "care" to improve this on
>> Windows and the few that use Ruby on Windows don't have the time to
>> dig into Ruby C code and extract what is the expected behavior of Ruby
>> to add to RubySpec.
>
> This may be the biggest problem.
>
> Matz: Is it official policy that all feature changes in Ruby must be
> accompanied by a test or spec? If not, why not? If so, why aren't they
> going into a non-MRI-specific test suite?
If someone change the behavior, they must add/change a test.
> Shouldn't ruby-core help maintain an implementation-agnostic Ruby
> behavioral test suite? If so, why isn't that happening, and why
> couldn't it be RubySpec? If not, why not?
CRuby test-all tests Windows specific behavior and skips non Windows
feature on Windows,
and 0F0E.
--
NARUSE, Yui <nar...@airemix.jp>
NARUSE, Yui
Thank your for summarize your needs.
Yeah, they are your needs.
In short, we just think they are not worth our cost.
Until you can show our merit, you should watch test/ and copy to RubySpec.
--
NARUSE, Yui <nar...@airemix.jp>
Luis Lavena
>
> CRuby test-all tests Windows specific behavior and skips non Windows
> feature on Windows,
> and 0F0E.
>
https://gist.github.com/1079523
ruby 1.9.3dev (2011-07-13 revision 32531) [i386-mingw32]
9618 tests, 1881925 assertions, 11 failures, 12 errors, 89 skips
11 failures and 12 errors that covers CGI, M17N (previously mentioned
on the list) and a few segv that caused test-all to stop for manual
input (so can't be run on CI environments)
NARUSE, Yui
> On Thu, Sep 8, 2011 at 7:40 PM, SASADA Koichi <k...@atdot.net> wrote:
>> (2) We can add code modification and test cases in one commit. If we
>> can't do it, we need commit commit modifications and RubySpec isolation.
>
> I believe that tests that fix bugs in MRI can be done in a single commit.
>
> Changes in MRI that do change the behavior of Ruby itself need to be
> reflected in RubySpec.
>
> If not, how other implementations will match?
Who apply the change to RubySpec is not self-evident.
Yeah, I know you think the person who change it is.
>> (3) It is difficult to judge test is an independent or not.
>> Additionally, we don't need to insert "guard" about version or
>> implementation on our "test/*". It seems tough work to insert correct
>> "guard".
>>
>> (4) Current RubySpec is not portable (especially Windows)
>> [ruby-core:39379].
>>
>
> RubySpec is portable, see [ruby-core:39382]
Say portable after it works in the real world.
Current framework is just a paper tiger.
I can't think it works from the result, 99 failures 608 errors.
>> I think this problem is "who pay efforts on it?".
>
> We all pay, MRI and other implementation developers pay the price.
>
> There is no consolidated place of reference to look into the language
> specification.
I think rdoc should describe it.
>> One solution is continuing current style (using test/*) and someone
>> migrate tests to RubySpec if needed. But we need "someone".
Why *you* don't do it?
>> Other solution is MRI committer join to maintain RubySpec. MRI
>> developer (including me) should change development style and several
>> overhead to do (I think inserting correct "guard" is difficult).
Run `git shortlog -sn` on RubySpec before say that.
(yeah I rarely add new spec)
--
NARUSE, Yui <nar...@airemix.jp>
NARUSE, Yui
> On Thu, Sep 8, 2011 at 9:47 PM, NARUSE, Yui <nar...@airemix.jp> wrote:
>>
>> CRuby test-all tests Windows specific behavior and skips non Windows
>> feature on Windows,
>> and 0F0E.
>
> https://gist.github.com/1079523
>
> ruby 1.9.3dev (2011-07-13 revision 32531) [i386-mingw32]
> 9618 tests, 1881925 assertions, 11 failures, 12 errors, 89 skips
minw and mswin are different thing, and we usually think mswin as Windows.
--
NARUSE, Yui <nar...@airemix.jp>
Urabe Shyouhei
>> Shouldn't ruby-core help maintain an implementation-agnostic Ruby
>> behavioral test suite? If so, why isn't that happening, and why
>> couldn't it be RubySpec? If not, why not?
>
> I'm going to try to dial back the rhetoric a bit and lay out the
> facts. Jump in where I go astray.
>
> THE NEED:
>
> * There must be a test suite all Ruby implementations can use to
> confirm that they are compatible with "Ruby" the
> language/core/libraries
>
> * This suite must not be specific to any one implementation, and
> should only include tests for features considered "standard" Ruby
> (i.e. not implementation-specific).
And I think RubySpec is insufficient for this purpose. It describes
what CRuby is now and never specifies what it should be. I don't
want to say that's a standard to obey. For Ruby alternatives, RubySpec
shows you a way to go. Sadly it won't for CRuby.
Charles Oliver Nutter
> Thank you for summary of needs. Following is problems I feel (and
> discussed on IRC).
>
> Problems:
>
> (1) Most of MRI developers (is it true?) can't change RubySpec. Also
> most of MRI developers don't know RubySpec rules.
I don't think there would be a problem getting MRI developers access
to RubySpec for direct commits. I understand the issues with "rules"
though. Personally I'd be happy to see *any* specs getting in, even if
they require some cleanup after the fact. It would be better than
nothing.
> (2) We can add code modification and test cases in one commit. If we
> can't do it, we need commit commit modifications and RubySpec isolation.
This is an issue, but a small one. You point out later that you could
just keep a copy of RubySpec in MRI's repository and migrate changes
over. That's not a difficult task, if it's done periodically. This is
how Rubinius uses RubySpec.
For JRuby, we chose not to keep our own copy of RubySpec. Instead, our
build will fetch a "known stable" revision of RubySpec as part of the
build and run against that. We make all our RubySpec commits directly
to RubySpec.
> (3) It is difficult to judge test is an independent or not.
> Additionally, we don't need to insert "guard" about version or
> implementation on our "test/*". It seems tough work to insert correct
> "guard".
That's certainly true. I wouldn't expect that all tests would
immediately start going to the "right place". I believe the
frustration I and others feel mostly stems from the fact that there's
almost no specs coming from Ruby core now, which means those of us
*not* in ruby-core have to reverse-engineer behavior changes and write
specs ourselves. It's a dismal situation, and any improvement would be
welcome.
> (4) Current RubySpec is not portable (especially Windows)
> [ruby-core:39379].
This is just from lack of maintenance. As early as last year the whole
suite ran green on Windows. We can do it again.
> (5) As you know, trunk is not unstable. For example, some modifications
> are revertd soon. Should we add such features as "Specifications"? (I
> know that there is an opinion that the experimental code should have a
> spec).
If MRI maintained a copy of RubySpec in the MRI repository, then it
would be easy to revert spec changes as well. Maybe this would be a
good guide (edits welcome):
* If it is a new experimental feature, tests/specs (if they exist)
could certainly live only in MRI's repository. If that feature becomes
"official" those tests should be migrated to RubySpec.
* If an existing feature changes, the associated specs should change too.
* Anything that's MRI-specific or which Matz deems "implementation
specific" should only be tested by MRI's own suite.
> (6) test/* contains many many corner cases depend on MRI implementation
> to increase code coverage (Thanks Endo-san). It is independent as Ruby
> language, but dependent on MRI. Where should we write such tests?
Those all stay in MRI's suite.
There will never be a clear line between what's
implementation/MRI-specific and what is true specified behavior since
there's no specification (or rather, the only specification that
exists encompasses only a small subset of Ruby). Nobody expects
ruby-core to be perfect here. But when a commit includes visible
behavioral changes that other implementations will eventually want to
duplicate, we *must* have a test/spec somewhere independent of MRI's
tests.
> I think this problem is "who pay efforts on it?".
>
> One solution is continuing current style (using test/*) and someone
> migrate tests to RubySpec if needed. But we need "someone".
In the past, there have been many contributors migrating tests (and
reverse-engineering MRI to write new ones) by hand. That works, if we
can keep up such efforts and keep finding new people as old ones
leave. Migrating tests is a thankless job, so perhaps we need to thank
people more often :)
One way to start this process would be to start at "a" in the existing
test/* tests and just start moving bits to RubySpec. I've done this
for some MRI tests in the past and for many JRuby tests too. It's a
slow process, but it feels great to delete the old test in favor of a
new spec.
> Other solution is MRI committer join to maintain RubySpec. MRI
> developer (including me) should change development style and several
> overhead to do (I think inserting correct "guard" is difficult).
Guarding isn't too bad. The main guards that ruby-core would need to know are:
ruby_version_is/is_not - for specifying a range of Ruby versions, as
Strings, where the behavior is expected to exist
platform_is/is_not - for specifying platform-specific behaviors (e.g. :windows).
There are more complex forms in mspec, but we can help teach them to
ruby-core as we get going.
I think ruby-core would only be responsible for making sure specs pass
on the Ruby versions that RubySpec covers: 1.8.7 and 1.9.2. 1.9.3
changes are probably not covered in specs currently.
> Brian Ford told me that we can have our own RubySpec repository in MRI
> repository and migrating updating specs from MRI's RubySpec repository
> to the RubySpec repository is easy. It is also one solution. However,
> we need "someone" who can migrate. It solves (1), (2) and maybe (3).
For 1.9.2, I believe yugui took the lead on making sure specs were
written for 1.9.2 behaviors (as much as possible...time was limited)
and also ensuring that 1.9.2 passed all specs (with appropriate
version guards, etc). Could it be release manager's responsibility to
ensure specs get written or migrated, perhaps by asking the developers
making changes to write/migrate those specs?
Those of us outside ruby-core really, really want to help make this
happen. Let me know what I personally can do to help.
- Charlie
NARUSE, Yui
> 2011/9/9 Luis Lavena <luisl...@gmail.com>:
>>> One solution is continuing current style (using test/*) and someone
>>> migrate tests to RubySpec if needed. But we need "someone".
>
> Why *you* don't do it?
>
>>> Other solution is MRI committer join to maintain RubySpec. MRI
>>> developer (including me) should change development style and several
>>> overhead to do (I think inserting correct "guard" is difficult).
>
> Run `git shortlog -sn` on RubySpec before say that.
> (yeah I rarely add new spec)
I misunderstood that those original comments are written by Luis,
so those my comments wrong, sorry.
--
NARUSE, Yui <nar...@airemix.jp>
"Martin J. Dürst"
On 2011/09/09 0:29, Charles Oliver Nutter wrote:
> On Wed, Sep 7, 2011 at 4:17 AM, NARUSE, Yui<nar...@airemix.jp> wrote:
>> I don't think RubySpec (RSpec syntax) is beautiful.
>> I want to write assertion with declarative.
>>
>> For example I can't find which is the expectation 4 or 5.
>> Fixnum#div coerces self and the given argument to Floats and returns
>> self divided by other as Fixnum FAILED
>> Expected 4
>> to equal 5
>
> I think this is part of the language barrier I sympathize with.
For most MRI developers, this language barrier is indeed quite high. It
is easy for them to understand that if they want to talk to Ruby
developers who don't speak Japanese, they have to write in English. But
RubySpec is essentially a program, executed by a computer. Why should
they have to write in English for the computer? If you had to choose
between writing Ruby tests in straightforward Ruby and writing them in
convoluted French or Japanese (or pick whatever foreign language you may
(barely) know), which one would you prefer?
Of course some will say that the English used in RubySpec isn't for the
computer, it's for other people who read RubySpec as a specification of
Ruby behavior or who read the error messages. That's partially true. But
contrary to some business-oriented application, implementers of Ruby
should be able to deal with much simpler error messages, even if that
may mean a tad more work when investigating the bug. Also, you may say
that the English contains not only the actual test, but a more general
specification. But then we get into the issue of passing the tests and
believing (potentially wrongly, because the tests only cover individual
cases) that an implementation conforms. That danger is lower if the
tests are just given as tests.
[the following two lines recited for convenience]
>> Expected 4
>> to equal 5
> It's a
> very terse way to say "I expected 5, but the code returned 4". I agree
> it might be a bit clearer to say "Expected 5 but got 4", but this is
> mostly a language issue; as written, it is correct, if confusing for
> you (and others).
Sorry, but here I have to disagree strongly.
It's definitely possible to read this as "Expected 4 (the result of
executing the test code) to equal 5 (the expected result)" if one has
used RubySpec for a while and got used to this form of output. But it's
difficult to do so without such preconditioning.
The straightforward parsing of this sentence is "Expected (4 to equal
5)". This is what anybody with knowledge of English syntax (but no
knowledge of program testing) would give you. In this parsing, it's
totally unclear which value is which.
Another way to look at it is by line. The first line says "Expected 4".
So that must be the expected value. And it's expected to equal the
result of executing the test, which was 5. It's not only by chance that
in your rewording ("Expected 5 but got 4"), you switched values to move
5 closer to Expected. This interpretation is false.
So without knowledge of RubySpec, we either don't know or get it wrong.
[I agree with Michael Klishin that this is just a minor detail, which
can be addressed. But currently, there are quite a few of these "minor
details", and they pile up. It would be a move forward if they were
addressed. And there are also a few not so minor issues. See below.]
> I'm starting to feel like the RubySpec resistance boils down to "it
> has too much English in it."
It doesn't "boil down to". But "it has too much English in it, and often
weird English" is probably one of the major points.
('weird English' includes other aspects of RubySpec such as the very
formalized specification language (which is to some extent necessary for
a specification), the fact that sentences appear piecemeal and the
writer has to put them together in his/her head when writing RubySpec,
and the use of method names that express half of a concept (.should and
friends).)
Other major issues may be:
- Using git. Many CRuby committers are not familiar with git.
- Using RSpec-style specs rather than unit tests. This is to some
extent a matter of taste, and that's difficult to change
(and there have been a lot of discussions about the pros and cons
of both approaches elsewhere, which we don't need to repeat here).
- Cognitive overhead. Fixing CRuby (often very low level C) and writing
unit tests and writing RubySpec and using svn and git all together may
be easy for some people, but not for many.
Regards, Martin.
"Martin J. Dürst"
> I think ruby-core would only be responsible ...
Please don't think of ruby-core as a single entity. It's a set of
people, each of them with different abilities and preferences.
> Those of us outside ruby-core really, really want to help make this
> happen. Let me know what I personally can do to help.
I think that anything that addresses any of the smaller and larger
issues that have come up in the discussion may help. That may range from
fixing the error format (should be very easy) to switching RubySpec from
RSpec-like style to unit test style (I'm half joking here, but half
serious, because I'm sure that would increase contributions from CRuby
people).
If the goal is to have every CRuby committer to add to RubySpec for a
feature change, then that's a very long term goal. It's better if you
take as your goal to increase contributions to RubySpec from CRuby
committers.
Regards, Martin.
NARUSE, Yui
> immediately start going to the "right place". I believe the
> frustration I and others feel mostly stems from the fact that there's
> almost no specs coming from Ruby core now, which means those of us
> *not* in ruby-core have to reverse-engineer behavior changes and write
> specs ourselves. It's a dismal situation, and any improvement would be
> welcome.
You use "ruby-core" as ambiguous meaning.
You should write "the author".
And why you need reverse engineering?
Commit message and rdoc shall describe such information.
If not, you can blame the author.
--
NARUSE, Yui <nar...@airemix.jp>
Charlie Savage
> minw and mswin are different thing, and we usually think mswin as Windows.
Well maybe I am doing something wrong, but when I run the tests on
mswin, using the latest 1.9.3 source from today, they don't even complete.
> ruby -v
ruby 1.9.3dev (2011-09-08 revision 33226) [i386-mswin32_100]
> C:\MinGW\local\src\ruby\win32>nmake test-all
Microsoft (R) Program Maintenance Utility Version 10.00.30319.01
Copyright (C) Microsoft Corporation. All rights reserved.
.\ruby.exe -I./../lib -I".ext/i386-mswin32_100" -I.
"./../tool/runruby.rb" --extout=".ext" -- --disable-gems
"./../test/runner.rb" --ruby=".\ruby.exe -I./../lib
-I".ext/i386-mswin32_100" -I. "./../tool/runruby.rb" --extout=".ext"
-- --disable-gems"
C:/MinGW/local/src/ruby/lib/rdoc/task.rb:322: warning: already initialized
constant RDocTask
Run options: "--ruby=.\\ruby.exe -I./../lib -I.ext/i386-mswin32_100 -I.
./../tool/runruby.rb --extout=.ext -- --disable-gems"
# Running tests:
................................................................................................................................................................................................................................S..............S..................................................................................................................................................................NMAKE
: fatal error U1077: '.\ruby.exe' : return code '0x1'
Stop.
>
Luis Lavena
>>
>> minw and mswin are different thing, and we usually think mswin as Windows.
>
> Well maybe I am doing something wrong, but when I run the tests on mswin,
> using the latest 1.9.3 source from today, they don't even complete.
>
What about nmake test only?
Charlie Savage
> On Fri, Sep 9, 2011 at 1:49 AM, Charlie Savage<cf...@savagexi.com> wrote:
>>>
>>> minw and mswin are different thing, and we usually think mswin as Windows.
>>
>> Well maybe I am doing something wrong, but when I run the tests on mswin,
>> using the latest 1.9.3 source from today, they don't even complete.
>>
>
> What about nmake test only?
Yes, that works, no failures.
But that seems like a subset of all the tests, so not comparable to what
you showed with mingw.
Charlie
-------
C:\MinGW\local\src\ruby\win32>nmake test
PASS all 943 tests
.\ruby.exe -I./../lib -I".ext/i386-mswin32_100" -I.
"./../tool/runruby.rb" --extout=".ext" -- --disable-gems
"./../bootstraptest/runner.rb" --ruby="ruby.exe" ./../KNOWNBUGS.rb
2011-09-08 23:08:10 -0600
Driver is ruby 1.9.3dev (2011-09-08 revision 33226) [i386-mswin32_100]
Target is ruby 1.9.3dev (2011-09-08 revision 33226) [i386-mswin32_100]
Vít Ondruch
There are at least two of the test which fails also on my Fedora, namely:
test_reorganize(TestGDBM)
[C:/Users/Luis/Projects/oss/ruby/test/gdbm/test_gdbm.rb:592]:
test_cgi(TestWEBrickCGI)
[C:/Users/Luis/Projects/oss/ruby/test/webrick/test_cgi.rb:40]:
It doesn't look to be just windows issue ...
Vit
Kirk Haines
of Ruby 1.8.6 that I have done, I have tested against RubySpec prior
to making the release.
Also, in my role at Engine Yard, I am often building packages for the
various Rubies (CRuby 1.8.x/1.9.x, REE, JRuby, Rubinius). Sometimes
these are straight builds. Sometimes we add patches. It depends on the
build. I try to test these builds thoroughly, and that is especially
important if it is a build that has patches applied to it. For
CRubies, this generally involves running both the test/* suite, and
RubySpec. I have found that I need to run both, because there have
been times when differing test coverage has resulted in one suite
running cleanly while the other has an unexpected failure. I have
always found that somewhat frustrating, and I have wondered for a long
time why Ruby behavioral tests are not being added directly to
RubySpec.
The benefit to doing so seems clear to me. While the testing
antecedents relied on test/* in order to regression test Ruby, back
when there was only a single implementation, the world has changed
since then. Ruby the language is bigger than any one implementation.
So while tests that are specific to MRI should clearly go in test/*,
tests which verify that an implementation has the correct Ruby
behavior would be more globally useful if they were added to RubySpec,
as that test suite is intended to support the language as a whole, and
not any one specific implementation.
If there is an untested old behavior, or a new behavior that needs a
corresponding test, placing that test in RubySpec benefits all Ruby
implementations. I've done it in the past, when working on a 1.8.6
release. That is actually what I originally got RubySpec commit access
for.
I was thinking about Urabe's comment that RubySpec only describes
current CRuby behavior, but doesn't proscribe what future behavior
should be. That sort of prescriptive behavior could be implemented
using a guard. A spec that isn't expected to pass currently, on any
version of Ruby, but that describes a behavior that is desired could
be flagged as such. Then, at such a time as an implementation of Ruby
has that behavior implemented, the guard is changed to reflect this.
I have been contemplating a role to take on with ruby-core after the
1.8.6 era closes for good. If there would be value in it, maybe I
could start working on identifying and duplicating tests that are in
test/*, but are lacking in RubySpec.
However, I struggle to understand the fundamental perspective that it
is better for Ruby (not MRI-specific) tests to go into test/* instead
of into RubySpec. Perhaps I am misunderstanding some important
component to the objections?
I understand not being comfortable with git. I will admit that I curse
git sometimes. But for all of the basic capabilities, git is trivial
to learn and to use.
I also understand not liking RSpec, and preferring the declarative
test/unit style. I prefer the test/unit style, personally. But RSpec
is also pretty trivial to learn sufficiently to make it a useful tool.
I am far from being a RubySpec expert, RSpec expert, or git expert,
but I would also offer to help anyone in ruby-core who wants to try
using RubySpec or adding a test to RubySpec, if they need someone who
has done it before to provide some assistance.
Thanks much,
Kirk Haines
Urabe Shyouhei
> I was thinking about Urabe's comment that RubySpec only describes
> current CRuby behavior, but doesn't proscribe what future behavior
> should be. That sort of prescriptive behavior could be implemented
> using a guard. A spec that isn't expected to pass currently, on any
> version of Ruby, but that describes a behavior that is desired could
> be flagged as such. Then, at such a time as an implementation of Ruby
> has that behavior implemented, the guard is changed to reflect this.
No, sorry. I was not talking about such things.
For instance, when you look at current rubyspec, you see
sprintf("% u", -123).should == "-123"
and you never know WHY IT SHOULD. I don't believe this is a proper
"standard" behaviour that all CRuby committers can agree. Giving
negative numbers for %u a programmer should have intended to cast it
to an unsigned integer. This spec does describe the status quo of
our implementation but not for the "ideal" figure.
There are many such specs in the project.
Michael Klishin
> and you never know WHY IT SHOULD. I don't believe this is a proper
> "standard" behaviour that all CRuby committers can agree. Giving
> negative numbers for %u a programmer should have intended to cast it
> to an unsigned integer. This spec does describe the status quo of
> our implementation but not for the "ideal" figure.
test/* tests do not explain that either. And no documentation does. It is not surprising for a project that describes some existing implementation to only claim how things work, not why.
Nobody says that RubySpec is "ideal". Specs that explain *why* things work in certain ways take forever to write and only can be written in retrospect after years of trial and error, take a look at HTML5 and related specifications for example.
I think the point of this thread originally was to encourage MRI developers to *use* RubySpec more (it needs some work on Windows, but other than that, it is a giant body of work that is already done) and contribute to it when behavior of the language changes, not claim that RubySpec is ideal or an absolute of some kind.
MK
Urabe Shyouhei
> Urabe Shyouhei:
>
>> and you never know WHY IT SHOULD. I don't believe this is a proper
>> "standard" behaviour that all CRuby committers can agree. Giving
>> negative numbers for %u a programmer should have intended to cast it
>> to an unsigned integer. This spec does describe the status quo of
>> our implementation but not for the "ideal" figure.
>
> test/* tests do not explain that either. And no documentation does. It is not surprising for a project that describes some existing implementation to only claim how things work, not why.
I see RubySpec is too careless about adding those questionable specs.
FYI test/* do not have tests for %u, because we believe it's not yet
the time to specify this.
> Nobody says that RubySpec is "ideal". Specs that explain *why* things work in certain ways take forever to write and only can be written in retrospect after years of trial and error, take a look at HTML5 and related specifications for example.
But headius wants the ideal standard. That was the starting point
(look again).
> I think the point of this thread originally was to encourage MRI developers to *use* RubySpec more (it needs some work on Windows, but other than that, it is a giant body of work that is already done) and contribute to it when behavior of the language changes, not claim that RubySpec is ideal or an absolute of some kind.
Did you know that almost all members in this thread (except me, sorry)
was actually top 10 most active rubyspec committers? Yui's being
against it after contributing this much should be considered.
Yusuke ENDOH
2011/9/9 Urabe Shyouhei <shyo...@ruby-lang.org>:
> FYI test/* do not have tests for %u, because we believe it's not yet
> the time to specify this.
Really? I don't think that writing a test into test/* does means
specifying a spec. One of test/* roles is to prevent an unintentional
behavior change (or, regression). So, even if it is a spec or not,
test/* should have tests for %u. In fact, there is a test for %u in
"test_printf_diu" of test/ruby/test_integer_comb.rb, which seems to
check if: ("%u" % n).to_i == n
--
Yusuke Endoh <ma...@tsg.ne.jp>
Urabe Shyouhei
Oops, sorry then. I was not aware of it.
So both RubySpec and test/* cannot be what headius want ("tests for
features considered standard Ruby" he says).
KOSAKI Motohiro
>
> test_reorganize(TestGDBM)
> [C:/Users/Luis/Projects/oss/ruby/test/gdbm/test_gdbm.rb:592]:
> test_cgi(TestWEBrickCGI)
> [C:/Users/Luis/Projects/oss/ruby/test/webrick/test_cgi.rb:40]:
>
> It doesn't look to be just windows issue ...
My fedora box can't reproduce neither. We need more accurate bug reports...