io_rewiring branch needs testing
Andrew Whitworth
point where we would like to get some extra testing. Not only do we
need testing to prove that the branch isn't broken and stupid, but we
also want verification that the work in this branch does represent an
actual performance improvement. Infinoid has done one test that
suggests a 4x speedup for IO-intensive programs, which would be very
nice indeed.
What the branch does, most basically, is to optimize for the common
cases of built-in IO PMC types: FileHandle and StringHandle mostly. If
a different type of PMC is passed to an IO routine, it falls back to a
PCCINVOKE on a method to handle the operation. So, in the case where
we use built-in PMC types and direct subclasses of them, things should
be faster. Otherwise, performance should stay the same as it was.
While there is more work to do in the IO system, this branch is stable
and I think is suitable for merging as-is, pending testing.
Speciflcally I would like to hear if Rakudo and Partcl work well on
this branch, and if they experience any performance improvements,
however small.
Thanks,
--Andrew Whitworth
_______________________________________________
http://lists.parrot.org/mailman/listinfo/parrot-dev
Mark Glines
> Infinoid++ have been working on the io_rewiring branch and are at a
> point where we would like to get some extra testing. Not only do we
> need testing to prove that the branch isn't broken and stupid, but we
> also want verification that the work in this branch does represent an
> actual performance improvement. Infinoid has done one test that
> suggests a 4x speedup for IO-intensive programs, which would be very
> nice indeed.
Andrew didn't give himself enough credit in the above paragraph; he did
the vast majority of the work on this branch. Whiteknight++.
We're looking for better ways to test and benchmark it, and we'd also
love some assurance that things work (or reports that they don't work)
for lots of HLLs.
Thanks!
Mark
_______________________________________________
http://lists.parrot.org/mailman/listinfo/parrot-dev
Will Coleda
This branch causes quite a few failures for partcl.
Some of them seem to be issues that have already been resolved in
trunk, including having TclInts change into TclFloats, namespace
pollution... I don't see any merges in from trunk since the branch was
created, so this is hopefully just a matter of merging trunk into the
branch, which is probably a good idea to do before asking for non-core
testing anyway.
Here's the results from 'prove' in partcl's build dir against an
installed copy of the io_rewiring branch.
Test Summary Report
-------------------
t/cmd_break.t (Wstat: 0 Tests: 4 Failed: 1)
Failed test: 2
t/cmd_catch.t (Wstat: 0 Tests: 13 Failed: 1)
Failed test: 8
t/cmd_expr.t (Wstat: 0 Tests: 130 Failed: 0)
Parse errors: Bad plan. You planned 322 tests but ran 130.
t/cmd_foreach.t (Wstat: 0 Tests: 16 Failed: 1)
Failed test: 14
t/cmd_incr.t (Wstat: 0 Tests: 15 Failed: 4)
Failed tests: 2-3, 6, 12
t/cmd_lsort.t (Wstat: 256 Tests: 22 Failed: 0)
Non-zero exit status: 1
t/cmd_namespace.t (Wstat: 0 Tests: 42 Failed: 3)
Failed tests: 5-6, 9
t/cmd_parray.t (Wstat: 512 Tests: 9 Failed: 2)
Failed tests: 2-3
Non-zero exit status: 2
t/cmd_rename.t (Wstat: 256 Tests: 9 Failed: 1)
Failed test: 2
Non-zero exit status: 1
t/cmd_while.t (Wstat: 0 Tests: 6 Failed: 1)
Failed test: 5
t/tcl_misc.t (Wstat: 0 Tests: 32 Failed: 1)
Failed test: 11
t/tcl_namespace.t (Wstat: 0 Tests: 6 Failed: 1)
Failed test: 4
Files=70, Tests=1065, 237 wallclock secs ( 0.26 usr 0.99 sys + 213.01
cusr 7.96 csys = 222.22 CPU)
Result: FAIL
Against trunk, with all tests passing, the result was:
Files=70, Tests=1257, 221 wallclock secs ( 0.23 usr 0.21 sys + 198.28
cusr 9.88 csys = 208.60 CPU)
So it's taking longer to fail in the branch than it is to pass in trunk.
Here's some timings on some of the tests that does still run to completion.
Branch @39443
[16:41:53] t/cmd_after.t .......... ok 1572 ms
[16:41:55] t/cmd_append.t ......... ok 2017 ms
[16:41:57] t/cmd_apply.t .......... ok 2489 ms
[16:41:59] t/cmd_array.t .......... ok 6444 ms
[16:42:06] t/cmd_binary.t ......... ok 2629 ms
Trunk @39443
[16:45:17] t/cmd_after.t .......... ok 1273 ms
[16:45:18] t/cmd_append.t ......... ok 1679 ms
[16:45:20] t/cmd_apply.t .......... ok 2033 ms
[16:45:22] t/cmd_array.t .......... ok 4932 ms
[16:45:27] t/cmd_binary.t ......... ok 2179 ms
Looks like the branch is consistently slower than trunk, at least for partcl.
Sorry.
--
Will "Coke" Coleda
_______________________________________________
http://lists.parrot.org/mailman/listinfo/parrot-dev
Andrew Whitworth
> This branch causes quite a few failures for partcl.
>
> Some of them seem to be issues that have already been resolved in
> trunk, including having TclInts change into TclFloats, namespace
> pollution... I don't see any merges in from trunk since the branch was
> created, so this is hopefully just a matter of merging trunk into the
> branch, which is probably a good idea to do before asking for non-core
> testing anyway.
I merged trunk into branch so we could try to get benefit from some of
the improvements that have been made to trunk since the branch
started. There are some packfile-related failures because we bumped
PBC_COMPAT and we're fixing that now. Could you do me a favor and run
the tests again with trunk and branch at r39444, which is when I
merged? that should show a pretty even footing.
Does TCL use FileHandle and subclasses of it, or does it use other
custom types for IO?
Will Coleda
> On Sun, Jun 7, 2009 at 10:48 AM, Will Coleda<wi...@coleda.com> wrote:
>> This branch causes quite a few failures for partcl.
>>
>> Some of them seem to be issues that have already been resolved in
>> trunk, including having TclInts change into TclFloats, namespace
>> pollution... I don't see any merges in from trunk since the branch was
>> created, so this is hopefully just a matter of merging trunk into the
>> branch, which is probably a good idea to do before asking for non-core
>> testing anyway.
>
> I merged trunk into branch so we could try to get benefit from some of
> the improvements that have been made to trunk since the branch
> started. There are some packfile-related failures because we bumped
> PBC_COMPAT and we're fixing that now. Could you do me a favor and run
> the tests again with trunk and branch at r39444, which is when I
> merged? that should show a pretty even footing.
trunk @39444 == trunk @39443
> Does TCL use FileHandle and subclasses of it, or does it use other
> custom types for IO?
>
> --Andrew Whitworth
>
partcl isn't using any custom IO (yet). Everything is, I think, just
straight opcodes.
Branch now completes. 'prove' results:
Files=70, Tests=1257, 215 wallclock secs ( 0.30 usr 0.37 sys + 195.62
cusr 9.39 csys = 205.68 CPU)
Result: FAIL
So, that's 215s now in branch vs. 221s a few hours ago in trunk. (only
ran 'prove' once each time)
Andrew Whitworth
>> I merged trunk into branch so we could try to get benefit from some of
>> the improvements that have been made to trunk since the branch
>> started. There are some packfile-related failures because we bumped
>> PBC_COMPAT and we're fixing that now. Could you do me a favor and run
>> the tests again with trunk and branch at r39444, which is when I
>> merged? that should show a pretty even footing.
>
> trunk @39444 == trunk @39443
Sorry, I should have been more clear. I merged trunk INTO the branch,
not the other way around.
iorewiring @39444 != io_rewiring@39443 != trunk @39444
> partcl isn't using any custom IO (yet). Everything is, I think, just
> straight opcodes.
Striaght opcode on vanilla IO PMCs?
> Branch now completes. 'prove' results:
>
> Files=70, Tests=1257, 215 wallclock secs ( 0.30 usr 0.37 sys + 195.62
> cusr 9.39 csys = 205.68 CPU)
> Result: FAIL
>
> So, that's 215s now in branch vs. 221s a few hours ago in trunk. (only
> ran 'prove' once each time)
So that's a small improvement, which I guess is better then nothing.
At least it's not a regression, which is always something.
Will Coleda
> On Sun, Jun 7, 2009 at 8:25 PM, Will Coleda<wi...@coleda.com> wrote:
>>> I merged trunk into branch so we could try to get benefit from some of
>>> the improvements that have been made to trunk since the branch
>>> started. There are some packfile-related failures because we bumped
>>> PBC_COMPAT and we're fixing that now. Could you do me a favor and run
>>> the tests again with trunk and branch at r39444, which is when I
>>> merged? that should show a pretty even footing.
>>
>> trunk @39444 == trunk @39443
>
> Sorry, I should have been more clear. I merged trunk INTO the branch,
> not the other way around.
Yes, I know; just commenting on your request to test trunk @39444 when
I had already tested it at an equivalent revision.
> iorewiring @39444 != io_rewiring@39443 != trunk @39444
>
>> partcl isn't using any custom IO (yet). Everything is, I think, just
>> straight opcodes.
>
> Striaght opcode on vanilla IO PMCs?
Yes. I'm not explicitly creating ANY pmcs for IO, let alone custom ones.
>> Branch now completes. 'prove' results:
>>
>> Files=70, Tests=1257, 215 wallclock secs ( 0.30 usr 0.37 sys + 195.62
>> cusr 9.39 csys = 205.68 CPU)
>> Result: FAIL
>>
>> So, that's 215s now in branch vs. 221s a few hours ago in trunk. (only
>> ran 'prove' once each time)
>
> So that's a small improvement, which I guess is better then nothing.
> At least it's not a regression, which is always something.
>
> --Andrew Whitworth
Without a more thorough benchmark test, that could just be line noise.
(That's why I pointed out I just ran prove once each time.)
Regards.
Patrick R. Michaud
> Andrew Whitworth wrote:
> > Infinoid++ have been working on the io_rewiring branch and are at a
> > point where we would like to get some extra testing. Not only do we
> > need testing to prove that the branch isn't broken and stupid, but we
> > also want verification that the work in this branch does represent an
> > actual performance improvement. Infinoid has done one test that
> > suggests a 4x speedup for IO-intensive programs, which would be very
> > nice indeed.
>
> love some assurance that things work (or reports that they don't work)
> for lots of HLLs.
In early February I posted a message [1] to parrot-dev that
demonstrated how Parrot's (current) IO system is very slow.
In the new branch the test program I ran is indeed about 4 times
faster:
trunk : 48.394s
io_rewiring: 11.151s
So, good work thus far!
By way of comparison, the 0.8.0 version of Parrot (pre-I/O-refactor)
runs the benchmark in 10.437s, which means the branch is getting
close to what Parrot once had. It's entirely possible that the
difference here is due to startup or other non-I/O items, too.
Another comparison: Perl 5 is able to perform the equivalent
set of operations in 4.554s. It seems to me that for
straightforward I/O such as this Parrot really ought to be a
bit closer to the Perl 5 speed.
Pm
[1] http://lists.parrot.org/pipermail/parrot-dev/2009-February/001356.html
_______________________________________________
http://lists.parrot.org/mailman/listinfo/parrot-dev
Mark Glines
> In the new branch the test program I ran is indeed about 4 times
> faster:
>
> trunk : 48.394s
> io_rewiring: 11.151s
>
> So, good work thus far!
>
> By way of comparison, the 0.8.0 version of Parrot (pre-I/O-refactor)
> runs the benchmark in 10.437s, which means the branch is getting
> close to what Parrot once had. It's entirely possible that the
> difference here is due to startup or other non-I/O items, too.
>
> Another comparison: Perl 5 is able to perform the equivalent
> set of operations in 4.554s. It seems to me that for
> straightforward I/O such as this Parrot really ought to be a
> bit closer to the Perl 5 speed.
Branch performance for your benchmark should be much improved by r39450.
Here are my numbers:
trunk : ./parrot x.pir 12.37s user 1.67s system 56% cpu 24.747 total
branch: ./parrot x.pir 0.42s user 0.06s system 64% cpu 0.741 total
Executive summary: We were spending all our time calling fsync() after
every call to write(), for no good reason. This also has a drastic
effect on "coretest".
Mark
_______________________________________________
http://lists.parrot.org/mailman/listinfo/parrot-dev