Issue 20: performance plugin
James Casbon
I cut a branch with has an implementation of a performance plugin.
See branches/20-performance-plugin.
At the moment, it simply reruns each test a number of times and gives
you a report such as this:
"""
----------------------------------------------------------------------
Test Name Min time Avg time Max time
test_fasta.perf_large_sequence_string_join_parser 9.667s 9.747s 9.871s
test_fasta.perf_large_sequence_stringio_parser 9.726s 9.810s 9.912s
test_fasta.perf_lots_of_fragments_string_append_parser 0.240s 0.243s
0.247s
test_fasta.perf_lots_of_fragments_string_join_parser 0.245s 0.245s
0.245s
test_fasta.perf_lots_of_fragments_stringio_parser 0.263s 0.266s 0.272s
----------------------------------------------------------------------
Ran 15 tests in 60.938s
"""
It's already useful, but any feedback appreciated.
ticket link: http://code.google.com/p/python-nose/issues/detail?id=20
code link: http://code.google.com/p/python-nose/source/browse/branches/20-performance-plugin/nose/plugins/performance.py
cheers,
James
PS would it be better if ticket changes were cced to the list to give
some visibility?
Kumar McMillan
that's really neat! Here are some quick wins that would help get this
into trunk sooner:
- Add docstrings for each plugin hook implemented so we can
incorporate it into the new sphinx autodoc pages for plugins
- Add at least one functional test (using PluginTester, etc) and/or
add some unit tests for each plugin hook so we can maintain it
properly. The nose suite is tested against 2.3, 2.4, 2.5, 2.6 + so
having tests for the plugin itself would ensure interoperability.
Also what would be nice -- but this is a bigger feature and not a
necessity -- is to have the output be sortable somehow so you can
identify the slowest tests in a suite easily. I don't know how best
to present that to the user. Maybe dump it to a file in finalize() in
some sorted way for inspection.
-Kumar
James Casbon
>
> Hi James,
> that's really neat! Here are some quick wins that would help get this
> into trunk sooner:
Thanks :)
> - Add docstrings for each plugin hook implemented so we can
> incorporate it into the new sphinx autodoc pages for plugins
mkay - I kind of dislike stating the bleedin' obvious but will do.
> - Add at least one functional test (using PluginTester, etc) and/or
> add some unit tests for each plugin hook so we can maintain it
> properly. The nose suite is tested against 2.3, 2.4, 2.5, 2.6 + so
> having tests for the plugin itself would ensure interoperability.
Already got functional tests checked in.
> Also what would be nice -- but this is a bigger feature and not a
> necessity -- is to have the output be sortable somehow so you can
> identify the slowest tests in a suite easily. I don't know how best
> to present that to the user. Maybe dump it to a file in finalize() in
> some sorted way for inspection.
Should be possible to pass the sort parameter as a command line option
easily enough. Could also output csv so you can load it into a
spreadsheet?
Kumar McMillan
>> - Add docstrings for each plugin hook implemented so we can
>> incorporate it into the new sphinx autodoc pages for plugins
>
> mkay - I kind of dislike stating the bleedin' obvious but will do.
yeah me too; sphinx sheepishly will not add the methods otherwise
>
>> - Add at least one functional test (using PluginTester, etc) and/or
>> add some unit tests for each plugin hook so we can maintain it
>> properly. The nose suite is tested against 2.3, 2.4, 2.5, 2.6 + so
>> having tests for the plugin itself would ensure interoperability.
>
> Already got functional tests checked in.
cool, sorry, missed those.
>
>> Also what would be nice -- but this is a bigger feature and not a
>> necessity -- is to have the output be sortable somehow so you can
>> identify the slowest tests in a suite easily. I don't know how best
>> to present that to the user. Maybe dump it to a file in finalize() in
>> some sorted way for inspection.
>
> Should be possible to pass the sort parameter as a command line option
> easily enough. Could also output csv so you can load it into a
> spreadsheet?
sure, that would be useful.
K
casbon
I added a csv output option. It's not ideal but it gives you some
idea. You really need all sorts of other information in there for it
to be useful: date and time, revision, system information. That
doesn't fit too well into a csv file though.
Kumar McMillan
>
>
>
> On Mar 30, 5:40 pm, Kumar McMillan <kumar.mcmil...@gmail.com> wrote:
>> On Mon, Mar 30, 2009 at 11:20 AM, James Casbon <cas...@gmail.com> wrote:
>> >> Also what would be nice -- but this is a bigger feature and not a
>> >> necessity -- is to have the output be sortable somehow so you can
>> >> identify the slowest tests in a suite easily. I don't know how best
>> >> to present that to the user. Maybe dump it to a file in finalize() in
>> >> some sorted way for inspection.
>>
>> > Should be possible to pass the sort parameter as a command line option
>> > easily enough. Could also output csv so you can load it into a
>> > spreadsheet?
>>
>> sure, that would be useful.
>
> Docstrings in. Where does this sphinx documentation live?
Not sure if you are working off an hg clone or off of svn. There is
an hg clone which has been pulled from svn but I don't think much has
been pushed back to svn yet:
http://bitbucket.org/jpellerin/nose-pycon/
The sphinx branch is 170-sphinx-docs but I wouldn't worry too much
about adding the plugin there. It would be as simple as ..
automodule:: nose.plugins.performance or whatever. The sphinx docs
still need some organization work. Pam was working on that but I'm
not sure if she is on devs (cc'd).
>
> I added a csv output option. It's not ideal but it gives you some
> idea. You really need all sorts of other information in there for it
> to be useful: date and time, revision, system information. That
> doesn't fit too well into a csv file though.
Neat
Pamela Zerbinos
the organization itself is largely done, but there are a lot of holes
and other issues that need dealing with. The stuff I did is, as far as
I know, still in my fork:
http://bitbucket.org/catechism/nose-pycon
-Pam
jason pellerin
Are you ready for me to pull your changes? I'm back on toddler time
now that I'm home so I can't guarantee it'll be tonight, but if you're
ready I'll get it pulled ASAP.
JP
Pamela Zerbinos
-p.
Pamela Zerbinos
-p.