Running tests

[Leo Franchi]

Hi guys,

In adding some features I noticed that it's taking me around 17s to run `nosetests` each time, which is a frustrating experience :) Are there any ways to cut down on the number of tests run that actually help? I've tried to tell nose to only run the bootstrap tests (even going so far as to delete all the other tests) but I'm still seeing at least 4 or 5s of launch time before any tests are written.

Tips appreciated :) I'm happy to look into making it faster if there's no way right now---if anyone has any idea where the time is going (or ways to profile it) I'll gladly take a look.

cheers,
leo

[Eric Shull]

If I recall, the startup time is due to bootstrapping Clojure-Py. It takes a while to execute the core, which is why it still takes a few seconds even after removing all the tests. Feel free to look into speeding it up :)

The way our tests are bootstrapped, I'm not sure if nose can distinguish between different sets of them like it does for straight Python tests. If you want to fiddle around with it, check out tests/bootstrop-clj-tests.py. I believe Tim wrote that in short order when we hooked Clojure-Py up to TravisCI, and it's pretty much been left as is. I imagine there's room for improvement, so have at it.

Eric

[Antony Lee]

To be precise, the problem comes from the fact that we currently cannot
produce "compiled" clojure-py files (� la .pyc), so clojure.py has to
parse the ~4k lines of core.clj every time. I haven't looked at that in
detail but I remember this having something to do with the low-level
bytecodes themselves so it may not be that easy to fix.

The project has not seen much activity recently but I'd actually like to
start working on it again too. A few possible directions I can think of:

- bytecode issues: support compiled clj-py, support python3 (I am all
for using python3 as much as possible) -- I should look at the treadle
branch more in detail.

- protocols: perhaps I'm just too perfectionnist but I *hate* the way
this is currently implemented -- (clojure) builtin classes inherit
from the protocols they extend which is contrary to the basic ideas
behind protocols; also the existence of the MRO in python creates
some problems that do not exist in clojure-jvm (say protocol P
extends classes A and B, and class C inherits from A and B; now P's
functions will need to know whether A or B comes first in C's MRO to
find which implementation to use) and I do not have a trivial answer
to these (I mean sure you can check C.__mro__ but at that point I just
don't see the difference between protocols and normal python methods
anymore) (see previous postings on the list for more of my rants on
this).

- pypy: see how much speed we can get from that?

Antony

[John Gabriele]

On Mon, Oct 22, 2012 at 3:13 PM, Antony Lee <anntz...@gmail.com> wrote:
>
> The project has not seen much activity recently but I'd actually like to
> start working on it again too. A few possible directions I can think of:
>
> - bytecode issues: support compiled clj-py, support python3 (I am all
> for using python3 as much as possible) -- I should look at the treadle
> branch more in detail.
>

> - protocols: {snip}

>
> - pypy: see how much speed we can get from that?

It appears to me that there's always been a split in goals for clojure-py:

1. On the one hand, I think Timothy and others had goals
for serious execution speed, likely via pypy.

2. And on the other hand, there's surely also interest in just
having something run reasonably fast on CPython (with
pretty quick startup time).

My guess is that that split has held the project back a bit.

---John

[John Gabriele]

On Mon, Oct 22, 2012 at 4:46 PM, John Gabriele <jmg...@gmail.com> wrote:
>
> 2. And on the other hand, there's surely also interest in just
> having something run reasonably fast on CPython (with
> pretty quick startup time).
>
> My guess is that that split has held the project back a bit.
>

Actually, come to think of it, I'm sure there was more to it. This was
discussed here and there in a lot of detail on the Clojure ML, and ---
though I didn't understand all (much? :) ) of it --- I got the
impression that speeding up the startup time was a thorny enough
problem that other avenues were being explored.

[Antony Lee]

Running clojure-py through statprof.py shows that the 3 sec. or so it
takes to start (on my laptop) mostly comes from compiling core.clj
(anyways there is not much else to do on startup), so hopefully having
cljc's (aka pyc's) will be helpful.

Here is what Timothy said on that regard on that mailing list 6 months
ago:

===
If we can get pyc files generated from clojure code then yes, they'd
work just as well for pypy as they would for cpython. However,
actually pulling this off is going to be a bit more complex than I
thought. pyc files use the marshal module and this module can't
serialize user classes. So either we have to use something different
(like pickle) or we try to generate code differently. For example:

(:foo mymap)

compiles to:

LOAD_CONST :foo
LOAD_FAST mymap
CALL_FUNCTION 1

The issue is that marshal can't serialize :foo, and pickle can't
serialize functions at all. Now I could do this:

LOAD_CONST keyword
LOAD_CONST "foo"
CALL_FUNCTION 1
LOAD_FAST mymap
CALL_FUNCTION 1

but then we have the insane overhead of interning a keyword every time
we simply want get something from a map.

So all that to say, I'm still figuring out the best way to do this.

Timothy
===

Antony

[Leo Franchi]

On Oct 22, 2012, at 2:44 PM, Eric Shull <eric....@gmail.com> wrote:

If I recall, the startup time is due to bootstrapping Clojure-Py. It takes a while to execute the core, which is why it still takes a few seconds even after removing all the tests. Feel free to look into speeding it up :)

The way our tests are bootstrapped, I'm not sure if nose can distinguish between different sets of them like it does for straight Python tests. If you want to fiddle around with it, check out tests/bootstrop-clj-tests.py. I believe Tim wrote that in short order when we hooked Clojure-Py up to TravisCI, and it's pretty much been left as is. I imagine there's room for improvement, so have at it

I guess the reason I was wondering is that it seems to take so much longer to run the nosetests on my machine (15-20s w/ cold cache) than to launch a clojure-py repl. I imagine both need to run the whole core.clj bootstrap process, so they should be in the same ballpark---right? Even disabling the bootstrapper and the python-only tests, it seems to take longer to launch than a simple REPL.

Anyway, It was more a curiosity, I don't think I have the time to investigate that right now. Thanks for all the helpful responses though :)

cheers,

leo