Trying to rebuild the test bot

[Juan Jose Garcia-Ripoll]

Hi Anton,

I have finally recovered my broken computer and the data, but I seem to be unable to guess how to run the tests in a way that I keep the data and may do the comparisons.

I know you said that the API has changed and that you dropped the local storage completely, but how I am now to do the check for regressions? Could you drop some example here?

Sorry for the long time, but work and the failure of that computer have kept me from doing this earlier.

Best

Juanjo

--
Instituto de Física Fundamental, CSIC
c/ Serrano, 113b, Madrid 28006 (Spain) 
http://juanjose.garciaripoll.googlepages.com

[Juan Jose Garcia-Ripoll]

I just saw an old email with this text

To submit results into different storage configure agent like this:

    (setf (test-grid-agent::result-storage-name *agent*) "mystorage")

    ;; and to access the data

    (defparameter *r* (test-grid-storage:make-replica "mystorage" "db-my.lisp"))

For example, I am now testing development versions of ECL and submit results

into storage named "ecl". Anyone can access it using

    (defparameter *r* (test-grid-storage:make-replica "ecl" "db-ecl.lisp"))

I understand then I only have to take your code for comparing versions and use this procedure to download the database and do the comparison of just my data, am I right?

[Anton Vodonosov]

23.03.2013, 13:19, "Juan Jose Garcia-Ripoll" <juanjose.g...@gmail.com>:

> I just saw an old email with this text
>> To submit results into different storage configure agent like this:
>>

>> О©╫ О©╫ (setf (test-grid-agent::result-storage-name *agent*) "mystorage")
>> О©╫ О©╫ ;; and to access the data
>> О©╫ О©╫ (defparameter *r* (test-grid-storage:make-replica "mystorage" "db-my.lisp"))

>>
>> For example, I am now testing development versions of ECL and submit results
>> into storage named "ecl". Anyone can access it using
>>

>> О©╫ О©╫ (defparameter *r* (test-grid-storage:make-replica "ecl" "db-ecl.lisp"))

>
> I understand then I only have to take your code for comparing versions and use this procedure to download the database and do the comparison of just my data, am I right?

Right.

tg-storage:make-replica only creates "empty" lisp object. You should also add

(tg-storage:sync *r*)

It actually fetches data from online storage, saving the snapshot into "db-ecl.lisp" file for future faster access.

Then you do (tg-storage:data *r*) => returns exactly the same data structure as you previously had with
(test-grid-data:make-db (local-result-store:list-test-run-results *local-db-path*)).

Let me know if you if you need more info.

Best regards,
- Anton

[Anton Vodonosov]

Juan Jose, have you noticed that the test results
you collected in ecl-bytecode and ecl-lisp-to-c online storages
have every test :CRASHed.

Something must be wrong with your setup, because
I successfully tested ecl-12.2.1 on linux with
quicklisp 2013-03-12.

You can try to reproduce the crash by running ECL
manually with the quicklisp installed agent for testing
and trying to ql:quickload libraries.

The exact command line for ECL invocations performed
by agent may be found in <agent-work-dir>/logs/agent.log.

Best regards,
- Anton

[Juan Jose Garcia-Ripoll]

On Sun, Mar 24, 2013 at 5:18 AM, Anton Vodonosov <avodo...@yandex.ru> wrote:

You can try to reproduce the crash by running ECL
manually with the quicklisp installed agent for testing
and trying to ql:quickload libraries.

I noticed and the libraries work just fine when loaded independently :-/ I did not have a look at the commands that cl-test-grid uses, though.

[Juan Jose Garcia-Ripoll]

On Sun, Mar 24, 2013 at 5:18 AM, Anton Vodonosov <avodo...@yandex.ru> wrote:

Something must be wrong with your setup, because
I successfully tested ecl-12.2.1 on linux with
quicklisp 2013-03-12.

The tests were run with a more recent version of ECL. The problem is that my build bot stripped the .git directory and thus the commit id was missing. I confirmed that this more recent version had problems with loading prebuilt ASDF files -- I am looking into that and running an older ECL to be the baseline for comparisons.

Anton, will the old tests get deleted if I run new ones with the same versions of ECL and quicklisp? It would be nice to be able to overwrite them.

[Juan Jose Garcia-Ripoll]

On Sun, Mar 24, 2013 at 5:18 AM, Anton Vodonosov <avodo...@yandex.ru> wrote:

Something must be wrong with your setup, because
I successfully tested ecl-12.2.1 on linux with
quicklisp 2013-03-12.

ECL-12.2.1 does not work with that quicklisp. I have verified it. The problem lays in ASDF and prebuild files (like sockets.asd), but since this is an old release, I am not going to spend too much time tracking it.

ECL.12.12.1 seems to be the last release that works with quicklisp. I am running the tests again with this release, this time only for the compiled ECL, to save some time.

[Anton Vodonosov]

24.03.2013, 14:02, "Juan Jose Garcia-Ripoll" <juanjose.g...@gmail.com>:

> On Sun, Mar 24, 2013 at 5:18 AM, Anton Vodonosov <avodo...@yandex.ru> wrote:
>> Something must be wrong with your setup, because
>> I successfully tested ecl-12.2.1 on linux with
>> quicklisp 2013-03-12.
>
> ECL-12.2.1 does not work with that quicklisp. I have verified it. The problem lays in ASDF and prebuild files (like sockets.asd), but since this is an old release, I am not going to spend too much time tracking it.
>
> ECL.12.12.1 seems to be the last release that works with quicklisp. I am running the tests again with this release, this time only for the compiled ECL, to save some time.
>
> Juanjo

Sorry, I meant 12.12.1.

Yes. Older ECLs doesn't work due to ASDF upgrage problem.

[Anton Vodonosov]

24.03.2013, 13:20, "Juan Jose Garcia-Ripoll" <juanjose.g...@gmail.com>:

> Anton, will the old tests get deleted if I run new ones with the same versions of ECL and quicklisp? It would be nice to be able to overwrite them.

The results are not overwritten, but accumulated.

It is possible to delete results explicitly by tg-data:remove-test-runs transaction.
I just did so for the :CRASHed test runs.

Removal of test results is rare operation, when something crashed like we had now.

Typically we limit the results to a subset we are interested in using a lisp predicate.

(cl:remove-if-not (lambda (r)
(string= "ecl-12.12.1-256126ac-macosx-x86-lisp-to-c" (tg-reporting::lisp r)))
*all-results*)

This returns only results for ecl-12.12.1-256126ac-macosx-x86-lisp-to-c.

I use this so often, that I even have a more readable version of cl:remove-if-not in the test-grid-reporting package:

(tg-rep::subset *all-results*
(lambda (r)
(string= "ecl-12.12.1-256126ac-macosx-x86-lisp-to-c" (tg-rep::lisp r))))

It is equal to the previous call.

BTW, taking the above into account, you may find it more convenient to use a single online storage for your results,
instead of working with two "ecl-bytecode"and "ecl-lisp-to-c".

I already use a storage named "ecl" to publish results of ECL pre-release versions.
You can also publish into the "ecl" storage, or use your own, separate storage.

Best regards,
- Anton

[Juan Jose Garcia-Ripoll]

On Sun, Mar 24, 2013 at 11:27 AM, Anton Vodonosov <avodo...@yandex.ru> wrote:

I already use a storage named "ecl" to publish results of ECL pre-release versions.
You can also publish into the "ecl" storage, or use your own, separate storage.

Hi Anton, sorry for the inconvenience of creating multiple storages. I misunderstood how it works. I may now use the "ecl" storage, since filtering of tests now works.

The final results are shown here: http://ecls.sourceforge.net/reports-generated/ecl/index.html

The code I used for generating the files is attached. Please comment if you see something non-compliant with the cl-test-grid API

[Anton Vodonosov]

25.03.2013, 16:00, "Juan Jose Garcia-Ripoll" <juanjose.g...@gmail.com>:

> On Sun, Mar 24, 2013 at 11:27 AM, Anton Vodonosov <avodo...@yandex.ru> wrote:
>> I already use a storage named "ecl" to publish results of ECL pre-release versions.
>> You can also publish into the "ecl" storage, or use your own, separate storage.
>
> Hi Anton, sorry for the inconvenience of creating multiple storages. I misunderstood how it works. I may now use the "ecl" storage, since filtering of tests now works.

No problem, use whatever is convenient for you, 1 storage, 2 sotrgaes, 10 storages

> The final results are shown here:О©╫http://ecls.sourceforge.net/reports-generated/ecl/index.html

>
> The code I used for generating the files is attached. Please comment if you see something non-compliant with the cl-test-grid API

You mistakenly attached the shell script (the file contains shell command). Could you send the lisp code?

[Juan Jose Garcia-Ripoll]

On Tue, Mar 26, 2013 at 5:51 PM, Anton Vodonosov <avodo...@yandex.ru> wrote:

You mistakenly attached the shell script (the file contains shell command). Could you send the lisp code?

Sorry, here it goes!

[Anton Vodonosov]

Juan Jose,

I have reviewed the ecl-all-reports.lisp and don't have any more useful feedback.

Maybe only a hint that test-grid-reporting package has nicknames tg-reporting
and tg-rep. I see you probably have some inconveniences, as copy-pasting
print-pivot function only to change the template file, but the current cl-test-grid
API doesn't provide better approach than you did.

As you are preparing for release, let me remind you that ECL has two regressions
between commits b6519e5c and 89a8201b, see the quotation below.

Your latest reports also show these regressions (and some other).

Best regards,
- Anton

15.01.2013, 03:44, "Anton Vodonosov" <avodo...@yandex.ru>:
> Meantime I've tested 89a8201b and compared with the "base version" b6519e5c.
>
> Now we have some improvements, but also regressions and "infrastructure problems"
>
> bytecode compiler: http://common-lisp.net/project/cl-test-grid/ecl/ecl-diff-bytecode.html
>
> О©╫О©╫- genworks-gdl timeouts are not regressions.
> О©╫О©╫О©╫О©╫It's just that bytecode compiler really takes > 10 minutes to install genworks-gdl.
> О©╫О©╫О©╫О©╫Tried with the 12.12.1 release, it works the same.
> О©╫О©╫О©╫О©╫I have impression that the time is spent during unpacking of the 20 MB .tgz archive.
> О©╫О©╫О©╫О©╫The timeout didn't happened in previous testing because genworkd-gdl has already been
> О©╫О©╫О©╫О©╫downloaded by some other lisp implementation. But now bytecode ecl
> О©╫О©╫О©╫О©╫was the first lisp implementation run on this quicklisp so the download was performed
> О©╫О©╫О©╫О©╫by it.
> О©╫О©╫- opticl-doc not a regression.
> О©╫О©╫О©╫О©╫Again, we see it this time, but not in previous test runs because this time
> О©╫О©╫О©╫О©╫bytecode ecl was the first lisp running (ql:quickload :opticl-doc).
>
> О©╫О©╫О©╫О©╫opticl-doc generates .html file from .md file during asdf:load-op,
> О©╫О©╫О©╫О©╫and signals an error during this process. Nevertheless, the .html file
> О©╫О©╫О©╫О©╫is created, and subsequent asdf:load-ops just do nothing, therefore
> О©╫О©╫О©╫О©╫the error only signaled by first asdf:load-op.
>
> lisp-to-c compiler: http://common-lisp.net/project/cl-test-grid/ecl/ecl-diff-lisp-to-c.html
> О©╫О©╫- com.informatimago.common-lisp systems - real regression, confirmed manually
> О©╫О©╫- gbbopen - not a regression. This system, gbbopen, doesn't use ASDF but
> О©╫О©╫О©╫О©╫uses it's own module manage, and stores .fas files in a subdirectory along the source
> О©╫О©╫О©╫О©╫code. Therefore the asdf-output-translations installed by test-grid-agent do not
> О©╫О©╫О©╫О©╫work and the .fas files are not recompiled during every test run, but old .fas files
> О©╫О©╫О©╫О©╫are used left there from previous runs. In this case the .fas file was left here
> О©╫О©╫О©╫О©╫by ecl bytecode compiler. The .fas file is strange indeed, and you can
> О©╫О©╫О©╫О©╫reproduce the " invalid ELF header" problem during bytecode compilation by:
> О©╫О©╫О©╫О©╫О©╫О©╫О©╫rm -r quicklisp/dists/quicklisp/software/gbbopen-20121223-svn/linux86-ecl-12.12.1/
> О©╫О©╫О©╫О©╫О©╫О©╫О©╫lisps/ecl-bin/bin/ecl -eval "(ext:install-bytecodes-compiler)" О©╫-eval "(ql:quickload :gbbopen)" -eval "(quit)"
> О©╫О©╫О©╫О©╫But this is definitely not a regression, it happens on older versions too.
> О©╫О©╫- lucene-in-action-tests and montezuma - real regression, confirmed manually
>
> So, in total 2 new regressions on lisp-to-c compiler, and some problems which can
> not be qualified as regressions, which you may or may not want to investigate.
>
> As we have regressions, "base version" is not shifted forward, it remains at b6519e5c.
>
> Best regards,
> - Anton
>
> --

[Juan Jose Garcia-Ripoll]

On Thu, Apr 4, 2013 at 12:53 AM, Anton Vodonosov <avodo...@yandex.ru> wrote:

As you are preparing for release, let me remind you that ECL has two regressions
between commits b6519e5c and 89a8201b, see the quotation below.

I am working on this using my code, but when trying to rebuild older versions sometimes the test grid refuses to run the tests with an obscure log

[12:00:15] [info] <tg-agent> - Skipping the lisps already tested on quicklisp 2013-03-12: ("ecl-12.12.1-ab933fa5-linux-x64-bytecode" "ecl-12.12.1-ab933fa5-linux-x64-lisp-to-c")

that I cannot know how to work around. Most important: while it claims the tests were already run in the past, when I download the database, such tests are missing (I am now using the cl-test-grid space "ecl", btw)

[Anton Vodonosov]

18.04.2013, 18:38, "Juan Jose Garcia-Ripoll" <juanjose.g...@gmail.com>:

> On Thu, Apr 4, 2013 at 12:53 AM, Anton Vodonosov <avodo...@yandex.ru> wrote:
>> As you are preparing for release, let me remind you that ECL has two regressions
>> between commits b6519e5c and 89a8201b, see the quotation below.
>
> I am working on this using my code, but when trying to rebuild older versions sometimes the test grid refuses to run the tests with an obscure log
>
> [12:00:15] [info] <tg-agent> - Skipping the lisps already tested on quicklisp 2013-03-12: ("ecl-12.12.1-ab933fa5-linux-x64-bytecode" "ecl-12.12.1-ab933fa5-linux-x64-lisp-to-c")
>
> that I cannot know how to work around. Most important: while it claims the tests were already run in the past, when I download the database, such tests are missing (I am now using the cl-test-grid space "ecl", btw)

Edit file <agent-work-dir>/persistence.lisp.
Where <agent-work-dir> defaults to cl-test-grid/work-dir/agent/, unless you specified different value from your run-agent.lisp file.

Best regards,
- Anton

[Anton Vodonosov]

18.04.2013, 18:50, "Anton Vodonosov" <avodo...@yandex.ru>:

> 18.04.2013, 18:38, "Juan Jose Garcia-Ripoll" <juanjose.g...@gmail.com>:
>

>> О©╫On Thu, Apr 4, 2013 at 12:53 AM, Anton Vodonosov <avodo...@yandex.ru> wrote:
>>> О©╫As you are preparing for release, let me remind you that ECL has two regressions
>>> О©╫between commits b6519e5c and 89a8201b, see the quotation below.
>> О©╫I am working on this using my code, but when trying to rebuild older versions sometimes the test grid refuses to run the tests with an obscure log
>>
>> О©╫[12:00:15] [info] <tg-agent> - Skipping the lisps already tested on quicklisp 2013-03-12: ("ecl-12.12.1-ab933fa5-linux-x64-bytecode" "ecl-12.12.1-ab933fa5-linux-x64-lisp-to-c")
>>
>> О©╫that I cannot know how to work around. Most important: while it claims the tests were already run in the past, when I download the database, such tests are missing (I am now using the cl-test-grid space "ecl", btw)

>
> Edit file <agent-work-dir>/persistence.lisp.
> Where <agent-work-dir> defaults to cl-test-grid/work-dir/agent/, unless you specified different value from your run-agent.lisp file.

Some background info, why the lisp already tested are stored in presistence.lisp.
I initially designed the tg-agent:main procedure to be run daily from cron.
Agent checks if new quicklisp is available and if so runs tests of all lisps on new quicklisp.
If all lisps are already tested on the last quicklisp, agent started by cron does nothing.