continuous integration, again

[Qian Yun]

We have a discussion about Travis CI 3 years ago, now we have
moved to GitHub, I believe now it's time to have CI?

One point Waldek raised was "run test before commit".

If we have CI support, and you commit in your own master repo
before merge into main repo, you can "run test before commit".

For pull request, with CI we can run tests before commit.

Even if we commit directly into repo, with CI we can find
problems more quickly.

(Anyway, I assume we don't run a full build and full test
before commit, most of time, right? :-)

So, if we all agree to add CI support, I'll post a simple
CI script for discussion, and we can add functionality
gradually over time.

- Best,
- Qian

[Ralf Hemmecke]

Qian,

super! I actually wanted to ask you before about some instructions that
I (and others) could do in their github fork in order to make a full
build with every commit.

I don't understand what you mean by

> we don't run a full build and full test before commit, most of time

Before I push to the official fricas repo, I have of course committed to
my local repository and run the compilation, the tests, and now even
build the documentation. You certainly know well enough that a local
commit can easily be undone, so it's not a question of commit or not,
but whether it's pushed to the official gh repo.

I think it would be a good idea if every one who wants to have something
integrated into fricas must setup CI on his/her github fork and show the
logs of the build and tests so that it is clear that everything is OK
before it is merged into the official repo.

But perhaps you take the lead since you are the most experienced person
with that stuff.

Ralf

[Waldek Hebisch]

On Sat, Nov 21, 2020 at 10:56:17PM +0800, Qian Yun wrote:
> We have a discussion about Travis CI 3 years ago, now we have
> moved to GitHub, I believe now it's time to have CI?
>
> One point Waldek raised was "run test before commit".
>
> If we have CI support, and you commit in your own master repo
> before merge into main repo, you can "run test before commit".
>
> For pull request, with CI we can run tests before commit.
>
> Even if we commit directly into repo, with CI we can find
> problems more quickly.
>
> (Anyway, I assume we don't run a full build and full test
> before commit, most of time, right? :-)

I do run full build and full testsuite before almost every
commit. I sometime make exception when change is obvious
and and does not change code and I am goind to do another
commit with full build and full testsuite run just after
current commit. I expect other people to do the same
(those rules appeared here several times and should be
well known). This may look extreme but:
- project that use more relaxed rules freqently have
non-buildable repository and this would be bad
- IME before commit test from time to time catches problems
(fails). Catching/correcting them later would cause
more trouble.

> So, if we all agree to add CI support, I'll post a simple
> CI script for discussion, and we can add functionality
> gradually over time.

Well, more testing is better than less. Just I do not
consider CI as game changer.

--
Waldek Hebisch

[Qian Yun]

Bellow is a simple travis CI configuration file.

It does the most basic things, compile the source and run the tests.

I'd like to ask your opinions about a few questions:

1. Shall we use a newer or older version of Ubuntu as
testing environment?

2. Shall we test other lisp implementations?

I welcome other comments as well.

(To Ralf, after this file committed to repo, all you need to do
is to login travis-ci.org with GitHub credential, maybe tweak
one or two settings.)

- Best,
- Qian


diff --git a/.travis.yml b/.travis.yml
new file mode 100644
index 00000000..dff0dffa
--- /dev/null
+++ b/.travis.yml
@@ -0,0 +1,17 @@
+os: linux
+dist: bionic
+language: c
+addons:
+ apt:
+ packages:
+ - sbcl
+compiler: gcc
+
+script:
+ - |
+ ./configure --with-x=no && make -j2 2>&1 | tee build.log | \
+ sed -n 's/.*\(")compile".*spad\).*/\1/p'
+ - |
+ cd src/input && make -j2 | \
+ tee ../../tests.log | sed -n 's/.*tee \(.*output\);.*/\1/p'
+ - cd $TRAVIS_BUILD_DIR/src/input && ../scripts/test.sh
diff --git a/src/scripts/test.sh b/src/scripts/test.sh
new file mode 100755
index 00000000..d9e8bf28
--- /dev/null
+++ b/src/scripts/test.sh
@@ -0,0 +1,7 @@
+#! /bin/sh
+
+OUTPUTS=`ls *.output | grep -v -e elemnum -e unittest`
+
+grep ' unexpected \(failures\|passes\):' ${OUTPUTS} | \
+ grep -v '\(failures\|passes\): 0'
+test $? -eq 1

[Dima Pasechnik]

Travis CI has drastically cut on allowing tests for free:
https://blog.travis-ci.com/2020-11-02-travis-ci-new-billing
TL; DR: one gets about 1000 free CPU minutes per month per project.

GitHub actions are much more powerful nowadays, give much more free time, etc.

> --
> You received this message because you are subscribed to the Google Groups "FriCAS - computer algebra system" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to fricas-devel...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/fricas-devel/a4f2591d-508f-fd6f-d7a3-b6f1ebd768a4%40gmail.com.

[Dima Pasechnik]

On Sat, Nov 21, 2020 at 3:32 PM Waldek Hebisch <heb...@math.uni.wroc.pl> wrote:
>
> On Sat, Nov 21, 2020 at 10:56:17PM +0800, Qian Yun wrote:
> > We have a discussion about Travis CI 3 years ago, now we have
> > moved to GitHub, I believe now it's time to have CI?
> >
> > One point Waldek raised was "run test before commit".
> >
> > If we have CI support, and you commit in your own master repo
> > before merge into main repo, you can "run test before commit".
> >
> > For pull request, with CI we can run tests before commit.
> >
> > Even if we commit directly into repo, with CI we can find
> > problems more quickly.
> >
> > (Anyway, I assume we don't run a full build and full test
> > before commit, most of time, right? :-)
>
> I do run full build and full testsuite before almost every
> commit.

Do you say that you expect every contribution to be wrapped up in
exactly one commit?
(This is not very usual, although there are workflows where this is a
requirement, e.g.
https://en.wikipedia.org/wiki/Gerrit_(software) - used at Google, AFAIK is built
around such a model)

> I sometime make exception when change is obvious
> and and does not change code and I am goind to do another
> commit with full build and full testsuite run just after
> current commit. I expect other people to do the same
> (those rules appeared here several times and should be
> well known). This may look extreme but:
> - project that use more relaxed rules freqently have
> non-buildable repository and this would be bad
> - IME before commit test from time to time catches problems
> (fails). Catching/correcting them later would cause
> more trouble.

In danger to state the obvious (to me):
the idea of CI is that *you* only look at a contribution after it
passes pre-defined
automatic tests, and these are as extensive (and possibly much more
extensive, as it can be
done on many different platforms (i.e. OS+LISP) at the same time) as
*you* would do by hand.
It is meant to save *you* time.
And avoid SNAFUs such as "oops, it's already merged, but it does not
work on FooOS with BarCL".

>
> > So, if we all agree to add CI support, I'll post a simple
> > CI script for discussion, and we can add functionality
> > gradually over time.
>
> Well, more testing is better than less. Just I do not
> consider CI as game changer.
>
> --
> Waldek Hebisch
>

> --
> You received this message because you are subscribed to the Google Groups "FriCAS - computer algebra system" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to fricas-devel...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/fricas-devel/20201121153211.GA1000%40math.uni.wroc.pl.

[Ralf Hemmecke]

> GitHub actions are much more powerful nowadays, give much more free time, etc.

Do you have some hints in how to setup this stuff. I once tried to
understand what it does and gave up.

I read about compiling and testing inside a virtual machine that one can
setup. That sounded interesting since I would also like to test whether
libfricas.al would compile and so needed aldor installed.

Ralf

[Waldek Hebisch]

On Sun, Nov 22, 2020 at 09:15:36AM +0000, Dima Pasechnik wrote:
> On Sat, Nov 21, 2020 at 3:32 PM Waldek Hebisch <heb...@math.uni.wroc.pl> wrote:
> >
> > On Sat, Nov 21, 2020 at 10:56:17PM +0800, Qian Yun wrote:
> > > We have a discussion about Travis CI 3 years ago, now we have
> > > moved to GitHub, I believe now it's time to have CI?
> > >
> > > One point Waldek raised was "run test before commit".
> > >
> > > If we have CI support, and you commit in your own master repo
> > > before merge into main repo, you can "run test before commit".
> > >
> > > For pull request, with CI we can run tests before commit.
> > >
> > > Even if we commit directly into repo, with CI we can find
> > > problems more quickly.
> > >
> > > (Anyway, I assume we don't run a full build and full test
> > > before commit, most of time, right? :-)
> >
> > I do run full build and full testsuite before almost every
> > commit.
>
> Do you say that you expect every contribution to be wrapped up in
> exactly one commit?
> (This is not very usual, although there are workflows where this is a
> requirement, e.g.
> https://en.wikipedia.org/wiki/Gerrit_(software) - used at Google, AFAIK is built
> around such a model)
>

No. I expect commits to be reasonable self-contained and
logically connected units of work. If somebody wants
to contribute two unrelated domains, then it is natural
to do this as two commits. OTOH cluster of cooperating
domains and categories + changes needed to connect them
to rest of algebra naturaly fit in one commit. Bug
fixes naturally go as one fix per commit. And when
somebody fixes unrelated bug developing something else
(say output problem when working on computational
problem) then bug fix naturally comes as separate commit.

> > I sometime make exception when change is obvious
> > and and does not change code and I am goind to do another
> > commit with full build and full testsuite run just after
> > current commit. I expect other people to do the same
> > (those rules appeared here several times and should be
> > well known). This may look extreme but:
> > - project that use more relaxed rules freqently have
> > non-buildable repository and this would be bad
> > - IME before commit test from time to time catches problems
> > (fails). Catching/correcting them later would cause
> > more trouble.
>
> In danger to state the obvious (to me):
> the idea of CI is that *you* only look at a contribution after it
> passes pre-defined
> automatic tests, and these are as extensive (and possibly much more
> extensive, as it can be
> done on many different platforms (i.e. OS+LISP) at the same time) as
> *you* would do by hand.
> It is meant to save *you* time.
> And avoid SNAFUs such as "oops, it's already merged, but it does not
> work on FooOS with BarCL".

You previously wrote that CI can cover several platforms, that
is true. And for some folks CI may be more convenient than
local testing. I am just saying that for me build machine
takes something like 2 min 30s real time for full build
(parallel) and testing about 3 min real time. When platform
dependent code is involved, then I run more tests, but that
is rare (vast majority of FriCAS code is platform independent).

[Ralf Hemmecke]

> You previously wrote that CI can cover several platforms, that
> is true. And for some folks CI may be more convenient than
> local testing. I am just saying that for me build machine
> takes something like 2 min 30s real time for full build
> (parallel) and testing about 3 min real time. When platform
> dependent code is involved, then I run more tests, but that
> is rare (vast majority of FriCAS code is platform independent).

Waldek, that you can test quickly on many platforms does not mean that
everyone can.

I would really like to put FriCAS into an environment where more people
can easily test on several platforms. If public available systems like
travis-ci or whatever github offers can help with that, it would be
advantageous for FriCAS. It does not mean that you have to change your
workflow.


Ralf

[Dima Pasechnik]

On Sun, Nov 22, 2020 at 12:01 PM Waldek Hebisch

This gets very messy as soon as contributions are no longer individual, as
merging joint work in a single commit is tricky, error-prone, and removes
authorship from all but one contributor.

That's why feature branches and PRs (these are the same on the level
of git, GitHub's PRs are just feature branches)
are a much more common kind of a workflow.

While it's probably not often that one gets code in Spad (i.e. as
platform-independent
as it can be) that runs on one platform and fails on another, it looks
not at all impossible.

>
> > >
> > > > So, if we all agree to add CI support, I'll post a simple
> > > > CI script for discussion, and we can add functionality
> > > > gradually over time.
> > >
> > > Well, more testing is better than less. Just I do not
> > > consider CI as game changer.
>
> --
> Waldek Hebisch
>

> --
> You received this message because you are subscribed to the Google Groups "FriCAS - computer algebra system" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to fricas-devel...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/fricas-devel/20201122120132.GA22235%40math.uni.wroc.pl.

[Waldek Hebisch]

Authorship does not depend on version control. If you mean that
automatic commit information is no longer correct, we have
ChangeLog which allows specifying authorship without artifical
limits of version control. I would say that in case something
is a single unit of work splitting it makes things messy.

>
> That's why feature branches and PRs (these are the same on the level
> of git, GitHub's PRs are just feature branches)
> are a much more common kind of a workflow.

Well, some work fits naturaly into branches, but AFAICS most
cases relevant to FriCAS just commiting when ready is the
best way.

Impossible -- probably not. But past practice is that after
initial port Spad code just works. There were bunch of
cases with bugs in Lisp compilers. When we learn about bug
of course we report it. But actively searching for bugs
in Lisp compilers is not main point of FriCAS.

--
Waldek Hebisch

[Dima Pasechnik]

Times of ChangeLogs are gone.
Nowadays people would rather look at
https://github.com/fricas/fricas/graphs/contributors
and not at the ChangeLog.

> I would say that in case something
> is a single unit of work splitting it makes things messy.

splitting is messy, but working on something as a group is quite normal.
And then exchange of updates is done via commits, so the end result is a
branch with many commits from different people. Yes, it can be squashed,
but often it's not desirable.

> --
> You received this message because you are subscribed to the Google Groups "FriCAS - computer algebra system" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to fricas-devel...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/fricas-devel/20201122200653.GA29525%40math.uni.wroc.pl.

[Qian Yun]

Thanks! I didn't know that.

I'm playing with GitHub actions. Will report back soon...

[Qian Yun]

On 11/22/20 7:39 PM, Ralf Hemmecke wrote:
>> GitHub actions are much more powerful nowadays, give much more free time, etc.

Yes, GitHub actions provides more functionality, for public repos the
free time is almost unlimited, and a bit (around 30%) faster than travis CI.

> Do you have some hints in how to setup this stuff. I once tried to
> understand what it does and gave up.

Hi Ralf, it's pretty straight forward. GitHub actions are by default
enabled for everyone's every repo.

For FriCAS, I added this
https://github.com/oldk1331/fricas/blob/master/.github/workflows/ci.yml
and results are https://github.com/oldk1331/fricas/runs/1440786646
(Build and run tests with "make -j2" takes 7.5 minutes.)

====
name: FriCAS CI

on:
push:
branches: [ master ]
pull_request:
branches: [ master ]

jobs:
build:

runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v2
- name: install lisp
run: sudo apt install sbcl
- name: configure
run: ./configure
- name: make
run: make -j2
- name: make check
run: make check -j2
====

> I read about compiling and testing inside a virtual machine that one can
> setup. That sounded interesting since I would also like to test whether
> libfricas.al would compile and so needed aldor installed.
>
> Ralf
>

It is certainly possible to do that. But IIRC building aldor interface
takes a very long time?

- Best,
- Qian

[Dima Pasechnik]

You can actually use something like `make -j8`, or even `-j12` rather
than `make -j2` - I think their runners are quite beefy.
That's what we do in our (quite elaborate by now) Actions test framework:
https://github.com/sagemath/sage/tree/develop/.github/workflows
(we test on macOS and on Windows (Cygwin, and WSL) too, by the way).

> --
> You received this message because you are subscribed to the Google Groups "FriCAS - computer algebra system" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to fricas-devel...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/fricas-devel/3c3ef66c-1a3d-37cf-bcab-bfefecef973f%40gmail.com.

[Qian Yun]

Hi Dima,

From
https://docs.github.com/en/free-pro-team@latest/actions/reference/specifications-for-github-hosted-runners
it says each VM has 2 CPU cores. And my test shows that "make -j8"
doesn't make it faster.

And thanks for your sage example, I do plan to add additional platform
support later.

- Qian

[Dima Pasechnik]

On Tue, Nov 24, 2020 at 2:33 AM Qian Yun <oldk...@gmail.com> wrote:
>
> Hi Dima,
>
> From
> https://docs.github.com/en/free-pro-team@latest/actions/reference/specifications-for-github-hosted-runners
> it says each VM has 2 CPU cores. And my test shows that "make -j8"
> doesn't make it faster.

OK, thanks for pointing it out. GH Actions is a moving target, what
was correct a couple of months ago might be
different now.

>
> And thanks for your sage example, I do plan to add additional platform
> support later.
>
> - Qian
>
> On 11/23/20 6:50 PM, Dima Pasechnik wrote:
> > You can actually use something like `make -j8`, or even `-j12` rather
> > than `make -j2` - I think their runners are quite beefy.
> > That's what we do in our (quite elaborate by now) Actions test framework:
> > https://github.com/sagemath/sage/tree/develop/.github/workflows
> > (we test on macOS and on Windows (Cygwin, and WSL) too, by the way).
>

> --
> You received this message because you are subscribed to the Google Groups "FriCAS - computer algebra system" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to fricas-devel...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/fricas-devel/5b56fd5b-6bbf-8575-e2e1-94f831e52133%40gmail.com.

[Waldek Hebisch]

On Tue, Nov 24, 2020 at 10:33:18AM +0800, Qian Yun wrote:
> Hi Dima,
>
> From https://docs.github.com/en/free-pro-team@latest/actions/reference/specifications-for-github-hosted-runners
> it says each VM has 2 CPU cores. And my test shows that "make -j8"
> doesn't make it faster.

On real hardware IME it pays to have number of jobs larger than
number of cores. On uniprocessors I used '-j3', on dual
core '-j5', on quad core '-j7', on machine with 20 cores
I use '-j15' (with such number of jobs serial parts
dominate, so gains are limied and I prefer to limit
top load of machine). It is not clear how virtual machine
affects runtime.

--
Waldek Hebisch