Continuous Integration

[Jonathan Lawlor]

Hi,

I'd like to start using a continuous integration tool (specifically travis), which would provide a degree of quality control to the packages in gonum.  gonum/graph already uses it and has a bit of flair on its readme indicating its status, which is currently:   

It will also put warnings on github's ui for merging branches if CI is enabled for them and they fail.

There are a lot of options that we can use for travis, and there are also lots of alternate tools (like drone.io).  We are lucky because go's builds are very fast and the testing is standardized, so there isn't as much plumbing to do, but we still might want to decide on things like:

* Do we want a code coverage report?  Coveralls.io provides it for travis, and it looks like it is baked into drone.io

* Who should be notified about new builds?  Where should notifications go? (email, slack, irc, mailing list...) Do we want to know about all changes, or just failing tests?

* Which version(s) of go should the tests be built against?

* How to provide testing for BLAS & LAPACK?

Travis:

Requires a config file, .travis.yml.  It is the most popular CI.  For packages (like floats) which only depend on the standard lib, and which we only want to build against the current version of go, we can just use:

```

language: go

```

Which can be expanded like the one in gonum/graph:

```

branches:

 only:

  - master

language: go

go:

 - 1.2

script:

 - go get -d -v ./... && go build -v ./...

 - go test -v ./...

 - diff <(gofmt -d .) <("") 

after_failure: failure

after_error: failure

notifications:

  email:

    recipients:

      - jrago...@gmail.com

    on_success: change

    on_failure: always

```

Which only checks changes to the master branch, uses go 1.2 (?), runs the default tests and then makes sure that it is formatted correctly (and won't pass the build if it isn't), and notifies when builds either fail or when the status changes to success.  Most of these options are the same as the defaults.  (I couldn't find after_error in travis's docs, which isn't a great sign...)

I'm less familiar with drone.io (which isn't saying much...) but it looks like the same options are available there.  It doesn't use a config file in favor of configuration on their website. I'm sure there are other places we could look as well.

Personally I think we should use travis, try including code coverage flair, build against all versions of go that we are interested in supporting, build all branches, and use direct email to notify people that want to receive it.

If anyone has any thoughts or ideas I would like to hear them.  Also if anyone has experience with travis that would be helpful - I've only used a proprietary CI tool.

[Sebastien Binet]

Jonathan,

for my own projects, I first used travis.ci and then migrated to drone.io.

I dropped travis.ci because at the time, they were lagging in supporting new go releases.

I suspect this has improved, now.

the nice thing about drone.io is that they are a go-based shop :)

they also have support for teams:

 https://github.com/drone/drone/blob/master/cmd/droned/drone.go#L193

 http://docs.drone.io/teams.html

(but I haven't tested it and don't know whether the 0$-plan version is usable)

that said, I agree: the important thing is to have *a* continuous integration test tool, be it travis-ci or drone.io.

-s

[Jonathan Lawlor]

Yes, drone.io was a snap when I used it.  I don't know if it can handle the more complicated builds like gonum/blas and gonum/lapack would require - I'll take a look at their docs and see how configurable it is.  I'm pretty sure that with travis we can come up with something that works.  I'd prefer to have the same CI for all of the gonum projects, so if we can't figure out a way for drone.io to work with one then I'd like to use something else.

We can try these out, of course.  Just fork a gonum repo, and try to set up CI using your own github account.  I'll be trying it later.

[Sebastien Binet]

On Wed, Oct 8, 2014 at 2:38 PM, Jonathan Lawlor <jonatha...@gmail.com> wrote:

Yes, drone.io was a snap when I used it.  I don't know if it can handle the more complicated builds like gonum/blas and gonum/lapack would require - I'll take a look at their docs and see how configurable it is.  I'm pretty sure that with travis we can come up with something that works.  I'd prefer to have the same CI for all of the gonum projects, so if we can't figure out a way for drone.io to work with one then I'd like to use something else.

We can try these out, of course.  Just fork a gonum repo, and try to set up CI using your own github account.  I'll be trying it later.

drone.io can cater for cgo based packages. (I have a few)

see, for example, this one (github.com/go-hep/croot) where I need to install deb pkgs from the multi-verse and adjust various env.vars, then pass control to some Makefile.

https://drone.io/github.com/go-hep/croot/admin

the drone.io build script amounts to:

"""

uname -a

sudo add-apt-repository "deb http://archive.ubuntu.com/ubuntu $(lsb_release -sc) universe restricted multiverse"

sudo apt-get update

sudo apt-get install -y root-system || exit 0

export PATH=${ROOTSYS}/bin:${PATH}

export LD_LIBRARY_PATH=${ROOTSYS}/lib:${LD_LIBRARY_PATH}

make

make test

"""

 

hth,

-s

[Jonathan Lawlor]

Looks like the only restriction to drone we would care about is that free accounts are limited to notifying only 5 people. That might not be an issue though.

Their docs mention that they use go1.1 - I'd like to verify that, because I doubt it is true.

Also, are we intending to support multiple versions of go?

[Sebastien Binet]

On Wed, Oct 8, 2014 at 3:46 PM, Jonathan Lawlor
<jonatha...@gmail.com> wrote:
> Looks like the only restriction to drone we would care about is that free accounts are limited to notifying only 5 people. That might not be an issue though.
>
> Their docs mention that they use go1.1 - I'd like to verify that, because I doubt it is true.

they use (sadly) 1.2:
$ go version
go version go1.2 linux/amd64

(so, now they are lagging wrt travis-ci :)

>
> Also, are we intending to support multiple versions of go?

fwd compatibility is tough.
e.g. the recent use of sync.Pool breaks fwd compat with anything older than 1.3.

do we have the manpower to support more than go-1.N where N=current ?
where N=current,current-1 ?

I guess it really depends on how many users we have, "lagging" behind
or "stucked" with some go-1.X version.
I suspect our user base is relatively modest, fractionning it wouldn't
be a good move. (e.g. the python-2/python-3 split)

-s

[Jonathan Lawlor]

There is also a distinction between supporting multiple versions of go and being able to test on multiple versions of go.  We could use travis's build matrix to check:  http://docs.travis-ci.com/user/languages/go/

Which would also be nice because it looks like it could give us a head's up about go tip breaking gonum.

[Jonathan Lawlor]

So I put together a fork of gonum/floats using travis at jonlawlor/blas and I've got to say that the way it ties into github is pretty neat.  I have an example pull request that fails the formatting requirements at https://github.com/jonlawlor/floats/pull/1 so you can see what it looks like.

And then feeling ambitious I tried it with gonum/blas:  https://travis-ci.org/jonlawlor/blas

I can't seem to get the blas environment set up correctly, in part because I run osx, and the travis environment is not easy to interact with (I've had to make changes, trigger a build, wait, repeat) so maybe you'll have some suggestions Sebastien.  I'll come back for round 2 tomorrow.

Incidentally, I did get it to work with ATLAS, which indicated that several of the blas tests failed with very different outputs.  You can see the errors at https://travis-ci.org/jonlawlor/blas/builds/37464809#L132 - note that some appear to be rounding issues, but others are not.  Take that with a grain of salt though, because I'm not sure it is configured correctly.

[Dan Kortschak]

On Wed, 2014-10-08 at 22:41 -0700, Jonathan Lawlor wrote:
> Incidentally, I *did* get it to work with ATLAS, which indicated that

> several of the blas tests failed with very different outputs. You can
> see the errors at
> https://travis-ci.org/jonlawlor/blas/builds/37464809#L132 -
> note that some appear to be rounding issues, but others are not. Take
> that with a grain of salt though, because I'm not sure it is
> configured correctly.
>

That's a known issue. It should be written down somewhere, but I can't
find it. Brendan has put some effort into that - it is upstream.

[Jonathan Lawlor]

So I forked the repos for both floats and blas to get travis going, and see what kind of trouble we might run into.

https://github.com/jonlawlor/blas has it enabled, and if you go to https://travis-ci.org/jonlawlor/blas then you'll see the current status.  I've implemented, but not enabled, a set of builds that use ATLAS pending a fix to the bug mentioned previously, and it looks like travis is working on an OSX environment, which we can run tests against once it comes out of beta.

Take a look to see if there is anything you like / don't like about the setup.  I haven't turned on code coverage flair, but that is pretty simple.  Also, my bash scripting is terrible and the .travis.yml file is probably too wordy, but it gets the job done for now.

If there are no complaints I would like to start enabling travis-ci on gonum's repos, where appropriate.  To that end I'd like it if people who were interested in receiving the build status updates via email would chime in.

Thanks

[Jonathan Lawlor]

https://travis-ci.org/gonum/floats is up and running.  Do we want to build downstream (internal to gonum) packages and test them as well?  For example, gonum/stat uses gonum/floats.  During builds of gonum/floats, we could check gonum/stat as well to ensure we don't break anything.  However, this should really be handled by tests that live in gonum/floats.

Also, I suggest that we include coverage flair from coveralls.io, because the gonum packages generally have extremely good coverage (>95%) and that is something we should be proud of.

[Jonathan Lawlor]

There's now code coverage flair for gonum/floats, which lists coverage at 100% (it is actually 99.xx or something, but rounds up).  I'm going to start moving ahead with other libraries.

Right now I plan on going through in some kind of rough hierarchy of dependency, so the next ones that I'll try to add CI and coverage to will be: diff, blas, root, stat, unit, lapack, and matrix, in that order.  Due to some unresolved issues with ATLAS, we'll only test blas libs (and packages that import them) against OpenBLAS for now.

If you want to be notified when there are builds of any of these packages it is easy to be added to the email list.

Any concerns?

On Tuesday, October 7, 2014 8:19:42 PM UTC-4, Jonathan Lawlor wrote:

[Jonathan Lawlor]

I've added travis and coveralls to all of the packages listed, except for root.  There are a few packages that I didn't add testing to, because they were in an immature state.

[Jonathan Lawlor]

A small update.  Go 1.4 beta 1 is out, soI manually reran all of the travis-ci builds, which all passed on go tip.  Not quite the same thing, but probably close enough.

[Jonathan Lawlor]

I've reran all of the travis-ci builds with the new 1.4 release and everything passes.  I've also updated the docker images for lapack and blas.

[Brendan Tracey]

It was asked on one of the PRs, but Optimize is in a state where it can use travis.

[Jonathan Lawlor]

I've added travis-ci and coveralls to optimize.

[Jonathan Lawlor]

Hi,

It has become clear that my priorities have shifted since I joined this project, and I should probably no longer be listed as an owner in github.  A combination of dad-brain and the end to a lull in my professional life means that I haven't made a contribution in months, and I'm barely monitoring the CI updates.  I think someone else should have that responsibility (and be listed in the email notifications in the various .travis.yml files).

The biggest maintenance work is adding the various supported versions of go into the build matrix at the top of the .travis.yml files.  Currently:

go:

 - 1.3.3

 - 1.4.2

 - tip

which is pretty self explanatory.

The biggest outstanding issues (other than some cleanup) is building on other architectures than x86_64, testing in mac environments, and (for gonum/blas) testing on libs other than OpenBLAS.  The other architectures can probably be tackled by someone with sufficient familiarity with qemu, although build times may suffer - if that is the case, you may want to look into a CI service that uses docker images directly, such as shippable.  Travis has been testing mac build environments for some time and may eventually make the second point possible.  For the final point, some issues need to be resolved with respect to inconsistent implementation of various BLAS functionality in different BLAS libraries.

You may also want some method of automated benchmarking.  I don't know how consistent the hardware and environment is on travis-ci.

The CI infrastructure is not terribly complicated, and not terribly consistent.  You do need to be a repo owner to modify things at travis-ci and coveralls.io, which is the only reason I was one.  I think this infrastructure work was probably the biggest contribution I made, which hopefully let some of you clever folks focus more on writing gonum.  I may still make some contributions in the future.

I'll still be around to help out anyone who would like to take over these responsibilities, but I really don't have much time or attention to give to it.

On an unrelated note, for some of you in academia - if you have some bright students looking to sell out for either employment or internship in commodities trading in CT, near NYC, then feel free to send them my way.  I'm looking for people for front office power market modelling and dev roles - basically a junior quant.  Power markets lend themselves well to statistical analysis, if you can handle the firehose of data, and it can be an interesting, lucrative career.  They'd have to program in matlab though.

[Dan Kortschak]

Thanks for all the work you have put in, it has been invaluable.

I understand entirely the impact of small people on cognitive capacity. Best wishes with all that. For the same reason, I won't step forward.

Dan

[Brendan Tracey]

Thanks for the time that you've invested. Best of luck!

[Dan Kortschak]

As an aside and answering this to a degree. I tested mat64 on arm a
couple of days ago and it passes.

I also ran benches and they are not surprisingly very slow. I'll test
all of gonum soon and post the results.

[Jonathan Lawlor]

I'd like to do a cleanup of CI code in time for the go 1.5 release.  Right now I'm thinking:

• Move all travis-ci related code (except for .travis.yml) into a subfolder .travis
• Remove secure environment variables from .travis.yml in favor of entering them directly into travis (a feature they added recently).
• Remove testing of go tip, just test releases.
• Remove myself from notifications.
• Some code cleanup and commenting.

If you have any other suggestions, such as removing coverage, or keeping the go tip test, or adding notification to an IRC channel, I'm all ears.

[Dan Kortschak]

That all sounds good to me, Particularly removing tip, which was broken for us since yesterday (issue filed).

Welcome back BTW.

[Jonathan Lawlor]

This is complete, I think.  There was some speedup in gonum/matrix's testing, because we don't have to apt-get update, and don't need sudo.  I fiddled around a bit in lapack trying to get it to work with ATLAS, but I was not successful.  Someone more knowledgeable might have more luck.

Hopefully, when go 1.5 comes around, we'll just have to add it to the build matrix and be done with it.

On Thursday, June 4, 2015 at 3:45:04 PM UTC-4, kortschak wrote:

That all sounds good to me, Particularly removing tip, which was broken for us since yesterday (issue filed).

Welcome back BTW.

Thanks! 

[Brendan Tracey]

Thank you very much for that maintenance

--
You received this message because you are subscribed to the Google Groups "gonum-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gonum-dev+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jonathan Lawlor]

Would you like me to add go1.5beta1 to the travis build matrices?  It seems to work.

[Dan Kortschak]

Are you sure it works? I have an issue in golang/go about failure to build on travis with go1.5 due to incorrect internal package exclusion. The issue is still open as of this morning.

On 09/07/2015, at 10:47 AM, "Jonathan Lawlor" <jonatha...@gmail.com> wrote:

Would you like me to add go1.5beta1 to the travis build matrices?  It seems to work.

--

[Jonathan Lawlor]

I only tried it with gonum/matrix (actually a slightly outdated version, but I don't think that would change much...)

Completed successfully here.

[Dan Kortschak]

On Wed, 2015-07-08 at 19:23 -0700, Jonathan Lawlor wrote:
> I only tried it with gonum/matrix (actually a slightly outdated version,
> but I don't think that would change much...)
>
> Completed successfully here

> <https://travis-ci.org/jonlawlor/matrix/jobs/70157561>.

That's interesting. OK SGTM - the worst that can happen is a fail on the
PR. I certainly would like to be testing against go1.5.

thanks

[Jonathan Lawlor]

Alright, I'll do so by making a commit directly to the .travis.yml file in the master branch for each of the repos, unless you have an objection.  I only have to add the line - go1.5beta1 to it.

[Dan Kortschak]

On Wed, 2015-07-08 at 19:42 -0700, Jonathan Lawlor wrote:
> Alright, I'll do so by making a commit directly to the .travis.yml
> file in the master branch for each of the repos, unless you have an
> objection. I only have to add the line - go1.5beta1 to it.

SGTM

[Brendan Tracey]

SGTM

[Jonathan Lawlor]

All builds succeeded.  The 1.5beta1s took about 20 seconds longer, which I suspect is due to having to download the new go release, as (I think...) travis has the other releases cached.  Could also be the increased compilation times, but I wouldn't expect the increase to be so constant.

[Dan Kortschak]

Related, when do we want to drop 1.3? This should not happen before
go1.5 is out, but do we want to do it then?

This drop would allow two sets of clean up:

1. use of #include "textflag.h" in the asm code
2. "for _ = range" => "for range" in one place

These are changes that break 1.3 builds, but we don't *need* to do them.

[Jonathan Lawlor]

Actually I spoke too soon.  Somehow gonum/unit is failing in 1.5beta1 with:

--- FAIL: TestFormat (0.00s)
	unit_test.go:43: Format "%f": got: "%!(BADPREC)9.810000 kg s^-2" expected: "9.81 kg s^-2"
	unit_test.go:43: Format "%f": got: "%!(BADPREC)9.810000 kg s^-2" expected: "9.81 kg s^-2"
	unit_test.go:43: Format "%e": got: "%!(BADPREC)6.626070e-34 kg^2 s^-1" expected: "6.62606957e-34 kg^2 s^-1"
	unit_test.go:43: Format "%v": got: "%!(BADPREC)6.62606957e-34 kg^2 s^-1" expected: "6.62606957e-34 kg^2 s^-1"
	unit_test.go:43: Format "%s": got: "%!s(*Unit=%!(BADPREC)6.62606957e-34 kg^2 s^-1)" expected: "%!s(*Unit=6.62606957e-34 kg^2 s^-1)"
	unit_test.go:43: Format "%v": got: "%!(BADPREC)2.718281828459045" expected: "2.718281828459045"
	unit_test.go:43: Format "%v": got: "%!(BADPREC)1 kg" expected: "1 kg"
	unit_test.go:43: Format "%v": got: "%!(BADPREC)1.61619926e-35 m" expected: "1.61619926e-35 m"
	unit_test.go:43: Format "%v": got: "%!(BADPREC)15.2 s" expected: "15.2 s"

[Dan Kortschak]

That's our problem. I send a fix in a sec.

On Wed, 2015-07-08 at 20:23 -0700, Jonathan Lawlor wrote:
> Actually I spoke too soon. Somehow gonum/unit is failing in 1.5beta1

> <https://travis-ci.org/gonum/unit/jobs/70164130> with: