Continuous integration

[Dag Sverre Seljebotn]

That Other Thread contained some references to CI. So I'm mainly
wondering what the current NumFOCUS plans for supporting CI efforts are,
if any? Was there a mention on money being available for somebody to
work on that?

I think of Cython as one vertex in a CI graph:

a) Upstream, Cython depends on various versions of NumPy and Python as
part of its CI

b) Downstream, we rely on building and testing Sage as an extended
regression test suite. It's not just that our test suite doesn't have
100% coverage, but also sometimes that we intentionally break backwards
compatibility, and fixing up Sage gives a nice indication of the
consequences of a change. (I imagine NumPy is in a very similar
situations with lots of libraries depending on it and potential for very
subtle breakage of backwards compatibility.)

Ideally, we'd have LOTS of libraries using Cython in our CI -- mpi4py,
Pandas, scikits-learn... BUT, we're running into problems as it is with
the CI server hardware keeping up (if there was infinite CI there'd
probably be a lot more tests set up).

Since most scientific-python libraries have both dependencies and
dependees, it seems like there should be some benefit to having the same
CI system test all of them. That could conserve both hardware use and
administration overhead. And, which I believe is very important, make it
easier for small projects like scikits-sparse to participate in
automatic CI by simply participating in an environment maintained by the
larger projects like Cython and NumPy.

I think there's two ways of spinning this:

1) Build "Sciome" in a CI and approach it by "release managing" Sciome

 2) Focus on providing "infrastructure for library developers", where
there's a relatively big CI graph where each project has a node. I.e.
something like "ShiningPanda for scientific Python", but with a critical
difference being that each project can use the build artifacts of
others; Cython would flip a switch and have all projects depending on
Cython being rebuilt to try the new Cython version. This seems
complicated, but certainly Jenkins for one seems to support such setups
already so its mainly about hardware and administration...

I know I get a lot more excited by 2) than 1), even if it's perhaps
mainly the spin put on it rather than a technical difference. I guess I
worry that 1) would tend to duplicate the CI administration work already
done by Cython in its own CI, but Cython would then still have to do CI
on Sage servers.

Dag

[Anthony Scopatz]

On Sat, May 26, 2012 at 1:05 PM, Dag Sverre Seljebotn <d.s.se...@astro.uio.no> wrote:

[snip]

 

I think there's two ways of spinning this:

 1) Build "Sciome" in a CI and approach it by "release managing" Sciome

 2) Focus on providing "infrastructure for library developers", where there's a relatively big CI graph where each project has a node. I.e. something like "ShiningPanda for scientific Python", but with a critical difference being that each project can use the build artifacts of others; Cython would flip a switch and have all projects depending on Cython being rebuilt to try the new Cython version. This seems complicated, but certainly Jenkins for one seems to support such setups already so its mainly about hardware and administration...

I know I get a lot more excited by 2) than 1), even if it's perhaps mainly the spin put on it rather than a technical difference. I guess I worry that 1) would tend to duplicate the CI administration work already done by Cython in its own CI, but Cython would then still have to do CI on Sage servers.

I am also in favor of (2).  It seems to be a better representative of real world build environments than (1).  That is, (1) would only be fully representative if 'sciome' had perfect market penetration, which is unrealistic.  Furthermore, forcing CI, documentation, etc standards to be dependent on the meta-package seems a bit off since they can be orthogonal efforts. 

Additionally, I think if a project (like Cython or something domain specific like yt or PyNE) wants to adopt the CI facilities provided, there is some amount of work that should be expected.  Cython might have more to do here than others but I think we all have a similar set of concerns....

Be Well

Anthony

 

Dag

[Andy Ray Terrel]

The only place I saw support being offered thus far was for a build
engineer. It seems to me that one really needs to nail down what
"Sciome" is doing before we are able to launch a CI service for it.

I think just getting something like ShiningPanda to host a set of
testing structures so that devs can have a peek into how a library is
interacting with others is important. Figuring out how to spell the
version number you are testing against is also important which may be
why this thread is empty and "That Other Thread" is so full.

One issue I have found with such CI efforts is getting something
getting a nice balance between useful "you've broken the build"
messages when there is a genuine breakage that will postpone release,
and the usual noise of a dev team checking in broken code. My
recommendation would be for the "Sciome" solution to be only testing a
few branches from each project (something like a stable and a dev),
but to limit the number of tests for each one (as in not every
checkin). Then each project can have its own CI solution that checks
at a finer grain.

It would be good if it could also be engineered in a way that a
project could grab the test suite (AMI, disk image, or whatever) and
have their finer grained testing run in it. This way supporting Dag's
requirement of doing both upstream and downstream testing.

-- Andy

[Chris Kees]

On Monday, May 28, 2012 8:01:05 AM UTC-5, Andy R. Terrel wrote:

On Sun, May 27, 2012 at 2:04 PM, Anthony Scopatz <> wrote:
> On Sat, May 26, 2012 at 1:05 PM, Dag Sverre Seljebotn

> <> wrote:
>
> [snip]
>
>>
>> I think there's two ways of spinning this:
>>
>>  1) Build "Sciome" in a CI and approach it by "release managing" Sciome
>>
>>  2) Focus on providing "infrastructure for library developers", where
>> there's a relatively big CI graph where each project has a node. I.e.
>> something like "ShiningPanda for scientific Python", but with a critical
>> difference being that each project can use the build artifacts of others;
>> Cython would flip a switch and have all projects depending on Cython being
>> rebuilt to try the new Cython version. This seems complicated, but certainly
>> Jenkins for one seems to support such setups already so its mainly about
>> hardware and administration...
>>
>> I know I get a lot more excited by 2) than 1), even if it's perhaps mainly
>> the spin put on it rather than a technical difference. I guess I worry that
>> 1) would tend to duplicate the CI administration work already done by Cython
>> in its own CI, but Cython would then still have to do CI on Sage servers.
>
>
> I am also in favor of (2).  It seems to be a better representative of real
> world build environments than (1).  That is, (1) would only be
> fully representative if 'sciome' had perfect market penetration, which is
> unrealistic.  Furthermore, forcing CI, documentation, etc standards to be
> dependent on the meta-package seems a bit off since they can be
> orthogonal efforts.
>

I like the idea of infrastructure for a CI graph that allows a project to examine its upstream and downstream nodes.  I wonder if a something built on top of the existing packages' CI systems would work instead of trying to build something like shining panda for scientific python. For example, we're already invested in running buildbots on quite  few HPC systems, and they test much of the proposed core (both the core packages' own unit tests and our tests).  A hosted solution running some nice flavor of linux is not going to catch many of the problems we encounter on these systems, but we're also not going to be able to make buildbots on these platforms respond to the actions of "outside" developers (as much as I'd like to). Maybe what we need is a way to make the unit test output passively available to a central CI graph.

Allowing packages to use the build artifacts of upstream packages would also be great though. That seems like an issue internal to each  CI system and also dependent on the distribution tools. It also starts to cross the line between CI and automated packaging.

> Additionally, I think if a project (like Cython or something domain specific
> like yt or PyNE) wants to adopt the CI facilities provided, there is some
> amount of work that should be expected.  Cython might have more to do here
> than others but I think we all have a similar set of concerns....
>
> Be Well
> Anthony
>
>>
>>
>>
>> Dag
>
>

The only place I saw support being offered thus far was for a build
engineer.  It seems to me that one really needs to nail down what
"Sciome" is doing before we are able to launch a CI service for it.

I agree.

 

I think just getting something like ShiningPanda to host a set of
testing structures so that devs can have a peek into how a library is
interacting with others is important.  Figuring out how to spell the
version number you are testing against is also important which may be
why this thread is empty and "That Other Thread" is so full.

One issue I have found with such CI efforts is getting something
getting a nice balance between useful "you've broken the build"
messages when there is a genuine breakage that will postpone release,
and the usual noise of a dev team checking in broken code.  My
recommendation would be for the "Sciome" solution to be only testing a
few branches from each project (something like a stable and a dev),
but to limit the number of tests for each one (as in not every
checkin).  Then each project can have its own CI solution that checks
at a finer grain.

I agree, but I can't resist poking the "bid data" folks on this list: I'm expecting great new ways of visualizing/processing complex information that would allow me to comprehend more than just a graph of projects with each node having a stable and dev attribute.

 

It would be good if it could also be engineered in a way that a
project could grab the test suite (AMI, disk image, or whatever) and
have their finer grained testing run in it. This way supporting Dag's
requirement of doing both upstream and downstream testing.

When I read Dag's comment I thought he had in mind the sharing the build artifacts in the "store" or "cache" that they are working on as opposed to disk images. I recall that Nix (the basis for some of the ideas in Dag and Ondrej's  project) had some fairly sophisticated approaches for sharing build results.  Maybe they can clarify.

 

Chris

-- Andy

[David]

I know I prefer 2 as well. I know of at least one attempt of doing something like this (linux-only, though): the build service from open suse (http://www.open-build-service.org/). 4 years ago, it could already do useful bits that are not trivial in any CI I am aware of:

  - dependencies between packages is known: if you update the build service package numpy, and the build is successful, it will automatically rebuild all the dependent packages

  - it handles completely isolated build environments through vms

  - it produces rpm, debian, etc.. that can be easily installed in the supported distributions.

My ideal setup would be something like this, but working on windows and mac (and not developed in perl), and with easier set up ala travis CI. I don't know yet the best way to go there: is is based on existing infrastructure (jenkins, travis, something else), building our own, a hybrid between the two ?

David

[Dag Sverre Seljebotn]

What I was imagined was really something very simple (and I put it too
convoluted). Just have NumFOCUS cash out :-) for one or more servers,
then give everybody who wants shell access to a shared CI instance
(running Jenkins or whatever).

In the Cython Jenkins we currently have build artifacts for all Python
versions that we use in Cython. We *should* have build artifacts for all
NumPy versions too; but that's too expensive.

But obviously, if we shared the CI server with NumPy, then Cython could
just hook itself against *some* of those artifacts (the head for each
release + trunk, but no development branches), and we'd get early
warning when NumPy trunk broke Cython.

Also, cooperation could just reduce the amount of hassle for the
different communities to set up CI, and get better bus factors for
administration of CI etc..

(If I understood Travis correctly on the other thread, the "cash out"
part makes this a bit premature; which is fine, let's just revisit this
thread in a year then...)

Dag

[Travis Oliphant]

This sounds like a great plan. NumFOCUS can provide for this. We will just need a budget and board approval (but I suspect the board will be happy to approve this kind of plan).

So, who wants to spec out the machine? There is some money. The problem is actually a people problem. Who is going to be available to do the work. So, far, I have not been able to find anyone experienced who is willing to work on this stuff at 50% time.

If you are willing, let me know in a private email or an email to ad...@numfocus.org

Best,

-Travis

[Stéfan van der Walt]

On Mon, May 28, 2012 at 11:27 AM, Dag Sverre Seljebotn
<d.s.se...@astro.uio.no> wrote:
> What I was imagined was really something very simple (and I put it too
> convoluted). Just have NumFOCUS cash out :-) for one or more servers, then
> give everybody who wants shell access to a shared CI instance (running
> Jenkins or whatever).

FWIW, we already have an Amazon EC2 up and running just waiting to be
used (currently, it runs an experimental TeamCity and buildbot).

Stéfan

[Dag Sverre Seljebotn]

A "problem" is that Cython probably has the best CI around in scientific
Python, and has had for years (thanks to Stefan Behnel setting up
Jenkins on Sage servers). So there's little incentive to rock the boat
until there's something better than what we have.

I was mostly curious to see if there was any existing plan. The issue of
a new CI server came up on the Cython lists some months ago; I raised
the possibility of asking numfocus then, but nobody else wanted to do
anything to divert any funding away from funded hacking time. And right
now things work OK.

Dag

[Aron Ahmadia]

It sounds like the hardest part about this is finding somebody to work on administering and setting up the CI servers, though I have another option in mind for free hosting.

I've been meaning to write up a proposal to FutureGrid (https://portal.futuregrid.org/) requesting resources for a project in reproducible research (providing services such as continuous integration and bootable system images).  They operate several scientific research clouds and the CPU hours are awarded on a no-cost grant basis.  

If there's interest, I could expand the proposal to request one or more continuous integration servers for our needs.

Regards,

Aron

On Tue, May 29, 2012 at 12:29 PM, Dag Sverre Seljebotn <d.s.se...@astro.uio.no> wrote:

[Travis Oliphant]

We are definitely interested in donated compute cycles.   We are contacting several hardware manufactures to get the same kind of time.   Please do expand the proposal and let us know how it goes. 

Along the same lines, has anyone tried using Travis CI?   It looks like they offer a pretty interesting service for open source projects, but I wonder how well it works in practice (i.e. how much time to they give you for build testing). 

-Travis

[Nathaniel Smith]

On Tue, May 29, 2012 at 3:08 PM, Travis Oliphant <teoli...@gmail.com> wrote:
> We are definitely interested in donated compute cycles.   We are contacting
> several hardware manufactures to get the same kind of time.   Please do
> expand the proposal and let us know how it goes.
>
> Along the same lines, has anyone tried using Travis CI?   It looks like they
> offer a pretty interesting service for open source projects, but I wonder
> how well it works in practice (i.e. how much time to they give you for build
> testing).

Travis CI is pretty slick. You get IIRC 15 minutes per build+test
cycle, with no limit on the number of test cycles (though if they
notice you using up all their capacity then I suppose they'd get
grumpy). They don't support Python 2.4 out of the box, but you can
install it yourself as part of your build script. To give a sense of
how much time 15 minutes is, their boxes take ~3 minutes to build
Python 2.4 + numpy:
http://travis-ci.org/#!/charlton/charlton
(The NUMPY_VER=1.2.1 line is actually using Python 2.4, don't believe
the Python version column)

Main advantages:
-- can be set up in about 20 minutes
-- for a nominal donation (?) they'll turn on their service where
they automatically run tests against all your pending pull requests
and leave a comment with the results:
http://about.travis-ci.org/blog/announcing-pull-request-support/

Main disadvantages:
-- No real way to report results or get artifacts out.
-- You can test on any platform you like, as long as it's 64-bit Ubuntu.

IMO the whole point of a CI system is that you can count on it to tell
you when you've broken the build, and that only works if it actually
tests on all supported platforms. So Travis CI is handy (and that pull
request trick might be a nice complement to another system), but in
the long term the lack of platform support seems like a fatal flaw to
me. (Just like all the other hosted CI platforms in this regard, I
guess.)

- N

[Stéfan van der Walt]

On Tue, May 29, 2012 at 7:43 AM, Nathaniel Smith <n...@pobox.com> wrote:
> Main disadvantages:
>  -- No real way to report results or get artifacts out.
>  -- You can test on any platform you like, as long as it's 64-bit Ubuntu.

What's also been stopping me from using it is that you have to rebuild
the entire dependency list on each run (so skimage would need to
compile cython, numpy, scipy, freeimage and/or matplotlib).

Stéfan

[Frédéric Bastien]

There have been discussion about a GPU ndarray. So take into account
that in the futur, we will want this to be in the buildbot. We use our
own computer here, the problem is that it is homemade scripts, so
nothing fancy... And our full buildbot test take 10h for Theano. So I
don't think someone will want to host it :) But we will try to reuse
the same system if possible and not too long to set up.

Frédéric

[Nathaniel Smith]

Yeah. They don't seem to actually care how wasteful this is, and it's
their CPU cycles, so oh well. But if your dependencies take too long
to build then you might run into the time limit, and scipy at least
does take a while to build (haven't tried it).

You could at least test against the Ubuntu-packaged versions of
numpy/scipy/etc., on the versions of Python that have such packages,
though.

- N

[Chris Kees]

Yes, I want to underscore that it's a fatal flaw with the hosted CI
platforms. On HPC platforms, if you're not testing on the hardware in
the compute node environment, you're missing some of the most
difficult problems. I'm not saying hosted CI isn't an important part
of the overall solution, but I think CI on HPC platforms needs
explicit attention.

>
> - N