Travis backlog: moving packages out of JuliaLang?

[Tim Holy]

Hi all,

Given the amount of development activity these days, Travis has quite a long
backlog. I could be wrong about this, but I have the impression that JuliaLang
gets a fixed number of slots and that packages under JuliaLang
(https://github.com/JuliaLang?page=1) compete with julia-proper for those
slots.

If that's true, how about moving packages to other organizations? For example:

JuliaIO:
HDF5.jl
JLD.jl
JSON.jl
LightXML.jl

JuliaGraphics:
Graphics.jl
Gtk.jl
Tk.jl
Cairo.jl
Reactive.jl?

JuliaPackages? (doesn't exist yet):
METADATA.jl
BinDeps.jl
PkgDev.jl
WinRPM.jl
Compat.jl?

Other candidates are ODE, JuliaBox, IJulia, ZMQ, Iterators. The key,
presumably, is to move the most actively-developed packages, as these cause
the greatest Travis load.

Best,
--Tim

[Tim Holy]

I went ahead and did this for Gtk.jl, with dramatic results (from a ~2 hour wait for Travis down to nearly instantaneous). Unless I hear otherwise I'm going to start doing this with at least the packages I've played a big role in.

Moving METADATA.jl might be the single biggest win, but I won't do that unless others think it's a good idea.

--Tim

[Andreas Lobinger]

Hello colleague,

just-my-two-cents: Yes i agree JuliaLang and other organisations should have clearer agenda and focus what is inside (see similar on JuliaPlot and JuliaPlots). But moving packages because of travis overflow looks like working on a second order effect. Shouldn't there be some effort spend to make tests run faster, first?

My recommendation would be to have a slot at julia con to discuss structure of julia as a programming environment. Like questions on a standard library etc.

[Tim Holy]

There's already been quite a lot of effort to make the tests run faster, both
through streamlining the tests themselves and through improvements to codegen.
As a fairly extreme case, the subarray tests (among the longest-running) take
about 40% of the time they used to take, which is a huge savings, despite
testing more thoroughly than ever. Because test time has been a persistent
problem, both Jeff and I have recently made optimizations specifically targeted
at improving the performance of Base.Test. So, it's easy to say "make the
tests run faster," and I think it's fair to say that the situation will
continue to improve, but it's not like there's much low-hanging fruit just
lying around waiting to be picked. Presumably future improvements will come
mostly from improving the performance of codegen, which everyone wants, but
such improvements are never easy.

Worst of all, some extremely interesting PRs
(https://github.com/JuliaLang/julia/pull/6837) are currently blocked from
being merged for the main reason that they greatly increase the test time.

So let's extrapolate a bit. Currently julia itself takes ~45 minutes for its
tests. Let's say one gets this down to ~35 minutes. Now, each time someone
submits a new commit to an existing PR, the Travis tests run. If we have 2
Travis slots on JuliaLang, it only take 4 pushes per hour to have an
exponentially-growing queue of Travis jobs. Now, let's say (optimistically)
that each PR gets merged to master, but given rebases there are 1.5x as many
commits to branches as there are to master. If (pessimistically) people push
commits one at a time, then each of these triggers Travis. You trigger Travis
both when you submit the PR and when you merge to master, so we can estimate
(probably pessimistically) the number of Travis builds is something like 2.5x
the number of commits to master in a day. Consequently, with just an average
of 40 commits to master a day, we cannot manage the queue. Now, we don't
usually have 40 commits to master in a day, but some days we do and often
we're in that ballpark.

All this excludes the commits to METADATA.jl and to packages under the
JuliaLang umbrella. Each of those tends to be much faster (from 1-10 minutes),
but collectively there are many more of them.

As an example: yesterday I submitted a 12-line change to Compat.jl. The Travis
run only took 90s per job, but it had to wait in line for 4 hours because of
the backlog. The same thing happens when you tag a new package, because
METADATA.jl is under JuliaLang. Often when I'm working on packages, I might
update one "low level" package (e.g., ColorTypes), and then want to tag a
cascading series of other packages that depend on it. In the worst case, each
has to be tagged independently if the Travis tests are to succeed, so even a
trivial (but important) change to a low-level package can result in a half-day
of making small changes and then waiting for Travis. This is starting to get
so slow that it's hurting development.

Consequently, my conclusion is: Houston, we have a problem. Fortunately, it's
easy to end at least the drag on package development, by moving most of the
packages out from under the JuliaLang umbrella.

Best,
--Tim

[Tony Kelman]

Yes, packages should move from JuliaLang to separate organizations. Especially packages that only a handful of people have contributed to, since managing commit access will be simpler there. Commit access, badge links, and re-enabling CI services are the main things we shouldn't forget to do when we move things (did Gtk have AppVeyor enabled? AV doesn't handle repo moves very well, and Travis might not either).

I'm a bit hesitant about Compat, BinDeps, and METADATA since they're so central. But lots of the general math-related packages would be good to move to something like JuliaMath.

Another thing that may help is spreading out some of base Julia's Travis matrix to separate services. There are lots of competitors to Travis we can evaluate. Travis is the only one with a premade "language: julia" option, but only packages use that.

[Jeffrey Sarnoff]

+1

this has the additional benefit of easing the task of locating packages of a certain sort

e.g. Numerics, Geometry, Analytics, Statistics, DataStructures, IO, Graphics, Plotting, ...

[Tony Kelman]

Oh and don't forget to change the url in METADATA as well. Otherwise if someone were to create a new repo at JuliaLang/Gtk.jl it would break the github redirect and bad times would ensue.

A less work but non free option is that we can pay for additional capacity on a paid plan with Travis. They don't visibly list prices for this paid open source option last I checked, but have said to contact them with inquiries about price. It's also possible to ask for a temporary boost to the max number of concurrent jobs for JuliaLang, I just have to ask nicely on Travis' slack channel. I haven't taken advantage of this yet since I don't want to abuse it, but come JuliaCon I suspect we'll be really busy so someone should remind me to do this.

[Tim Holy]

On Wednesday, May 11, 2016 05:12:59 AM Tony Kelman wrote:
> I'm a bit hesitant about Compat, BinDeps, and METADATA since they're so
> central. But lots of the general math-related packages would be good to
> move to something like JuliaMath.

I understand the hesitancy, though I suspect you also see the advantage. We
can wait on deciding.

I noticed you did create JuliaPackages---can you give a list of the packages
you envision transitioning there?

Best,
--Tim

[Tony Kelman]

I created it mostly to reserve the name in case I change my mind. Those 3 really central repos tend to need the most careful review aside from base Julia, and it helps to have more eyes on them. Moving packages to other organizations isn't totally without cost in terms of visibility and organization, so it's an easier sell IMO (though less of a help with the backlog) for lower-traffic packages than for critical pieces of infrastructure. And I am against moving anything new into JuliaLang that isn't there right now.

I personally don't "watch" all that many repos as the notifications for threads I'm not involved in yet can be a bit much, but the JuliaLang organization page being sorted by modification date can be a convenient central way to find out when work is going on. If there are Travis issues that can't be found from local testing, the debugging turnaround can also be reduced by making pull requests from personal forks, enabling Travis and looking at the branch builds there (this is also true for work on base Julia, but to a lesser extent as it requires manual cache building and temporary modification of the yml). As long as our backlog catches up by the end of the day, I don't really hate the idea of imposing a bit of delay before merging things on high-traffic repos. I've expressed a preference for leaving base PR's that aren't time-critical or absolutely trivial open for at least 24 hours to allow for feedback for example. Tagging a series of related packages is the main case that justifies wanting to rush changes through faster, just so you can get more done in a single sitting.

METADATA is the highest traffic repo aside from base, but the run time for its single CI matrix job is tiny. And its URL is hard coded in several places that we would need to update to reduce the chances of any issues happening with the github redirect.

[Alex Arslan]

ODE could live in Tony's newly created JuliaMath organization. I put up a proposed list of features for JuliaMath which incorporates a healthy chunk of the JuliaLang packages. See https://github.com/JuliaMath/Roadmap.jl/issues/1. So hopefully that will help in this regard. :)

Perhaps JuliaBox would be suitable for JuliaWeb? Not sure about decoupling it from IJulia though since they kind of go hand-in-hand.

Anyway, that's my $0.02.

[Tony Kelman]

I just moved a bunch.

Cairo, Tk, and Graphics.jl to JuliaGraphics
LightXML and IniFile.jl to JuliaIO
Color.jl and juliawebstack.org to JuliaArchive

Didn't move it yet, but GZip.jl would be another good candidate for JuliaIO.

I also created a JuliaEditorSupport organization which we could potentially move julia-emacs, atom-language-julia, julia-vim, and Julia.tmbundle to.