Far too many packages broken on 0.2 + more general questions

[Iain Dunning]

Hi all,

I have finally started running PackageEvaluator nightly as a cron job, so status.julialang.org is going to stay up to date (at least the part that I feed).

I currently run only against Julia v0.2, and the results are pretty horrifying (if you are of the opinion that 0.2 support is important, which I know some people don't agree with). The real frightening ones are the "Package Unloadable" packages - some of those I'm hoping a false negatives due to some configuration shortcoming.

Please check for your packages here http://status.julialang.org/

If your tests (or even loading the package) are failing due to some binary dependency, please file an issue at https://github.com/IainNZ/PackageEvaluator.jl/issues and I will add an exception.

Next steps are

 - Also test against nightly Julia

 - Test against package master versions

 - Automatically generate github issues or something like that.

Couple of questions:

 - Should we have a policy for removing packages? Some seem both abandoned and broken.

 - If we do remove packages from METADATA, is Pkg going to freak out? Presumably we'd only remove packages that weren't are dependency.

 - Would a daily name-and-shame/testing report email to julia-dev be good or just spam/noise?

 - I think the best idea for automatic github issue generation is to only do it on a negative change, i.e. before it could at least do using, now can't, or tests were passing, now they're not. Any other ideas on this?

Cheers,

Iain

[Iain Dunning]

I've just opened a nagging issue on quite a few packages to encourage more "greens". Happy to report that even Gadfly, a complicated package in terms of dependencies, should begin passing as of tonight thanks to adding some manual exceptions for setup and teardown of tests.

I've also added a couple of untestable packages.

I'm broadening the "test file search" so that if there is a single file in the test directory I'll try to run that. Hopefully that'll cut down the number of "blues".

Thanks for your patience!

Cheers,

Iain

[Iain Dunning]

I got impatient and run it before the nightly cron job.

Update, after adding better package detection:

Category - Number Before Changes - Number Now - Delta

Tests Passing, 83, 99, +16
Tests Failing, 45, 59, +14
Tests Missing, 89, 65, -34
Package Unloadable, 23, 16, -7

Total packages now testable: 246

Package Excused, 27, 28, +1

So 40% of testable packages have tests and passing, and additional 26% don't have (detectable) tests but can at least be "using"ed.

[Andreas Lobinger]

Hello colleague,

just a quick question (and i may have overlooked this): On which OS/Platforms/Environments does the test run?

Wishing a happy day,
         Andreas

[Iain Dunning]

Hi Andreas,

The tests run with Julia 0.2 on Arch Linux (with no graphics running).

Cheers,

Iain

[Pierre-Yves Gérardy]

Just to be sure, is the exit status is enough to report the results (0 for success, 1 otherwise)?

[Iain Dunning]

Yes thats correct, see:

https://github.com/IainNZ/PackageEvaluator.jl/blob/master/src/package.jl#L243

[Pierre-Yves Gérardy]

Good, thanks :-)

[Iain Dunning]

Update with two new tweaks:

- I changed it so it now says PackageName - Version/SHA (time since that commit) - author

- Support for associating test results with a Julia version has been added but hasn't been integrated into the website yet.

- If I find a LICENSE file or similar but its empty, I'll still report that there is a file, just that its unknown. So now its a green-orange-red (no license file, no license)

We've now climbed to 99 packages passing, which is good to see - hopefully can break 100!

[Spencer Russell]

Is there a way to get the test output from these results? The AudioIO tests pass on Travis but apparently are not passing in this check. It's likely because I'm using BinDeps to install some dependencies, which I've tested in Ubuntu and OSX but not on Arch.

-s

[Iain Dunning]

Not yet, but its on the todo list.

Heres the output I see for AudioIO:

##### Current package: AudioIO

INFO: Cloning cache of AudioIO from git://github.com/ssfrr/AudioIO.jl.git

INFO: Installing AudioIO v0.0.2

INFO: Installing BinDeps v0.2.12

INFO: Installing URIParser v0.0.1

INFO: Building AudioIO

===============================[ ERROR: AudioIO ]===============================

None of the selected providers can install dependency libportaudio

at /home/idunning/pkgtest/.julia/v0.2/AudioIO/deps/build.jl:14

================================================================================

================================[ BUILD ERRORS ]================================

WARNING: AudioIO had build errors.

 - packages with build errors remain installed in /home/idunning/pkgtest/.julia/v0.2

 - build a package and all its dependencies with `Pkg.build(pkg)`

 - build a single package by running its `deps/build.jl` script

================================================================================

INFO: REQUIRE updated.

INFO: 

INFO: Running tests from "test_AudioIO.jl"...

INFO: ===================================================================

INFO: Testing Playing Float32 arrays...

INFO: Testing Playing Float64 arrays...

INFO: Testing Playing Int8(Signed) arrays...

INFO: Testing Playing Uint8(Unsigned) arrays...

INFO: Testing AudioNode Stopping...

INFO: ===================================================================

INFO: 

INFO: Running tests from "test_nodes.jl"...

INFO: ===================================================================

INFO: Testing AudioMixer...

INFO: Testing SinOSC...

INFO: Testing ArrayPlayer...

INFO: ===================================================================

INFO: Removing AudioIO v0.0.2

INFO: Removing BinDeps v0.2.12

INFO: Removing URIParser v0.0.1

INFO: REQUIRE updated.

If you'd like I can add an "exception" for AudioIO - many packages with special binary dependencies are excepted. Just open an issue.

Cheers,

Iain