Go builders for fsnotify?

[Nathan Youngman]

Hi,

Interest in the fsnotify project is bringing support for new architectures and operating systems. Docker is one of the applications depending on fsnotify, and likely the reason for contributions from employees of IBM and Oracle.

You probably saw recent CLs into the syscall package as well as x/sys/unix for linux/arm64 and power pc. The fsnotify test suite is made up entirely of integration tests that exercise the underlying notification systems. Our current build infrastructure uses freely available services (Travis CI, AppVeyor) for testing Linux, OS X, and Windows on a single 64-bit x86 architecture. Without more comprehensive testing, it's becoming rather scary for me to make or merge changes. 

The best remedy I know of is to utilize the Go builder machines, covering the whole gamut of platforms that Go itself supports.

From @bradfitz:

Currently our builders can only run stuff in golang.org/x* and hosted on Gerrit.

That's not fundamental, but it's just how things are currently implemented.

So my first question is whether the Go team would be willing to allow github.com/fsnotify/fsnotify to use the Go builder machines and what would it entail to modify the implementation to do so?

And secondly, is the Go team willing and interested in adopting fsnotify into the golang.org/x/* subrepositories? If that is the preferred course of action, I have a lot more questions to ask..

If I recall, there were plans to allow access to the builders from a local computer while debugging issues? Is that currently possible? CircleCI offers this via ssh, but again, on a very narrow set of platforms. My local testing environment requires a Mac due to OS X licensing and several VMs for BSD, Linux, and Windows. It's not easy or always possible for other contributors to setup and doesn't cover other architectures. 

Cheers,

Nathan.

[Damian Gryski]

On Thursday, April 21, 2016 at 7:28:12 AM UTC+2, Nathan Youngman wrote:

So my first question is whether the Go team would be willing to allow github.com/fsnotify/fsnotify to use the Go builder machines and what would it entail to modify the implementation to do so?

 Opening up builders to other projects would allow one other "would be nice" feature, which is the ability for Go to test "important" packages from the ecosystem as part of builds.   Rust already does this to detect packages that will break with language changes for the upcoming version.  It's not even a requirement that the builders run on every commit from the upstream project, or even that the tests themselves are run.  Just that a "known good" revision is checked out and built as a secondary smoke test outside of the main repo.

Damian

[brad.ry...@gmail.com]

Nathan had asked a similar question in the Drone CI issue tracker. I have posted a reply that details what would be required to support multiple architectures and operating systems in Drone:

https://github.com/drone/drone/issues/135#issuecomment-67228585

If anyone is interested in helping bring multi-architecture support to Drone and willing to submit a few pull requests let me know. I would be willing to sponsor some windows and arm machines for the community.

[Nathan Youngman]

Fortunately the Go language and libraries aren't changing a whole lot. It still would be a pretty good idea to ensure various third-party packages work on tip.

Drone is looking pretty great -- I really like the test matrix stuff. It would be pretty amazing to utilize Drone (Apache licensed) for the Go builders in the long-term.

Even if fsnotify doesn't utilize the Go builders directly, what would help build confidence is tests for the x/sys and/or syscall packages covering the functions being used by fsnotify, or a well-tested intermediary library. I opened an issue to propose such a library: https://github.com/golang/go/issues/15406

Thanks, Nathan.

--
You received this message because you are subscribed to a topic in the Google Groups "golang-dev" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-dev/SVrNHQU1oEM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to golang-dev+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Nathan Youngman 
Email: he...@nathany.com
Web: https://nathany.com

[Brendan Tracey]

On Apr 21, 2016, at 8:13 PM, Nathan Youngman <he...@nathany.com> wrote:

Fortunately the Go language and libraries aren't changing a whole lot. It still would be a pretty good idea to ensure various third-party packages work on tip.

In the recent past (go 1.5 cycle?) we had been testing the gonum packages on tip as well as the stable branches. However, the failures on tip were too frequent and were interfering with our development cycle, so we discontinued that practice.

You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.

[Damian Gryski]

On Friday, April 22, 2016 at 6:41:46 AM UTC+2, Brendan Tracey wrote:

In the recent past (go 1.5 cycle?) we had been testing the gonum packages on tip as well as the stable branches. However, the failures on tip were too frequent and were interfering with our development cycle, so we discontinued that practice.

I saw at least two projects with `tip` in travis.yml find bugs in the new SSA backend.

Damian 

[Nathan Youngman]

That's awesome. For fsnotify we use allow failures on tip. That way it doesn't break the build or prevent merging, but we can still see how it's working on tip.

https://github.com/fsnotify/fsnotify/blob/master/.travis.yml

Back to the original topic though. Im trying to figure out how to test more architectures for fsnotify. 

Thanks, Nathan.  

[Russ Cox]

We spend a lot of time just keeping the builders running for the main Go repositories. I am not sure we want to take on additional third-party repositories. I'm not saying no, but I'm saying we'd need to think very hard before a yes. 

It looks like you do have decent operating system coverage from Travis and AppVeyor. Have you been running into architecture-specific test failures? I wouldn't expect fsnotify on linux/ppc64 to be appreciably different from fsnotify on linux/amd64, for example.

Russ

[Nathan Youngman]

Hi Russ,

No architecture-specific failures yet, but we haven't supported linux/ppc64 and linux/arm64 until now. I'm more concerned more about keeping things working.

Pushing some testing down into sys or an intermediary "fsn" library could also ensure that architectures are behaving the same without needing to run the Go builders on fsnotify itself.

fsn proposal issue: https://github.com/golang/go/issues/15406

Nathan.

[Brad Fitzpatrick]

It is actually a medium-term goal of mine to support third-party repositories, once I complete some redesign of our build system and redo its scheduler. I'm putting off builder work until after the Go freeze.

The first trial customer will be grpc-go, which is hosted on Github. There are resource cost and isolation questions, so it'll likely be whitelist-only for a bit, and certain projects will probably only get spare resources which are otherwise idle.