Go betas on Travis or other continuous testing systems?

[Russ Cox]

Hello all,

David Chase and I were just brainstorming ways to get more testing of Go work ahead of the releases. I am aware that Travis provides continuous testing to Go projects but not really aware of the details. I assume they let you pick what release to test against (Go 1.8, Go 1.9, and so on).

Are there enough Go users using Travis that it would make sense to coordinate with them to make the betas and release candidates available as well, maybe even automatically on in parallel with the usual release people have configured? Or have they done this before and I just don't know about it?

Are there any other continuous testing services that we should be thinking about talking to?

Thanks.

Russ

[Avelino]

Hi,

I (particularly) like Travis CI (was my choice for Nuveo), I believe he meets all our needs of the Go language core.

I've tested it and I like it:

- Circle CI

- Coveralls

There are others on this list: https://github.com/ligurio/Continuous-Integration-services

Sent from ProtonMail Mobile

--
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Konstantin Kulikov]

Drone ci (https://github.com/drone/drone) for example uses docker
images to run tests. I think if you'd post betas on docker hub, more
people will use them.

[ra...@develer.com]

CircleCI uses Docker, as you can see in this example:

https://github.com/CircleCI-Public/circleci-demo-go/blob/master/.circleci/config.yml#L6

Looking into docker hub, it looks like they've already built betas and rcs for go 1.9:

https://hub.docker.com/r/circleci/golang/tags/

and they also a tag simply called "rc" which I can presume it's updated for every new "rc", but it's undocumented AFAICT.

I think it would help to have a documented tag that allows to automatically upgrade to a beta/rc as soon as it's released.

Giovanni

[Brad Fitzpatrick]

Avelino,

We're not looking for a CI system ourselves. We're talking about helping existing Travis users help us test Go betas more easily.

To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "golang-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.

[Avelino]

Tag `before_install` or used docker service

We can rotate everything inside the Docker (with image maintained by us) if we have a very different need.

[],s

Avelino

+1 630 652 5340

+55 11 981 574 800

E3B5 94B9 A7CF BD3A BF57 FA00 B345 B4D5 2E98 180A

[Pierre Durand]

Go betas are already available on Travis.

[Florin Pățan]

Hi,

Please see the reply inline:

On Friday, November 10, 2017 at 7:42:14 PM UTC, rsc wrote:

Hello all,

David Chase and I were just brainstorming ways to get more testing of Go work ahead of the releases. I am aware that Travis provides continuous testing to Go projects but not really aware of the details. I assume they let you pick what release to test against (Go 1.8, Go 1.9, and so on).

Travis has a tool called gimme which already allows you to fetch all Go stable versions and even specify Go commits if one wants to test something like that.

The tool can also be used in other CIs / environments. 

 

Are there enough Go users using Travis that it would make sense to coordinate with them to make the betas and release candidates available as well, maybe even automatically on in parallel with the usual release people have configured? Or have they done this before and I just don't know about it?

Some users prefer to have at least a couple of Go versions they are testing against, especially if they are releasing / maintaining an open-source project. Having tip as a version is very frequent as well.

 

Are there any other continuous testing services that we should be thinking about talking to?

Thanks.

Russ

There is also CircleCI which is very popular.

In my opinion both gimme and other solutions can easily add betas / RCs to their support list.

However, in my opinion, it would be useful to have a better Docker release in the form of automatic building / pushing of new images when new commits / tags happen. That would allow everyone to use them regardless of the environment they are in.

Hope this helps.

[Nathan Youngman]

Travis CI has a matrix which can build across multiple versions of Go, using gimme under the hood as Florin mentioned. Testing against tip is perfectly doable, though I don't know how common this practice is:

go:
	- 1.9
	- 1.8
	- tip
	matrix:
	allow_failures:
	- go: tip
	fast_finish: true

As soon as betas are available, gimme can grab them without changes on their end, though it does involve updating TravisCI files as new versions are released. It might be useful if there was something in between gimme stable and gimme master, something that went for the latest beta, or stable if no new betas exist.

https://github.com/travis-ci/gimme

CircleCI and Drone.io are both based on Docker images, which could be from anywhere, though the official Go image on DockerHub is a good place to start.

https://hub.docker.com/_/golang/

https://github.com/docker-library/golang/

Traditionally CircleCI was limited to just testing with one Go version at a time, as it didn't have a build matrix. But with CircleCI 2.0 workflows, I believe parallel builds could be accomplished.

It may be worth coordinating the beta release with folks from Travis CI, Docker, CircleCI, and so on. There are plenty of other options, such as AppVeyor on Windows. I think a blog post could present a few options to encourage more people to add the betas to CI without coming across as biased to particular CI systems.

With Go 1.10 supporting coverage across packages and Go 1.9 skipping vendor folders, it seems like a good time to refresh those CI configs with some best practices. Not to mention removing that gofmt test from CI. :-)

Cheers,

Nathan.

[Ian Davis]

For reference here is how I matrix test against several versions of Go in Travis and CircleCI:

https://github.com/iand/circuit/blob/master/.travis.yml

https://github.com/bankrs/bosgo/blob/master/.circleci/config.yml

Ian

--

You received this message because you are subscribed to the Google Groups "golang-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.

[Sebastien Binet]

FYI, `tip` should be spelled out `master` now.

To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "golang-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.

[Sam Whited]

On Fri, Nov 10, 2017, at 13:42, Russ Cox wrote:
> Are there any other continuous testing services that we should be
> thinking about talking to?

Having betas (or even nightlies) available on the Go Docker Hub account
may make it possible for many companies to add them to their internal
testing matrix, and as others have said you might also then get them for
"free" on public CI systems like CircleCI and Bitbucket Pipelines.

—Sam

[Sameer Ajmani]

+Carmen and I have chatted about this idea before. Ideally we'd like to be able to determine whether a given Go release candidate causes a test to transition from reliably passing to reliably failing, or flaky, or failing to build.

--
You received this message because you are subscribed to the Google Groups "golang-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.

[Damian Gryski]

On Friday, November 24, 2017 at 5:18:32 PM UTC-8, Sameer Ajmani wrote:

+Carmen and I have chatted about this idea before. Ideally we'd like to be able to determine whether a given Go release candidate causes a test to transition from reliably passing to reliably failing, or flaky, or failing to build.

I've tweeted a bunch previously to encourage people to add pre-release versions of Go to their build systems.  The issues I've run into stem from convincing people to investigate failures from the pre-release version and file bugs against Go, rather than just disabling the check again and getting on with their lives.

Damian

[Sameer Ajmani]

I was thinking of something more automated: if CI shows a transition from reliably passing to broken/failing/flaky with a Go release candidate, automatically create an issue with reproduction instructions, information about the failing target, and the failure log.  I could imagine this being very noisy, but we could do filtering/aggregation at the other end.

--

[Nathan Youngman]

Sameer,

It sounds like you’re after something like the Rust crater project? Or at least, similar goals.

https://brson.github.io/2017/07/10/how-rust-is-tested#s-ds

Was that the idea behind:

https://github.com/rsc/corpus

Nathan.

You received this message because you are subscribed to a topic in the Google Groups "golang-dev" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-dev/ONpj39nviNg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to golang-dev+...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Nathan Youngman 
Email: he...@nathany.com
Web: https://nathany.com

[David Chase]

I think Russ's corpus project is to look for examples that might influence Go 2 design.

I have a project to do something for benchmarks that also runs tests that is somewhat similar:

https://github.com/dr2chase/bent

But:

 - it needs more automation (to detect regressions/advances)

 - it only works for tests that are go-gettable

 - it runs stuff in a Docker image to reduce risk from unintentional errors, but I worry about intentional errors

 - it gets confused when test directories are refactored (e.g., k8s recently)

All this made Russ wonder if there was a better way.

On Thu, Nov 30, 2017 at 10:47 AM, Nathan Youngman <he...@nathany.com> wrote:

Sameer,

It sounds like you’re after something like the Rust crater project? Or at least, similar goals.

https://brson.github.io/2017/07/10/how-rust-is-tested#s-ds

Was that the idea behind:

https://github.com/rsc/corpus

Nathan.

On Mon, Nov 27, 2017 at 8:17 PM Sameer Ajmani <sam...@golang.org> wrote:

I was thinking of something more automated: if CI shows a transition from reliably passing to broken/failing/flaky with a Go release candidate, automatically create an issue with reproduction instructions, information about the failing target, and the failure log.  I could imagine this being very noisy, but we could do filtering/aggregation at the other end.

On Mon, Nov 27, 2017 at 8:25 PM Damian Gryski <dgr...@gmail.com> wrote:

On Friday, November 24, 2017 at 5:18:32 PM UTC-8, Sameer Ajmani wrote:

+Carmen and I have chatted about this idea before. Ideally we'd like to be able to determine whether a given Go release candidate causes a test to transition from reliably passing to reliably failing, or flaky, or failing to build.

I've tweeted a bunch previously to encourage people to add pre-release versions of Go to their build systems.  The issues I've run into stem from convincing people to investigate failures from the pre-release version and file bugs against Go, rather than just disabling the check again and getting on with their lives.

Damian

--
You received this message because you are subscribed to the Google Groups "golang-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to a topic in the Google Groups "golang-dev" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-dev/ONpj39nviNg/unsubscribe.

To unsubscribe from this group and all its topics, send an email to golang-dev+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Nathan Youngman 
Email: he...@nathany.com
Web: https://nathany.com

--

You received this message because you are subscribed to the Google Groups "golang-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.

[Nathan Youngman]

Hi,

Maybe this would help make Sameer's idea a little more concrete.

For CircleCI, some projects I work on use JUnit XML files, which CircleCI uses to build a nice little UI for test failures.

https://circleci.com/docs/1.0/test-metadata/

https://github.com/jstemmer/go-junit-report

On the CI side, it would be fairly simple for an (open source) project to opt-in to posting artifacts like this to a central server. It seems like most of the work would be in building a server to store and analyze the data. If a given project is very near 100% stable on the Go release build but failing on beta or tip builds, it could offer a way to drill into those failures.

That would provide the automation.

Working with CI companies to provide beta Docker containers, documentation, and so on seems like a good first step. Then perhaps working with specific project owners that want to opt-in to submitting build information?

Nathan.

[Kevin Burke]

I’m reasonably sure that enough open source Go projects test against “tip” or “master” that if a regression occurred as Sameer suggested it would show up as failure on those projects.

Some projects are relatively complete and don’t get many new PR’s and as a consequence don’t run the tip tests that often (they’re triggered by new activity). We could increase coverage by encouraging those projects to run the tests as a cron (you can configure this in the Travis config file as well.)

[Nathan Youngman]

That's a good suggestion Kevin.

CircleCI 2.0 also scheduling:

https://circleci.com/docs/2.0/workflows/#scheduling-a-workflow

So is it fair to say that this could be broken down as follows:

* Availability of beta/tip release of Go for commonly used CI services (and quickly enough to be useful)

* Guidance and documentation for the open source community, possibly through the CI services, as well as the Go blog, website, and Wiki

* Automation around the submission of test failures, and detection of failures that are isolated to tip/betas.

Nathan.

On 1 December 2017 at 10:08, Kevin Burke <kbur...@gmail.com> wrote:

I’m reasonably sure that enough open source Go projects test against “tip” or “master” that if a regression occurred as Sameer suggested it would show up as  failure on those projects.

Some projects are relatively complete and don’t get many new PR’s and as a consequence don’t run the tip tests that often (they’re triggered by new activity). We could increase coverage by encouraging those projects to run the tests as a cron (you can configure this in the Travis config file as well.)

--
You received this message because you are subscribed to a topic in the Google Groups "golang-dev" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-dev/ONpj39nviNg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to golang-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Sameer Ajmani]

Thank you all for the ideas and suggestions. I'll take a look at the links you provided.

On Fri, Dec 1, 2017 at 1:55 PM Nathan Youngman <he...@nathany.com> wrote:

That's a good suggestion Kevin.

CircleCI 2.0 also scheduling:

https://circleci.com/docs/2.0/workflows/#scheduling-a-workflow

So is it fair to say that this could be broken down as follows:

* Availability of beta/tip release of Go for commonly used CI services (and quickly enough to be useful)

* Guidance and documentation for the open source community, possibly through the CI services, as well as the Go blog, website, and Wiki

* Automation around the submission of test failures, and detection of failures that are isolated to tip/betas.

Nathan.

On 1 December 2017 at 10:08, Kevin Burke <kbur...@gmail.com> wrote:

I’m reasonably sure that enough open source Go projects test against “tip” or “master” that if a regression occurred as Sameer suggested it would show up as  failure on those projects.

Some projects are relatively complete and don’t get many new PR’s and as a consequence don’t run the tip tests that often (they’re triggered by new activity). We could increase coverage by encouraging those projects to run the tests as a cron (you can configure this in the Travis config file as well.)

--
You received this message because you are subscribed to a topic in the Google Groups "golang-dev" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-dev/ONpj39nviNg/unsubscribe.

To unsubscribe from this group and all its topics, send an email to golang-dev+...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Nathan Youngman 
Email: he...@nathany.com
Web: https://nathany.com

--

You received this message because you are subscribed to the Google Groups "golang-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.

[Alexey Palazhchenko]

One more link: Docker image with Go from master branch, updated hourly – https://hub.docker.com/r/captncraig/go-tip/. I use it in my CI, works great.

A note for people using gimme: to install the latest patch version of, for example, Go 1.9, you have to spell it "1.9.x" – "1.9" will install exactly that version. "1.x.x" will work too. That "x" handling is done by travis-build (https://github.com/travis-ci/travis-build/blob/master/public/version-aliases/go.json) and not by gimme itself.

–-–
Alexey «AlekSi» Palazhchenko