Proposal: regression testing tool

[Josh Austin]

Any thoughts about building a regression testing tool for new Elixir releases? I'm thinking about something like cargobomb which is used for Rust release regression testing.

Other cargobomb info:

- blog post: https://brson.github.io/2017/07/10/how-rust-is-tested#s-ds

- example report: http://cargobomb-reports.s3-website-us-west-1.amazonaws.com/nightly-2017-07-07/index.html

I'm thinking having excellent tools like mix and hex.pm could enable something similar for Elixir. I'm interested in knowing your thoughts about this.

Best,

Josh Austin

[Allen Madsen]

I personally love the idea of releases running against commonly used packages. I'm not sure how rust does it, but it would be cool if the work could be distributed. For example, you do x, y, and z in your project and submit a URL to the elixir team and they have something that hits those URLs and expects them to ping back success or failure.

Allen Madsen
http://www.allenmadsen.com

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-core+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/7387f1f0-7124-495d-81b2-a94fcb7efbde%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Bryan Joseph]

I'm also interested in seeing something like what Rust does exist for Elixir and willing to help make it happen

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAK-y3CsPJ6EH4kEOub4rW93zH1WHDrp-fjNWusdw65THdJ0CAQ%40mail.gmail.com.

[OvermindDL1]

I'd also be willing to run such a build node as well, though perhaps travis could do the heavy-lifting?

On Thursday, August 3, 2017 at 7:25:50 AM UTC-6, Bryan Joseph wrote:

I'm also interested in seeing something like what Rust does exist for Elixir and willing to help make it happen

On Wed, Aug 2, 2017 at 5:46 PM, Allen Madsen <allen.c...@gmail.com> wrote:

I personally love the idea of releases running against commonly used packages. I'm not sure how rust does it, but it would be cool if the work could be distributed. For example, you do x, y, and z in your project and submit a URL to the elixir team and they have something that hits those URLs and expects them to ping back success or failure.

Allen Madsen
http://www.allenmadsen.com

On Wed, Aug 2, 2017 at 1:23 PM, Josh Austin <josh....@gmail.com> wrote:

Any thoughts about building a regression testing tool for new Elixir releases? I'm thinking about something like cargobomb which is used for Rust release regression testing.

Other cargobomb info:

- blog post: https://brson.github.io/2017/07/10/how-rust-is-tested#s-ds

- example report: http://cargobomb-reports.s3-website-us-west-1.amazonaws.com/nightly-2017-07-07/index.html

I'm thinking having excellent tools like mix and hex.pm could enable something similar for Elixir. I'm interested in knowing your thoughts about this.

Best,

Josh Austin

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.

To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/7387f1f0-7124-495d-81b2-a94fcb7efbde%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.

To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.

[Josh Austin]

That's interesting, I wonder if you could have an integration where a webhook from elixir-lang/elixir could trigger Travis test runs on individual package repos but direct them to use a new build of Elixir and post the results to a central location (S3 bucket or something).

I'm not an expert on Travis but that sort of configuration might go a long way toward making the regression check workload manageable. Then we'd just need to aggregate the results and diff them.

My biggest concern was around deploying the temporary compute resources to run all these tests, but if we could get the existing Travis configs to run that could be great.

[Bryan Joseph]

That is an interesting way to handle it if possible. My coworker, Luke had the idea of using a docker container per package. So each container would have the latest Elixir build. Either on push to master or a daily build. It would download the latest version of a package from hex, run tests or whatever and then report back to something with the results. Repeat for each package in hex (or a subset). And that way work could be distributed.

To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-core+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/d674917a-cdb0-47f1-a7e4-243042c23e26%40googlegroups.com.

[Dan McClain]

I had an idea and the beginnings of a pet project in which a repo could "subscribe" to updates of another repo and kick off Travis builds of that project. One concern I had was overloading Travis: if the source project has frequent commits, you'd have a frequent stampeding herd (you could limit to a certain # per day/hour/etc). The other issue is if you have a repo with a large number of subscriptions, you'd flood Travis's queue and prevent other projects from getting timely builds. 

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAKRLFpbWbeJig4EcO3iNex5GdeBSwMBy%3D9qjHDLeXpeWkCYEsw%40mail.gmail.com.

[OvermindDL1]

Or just have a single travis task use a mini-project that just depends on all of those libraries, then builds it.  Then rebuild it on 'update' of any of them along with a throttle?

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/d674917a-cdb0-47f1-a7e4-243042c23e26%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Bryan Joseph]

That's a pretty good idea and I think would solve the problem if you only care about checking against a subset of packages. Like the top 10 from hex.

To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-core+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/7f3e5de7-d7a2-4501-8373-0b13e5d70641%40googlegroups.com.

[Dan McClain]

The pet project is a service that allows you to test your library against your upstream dependencies (start specifically around Ember and ember addons). A specific project (like top 10 packages from hex) could be kicked off via a webhook pretty easily (using something like AWS lambda). If you set up that top-10 project properly, and passed enough context to the webhook (like commit), you could kick off the project for that build against the said SHA, have it update github commit status, and at the end of the build send the info back to the PR as success/failure status.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAKRLFpZfJAfC-L79DBHmiECcX2ojmP0bmDv_9nhV0vd%3D_huk3A%40mail.gmail.com.

[Josh Austin]

For an Elixir release regression testing tool we really wouldn't have to run it except for when there was a new Elixir release candidate. I'm not sure we'd need to trigger on each commit to elixir-lang/elixir.

I wonder if you could build a mix task that could test against the latest release candidate and upload the results to a central location. Then one could opt-in by including that task in the Travis config. Maybe we wouldn't get complete coverage exactly but it could be a nice way to ramp-up and distribute the workload.

[Josh Austin]

I'm thinking then one could also write a task that checked the deps of a project for tested compatibility against a specific release.