Use of Hosted Testing (redux)

[ian.t...@gmail.com]

The promise of hosted testing is several fold:

1) don't have to maintain infrastructure

2) on-demand

3) available to all developers

4) growing number of platforms

Looking into this, I expect it'd be a fair amount of work.  Still, here's what I found:

I see a closed issue for using Travis:

https://github.com/cl-test-grid/cl-test-grid/issues/24

where I'd summarize the concerns as:

a) cl-test-grid tests all known libraries rather than just one

b) these tests may cause timeouts

c) multiple platforms must be supported

d) triggering builds may not cleanly map to a commit

These seem to stem from Travis as commonly used by GitHub projects, but it doesn't seem like they're 

fundamental to Travis in particular, or hosted testing services in general.

It looks like the timeout is configurable, and also may not apply if there's output:

https://docs.travis-ci.com/user/common-build-problems/#My-builds-are-timing-out

Was there some testing that revealed otherwise?

Also, in addition to Linux and OSX, it looks like you can test on embedded platforms:

https://docs.travis-ci.com/user/integration/platformio/

Another concern in that issue was in when to do builds.  What if the set of versions to 

test together were encoded in a file, that way commits to that file trigger builds.  In any 

case, there's a triggering API:

https://docs.travis-ci.com/user/triggering-builds

Also, there's both Circle CI and AppVeyor which seem to have similar features, as well as 

include a few mobile platforms:

https://circleci.com/docs/mobile/

And then AppVeyor for Windows (unfortunately seems like only Windows Server 2012 R2 (x64)):

https://www.appveyor.com/docs/build-configuration#time-limitations

https://www.appveyor.com/docs/api/projects-builds

Then there's the issue of bootstrapping lisp implementations across platforms.  The Roswell 

project looks promising in this regard:

https://github.com/roswell/roswell

Where there are a few related articles:

http://eudoxia.me/article/lisp-travis-coveralls

http://eudoxia.me/article/common-lisp-sotu-2015#testing-on-the-cloud

(this points to a howto that's broken, I'm pasting the correct link, I think)

https://github.com/roswell/roswell/wiki/4.1-Travis-CI

https://github.com/roswell/roswell/wiki/4.2-Circle-CI

(where AppVeyor support seems experimental)

Not sure where to start, and I couldn't commit to much work myself, but I thought I'd share these findings in case they're useful.  There's some activity around Roswell so perhaps some collaboration is possible.

[Anton Vodonosov]

Thanks for sharing your findings. They have some information new to me
(for example last time I looked Travis had 20 minutes hardcoded timeout for tests,
now you say tests timeout only if there is no output).

Nuances still exist, in particular when to start tests seems to be the most difficult one.

Anyway, integrating with some cloud platform would take a serious effort I think.



29.04.2016, 05:02, "ian.t...@gmail.com" <ian.t...@gmail.com>: