Automated builds for Vim

[Florian Walch]

Hi,

in an attempt to make testing Vim a bit easier, I created a Github repository [1] that uses Travis CI to compile Vim and run the tests.

This is a proof-of-concept right now and only builds for Linux (Ubuntu 12.04 with GCC 4.6.3 and Clang 3.4) with different feature sets (tiny, ...). See [2] for the build script.

It should be fairly easy to extend it to build on OS X (currently in Beta on Travis CI) and Windows (using AppVeyor), or e.g. to use different compiler versions.

It currently does two things:


1. Building of latest Vim version

Pulls latest Vim from its Mercurial sources, compiles it and runs the tests. See [3] for a build of current Vim (7.4.582), which I think already uncovered a problem: Running the tests fails with tiny and small features ("test16 NO OUTPUT"), but works with bigger feature sets.

These builds are currently triggered manually, but could easily be turned into nightly builds by using a cron job.


2. Testing of patches

A pull request containing patches can be created on Github (Example: [4]), triggering a build that applies these patches before Vim is compiled and tests are run. This could be used by contributors to test Vim patches.


As I said in the beginning, this is only an experiment for now, so please let me know whether you think this could be useful.

Cheers,
Florian

[1] https://github.com/fwalch/vim-ci
[2] https://github.com/fwalch/vim-ci/blob/master/.travis.yml
[3] https://travis-ci.org/fwalch/vim-ci/builds/47524660
[4] https://github.com/fwalch/vim-ci/pull/1

[Christian Brabandt]

Hi Florian!

Sounds very interesting. Is the build always triggered for pull requests
with all different feature sets?

If I open a new pull request the travis CI will immediatly start with
the test suite, right?

Best,
Christian
--
Egoistische Kleinstädterei, die sich Zentrum deucht.
-- Goethe, Maximen und Reflektionen, Nr. 667

[Christian Brabandt]

Hi Florian!

Thanks for contributing, I thinks this is really helpful.

On Mo, 19 Jan 2015, Florian Walch wrote:

> Pulls latest Vim from its Mercurial sources, compiles it and runs the tests. See [3] for a build of current Vim (7.4.582), which I think already uncovered a problem: Running the tests fails with tiny and small features ("test16 NO OUTPUT"), but works with bigger feature sets.

Interesting, I do not see this here locally, but when I run make
test16.out I see an error. This patch fixed
diff --git a/src/testdir/test16.in b/src/testdir/test16.in
--- a/src/testdir/test16.in
+++ b/src/testdir/test16.in
@@ -2,6 +2,7 @@ Tests for resetting "secure" flag after
For KDE set a font, empty 'guifont' may cause a hang.

STARTTEST
+:so small.vim
:if $DISPLAY == "" | e! test.ok | wq! test.out | endif
:set exrc secure
:if has("gui_kde")




Best,
Christian
--
"Heinrich der Vierte", von Shakespeare. Wenn alles verloren
wäre, was je dieser Art geschrieben zu uns gekommen, so könnte man
Poesie und Rhetorik daraus vollkommen wiederherstellen.
-- Goethe, Maximen und Reflektionen, Nr. 580

[Florian Walch]

Hi Christian,

> Sounds very interesting. Is the build always triggered for pull requests
> with all different feature sets?

Yes; that's the way it is configured in .travis.yml. If your pull request modifies the .travis.yml, though, this configuration will be used (e.g. only build certain feature sets).

> If I open a new pull request the travis CI will immediatly start with
> the test suite, right?

Exactly. Github will more or less immediately show a link to the Travis build at the location where it says "The Travis CI build failed" in the example build I linked above ([3]). So if you e.g. were to include a patch that fixes the tiny/small test errors, it should build successfully.

Cheers!

[Florian Walch]

I created a pull request at [1] with your patch; looks like that fixed the build for tiny, but not for small [2].

Seems to be a different error: "test_command_count FAILED". I could now add a second patch file to the PR, force-push to the branch (triggering a new build) and check again if that fixes it.

[1] https://github.com/fwalch/vim-ci/pull/2
[2] https://travis-ci.org/fwalch/vim-ci/builds/47535926

[Florian Walch]

Actually, instead of sending pull requests to my repository, it might be easier to enable Travis [1] for your fork of the vim-ci repository and just push changes to a branch. This will also trigger a build, which will show up at https://travis-ci.org/<your-username>/vim-ci/branches. I just added a commit to the .travis.yml to support this (previously it only built the master branch).

[1] https://travis-ci.org/profile

[Christian Brabandt]

Am 2015-01-19 17:00, schrieb Florian Walch:
> Seems to be a different error: "test_command_count FAILED". I could
> now add a second patch file to the PR, force-push to the branch
> (triggering a new build) and check again if that fixes it.

That is a different error. Looks like test_command_count.in needs a :so
small.vim
instead of :so tiny.vim

I'll test that patch later today and can then also test, how this
Travis-CI works.

BTW: Where do you get the vim source from and how often do you update
the source?

Best,
Christian

[Florian Walch]

> BTW: Where do you get the vim source from and how often do you update
> the source?

It's downloaded to the Travis worker with Mercurial [1] and cached [2,3] so that subsequent jobs only need to get updates (if there are any). That is done for each job.

[1] https://github.com/fwalch/vim-ci/blob/master/.travis.yml#L22
[2] https://github.com/fwalch/vim-ci/blob/master/.travis.yml#L15
[3] http://docs.travis-ci.com/user/caching/#Caching-directories-%28Bundler%2C-dependencies%29

[Christian Brabandt]

On Mo, 19 Jan 2015, Christian Brabandt wrote:

> Am 2015-01-19 17:00, schrieb Florian Walch:
> >Seems to be a different error: "test_command_count FAILED". I could
> >now add a second patch file to the PR, force-push to the branch
> >(triggering a new build) and check again if that fixes it.
>
> That is a different error. Looks like test_command_count.in needs a
> :so small.vim
> instead of :so tiny.vim

Okay, this patch fixes both problems

Best,
Christian
--
Strahlt der Mond ganz voll und hell, wächst dem Knecht ein Werwolffell.

[mattn]

Hi.

Here is an another one.

https://github.com/vim-jp/vim-ci

This works fine on CI. All of patches are applied automatically, and tested. Currently, vim-jp/vim-ci is working on three CIs

travis-ci project - https://travis-ci.org/vim-jp/vim-ci
coveralls project - https://coveralls.io/r/vim-jp/vim-ci
AppVeyor project - https://ci.appveyor.com/project/k-takata/vim-ci

* Windows version is also tested on AppVeyor.

Thanks.

[Florian Walch]

Heh, I was half-expecting that, but since I couldn't find anything when I had the idea this morning, I just went ahead. Looks very nice, thanks for sharing!

Maybe it could be improved to support building arbitrary patches. Having some way to test patches on different systems before proposing them for merge into Vim could be useful, I think.

[Bram Moolenaar]

Christian Brabandt wrote:

> Hi Florian!
>
> Thanks for contributing, I thinks this is really helpful.
>
> On Mo, 19 Jan 2015, Florian Walch wrote:
>
> > Pulls latest Vim from its Mercurial sources, compiles it and runs the tests. See [3] for a build of current Vim (7.4.582), which I think already uncovered a problem: Running the tests fails with tiny and small features ("test16 NO OUTPUT"), but works with bigger feature sets.
>
> Interesting, I do not see this here locally, but when I run make
> test16.out I see an error. This patch fixed
> diff --git a/src/testdir/test16.in b/src/testdir/test16.in
> --- a/src/testdir/test16.in
> +++ b/src/testdir/test16.in
> @@ -2,6 +2,7 @@ Tests for resetting "secure" flag after
> For KDE set a font, empty 'guifont' may cause a hang.
>
> STARTTEST
> +:so small.vim
> :if $DISPLAY == "" | e! test.ok | wq! test.out | endif
> :set exrc secure
> :if has("gui_kde")

I didn't see this problem, which is actually surprising, considering the
use of the +eval feature. Maybe the test passes by accident.

--
I AM THANKFUL...
...for the taxes that I pay because it means that I am employed.

/// Bram Moolenaar -- Br...@Moolenaar.net -- http://www.Moolenaar.net \\\
/// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
\\\ an exciting new programming language -- http://www.Zimbu.org ///
\\\ help me help AIDS victims -- http://ICCF-Holland.org ///

[Bram Moolenaar]

Christian Brabandt wrote:

> On Mo, 19 Jan 2015, Christian Brabandt wrote:
>
> > Am 2015-01-19 17:00, schrieb Florian Walch:
> > >Seems to be a different error: "test_command_count FAILED". I could
> > >now add a second patch file to the PR, force-push to the branch
> > >(triggering a new build) and check again if that fixes it.
> >
> > That is a different error. Looks like test_command_count.in needs a
> > :so small.vim
> > instead of :so tiny.vim
>
> Okay, this patch fixes both problems

I suppose that's good to fix, but I had not noticed a problem.
What was the error when only sourcing tiny.vim?
Maybe my test script somehow didn't spot this problem.

--
I AM THANKFUL...
...for the clothes that fit a little too snug because it
means I have more than enough to eat.

[Christian Brabandt]

Am 2015-01-20 12:39, schrieb Bram Moolenaar:
> Christian Brabandt wrote:
>
>> On Mo, 19 Jan 2015, Christian Brabandt wrote:
>>
>> > Am 2015-01-19 17:00, schrieb Florian Walch:
>> > >Seems to be a different error: "test_command_count FAILED". I could
>> > >now add a second patch file to the PR, force-push to the branch
>> > >(triggering a new build) and check again if that fixes it.
>> >
>> > That is a different error. Looks like test_command_count.in needs a
>> > :so small.vim
>> > instead of :so tiny.vim
>>
>> Okay, this patch fixes both problems
>
> I suppose that's good to fix, but I had not noticed a problem.

According to the documentation and feature.h user commands are only
available starting with normal builds and not in the tiny or small
build.
So I figured, one should actually source small.vim instead of tiny.vim

> What was the error when only sourcing tiny.vim?

Logfile of the failing run is here:
https://s3.amazonaws.com/archive.travis-ci.org/jobs/47535928/log.txt

If you grep for test_command_count you'll see many:
E319: Sorry, the command is not available in this version
and then a final
test_command_count FAILED


Best,
Christian

[Bram Moolenaar]

I tried again and now my test script does fail...
Not sure what happened before.



--
I AM THANKFUL...
...for the piles of laundry and ironing because it means I
have plenty of clothes to wear.