tinytests can fail if .ok fileformat is wong (Make_ming.mak)

[Michael Soyka]

Vim developers,

Using Make_ming.mak, the "tinytests" collection of build tests will fail if the associated .ok file has Windows line endings.  This happens because the test.out file is explicitly set to have unix/linux line endings but the .ok file has not been converted and so "diff" concludes they're different.  I have attached a patch which converts the .ok file and cleans-up. 

This was introduced by patch v8.2.1439 which removed the "fixff" target.  I noticed it because I use git-for-windows and its default is to use Windows line endings in the workspace. 

I suspect the same problem occurs in Make_dos.mak but I have no way of verifying that.

-mike

[Bram Moolenaar]

I had been wondering if "fixff" would still be needed. The very old
reason was that after unpacking the source tar archive some versions of
tar would change the line endings. So git-for-windows does the same
thing? Or should we somehow set the git metadata so that it doesn't
change the line endings?

--
Communication is one of the most compli..., eh, well, it's hard.
You know what I mean. Not?

/// Bram Moolenaar -- Br...@Moolenaar.net -- http://www.Moolenaar.net \\\
/// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
\\\ an exciting new programming language -- http://www.Zimbu.org ///
\\\ help me help AIDS victims -- http://ICCF-Holland.org ///

[M. Soyka]

On Wed, Sep 16, 2020 at 9:42 AM Bram Moolenaar <Br...@moolenaar.net> wrote:

Michael Soyka wrote:

> Using Make_ming.mak, the "tinytests" collection of build tests will fail if
> the associated .ok file has Windows line endings.  This happens because the
> test.out file is explicitly set to have unix/linux line endings but the .ok
> file has not been converted and so "diff" concludes they're different.  I
> have attached a patch which converts the .ok file and cleans-up. 
>
> This was introduced by patch v8.2.1439 which removed the "fixff" target.  I
> noticed it because I use git-for-windows and its default is to use Windows
> line endings in the workspace. 
>
> I suspect the same problem occurs in Make_dos.mak but I have no way of
> verifying that.

I had been wondering if "fixff" would still be needed.  The very old
reason was that after unpacking the source tar archive some versions of
tar would change the line endings.  So git-for-windows does the same
thing?  Or should we somehow set the git metadata so that it doesn't
change the line endings?

Assuming there is a preferred line ending for the tinytests, how can you then force it on the end-user?  For git, I think the .gitattributes file is the way forward but what about Mercurial, tar-files, and whatever other distribution methods you support?  It is also unclear to me what determines the fileformat of the test.out files created by the tests.  For diff to work, it needs to be the same as the .ok file. 

For these reasons, I think that your only choice is to control it in the Makefiles. 

As to the preferred line ending, I suspect it should be LF but I don't know if that is appropriate for non-Windows and non-linux systems.

[Michael Soyka]

Bram,

I've done some additional investigations and have concluded that the "dostmp" scratch directory can be eliminated from testdir/Make_ming.mak.  I have attached two patches that do just that.  The second patch is a merge of the 7 steps/commits I used to arrive at this.   If anyone is interested in the individual commits I can provide them.

Unfortunately, I don't see how one can reliably eliminate the fileformat conversion of the test.out file.  In my git workspace, the tinytests line endings are as follows:

    i/-text w/-text attr/                   dotest.in
    i/lf    w/crlf  attr/                   test20.in
    i/lf    w/crlf  attr/                   test20.ok
    i/mixed w/mixed attr/                   test21.in
    i/mixed w/mixed attr/                   test21.ok
    i/lf    w/crlf  attr/                   test22.in
    i/lf    w/crlf  attr/                   test22.ok
    i/-text w/-text attr/                   test23.in
    i/lf    w/crlf  attr/                   test23.ok
    i/-text w/-text attr/                   test24.in
    i/lf    w/crlf  attr/                   test24.ok
    i/-text w/-text attr/                   test25.in
    i/-text w/-text attr/                   test25.ok
    i/lf    w/crlf  attr/                   test26.in
    i/lf    w/crlf  attr/                   test26.ok
    i/-text w/-text attr/                   test27.in
    i/-text w/-text attr/                   test27.ok
 

Files that have unix line endings in the index (leftmost column) have been converted to dos line endings in the workspace (2'nd column).  The remaining files git views as binary or as having several line endings and does not change them but Vim treats them as fileformat unix.  I think, correct me if I'm mistaken, the fileformat of the test.out files is determined by that of the .in file (set to dos by dotest.in) but that may differ from the line endings used in the .ok file.  If they are different, diff will report differences.  

Sorry but I think this is the best I can do but stand ready to be corrected. 

-mike

 

 

[Bram Moolenaar]

Michael Soyka wrote:

> > > Using Make_ming.mak, the "tinytests" collection of build tests will fail if
> > > the associated .ok file has Windows line endings. This happens because the
> > > test.out file is explicitly set to have unix/linux line endings but the .ok
> > > file has not been converted and so "diff" concludes they're different. I
> > > have attached a patch which converts the .ok file and cleans-up.
> > >
> > > This was introduced by patch v8.2.1439 which removed the "fixff" target. I
> > > noticed it because I use git-for-windows and its default is to use Windows
> > > line endings in the workspace.
> > >
> > > I suspect the same problem occurs in Make_dos.mak but I have no way of
> > > verifying that.
> >
> > I had been wondering if "fixff" would still be needed. The very old
> > reason was that after unpacking the source tar archive some versions of
> > tar would change the line endings. So git-for-windows does the same
> > thing? Or should we somehow set the git metadata so that it doesn't
> > change the line endings?
>

This has changed a few times in the early days, and settled down on
something that worked. I suspect going back to that situation might be
best, even when this involves some scripting to do conversions.

Part of the problem is that originally files were distributed for Unix
and for DOS/Windows separately, with the latter in CR/LF format so they
could be handled by other tools. But some users unpacked the Unix files
on Windows and expect them to work.

Now with version control I don't really know what you end up with.
Converting from LF to CR/LF is then the simplest, and the tests rely on
the .in and .ok files to use CF/LF.

Doing this the other way around is tricky and is not reliable.
Making the tests work with either format is also tricky, since they are
Normal mode commands.

If you think your changes work, please create a pull request, so it gets
tested with different setups by CI.

--
LARGE MAN: Who's that then?
CART DRIVER: (Grudgingly) I dunno, Must be a king.
LARGE MAN: Why?
CART DRIVER: He hasn't got shit all over him.
"Monty Python and the Holy Grail" PYTHON (MONTY) PICTURES LTD