[windows] BOOT compiler doesn't recognize windows line ending

[Qian Yun]

The error message is:

"The character whose number is 13 is not a Boot character"

ASCII char number 13 is '\r', which is part of windows line ending.

First, current BOOT compiler does not recognize this character.
Shall we treat it properly?

Second, how did this bug happen?

It is said that on Windows, the "git for windows" will by default
transform text file from linux line ending to windows line ending.

So, apart from the above issue, shall we add file ".gitattributes" with:

* text=auto eol=lf

Or something else, see `man gitattributes`.

- Qian

[Qian Yun]

Spent some time on this, the fix in BOOT compiler is not that
obvious, because of the complexity of escape char at the end
of line.

Shall we simply go with ".gitattributes"?

- Qian

[Waldek Hebisch]

On Sun, Mar 07, 2021 at 02:32:47PM +0800, Qian Yun wrote:
> Spent some time on this, the fix in BOOT compiler is not that
> obvious, because of the complexity of escape char at the end
> of line.
>
> Shall we simply go with ".gitattributes"?

First we need to decide what carriage return in variuos FriCAS
input files should do and check what it is doing now. In
particular, what carriage return in middle of line should do?

For Windows users we should allow carriage return at end of
line in .input files. If we implement this (I do not know
if this works now) we will have also handling for .spad
files. Then it is only natural to do it also for Boot
(and scanner code in 'boot' subdirectory is very similar
to scanner in 'interp' used for .input and .spad files).

Concerning carriage return in middle of line, we probably
should signal error in such case (or maybe treat is as
end of line for benefit of old Macs).

Setting ".gitattributes" is good idea, but will not
resolve issues with user files.

--
Waldek Hebisch

[Waldek Hebisch]

On Sun, Mar 07, 2021 at 01:04:42PM +0100, Waldek Hebisch wrote:
> On Sun, Mar 07, 2021 at 02:32:47PM +0800, Qian Yun wrote:
> > Spent some time on this, the fix in BOOT compiler is not that
> > obvious, because of the complexity of escape char at the end
> > of line.
> >
> > Shall we simply go with ".gitattributes"?
>
> First we need to decide what carriage return in variuos FriCAS
> input files should do and check what it is doing now.

A little experiments shows that FriCAS ignores carriage return at
and of line in .input files and signals error when carriage return
is in middle of line. Boot scanner currently signals error everywhere.
I will try to modify Boot scanner to match .input scanner.

--
Waldek Hebisch

[Qian Yun]

>
> Setting ".gitattributes" is good idea, but will not
> resolve issues with user files.
>

OK, is this setting good to commit?

````
* text=auto eol=lf
````

Docs from man page are:

text
Set to string value "auto"
When text is set to "auto", the path is marked for
automatic end-of-line conversion. If Git decides that the content is
text, its line
endings are converted to LF on checkin. When the file
has been committed with CRLF, no conversion is done.

eol
Set to string value "crlf"
This setting forces Git to normalize line endings for
this file on checkin and convert them to CRLF when the file is checked out.

[Waldek Hebisch]

.input scanner simply strips final carriage return in 'expand-tabs'
(in 'macros.lisp'). Boot scanner have no equivalent function,
but I added equivalent code (patch attached) to 'shoeread-line'
and with this code Boot accepts carriage return at end of line.

BTW: I have now enough reasons to rewrite our scanners, but
must wait after release.

--
Waldek Hebisch

[Waldek Hebisch]

On Sun, Mar 07, 2021 at 11:39:15PM +0800, Qian Yun wrote:
>
> >
> > Setting ".gitattributes" is good idea, but will not
> > resolve issues with user files.
> >
>
> OK, is this setting good to commit?
>
> ````
> * text=auto eol=lf
> ````

Yes, OK.

--
Waldek Hebisch

[Qian Yun]

The patch looks good to me, and the test on Windows CI
passed without problem.

- Qian