issue with CRLF line endings

[John Cremona]

Has anyone else been seeing the following problem, which has been
plaguing me for a week or two. Here's a simple example. All
computers mentioned here are running ubuntu. On a machine I had not
used for a while I had a sage build of the develop branch at an old
version (pre 9.0). There were no modified files (git status showed
nothing). Then I did "git pull trac develop", after which one file is
marked as having changed:

modified: src/sage/misc/element_with_label.py

It is always this file, on several machines where I have gone through
similar steps. file shows this:
src/sage/misc/element_with_label.py: Python script, ASCII text
executable, with CRLF line terminators

git diff shows this:

$ git diff -w src/sage/misc/element_with_label.py
warning: CRLF will be replaced by LF in src/sage/misc/element_with_label.py.
The file will have its original line endings in your working directory.

I cannot get this to go away. git stash does nothing, and I cannot
change branches since git thinks there is an uncommitted change. I
have been reduced to deleting the entire build and starting again from
scratch.

No, I have not knowingly changed any configuration on these computers,
though I have done some normal package upgrades using apt.

I just went through the same steps in another directory on the same
machine where I had an old build (8.7) and after "git pull trac
develop" there are now 5 modified files:

modified:
build/pkgs/sage_sws2rst/src/test/Adding_Pictures_and_screenshots.sws
modified: src/doc/en/reference/combinat/media/k-rim.JPG
modified: src/sage/ext_data/doctest/rich_output/example.png
modified: src/sage/ext_data/notebook-ipython/logo-64x64.png
modified: src/sage/misc/element_with_label.py

-- the same one as before and 3 others which are not even normal text
files. I don't want to trash all these since I am trying to resurrect
and complete work done on some old trac tickets.

John

[Dima Pasechnik]

On Wed, Feb 10, 2021 at 2:58 PM John Cremona <john.c...@gmail.com> wrote:
>
> Has anyone else been seeing the following problem, which has been
> plaguing me for a week or two. Here's a simple example. All
> computers mentioned here are running ubuntu. On a machine I had not
> used for a while I had a sage build of the develop branch at an old
> version (pre 9.0). There were no modified files (git status showed
> nothing). Then I did "git pull trac develop", after which one file is
> marked as having changed:
>
> modified: src/sage/misc/element_with_label.py
>
> It is always this file, on several machines where I have gone through
> similar steps. file shows this:
> src/sage/misc/element_with_label.py: Python script, ASCII text
> executable, with CRLF line terminators

does

git config core.autolf

show 'true'?

Set it to false, IMHO this should fix this issue.

>
> git diff shows this:
>
> $ git diff -w src/sage/misc/element_with_label.py
> warning: CRLF will be replaced by LF in src/sage/misc/element_with_label.py.
> The file will have its original line endings in your working directory.
>
> I cannot get this to go away. git stash does nothing, and I cannot
> change branches since git thinks there is an uncommitted change. I
> have been reduced to deleting the entire build and starting again from
> scratch.
>
> No, I have not knowingly changed any configuration on these computers,
> though I have done some normal package upgrades using apt.
>
> I just went through the same steps in another directory on the same
> machine where I had an old build (8.7) and after "git pull trac
> develop" there are now 5 modified files:
>
> modified:
> build/pkgs/sage_sws2rst/src/test/Adding_Pictures_and_screenshots.sws
> modified: src/doc/en/reference/combinat/media/k-rim.JPG
> modified: src/sage/ext_data/doctest/rich_output/example.png
> modified: src/sage/ext_data/notebook-ipython/logo-64x64.png
> modified: src/sage/misc/element_with_label.py
>
> -- the same one as before and 3 others which are not even normal text
> files. I don't want to trash all these since I am trying to resurrect
> and complete work done on some old trac tickets.
>
> John
>

> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/CAD0p0K7KrVUvvkmqt6Nwhe1Tt9ef_w3ZWAqU5GK6Y6%3Dysqq3Uw%40mail.gmail.com.

[John Cremona]

On Wed, 10 Feb 2021 at 16:56, Dima Pasechnik <dim...@gmail.com> wrote:
>
> On Wed, Feb 10, 2021 at 2:58 PM John Cremona <john.c...@gmail.com> wrote:
> >
> > Has anyone else been seeing the following problem, which has been
> > plaguing me for a week or two. Here's a simple example. All
> > computers mentioned here are running ubuntu. On a machine I had not
> > used for a while I had a sage build of the develop branch at an old
> > version (pre 9.0). There were no modified files (git status showed
> > nothing). Then I did "git pull trac develop", after which one file is
> > marked as having changed:
> >
> > modified: src/sage/misc/element_with_label.py
> >
> > It is always this file, on several machines where I have gone through
> > similar steps. file shows this:
> > src/sage/misc/element_with_label.py: Python script, ASCII text
> > executable, with CRLF line terminators
>
> does
>
> git config core.autolf
>
> show 'true'?

No.

>
> Set it to false, IMHO this should fix this issue.

It is not set at all. I saw that option after googling for help, but
the issue it exists to solve is not one which has ever hit me (in sage
anyway, of course I know about using dos2unix sometimes when a windows
users sends a file).

So, I did not change anything (as far as I can tell -- except get
older) but my git state changed. These files stay changed even after
trying to reset them (though *I* did not change them) using git
checkout --.

John

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/CAAWYfq10WrFPb-2wQfLVyhSSRRGdWr_eEQ7gyanQ9Pbgm6o6qQ%40mail.gmail.com.

[Dima Pasechnik]

On Wed, Feb 10, 2021 at 5:49 PM John Cremona <john.c...@gmail.com> wrote:
>
> On Wed, 10 Feb 2021 at 16:56, Dima Pasechnik <dim...@gmail.com> wrote:
> >
> > On Wed, Feb 10, 2021 at 2:58 PM John Cremona <john.c...@gmail.com> wrote:
> > >
> > > Has anyone else been seeing the following problem, which has been
> > > plaguing me for a week or two. Here's a simple example. All
> > > computers mentioned here are running ubuntu. On a machine I had not
> > > used for a while I had a sage build of the develop branch at an old
> > > version (pre 9.0). There were no modified files (git status showed
> > > nothing). Then I did "git pull trac develop", after which one file is
> > > marked as having changed:
> > >
> > > modified: src/sage/misc/element_with_label.py
> > >
> > > It is always this file, on several machines where I have gone through
> > > similar steps. file shows this:
> > > src/sage/misc/element_with_label.py: Python script, ASCII text
> > > executable, with CRLF line terminators
> >
> > does
> >
> > git config core.autolf
> >
> > show 'true'?
>
> No.
>
> >
> > Set it to false, IMHO this should fix this issue.
>
> It is not set at all. I saw that option after googling for help, but
> the issue it exists to solve is not one which has ever hit me (in sage
> anyway, of course I know about using dos2unix sometimes when a windows
> users sends a file).

did you try setting this option to false, and see if it helps?

>
> So, I did not change anything (as far as I can tell -- except get
> older) but my git state changed. These files stay changed even after
> trying to reset them (though *I* did not change them) using git
> checkout --.

It's a new version of git that got you, I suppose.

We've had some "fun" with this lately, as there are Sage spkgs which
need patches for files
with CRLF endings, and so these patches had to be dealt with somehow.
It was not pleasant.
See https://trac.sagemath.org/ticket/30403

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/CAD0p0K41axYDsedDj4-Dc7TkANM7T3CmVyMmS%3DDHhEXQQZqn4A%40mail.gmail.com.

[John Cremona]

Not yet systematically. After setting it to false and doing nothing
else, git status still shows
src/sage/misc/element_with_label.py as modified, even after 'git
checkout --'. Trying to get back to sanity a different way, 'git
stash' now outputs

warning: CRLF will be replaced by LF in src/sage/misc/element_with_label.py.
The file will have its original line endings in your working directory.
warning: CRLF will be replaced by LF in src/sage/misc/element_with_label.py.
The file will have its original line endings in your working directory.

Saved working directory and index state WIP on develop: 8453ffb
Updated SageMath version to 9.3.beta7
HEAD is now at 8453ffb Updated SageMath version to 9.3.beta7

but git status still shows that file as modified. Now I cannot just
checkout a different existing branch (since there are apparently
modified files), but I can checkout a new branch (g = alias for git)

$ g co -b dud
M src/sage/misc/element_with_label.py
Switched to a new branch 'dud'

the delete the 'develop' branch (which was the same as upstream anyway):

$ g branch -d develop
Deleted branch develop (was 8453ffb).

and now recreate the develop branch

$ g remote update trac
(...)
$ g co -b develop trac/develop
M src/sage/misc/element_with_label.py
Branch develop set up to track remote branch develop from trac.
Switched to a new branch 'develop'

and we are back to where we started. All of the above was with
core.autolf set to false.

On the same machine, in a completely new clone made and built
yesterday, git status shows

$ g st
On branch develop
Your branch is up-to-date with 'origin/develop'.
Changes not staged for commit:
(use "git add <file>..." to update what will be committed)
(use "git checkout -- <file>..." to discard changes in working directory)

modified: src/sage/rings/invariants/__init__.py

no changes added to commit (use "git add" and/or "git commit -a")

So the same problem with a new file -- which only has 2 bytes in it!
AFter dos2unix-ing it (so it now has 1 byte) it is still showing up in
git status. AFter git checkout -- it goes back to having 2 bytes and
still shows up.

It seems to be impossible to do any sage development here. I am now
trying yet another git clone -- so the global config of core.autolf is
in force as the clone is made -- but then after the clone from
git://github.com/sagemath/sage.git that same file is shown as
modified. For some reason I have been making the cones from github
and then adding trac as a remote. As a variant I made a fresh clone
from trac -- but see exactly the same.

A similar problem was reported on stackexchange *over 9 years ago* by
a Mac users whose collaborators were all ubuntu users. There, it is
suggest to look at the .gitattributes file, and *in this fresh clone*
that file contains

# Force LF normalization
* text=auto eol=lf
# except for Windows batch files
*.{cmd,[cC][mM][dD]} text eol=crlf
*.{bat,[bB][aA][tT]} text eol=crlf
*.diff_bin binary

This is commit 5328570 which is 9.2, master branch, but the develop
branch has the same.

To me it seems (from reading all that stackexchange page) that this is
not just a problem caused by something in my local configuration, but
something which has happened to the repository itself, which needs to
be fixed. But I am no expert.

Just to add to the fun, on a second machine with the same ubuntu
version as the one above (16.04.7 LTS) and same git version 2.7.4, a
clone I made yesterday shows no problems at all, while and older clone
on which I had done development for a long time has some CRLF file
issues. The only differences in the global .gitconfig between the two
machines is the core.autolf which I just added to the first.

I do have some clones without these problems so can keep on working,
but I also have branches I wanted to do more work on in clones which
are unusable.

John

>
> >
> > So, I did not change anything (as far as I can tell -- except get
> > older) but my git state changed. These files stay changed even after
> > trying to reset them (though *I* did not change them) using git
> > checkout --.
>
> It's a new version of git that got you, I suppose.

Yes, that is likely, since I did do package updates across the board
before this started to happen. I am seeing it on machines running:
ubuntu 16.04.7 LTS and git 2.7.4
ubuntu 18.04.5 LTS and git 2.17.1
ubuntu 20.04.2 LTS and git 2.25.1

In no cases was the whole OS updated, just packages so possibly git,
but I don't know what git version any had before this.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/CAAWYfq19RqXJ1oNgiTo3hX0yV06dXtMEenu-yuXjoM5S9XLhfg%40mail.gmail.com.

[Frédéric Chapoton]

I have been hit by this kind of annoying issue some time ago. No idea what is happening.

There are only 3 offending files containing CRLF:

sage$ git grep -Il $'\r' src/

src/sage/combinat/crystals/star_crystal.py

src/sage/misc/element_with_label.py

src/sage/rings/invariants/__init__.py

We should probably change them to use the correct line break

[Frédéric Chapoton]

I have made https://trac.sagemath.org/ticket/31379 to get rid of the CRLF in these 3 files.

[John Cremona]

On Thu, 11 Feb 2021, 10:00 Frédéric Chapoton, <fchap...@gmail.com> wrote:

I have made https://trac.sagemath.org/ticket/31379 to get rid of the CRLF in these 3 files.

Thanks!

Can we do something to stop this happening again?

John

To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/0e364799-33a8-4623-8faa-c0d8a17f5ac4n%40googlegroups.com.

[Dima Pasechnik]

On Thu, Feb 11, 2021 at 10:43 AM John Cremona <john.c...@gmail.com> wrote:
>
>
>
> On Thu, 11 Feb 2021, 10:00 Frédéric Chapoton, <fchap...@gmail.com> wrote:
>>
>> I have made https://trac.sagemath.org/ticket/31379 to get rid of the CRLF in these 3 files.

I guess one can ensure that CRLF get automatically converted into LF
on git push to trac,
for all files which are not specifically declared as binary in .gitattributes

I believe that the trouble came from pulling from a github repo, as
github might have its own default ideas
about dealing with CRLF...

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/CAD0p0K5YysFPDs8Z%3DWGg7%2B7Vyz0kP-OUgPFMMbWVgybC2e8ynA%40mail.gmail.com.

[Dima Pasechnik]

It could also be that core.autocrlf is true by default for you.
(this is the option that does automatic convesion to LF from CRLF)
Try

git config core.autocrlf false

and see if it helps

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/CAD0p0K4PgJG0PsV1cL8agta4262O5PU%2B7QqeESdSot%3D7-x1dpw%40mail.gmail.com.

[seb....@gmail.com]

I had the exactly same issue a while ago and finally got rid of it after many tries. It even appeared after a fresh git clone. I had to unset all (not just core.autocrlf) global Git configurations except username and password, even push.default (which I do on all other Git repositories locally, now).

BTW: I suspect that the issue with the broken GitLab pipelines for Trac tickets that I mentioned in this sage-devel thread is related.

[John Cremona]

On Thu, 11 Feb 2021 at 11:24, Dima Pasechnik <dim...@gmail.com> wrote:
>
> It could also be that core.autocrlf is true by default for you.
> (this is the option that does automatic convesion to LF from CRLF)
> Try
>
> git config core.autocrlf false

I think it is false by default (someone on stackexchange actually
looked at git source code to determine this), at least on unix-like
systems. As far as I can tell the only people who might want it set
to anything else are windows users collaborating with non-windows
users in a repository.

ANyway, Frederic has fixed the offending files in #31379.

John

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/CAAWYfq0sk-rHs_v2b-%2BaWiVb2%2B4vOAPUN9wo-qpe5QJM8Njmaw%40mail.gmail.com.