Hi,
We’ve got a repository containing a lot of text-files committed from windows systems without the svn:eol-style=native setting. Some developers now want to work on these from Linux.
I would therefore like to fix the eol-style. Writing a script on linux to find all the relevant files and run “propset svn:eol-style native” on them is easy enough. However I’ve noticed that if the change is committed from *linux*, then the commit not only changes the file properties, but also *every line-ending in the file*, ie looking at diffs between files before and after this change is now pointless. Committing exactly the same propset change from *windows* just sets the property without the whitespace problems; checking out on both systems then works as expected, and diffs between repository versions are sane.
However it’s much more convenient to write and run this script from linux. Is there any way from linux to persuade subversion to just set the eol-style property?
Thanks,
Simon
Good planets are hard to find - please think of the environment before you print this email. ____________________________________________________________________ CAUTION - This message may contain privileged and confidential information intended only for the use of the addressee named above. If you are not the intended recipient of this message you are hereby notified that any use, dissemination, distribution or reproduction of this message is prohibited. If you have received this message in error please notify Air New Zealand immediately. Any views expressed in this message are those of the individual sender and may not necessarily reflect the views of Air New Zealand. _____________________________________________________________________ For more information on the Air New Zealand Group, visit us online at http://www.airnewzealand.com _____________________________________________________________________
Without the "svn:eol-style" property any file content is considered as binary.
At that point, checkouted files on Windows and Linux have "CR-LF" eol.
If you want to run your script on Linux, you should run a "dos2unix" (or equivalent)
and add the property "svn:eol-style" on your files before commit.
Beware, that is just a guess, I have not tested myself !
On Windows, either use Cygwin for Bash, Perl, Ruby... or the win32 port of your scripting langage. Sure it exists !
By the way, I doubt "diff" is sane in any case... just check a "svn diff" from Linux for instance.
The "perfect" fix whould be to dump the repository content,
add "svn:eol-style" properties at file addition in dump and the load it into a new repository.
OK that is really too heavy...
Another advice: you should distribute a correct "autoprops" settings
to your dev team to avoid that issue at source
Best regards
Yves
----- "Simon Kitching" <Simon.K...@airnz.co.nz> a écrit :
> We’ve got a repository containing a lot of text-files committed from
> windows systems without the svn:eol-style=native setting. Some
> developers now want to work on these from Linux.
>
> I would therefore like to fix the eol-style. Writing a script on linux
> to find all the relevant files and run “propset svn:eol-style native”
> on them is easy enough. However I’ve noticed that if the change is
> committed from * linux *, then the commit not only changes the file
> properties, but also * every line-ending in the file *, ie looking at
> diffs between files before and after this change is now pointless.
> Committing exactly the same propset change from *windows* just sets
> the property without the whitespace problems; checking out on both
> systems then works as expected, and diffs between repository versions
> are sane.
>
> However it’s much more convenient to write and run this script from
> linux. Is there any way from linux to persuade subversion to just set
> the eol-style property?
--
Yves Martin
ASAIK this is not true. Subversion has a clever way to tell whether a file is text or binary by looking at the file itself. I believe that you can "trick" Subversion by setting svn:mime-type, although I'm not sure why someone would want to treat a binay file as text or vice-versa.
Giulio
Linedata Limited
Registered Office: 85 Gracechurch St., London, EC3V 0AA
Registered in England and Wales No 3475006 VAT Reg No 710 3140 03
Pretty much all files on my hard disk are binary ;)
Anyhow, the point is that there are no transformations going on between the
client and the repository by default (i.e. the file is treated as raw
binary). Only if you configure lineendings, SVN will look at the content and
validate/transform the lines accordingly.
Uli
--
ML: http://subversion.tigris.org/mailing-list-guidelines.html
FAQ: http://subversion.tigris.org/faq.html
Docs: http://svnbook.red-bean.com/
Sator Laser GmbH, Fangdieckstraße 75a, 22547 Hamburg, Deutschland
Geschäftsführer: Thorsten Föcking, Amtsgericht Hamburg HR B62 932
**************************************************************************************
Sator Laser GmbH, Fangdieckstraße 75a, 22547 Hamburg, Deutschland
Geschäftsführer: Thorsten Föcking, Amtsgericht Hamburg HR B62 932
**************************************************************************************
Visit our website at <http://www.satorlaser.de/>
**************************************************************************************
Diese E-Mail einschließlich sämtlicher Anhänge ist nur für den Adressaten bestimmt und kann vertrauliche Informationen enthalten. Bitte benachrichtigen Sie den Absender umgehend, falls Sie nicht der beabsichtigte Empfänger sein sollten. Die E-Mail ist in diesem Fall zu löschen und darf weder gelesen, weitergeleitet, veröffentlicht oder anderweitig benutzt werden.
E-Mails können durch Dritte gelesen werden und Viren sowie nichtautorisierte Änderungen enthalten. Sator Laser GmbH ist für diese Folgen nicht verantwortlich.
**************************************************************************************
Subversion supports the use of "pre-commit hooks", that can auto-set
such properties based on the file name. I usually personally consider
it a bad idea: changing EOL behind my back as I switch from system to
system is nasty, especially when my files are on a shared directory
that may be used on both Windows and UNIX style systems.
It's often most effective to have a project or site policy. "No
Windows-EOL files" works well for me, and the dos2unix and unix2dos
files available on UNIX systems and withing CygWin for Windows work
well. Mind you, it gets even more fun for MacOS....
Errr... Shared directories are the bad idea here. Svn is perfectly capable of
managing a copy for everyone and text files are not portable.
--
Les Mikesell
lesmi...@gmail.com
Les Mikesell wrote:
> Errr... Shared directories are the bad idea here. Svn is
> perfectly capable of managing a copy for everyone and text
> files are not portable.
I don't necessarily disagree about shared directories being
a bad idea ... but the idea that text files are not portable
is slowly getting out of date. I suspect some old *nix
utilities may still suffer from some vulnerability here but
a lot (most?) newer (Windows) software is much more tolerant.
If, as in Nico's case, you stick with LF (no Windows EOL)
then your files will continue to work with the unnecessarily
fussy *nix tools and also with most modern Windows software.
--
Geoff Worboys
Telesis Computing
There are workarounds but you can't just expect things to work in general and
pretend it is normal. And some of those workarounds may involve silently
changing to the preferred local line endings - and if anything but subversion
itself does that it becomes a change in every line when committed, making diffs
useless.
--
Les Mikesell
lesmi...@gmail.com
This may be true for some *nix systems/tools. But...
For most modern (Windows) development software it is not a
matter of "workarounds". They simply process the files and
accept whatever line endings they find. Good editors will
automatically switch between CRLF, LF and CR EOL modes
according to what they find. They don't change the file
(unless asked) they change their own mode operation.
If your developers all work with LF then there will be no
confusion and no need for workarounds. If anyone makes a
mistake and accidently commits a file with CRLF it is an
easy fix to resolve ... although most Windows software wont
care even if you don't bother to fix it.
If you have to worry about mistakes, you are doing workarounds. What
windows editor will create new files with LF endings by default?
--
Les Mikesell
lesmi...@gmail.com
> (Forcing users to deal with the reply-to stuff on this list
> is something of a pain don't you think. As if I need the list
> copy and a CC copy. Never mind...)
No, we don't think so.
http://subversion.apache.org/docs/community-guide/mailing-lists.html#replying
Never-ending debate/troll...
The best option: CR-LF on Windows and LF on Linux and autoprops for
everyone !
To illustrate, worst examples I have seen:
* Cygwin installed with "LF" text file support
to run "cvs" command line. Result: appending a new partial
end-of-line at each commit - not fun !
* Unix scripts edited on Windows without eol-style.
Result: on Linux, the shebang (#!) is invalid and the interpreter
command is invalid: /bin/bash^M not found !
Best regards
Yves
Notepad++ can be configured to do so:
http://notepad-plus.sourceforge.net/
as can Programmer's Notepad:
http://www.pnotepad.org/
both based on Scintilla so there are probably more out there.
Yves Martin wrote:
> Never-ending debate/troll...
My apologies if my raising of this was seen as trolling. I
was not attempting too suggest which was the best for all
situations, simply that Nico's solution is viable for many
projects these days.
Obviously if you are going to continue to use old *nix based
scripts that remain vulnerable after all these years then
of course you must work-around such limitations, for which
the features of svn are very convenient.
Emacs and VIM.
You asked.....
Ryan Schmidt wrote:
> No, we don't think so.
> http://subversion.apache.org/docs/community-guide/mailing-lists.html#replying
Thanks for pointing that out. Having been a member of many
lists from the days of compuserve on I have tended to get a bit
lax in my reading for the forum rules ... they all tend to be
the same. That'll teach me, here is one that works differently
to anything else I have been on for many years.
If that's the way you want to work that's the way I'll work.
> My apologies if my raising of this was seen as trolling. I
> was not attempting too suggest which was the best for all
> situations, simply that Nico's solution is viable for many
> projects these days.
>
> Obviously if you are going to continue to use old *nix based
> scripts that remain vulnerable after all these years then
> of course you must work-around such limitations, for which
> the features of svn are very convenient.
Well, that is not "limitation" but a de-facto situation.
On Unix, a script with CR-LF is really unlikely to work, either with Bash, Perl or others...
The same way a .bat script on Windows fails if LF.
For me, this SVN feature is not for "work-around" but to
support multi-platform development "the right way".
http://svnbook.red-bean.com/nightly/en/svn.advanced.props.file-portability.html
And what about text file encoding ??
That is so easy to find two Linux boxes in UTF-8 and latin-1 encodings even in the same company in Europe,
it is already a nightmare.
We really have to "work-around" non-ANSI-compliant developers languages !
As Subversion does not provide a proper encoding support, developers must be fixed
so they do not write comments in non-ANSI characters with their non-US keyboards ;)
In fact, I could not wait and I use pre-commit hooks to validate text content
compliance with project policy concerning charset/encodings.
So the end-of-line problem is really simple compared to encodings.
And with developpers all around the world working on the same codebase,
it is difficult to check/enforce everyone use the "right tool"
with the proper configuration on any platform.
--
Yves Martin
I will take your word for this, but this:
> The same way a .bat script on Windows fails if LF.
is no longer true (XP+, not tested earlier) - or at least not
100% true, there may be situations I have not come across that
still cause issues. (This could inspire a comment about
Windows attempting to learn from it's past mistakes so why
isn't *nix, but people may get the impression that I'm a
Microsoft fan and that is not true either.)
[...]
> So the end-of-line problem is really simple compared to
> encodings. And with developpers all around the world working
> on the same codebase, it is difficult to check/enforce
> everyone use the "right tool" with the proper configuration
> on any platform.
Everyone's projects have their own problems. I am not about
to tell anyone that using only LF is always safe for all
projects, my only contention was that it can be safe for some,
that EOL issues are not what they once were (at least not on
Windows).
What IDE are you using? If you're not using an IDE, what program
editor are you using. Please, tell me, for the sake of everything we
hold dear, that your developers aren't using Notepad for editing their
files. Please tell me they're not.
Your developers on your Windows side need a good program editor. A
good program editor handles various line endings with aplomb. That
means it doesn't matter if a file has either Windows or Unix style
line endings. The text editor will handle that and there is no need to
convert from one to the other. In fact, your Linux developers should
also be working with a similar editor.
I like VIM. It is powerful, flexible, fast and is available on both
Linux and Windows. Many people don't like VIM because it can take a
while to learn to use. You can try TextPad on Windows, but it is not a
free program. If you make this the official program on your site,
please pay the licensing fee! SciTE is also nice and it is open source
and free.
Many developers simply use Eclipse which provides a full IDE for both
Windows and Linux.
If you go the pre-commit hook root, you'll need to checkout all files,
change the line endings, add the pre-commit hook, and commit the
files. This will change the line endings stored in the Subversion
repository. And unfortunately, the "svn diff" command cannot be easily
set to ignore whitespace or the change in line endings.
However, you can specify your own diff command when using "svn diff"
either through the configuration or via the --diff-cmd parameter. And,
you could set that diff command to ignore line ending differences or
white spaces. You can use a batch or shell script to make calling that
program easier. I use vimdiff when I compare only one or two files.
This gives me a visual diff, and I call it via:
$ svn diff --diff-cmd mydiff <file>
Here's what mydiff looks like:
#! /usr/bin/env perl
use strict;
use warnings;
my $DIFF = "/usr/bin/vimdiff";
my $parameters = $#ARGV;
my $file1 = $ARGV[$parameters - 1];
my $file2 = $ARGV[$parameters];
my $title1 = $ARGV[$parameters - 4];
my $title2 = $ARGV[$parameters - 2];
my $title = "$title1 - $title2";
$title =~ s/\t/ /g;
$title =~ s/ /\\ /g;
my $command = qq($DIFF -c "set titlestring=$title" "$file1" "$file2");
system qq($command);
--
David Weintraub
qaz...@gmail.com
Maybe you could write your script to check for the need for the change
to run on linux but instead of setting the property there, output a
batch file of the commands to be executed, transfer the file to windows
box (perhaps by committing, and checking back out there with the correct
eol's), and then run the commands from the windows side.
--
Les Mikesell
lesmi...@gmail.com
Look, *BREAK* the history. History is overvalued: Make a clean tag
with your final pre-switchover release, with a note explaining what
happened, and make an entirely new repository or branch with entirely
imported code. It will be much cleaner to track and follow what
happened without trying to back-revise history of mixed EOL
configurations.
I have to disagree very much with this. The ability to easily see what
changed between any two points is most of the value of using version
control systems. Sometimes you won't know why the old way was better
until long after you've forgotten the details of the changes - or the
person who made them may no longer be around.
--
Les Mikesell
lesmi...@gmail.com
I'm thinking aloud here, you may not be able to do it, but if you can add
props. as a direct repo. mod., them maybe it'd be easier for you to:
- create a temp working branch off trunk
- use a 'find' loop to find the files, but instead of 'svn ps svn:eol-style
{}' use 'http:/part/to/repo/'{} [if that will work with find!] with a '-m'
comment.
- merge the whole bally lot back to trunk and delete the branch
Yes, you'll add a gazillion revs. via the branch, but the mod. will
appear as atomic on trunk.
--
[neil@fnx ~]# rm -f .signature
[neil@fnx ~]# ls -l .signature
ls: .signature: No such file or directory
[neil@fnx ~]# exit
> On Wed, 2010-07-14 at 10:35 -0400, David Weintraub wrote:
>> I have a pre-commit hook that will verify that particular files have
>> svn:eol-style set before allowing a commit. This way, you can make
>> sure that the property is set before anyone can do a commit.
> Yes, that might be the best way to prevent the problem reoccurring in
> the future. Unfortunately, it doesn't help with fixing a pre-existing
> repository (ie setting eol:style=native on existing files that really
> should have had it set when they were created but didn't).
Here's a solution you haven't considered: Force all text files to have
"svn:eol-style" set to "CRLF" instead of "native".
I've found that Windows tends to be the system that messes up line
endings. Many developers will open up Notepad or Wordpad which will
muck up line endings if the file doesn't already have Windows style
line endings. However, Linux/Unix users tend to use tools like
Eclipse, VIM, or Emacs which will not mess with the line endings of
files that it is editing. If a file has "CRLF", these editors wont'
change it.
These Unix/Linux tools can also be configured to default to Windows'
CRLF on new files too.
Then, you simply need a hook to enforce the use of Windows line
endings on all text files (except maybe Unix specific files like Shell
scripts and Makefiles which fail if they have CRLF line endings).
--
David Weintraub
qaz...@gmail.com
> I'm thinking aloud here, you may not be able to do it, but if you can add props. as a direct repo. mod., them maybe it'd be easier for you to:
>
> - create a temp working branch off trunk
> - use a 'find' loop to find the files, but instead of 'svn ps svn:eol-style {}' use 'http:/part/to/repo/'{} [if that will work with find!] with a '-m' comment.
> - merge the whole bally lot back to trunk and delete the branch
>
>
> Yes, you'll add a gazillion revs. via the branch, but the mod. will appear as atomic on trunk.
Or use svnmucc to do it in a single revision in the first place.
The EOL problem makes this a seriously broken problem. The wreckage of
the old code is in a clean location, where the changes can be tracked
too, but there is *NO WAY* to "easily see what changed" between files
with unpredictably changing EOL's. With a clean demarcation, you have
a well denoted break point to mark where you'll have to change
procedures to get your diffs.