I received a text file with tab-separated values. However, the
entire file is displayed in one line and vim shows ^M symbols
all over the text, which are U+000d characters (CR).
I ran a search-replace to replace them with <CTRL-V><CR> and the
file displays as it should. But what's the cause? Is the file
malformed? I thought CR is the usual way of encoding a line ending.
What do I need to change to make vim display the file correctly
without changing it?
What system might have saved the file? Usually I never have any
problems opening files from Windows or Mac.
Maybe I should add that the file displays fine in other editors
(tested in nano). I used my usual .vimrc with plenty of plugins and
customization, as well as
vim -u NONE -U NONE -i NONE
to exclude any weird configuration, with the same result.
On Fri, Nov 16, 2012 at 10:18 AM, Marco <net...@lavabit.com> wrote:
> Hi!
> I received a text file with tab-separated values. However, the
> entire file is displayed in one line and vim shows ^M symbols
> all over the text, which are U+000d characters (CR).
> I ran a search-replace to replace them with <CTRL-V><CR> and the
> file displays as it should. But what's the cause? Is the file
> malformed? I thought CR is the usual way of encoding a line ending.
> What do I need to change to make vim display the file correctly
> without changing it?
> What system might have saved the file? Usually I never have any
> problems opening files from Windows or Mac.
> VIM - Vi IMproved 7.3 (2010 Aug 15, compiled Nov 12 2012 19:41:47)
> Included patches: 1-712
> on Debian Linux
Different operating systems use different line endings for text files. From
:help 'ff':
dos <CR><NL>
unix <NL>
mac <CR>
What OS are you opening the file in? Next time you open the offending file,
try ":set ff?" (without the quotes) to see what it says. The <CR> way is
"mac".
My Windows GVim opened the file exactly as you described: one long line.
However, by using ":e ++ff=mac" (again, no quotes) I was able to reload the
file correctly.
> I received a text file with tab-separated values. However, the
> entire file is displayed in one line and vim shows ^M symbols
> all over the text, which are U+000d characters (CR).
> I ran a search-replace to replace them with <CTRL-V><CR> and the
> file displays as it should. But what's the cause? Is the file
> malformed? I thought CR is the usual way of encoding a line ending.
> What do I need to change to make vim display the file correctly
> without changing it?
> What system might have saved the file? Usually I never have any
> problems opening files from Windows or Mac.
> Different operating systems use different line endings for text files. From
> :help 'ff':
> dos <CR><NL>
> unix <NL>
> mac <CR>
> What OS are you opening the file in?
> > on Debian Linux
> Next time you open the offending file,
> try ":set ff?" (without the quotes) to see what it says.
unix
> My Windows GVim opened the file exactly as you described: one long line.
> However, by using ":e ++ff=mac" (again, no quotes) I was able to reload the
> file correctly.
Thanks a lot, that works. Can I automate this somehow, so that vim
opens <CR> (mac) files automatically with the ff=mac setting?
On Friday, November 16, 2012 9:55:10 AM UTC-6, Marco wrote:
> > My Windows GVim opened the file exactly as you described: one long line. > > However, by using ":e ++ff=mac" (again, no quotes) I was able to reload the > > file correctly.
> Thanks a lot, that works. Can I automate this somehow, so that vim > opens <CR> (mac) files automatically with the ff=mac setting?
> On Friday, November 16, 2012 9:55:10 AM UTC-6, Marco wrote:
> > > My Windows GVim opened the file exactly as you described: one long line.
> > > However, by using ":e ++ff=mac" (again, no quotes) I was able to reload the
> > > file correctly.
> > Thanks a lot, that works. Can I automate this somehow, so that vim
> > opens <CR> (mac) files automatically with the ff=mac setting?
On Fri, Nov 16, 2012 at 11:00 AM, Ben Fritz <fritzophre...@gmail.com> wrote:
> On Friday, November 16, 2012 9:55:10 AM UTC-6, Marco wrote:
> > Thanks a lot, that works. Can I automate this somehow, so that vim
> > opens <CR> (mac) files automatically with the ff=mac setting?
> :help 'fileformats' (note the s at the end).
To elaborate on what Ben said, you could just make sure your 'ffs' also has
"mac" in there and this problem should go away. I have this function that I
fire off every time I edit a file. (The autocommand is right after the
endfunction.) Occasionally, I will get a file that was edited by both
developers on DOS and Unix systems and their editors will quietly leave
mixed newlines in there. I believe that Vim will open a file with mixed
newlines as Unix (at least, this is what I see).
Basically, this counts the number of ^M characters and prints a message
reporting the number of ^M characters (if any) and, if more than half the
file contains them, suggests that the file might be DOS--otherwise, Unix.
It also returns Vim script commands to fix the file format, but the
autocommand doesn't use that. I put an arbitrary line limit of 10,000 lines
so that the file opening process isn't slowed down too much by this check.
Admittedly, it's going to be have to refactored for Mac; also, that
'silent! execute' line contains a ^M (literal control-M) where there's a
newline in the paste below.
function! NewFileReport()
if ( &binary )
return
endif
let result = ''
let newFormat = ''
if ( line( '$' ) > 10000 )
let result = printf( 'File too large to check for newlines (%d
lines).', line( '$' ) )
else
let num = 0
let savedSearch = @/
let pos = winsaveview()
silent! execute "g/
$/let num+= 1"
let @/ = savedSearch
call winrestview( pos )
if ( num > 0 )
let result = printf( 'Number of DOS newlines: %d (out of %d total).',
num, line( '$' ) )
if ( &fileformat != 'unix' )
let result .= ' File is ' . toupper( &fileformat ) . '.'
else
let newFormat = num > ( line( '$' ) / 2 ) ? 'DOS' : 'Unix'
let result .= ' File is probably ' . newFormat . '.'
endif
endif
endif
let result .= &eol ? '' : ' No end-of-line.'
if ( result != '' )
redraw
echo result
endif
let result = ''
if ( newFormat != '' )
let result = 'set ff=' . tolower( newFormat )
endif
if ( !&eol )
let result .= "\<nl>set eol"
endif
return result
endfunction
au BufReadPost * call NewFileReport()
On Friday, November 16, 2012 10:11:34 AM UTC-6, Salman Halim wrote:
> Occasionally, I will get a file that was edited by both developers on DOS and Unix systems and their editors will quietly leave mixed newlines in there. I believe that Vim will open a file with mixed newlines as Unix (at least, this is what I see).
> Basically, this counts the number of ^M characters and prints a message reporting the number of ^M characters (if any) and, if more than half the file contains them, suggests that the file might be DOS--otherwise, Unix. It also returns Vim script commands to fix the file format, but the autocommand doesn't use that. I put an arbitrary line limit of 10,000 lines so that the file opening process isn't slowed down too much by this check.
> Admittedly, it's going to be have to refactored for Mac; also, that 'silent! execute' line contains a ^M (literal control-M) where there's a newline in the paste below.
Here's a script to do the automatic reload for you (between DOS and Unix still unfortunately):
> On Friday, November 16, 2012 5:09:19 PM UTC+1, Marco wrote:
>> 2012-11-16 Ben Fritz:
>>> On Friday, November 16, 2012 9:55:10 AM UTC-6, Marco wrote:
>>>>> My Windows GVim opened the file exactly as you described: one long line.
>>>>> However, by using ":e ++ff=mac" (again, no quotes) I was able to reload the
>>>>> file correctly.
>>>> Thanks a lot, that works. Can I automate this somehow, so that vim
>>>> opens <CR> (mac) files automatically with the ff=mac setting?
>>> :help 'fileformats' (note the s at the end).
>> Thanks
>> set fileformats=unix,dos,mac
>> did the trick.
> Is there any reason why this is not the default on unix?
> Jeroen
I'm not sure, but on Unix (well, on Linux) I have occasionally met files which had a lot of lone carriage-returns and yet weren't Mac files. This happens for instance when logging the stdout of a console program which displays a text-mode "progess bar" by using a CR to go to the left margin without advancing to the next line, in order to overwrite the line just written. ISTR that rsync used to do that (when I used it to keep my Vim source in sync before there was a Mercurial repository), and maybe Mercurial (with the "progress" extension), or the command-line "ftp" utility, do too. In that case you don't want to break the line at a lone CR but you may want to delete everything that precedes a CR which is not at the end of a line (CR at the end of a line, i.e. followed by a line-feed character, can be taken care of by reading the file with ++ff=dos).
Best regards,
Tony.
-- Bees are very busy souls
They have no time for birth controls
And that is why in times like these
There are so many Sons of Bees.
> On 17/11/12 09:00, jeroen wrote:
> >On Friday, November 16, 2012 5:09:19 PM UTC+1, Marco wrote:
> >>2012-11-16 Ben Fritz:
> >>>On Friday, November 16, 2012 9:55:10 AM UTC-6, Marco wrote:
> >>>>>My Windows GVim opened the file exactly as you described: one long line.
> >>>>>However, by using ":e ++ff=mac" (again, no quotes) I was able to reload the
> >>>>>file correctly.
> >>>>Thanks a lot, that works. Can I automate this somehow, so that vim
> >>>>opens <CR> (mac) files automatically with the ff=mac setting?
> >>>:help 'fileformats' (note the s at the end).
> >>Thanks
> >> set fileformats=unix,dos,mac
> >>did the trick.
> >Is there any reason why this is not the default on unix?
> >Jeroen
> I'm not sure, but on Unix (well, on Linux) I have occasionally met
> files which had a lot of lone carriage-returns and yet weren't Mac
> files. This happens for instance when logging the stdout of a
> console program which displays a text-mode "progess bar" by using a
> CR to go to the left margin without advancing to the next line, in
> order to overwrite the line just written. ISTR that rsync used to do
> that (when I used it to keep my Vim source in sync before there was
> a Mercurial repository), and maybe Mercurial (with the "progress"
> extension), or the command-line "ftp" utility, do too. In that case
> you don't want to break the line at a lone CR but you may want to
> delete everything that precedes a CR which is not at the end of a
> line (CR at the end of a line, i.e. followed by a line-feed
> character, can be taken care of by reading the file with ++ff=dos).
Sure enough, but that wouldn't have triggered Vim to set the fileformat to mac. As long there are some newlines in there, you would be safe.
>> On 17/11/12 09:00, jeroen wrote:
>>> On Friday, November 16, 2012 5:09:19 PM UTC+1, Marco wrote:
>>>> 2012-11-16 Ben Fritz:
>>>>> On Friday, November 16, 2012 9:55:10 AM UTC-6, Marco wrote:
>>>>>>> My Windows GVim opened the file exactly as you described: one long line.
>>>>>>> However, by using ":e ++ff=mac" (again, no quotes) I was able to reload the
>>>>>>> file correctly.
>>>>>> Thanks a lot, that works. Can I automate this somehow, so that vim
>>>>>> opens <CR> (mac) files automatically with the ff=mac setting?
>>>>> :help 'fileformats' (note the s at the end).
>>>> Thanks
>>>> set fileformats=unix,dos,mac
>>>> did the trick.
>>> Is there any reason why this is not the default on unix?
>>> Jeroen
>> I'm not sure, but on Unix (well, on Linux) I have occasionally met
>> files which had a lot of lone carriage-returns and yet weren't Mac
>> files. This happens for instance when logging the stdout of a
>> console program which displays a text-mode "progess bar" by using a
>> CR to go to the left margin without advancing to the next line, in
>> order to overwrite the line just written. ISTR that rsync used to do
>> that (when I used it to keep my Vim source in sync before there was
>> a Mercurial repository), and maybe Mercurial (with the "progress"
>> extension), or the command-line "ftp" utility, do too. In that case
>> you don't want to break the line at a lone CR but you may want to
>> delete everything that precedes a CR which is not at the end of a
>> line (CR at the end of a line, i.e. followed by a line-feed
>> character, can be taken care of by reading the file with ++ff=dos).
> Sure enough, but that wouldn't have triggered Vim to set the fileformat
> to mac. As long there are some newlines in there, you would be safe.
> regards,
> Christian
Quoting options.txt lines 2905 sqq under 'fileformats':
> This means that "mac" is only chosen when:
> "unix" is not present or no <NL> is found in the file, and
> "dos" is not present or no <CR><NL> is found in the file.
> Except: if "unix" was chosen, but there is a <CR> before
> the first <NL>, and there appear to be more <CR>s than <NL>s in
> the first few lines, "mac" is used.
Depending on how the logging is done, in the case I mentioned there could very well be quite a lot of <CR>s before the first <NL>, and, let's say, something like fifty <CR>s before the fifth <NL>.
Best regards,
Tony.
-- UNIX was half a billion (500000000) seconds old on
Tue Nov 5 00:53:20 1985 GMT (measuring since the time(2) epoch).
-- Andy Tannenbaum
> Depending on how the logging is done, in the case I mentioned there
> could very well be quite a lot of <CR>s before the first <NL>, and,
> let's say, something like fifty <CR>s before the fifth <NL>.
It is more likely, the file is really in MAC format rather than some obscure logfile.
regards,
Christian
-- Es gibt problematische Naturen, die keiner Lage gewachsen sind, in der sie sich befinden, und denen keine genugtut. Daraus entsteht der ungeheure Widerstreit, der das Leben ohne Genuss verzehrt.
-- Goethe, Maximen und Reflektionen, Nr. 278
On Sat, Nov 17, 2012 at 11:39 AM, Christian Brabandt <cbli...@256bit.org> wrote:
> Hi Tony!
> On Sa, 17 Nov 2012, Tony Mechelynck wrote:
>> Depending on how the logging is done, in the case I mentioned there
>> could very well be quite a lot of <CR>s before the first <NL>, and,
>> let's say, something like fifty <CR>s before the fifth <NL>.
> It is more likely, the file is really in MAC format rather than some
> obscure logfile.
Just adding a comment.
A couple weeks ago I had 4 files that I created on my Linux system and
did not move them or share them and some how they were showing a ^M
when I was doing a grep and sed search and replace from the shell. So
it must be possible to create that condition on Linux but I have no
idea how it happened but the files are no longer showing that
condition.
On Saturday, November 17, 2012 10:46:01 PM UTC-6, stosss wrote:
> A couple weeks ago I had 4 files that I created on my Linux system and
> did not move them or share them and some how they were showing a ^M
> when I was doing a grep and sed search and replace from the shell.
The problem wasn't "showing a ^M", that's pretty common. The problem was that EVERY line was terminated ONLY by a ^M and there were NO linefeed characters in the file at all, so the file was showing as one long line with a bunch of ^M characters.
On Sun, Nov 18, 2012 at 8:50 AM, Ben Fritz <fritzophre...@gmail.com> wrote:
> On Saturday, November 17, 2012 10:46:01 PM UTC-6, stosss wrote:
>> A couple weeks ago I had 4 files that I created on my Linux system and
>> did not move them or share them and some how they were showing a ^M
>> when I was doing a grep and sed search and replace from the shell.
> The problem wasn't "showing a ^M", that's pretty common. The problem was that EVERY line was terminated ONLY by a ^M and there were NO linefeed characters in the file at all, so the file was showing as one long line with a bunch of ^M characters.
Okay :-) thanks. I didn't realize that happens a lot with the "^M".
They showed up again in a file last night but not the continuous
single line. Then they were gone without me doing anything. I have
never actually seen them when opening the actual file. Only when doing
a shell command that out puts to the screen. Like grep, cat, head,
tail etc. I have also never seen them in a file created when using a
redirect away from the screen.
