New line not recognized

[Marco]

Hi!

I received a text file with tab-separated values. However, the
entire file is displayed in one line and vim shows ^M symbols
all over the text, which are U+000d characters (CR).

I ran a search-replace to replace them with <CTRL-V><CR> and the
file displays as it should. But what's the cause? Is the file
malformed? I thought CR is the usual way of encoding a line ending.
What do I need to change to make vim display the file correctly
without changing it?

What system might have saved the file? Usually I never have any
problems opening files from Windows or Mac.

The file in question can be downloaded here:
http://freeshell.de/~mpfusion/1.txt (15 KiB)

VIM - Vi IMproved 7.3 (2010 Aug 15, compiled Nov 12 2012 19:41:47)
Included patches: 1-712

on Debian Linux


Marco

[Marco]

2012-11-16 Marco:

Maybe I should add that the file displays fine in other editors
(tested in nano). I used my usual .vimrc with plenty of plugins and
customization, as well as

vim -u NONE -U NONE -i NONE

to exclude any weird configuration, with the same result.


Marco

[Salman Halim]

Different operating systems use different line endings for text files. From :help 'ff':

dos  <CR><NL>
unix  <NL>
mac  <CR>

What OS are you opening the file in? Next time you open the offending file, try ":set ff?" (without the quotes) to see what it says. The <CR> way is "mac".

My Windows GVim opened the file exactly as you described: one long line. However, by using ":e ++ff=mac" (again, no quotes) I was able to reload the file correctly.

Hope this helps,

Salman

[taco]

it is windows format , use dos2unix convert it to unix.

[Marco]

2012-11-16 Salman Halim:

> > on Debian Linux
> >
> >
> Different operating systems use different line endings for text files. From
> :help 'ff':
>
> dos <CR><NL>
> unix <NL>
> mac <CR>
>
> What OS are you opening the file in?

> > on Debian Linux

> Next time you open the offending file,
> try ":set ff?" (without the quotes) to see what it says.

unix

> My Windows GVim opened the file exactly as you described: one long line.
> However, by using ":e ++ff=mac" (again, no quotes) I was able to reload the
> file correctly.

Thanks a lot, that works. Can I automate this somehow, so that vim
opens <CR> (mac) files automatically with the ff=mac setting?


Marco

[Ben Fritz]

On Friday, November 16, 2012 9:55:10 AM UTC-6, Marco wrote:
>
> > My Windows GVim opened the file exactly as you described: one long line.
> > However, by using ":e ++ff=mac" (again, no quotes) I was able to reload the
> > file correctly.
>
> Thanks a lot, that works. Can I automate this somehow, so that vim
> opens <CR> (mac) files automatically with the ff=mac setting?
>

:help 'fileformats' (note the s at the end).

[Marco]

2012-11-16 Ben Fritz:

Thanks

set fileformats=unix,dos,mac

did the trick.

Marco

[Salman Halim]

To elaborate on what Ben said, you could just make sure your 'ffs' also has "mac" in there and this problem should go away. I have this function that I fire off every time I edit a file. (The autocommand is right after the endfunction.) Occasionally, I will get a file that was edited by both developers on DOS and Unix systems and their editors will quietly leave mixed newlines in there. I believe that Vim will open a file with mixed newlines as Unix (at least, this is what I see).

Basically, this counts the number of ^M characters and prints a message reporting the number of ^M characters (if any) and, if more than half the file contains them, suggests that the file might be DOS--otherwise, Unix. It also returns Vim script commands to fix the file format, but the autocommand doesn't use that. I put an arbitrary line limit of 10,000 lines so that the file opening process isn't slowed down too much by this check.

Admittedly, it's going to be have to refactored for Mac; also, that 'silent! execute' line contains a ^M (literal control-M) where there's a newline in the paste below.

function! NewFileReport()
  if ( &binary )
    return
  endif

  let result    = ''
  let newFormat = ''

  if ( line( '$' ) > 10000 )
    let result = printf( 'File too large to check for newlines (%d lines).', line( '$' ) )
  else
    let num         = 0
    let savedSearch = @/
    let pos         = winsaveview()

    silent! execute "g/
$/let num+= 1"

    let @/ = savedSearch
    call winrestview( pos )

    if ( num > 0 )
      let result = printf( 'Number of DOS newlines: %d (out of %d total).', num, line( '$' ) )

      if ( &fileformat != 'unix' )
        let result .= ' File is ' . toupper( &fileformat ) . '.'
      else
        let newFormat = num > ( line( '$' ) / 2 ) ? 'DOS' : 'Unix'

        let result .= ' File is probably ' . newFormat . '.'
      endif
    endif
  endif

  let result .= &eol ? '' : ' No end-of-line.'

  if ( result != '' )
    redraw
    echo result
  endif

  let result = ''

  if ( newFormat != '' )
    let result = 'set ff=' . tolower( newFormat )
  endif

  if ( !&eol )
    let result .= "\<nl>set eol"
  endif

  return result
endfunction
au BufReadPost * call NewFileReport()

[Ben Fritz]

On Friday, November 16, 2012 10:11:34 AM UTC-6, Salman Halim wrote:
>
> Occasionally, I will get a file that was edited by both developers on DOS and Unix systems and their editors will quietly leave mixed newlines in there. I believe that Vim will open a file with mixed newlines as Unix (at least, this is what I see).
>
>
> Basically, this counts the number of ^M characters and prints a message reporting the number of ^M characters (if any) and, if more than half the file contains them, suggests that the file might be DOS--otherwise, Unix. It also returns Vim script commands to fix the file format, but the autocommand doesn't use that. I put an arbitrary line limit of 10,000 lines so that the file opening process isn't slowed down too much by this check.
>
>
> Admittedly, it's going to be have to refactored for Mac; also, that 'silent! execute' line contains a ^M (literal control-M) where there's a newline in the paste below.
>

Here's a script to do the automatic reload for you (between DOS and Unix still unfortunately):

http://vim.wikia.com/wiki/Automatically_reload_files_with_mixed_line-endings_in_DOS_fileformat

This version uses a timeout in milliseconds on a search() rather than limiting by number of lines.

[jeroen]

Is there any reason why this is not the default on unix?

Jeroen

[Tony Mechelynck]

I'm not sure, but on Unix (well, on Linux) I have occasionally met files
which had a lot of lone carriage-returns and yet weren't Mac files. This
happens for instance when logging the stdout of a console program which
displays a text-mode "progess bar" by using a CR to go to the left
margin without advancing to the next line, in order to overwrite the
line just written. ISTR that rsync used to do that (when I used it to
keep my Vim source in sync before there was a Mercurial repository), and
maybe Mercurial (with the "progress" extension), or the command-line
"ftp" utility, do too. In that case you don't want to break the line at
a lone CR but you may want to delete everything that precedes a CR which
is not at the end of a line (CR at the end of a line, i.e. followed by a
line-feed character, can be taken care of by reading the file with
++ff=dos).


Best regards,
Tony.
--
Bees are very busy souls
They have no time for birth controls
And that is why in times like these
There are so many Sons of Bees.

[Christian Brabandt]

Hi Tony!

Sure enough, but that wouldn't have triggered Vim to set the fileformat
to mac. As long there are some newlines in there, you would be safe.

regards,
Christian
--

[Tony Mechelynck]

Quoting options.txt lines 2905 sqq under 'fileformats':

> This means that "mac" is only chosen when:
> "unix" is not present or no <NL> is found in the file, and
> "dos" is not present or no <CR><NL> is found in the file.
> Except: if "unix" was chosen, but there is a <CR> before
> the first <NL>, and there appear to be more <CR>s than <NL>s in
> the first few lines, "mac" is used.

Depending on how the logging is done, in the case I mentioned there
could very well be quite a lot of <CR>s before the first <NL>, and,
let's say, something like fifty <CR>s before the fifth <NL>.


Best regards,
Tony.
--
UNIX was half a billion (500000000) seconds old on
Tue Nov 5 00:53:20 1985 GMT (measuring since the time(2) epoch).
-- Andy Tannenbaum

[Christian Brabandt]

It is more likely, the file is really in MAC format rather than some
obscure logfile.

regards,
Christian
--
Es gibt problematische Naturen, die keiner Lage gewachsen sind,
in der sie sich befinden, und denen keine genugtut. Daraus entsteht
der ungeheure Widerstreit, der das Leben ohne Genuss verzehrt.
-- Goethe, Maximen und Reflektionen, Nr. 278

[stosss]

On Sat, Nov 17, 2012 at 11:39 AM, Christian Brabandt <cbl...@256bit.org> wrote:
> Hi Tony!
>
> On Sa, 17 Nov 2012, Tony Mechelynck wrote:
>
>> Depending on how the logging is done, in the case I mentioned there
>> could very well be quite a lot of <CR>s before the first <NL>, and,
>> let's say, something like fifty <CR>s before the fifth <NL>.
>
> It is more likely, the file is really in MAC format rather than some
> obscure logfile.
>

Just adding a comment.

A couple weeks ago I had 4 files that I created on my Linux system and
did not move them or share them and some how they were showing a ^M
when I was doing a grep and sed search and replace from the shell. So
it must be possible to create that condition on Linux but I have no
idea how it happened but the files are no longer showing that
condition.

[Ben Fritz]

On Saturday, November 17, 2012 10:46:01 PM UTC-6, stosss wrote:
>
>
> A couple weeks ago I had 4 files that I created on my Linux system and
>
> did not move them or share them and some how they were showing a ^M
>
> when I was doing a grep and sed search and replace from the shell.

The problem wasn't "showing a ^M", that's pretty common. The problem was that EVERY line was terminated ONLY by a ^M and there were NO linefeed characters in the file at all, so the file was showing as one long line with a bunch of ^M characters.

[stosss]

Okay :-) thanks. I didn't realize that happens a lot with the "^M".
They showed up again in a file last night but not the continuous
single line. Then they were gone without me doing anything. I have
never actually seen them when opening the actual file. Only when doing
a shell command that out puts to the screen. Like grep, cat, head,
tail etc. I have also never seen them in a file created when using a
redirect away from the screen.