ex program inserts null char for newline upon insert/append of backslash newline

[Frush]

VIM Developers-

One of the engineers in our lab found this one, and even wrote up a
simple test case. Apparently, some change was introduced in VIM 7.0
that changes the behavior of the 'ex' command. I wasn't able to
quickly find a VIM bugtracker, so if there is one, please point me
there so I can submit this issue!


This problem occurs on version 7.0.109-7 of vim on RHEL 5. It also
seems to occur on a self-compiled 7.2 version (VIM - Vi IMproved 7.2
(2008 Aug 9, compiled Jul 28 2010 12:12:14))
The problem occurs both when executing ex from a script (as in the
example shown below) or in an interactive ex session, but not with a
vim interactive session.

The problem did not occur with VIM 6.3.

To recreate the problem, perform the following at the bash command
prompt:

#Create Input file to ex
cat - > ex.input << EOF
a
X \\
Y
.
w ex.output
EOF

# Show input
cat -tvse ex.input
# Submit input to ex
ex - -u NONE < ex.input
#Show output
cat -tvse ex.output


When working correctly VIM 6.3 (rhel4), the output from cat of
ex.output is:

$
X \$
Y$

On VIM 7.x , the output from cat of ex.output is:

X \^@Y$

Note that the '^@' indicates that we have a null character where there
should have been a newline.


Hopefully someone familiar with the code will be able to track down
where this is getting messed up.

[Frush]

We tracked down the issue to a change in the file src/ex_getln.c.
Here is the change log for the change that introduced the problem:

CVS revision 1.14
HG revision 4d9eabb139
date: 2005/02/22 08:44:08; author: vimboss; state: Exp; lines: +181
-182
updated for version 7.0051

< if (line_ga.ga_len && p[-1] == '\n')
---
> /* we are done when a NL is entered, but not when it comes after a
> * backslash */
> if (line_ga.ga_len > 0 && pend[-1] == '\n'
> && (line_ga.ga_len <= 1 || pend[-2] != '\\'))


This patch seems to be the cause of the behavior we're seeing. Can
someone (VIMBOSS?) more familiar with this comment on this behavior,
and if it correct for ex?

--
Ray Frush
IT Infrastructure Engineer

[John Beckett]

Frush wrote:
> We tracked down the issue to a change in the file src/ex_getln.c.

Vimboss is Bram Moolenaar, the author of Vim.

I noticed your first post and reproduced the issue, but have no
idea what the correct behaviour should be. The thread is:
http://groups.google.com/group/vim_dev/browse_thread/thread/57baa4b3995d1e81

The issue concerns the result from this ex input:

a
X \\
Y
.

The ex command 'a' (which is :a in Vim) appends the following
lines (starting with 'X' and 'Y'), stopping at the '.' line.

In Vim (entering :a) you get two lines appended:
X \\
Y

When fed via stdin to 'ex' by the shell, the result is one line:
X \^@Y

When you do this, you get a NULL byte where "^@" is shown above.
Clearly some tricky quoting is going on, and the newline ends up
in its internal representation (a zero byte).

The above summary might prompt someone to look at the issue, but
people are distracted by the impending release of Vim 7.3.
If you get no answer, I suggest trying again in a month.

John

[Lech Lorens]

On 11 August 2010 10:22, John Beckett <johnb....@gmail.com> wrote:
> The issue concerns the result from this ex input:
>
>    a
>    X \\
>    Y
>    .
>
> The ex command 'a' (which is :a in Vim) appends the following
> lines (starting with 'X' and 'Y'), stopping at the '.' line.
>
> In Vim (entering :a) you get two lines appended:
>    X \\
>    Y
>
> When fed via stdin to 'ex' by the shell, the result is one line:
>    X \^@Y

I get (double backslash)
X \\^@Y

which is still incorrect (I suppose the difference between your and
mine results might come from the quoting done by the shell). This is
on Vim 7.2.440.

--
Cheers,
Lech

[Frush]

Thanks for the replies and the insight into folk's focus on bug
smashing for 7.3. Perhaps this will be of interest to that effort:

I got this hint from another source: Apparently in FreeBSD, VIM 7.2
with a select set of patches manages to get the behavior correct:

<quote>
Does not occur in:
VIM - Vi IMproved 7.2 (2008 Aug 9, compiled Apr 14 2010 06:22:53)
Included patches: 1-6, 8-35, 37-48, 50-70, 73, 75-87, 90-92, 94-100,
102-137, 139-149, 151-171, 173-190, 192-193, 195-203, 206-211,
213-215, 217-218, 220-232, 234-246, 251-259, 261-301, 303-319,
321-322, 324-335, 337-351, 353-361, 363, 366-371, 373, 375-376,
378-383, 385-387, 389-398, 401-402, 404-411
on FreeBSD; I don't track the reasons for FreeBSD's port maintainer of
vim to declare certain patches bad.

</quote>

we're looking into reproducing the results on Red Hat Linux.... it's
quite a list of patches.

[James Vega]

On Wed, Aug 11, 2010 at 10:10 AM, Frush <ray....@gmail.com> wrote:
> Thanks for the replies and the insight into folk's focus on bug
> smashing for 7.3.   Perhaps this will be of interest to that effort:
>
> I got this hint from another source:   Apparently in FreeBSD, VIM 7.2
> with a select set of patches manages to get the behavior correct:
>
> <quote>
> Does not occur in:
>  VIM - Vi IMproved 7.2 (2008 Aug 9, compiled Apr 14 2010 06:22:53)
>  Included patches: 1-6, 8-35, 37-48, 50-70, 73, 75-87, 90-92, 94-100,
>  102-137, 139-149, 151-171, 173-190, 192-193, 195-203, 206-211,
>  213-215, 217-218, 220-232, 234-246, 251-259, 261-301, 303-319,
>  321-322, 324-335, 337-351, 353-361, 363, 366-371, 373, 375-376,
>  378-383, 385-387, 389-398, 401-402, 404-411
> on FreeBSD; I don't track the reasons for FreeBSD's port maintainer of
> vim to declare certain patches bad.

That's probably due to some of the patches only applying to the "extra"
tarball, which contains sources for non-unix systems and other
non-essential code. If they're not using the extra tarball, then they
have no need to apply the patches for it.

--
James
GPG Key: 1024D/61326D40 2003-09-02 James Vega <jame...@jamessan.com>