<NL> in sub-replace-expression('\=')

[motz]

Hi all;

Could anyone tell me why <NL>(0x0a) should be replaced by <CR>(0x0d)
in sub-replace-expression('\=')?

See the following code snippet;

function! s:nl()
return "\n"
endfunction
echomsg substitute('a', '.', "\n", "") "(1)
echomsg substitute('a', '.', '\=s:nl()', "") "(2)

(1) returns <NL>(0x0a), (2) returns <CR>(0x0d).

Why such design choice was done?

Thank you in advance.

Motoya Kurotsu

[Ben Schmidt]

> Could anyone tell me why<NL>(0x0a) should be replaced by<CR>(0x0d)
> in sub-replace-expression('\=')?
>
> See the following code snippet;
>
> function! s:nl()
> return "\n"
> endfunction
> echomsg substitute('a', '.', "\n", "") "(1)
> echomsg substitute('a', '.', '\=s:nl()', "") "(2)
>
> (1) returns<NL>(0x0a), (2) returns<CR>(0x0d).
>
> Why such design choice was done?

I think you are doing something you're not really supposed to do. The
bug is probably that it doesn't trigger an error, not that it works
unexpectedly.

I think there are a bunch of different things going on here.

For (1), I don't get <NL> returned, but ^@ (<Nul>). This is somewhat
expected, given :help keycodes, but still a little mysterious, as that
help topic is about bytes being represented in a buffer, not a string.
Strings usually behave a bit differently and more normally.

But I get different results with :echo to what I get with :echomsg, too.
:echo behaves in a more expected manner. So I suspect :echomsg is not
really printing a Vim string properly, treating it more like raw text
out of a buffer, which is slightly different, and :echo is doing more
the right thing. There are a lot of funny little behaviours with things
like this. Doing Ctrl-R = in insert mode and then using strings like
"\<NL>", "\<CR>", "\<Nul>", "\<Up>", etc. all do different and strange
things, too. Likewise trying them with :echo and :echomsg. Lots of
strangeness.

Regarding (2), note this comment in the help (:help
sub-replace-expression):

"This does not work recursively: a substitute() function inside the
expression cannot use "\=" for the substitute string."

This suggests you can't really expect \= to work as the substitute
string in a substitute() function call.

However, the help text at that place, as well as at :help
sub-replace-special sheds some light on the matter, and perhaps explains
why things are happening the way they are. Of course, the differences
between :echo and :echomsg make it even more confusing.

I'm not sure how much, if any, of this could be regarded as bugs, and
how much, if any, would be fixed, given backwards-compatibility issues.

Ben.

[motz]

Hi, Ben and all;

Thank you for your reply. At first, I should have made my system
environment clear at the previous post.

OS - Fedora 12
VIM - Vi IMproved 7.3 (2010 Aug 15, compiled Oct 18 2010 15:14:40)
Included patches: 1-29
Modified by <bugz...@redhat.com>
Compiled by <bugz...@redhat.com>

I would like to exclude the issue of echo/echomsg dependency here.
I updated the previous code like the following;

function! s:nl()
return "\n"
endfunction

let x = substitute('a', '.', "\n", "")
let y = substitute('a', '.', '\=s:nl()', "")
echomsg x
echomsg y
echomsg len(y)
echomsg strpart(y, 0, 1) == "\x0a"
echomsg strpart(y, 0, 1) == "\x0d"

The result is,

^@
^M
1
0
1

I hope that I could have made my point clear and would like to
focus on the behaviour of substitute() and sub-replace-expression.

>"This does not work recursively: a substitute() function inside the
>expression cannot use "\=" for the substitute string."
>
>This suggests you can't really expect \= to work as the substitute
>string in a substitute() function call.

I think that you misunderstand the help above. I think that it means,
for example, the following code does not work.

let y = substitute(str, '.', "\=substitute(v:val, 'x', 'X', '')",
'g')

Please refer to the function vim_regsub_both in regexp.c if you like.

>I think you are doing something you're not really supposed to do. The
>bug is probably that it doesn't trigger an error, not that it works
>unexpectedly.

As you mention, if I'm doing something what is not supposed to do, Vim
should generate an error. However I suspect that what I'm doing
something not supposed to do.

The attached patch is the example which works around this problem. It
is
assumed that the variable reg_line_lbr is true if the function
vim_regsub_both is called from the substitute(), vice versa if it is
called from the :substitute command as the current implementation is.

The result of the code above;

^@
^@
1
1
0

It seems OK. Next test :substitute command with the pattern
written at the comment of the function vim_regsub_both. On text;

abc\
def

execute ':%s/abc\\\ndef/\="aaa\\\nbbb"/'.

aaa\
bbb

It seems OK, too.

In summary, I think that this issue is a bug and it should be fixed
so that the result is as same as my patch works around. Otherwise,
Vim should inform us of illegal usage by generating error.

Regards,

Motoya Kurotsu

===================== attachment =================
--- vim73/src/regexp.c 2011-03-06 16:45:15.000000000 +0900
+++ vim73.1/src/regexp.c 2011-03-06 16:36:12.000000000 +0900
@@ -7011,7 +7011,7 @@
{
/* Change NL to CR, so that it becomes a line
break.
* Skip over a backslashed character. */
- if (*s == NL)
+ if (*s == NL && !reg_line_lbr)
*s = CAR;
else if (*s == '\\' && s[1] != NUL)
{
@@ -7021,7 +7021,7 @@
* abc\
* def
*/
- if (*s == NL)
+ if (*s == NL && !reg_line_lbr)
*s = CAR;
had_backslash = TRUE;

[Ben Schmidt]

> I would like to exclude the issue of echo/echomsg dependency here.

OK, sounds good!

>> "This does not work recursively: a substitute() function inside the
>> expression cannot use "\=" for the substitute string."
>>
>> This suggests you can't really expect \= to work as the substitute
>> string in a substitute() function call.
>
> I think that you misunderstand the help above. I think that it means,
> for example, the following code does not work.
>
> let y = substitute(str, '.', "\=substitute(v:val, 'x', 'X', '')",
> 'g')

I don't think that's the correct interpretation of the help, either.
Actually, I think it means you can't do

:s/./\=substitute(submatch(0),'x','\=whatever','')

My main point, though, which has basically just been proven, is that
it's hard to know exactly what the help means, or was intended to
describe; that fact merely suggests there might be problems if you have
\= inside a substitute() call.

> The attached patch is the example which works around this problem. It
> is assumed that the variable reg_line_lbr is true if the function
> vim_regsub_both is called from the substitute(), vice versa if it is
> called from the :substitute command as the current implementation is.
>
> The result of the code above;
>
> ^@
> ^@
> 1
> 1
> 0
>
> It seems OK. Next test :substitute command with the pattern
> written at the comment of the function vim_regsub_both. On text;
>
> abc\
> def
>
> execute ':%s/abc\\\ndef/\="aaa\\\nbbb"/'.
>
> aaa\
> bbb
>
> It seems OK, too.
>
> In summary, I think that this issue is a bug and it should be fixed
> so that the result is as same as my patch works around. Otherwise,
> Vim should inform us of illegal usage by generating error.

Your workaround seems sensible to me, though yes, of course, it should
either be ensured that reg_line_lbr is always set as you expect, and
documented in a code comment to be that way, or another variable should
be introduced to give the distinction you need.

Also, there may be other issues that are probably worth
checking/addressing at the same time. As :help sub-replace-expression
notes, using \n to break lines is a convenience. The 'proper' way to do
it is to use \r. Does \r currently 'do the right thing', or does it
result in \r in the string? Also, does \\\r do the right thing (insert a
\r), and how about \\\n--does it insert a null (i.e. probably end the
string) as it should if the documentation is followed?

>> I'm not sure how much, if any, of this could be regarded as bugs, and
>> how much, if any, would be fixed, given backwards-compatibility issues.

I still stand by this, though. We'll need to hear from Bram whether he's
willing to change this.

Ben.

[motz]

Hi, Ben and all.

Thank you for following up. At first, I would like to correct the
previous my post.

> I think that you misunderstand the help above. It means, for example,
> the following code does not work, I think. See the function
> vim_regsub_both in regexp.c.

>
> let y = substitute(str, '.', "\=substitute(v:val, 'x', 'X', '')", 'g')

This code is wrong and is not what I intended.

let y = substitute(str, '.', '\=substitute(submatch(0), "x", "X",
"")', 'g')

This one is what I intended. And this code "works" correctly.
So I would like to delete the part of the previous post and apologize
you Ben for my carelessness.

> :s/./\=substitute(submatch(0),'x','\=whatever','') - (1)

>
> My main point, though, which has basically just been proven, is that
> it's hard to know exactly what the help means, or was intended to
> describe; that fact merely suggests there might be problems if you have
> \= inside a substitute() call.

Yes, I agree that help is only suggestion. In this case, code says us
that
there would be problems if we have '\=' inside a 'recursively' called
substitute(). For example, the (1) you showed is one case and the
below
is another in the context of function call.

let y = substitute(str, '.', '\=substitute(submatch(0), "x",
"\=whatever", "")', '')

I assume that we can have '\=' inside a 'non-recursively' called
substitute()
like the following;

let y = substitute('a', '.', '\=s:nl()', "")

The issue is that evaluation denoted by '\=' includes the
interpretation
such as ':help sub-replace-expression' describes. Actually we don't
have
any way to replace something with the result of evaluation including
"\n".

> Also, there may be other issues that are probably worth
> checking/addressing at the same time. As :help sub-replace-expression
> notes, using \n to break lines is a convenience. The 'proper' way to do
> it is to use \r. Does \r currently 'do the right thing', or does it
> result in \r in the string? Also, does \\\r do the right thing (insert a
> \r), and how about \\\n--does it insert a null (i.e. probably end the
> string) as it should if the documentation is followed?

I think that ':help sub-replace-expression' describes about
substitution
of buffer. I think that substitution of string should not do such
interpretation. What really we want seems the flag which disable such
interpretation. The variable reg_line_lbr seems to work as such flag,
but as you mention, we may need another flag.

>>> I'm not sure how much, if any, of this could be regarded as bugs, and
>>> how much, if any, would be fixed, given backwards-compatibility issues.
>
> I still stand by this, though.

I understand.

> We'll need to hear from Bram whether he's
> willing to change this.

Yes, I would. Is he check this group periodically? Or is it better to
post this
issue to vim_dev?

Regards,

Motoya Kurotsu

[Ben Schmidt]

> So I would like to delete the part of the previous post and apologize
> you Ben for my carelessness.

No need at all to apologise.

> I think that ':help sub-replace-expression' describes about
> substitution of buffer. I think that substitution of string should not
> do such interpretation. What really we want seems the flag which
> disable such interpretation. The variable reg_line_lbr seems to work
> as such flag, but as you mention, we may need another flag.

The help section is definitely written with a buffer perspective.
However, it doesn't have anything in it to say it is restricted to that.
I'm not sure what makes more sense. I lean towards doing the
substitutions, though, so that \= works the same way regardless of
whether you use it in a substitute() or a :substitute.

For buffers, this means:

<CR> => line break
\<CR> => carriage return
<NL> => line break (for convenience)
\<NL> => linefeed, which is interpreted as a null in a buffer

For strings:

<CR> => linefeed
\<CR> => carriage return
<NL> => line break, linefeed (unchanged)
\<NL> => null (ends the string)

Part of my reasoning for this is that I can't think of any situation
where you would want to use \= in a substitute() call, unless you were
writing some plugin that took a \= originally intended for a :substitute
command and used it in substitute(). Ordinarily if you want an
expression in substitute(), you just write the expression as the
argument.

>> We'll need to hear from Bram whether he's willing to change this.
>
> Yes, I would. Is he check this group periodically? Or is it better to
> post this issue to vim_dev?

He definitely follows vim_use a bit, but appears more frequently in
vim_dev. I'm sure he wouldn't mind being CCed or emailed directly with a
specific question about a patch (or proposed patch), though, as it is a
question only he can answer. :-)

Ben.