syntax highlighting not correct in bash script

[kamaraju kusumanchi]

The following shell script is not highlighted correctly.

. % cat foo.sh
#!/usr/bin/bash

UPSTREAM=${1:-'@{u}'}

Please find the screenshot at https://pasteboard.co/HwyXxZU.png .

As you can see in the screenshot, the first and second left curly
braces are blue and yellow while the third and fourth right curly
braces are blue and magenta. Any idea why?

Using vim 8.0, on Debian 9.5 (Stretch).

thanks
raju
--
Kamaraju S Kusumanchi | http://raju.shoutwiki.com/wiki/Blog

[tooth pik]

my vim is newer than yours, so i'll wager my syntax/sh.vim is newer too

anyway, that exact command renders just fine in my vim (8.1.223)

> --
> --
> You received this message from the "vim_use" maillist.
> Do not top-post! Type your reply below the text you are replying to.
> For more information, visit http://www.vim.org/maillist.php
>
> ---
> You received this message because you are subscribed to the Google Groups "vim_use" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to vim_use+u...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[kamaraju kusumanchi]

On Sat, Jul 28, 2018 at 3:00 PM, tooth pik <toot...@gmail.com> wrote:
> my vim is newer than yours, so i'll wager my syntax/sh.vim is newer too
>
> anyway, that exact command renders just fine in my vim (8.1.223)
>

My /usr/share/vim/vim80/syntax/sh.vim is

" Last Change: Sep 22, 2016
" Version: 165

I see the 8.1.223 file in
https://github.com/vim/vim/blob/v8.1.0223/runtime/syntax/sh.vim . I
downloaded it to ~/.vim/syntax/sh.vim . Even then I get the same
behavior. Do I need to copy any other files besides that?

[John Little]

With the vim 8.1.0224 from git, I see the problem. (BTW, the colours depend totally upon the colour scheme being used, so characterizing the problem by the colours seen isn't very useful. Maybe using the default scheme, starting vim with --clean -N could be.)

The problem causes the syntax colouring to lose track of what is quoted, and following lines of script are coloured as if quoted.

The maintainer of sh.vim has fixed quoting problems in his latest version, 179, at http://www.drchip.org/astronaut/vim/index.html#SYNTAX_SH and indeed version 179 colours your snippet correctly for me; it works with the 8.0.1453 I get with my distro, derived from Ubuntu 18.04, so I expect it will work for you.

I imagine your snippet is a minimal example of the problem, kudos for that, but still looks pathological to me; I avoid tricky stuff like that for the sake of my sanity.

Regards, John Little

[kamaraju kusumanchi]

On Sun, Jul 29, 2018 at 7:37 AM, John Little <john.b...@gmail.com> wrote:
> With the vim 8.1.0224 from git, I see the problem.

Thanks for your help.

> The maintainer of sh.vim has fixed quoting problems in his latest version, 179, at http://www.drchip.org/astronaut/vim/index.html#SYNTAX_SH and indeed version 179 colours your snippet correctly for me; it works with the 8.0.1453 I get with my distro, derived from Ubuntu 18.04, so I expect it will work for you.
>

Confirm that using the 179 version fixed the problem. For the record,
my vim version is 8.0.707 and I just copied the sh.vim to
~/.vim/syntax .

> I imagine your snippet is a minimal example of the problem, kudos for that, but still looks pathological to me; I avoid tricky stuff like that for the sake of my sanity.

Yes.The snippet I provided is a minimal example. The
UPSTREAM=${1:-'@{u}'} sets UPSTREAM variable to the first argument. If
no such argument is supplied, it uses @u which represents the upstream
branch (in git parlance). If you are curious, the complete script is
at https://github.com/KamarajuKusumanchi/rutils/blob/master/bin/git-up
.

regards

[L A Walsh]

May not be related to your problem, BUT you also need to tell vim that
you are
using bash, as it shares syntax with 'sh'.

In my ~/.vimrc, I have
let g:is_bash=1
let b:is_bash=1

[Richard Riegner]

I consistently see text display corruption when
editing a file on a remote Linux system with vim.

I run an xterm window on macOS and ssh into a
remote Linux machine. The edit session is
displayed back to an XQuartz server running on
the same macOS.

I have only seen this problem when displaying
back to an XQuartz server running on macOS.

I have not seen the problem when displaying back
to an X11 server running on various Linux
systems.

This problem occurs with newer versions of vim,
but not older versions.

Inserting lines causes the INSERT message to be
displayed in the command line, but it is not
cleared after hitting the ESC key.

Inserting text also causes extraneous lines of
text to be added to the body of the original
text, corrupting the edit display.

I suspect that I am seeing the same vim behavior
as was reported in this incident:

https://groups.google.com/forum/#!topic/vim_dev/GR9YG8TZy6o

A work-around is to add this statement to the
.vimrc file:

set ambiwidth=single

I do not know if this is a bug in vim, XQuartz,
or both. So I am asking for vim help here and
XQuartz help in another forum.

Here is a summary of editing a file using two
versions of vim, each with and without, the
ambiwidth work-around.


New Vim
-------
VIM - Vi IMproved 7.4 (2013 Aug 10)
Included patches: 1-207, 209-326
Compiled by 'http://www.opensuse.org/'

Works with: set ambiwidth=single
Fails without: set ambiwidth=single


Old Vim
-------
VIM - Vi IMproved 7.2 (2008 Aug 9, compiled Feb 4 2012 09:50:29)
Included patches: 1-108
Compiled by 'http://www.opensuse.org/'

Works with: set ambiwidth=single
Works without: set ambiwidth=single

--

Dick Riegner

[Bram Moolenaar]

What is the version of the xterm you are using?
Also, what is v:termresponse?

--
To the optimist, the glass is half full.
To the pessimist, the glass is half empty.
To the engineer, the glass is twice as big as it needs to be.

/// Bram Moolenaar -- Br...@Moolenaar.net -- http://www.Moolenaar.net \\\
/// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
\\\ an exciting new programming language -- http://www.Zimbu.org ///
\\\ help me help AIDS victims -- http://ICCF-Holland.org ///

[Eli the Bearded]

Richard Riegner <dickie@integra> wrote:

> I run an xterm window on macOS and ssh into a
> remote Linux machine. The edit session is
> displayed back to an XQuartz server running on
> the same macOS.

I have seen lots of problems with that sort of setup:

local Mac with XQuartz and xterm used to
remote ssh to Linux or NetBSD and
running vim on the remote end

My personal guess is that there's an issue with the xterms that come
with XQuartz, but I am not certain. And I no longer have a mac to test
with. As I recall, the workaround I found was to run screen or tmux
to provide an intermediary terminal interpreter. I don't think it is a
vim bug (but maybe vim needs to work around this bug).

> I have only seen this problem when displaying
> back to an XQuartz server running on macOS.

I've seen other people complain about this, too.

> This problem occurs with newer versions of vim,
> but not older versions.

...

> Inserting text also causes extraneous lines of
> text to be added to the body of the original
> text, corrupting the edit display.

This is the exact behavior I found most annoying.

> I suspect that I am seeing the same vim behavior
> as was reported in this incident:
>
> https://groups.google.com/forum/#!topic/vim_dev/GR9YG8TZy6o

That's a 7.3 change which would explain why your 7.2 is fine and 7.4
needs help.

> A work-around is to add this statement to the
> .vimrc file:
>
> set ambiwidth=single

Would I have known that when I had the issue.

> I do not know if this is a bug in vim, XQuartz,
> or both. So I am asking for vim help here and
> XQuartz help in another forum.

The other fix I found was to ditch the XQuartz xterms and use iterm2.
Just as another option.

Elijah

[Richard Riegner]

Elijah,

Thanks for the help and insight.

I guess it is good to know that I am not the only one seeing these kinds
of problems with vim and XQuartz.

Norm Wood found that you can also work-around this problem if you reset
your vim term variable to xterm after starting the edit session:

set term=xterm

This makes me wonder if there is some kind of timing issue at start-up
between vim and XQuartz? But none of my X11 servers on Linux have this
problem.

Another work-around is to set TERM to screen-256color instead of xterm.

I have minimized the impact of this issue by adding ambiwidth to the
.vimrc files on systems I control. But there are many other shared lab
systems where updating .vimrc on an ongoing basis is not practical. This
is why I would like to get this fixed.

Thanks again.

Dick Riegner

--

[Richard Riegner]

Bram,

Thanks for your help.

The macOS xterm version is:

XTerm(326)

The macOS XQuartz version is:

XQuartz 2.7.11 (xorg-server 1.18.4)

The v:termresponse value is:

v:termresponse ^[[>41;326;0c

--

Dick Riegner

--

[Dick Riegner]

Folks,

After almost two years, I may have stumbled onto the root cause of this vi display corruption problem.

It looks to be related to the character encoding difference between the Linux system that uses UTF-8, and macOS (XQuartz) (X11) which apparently uses something else.

I solved this problem by starting my xterm sessions on macOS with this xterm command option:

-en UTF-8

and removed all other work-arounds on macOS and the Linux systems.

Now the vi edit display on macOS is no longer corrupted.

The key seems to be the the encoding difference seen on macOS as:

Riegner-iMac3:HPE dick$ locale
LANG=
LC_COLLATE="C"
LC_CTYPE="C"
LC_MESSAGES="C"
LC_MONETARY="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_ALL=

And that on one of our typical Linux systems:

linux-pu89:/dickie/text/$ locale
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

The root cause seems to be the extended non-ASCII characters in the text file. This solution does not handle all the extended non-ASCII characters exactly the same on macOS and Linux.

For example, on my local Linux xterm, the Linux vi shows the correct text for the local file:

491 If it's not logged in, then login as:

But on my macOS xterm, looking remotely at the same file on Linux, the Linux vi shows that the quote is missing; so vi displaying from Linux back to macOS is close, but not exact:

491 If it s not logged in, then login as:

I copied the Linux file locally to macOS and used the macOS version of vi to display the non-ASCII characters as hex using "set display=uhex":

491 If it <80><99>s not logged in, then login as:

So there are definitely extended non-ASCII characters in this text file contributing to the macOS, Linux, and vi confusion.

Another option is to set the macOS locale environment variables, but I do not know what side-effects and problems that may cause. So I am going with this macOS (XQuartz) (X11) solution:

xterm -en UTF-8

None of the vi display corruption is seen using the macOS Terminal application. This sure looks to be a macOS (XQuartz) (X11) issue, but vi is front-and-center when the display corruption is seen.

I hope this solution turns out to be solid and helps other vi users running XQuartz on macOS.

--
Dick Riegner
dic...@nuveramail.net
--