[vim/vim] sh FileType syntax folding lost when re-enabling syntax highlighting (Issue #11536)

[foshoz]

Steps to reproduce

(I apologize for the terrible formatting -- I don't know Markdown. There is a much more readable bug-report.txt file attached if that's easier to read.)

1. Enable sh FileType syntax folding in Vim user configuration files.
   i. See attached files for minimum working example of my ~/.vimrc
   configuration.
2. Invoke vim without its '-u' argument to open one or more sh or Bash shell
   scripts.
   i. See attached files for an example of my /etc/profile, which exhibits the
   problem for me.
   ii. See Discussion section for additional information.
3. Navigate to any foldable region.
   i. I used the case statement on line 31.
   ii. I used arrow keys and the 'w' key, but any method is likely to
   demonstrate the issue.
4. Attempt to fold the region.
   i. I use the 'za' command in Normal mode.
5. Vim produces the error message "E490: No fold found".
   i. I observe the error message in the usual location, on the left side of
   the window, under the status line.

Notes:

A.  Syntax highlighting appears to continue working for the sh FileType.
B.  Syntax highlighting AND syntax folding both appear to continue working
    for other FileTypes.
C.  I did not exhaustively test either A. or B.

vimrc.txt
suse.vimrc.txt
sh.vim.patch.txt
profile.txt
scriptnames.txt
bug-report.txt

Expected behaviour

The foldable region should fold, and there should not be an error message.
Syntax highlighting and syntax folding should both be enabled.

Note: in this configuration, all folds should be closed by default. That no
folds are show when the buffer first opens is itself an indicator of a problem.

Version of Vim

9.0.0313-150000.5.25.1

Environment

Operating System: openSUSE Leap 15.3
Terminal : PuTTY, version 0.77-1
$TERM: : putty-256color
Shell : GNU bash, version 4.4.23(1)-release (x86_64-suse-linux-gnu)

Notes:
1. I am using this machine from a Windows 10 host over an SSH connection.
It's possible, but very, very, very unlikely that this is an SSH or
Windows problem. I don't currently have physical access to the machine
so I cannot check this assumption though.

Logs and stack traces

Discussion

I initially suspected a problem because none of my attempts to add this feature
to my Vim user configuration were working. I was tipped off to a possible Vim
initialization bug when I found that
    
    $ vim /etc/profile
    
would not show syntax folding, but
    
    $ vim -u ~/.vimrc /etc/profile
    
WOULD show shell syntax folding. Note that both of these Vim invocations use the
EXACT SAME user vimrc file (~/.vimrc), but the second command causes Vim to skip
certain initializations before sourcing the user vimrc.

I spent some time isolating the problem, and I believe I have assembled a 
minimum reproducible example. I can use the attached files to replace
my full Vim user configuration and the distribution's default Vim configuration
and produce the issue. I make the replacements as:

    $ cp -p "${ATTACHED_VIMRC}" ~/.vimrc
    $ sudo cp -p "${ATTACHED_SUSEVIMRC}" /usr/share/vim/vim90/suse.vimrc
    
where the ATTACHED_* shell variables represent files attached to this issue.

With this minimum configuration in place, I am able to work around the problem  
by commenting out the distribution's 'syntax on' command. And after additional
investigation, I am concluding that this behavior is likely the result of a bug
in Vim's sh FileType syntax handling.

Inspection of the Vim runtime file
    
    /usr/share/vim/vim90/syntax/sh.vim
    
suggests that the sh FileType syntax logic cannot support multiple invocations
of the syntax command, e.g. 'syntax on' or 'syntax enable'. When this vimscript
is sourced, it defines the syntax highlighting and folding rules for sh and Bash
script FileTypes. Below is an outline of its behavior as relevant to this issue.

    Line(s)     Comments
    88-89       
                Check for the 'g:sh_fold_enabled' global variable, and define it
                and set it to zero if it doesn't already exist.
    94-102      
                Define script variable 's:sh_fold_functions' (and similar script
                variables) based on the bits set in the 'g:sh_fold_enabled'
                global variable, but ONLY when the script variables do not
                already exist.
    123-137     
                Define folding functions, based on the value of the
                's:sh_fold_functions' script variable (and similar script
                variables.)
    482-492 (e.g.)
                Call folding functions, based on which shell type is detected
                (sh or Bash). The folding functions are used to append the
                'syntax' command's 'fold' parameter (as appropriate) when
                defining foldable syntax regions.
    802-804
                Delete folding functions.

I propose that the following Vim runtime behavior is occurring.
     
    1.  Vim begins initialization.
         
    2.  Vim sources the distribution's default vimrc.
         
        ** Assumption: the distribution's Vim default configuration DOES enable
        syntax highlighting and DOES NOT enable syntax folding for the sh
        FileType. (This is the configuration for the openSUSE Leap distribution,
        15.3 release; I don't know what other distributions do.) **
         
        Vim reaches the 'syntax' command and eventually invokes the
        'syntax/sh.vim' script for the VERY FIRST time.
         
    3.  The 'syntax/sh.vim' script defines and initializes the
        'g:sh_fold_enabled' global variable to zero; and defines the
        's:sh_fold_functions' script variable (and similar script variables),
        and initializes each to zero. The values of the 's:sh_fold_functions'
        script variable (and similar script variables) cause the folding
        functions to be defined without the fold parameter, which disables
        syntax folding. The folding function commands are then deleted just
        before the script ends. The 's:sh_fold_functions' script variable (and
        similar script variables) are not deleted.
         
    4.  Vim continues initialization.
         
    5.  Vim sources the Vim user configuration files.
         
        ** Assumption: the Vim user configuration DOES enable syntax
        highlighting and DOES enable syntax folding for the sh FileType.
         
        Vim reaches the 'syntax' command and eventually invokes the
        'syntax/sh.vim' script for (at least) the SECOND time.
         
    6.  The 'syntax/sh.vim' script observes that the 'g:sh_fold_enabled' global
        variable already exists. If the user's Vim configuration configures this
        variable, that value is respected and preserved. HOWEVER, because the
        's:sh_fold_functions' script variable (and similar script variables)
        already exist, they are not updated and so do not respect the user's Vim
        configuration choices. As a result, when the folding functions are
        defined, they are defined according to the value of the
        'g:sh_fold_enabled' global variable present when the script was first
        invoked. Therefore syntax folding continues to be disabled.
         
    7.  Vim continues initialization. If Vim encounters additional 'syntax on'
        or 'syntax enable' commands, their effect will be as described in para.
        6: that is, syntax folding will continue to be disabled.
         
    8.  Vim completes initialization.
         
    9.  Vim completes loading the file(s) normally.
         
    10. The user attempts to use syntax folding with the shell script file(s).
        Syntax highlighting works, but syntax folding does not work.
         
        ** Assumption: the Vim user has opened sh or Bash shell scripts. **
         
    11. The user attempts to manually set the FileType ('set filetype=sh').
        
        Vim eventually invokes the 'syntax/sh.vim' script for (at least) the
        THIRD time.
         
    12. Vim sets the filetype (otherwise) normally. The effect of the
        'syntax/sh.vim' script will be as described in para. 6: that is, syntax
        folding will continue to be disabled.

Note that some of this is speculation, because I can't find in
':help initialization' exactly where the distribution's Vim default 
configuration file is read. There is nothing named /etc/vimrc, so I am unclear
exactly how and when the distribution's Vim default configuration file is read.
The help _seems_ to say that if the user configuration files exist, they are
read _instead of_ the distribution's default configuration file, which seems
prima facie incorrect. And testing has proven that the distribution's Vim
default configuration file is read BEFORE the Vim user configuration files (see
attached 'startscripts' output).

Based on the behaviors outlined above, it also appears that if the
distribution's Vim default configuration file sets the 'g:sh_fold_enabled'
global variable BEFORE enabling syntax highlighting, that setting will be
respected, and sh FileType syntax folding will work correctly. However, even in
this case, the user would still be unable to change the behavior of the sh
FileType syntax folding (the value of the 'g:sh_fold_enabled' global variable)
from the distribution's defaults. This is because the 's:sh_fold_functions'
script variable (and similar script variables) still cannot be re-evaluated,
preventing re-definition of the folding function commands.

I am new to working in the Vim runtime, so my descriptions of this issue and its
implications are likely to be wrong in some places. I have nevertheless included
a patch for the 'syntax/sh.vim' script. The patch simply unlets the
's:sh_fold_functions' script variable (and similar script variables) when
deleting the folding function commands. I still have a lot to learn about Vim,
so there may be better solutions. But hopefully my description helps identify
the problem.

The patch works on my openSUSE Leap 15.3 machine, but I have not tested any Vim
user configurations aside from my minimal reproductible example and my full
configuration. And I definitely have not done any done any regression testing.

—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.Message ID: <vim/vim/issues/11536@github.com>

[Bram Moolenaar]

I can't tell whether this is really a bug or you need to adjust your setup. @cecamp

Note that setting 'foldcolumn' to 4 might give you extra information.

—
Reply to this email directly, view it on GitHub.

You are receiving this because you are subscribed to this thread.Message ID: <vim/vim/issues/11536/1312714908@github.com>

[foshoz]

All indications I have suggest this is a bug. I also observe the same issue on another machine with a near-identical environment, although I did not test my MRE on that machine, and I don't have privileges to attempt my patch on that machine.

I did not know about 'foldcolumn', but I added it to my full configuration, and checked before and after applying my patch. I did not repeat with my MRE, just my full configuration.

WITHOUT my patch (sh.vim.patch.txt above) applied:

WITH my patch (sh.vim.patch.txt above) applied:

—
Reply to this email directly, view it on GitHub.

You are receiving this because you are subscribed to this thread.Message ID: <vim/vim/issues/11536/1312777393@github.com>