foldmethod=syntax with Ruby slows Vim down tremendously

[R. Tyler Croy]

Somebody on IRC recommended I bring this issue up on the dev list last week, so
here I am.

I've done a fair bit of debugging/profiling to determine what was making my Vim
instances so slow with larger (>1k lines) Ruby files.

I've tracked it down to foldmethod=syntax, which left me with the following
backtrace from gdb:

#0 0x0000000000584f8a in fast_breakcheck ()
#1 0x00000000005d5e75 in ?? ()
#2 0x00000000005d8c2d in ?? ()
#3 0x00000000005d9010 in vim_regexec_multi ()
#4 0x0000000000613b45 in ?? ()
#5 0x000000000061b263 in ?? ()
#6 0x000000000061c048 in ?? ()
#7 0x000000000061c4fc in syntax_start ()
#8 0x000000000061cdd1 in syn_get_foldlevel ()
#9 0x000000000054a654 in ?? ()
#10 0x000000000054bfbb in ?? ()
#11 0x000000000054bd7e in ?? ()
#12 0x000000000054bd7e in ?? ()
#13 0x000000000054bd7e in ?? ()
#14 0x000000000054bd7e in ?? ()
#15 0x000000000054c9d7 in foldUpdate ()
#16 0x000000000057b85a in ?? ()
#17 0x000000000057c039 in changed_bytes ()
#18 0x000000000057c9e7 in ins_char_bytes ()
#19 0x000000000057cbb2 in ins_char ()
#20 0x00000000004e28eb in insertchar ()
#21 0x00000000004e54f1 in edit ()
#22 0x0000000000597e9a in ?? ()
#23 0x00000000005a201c in normal_cmd ()
#24 0x0000000000658add in main_loop ()
#25 0x000000000065a79e in main ()
(gdb) bt

Any more details I can help with to isolate and fix the issue? Unfortunately I
can't share the files that cause the issue, but I'm not afraid of a little gdb
or valgrindin' if necessary.

Cheers
- R. Tyler Croy
--------------------------------------
Code: http://github.com/rtyler
Chatter: http://twitter.com/agentdero
rty...@jabber.org

[Ben Fritz]

On Nov 8, 2:33 am, "R. Tyler Croy" <ty...@monkeypox.org> wrote:
> Somebody on IRC recommended I bring this issue up on the dev list last week, so
> here I am.
>
> I've done a fair bit of debugging/profiling to determine what was making my Vim
> instances so slow with larger (>1k lines) Ruby files.
>
> I've tracked it down to foldmethod=syntax, which left me with the following
> backtrace from gdb:
>

Have you tried the advice at http://vim.wikia.com/wiki/Keep_folds_closed_while_inserting_text
?

It's obviously not originally meant to speed things up, but has that
nice side effect. The FAQ actually menions this:

http://vimhelp.appspot.com/vim_faq.txt.html#faq-29.7

I was going to respond with a sarcastic "let me google that for you"
but realized this particular version of the FAQ is not readily found
via Google search (however item 3 on the first page for "vim faq" is a
wiki page with a direct link: http://vim.wikia.com/wiki/Vim_FAQ_and_other_documentation
).

[Ingo Karkat]

On 08-Nov-2011 17:59, Ben Fritz wrote:

> Have you tried the advice at
> http://vim.wikia.com/wiki/Keep_folds_closed_while_inserting_text ?
>
> It's obviously not originally meant to speed things up, but has that
> nice side effect.

I use this tip, and yet occasionally encounter such slowdowns (often when
editing large Vim scripts (ft=vim)). There's noticeable delay in moving the
cursor, and insert-completion (i_CTRL-N) is also painfully slow (so that I see
the "scanning buffers, tags, ..." messages). A restart of Vim fixes things (at
least for a while). Sorry to be so unspecific, but this can be a real pain.

-- regards, ingo

[Ben Fritz]

Yeah I realize it's not a full solution. Is the issue pretty much
known, it just needs a good fix figured out? Or does it still need
investigation? Does the OP's backtrace help at all? I admit I've not
used gdb enough to know how to use it effectively, but it looks like
all it points to is "somewhere in the syntax folding routines Vim is
being slow".

[Peter Odding]

Me and a friend of mine started a Python file type plug-in. I'd never
used syntax folding but my friend took on the challenge of creating a
decent automatic folding mode for Python source code. Ever since we
added the syntax folding I've loved the feature (no more fold markers!)
but since I started using it my Vim sometimes gets unbearably slow.

There's no real pattern (except that it takes a while for Vim to slow
down -- I only notice the problem during >= 30 minute editing sessions)
but I have noticed that once Vim gets slow, lots of non-specific and
seemingly unrelated features become very slow -- insert mode editing,
the completion menu, indenting and dedenting.

My friend suggested I use the tip on the Vim wiki but it doesn't really
solve the root problem, it just makes it occur less often, but once the
"bug" triggers it's just as big of a problem :-\

During one of the slowdowns I used ltrace to see what the hell Vim was
doing while it was unresponsive and all I could see was thousands of
string comparisons against substrings of lines (not full lines).

Like Ingo I'm sorry I can't get more specific, but that's kind of the
problem: This issue is really hard to diagnose properly because it
basically seems like syntax folding has some horrible worst case
performance which only shows once in a while, in large buffers, but once
it manifests you can basically give up on using Vim properly.

When I disable syntax folding and reload the buffer using :edit Vim is
instantly fast again. Strange enough it seems that changing the text
folding options does not have the same effect...

- Peter

[Charles Campbell]

Does

https://vimhelp.appspot.com/vim_faq.txt.html#faq-29.7

help?
Chip Campbell

[Peter Odding]

>>>>> Have you tried the advice at
>>>>> http://vim.wikia.com/wiki/Keep_folds_closed_while_inserting_text?
>>>>

>>>> I use this tip, and yet occasionally encounter such slowdowns (often

>>>> whenediting large Vim scripts (ft=vim)). There's noticeable delay in

>>>> moving the cursor, and insert-completion (i_CTRL-N) is also painfully slow
>>>

>>> Yeah I realize it's not a full solution. Is the issue pretty much
>>> known, it just needs a good fix figured out? Or does it still need
>>> investigation? Does the OP's backtrace help at all? I admit I've not
>>> used gdb enough to know how to use it effectively, but it looks like
>>> all it points to is "somewhere in the syntax folding routines Vim is
>>> being slow".
>>

>> ...

>>
>> My friend suggested I use the tip on the Vim wiki but it doesn't
>> really solve the root problem, it just makes it occur less often, but
>> once the "bug" triggers it's just as big of a problem :-\
>>

> Does
>
> https://vimhelp.appspot.com/vim_faq.txt.html#faq-29.7
>
> help?

Thanks for the suggestion Charles, but the tip you linked to comes down
to the same automatic commands in the link at the top of the quoted
text, which is the tip I was referring to when I said I tried the tip on
the Vim wiki but it didn't help that much -- I've been using that tip
for more than a month and the slow downs still bug me every day.

Maybe Vim is just not ready for syntax based folding of large buffers. I
know some of my file type plug-ins are pushing Vim to its limits, but
after investing several years getting to know Vim and its scripting
language and writing thousands of lines of Vim script I'm not yet ready
to give up on Vim :-)

Anyway in the worst case I'll just disable syntax folding for buffers
where line('$') > 1000 holds. At least then I still get to enjoy all of
the other goodies Vim has to offer. I'm definitely not switching back to
an IDE :-P. I'll keep vim-dev up to date in case I make any progress on
this issue.

- Peter

PS. I did find a nasty bug in the file type plug-in I mentioned earlier;
automatic commands were stacking up instead of being cleared before
definition. If this turns out to have caused my issue (and the tip is in
fact a valid workaround) I'll report back here in a few days.

[Ben Fritz]

On Nov 9, 4:22 pm, Peter Odding <pe...@peterodding.com> wrote:
>
> Anyway in the worst case I'll just disable syntax folding for buffers
> where line('$') > 1000 holds. At least then I still get to enjoy all of
> the other goodies Vim has to offer. I'm definitely not switching back to
> an IDE :-P. I'll keep vim-dev up to date in case I make any progress on
> this issue.
>

I think another good worst-case workaround, would be to enable syntax
folding for the initial buffer load, then switch to manual. Folds
won't update, but the folds present when opening the file will at
least be there. You could even switch to syntax folding and back to
manual again on a BufWritePost autocmd to refresh the folds at a point
in your editing where they should be pretty stable.

Still not as good as fixing the underlying issue, obviously.

[Adrien "Axioplase" Piérard]

2011/11/10 Peter Odding <pe...@peterodding.com>:

> Like Ingo I'm sorry I can't get more specific, but that's kind of the
> problem: This issue is really hard to diagnose properly because it basically…
> seems like syntax folding has some horrible worst case performance which
> only shows once in a while, in large buffers, but once it manifests you can
> basically give up on using Vim properly.
>
> When I disable syntax folding and reload the buffer using :edit Vim is
> instantly fast again. Strange enough it seems that changing the text folding
> options does not have the same effect...

That reminds me of a "bug" I found last week after *years* of painful
latex edition.
Sometimes, for a reason, editing would get really slow. It'd take a
second or two to just go into insert mode.
I would solve (temporarily) the issue with :syntax sync minlines=200,
until I trigger it again.

Here's the fix:

" Ok, so the one below *kills* LaTeX edition! Let's never do that damn mistake
" again. Do not uncomment!
"autocmd BufEnter * :syntax sync fromstart " ensure every file does
syntax highlighting (full)

The trigger was a shell script doing a remote call to vim and which,
in some cases, would reload the file and jump at a given line.
A sufficiently big latex file (500 lines) would often be enough to
start the slow-down…

Since syntax based folding seems to be, well, syntax, I wonder whether
you have the same issue as me before…

--
Français, English, 日本語, 한국어

[Peter Odding]

Thanks for the suggestion. I wasn't sure whether this affected me so I
looked through the code and documentation. The slow down happens for me
in Python scripts and I'm using the script $VIMRUNTIME/syntax/python.vim
from Vim 7.3 except that my customized file type plug-in also defines
several syntax fold rules (but doesn't change syncing). The Python
syntax script contains the following lines:

if exists("python_slow_sync") && python_slow_sync != 0
syn sync minlines=2000
else
" This is fast but code inside triple quoted strings screws it up. It
" is impossible to fix because the only way to know if you are inside
" a triple quoted string is to start from the beginning of the file.
syn sync match pythonSync grouphere NONE "):$"
syn sync maxlines=200
endif

I don't have python_slow_sync defined so it's executing the else block.
The code there is not exactly equivalent to "syntax sync fromstart" but
of course it can still perform badly.

- Peter

[Peter Odding]

Thanks for the suggestion, that's much better than just disabling syntax
folding altogether! I actually think I won't mind that much if the
folding is only updated when I :write.

- Peter

[Christian Brabandt]

Hi Peter!

Has anybody an easy to reproduce example of the problem?

Mit freundlichen Gr��en
Christian
--