Bug in ftdetect?

[Frew Schmidt]

I recently discovered that using ftdetect is absurdly slower than filetype.vim. To reproduce, try this:

mkdir -p ~/.vim/ftdetect
echo 'autocmd BufNew,BufNewFile,BufRead *.md :set filetype=markdown' > ~/.vim/ftdetect/markdown.vim
mkdir testing
cd testing
touch {1..500}.md
vim
:args *

Here's vim --version:

VIM - Vi IMproved 8.0 (2016 Sep 12, compiled Mar 17 2017 12:13:35)
Included patches: 1-95
Modified by pkg-vim-m...@lists.alioth.debian.org
Compiled by pkg-vim-m...@lists.alioth.debian.org
Huge version with GTK3 GUI. Features included (+) or not (-):
+acl +file_in_path +mouse_sgr +tag_old_static
+arabic +find_in_path -mouse_sysmouse -tag_any_white
+autocmd +float +mouse_urxvt +tcl
+balloon_eval +folding +mouse_xterm +termguicolors
+browse -footer +multi_byte +terminfo
++builtin_terms +fork() +multi_lang +termresponse
+byte_offset +gettext -mzscheme +textobjects
+channel -hangul_input +netbeans_intg +timers
+cindent +iconv +num64 +title
+clientserver +insert_expand +packages +toolbar
+clipboard +job +path_extra +user_commands
+cmdline_compl +jumplist +perl +vertsplit
+cmdline_hist +keymap +persistent_undo +virtualedit
+cmdline_info +lambda +postscript +visual
+comments +langmap +printer +visualextra
+conceal +libcall +profile +viminfo
+cryptv +linebreak -python +vreplace
+cscope +lispindent +python3 +wildignore
+cursorbind +listcmds +quickfix +wildmenu
+cursorshape +localmap +reltime +windows
+dialog_con_gui +lua +rightleft +writebackup
+diff +menu +ruby +X11
+digraphs +mksession +scrollbind -xfontset
+dnd +modify_fname +signs +xim
-ebcdic +mouse +smartindent +xpm
+emacs_tags +mouseshape +startuptime +xsmp_interact
+eval +mouse_dec +statusline +xterm_clipboard
+ex_extra +mouse_gpm -sun_workshop -xterm_save
+extra_search -mouse_jsbterm +syntax
+farsi +mouse_netterm +tag_binary
system vimrc file: "$VIM/vimrc"
user vimrc file: "$HOME/.vimrc"
2nd user vimrc file: "~/.vim/vimrc"
user exrc file: "$HOME/.exrc"
system gvimrc file: "$VIM/gvimrc"
user gvimrc file: "$HOME/.gvimrc"
2nd user gvimrc file: "~/.vim/gvimrc"
defaults file: "$VIMRUNTIME/defaults.vim"
system menu file: "$VIMRUNTIME/menu.vim"
fall-back for $VIM: "/usr/share/vim"
Compilation: gcc -c -I. -Iproto -DHAVE_CONFIG_H -DFEAT_GUI_GTK -pthread -I/usr/include/gtk-3.0 -I/usr/include/at-spi2-atk/2.0 -I/usr/include/at-spi-2.0 -I/usr/include/dbus-1.0 -I/usr/lib/x86_64-linux-gnu/dbus-1.0/include -I/usr/include/gtk-3.0 -I/usr/include/gio-unix-2.0/ -I/usr/include/mirclient -I/usr/include/mircore -I/usr/include/mircookie -I/usr/include/cairo -I/usr/include/pango-1.0 -I/usr/include/harfbuzz -I/usr/include/pango-1.0 -I/usr/include/atk-1.0 -I/usr/include/cairo -I/usr/include/pixman-1 -I/usr/include/freetype2 -I/usr/include/libpng16 -I/usr/include/gdk-pixbuf-2.0 -I/usr/include/libpng16 -I/usr/include/glib-2.0 -I/usr/lib/x86_64-linux-gnu/glib-2.0/include -Wdate-time -g -O2 -fdebug-prefix-map=/build/vim-8krYYf/vim-8.0.0095=. -fPIE -fstack-protector-strong -Wformat -Werror=format-security -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1
Linking: gcc -L. -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,now -fstack-protector -rdynamic -Wl,-export-dynamic -Wl,-E -Wl,-Bsymbolic-functions -fPIE -pie -Wl,-z,relro -Wl,-z,now -Wl,--as-needed -o vim -lgtk-3 -lgdk-3 -lpangocairo-1.0 -lpango-1.0 -latk-1.0 -lcairo-gobject -lcairo -lgdk_pixbuf-2.0 -lgio-2.0 -lgobject-2.0 -lglib-2.0 -lSM -lICE -lXpm -lXt -lX11 -lXdmcp -lSM -lICE -lm -ltinfo -lnsl -lselinux -lacl -lattr -lgpm -ldl -L/usr/lib -llua5.2 -Wl,-E -fstack-protector-strong -L/usr/local/lib -L/usr/lib/x86_64-linux-gnu/perl/5.24/CORE -lperl -ldl -lm -lpthread -lcrypt -L/usr/lib/python3.5/config-3.5m-x86_64-linux-gnu -lpython3.5m -lpthread -ldl -lutil -lm -L/usr/lib/x86_64-linux-gnu -ltcl8.6 -ldl -lz -lpthread -lieee -lm -lruby-2.3 -lpthread -lgmp -ldl -lcrypt -lm

The docs *do* say that if you are detecting files based purely on names you should instead use an autload group in a filetype.vim file, but it's pretty far down in the docs and a *lot* of people seem to have made this mistake: https://github.com/search?p=4&q=%22set+ft%3Dmarkdown%22+language%3Avim&type=Code&utf8=%E2%9C%93

I would like to know why this is slower, but more, if filetype.vim needs to be used for filename based filetype detection, and functions need to be defined in scripts.vim for content detection, what good is ftdetect anyway?

Thanks,
fREW

[Frew Schmidt]

FYI I reproduced this with a recent checkout and also a 7.3 build, so I'm pretty sure it's neither fixed nor a regression. I will look into profiling the issue, but I am still baffled.

[Frew Schmidt]

See attached the profiles for both versions.

[Bram Moolenaar]

Frew Schmidt wrote:

> I recently discovered that using ftdetect is absurdly slower than filetype.vim. To reproduce, try this:
>
> mkdir -p ~/.vim/ftdetect
> echo 'autocmd BufNew,BufNewFile,BufRead *.md :set filetype=markdown' > ~/.vim/ftdetect/markdown.vim
> mkdir testing
> cd testing
> touch {1..500}.md
> vim
> :args *

What do you compare this with? Filetype detection already has this rule
for markdown, thus the extra markdown.vim file isn't needed.

Perhaps you can get some more information by setting 'verbose' to a
non-zero number, e.g. ten.

From your logs it looks like one run detects the filetype only for the
one file that is loaded, while the other does it for every file in the
argiument list. Looks like something causes loading of every file.

Each sourced file has overhead, the more files you add that are sourced
when opening a file the slower Vim gets.

The ftdetect files are good when you want to just drop a file in a
directory. Installing a package is simpler that way. Otherwise
installing a package for a file type would require editing
the filetype.vim script.

--
SOLDIER: What? Ridden on a horse?
ARTHUR: Yes!
SOLDIER: You're using coconuts!
"Monty Python and the Holy Grail" PYTHON (MONTY) PICTURES LTD

/// Bram Moolenaar -- Br...@Moolenaar.net -- http://www.Moolenaar.net \\\
/// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
\\\ an exciting new programming language -- http://www.Zimbu.org ///
\\\ help me help AIDS victims -- http://ICCF-Holland.org ///

[fREW Schmidt]

The fast version is a filetype.vim script in my ~/. vim. If there is a cost to these settings, why does it show up with ftdetect but not a local filetype.vim?

Also most machines are not on vim 8, and thus still think .md is modula 2.

--
Sent from a telephone. Pardon my brevity.

On May 23, 2017 1:52 AM, "Bram Moolenaar" <Br...@moolenaar.net> wrote:

Frew Schmidt wrote:

> I recently discovered that using ftdetect is absurdly slower than filetype.vim.  To reproduce, try this:
>
>      mkdir -p ~/.vim/ftdetect
>      echo 'autocmd BufNew,BufNewFile,BufRead *.md :set filetype=markdown' >  ~/.vim/ftdetect/markdown.vim
>      mkdir testing
>      cd testing
>      touch {1..500}.md
>      vim
>      :args *

What do you compare this with?  Filetype detection already has this rule
for markdown, thus the extra markdown.vim file isn't needed.

Perhaps you can get some more information by setting 'verbose' to a
non-zero number, e.g. ten.

From your logs it looks like one run detects the filetype only for the
one file that is loaded, while the other does it for every file in the
argiument list.  Looks like something causes loading of every file.

> Here's vim --version:
>
>      VIM - Vi IMproved 8.0 (2016 Sep 12, compiled Mar 17 2017 12:13:35)
>      Included patches: 1-95

>      Modified by pkg-vim-maintainers@lists.alioth.debian.org
>      Compiled by pkg-vim-maintainers@lists.alioth.debian.org

[Frew Schmidt]

> Perhaps you can get some more information by setting 'verbose' to a
>
> non-zero number, e.g. ten.

Well, I ran it with -V10 and even after deleting both ~/.vim/ (except for the ftdetect file) and ~/.vimrc and I can still reproduce it. It looks about the same as before: the markdown support is getting reloaded for every file instead of once at the beginning. I don't know how to store :messages in a file or I'd attach them.

[Frew Schmidt]

Another interesting detail: it's not slow if I set the filetype to sillybonk or perl, so this is somehow related to the markdown support. I think I'll bisect the markdown files and see which one causes it, and then bisect the file itself.

[Frew Schmidt]

If I replace the contents of /usr/share/vim/vim80/syntax/vb.vim with `let b:current_syntax = "vb"` everything is fast and, as I would hope, the support files are not loaded N times per buffer but instead exactly once.

I verified as well by putting the contents back and setting the filetype to `vb` and I can continue to reproduce the issue.

I still don't know why ftdetect causes this and filetype.vim does not, and I also don't see what's wrong with the vb support files.

[Bram Moolenaar]

I'm afraid this doesnt make any sense. I would suggest to reinstall
Vim, making sure you first delete all Vim related files (make a backup
of your preferences somewhere).

--
THEOREM: VI is perfect.
PROOF: VI in roman numerals is 6. The natural numbers < 6 which divide 6 are
1, 2, and 3. 1+2+3 = 6. So 6 is a perfect number. Therefore, VI is perfect.
QED
-- Arthur Tateishi

[Frew Schmidt]

On Tuesday, May 23, 2017 at 12:58:16 PM UTC-7, Bram Moolenaar wrote:
> I'm afraid this doesnt make any sense. I would suggest to reinstall
> Vim, making sure you first delete all Vim related files (make a backup
> of your preferences somewhere).

Well I have already repro'd this on a fresh 7.3, 7.4, 8.0, and system 8.0. Here's a reproduction using docker, with no dotfiles or even system vim installed:

$ docker run -it --rm library/buildpack-deps bash -l
$ git clone git://github.com/vim/vim
$ make
$ make install
$ mkdir -p ~/.vim/ftdetect
$ echo 'autocmd BufNew,BufNewFile,BufRead *.md :set filetype=markdown' > ~/.vim/ftdetect/markdown.vim
$ mkdir testing
$ cd testing
$ touch {1..500}.md
$ vim
:args *

It still happens, and this is from master, today.

If you don't have time to check this yourself I get it. I have multiple
workarounds at this point.

Thanks,
fREW Schmidt

[Antony Scriven]

Try taking out the BufNew autocmd. That'll get triggered for every
file in the args list, whereas BufRead is only triggered for the first
one that's actually edited. FWIW 'BufNew' is not in the template
ftdetect file in the docs. --Antony

> --
> --
> You received this message from the "vim_use" maillist.
> Do not top-post! Type your reply below the text you are replying to.
> For more information, visit http://www.vim.org/maillist.php
>
> ---
> You received this message because you are subscribed to the Google Groups "vim_use" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to vim_use+u...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Antony Scriven]

P.S. If you want to detect name changes, there's BufFilePost. --Antony

[fREW Schmidt]

It is sorta weird that the vb syntax, is somehow part of the problem,
but whatever I understand that this was the wrong way to configure the
editor.

> You received this message because you are subscribed to a topic in the Google Groups "vim_use" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/vim_use/m7t32w_n6wc/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to vim_use+u...@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.

--
fREW Schmidt
https://blog.afoolishmanifesto.com

[fREW Schmidt]

Aha! That's it Antony; thanks; that explains it and a lot more. Sorry
for inexplicably missing this myself.

On Wed, May 24, 2017 at 08:17:46AM +0100, Antony Scriven wrote:

> You received this message because you are subscribed to a topic in the Google Groups "vim_use" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/vim_use/m7t32w_n6wc/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to vim_use+u...@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.

[Christian Brabandt]

On Mi, 24 Mai 2017, fREW Schmidt wrote:

> It is sorta weird that the vb syntax, is somehow part of the problem,
> but whatever I understand that this was the wrong way to configure the
> editor.

Not sure I understand. Are you saying, that the vb syntax script is
responsible for the long loading time?

Best,
Christian
--
Du bekommst einen Computervirus genau dann, wenn Du ihn am wenigsten brauchen
kannst.

[fREW Schmidt]

Yes. "If I replace the contents of /usr/share/vim/vim80/syntax/vb.vim with `let b:current_syntax = "vb"` everything is fast and, as I would hope, the support files are not loaded N times per buffer but instead exactly once." 

--
Sent from a telephone. Pardon my brevity.

--
--
You received this message from the "vim_use" maillist.
Do not top-post! Type your reply below the text you are replying to.
For more information, visit http://www.vim.org/maillist.php

---
You received this message because you are subscribed to a topic in the Google Groups "vim_use" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/vim_use/m7t32w_n6wc/unsubscribe.

To unsubscribe from this group and all its topics, send an email to vim_use+unsubscribe@googlegroups.com.

[Christian Brabandt]

On Mi, 24 Mai 2017, fREW Schmidt wrote:

> Yes. "If I replace the contents of /usr/share/vim/vim80/syntax/vb.vim with `let
> b:current_syntax = "vb"` everything is fast and, as I would hope, the support
> files are not loaded N times per buffer but instead exactly once." 

Well, I see the problem, although I cannot reproduce it.

The markdown syntax script sources the html syntax script and that one
does include a couple of other syntax scripts, like css.vim,
javascript.vim and also vb.vim. However from my testing vb.vim is not
guilty alone. If I uncomment vb.vim, the overall loading time only
shrinks by 0.2 seconds or so (it needs here a total of around 7
seconds). So you would need to disable all other to be included syntax
scripts as well (for a test, you could set main_syntax='java').

However, I think you might have found a bug here. Your ftdetect script
does this:

autocmd BufNew,BufNewFile,BufRead *.md :set filetype=markdown

The :args command does fill the argument list, calls ex_next() which
calls eventually alist_set() and then iterates over the argument list
and calls adlist_add(). This will finally call buflist_add() which in
turn calls buflist_new(), which triggers the BufNew autocommand, which
then triggers the FileType and finally the Syntax autocommands. So in
the end, we are iterating over the whole list of arguments and call
autocommands, for buffers that are not even going to be displayed next.

So I wonder, if we can't do a little bit better, similar to how vimgrep
sets the eventignore option to prevent at least the expensive FileType
autocommand:

diff --git a/src/ex_cmds2.c b/src/ex_cmds2.c
index 36ee57e8e..e0fbbfb9b 100644
--- a/src/ex_cmds2.c
+++ b/src/ex_cmds2.c
@@ -2757,8 +2757,22 @@ ex_next(exarg_T *eap)
{
if (*eap->arg != NUL) /* redefine file list */
{
+ /* do_arglist() could trigger BufNew events, which might
+ * in turn trigger expensive FileType and Syntax autocommands
+ * prevent this for now */
+#if defined(FEAT_AUTOCMD) && defined(FEAT_SYN_HL)
+ char_u *save_ei = au_event_disable(",Filetype");
+#endif
if (do_arglist(eap->arg, AL_SET, 0) == FAIL)
+ {
+#if defined(FEAT_AUTOCMD) && defined(FEAT_SYN_HL)
+ au_event_restore(save_ei);
+#endif
return;
+ }
+#if defined(FEAT_AUTOCMD) && defined(FEAT_SYN_HL)
+ au_event_restore(save_ei);
+#endif
i = 0;
}
else


Best,
Christian
--
Wie man sein Kind nicht nennen sollte:
Don Erstag

[Frew Schmidt]

I'm glad you were able to come up with a solution! So I guess `vim *` goes through a different codepath than `:args *`? Note that this also affects the quickfix, the location list, and the local args list. Initially I discovered this issue with `:cexpr` and reproduced it with `:lexpr` and `:args`/`:largs`, but it was easiest to reproduce with `:args *`.

Maybe a dumb question, but why would BufNew be more expensive than BufRead? Shouldn't the filetype stuff get run either way? I wouldn't expect BufRead to take 500ms and BufNew to take multiple seconds, would you?

[Antony Scriven]

On May 24, Frew Schmidt wrote:

> [...]

>
> Maybe a dumb question, but why would BufNew be more expensive
> than BufRead? Shouldn't the filetype stuff get run either way?
> I wouldn't expect BufRead to take 500ms and BufNew to take
> multiple seconds, would you?

The bufnew autocommands happen all at once. Change your autocommand
to this instead:

au bufnew *.md echo 'markdown bufnew'
au bufread *.md echo 'markdown bufread'

Then :args * and see what happens.

I think you should still remove the 'BufNew' from your autocommand
(Christian's helpful-looking patch notwithstanding). I can't see
what purpose it serves other than to detect name changes. If that's
a real requirement, there's buffilepost. --Antony

[fREW Schmidt]

Ok so the BufNew's happen at *allocation* time, where the BufRead's
happen when you actually pull up the buffer. That makes sense.
Thanks, that explains a lot.

Yeah the BufNew was definitely cargo-cult and I'll remove it, but it
just seemed weird that it would cause this kind of slowdown.

[Antony Scriven]

On May 24, Christian Brabandt wrote:

> [...]

>
> However, I think you might have found a bug here. Your
> ftdetect script does this:

I don't know that I'd characterise it as a bug, since this:

> autocmd BufNew,BufNewFile,BufRead *.md :set filetype=markdown
>
> The :args command does fill the argument list, calls
> ex_next() which calls eventually alist_set() and then

> iterates over the argument list [... firing off
> autocommands ...]. So in the end, we are iterating over

> the whole list of arguments and call autocommands, for
> buffers that are not even going to be displayed next.

... is what I'd expect from reading :h bufnew.

> [... patch to prevent filetype autocmd firing during
> loading the args list ...]

Bug or not, this seems like it could be helpful. --Antony

[Christian Brabandt]

Fullquote, because I am CC'ing Bram for comment

I am not sure, if this behaviour is a bug. However after thinking some
more about it, I am wondering, whether the fact that the BufNew
autocommand triggers a FileType autocommand is a bug itself, since
autocommands shouldn't nest (see :h autocmd-nested). But I haven't
looked into the detail, why the FileType autocommand is triggered. (And
I have a feeling, that when this behaviour is changed, it might actually
break a lot of ftdetect scripts.)

Best,
Christian
--
Glaube heißt nicht wissen wollen, was wahr ist.
-- Friedrich Wilhelm Nietzsche (Der Antichrist)

[Christian Brabandt]

On Mi, 24 Mai 2017, Frew Schmidt wrote:

> I'm glad you were able to come up with a solution!

So much for the unwillingness of the vim maintainers to fix this issue.

> So I guess `vim *` goes through a different codepath than `:args *`?

Don't think so. Perhaps the autocommand wasn't set up by the time the
argument list was filled.

> Note that this also affects the quickfix, the location list, and the
>local args list. Initially I discovered this issue with `:cexpr` and
>reproduced it with `:lexpr` and `:args`/`:largs`, but it was easiest to
>reproduce with `:args *`.

That is a different story here, I think my proposed patch wouldn't
change that, but I think the reason is similar, the BufNew autocommand
triggering the FileType detection. In fact while looking into a profile,
it looked like the most expensive call was the syn clear command from
the SynSet function that was called for each file. However I am not
sure, why the FileType and Syntax autocommands are actually triggered
from the BufNew event, since they shouldn't nest (see other message from
me in this thread). So perhaps we need a similar change here as well.
Can you provide a reproducible example for the :cexpr case please)?

Best,
Christian
--
Tugend nennt man die Summe der Dinge, die wir aus Trägheit, Feigheit
oder Dummheit nicht getan haben.
-- Henry Miller

[Antony Scriven]

On May 25, Christian Brabandt wrote:

> [...]

>
> I am not sure, if this behaviour is a bug. However after
> thinking some more about it, I am wondering, whether the
> fact that the BufNew autocommand triggers a FileType
> autocommand is a bug itself, since autocommands shouldn't
> nest (see :h autocmd-nested). But I haven't looked into
> the detail, why the FileType autocommand is triggered.
> (And I have a feeling, that when this behaviour is
> changed, it might actually break a lot of ftdetect
> scripts.)

I was under the impression that nesting only applied to :e
and :w. The documentation at `:h setfiletype` implies to me
that it does nest, hence the inclusion of that command. You
can even set 'ft' within a FileType autocommand and then
trigger a different FileType autocommand. --Antony

[fREW Schmidt]

On Thu, May 25, 2017 at 09:14:06AM +0200, Christian Brabandt wrote:
> So perhaps we need a similar change here as well. Can you provide a
> reproducible example for the :cexpr case please)?

Sure; same setup as before, and in vim:

:cexpr system('ls * \| xargs -n1 -I{} echo {}:1:{}')

[Bram Moolenaar]

Loading syntax, indent and other plugins takes time. No way to avoid that.
The filetype is only detected (and side effects applied) when the buffer is
actually used (loaded), exactly to avoid that overhead.

Even detecting the file type may require loading the file (e.g. for shell
scripts), thus creating the buffer is too early for that.

--
Friends? I have lots of friends! In fact, I have all episodes ever made.

[Bram Moolenaar]

Christian Brabandt wrote:

> On Mi, 24 Mai 2017, fREW Schmidt wrote:
>
> > Yes. "If I replace the contents of /usr/share/vim/vim80/syntax/vb.vim with `let
> > b:current_syntax = "vb"` everything is fast and, as I would hope, the support
> > files are not loaded N times per buffer but instead exactly once." 
>
> Well, I see the problem, although I cannot reproduce it.
>
> The markdown syntax script sources the html syntax script and that one
> does include a couple of other syntax scripts, like css.vim,
> javascript.vim and also vb.vim. However from my testing vb.vim is not
> guilty alone. If I uncomment vb.vim, the overall loading time only
> shrinks by 0.2 seconds or so (it needs here a total of around 7
> seconds). So you would need to disable all other to be included syntax
> scripts as well (for a test, you could set main_syntax='java').

I haven't read the whole thread, but I suspect that the trick is to set
b:current_syntax (to anything), which will cause further syntax scripts to bail
out. Anyway, this is all working around the actual problem.

> However, I think you might have found a bug here. Your ftdetect script
> does this:
>
> autocmd BufNew,BufNewFile,BufRead *.md :set filetype=markdown
>
> The :args command does fill the argument list, calls ex_next() which
> calls eventually alist_set() and then iterates over the argument list
> and calls adlist_add(). This will finally call buflist_add() which in
> turn calls buflist_new(), which triggers the BufNew autocommand, which
> then triggers the FileType and finally the Syntax autocommands. So in
> the end, we are iterating over the whole list of arguments and call
> autocommands, for buffers that are not even going to be displayed next.
>
> So I wonder, if we can't do a little bit better, similar to how vimgrep
> sets the eventignore option to prevent at least the expensive FileType
> autocommand:

I'm quite sure this will make other things fail. Just don't use BufNew for
detecting the filetype, that is the cause of the problem. The file type should
only be detected when the buffer is loaded. Doing it earlier indeed causes the
overhead experienced.

--
Looking at Perl through Lisp glasses, Perl looks atrocious.

[Christian Brabandt]

On Do, 25 Mai 2017, Bram Moolenaar wrote:

> Christian Brabandt wrote:
>
> > On Mi, 24 Mai 2017, fREW Schmidt wrote:
> >
> > > Yes. "If I replace the contents of /usr/share/vim/vim80/syntax/vb.vim with `let
> > > b:current_syntax = "vb"` everything is fast and, as I would hope, the support
> > > files are not loaded N times per buffer but instead exactly once." 
> >
> > Well, I see the problem, although I cannot reproduce it.
> >
> > The markdown syntax script sources the html syntax script and that one
> > does include a couple of other syntax scripts, like css.vim,
> > javascript.vim and also vb.vim. However from my testing vb.vim is not
> > guilty alone. If I uncomment vb.vim, the overall loading time only
> > shrinks by 0.2 seconds or so (it needs here a total of around 7
> > seconds). So you would need to disable all other to be included syntax
> > scripts as well (for a test, you could set main_syntax='java').
>
> I haven't read the whole thread, but I suspect that the trick is to set
> b:current_syntax (to anything), which will cause further syntax scripts to bail
> out. Anyway, this is all working around the actual problem.

Not for html, which explicitly resets b:current_syntax to be able to
include several other syntax scripts and even provides the main_syntax
global variable to make this work with other syntax scripts.

>
> > However, I think you might have found a bug here. Your ftdetect script
> > does this:
> >
> > autocmd BufNew,BufNewFile,BufRead *.md :set filetype=markdown
> >
> > The :args command does fill the argument list, calls ex_next() which
> > calls eventually alist_set() and then iterates over the argument list
> > and calls adlist_add(). This will finally call buflist_add() which in
> > turn calls buflist_new(), which triggers the BufNew autocommand, which
> > then triggers the FileType and finally the Syntax autocommands. So in
> > the end, we are iterating over the whole list of arguments and call
> > autocommands, for buffers that are not even going to be displayed next.
> >
> > So I wonder, if we can't do a little bit better, similar to how vimgrep
> > sets the eventignore option to prevent at least the expensive FileType
> > autocommand:
>
> I'm quite sure this will make other things fail. Just don't use
> BufNew for detecting the filetype, that is the cause of the problem.
> The file type should only be detected when the buffer is loaded.
> Doing it earlier indeed causes the overhead experienced.

Indeed, however, that doesn't mean, we shouldn't try to avoid this
problem. Vimgrep does the same.

Best,
Christian
--
Am Ende ist alles ein Witz.
-- Charlie Chaplin

[Bram Moolenaar]

Christian Brabandt wrote:

> On Do, 25 Mai 2017, Bram Moolenaar wrote:
>
> > Christian Brabandt wrote:
> >
> > > On Mi, 24 Mai 2017, fREW Schmidt wrote:
> > >
> > > > Yes. "If I replace the contents of /usr/share/vim/vim80/syntax/vb.vim with `let
> > > > b:current_syntax = "vb"` everything is fast and, as I would hope, the support
> > > > files are not loaded N times per buffer but instead exactly once." 
> > >
> > > Well, I see the problem, although I cannot reproduce it.
> > >
> > > The markdown syntax script sources the html syntax script and that one
> > > does include a couple of other syntax scripts, like css.vim,
> > > javascript.vim and also vb.vim. However from my testing vb.vim is not
> > > guilty alone. If I uncomment vb.vim, the overall loading time only
> > > shrinks by 0.2 seconds or so (it needs here a total of around 7
> > > seconds). So you would need to disable all other to be included syntax
> > > scripts as well (for a test, you could set main_syntax='java').
> >
> > I haven't read the whole thread, but I suspect that the trick is to set
> > b:current_syntax (to anything), which will cause further syntax scripts to bail
> > out. Anyway, this is all working around the actual problem.
>
> Not for html, which explicitly resets b:current_syntax to be able to
> include several other syntax scripts and even provides the main_syntax
> global variable to make this work with other syntax scripts.

Well, it must be used somehwere to explain why making the file empty has
a different effect from only setting b:current_syntax in the file.

> > > However, I think you might have found a bug here. Your ftdetect script
> > > does this:
> > >
> > > autocmd BufNew,BufNewFile,BufRead *.md :set filetype=markdown
> > >
> > > The :args command does fill the argument list, calls ex_next() which
> > > calls eventually alist_set() and then iterates over the argument list
> > > and calls adlist_add(). This will finally call buflist_add() which in
> > > turn calls buflist_new(), which triggers the BufNew autocommand, which
> > > then triggers the FileType and finally the Syntax autocommands. So in
> > > the end, we are iterating over the whole list of arguments and call
> > > autocommands, for buffers that are not even going to be displayed next.
> > >
> > > So I wonder, if we can't do a little bit better, similar to how vimgrep
> > > sets the eventignore option to prevent at least the expensive FileType
> > > autocommand:
> >
> > I'm quite sure this will make other things fail. Just don't use
> > BufNew for detecting the filetype, that is the cause of the problem.
> > The file type should only be detected when the buffer is loaded.
> > Doing it earlier indeed causes the overhead experienced.
>
> Indeed, however, that doesn't mean, we shouldn't try to avoid this
> problem. Vimgrep does the same.

Avoid what problem? Detecting the filetype in cases it isn't used? Not
sure how we can predict that. For :vimgrep it's obvious, and it then
has to carefully jump to hoops later for when a buffer ends up not being
unloaded. Probably can't be turned into a generic solution, since we
don't know what happens next.

--
OLD WOMAN: Well, how did you become king, then?
ARTHUR: The Lady of the Lake, her arm clad in the purest shimmering samite,
held Excalibur aloft from the bosom of the water to signify by Divine
Providence ... that I, Arthur, was to carry Excalibur ... That is
why I am your king!

"Monty Python and the Holy Grail" PYTHON (MONTY) PICTURES LTD