Wrapg text at col. 80 on large-ish files - performance aspects.

[Chris Jones]

What is the recommended way to wrap files at column 80 or thereabout
when dealing with 5,000-10,000 lines files without a major performance
hit?

I have tried issuing a ':set syntax off' command followed by running
different flavors of the gw command in normal or visual mode but this
may not be the most efficient approach performance-wise.

I tried searching for lines longer than 80 characters and running 'gwap'
on the paragraph where they appear using the :g/long-line/gwap but this
doesn't seem to help.

It takes something like 10 minutes to complete this task suggesting I am
not doing it right.

Is there a better way to do it inside a vim session or should I exit vim
and run one of a number of utilities from the shell command prompt?

I would prefer staying with my vim session for reasons not directly
related to the issue.

Thank you.

CJ

[Tim Chase]

On 2020-03-15 11:44, Chris Jones wrote:
> What is the recommended way to wrap files at column 80 or thereabout
> when dealing with 5,000-10,000 lines files without a major
> performance hit?

Strange. I do this regularly and have never encountered any issues.

Have you attempted it without any plugins/non-defaults?

$ vim -u NONE file_with_long_lines.txt

Also, is there any difference if you use "gq" to reformat instead of
"gw" (maybe there's something weird in the "maintain the cursor
location" code)?

> I tried searching for lines longer than 80 characters and running
> 'gwap' on the paragraph where they appear using the
> :g/long-line/gwap but this doesn't seem to help.

I presume you mean something like

:g/\%80c/norm gww

> It takes something like 10 minutes to complete this task suggesting
> I am not doing it right.

For a file that small, that's a ridiculous amount of time, so your
gut reaction is spot on. But I'd far sooner point the finger at some
rogue plugin than point it at something you're "not doing right",
'cuz you *are* doing it right.

> Is there a better way to do it inside a vim session or should I
> exit vim and run one of a number of utilities from the shell
> command prompt?

The fmt(1) utility might lend a hand here, whether invoked inside vim:

:%!fmt

or externally

$ fmt -80 file_with_long_lines.txt > wrapped_file.txt

There's also a fold(1) utility that behaves similarly, with subtle
nuances in their differences, but for the most part, they should
behave roughly the same for this use-case.

Sorry that doesn't help on Windows by default, but fmt(1)/wrap(1)
should work on any Linux, Mac, or BSD box.

-tim

[Chris Jones]

[...]

With vim - u NONE... & a set tw=80 a (normal) gggwG is quasi instaneous
on a 5830 lines test file. With vim the way I have it set up ( that's
what I really meant by 'doing it wrong'...) the same scenario took 15-16
minutes to complete.

I don't have that many plugins installed/loaded (cscope, zoomwin, and
a few others that shouldn't interfere with buffer reformatting (they
look like the kind that presumably just sits there doing nothing until
you call them explicitly) so my impression is that this must be related
to stuff that might cause vim to refresh the screen automatically such
as perhaps my custom tabline and statusline (?)... after all if either
vim or a widely-used plugin was the culprit the chances are other people
would have had the problem & reported it.

In any event I'll take a hard look at all the crud that has accumulated
in my vim environment (~/.vimrc & ~/.vim/ directory) over the last 20
years and disable it.

As recommended I also gave the :%!fmt workaround a shot and likewise it
completed before I had time to take my hands off the keyboard.

Since I do not have the time now to look into this further I am going to
use this for the time being and do some vim spring cleaning in a big way
some time next month. I am thinking rebuilding my vim environment one
brick at a time... something long overdue... is probably what needs to
be done at this point.

Thank you very much indeed for your quick response and excellent advice.

CJ

[Chris Jones]

Fix typo in issue Subject (:s//Wrapg/wrap/)

[Tony Mechelynck]

To see if it's your custom tabline and/or statusline, you can load Vim
with your usual vimrc, then type

:set tal= stl=

and see if it makes a difference. If it does, then retry by setting
the one but not the other to empty (both ways) to see if it's the one,
the other, or both. Then look for "slow stuff" in whichever is/are the
culprit.


Best regards,
Tony.

[Chris Jones]

On Mon, Mar 16, 2020 at 01:35:45PM EDT, Tony Mechelynck wrote:
[..]

>
> To see if it's your custom tabline and/or statusline, you can load Vim
> with your usual vimrc, then type
>
> :set tal= stl=
>
> and see if it makes a difference. If it does, then retry by setting
> the one but not the other to empty (both ways) to see if it's the one,
> the other, or both. Then look for "slow stuff" in whichever is/are the
> culprit.

Thanks. That'll save me some work/research.

CJ

[Chris Jones]

Taking the approach that this is just another symptom that cleaning up
all the crud that thas accumulated in my vim environment over the years
is long overdue... I decided to look into this further and found the
culprit: I ended up looking at an ~8-year-old mediawiki.vim (!) file in
my ~/.vim/ftdetect.

The lines that cause the problem:

setlocal foldexpr=getline(v:lnum)=~'^\\(=\\+\\)[^=]\\+\\1\\(\\s*<!--.*-->\\)\\=\\s*$'?\">\".(len(matchstr(getline(v:lnum),'^=\\+'))-1):\"=\"
setlocal fdm=expr

I imagine someone familiar with foldexpr's would see at a glance what's
wrong with the fold expression when the number of lines in the file runs
in the thousands...

A bit odd that this should take effect on markdown *.md files or *.txt
files that have nothing to do with *.mw, *.wiki, or *.mediawiki...

Anyway, after commenting out those two lines I'm no longer able to
recreate the problem... and more importantly stuff that I had initially
suspected has now been proven innocent.

I looked at the current version of the mediawiki stuff on github and it
turns out that the foldexr/foldmethod logic was moved to
ftplugin/mediawiki.vim... When I have time I should probably take a look
at the current version... see if this issue (at least in my context) has
been addressed.

Thanks,

CJ

[Mateusz Okulus]

On 20/06/08 01:23PM, Chris Jones wrote:
> I imagine someone familiar with foldexpr's would see at a glance what's
> wrong with the fold expression when the number of lines in the file runs
> in the thousands...

Fold expressions are powerful, but they are evaluated for every line,
and can be slow because of that. It's slow because of the expensive
regex, equivalent to: https://regex101.com/r/fx4iPC/2.

You can simplify it to just match several equal signs at the beginning
of the line:

setlocal foldexpr=getline(v:lnum)=~'^=\\+'?\">\".(len(matchstr(getline(v:lnum),'^=\\+'))-1):\"=\"

It should be much faster.

What it does:

getline(v:lnum) - get text of the current line
=~ - check if string matches a regex
'^=\\+' - one or more = at the beginning of line; this is my
simplification, might give you false positives, but
if you have correct mediawiki file it should be ok
? - part of ternary operator, like in C, check :h expr1
\">\" - literal string ">", part fold level indication,
check :h fold-expr
. - string concatenation :h expr-.
len(...)-1 - counts how many = are at the beginning of the line
and subtracts 1, so top level headings are
not folded. With previous ">" it will evaluate
to e.g. ">1" and set fold level.
: - separation in ternary operator
\"=\" - literal "=" string; If there is no match it
will use fold level from previous line,
extending the fold to what's under the header.

Regards,
Matt

[Paul]

On Mon, Jun 08, 2020 at 01:23:24PM -0400, Chris Jones wrote:
>I ended up looking at an ~8-year-old mediawiki.vim (!) file in
>my ~/.vim/ftdetect.
>
>The lines that cause the problem:
>
>setlocal foldexpr=getline(v:lnum)=~'^\\(=\\+\\)[^=]\\+\\1\\(\\s*<!--.*-->\\)\\=\\s*$'?\">\".(len(matchstr(getline(v:lnum),'^=\\+'))-1):\"=\"
>setlocal fdm=expr

…

>A bit odd that this should take effect on markdown *.md files or *.txt
>files that have nothing to do with *.mw, *.wiki, or *.mediawiki...

You have this in ftdetect, which runs regardless of which filetype you open. ftdetect is for specification of filetype. Settings per filetype should go into after/ftplugin, and will only be executed when that filetype is detected.

[Chris Jones]

On Wed, Jun 10, 2020 at 04:52:37AM EDT, Paul wrote:
>
> You have this in ftdetect, which runs regardless of which filetype you open. ftdetect is for specification of filetype. Settings per filetype should go into after/ftplugin, and will only be executed when that filetype is detected.
>

Presumably the reason was this code was moved to a plugin in later
versions of mediawiki.vim.

Thanks for clarifying.

CJ