[vim] :g in diff mode is slow (#603)

[依云]

The steps to reproduce:

1. get a large enough text file, say tens of thousands of lines (there don't need to be long lines)
2. copy it as file a and b (it doesn't matter if they are the same or different)
3. run vim -N -u NONE a, then do :g/4/d.
4. run vim -N -u NONE -d a b, then do :g/4/d in one of the buffer. it's much slower. Also undo is really slow too.

—
Reply to this email directly or view it on GitHub.

[Antonio Giovanni Colombo]

Hi everybody,

not sure if it is relevant or not, but I happen to edit VERY long files, and the problem is the "undo". The undo file can grow VERY large, for natural reasons. Since what I usually need is to filter out unneeded lines, I usually type:

:setlocal ul=-1

(that means "no undo file, please")

for the interested file. That speeds up things a lot. If I need to "remember" how the file was at a particular point, I just do :w now and then. If I want to save the original file, I just get out from the editing session with :q! (without having typed any intermediate :w ). The only inconvenience is that you have to be careful with the commands you type...

Hope it helps,

Antonio 

[Bram Moolenaar]

依云 wrote:

> The steps to reproduce:
>

> 1. get a large enough text file, say tens of thousands of lines (there

> don't need to be long lines)

> 2. copy it as file `a` and `b` (it doesn't matter if they are the same
> or different)
> 3. run `vim -N -u NONE a`, then do `:g/4/d`.
> 4. run `vim -N -u NONE -d a b`, then do `:g/4/d` in one of the buffer.

> it's much slower. Also undo is really slow too.

Try switching off folding before executing the command. Keeping folds
updated can be slow. Unfortunately Vim doesn't know if it's faster to
switch off folding, apply the changes, then switch folding back on.
With a small change it could be much slower.

--
hundred-and-one symptoms of being an internet addict:
59. Your wife says communication is important in a marriage...so you buy
another computer and install a second phone line so the two of you can
chat.

/// Bram Moolenaar -- Br...@Moolenaar.net -- http://www.Moolenaar.net \\\
/// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
\\\ an exciting new programming language -- http://www.Zimbu.org ///
\\\ help me help AIDS victims -- http://ICCF-Holland.org ///

[依云]

On Thu, Jan 28, 2016 at 06:43:31AM -0800, Bram Moolenaar wrote:
>
> 依云 wrote:
>
> > The steps to reproduce:
> >

> > 1. get a large enough text file, say tens of thousands of lines (there

> > don't need to be long lines)

> > 2. copy it as file `a` and `b` (it doesn't matter if they are the same
> > or different)

> > 3. run `vim -N -u NONE a`, then do `:g/4/d`.

> > 4. run `vim -N -u NONE -d a b`, then do `:g/4/d` in one of the buffer

[Coot]

To switch off highlighting you can use :syn off and then to re-enable it :syn enable.

[依云]

Hi, I've discovered that most of the time is used in diff_infold. Here's the flamegraph: https://img.vim-cn.com/1f/223148527b0787a5cb427ed713ab46a513859f.svg

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub

[esiegerman]

set nofoldenable doesn't improve things any, but set foldmethod=manual does the trick. (I've been testing with 8.0..42, and have actually been doing _windo_ set foldmethod=manual. Not sure whether it actually matters, though, what state the other windows are in.)

@brammool wrote:

Unfortunately Vim doesn't know if it's faster to switch off folding, apply the changes, then switch folding back on. With a small change it could be much slower.

Vim could make that decision after the first (line-marking) pass, once it knows how many lines need to be touched.

The thing is, once that hot spot (which is specifically diff_infold, as @lilydjwg says) has been dealt with, another one rears its head: diff_mark_adjust_tp. This one isn't nearly as bad, but it's still painful once files get large enough. Can that also be disabled temporarily?

FYI: Empirical results from my very limited testing -- which consists of vimdiffing two identical large files of consecutive integers and doing :g/0$/d in one of them -- show diff_infold scaling at something like O(D^3) (where D is the number of lines being deleted). The number of calls scales at O(D^2), and the CPU per call scales at about O(C^1.5) (where C is the number of calls made). Yup: as the number of deletions increases, diff_infold is called more times per deletion, and each one of those calls gets more expensive. Not sure whether that's surprising or not, to folks who know the code. I'm also not sure how important it is in practice, if folding can indeed be temporarily disabled for large values of D.

diff_mark_adjust_tp is called exactly once per deletion, i.e. it's precisely O(D), but again, its CPU usage grows, at about O(C^2).

(If these results point to things that could use improvement independent of the :g situation, let me know and I'll file separate issues for them.)

—
You are receiving this because you commented.

Reply to this email directly, view it on GitHub

[依云]

I think this has been fixed by Patch 8.1.1922.

[依云]

Closed #603.