Asynchronous grep plugin - thoughts and issues

[Ramel Eshed]

Hi All,

Thanks to Bram and his recent work on channels, I have a preliminary version of an asynchronous grep plugin which lets you work with the available results while grep is still running. I would like to take advantage of this relatively short script to raise some issues and thoughts I have. The plugin can be found at https://github.com/ramele/agrep.

My original idea was to use the quickfix list and add each match to it from the out callback. I had some issues with it, so temporarily (I hope..) I’m using my own window to display the results (“agrep window”). This is the default now and it should demonstrate what this plugin should do eventually. In order to reproduce the quickfix issues, you'll need to tell agrep to use quickfix list instead of agrep window (:let qgrep_use_qf=1). You can install the plugin as is or load it with $vim -u NONE -N -S <path to agrep.vim>

1. Stability:
Currently (7.4.1819) when :Agrep returns high number of results (~2800. So we get similar number of callbacks in a time frame of ~2:45 seconds) Vim crashes with:

Caught deadly signal SEGV
Vim: Finished.
Segmentation fault (core dumped)

This is more likely to happen when using the quickfix list. But it happens sometimes with agrep window as well. I never see this when I change the out_cb to only process the msg without adding the results to neither agrep window or qf list (-with lines 108-118 commented out).

2. Quickfix issues
a. As long as there are not too many matches and the quickfix window is closed you can fire up a search and start jumping to the available matches while the search is still running using the quickfix navigation commands (:cn, for example). The problem is when the quickfix window is opened while the search is active – here there is a serious performance issue, and Vim hangs for a while. You can see this by running :copen and then :Agrep. Choose a search that will generate many results (> 1000).

b. As the matches rate becomes higher, Vim will become unresponsive. It may happen for short intervals or it can hangs for a while (for very high rate). By “rate” I simply mean to number_of_matches / sec. To compare, the performance is much better with the agrep window.

As I understand (and please correct me if I’m wrong) when using setqflist() to create a new list or add items to the current list, Vim reads all the buffers (as an unlisted buffers) in order to set the hidden marks. I think that this is the reason of the slowness I’m experiencing. If I'm right, maybe the way Vim loads the buffers of the qf list should be changed so buffers will be loaded only when they're been accessed and not when they're added to the list. What do you think?

3. Changing other buffers:
As far as I know, there is no nice way to change buffers other than the current one. There is the getbufline() function, but the symmetry is broken since there is no setbufline(). I found a discussion about this from 10 years ago (http://vim.1045645.n5.nabble.com/missing-setbufline-td1155970.html) but I'm not sure if Bram added it to his TODO list or not. In this script I'm using the move_to_buf() and back_from_buf() functions. The problems with this approach are:

- It is ugly.
- Poor performance (it is slow, you can see the cursor jumping between windows, it can cause Vim to hang).
- Difficult to change hidden buffers or buffers in other tab pages.
- Is has side effects - It seems to break my undo tree and I'm sure that there are more.

I'm not very much experienced with external plugins (like Python) which can change any buffer, but it seems that when using a channel with "out_io": "buffer" the performance is much better. Is it possible to implement such a function based on how it's done there?

I think I'll stop here for now. I'm looking forward to get your feedback.

Thanks,
Ramel

[Bram Moolenaar]

Ramel Eshed wrote:

> Thanks to Bram and his recent work on channels, I have a preliminary
> version of an asynchronous grep plugin which lets you work with the
> available results while grep is still running. I would like to take
> advantage of this relatively short script to raise some issues and
> thoughts I have. The plugin can be found at
> https://github.com/ramele/agrep.
>
> My original idea was to use the quickfix list and add each match to it
> from the out callback. I had some issues with it, so temporarily (I
> hope..) I’m using my own window to display the results (“agrep
> window”). This is the default now and it should demonstrate what this
> plugin should do eventually. In order to reproduce the quickfix
> issues, you'll need to tell agrep to use quickfix list instead of
> agrep window (:let qgrep_use_qf=1). You can install the plugin as is
> or load it with $vim -u NONE -N -S <path to agrep.vim>
>
> 1. Stability:
> Currently (7.4.1819) when :Agrep returns high number of results
> (~2800. So we get similar number of callbacks in a time frame of ~2:45
> seconds) Vim crashes with:
>
> Caught deadly signal SEGV
> Vim: Finished.
> Segmentation fault (core dumped)

What system are you using? If on Windows, what compiler?

> This is more likely to happen when using the quickfix list. But it
> happens sometimes with agrep window as well. I never see this when I
> change the out_cb to only process the msg without adding the results
> to neither agrep window or qf list (-with lines 108-118 commented
> out).
>
> 2. Quickfix issues
> a. As long as there are not too many matches and the quickfix
> window is closed you can fire up a search and start jumping to the
> available matches while the search is still running using the quickfix
> navigation commands (:cn, for example). The problem is when the
> quickfix window is opened while the search is active – here there is a
> serious performance issue, and Vim hangs for a while. You can see this
> by running :copen and then :Agrep. Choose a search that will generate
> many results (> 1000).

You could use the channel log to find out where Vim spends time. You
may need to add more log statements in the code and recompile.

> b. As the matches rate becomes higher, Vim will become
> unresponsive. It may happen for short intervals or it can hangs for a
> while (for very high rate). By “rate” I simply mean to
> number_of_matches / sec. To compare, the performance is much better
> with the agrep window.
>
> As I understand (and please correct me if I’m wrong) when using
> setqflist() to create a new list or add items to the current list, Vim
> reads all the buffers (as an unlisted buffers) in order to set the
> hidden marks. I think that this is the reason of the slowness I’m
> experiencing. If I'm right, maybe the way Vim loads the buffers of the
> qf list should be changed so buffers will be loaded only when they're
> been accessed and not when they're added to the list. What do you
> think?

Then it would also be slow if you add all the matches at once. Does
that happen?

> 3. Changing other buffers:
> As far as I know, there is no nice way to change buffers other than
> the current one. There is the getbufline() function, but the symmetry
> is broken since there is no setbufline(). I found a discussion about
> this from 10 years ago
> (http://vim.1045645.n5.nabble.com/missing-setbufline-td1155970.html)
> but I'm not sure if Bram added it to his TODO list or not. In this
> script I'm using the move_to_buf() and back_from_buf() functions. The
> problems with this approach are:
>
> - It is ugly.
> - Poor performance (it is slow, you can see the cursor jumping between windows, it can cause Vim to hang).
> - Difficult to change hidden buffers or buffers in other tab pages.
> - Is has side effects - It seems to break my undo tree and I'm sure that there are more.
>
> I'm not very much experienced with external plugins (like Python)
> which can change any buffer, but it seems that when using a channel
> with "out_io": "buffer" the performance is much better. Is it possible
> to implement such a function based on how it's done there?
>
> I think I'll stop here for now. I'm looking forward to get your feedback.

Thanks. I'm quite busy the coming days, but perhaps someone else can
have a go at it.

--
Wi n0t trei a h0liday in Sweden thi yer?
"Monty Python and the Holy Grail" PYTHON (MONTY) PICTURES LTD

/// Bram Moolenaar -- Br...@Moolenaar.net -- http://www.Moolenaar.net \\\
/// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
\\\ an exciting new programming language -- http://www.Zimbu.org ///
\\\ help me help AIDS victims -- http://ICCF-Holland.org ///

[Ramel Eshed]

On Sunday, May 8, 2016 at 8:59:24 AM UTC+3, Bram Moolenaar wrote:
> Ramel Eshed wrote:
>
> > Thanks to Bram and his recent work on channels, I have a preliminary
> > version of an asynchronous grep plugin which lets you work with the
> > available results while grep is still running. I would like to take
> > advantage of this relatively short script to raise some issues and
> > thoughts I have. The plugin can be found at
> > https://github.com/ramele/agrep.
> >
> > My original idea was to use the quickfix list and add each match to it
> > from the out callback. I had some issues with it, so temporarily (I
> > hope..) I’m using my own window to display the results (“agrep
> > window”). This is the default now and it should demonstrate what this
> > plugin should do eventually. In order to reproduce the quickfix
> > issues, you'll need to tell agrep to use quickfix list instead of
> > agrep window (:let qgrep_use_qf=1). You can install the plugin as is
> > or load it with $vim -u NONE -N -S <path to agrep.vim>
> >
> > 1. Stability:
> > Currently (7.4.1819) when :Agrep returns high number of results
> > (~2800. So we get similar number of callbacks in a time frame of ~2:45
> > seconds) Vim crashes with:
> >
> > Caught deadly signal SEGV
> > Vim: Finished.
> > Segmentation fault (core dumped)
>
> What system are you using? If on Windows, what compiler?
>

This plugin is for Linux. I'm using Ubuntu 16.04 (and tested it also on RHEL 5.5).

> > This is more likely to happen when using the quickfix list. But it
> > happens sometimes with agrep window as well. I never see this when I
> > change the out_cb to only process the msg without adding the results
> > to neither agrep window or qf list (-with lines 108-118 commented
> > out).
> >
> > 2. Quickfix issues
> > a. As long as there are not too many matches and the quickfix
> > window is closed you can fire up a search and start jumping to the
> > available matches while the search is still running using the quickfix
> > navigation commands (:cn, for example). The problem is when the
> > quickfix window is opened while the search is active – here there is a
> > serious performance issue, and Vim hangs for a while. You can see this
> > by running :copen and then :Agrep. Choose a search that will generate
> > many results (> 1000).
>
> You could use the channel log to find out where Vim spends time. You
> may need to add more log statements in the code and recompile.
>

I think that this is a quickfix issue rather than channels. I don't see that problem when running with agrep window or even with quickfix without the qf window opened (there is also the other issue (see #b below) but this one is much more serious). The channels code does exactly the same, no matter which mode we're using to display the results.

> > b. As the matches rate becomes higher, Vim will become
> > unresponsive. It may happen for short intervals or it can hangs for a
> > while (for very high rate). By “rate” I simply mean to
> > number_of_matches / sec. To compare, the performance is much better
> > with the agrep window.
> >
> > As I understand (and please correct me if I’m wrong) when using
> > setqflist() to create a new list or add items to the current list, Vim
> > reads all the buffers (as an unlisted buffers) in order to set the
> > hidden marks. I think that this is the reason of the slowness I’m
> > experiencing. If I'm right, maybe the way Vim loads the buffers of the
> > qf list should be changed so buffers will be loaded only when they're
> > been accessed and not when they're added to the list. What do you
> > think?
>
> Then it would also be slow if you add all the matches at once. Does
> that happen?
>

You're right, I forgot to mention that. It is noticeable also when adding many results at once (you can see this by running :Agrepsetqf after you have a long list of results).

> > 3. Changing other buffers:
> > As far as I know, there is no nice way to change buffers other than
> > the current one. There is the getbufline() function, but the symmetry
> > is broken since there is no setbufline(). I found a discussion about
> > this from 10 years ago
> > (http://vim.1045645.n5.nabble.com/missing-setbufline-td1155970.html)
> > but I'm not sure if Bram added it to his TODO list or not. In this
> > script I'm using the move_to_buf() and back_from_buf() functions. The
> > problems with this approach are:
> >
> > - It is ugly.
> > - Poor performance (it is slow, you can see the cursor jumping between windows, it can cause Vim to hang).
> > - Difficult to change hidden buffers or buffers in other tab pages.
> > - Is has side effects - It seems to break my undo tree and I'm sure that there are more.
> >
> > I'm not very much experienced with external plugins (like Python)
> > which can change any buffer, but it seems that when using a channel
> > with "out_io": "buffer" the performance is much better. Is it possible
> > to implement such a function based on how it's done there?
> >
> > I think I'll stop here for now. I'm looking forward to get your feedback.
>
> Thanks. I'm quite busy the coming days, but perhaps someone else can
> have a go at it.
>

Thanks Bram!

[Bram Moolenaar]

I fixed two related problems: processes hanging in "defunc" state and
not redirecting stdout to /dev/null. Probably won't have time to check
out the other problems this week.

--
A M00se once bit my sister ...

[Ramel Eshed]

It didn't help. I still get the SEGV error.

[Ramel Eshed]

Hi Bram,

I'm not sure what exactly you've done in the last few patches, but it seems like the SEGV issue is solved now, Thanks. Unfortunately, I have a different issue now when using the gui; Sometimes, when I perform a large search (~9000 results) Vim hangs and never restored. I enabled the channel log and I see the following Netbeans messages even though I'm not using the Netbeans protocol:

ERR on 1: messageFromNetbeans(): Cannot read from channel, will close it soon

When running Vim in a terminal everything works fine, but for some reason I still see errors in the channel's log. The terminal error is:

ERR on 1: channel_select_check(): Cannot read from channel, will close it soon

Any idea what went wrong?
The gui issue doesn't seem deterministic, so I can't give a way to reproduce it, but you can download my plugin and play with it a bit (It takes me no more than 10 searches in the linux kernel code base until I see the issue).

Thanks,
Ramel

[Bram Moolenaar]

Ramel Eshed wrote:

> I'm not sure what exactly you've done in the last few patches, but it
> seems like the SEGV issue is solved now, Thanks.

Glad to hear that.

> Unfortunately, I have
> a different issue now when using the gui; Sometimes, when I perform a
> large search (~9000 results) Vim hangs and never restored. I enabled
> the channel log and I see the following Netbeans messages even though
> I'm not using the Netbeans protocol:
>
> ERR on 1: messageFromNetbeans(): Cannot read from channel, will close it soon

That's just an old name for what is now a generic function.

> When running Vim in a terminal everything works fine, but for some
> reason I still see errors in the channel's log. The terminal error is:
>
> ERR on 1: channel_select_check(): Cannot read from channel, will close it soon
>
> Any idea what went wrong?

Not really. It's possible the job just finished. Do you have a way to
check for that?

> The gui issue doesn't seem deterministic, so I can't give a way to
> reproduce it, but you can download my plugin and play with it a bit
> (It takes me no more than 10 searches in the linux kernel code base
> until I see the issue).

Let's first find out if the output is truncated or not. You could run
the same command and redirect the output to a file, then check the
channel log to see what's the last thing that Vim received.

--
Everybody wants to go to heaven, but nobody wants to die.

[Ramel Eshed]

You're right. I checked and it seems like the job is finished, I can see that Vim got all the job's output. That's however what I see at the end of the log (the first line is the last line of the grep process):

/home/ramele/linux-4.6-rc5/block/blk-mq-sysfs.c:457: blk_mq_enable_h [01m [Kotp [m [Klug();
'
3.396833 : looking for messages on channels
3.396842 on 0: Invoking channel callback Agrep_cb
3.397006 on 0: Invoking channel callback Agrep_cb
3.397174 on 0: Invoking channel callback Agrep_cb
3.397333 on 0: Invoking channel callback Agrep_cb
3.397424 ERR on 0: channel_read_fd(): Cannot read from channel, will close it soon
3.397433 ERR on 0: channel_read_fd(): Cannot read from channel, will close it soon
3.397445 ERR on 0: channel_read_fd(): Cannot read from channel, will close it soon
3.397451 ERR on 0: channel_read_fd(): Cannot read from channel, will close it soon
3.397462 ERR on 0: channel_read_fd(): Cannot read from channel, will close it soon
3.397467 ERR on 0: channel_read_fd(): Cannot read from channel, will close it soon
3.397478 ERR on 0: channel_read_fd(): Cannot read from channel, will close it soon
3.397484 ERR on 0: channel_read_fd(): Cannot read from channel, will close it soon
.
.
.

It seems that Vim hangs in an infinite loop here. For some reason I didn't see it in the previous log, not sure why (maybe it's related to some changes I made to simplify the debug).

I hope that helps you. Let me know if you want me to chack further.

Thanks,
Ramel

[Bram Moolenaar]

Ramel Eshed wrote:

> > > I'm not sure what exactly you've done in the last few patches, but it
> > > seems like the SEGV issue is solved now, Thanks.
> >
> > Glad to hear that.
> >
> > > Unfortunately, I have
> > > a different issue now when using the gui; Sometimes, when I perform a
> > > large search (~9000 results) Vim hangs and never restored. I enabled
> > > the channel log and I see the following Netbeans messages even though
> > > I'm not using the Netbeans protocol:
> > >
> > > ERR on 1: messageFromNetbeans(): Cannot read from channel, will close it soon
> >
> > That's just an old name for what is now a generic function.
> >
> > > When running Vim in a terminal everything works fine, but for some
> > > reason I still see errors in the channel's log. The terminal error is:
> > >
> > > ERR on 1: channel_select_check(): Cannot read from channel, will close it soon
> > >
> > > Any idea what went wrong?
> >
> > Not really. It's possible the job just finished. Do you have a way to
> > check for that?
> >
> > > The gui issue doesn't seem deterministic, so I can't give a way to
> > > reproduce it, but you can download my plugin and play with it a bit
> > > (It takes me no more than 10 searches in the linux kernel code base
> > > until I see the issue).
> >
> > Let's first find out if the output is truncated or not. You could run
> > the same command and redirect the output to a file, then check the
> > channel log to see what's the last thing that Vim received.
>

Thanks for finding this. I can avoid giving the error more than once by
not trying to read if the channel is about to be closed.

However, I still wonder why this happens. Is this only with gvim or
also in a terminal?

Simplest would be if you could run Vim in a debugger and set a
breakpoint where the error is reported a second time.

Anyway, I'll make a patch to avoid giving the error multiple times, that
might already help, since it will skip trying to read.

--
When a fly lands on the ceiling, does it do a half roll or
a half loop?

[Ramel Eshed]

I tried the patch from shougo's thread and it solved my problem. thanks!

There is another (minor) issue I noticed; Because I had problems with the close_cb (I got out-callbacks after it was called) I started a timer to 200ms so I'll be able to know when it's safe to cleanup. I'm not sure if it is still needed, however I see that when running in the terminal the timer callback is called almost immediately as expected, but in the gui it takes more than 2 seconds from when the timer starts until I get the callback. Any idea why?

Also, did you have a chance to check the two quickfix issues from the first message?

[Bram Moolenaar]

Ramel Eshed wrote:

> I tried the patch from shougo's thread and it solved my problem. thanks!

So the current Vim works for you?

> There is another (minor) issue I noticed; Because I had problems with
> the close_cb (I got out-callbacks after it was called) I started a
> timer to 200ms so I'll be able to know when it's safe to cleanup. I'm
> not sure if it is still needed, however I see that when running in the
> terminal the timer callback is called almost immediately as expected,
> but in the gui it takes more than 2 seconds from when the timer starts
> until I get the callback. Any idea why?

I did a simple timer setup and in my GUI it works like in the terminal.
This is on Unix.

function! Done(timer)
echomsg "Done"
endfunction

call timer_start(200, 'Done')

> Also, did you have a chance to check the two quickfix issues from the
> first message?

Which ones are that? This thread has gotten a bit long. Is this about
parsing errors line by line? I was wondering if ":caddexpr" comes
close. Perhaps we should have a function for this.

--
How To Keep A Healthy Level Of Insanity:
4. Put your garbage can on your desk and label it "in".

[Yegappan Lakshmanan]

Hi,

On Sat, May 7, 2016 at 2:42 PM, Ramel Eshed <ram...@gmail.com> wrote:
> Hi All,
>
> Thanks to Bram and his recent work on channels, I have a preliminary
> version of an asynchronous grep plugin which lets you work with the
> available results while grep is still running. I would like to take
> advantage of this relatively short script to raise some issues and
> thoughts I have. The plugin can be found at
> https://github.com/ramele/agrep.
>

> 2. Quickfix issues
> a. As long as there are not too many matches and
> the quickfix window is closed you can fire up a search and start
> jumping to the available matches while the search is still running
> using the quickfix navigation commands (:cn, for example). The problem
> is when the quickfix window is opened while the search is active –
> here there is a serious performance issue, and Vim hangs for a while.
> You can see this by running :copen and then :Agrep. Choose a search
> that will generate many results (> 1000).
>
> b. As the matches rate becomes higher, Vim will become
> unresponsive. It may happen for short intervals or it can hangs for a
> while (for very high rate). By “rate” I simply mean to
> number_of_matches / sec. To compare, the performance is much better
> with the agrep window.
>
> As I understand (and please correct me if I’m wrong) when using
> setqflist() to create a new list or add items to the current list, Vim
> reads all the buffers (as an unlisted buffers) in order to set the
> hidden marks. I think that this is the reason of the slowness I’m
> experiencing. If I'm right, maybe the way Vim loads the buffers of the
> qf list should be changed so buffers will be loaded only when they're
> been accessed and not when they're added to the list. What do you
> think?
>

When adding a new entry to the quickfix list, a search is made to locate
the last entry. I think, this is the reason for the long processing time.
To confirm this, I created a test script to add 100000 entries to the
quickfix list. It took almost 120 seconds to complete. After I changed
the quickfix code to keep track of the last entry, this reduced to 2 seconds.
Can you try using the attached patch?

- Yegappan

[Ramel Eshed]

On Sunday, May 29, 2016 at 4:44:12 PM UTC+3, Bram Moolenaar wrote:
> Ramel Eshed wrote:
>
> > I tried the patch from shougo's thread and it solved my problem. thanks!
>
> So the current Vim works for you?

Yes.

>
> > There is another (minor) issue I noticed; Because I had problems with
> > the close_cb (I got out-callbacks after it was called) I started a
> > timer to 200ms so I'll be able to know when it's safe to cleanup. I'm
> > not sure if it is still needed, however I see that when running in the
> > terminal the timer callback is called almost immediately as expected,
> > but in the gui it takes more than 2 seconds from when the timer starts
> > until I get the callback. Any idea why?
>
> I did a simple timer setup and in my GUI it works like in the terminal.
> This is on Unix.
>
> function! Done(timer)
> echomsg "Done"
> endfunction
>
> call timer_start(200, 'Done')
>

Please try this:

func! Close_cb(channel)
let g:rt = reltime()
call timer_start(200, 'Timer_cb')
endfunc

func! Timer_cb(timer)
echo reltime(g:rt)
endfunc

call job_start('ls', {'close_cb': 'Close_cb'})

>
> > Also, did you have a chance to check the two quickfix issues from the
> > first message?
>
> Which ones are that? This thread has gotten a bit long. Is this about
> parsing errors line by line? I was wondering if ":caddexpr" comes
> close. Perhaps we should have a function for this.
>

In my plugin I'm doing the message parsing manually, then I'm using setqflist() to add the results to the list. The problems I mentioned are quoted in Yegappan's response.

[Ramel Eshed]

Hi Yegappan,

Thanks for checking this. I didn't see a noticeable improvement, at least regarding to my second issue (b) - The cursor is still not fully responsive in my test case (search which returns 10000 results and add each one to the qf list on the fly). It also does not solve the first issue.

Also, I noticed a very strange issue (which also exists without your patch, but I think that your patch makes it even worse):
When I run my search command without using the quickfix list I get the 10000 results in about 2.2 seconds (and it's also non blocking of course -pretty nice :)). But -when I run this search after using the quickfix list the performance (of my search command, which is not related to the quickfix mechanism) becomes very bad. Actually, the performance degrades in a correlation with how many times I used the quickfix before. Example:

:Agrep -r <pattern> <directory>

- This is my search command. It returns all the results in 2.2 seconds (no quickfix list here).

But:

:grep -r <pattern> <directory> " regular grep that uses the quickfix list
and then -
:Agrep -r <pattern> <directory>

- Here, my Agrep command takes ~4 seconds

And
:grep -r <pattern> <directory>
:grep -r <pattern> <directory>
:grep -r <pattern> <directory>
:Agrep -r <pattern> <directory>

- Now Agrep takes ~7 seconds.

Any idea what is going on here?

[Yegappan Lakshmanan]

Hi,

> Thanks for checking this. I didn't see a noticeable improvement, at
> least regarding to my second issue (b) - The cursor is still not fully
> responsive in my test case (search which returns 10000 results and add
> each one to the qf list on the fly). It also does not solve the first
> issue.
>

I think, you have the quickfix window open when running this command.
Can you try without opening the quickfix window and see whether my
patch reduces the time it takes to add all the matches to the quickfix list?

I think, the problem you are describing is related to refreshing the quickfix
window when adding entries to the quickfix list.

After adding an entry to the quickfix list, if the quickfix window is open,
the quickfix buffer is refreshed to display the latest list. As part of this
all the entries in the quickfix buffer/window are removed and then added
back again. When you add 10000 entries one at a time, this operation
takes a long time to complete.

>
> Also, I noticed a very strange issue (which also exists without your
> patch, but I think that your patch makes it even worse): When I run my
> search command without using the quickfix list I get the 10000 results
> in about 2.2 seconds (and it's also non blocking of course -pretty
> nice :)). But -when I run this search after using the quickfix list
> the performance (of my search command, which is not related to the
> quickfix mechanism) becomes very bad. Actually, the performance
> degrades in a correlation with how many times I used the quickfix
> before. Example:
>
> :Agrep -r <pattern> <directory>
>
> - This is my search command. It returns all the results in 2.2 seconds (no
> quickfix list here).
>
> But:
>
> :grep -r <pattern> <directory> " regular grep that uses the quickfix list
> and then -
> :Agrep -r <pattern> <directory>
>
> - Here, my Agrep command takes ~4 seconds
>
> And
> :grep -r <pattern> <directory>
> :grep -r <pattern> <directory>
> :grep -r <pattern> <directory>
> :Agrep -r <pattern> <directory>
>
> - Now Agrep takes ~7 seconds.
>
> Any idea what is going on here?
>

I am not sure.

- Yegappan

[Bram Moolenaar]

Ramel Eshed wrote:

> > > There is another (minor) issue I noticed; Because I had problems with
> > > the close_cb (I got out-callbacks after it was called) I started a
> > > timer to 200ms so I'll be able to know when it's safe to cleanup. I'm
> > > not sure if it is still needed, however I see that when running in the
> > > terminal the timer callback is called almost immediately as expected,
> > > but in the gui it takes more than 2 seconds from when the timer starts
> > > until I get the callback. Any idea why?
> >
> > I did a simple timer setup and in my GUI it works like in the terminal.
> > This is on Unix.
> >
> > function! Done(timer)
> > echomsg "Done"
> > endfunction
> >
> > call timer_start(200, 'Done')
> >
> Please try this:
>
> func! Close_cb(channel)
> let g:rt = reltime()
> call timer_start(200, 'Timer_cb')
> endfunc
>
> func! Timer_cb(timer)
> echo reltime(g:rt)
> endfunc
>
> call job_start('ls', {'close_cb': 'Close_cb'})

OK, that I can reproduce, in the GUI the timer is only invoked after
'updatetime'.

> > > Also, did you have a chance to check the two quickfix issues from the
> > > first message?
> >
> > Which ones are that? This thread has gotten a bit long. Is this about
> > parsing errors line by line? I was wondering if ":caddexpr" comes
> > close. Perhaps we should have a function for this.
> >
> In my plugin I'm doing the message parsing manually, then I'm using
> setqflist() to add the results to the list. The problems I mentioned
> are quoted in Yegappan's response.

I think we should do profiling to find out why it's slow. Guessing
often leads to optimizing code that isn't the bottleneck. Search for
PROFILING in src/Makefile for hints.

Otherwise, you could check if setting the quickfix list, instead of
appending to it, makes any difference.

--
How To Keep A Healthy Level Of Insanity:

18. When leaving the zoo, start running towards the parking lot,
yelling "run for your lives, they're loose!!"

[Bram Moolenaar]

Yegappan Lakshmanan wrote:

> On Sat, May 7, 2016 at 2:42 PM, Ramel Eshed <ram...@gmail.com> wrote:
> > Hi All,
> >
> > Thanks to Bram and his recent work on channels, I have a preliminary
> > version of an asynchronous grep plugin which lets you work with the
> > available results while grep is still running. I would like to take
> > advantage of this relatively short script to raise some issues and
> > thoughts I have. The plugin can be found at
> > https://github.com/ramele/agrep.
> >
>
> > 2. Quickfix issues
> > a. As long as there are not too many matches and
> > the quickfix window is closed you can fire up a search and start
> > jumping to the available matches while the search is still running
> > using the quickfix navigation commands (:cn, for example). The problem

> > is when the quickfix window is opened while the search is active =E2=80=
> =93

> > here there is a serious performance issue, and Vim hangs for a while.
> > You can see this by running :copen and then :Agrep. Choose a search
> > that will generate many results (> 1000).
> >
> > b. As the matches rate becomes higher, Vim will become
> > unresponsive. It may happen for short intervals or it can hangs for a

> > while (for very high rate). By =E2=80=9Crate=E2=80=9D I simply mean to

> > number_of_matches / sec. To compare, the performance is much better
> > with the agrep window.
> >

> > As I understand (and please correct me if I=E2=80=99m wrong) when using

> > setqflist() to create a new list or add items to the current list, Vim
> > reads all the buffers (as an unlisted buffers) in order to set the

> > hidden marks. I think that this is the reason of the slowness I=E2=80=99m

> > experiencing. If I'm right, maybe the way Vim loads the buffers of the
> > qf list should be changed so buffers will be loaded only when they're
> > been accessed and not when they're added to the list. What do you
> > think?
> >
>
> When adding a new entry to the quickfix list, a search is made to locate
> the last entry. I think, this is the reason for the long processing time.
> To confirm this, I created a test script to add 100000 entries to the
> quickfix list. It took almost 120 seconds to complete. After I changed

> the quickfix code to keep track of the last entry, this reduced to 2 second=

> s.
> Can you try using the attached patch?

Is finding the end of the list really that slow?

If this helps, I would propose to set qf_last in qf_add_entry() and drop
the use of qfprev. That keeps the logic of updating the last entry in
one place, and we could drop the "last element points to itself"
perhaps. And "first element points to itself". Using NULL makes more
sense.

--
How To Keep A Healthy Level Of Insanity:

17. When the money comes out the ATM, scream "I won!, I won! 3rd
time this week!!!!!"

[Bram Moolenaar]

Ramel Eshed wrote:

> Hi All,
>
> Thanks to Bram and his recent work on channels, I have a preliminary
> version of an asynchronous grep plugin which lets you work with the
> available results while grep is still running. I would like to take
> advantage of this relatively short script to raise some issues and
> thoughts I have. The plugin can be found at
> https://github.com/ramele/agrep.
>
> My original idea was to use the quickfix list and add each match to it
> from the out callback. I had some issues with it, so temporarily (I
> hope..) I’m using my own window to display the results (“agrep
> window”). This is the default now and it should demonstrate what this
> plugin should do eventually. In order to reproduce the quickfix
> issues, you'll need to tell agrep to use quickfix list instead of
> agrep window (:let qgrep_use_qf=1). You can install the plugin as is
> or load it with $vim -u NONE -N -S <path to agrep.vim>

Several of the problems have been fixed. Especially appending to the
quickfix list should now be much faster.

Let us know if you still encounter problems.

> 3. Changing other buffers:
> As far as I know, there is no nice way to change buffers other than
> the current one. There is the getbufline() function, but the symmetry
> is broken since there is no setbufline(). I found a discussion about
> this from 10 years ago
> (http://vim.1045645.n5.nabble.com/missing-setbufline-td1155970.html)
> but I'm not sure if Bram added it to his TODO list or not. In this
> script I'm using the move_to_buf() and back_from_buf() functions. The
> problems with this approach are:
>
> - It is ugly.
> - Poor performance (it is slow, you can see the cursor jumping between
> windows, it can cause Vim to hang).
> - Difficult to change hidden buffers or buffers in other tab pages.
> - Is has side effects - It seems to break my undo tree and I'm sure
> that there are more.

This is actually difficult to do. The internals of Vim only expect a
buffer to change when it's in a window. Also when invoking
autocommands, which expect a valid buffer and window. And undo expects
the buffer to be the current buffer. This would be a lot of work to
change, and it's likely to introduce a few bugs.

> I'm not very much experienced with external plugins (like Python)
> which can change any buffer, but it seems that when using a channel
> with "out_io": "buffer" the performance is much better. Is it possible
> to implement such a function based on how it's done there?

Python uses switch_to_win_for_buf(). It doesn't work perfectly though.
Marks may be wrong and it may redraw the current window even though it
didn't change.


--
Micro$oft: where do you want to go today?
Linux: where do you want to go tomorrow?
FreeBSD: are you guys coming, or what?

[Yegappan Lakshmanan]

Hi Bram,

On Thu, Jun 2, 2016 at 9:53 AM, Bram Moolenaar <Br...@moolenaar.net> wrote:
>
> Ramel Eshed wrote:
>
>> Hi All,
>>
>> Thanks to Bram and his recent work on channels, I have a preliminary
>> version of an asynchronous grep plugin which lets you work with the
>> available results while grep is still running. I would like to take
>> advantage of this relatively short script to raise some issues and
>> thoughts I have. The plugin can be found at
>> https://github.com/ramele/agrep.
>>
>> My original idea was to use the quickfix list and add each match to it
>> from the out callback. I had some issues with it, so temporarily (I
>> hope..) I’m using my own window to display the results (“agrep
>> window”). This is the default now and it should demonstrate what this
>> plugin should do eventually. In order to reproduce the quickfix
>> issues, you'll need to tell agrep to use quickfix list instead of
>> agrep window (:let qgrep_use_qf=1). You can install the plugin as is
>> or load it with $vim -u NONE -N -S <path to agrep.vim>
>
> Several of the problems have been fixed. Especially appending to the
> quickfix list should now be much faster.
>
> Let us know if you still encounter problems.
>

The problem with adding large number of entries to a quickfix list is not
fixed yet. I am attaching a test script to illustrate the problem. Source the
script and run "call Measure_Caddexpr_Time()". You will see that it
takes around 2 minutes to add the entries.

The patch I sent earlier address this problem.

- Yegappan

[Bram Moolenaar]

Yes, but I wanted to use another solution: add qf_last.
I'm doing that now, the time goes down from 150 seconds to 2 seconds.

--
Those who live by the sword get shot by those who don't.

[Ramel Eshed]

Hi Bram and Yegappan,

Yegappan, You were right -saving the qf_last makes a big difference. I probably checked your patch after using many quickfix operations (see below).

Let me summarize the problems found so far using the agrep plugin:

1. SEGV crashes - fixed
2. Vim hangs while quickfix window is opened - fixed.
3. GUI hangs - fixed
4. Timer issue in the GUI - fixed

5. quickfix is slow in general - it's much better with Yegappan's patch or 1881 but Vim is still not as responsive as it is when using the same search without quickfix (with Agrep window). Following are the top lines from the profiler log, in case you'll see something that can be optimized:

% cumulative self self total
time seconds seconds calls ms/call ms/call name
11.34 0.11 0.11 11084 0.01 0.01 buf_valid
10.31 0.21 0.10 11378 0.01 0.01 do_cmdline
5.15 0.26 0.05 9912382 0.00 0.00 otherfile_buf
5.15 0.31 0.05 452997 0.00 0.00 get_func_tv
5.15 0.36 0.05 11083 0.00 0.02 buflist_new
5.15 0.41 0.05 11082 0.00 0.01 buflist_findname_stat
5.15 0.46 0.05 2131 0.02 0.02 buflist_findnr

6. Agrep become very slow after using the quickfix list many times. Well, this is what I see in the profiler log:
% cumulative self self total
time seconds seconds calls ms/call ms/call name
55.98 4.21 4.21 33246 0.13 0.13 qf_mark_adjust <<<
12.10 5.12 0.91 50685 0.02 0.02 buf_valid
6.38 5.60 0.48 11091 0.04 0.05 buflist_findpat
5.05 5.98 0.38 13256 0.03 0.03 buflist_findnr

It looks like qf_mark_adjust is called each time line is appended to a buffer using 'out_io': 'buffer' for each entry in any quickfix list available. Can we avoid this?

Thanks,
Ramel

[Ramel Eshed]

Is it different from how 'out_io': 'buffer' is handled? Also, I tried this patch (which I found eventually in the TODO list): https://gist.github.com/mattn/23c1f50999084992ca98 and it worked very well for me. I guess that it can break other things, but maybe something like this can be added with an appropriate documentation which specify when it can be used?

In order to update my buffer now, I'm using a ridiculous workaround - Vim is writing to itself through a pipe...

[Bram Moolenaar]

No, this is needed. But why is a buffer changed? I thought you were
adding quickfix entries, which doesn't trigger adjusting marks. Only
inserting/deleting a line in a buffer triggers this.

The reason this is needed, is that when you have a list of errors in
various files, which are at specific line numbers, and you make changes
in files, the line numbers need to be adjusted.

Since the qf_last change helped a lot, I susped just going through all
the quickfix entries is making it slow. We would need to use another
data structure, which lists all the quickfix entries related to a
buffer. Then we only need to look at the ones that might actually
change. Keeping that list updated will be extra work though. In
different circumstances it may actually make it slower.

--
Nothing is fool-proof to a sufficiently talented fool.

[Ramel Eshed]

Because of adding quickfix entries was slow I've created my own buffer to display the search results (you can see an animated gif here: http://i.imgur.com/epffEDH.gif).
I need to perform some manipulations on the grep results in order to get the column numbers and to highlight the matching text. Currently, this is done in the out_cb function which sends the modified line to my special buffer via a separate 'cat' job (this is the workaround I found as a replacement to setbufline() :)).

> The reason this is needed, is that when you have a list of errors in
> various files, which are at specific line numbers, and you make changes
> in files, the line numbers need to be adjusted.
>
> Since the qf_last change helped a lot, I susped just going through all
> the quickfix entries is making it slow. We would need to use another
> data structure, which lists all the quickfix entries related to a
> buffer. Then we only need to look at the ones that might actually
> change. Keeping that list updated will be extra work though. In
> different circumstances it may actually make it slower.
>

I think that in cases like this -when lines are appended to a buffer by a job (using 'out_io': 'buffer') we don't need to adjust the quickfix marks since the target buffer is probably not included in any quickfix list. Is there a corresponding option to the :lockmarks command (like 'eventignore' and :noautocmd)?

Because of the serious performance impact of many quickfix entries, I think we should have a built in command for freeing a quickfix list. I guess I can use :call setqflist([], 'r'), but it'll leave an empty list and it'd be nicer to remove the list completely.
Also, I noticed that using :call setqflist([]) while there is only one list will add a new empty list, and will delete the last list when there is more than one list. According to the help it should behave like setqflist([], 'r'):

If you supply an empty {list}, the quickfix list will be
cleared.

[Bram Moolenaar]

So this buffer that's changing does not have entries in the quickfix
list, right? I think I can add a trick to skip buffers that don't have
any quickfix entries. That should solve your problem and is fairly
simple to implement.

> > The reason this is needed, is that when you have a list of errors in
> > various files, which are at specific line numbers, and you make changes
> > in files, the line numbers need to be adjusted.
> >
> > Since the qf_last change helped a lot, I susped just going through all
> > the quickfix entries is making it slow. We would need to use another
> > data structure, which lists all the quickfix entries related to a
> > buffer. Then we only need to look at the ones that might actually
> > change. Keeping that list updated will be extra work though. In
> > different circumstances it may actually make it slower.
>
> I think that in cases like this -when lines are appended to a buffer
> by a job (using 'out_io': 'buffer') we don't need to adjust the
> quickfix marks since the target buffer is probably not included in any
> quickfix list. Is there a corresponding option to the :lockmarks
> command (like 'eventignore' and :noautocmd)?

Adjusting marks for adding a line below the last one indeed doesn't have
much effect. It's possible that there is a mark when lines have been
deleted, but it will stick to the end anyway.

> Because of the serious performance impact of many quickfix entries, I
> think we should have a built in command for freeing a quickfix list. I
> guess I can use :call setqflist([], 'r'), but it'll leave an empty
> list and it'd be nicer to remove the list completely.
> Also, I noticed that using :call setqflist([]) while there is only one
> list will add a new empty list, and will delete the last list when
> there is more than one list. According to the help it should behave
> like setqflist([], 'r'):
>
> If you supply an empty {list}, the quickfix list will be
> cleared.

Yes, that is supposed to work.

--
Just remember...if the world didn't suck, we'd all fall off.

[Ramel Eshed]

Hi Bram,

1) Actually, there is a bug here which is not exactly what I've described earlier. The issue is that :call setqflist([]), instead of adding one more list after the last list, will clear the next list and delete the ones after. For example: let's say I used :grep 4 times so I have now 4 lists. Now, if I do :colder 3, the first list becomes the current list. :call setqflist([]) will empty list 2, and delete lists 3 and 4.

2) Although quickfix performance got much better, I'm afraid there is more to do:
a) It takes me about 9 seconds(!) to add 68,505 entries to the qf list (using simple :grep command. Time was measured from after the grep command output finished, of course). Yegappan's test takes only 2 seconds because all the results are from the same file. Try to add the loop variable to the file name in his test and see what happens...

This is the profiler log of the search I did:

% cumulative self self total
time seconds seconds calls ms/call ms/call name

40.63 1.69 1.69 68510 0.02 0.02 buf_valid
21.88 2.60 0.91 68505 0.01 0.02 buflist_findname_stat
12.38 3.12 0.52 127664549 0.00 0.00 otherfile_buf
4.81 3.32 0.20 4345 0.05 0.05 buflist_findnr

b) I remember you did 2 things in order to avoid the line adjustments when it's not necessary: check first if the buffer has a quickfix entry and don't call line adjustment when adding a line at the end.

I still see that adding to a buffer when there are many qf entries is very slow. After I had the 68505 results of the :grep command, adding ~84000 lines to a buffer (from channel output) became really slow. This is the profiler log:

Each sample counts as 0.01 seconds.

% cumulative self self total

time seconds seconds calls s/call s/call name
31.46 6.98 6.98 149100 0.00 0.00 buf_valid
29.20 13.46 6.48 80590 0.00 0.00 buflist_findpat
26.66 19.38 5.92 84950 0.00 0.00 buflist_findnr
3.56 20.17 0.79 68507 0.00 0.00 buflist_findname_stat
2.28 20.67 0.51 127673186 0.00 0.00 otherfile_buf

It seems like checking the buffer each time alone takes a lot of time. Is there any way to optimize the above functions?
Also, why I still see all these buffer functions even though all the lines were added at the end of the buffer?

Thanks,
Ramel

[Bram Moolenaar]

Ramel Eshed wrote:

> 1) Actually, there is a bug here which is not exactly what I've
> described earlier. The issue is that :call setqflist([]), instead of
> adding one more list after the last list, will clear the next list and
> delete the ones after. For example: let's say I used :grep 4 times so
> I have now 4 lists. Now, if I do :colder 3, the first list becomes the
> current list. :call setqflist([]) will empty list 2, and delete lists
> 3 and 4.

We do remove newer lists when adding a new list before the end, but in
this case it should not happen.

> 2) Although quickfix performance got much better, I'm afraid there is
> more to do:
> a) It takes me about 9 seconds(!) to add 68,505 entries to the qf
> list (using simple :grep command. Time was measured from after the
> grep command output finished, of course). Yegappan's test takes
> only 2 seconds because all the results are from the same file. Try
> to add the loop variable to the file name in his test and see what
> happens...
>
> This is the profiler log of the search I did:
>
> % cumulative self self total
> time seconds seconds calls ms/call ms/call name
> 40.63 1.69 1.69 68510 0.02 0.02 buf_valid
> 21.88 2.60 0.91 68505 0.01 0.02 buflist_findname_stat
> 12.38 3.12 0.52 127664549 0.00 0.00 otherfile_buf
> 4.81 3.32 0.20 4345 0.05 0.05 buflist_findnr

Well, when adding an entry a buffer is created, so that we keep track of
the actual name when doing ":cd". And it avoids duplicates.
buf_valid() is often needed to check if an autocommand didn't delete a
buffer. Both of these are hard to get rid of.

> b) I remember you did 2 things in order to avoid the line
> adjustments when it's not necessary: check first if the buffer has a
> quickfix entry and don't call line adjustment when adding a line at
> the end.
>
> I still see that adding to a buffer when there are many qf entries
> is very slow. After I had the 68505 results of the :grep command,
> adding ~84000 lines to a buffer (from channel output) became really
> slow. This is the profiler log:

How are the lines added?

> Each sample counts as 0.01 seconds.
> % cumulative self self total
> time seconds seconds calls s/call s/call name
> 31.46 6.98 6.98 149100 0.00 0.00 buf_valid
> 29.20 13.46 6.48 80590 0.00 0.00 buflist_findpat
> 26.66 19.38 5.92 84950 0.00 0.00 buflist_findnr
> 3.56 20.17 0.79 68507 0.00 0.00 buflist_findname_stat
> 2.28 20.67 0.51 127673186 0.00 0.00 otherfile_buf
>
> It seems like checking the buffer each time alone takes a lot of time.
> Is there any way to optimize the above functions?
> Also, why I still see all these buffer functions even though all the
> lines were added at the end of the buffer?

If you use 'errorformat' it will locate the file name and find out what
buffer has that file name.

--
Q: How many legs does a giraffe have?
A: Eight: two in front, two behind, two on the left and two on the right

[Ramel Eshed]

On Thursday, June 9, 2016 at 11:52:41 AM UTC+3, Bram Moolenaar wrote:
> Ramel Eshed wrote:
>
> > 1) Actually, there is a bug here which is not exactly what I've
> > described earlier. The issue is that :call setqflist([]), instead of
> > adding one more list after the last list, will clear the next list and
> > delete the ones after. For example: let's say I used :grep 4 times so
> > I have now 4 lists. Now, if I do :colder 3, the first list becomes the
> > current list. :call setqflist([]) will empty list 2, and delete lists
> > 3 and 4.
>
> We do remove newer lists when adding a new list before the end, but in
> this case it should not happen.

I'm not sure I understand. Is removing all the newer lists are the expected behavior? why? and why is it different here?

>
> > 2) Although quickfix performance got much better, I'm afraid there is
> > more to do:
> > a) It takes me about 9 seconds(!) to add 68,505 entries to the qf
> > list (using simple :grep command. Time was measured from after the
> > grep command output finished, of course). Yegappan's test takes
> > only 2 seconds because all the results are from the same file. Try
> > to add the loop variable to the file name in his test and see what
> > happens...
> >
> > This is the profiler log of the search I did:
> >
> > % cumulative self self total
> > time seconds seconds calls ms/call ms/call name
> > 40.63 1.69 1.69 68510 0.02 0.02 buf_valid
> > 21.88 2.60 0.91 68505 0.01 0.02 buflist_findname_stat
> > 12.38 3.12 0.52 127664549 0.00 0.00 otherfile_buf
> > 4.81 3.32 0.20 4345 0.05 0.05 buflist_findnr
>
> Well, when adding an entry a buffer is created, so that we keep track of
> the actual name when doing ":cd". And it avoids duplicates.
> buf_valid() is often needed to check if an autocommand didn't delete a
> buffer. Both of these are hard to get rid of.

This can really slow down Vim. I'm not sure if my test cases here are realistic, but I'm sure that if you keep Vim session active for enough time you'll probably get thousands of unlisted buffers which will slow down everything.
In this case, if there is nothing else can be done (like using another data structure for qf entries) I think that command for deleting quickfix list is necessary. As I mentioned earlier, :call setqflist([], 'r') is not good enough and it also doesn't free the associated buffers.
Maybe a simple mechanism for search, which does not use hidden marks, should be considered. When you're starting a search command you don't really know how many results you'll get so the last thing you want to do is to add 100000 results to the qf list. You need to be able to review the results, maybe filter them, and then add them to the quickfix if needed (in many cases it is not needed, like when you only want to review the code without changing anything). Actually, this is what I've done in the Agrep plugin..

>
> > b) I remember you did 2 things in order to avoid the line
> > adjustments when it's not necessary: check first if the buffer has a
> > quickfix entry and don't call line adjustment when adding a line at
> > the end.
> >
> > I still see that adding to a buffer when there are many qf entries
> > is very slow. After I had the 68505 results of the :grep command,
> > adding ~84000 lines to a buffer (from channel output) became really
> > slow. This is the profiler log:
>
> How are the lines added?

The lines are added by using 'out_io': 'buffer'. Without having many (4000) unlisted buffers adding 80000 lines takes about 1.2 seconds. The unlisted buffers slow this down to ~6 seconds. If, in addition, I add an out_cb to this job which calls setbufvar() it can take even 15 seconds.

>
> > Each sample counts as 0.01 seconds.
> > % cumulative self self total
> > time seconds seconds calls s/call s/call name
> > 31.46 6.98 6.98 149100 0.00 0.00 buf_valid
> > 29.20 13.46 6.48 80590 0.00 0.00 buflist_findpat
> > 26.66 19.38 5.92 84950 0.00 0.00 buflist_findnr
> > 3.56 20.17 0.79 68507 0.00 0.00 buflist_findname_stat
> > 2.28 20.67 0.51 127673186 0.00 0.00 otherfile_buf
> >
> > It seems like checking the buffer each time alone takes a lot of time.
> > Is there any way to optimize the above functions?
> > Also, why I still see all these buffer functions even though all the
> > lines were added at the end of the buffer?
>
> If you use 'errorformat' it will locate the file name and find out what
> buffer has that file name.
>

I'm not using this in this case.

[Bram Moolenaar]

Ramel Eshed wrote:

> > > 1) Actually, there is a bug here which is not exactly what I've
> > > described earlier. The issue is that :call setqflist([]), instead of
> > > adding one more list after the last list, will clear the next list and
> > > delete the ones after. For example: let's say I used :grep 4 times so
> > > I have now 4 lists. Now, if I do :colder 3, the first list becomes the
> > > current list. :call setqflist([]) will empty list 2, and delete lists
> > > 3 and 4.
> >
> > We do remove newer lists when adding a new list before the end, but in
> > this case it should not happen.
> I'm not sure I understand. Is removing all the newer lists are the
> expected behavior? why? and why is it different here?

Removing newer lists is what happens when inserting a list. But you are
not inserting here, you are clearing it.

The only way to avoid creating buffers is by intentionally dropping some
functionality and only keeping the file name. Perhaps we can still
expand it to a full path (that's also not a cheap operation, but at
least it's only once per entry).

The file name would be turned into a buffer only when jumping to the
location then.

I suppose this is a property of the quickfix list. Perhaps with a
function like "setqfflag()"? Would not work with commands like :cfile
though.

> > > b) I remember you did 2 things in order to avoid the line
> > > adjustments when it's not necessary: check first if the buffer has a
> > > quickfix entry and don't call line adjustment when adding a line at
> > > the end.
> > >
> > > I still see that adding to a buffer when there are many qf entries
> > > is very slow. After I had the 68505 results of the :grep command,
> > > adding ~84000 lines to a buffer (from channel output) became really
> > > slow. This is the profiler log:
> >
> > How are the lines added?
> The lines are added by using 'out_io': 'buffer'. Without having many
> (4000) unlisted buffers adding 80000 lines takes about 1.2 seconds.
> The unlisted buffers slow this down to ~6 seconds. If, in addition, I
> add an out_cb to this job which calls setbufvar() it can take even 15
> seconds.

OK, so it's not adding the lines that's slow, but checking every time
whether the buffer pointer we have is still valid.

> > > Each sample counts as 0.01 seconds.
> > > % cumulative self self total
> > > time seconds seconds calls s/call s/call name
> > > 31.46 6.98 6.98 149100 0.00 0.00 buf_valid
> > > 29.20 13.46 6.48 80590 0.00 0.00 buflist_findpat
> > > 26.66 19.38 5.92 84950 0.00 0.00 buflist_findnr
> > > 3.56 20.17 0.79 68507 0.00 0.00 buflist_findname_stat
> > > 2.28 20.67 0.51 127673186 0.00 0.00 otherfile_buf
> > >
> > > It seems like checking the buffer each time alone takes a lot of time.
> > > Is there any way to optimize the above functions?
> > > Also, why I still see all these buffer functions even though all the
> > > lines were added at the end of the buffer?
> >
> > If you use 'errorformat' it will locate the file name and find out what
> > buffer has that file name.
> >
> I'm not using this in this case.

Ehm, you do get those file names added as buffers, thus that must happen
somewhere.

Anyway, it seems that your problem is that you use thousands of buffers
and that's something Vim wasn't prepared for.

--
hundred-and-one symptoms of being an internet addict:
80. At parties, you introduce your spouse as your "service provider."

[Ramel Eshed]

On Thursday, June 9, 2016 at 6:53:06 PM UTC+3, Bram Moolenaar wrote:
> Ramel Eshed wrote:
>
> > > > 1) Actually, there is a bug here which is not exactly what I've
> > > > described earlier. The issue is that :call setqflist([]), instead of
> > > > adding one more list after the last list, will clear the next list and
> > > > delete the ones after. For example: let's say I used :grep 4 times so
> > > > I have now 4 lists. Now, if I do :colder 3, the first list becomes the
> > > > current list. :call setqflist([]) will empty list 2, and delete lists
> > > > 3 and 4.
> > >
> > > We do remove newer lists when adding a new list before the end, but in
> > > this case it should not happen.
> > I'm not sure I understand. Is removing all the newer lists are the
> > expected behavior? why? and why is it different here?
>
> Removing newer lists is what happens when inserting a list. But you are
> not inserting here, you are clearing it.

I'm using setqflist([]) with no 'r' flag. Doesn't it suppose to insert a list?
Anyway, is there a reason to delete the newer lists?

I'm not sure, but maybe something like file inode index number can be used for that and will be faster?

>
> The file name would be turned into a buffer only when jumping to the
> location then.
>
> I suppose this is a property of the quickfix list. Perhaps with a
> function like "setqfflag()"? Would not work with commands like :cfile
> though.

Sounds good to me, although I didn't understand the :cfile problem you mentioned.

>
> > > > b) I remember you did 2 things in order to avoid the line
> > > > adjustments when it's not necessary: check first if the buffer has a
> > > > quickfix entry and don't call line adjustment when adding a line at
> > > > the end.
> > > >
> > > > I still see that adding to a buffer when there are many qf entries
> > > > is very slow. After I had the 68505 results of the :grep command,
> > > > adding ~84000 lines to a buffer (from channel output) became really
> > > > slow. This is the profiler log:
> > >
> > > How are the lines added?
> > The lines are added by using 'out_io': 'buffer'. Without having many
> > (4000) unlisted buffers adding 80000 lines takes about 1.2 seconds.
> > The unlisted buffers slow this down to ~6 seconds. If, in addition, I
> > add an out_cb to this job which calls setbufvar() it can take even 15
> > seconds.
>
> OK, so it's not adding the lines that's slow, but checking every time
> whether the buffer pointer we have is still valid.
>

Sorry, my mistake. I don't know what was wrong in my previous test (maybe I was testing it with the profiler build..) but the only noticed difference I see is when calling to setbufvar(). No real difference when only adding lines.

> > > > Each sample counts as 0.01 seconds.
> > > > % cumulative self self total
> > > > time seconds seconds calls s/call s/call name
> > > > 31.46 6.98 6.98 149100 0.00 0.00 buf_valid
> > > > 29.20 13.46 6.48 80590 0.00 0.00 buflist_findpat
> > > > 26.66 19.38 5.92 84950 0.00 0.00 buflist_findnr
> > > > 3.56 20.17 0.79 68507 0.00 0.00 buflist_findname_stat
> > > > 2.28 20.67 0.51 127673186 0.00 0.00 otherfile_buf
> > > >
> > > > It seems like checking the buffer each time alone takes a lot of time.
> > > > Is there any way to optimize the above functions?
> > > > Also, why I still see all these buffer functions even though all the
> > > > lines were added at the end of the buffer?
> > >
> > > If you use 'errorformat' it will locate the file name and find out what
> > > buffer has that file name.
> > >
> > I'm not using this in this case.
>
> Ehm, you do get those file names added as buffers, thus that must happen
> somewhere.
>
> Anyway, it seems that your problem is that you use thousands of buffers
> and that's something Vim wasn't prepared for.

I know that this is not the typical use case but, as I said, during a long Vim session there might be thousands of unlisted buffers eventually. I think that the solution you've proposed should make quickfix much more robust.

[Bram Moolenaar]

I wrote:

> Ramel Eshed wrote:
>
> > > > 1) Actually, there is a bug here which is not exactly what I've
> > > > described earlier. The issue is that :call setqflist([]), instead of
> > > > adding one more list after the last list, will clear the next list and
> > > > delete the ones after. For example: let's say I used :grep 4 times so
> > > > I have now 4 lists. Now, if I do :colder 3, the first list becomes the
> > > > current list. :call setqflist([]) will empty list 2, and delete lists
> > > > 3 and 4.
> > >
> > > We do remove newer lists when adding a new list before the end, but in
> > > this case it should not happen.
> > I'm not sure I understand. Is removing all the newer lists are the
> > expected behavior? why? and why is it different here?
>
> Removing newer lists is what happens when inserting a list. But you are
> not inserting here, you are clearing it.

Somehow I managed to read that you were using:

call setqflist([], 'r')

Which clears the list. But it appears you were doing:

call setqflist([])

Which appends a new list. Then all following (newer) lists are deleted.
That's how it is supposed to work.


--
ASCII stupid question, get a stupid ANSI.

[Bram Moolenaar]

Ramel Eshed wrote:

> 1) Actually, there is a bug here which is not exactly what I've
> described earlier. The issue is that :call setqflist([]), instead of
> adding one more list after the last list, will clear the next list and
> delete the ones after. For example: let's say I used :grep 4 times so
> I have now 4 lists. Now, if I do :colder 3, the first list becomes the
> current list. :call setqflist([]) will empty list 2, and delete lists
> 3 and 4.

Looking into this again, this is intended behavior. The problem is that
you can't easily see an overview of the error lists. I'll add the
:chistory command for that.

The help explains this:

When ":colder" has been used and ":make" or ":grep" is used to add a new error
list, one newer list is overwritten. This is especially useful if you are
browsing with ":grep" |grep|. If you want to keep the more recent error
lists, use ":cnewer 99" first.


You can also use setqflist([], 'r') to clear one error list.

--
You were lucky to have a LAKE! There were a hundred and sixty of
us living in a small shoebox in the middle of the road.