Search & Replace Too Slow

[Sadi]

Hi,
I've just discovered Editra -- hoping that I could perhaps saved from
jEdit's (actually Java's) slow speed in loading largish (e.g. more
than 1MB) plain text files.
It was good to see how quickly you can open the same file in Editra.
But I became disappointed when I tried performing a simple RegEx
Search & Replace (e.g. removing all HTML tags: <.+?>)
I had to wait about 10 secs each time I press "Find & Replace"
I wonder if there's something I'm missing here or this is a bug or
this is how Python works: open file faster, but perform some
operations slower quickly than Java.
Regards,
Sadi

[Cody Precord]

Hi,

Do you have a sample file I can test with or is it just some HTML file
that is about 1 MB in size?

I haven't ever noticed much issue with the search speed so haven't
spent much time on optimizing it so I am sure there are some things
that can be done to make it faster.


Thanks,

Cody

[Sadi]

Hi,

Here's the test I've carried out this morning:

1) Open "http://en.wikipedia.org/wiki/Human_rights_declaration" and
Save As HTML.
2) Open the saved HTML file, Select All, Copy and Paste about 10 times
and then save it so that you have a large (>1MB) HTML file.
3) Go to the beginning of the file and press Ctrl+R (Find & Replace)
4) Try finding "human" and replacing with "animal" (using "Find and
Replace" and then "Replace All")
(a) It finds the text, but it does not replace!!! *
(b) "Find and Replace" operation is incredibly slow
(c) During the "Replace All" operation you might think the application
has frozen.

* Once I managed to get the text replaced when I ticked "Match case"
and "Whole word" checkboxes but I couldn't repeat it, which is
strange...

The problem is the same with "Regular expression" (e.g. when you try
removing all HTML tags using "<.+?>" )

My conclusion: it seems there is still a lot to be done on Find and
Replace so that it's not so buggy and so slow (practically unusable)

I hope this helps your valuable efforts to develop a Python-based
alternative to the Java-based jEdit ;-)

Kind regards,

Sadi

[Cody]

Hi,

On Tue, Oct 11, 2011 at 2:30 AM, Sadi <sadiy...@gmail.com> wrote:
> Hi,
>
> Here's the test I've carried out this morning:
>
> 1) Open "http://en.wikipedia.org/wiki/Human_rights_declaration" and
> Save As HTML.
> 2) Open the saved HTML file, Select All, Copy and Paste about 10 times
> and then save it so that you have a large (>1MB) HTML file.

Ok, did this and have an html file that about 1.5 MB in size

> 3) Go to the beginning of the file and press Ctrl+R (Find & Replace)
> 4) Try finding "human" and replacing with "animal" (using "Find and
> Replace" and then "Replace All")
> (a) It finds the text, but it does not replace!!! *

I clicked Find and it instantly located and selected the first
instance of 'human'. Its expected that it wont replace until you click
Replace.

> (b) "Find and Replace" operation is incredibly slow

Not seeing this at all, I continue hitting replace after the first
find and it instantly jumps and replaces the next match as fast as I
can click the button. This holds true for both the Find and Replace
buttons using either a basic search or regex search.

> (c) During the "Replace All" operation you might think the application
> has frozen.
>

I can reproduce this, it blocked for about 15 seconds or so while
replacing the 3200~ instances in this example. I think this should be
easy to improve.

>
> My conclusion: it seems there is still a lot to be done on Find and
> Replace so that it's not so buggy and so slow (practically unusable)
>

This is quite an exaggeration, the find/replace feature is very
powerful and works very well. Thousands of people use it everyday for
several years without issue.

What operating system are you using Editra on?
What kind of computer? (CPU/RAM)?
Are you running a binary version or from source?
If from source what version of Python/wxPython are you using?

Cody

[Sadi]

Thanks a lot Cody for trying to help me.

Apologies for my "harsh" comment at the end, it only reflects the situation I've found myself in, which might as well be a result of my own fault somewhere ;-)

Here are my hardware/software details:

I'm using a netbook connected to a 19" CRT monitor with Intel Atom N270 processor, 1GB RAM, Kubuntu 11.04, Python 2.7.1-0ubuntu5, python-wxversion 2.8.11.0-0ubuntu8, python-wxgtk2.8 2.8.11.0-0ubuntu8

(I'm a relatively new Linux user who doesn't know much about Python and wxPython)

Following your last mail, I tried both the older and stable version from the repositories (automatic install) and the latest development version (manual install) but it didn't make any apparent change in my problems.

First, it seems the problem of seeing no action is another strange problem. The open file sometimes is treated as "read-only" by Editra - I cannot even delete anything in the file. It happened with the HTML file I've created for testing purposes. I then opened a blank file, copied and pasted the contents into the new blank file, and saved it with a different name, and then I could edit it. Nothing like that happens in other text editors (i.e. Kate and jEdit)

But I suggest we ignore this for the moment, and focus on my Find/Replace problem.

Here are my latest test results using the same file and search texts:

Application          Opening       Text Find/Replace      RegEx Find/Replace

Kate                   4 secs       5 secs              4 secs

jEdit                   130 secs       25 secs                    20 secs

Editra           15 secs       140 secs                  Ages (I gave up and terminated Python after 5 minutes)

By the way, why I'm not content with Kate is that it doesn't have Macro feature, which I often use (mostly involving various Search&Replace operations)

In case it might be helpful, here are some screen messages I've seen during the installation of Editra from terminal as per your instructions.

(python:6041): Gtk-CRITICAL **: IA__gtk_widget_add_accelerator: assertion `GTK_IS_WIDGET (widget)' failed

(python:6041): Gtk-CRITICAL **: IA__gtk_widget_add_accelerator: assertion `GTK_IS_WIDGET (widget)' failed

Traceback (most recent call last):

  File "/usr/lib/pymodules/python2.7/Editra/src/ed_pages.py", line 968, in OnIdle

    page.DoOnIdle()

  File "/usr/lib/pymodules/python2.7/Editra/src/ed_editv.py", line 181, in DoOnIdle

    if self.IsLoading():

  File "/usr/lib/pymodules/python2.7/Editra/src/ed_stc.py", line 1195, in IsLoading

    return self._loading is not None

AttributeError: 'EdEditorView' object has no attribute '_loading'

Also, for your information, there is no such folder as /usr/lib/pymodules/python2.7/Editra in my system.

I think the correct location (and where Editra is installed at the end) is: /usr/local/lib/python2.7/dist-packages/Editra

Thank you once again very much.

Kind regards,

Sadi

[Cody]

Hi,

On Wed, Oct 12, 2011 at 2:11 AM, Sadi <sadiy...@gmail.com> wrote:
> Thanks a lot Cody for trying to help me.
> Apologies for my "harsh" comment at the end, it only reflects the situation
> I've found myself in, which might as well be a result of my own fault
> somewhere ;-)
> Here are my hardware/software details:
> I'm using a netbook connected to a 19" CRT monitor with Intel Atom N270
> processor, 1GB RAM, Kubuntu 11.04, Python 2.7.1-0ubuntu5, python-wxversion
> 2.8.11.0-0ubuntu8, python-wxgtk2.8 2.8.11.0-0ubuntu8

Ok, I think we need to do a little managing of expectations here. You
are running on a low powered netbook, there is only so fast that most
applications can be running on this setup.

> (I'm a relatively new Linux user who doesn't know much about Python and
> wxPython)
>
> Following your last mail, I tried both the older and stable version from the
> repositories (automatic install) and the latest development version (manual
> install) but it didn't make any apparent change in my problems.

The one in the Ubuntu repository is very old, guessing that the person
that was managing that package may have moved on to something else
sometime back. The ones released on editra.org are the way to go
(current is 0.6.77).

> First, it seems the problem of seeing no action is another strange problem.
> The open file sometimes is treated as "read-only" by Editra - I cannot even
> delete anything in the file. It happened with the HTML file I've created for
> testing purposes. I then opened a blank file, copied and pasted the contents
> into the new blank file, and saved it with a different name, and then I
> could edit it. Nothing like that happens in other text editors (i.e. Kate
> and jEdit)

Is the file marked with a Read Only bit on your hard disk? What does
the encoding say in the status bar after Editra has opened the file?

Currently Editra does not support overwriting locked or read only
files, user needs make the file writable. Allowing overwrite will be
added in future versions. For now you would need to make the file
writable before opening it.

The other case where it could have been read only is if the editor
failed to properly detect the file's encoding and just loaded it as
raw bytes. Editing is disabled in the case and the encoding is listed
as 'binary' since it cant be guaranteed that the program will be able
to properly write out the file again afterwards without the encoding.

A final case that the file will be read only is during a long running
file load, if a file is more than a few MB it is loaded asynchronously
in a background thread and incrementally loaded into the UI to speed
up the process. Until the file is completely loaded the buffer will be
locked as read only.

> But I suggest we ignore this for the moment, and focus on my Find/Replace
> problem.
> Here are my latest test results using the same file and search texts:
> Application          Opening       Text Find/Replace      RegEx Find/Replace
> Kate                   4 secs       5 secs              4 secs
> jEdit                   130 secs       25 secs                    20 secs
> Editra           15 secs       140 secs                  Ages (I gave up and
> terminated Python after 5 minutes)

I retested on my netbook which is running Windows 7 (Intel Atom n550 /
2GB ram). Using the incremental search/replace it jumps to the next
item instantly in most cases a few times there was a .5 to 1 second
pause. As with the previous test the replace all was rather slow, I
think there are a few things that could be done to make the replace
all faster but don't think there is much that can be done for the
incremental search while still retaining all the search features.

So, if your planning on sticking with using a low powered netbook for
development work I would suggest maybe using VIM (gvim) or possibly
checking out geany. Because in all reality Editra is probably going to
require a little more memory and (to an extent) cpu power than your
system has in order to perform at its best.

>
> (python:6041): Gtk-CRITICAL **: IA__gtk_widget_add_accelerator: assertion
> `GTK_IS_WIDGET (widget)' failed
> (python:6041): Gtk-CRITICAL **: IA__gtk_widget_add_accelerator: assertion
> `GTK_IS_WIDGET (widget)' failed

These are harmless and can be ignored.

> Traceback (most recent call last):
>   File "/usr/lib/pymodules/python2.7/Editra/src/ed_pages.py", line 968, in
> OnIdle
>     page.DoOnIdle()
>   File "/usr/lib/pymodules/python2.7/Editra/src/ed_editv.py", line 181, in
> DoOnIdle
>     if self.IsLoading():
>   File "/usr/lib/pymodules/python2.7/Editra/src/ed_stc.py", line 1195, in
> IsLoading
>     return self._loading is not None
> AttributeError: 'EdEditorView' object has no attribute '_loading'
>

This looks like a crash while loading a file. This bug existed in an
old version of Editra (guessing you were running the 0.5.30 version
from the ubuntu repo?)

Cody

[Sadi Yumusak]

Thank you very much Cody...

It seems the problems I've encountered (including the file being opened as "read only" although it isn't) are probably related with the fact that my hardware is below the level Editra can peform well.
I'll try Editra with a better hardware later, and probably start using it again when I get a better machine ;-)

Kind regards,

Sadi

[Cody]

Hi,

No, the read only thing has nothing to do with performance. Without
more information I can't even comment if it is problem or expected
behavior.

The find/replace speed issue though is performance related though ;) I
will look into optimizing the replace all routine sometime soon but as
mentioned the Find Next is probably about as good as it can get.
Thanks for the feedback.

Cody

[Cody]

Hi,

On Wed, Oct 12, 2011 at 11:11 AM, Cody <codyp...@gmail.com> wrote:
> The find/replace speed issue though is performance related though ;) I
> will look into optimizing the replace all routine sometime soon but as
> mentioned the Find Next is probably about as good as it can get.

Took awhile to get time to look into this but I have found a way to
massively improve the Replace All performance and have committed a fix
to the current svn.

Didn't do exact timings but on my old Pentinum 4 Desktop doing a
Replace All in a 3 MB file with over 9000 matches to replace:
* Old Time: ~5 minutes
* New Time: ~5 seconds

Cody

[Sadi Yumuşak]

Hooray! Hooray! Hooray!

And what an improvement!

Now my poor netbook successfully performed 30 thousand something RegEx replacements in a 1.1 MB HTML file within about 5 seconds thanks to this imrovement :-)

I very much look forward to replacing jEdit with Editra as my Most Frequently Used Text Editor very soon now :-)

Thanks a lot, and a Happy New Year to you!

Sadi

---

[werner]

Can you please check that this is not causing a problem with find in
directory. I right click in a project folder (wxPythons' demo folder)
and search for "comtypes" and I get zero hits with Find options set to
default.

Werner

[Cody]

Hi,

This only changed how the replacements were performed in the STC.

You are getting zero hits because there seems to be no matches in that
folder. To verify I ran "grep -R comtypes demo/" on the 2.8.12 demo
folder I had on my system and it also found no hits.

Also make sure that "Search Recursively" is checked when doing
directory searches to get the most results.

Cody

[werner]

I don't know, either I didn't have my coffee or .....

You are right there aren't any in demo, I did some testing with the demo
stuff and got these import errors but they are from wx.lib. and not the
demo.

Sorry for having you chase a non issue.
Werner