Whoosh, full text search status (bigdash.py)

[Ville M. Vainio]

I have now pushed full text search support (operational already, not
as nice as it will be later on ;-).

Screenshot here:

https://plus.google.com/103097156557482112329/posts/HnobiCY4QJR

Usage in a nutshell:

- easy_install Whoosh
- enable plugin "bigdash.py" (implementation itself is in leofts.py,
but it's enough to enable bigdash.py).
- Open the .leo files you want to search
- alt-x global-search

And in the dashboard line entry:

- Do "fts init" to create the index from all the open files
- Do "f foo bar" to search for terms "foo" and "bar".

(quick ref in the attached screenshot)

Note that you don't need to have the .leo files open anymore.

What's missing:

- Clicking on links won't select the position if the .leo file is not
open. That can be arranged later on (e.g. clicking on UNL could open
the file and navigate to the tree position).

- Source snippets won't be shown (as you can see from screenshots). It
will be quite easy to show snippets with highlighted items FOR LEO
FILES THAT ARE OPEN, I just didn't have time to do it now. Index
doesn't have full contents of the nodes (by design, I could store
those in the index too but I don't want to bloat it up), but yet I
have some ambitions to show also snippets from closed leo files. For
this, I need a lightweight xml file parsing and some way to drill into
@thin files and parse sentinels as well,

[Kent Tenney]

Should clicking on a result take me to the file?

> --
> You received this message because you are subscribed to the Google Groups "leo-editor" group.
> To post to this group, send email to leo-e...@googlegroups.com.
> To unsubscribe from this group, send email to leo-editor+...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/leo-editor?hl=en.
>

[Ville M. Vainio]

Clicking on the hyperlink (blue underlined text, of headstring) takes
you to the node.

If the .leo file is not already open, it doesn't work in current
implementation. So you have to open the file manually, then click on
the link.

[Kent Tenney]

On Mon, Mar 5, 2012 at 3:21 PM, Ville M. Vainio <viva...@gmail.com> wrote:
> Clicking on the hyperlink (blue underlined text, of headstring) takes
> you to the node.

Not happening for me. The link is blue underlined, clicking it doesn't
change focus.

Missing a plugin?

[Ville M. Vainio]

Ensure you have the correct tab selected.

Depending on what you mean by focus.... it moves the selection in the
tree, but does not change the UI focus to the right window or tab.

[Ville M. Vainio]

On Tue, Mar 6, 2012 at 12:05 AM, Kent Tenney <kte...@gmail.com> wrote:

> Not happening for me. The link is blue underlined, clicking it doesn't
> change focus.
>
> Missing a plugin?

Can you paste what you see on terminal where you launched leo, when
clicking on the links?

[Kent Tenney]

Clicking a link doesn't put anything in the terminal, but I now see
what is happening.

I tend to work with many Leo files opened in the tabbed interface.
When I click a result link, I was expecting to be transported to that node.
I now see that if I click the file tab for a result link, the found
node is focused.

I certainly _expected_ to be taken to the result (as a 'nav' result does),
do you prefer staying on the result window and selecting the file?

Also, the filename is in small italics after some context lines, if focus
is to stay on the search pane, I'd prefer the filename be larger, near
the result link.

I guess the advantage of this scheme is the ability to first click several
results, close the search, go from tab to tab, each one focused per the search.

Maybe that makes more sense than what I expect ...

Thanks,
Kent

[Ville M. Vainio]

No, this is a missing feature. I want it to select the tab so you see
where the focus was moved.

[Terry Brown]

On Wed, 7 Mar 2012 00:06:34 +0200
"Ville M. Vainio" <viva...@gmail.com> wrote:

> No, this is a missing feature. I want it to select the tab so you see
> where the focus was moved.

c.bringToFront() brings the tab to front, I think. Something I hacked
in a while ago.

Cheers -Terry

[Ville M. Vainio]

On Wed, Mar 7, 2012 at 2:44 AM, Terry Brown <terry_...@yahoo.com> wrote:

>> No, this is a missing feature. I want it to select the tab so you see
>> where the focus was moved.
>
> c.bringToFront() brings the tab to front, I think.  Something I hacked
> in a while ago.

Indeed it does, thanks for the feature :).

global-search now brings focus to right tab + node when clicking on a
link in search hits.

[lewis]

Ville,

This is great thankyou.
I notice when I do a search for a term I get this traceback in the
log:
Leo Log Window
Leo 4.9.1 devel, build 5091, 2012-03-08 23:35:48
Python 3.2.3, qt version 4.8.0
Windows 6, 1, 7601, 2, Service Pack 1
Traceback (most recent call last):
File "C:\Python32\Lib\site-packages\leo-editor\leo\plugins
\bigdash.py", line 199, in docmd

r = h(self, t)

File "C:\Python32\Lib\site-packages\leo-editor\leo\plugins
\bigdash.py", line 119, in do_fts

ss = unicode(qs)

NameError
:
global name 'unicode' is not defined
[end]

Regards
Lewis

[Edward K. Ream]

On Mar 9, 5:40 am, lewis <lewisn...@operamail.com> wrote:

> ss = unicode(qs)
>
> NameError
> :
> global name 'unicode' is not defined
> [end]

Fixed on the trunk at rev 5099. I changed unicode to g.toUnicode,
which is the Leo way.

Also added bigdash.py to leoPluginsRef.py.

Ville, presumably there is a reason why bigdash.py doesn't import
leo.core.leoGlobals as g at the top level. Can you tell us why?

Edward

[Ville M. Vainio]

It's because I started prototyping it outside Leo, to get quicker
turnaround (just edit the code, and run "python bigdash.py" on the
terminal.

[Edward K. Ream]

On Mar 9, 9:40 am, "Ville M. Vainio" <vivai...@gmail.com> wrote:
> It's because I started prototyping it outside Leo, to get quicker
> turnaround (just edit the code, and run "python bigdash.py" on the
> terminal.

How do you feel about importing leoGlobals at the top level now?

EKR

[Ville M. Vainio]

On Fri, Mar 9, 2012 at 5:50 PM, Edward K. Ream <edre...@gmail.com> wrote:

> How do you feel about importing leoGlobals at the top level now?

I'm ok with that, but don't really see a benefit. Code that can run
without Leo running is more valuable than one that can't, so I'd
rather leave it out for quick tests in the future.

[Edward K. Ream]

I see. I was confused by this import::

try:
import leo.plugins.leofts as leofts
except ImportError:
leofts = None

But we can't use this approach for leoGlobals: g.toUnicode is
necessary for Python 3k. It might be best to copy g.toUnicode (and
its several helpers) into bigdash.py.

Edward

[viva...@gmail.com]

Typically g is availabe at top level (see set_leo), so it's no problem leaving it like it is

On 9.3.2012 18:40 Edward K. Ream wrote:

On Fri, Mar 9, 2012 at 9:57 AM, Ville M. Vainio <viva...@gmail.com> wrote:

> On Fri, Mar 9, 2012 at 5:50 PM, Edward K. Ream <viva...@gmail.com> wrote:
>
>> How do you feel about importing leoGlobals at the top level now?
>
> I'm ok with that, but don't really see a benefit. Code that can run
> without Leo running is more valuable than one that can't, so I'd
> rather leave it out for quick tests in the future.

I see. I was confused by this import::

try:
import leo.plugins.leofts as leofts
except ImportError:
leofts = None

But we can't use this approach for leoGlobals: g.toUnicode is
necessary for Python 3k. It might be best to copy g.toUnicode (and
its several helpers) into bigdash.py.

Edward

--

You received this message because you are subscribed to the Google Groups "leo-editor" group.

To post to this group, send email to viva...@gmail.com
To unsubscribe from this group, send email to viva...@gmail.com

[Ville M. Vainio]

Full text search now highlights hits in the documents that are currently open:

https://plus.google.com/103097156557482112329/posts/HiHcvLgiJ47

Documents that are not open will (still) just show heading and UNL.

[Ville M. Vainio]

Next task 1: make it tokenize source code better.

As it is, it can't treat foo(bar) as tokens "foo" and "bar", which is
not great for scanning throught sources.

Next task 2: provide "text completion" from the words in full text search index.

[lewis]

Immediately after alt-x global-search, the Leo search window opens and
I get this traceback:

Leo Log Window
Leo 4.9.1 devel, build 5113, 2012-03-11 17:04:29

Python 3.2.3, qt version 4.8.0
Windows 6, 1, 7601, 2, Service Pack 1
Traceback (most recent call last):
File "C:\Python32\Lib\site-packages\leo-editor\leo\plugins

\qtGui.py", line 8657, in eventFilter

ret = k.masterKeyHandler(event)

File "C:\Python32\Lib\site-packages\leo-editor\leo\core\leoKeys.py",
line 2893, in masterKeyHandler

done,val = k.doMode(event,state,stroke)

File "C:\Python32\Lib\site-packages\leo-editor\leo\core\leoKeys.py",
line 2971, in doMode

val = k.callStateFunction(event) # Calls end-command.

File "C:\Python32\Lib\site-packages\leo-editor\leo\core\leoKeys.py",
line 2945, in callStateFunction

val = k.state.handler(event)

File "C:\Python32\Lib\site-packages\leo-editor\leo\core\leoKeys.py",
line 2033, in fullCommand

k.callAltXFunction(k.mb_event)

File "C:\Python32\Lib\site-packages\leo-editor\leo\core\leoKeys.py",
line 2079, in callAltXFunction

func(event)

File "C:\Python32\Lib\site-packages\leo-editor\leo\core
\leoCommands.py", line 1086, in minibufferCallback

retval = function(keywords)

File "C:\Python32\Lib\site-packages\leo-editor\leo\plugins

\bigdash.py", line 61, in global_search_f

leofts.init()

AttributeError
:
'NoneType' object has no attribute 'init'
[end]

Regards
Lewis

[Ville M. Vainio]

Good catch, I have to improve error handling.

You don't have Whoosh installed, install it:

easy_install Whoosh

> --
> You received this message because you are subscribed to the Google Groups "leo-editor" group.

> To post to this group, send email to leo-e...@googlegroups.com.
> To unsubscribe from this group, send email to leo-editor+...@googlegroups.com.

[zpcspm]

On Sunday, March 11, 2012 11:40:49 AM UTC+2, Ville M. Vainio wrote:

easy_install Whoosh

Sorry for offtopic, may ask why are you using easy_install? Is it a developed habit or you have any particular reasons for installing third party python software this way? I am asking because I keep hearing good things about pip (though I've never started to use it yet, when I need a third party python thing, I usually build a package and install it using the package manager of my Linux distro).

[viva...@gmail.com]

Both work for whoosh; setuptools is a devil i know ;).

--
You received this message because you are subscribed to the Google Groups "leo-editor" group.

To view this discussion on the web visit https://groups.google.com/d/msg/leo-editor/-/h3Kz_WZOcdcJ.

[lewis]

Thanks. There are no errors with build 5125.
I haven't installed Whoosh but am following the easy_install/pip
debate....
http://stackoverflow.com/questions/4750806/how-to-install-pip-on-windows

Lewis

[Ville M. Vainio]

I don't see much benefit in tracking the debate, since both seem to do the job. Use the one that is easier to install on your computer (setuptools, I.e. easy_install seemed easier to install on windows machine)

[Matt Wilkie]

> I don't see much benefit in tracking the debate, since both seem to do the
> job. Use the one that is easier to install on your computer (setuptools,
> I.e. easy_install seemed easier to install on windows machine)

FWIW, I just installed pip for the first time on windows today.
Relatively straightforward if one has curl or wget in path:

{{{
curl http://python-distribute.org/distribute_setup.py | python
curl --insecure
https://raw.github.com/pypa/pip/master/contrib/get-pip.py | python

copy con c:\pip.bat
%pythonhome%\Scripts\pip.exe %* [ctrl-Z]
}}}

--
-matt

[Terry Brown]

On Mon, 5 Mar 2012 22:58:33 +0200

"Ville M. Vainio" <viva...@gmail.com> wrote:

> but yet I
> have some ambitions to show also snippets from closed leo files. For
> this, I need a lightweight xml file parsing and some way to drill into
> @thin files and parse sentinels as well,

leosax is lightweight xml file parsing, but doesn't drill into @thin
files etc.

./leo/external/leosax.py

Thought there was a demo .leo file for it, but don't see it now.

Cheers -Terry

[Terry Brown]

On Mon, 19 Mar 2012 22:05:58 -0500
Terry Brown <terry_...@yahoo.com> wrote:

> Thought there was a demo .leo file for it, but don't see it now.

This example probably suffices :-)

http://groups.google.com/group/leo-editor/browse_thread/thread/cf5ab54f29a6c128

Cheers -Terry

[Terry Brown]

On Mon, 5 Mar 2012 22:58:33 +0200
"Ville M. Vainio" <viva...@gmail.com> wrote:

> I have now pushed full text search support (operational already, not
> as nice as it will be later on ;-).

That's very cool, nice job. I wonder if we can write a headless
nullgui based spider to regularly index all the .leo files it can find.

Cheers -Terry

[Ville M. Vainio]

If leoBridge still work, sure! This would be very handy - we could also use it to do indexing when explicitly asked, as it would allow doing it in a background process instead of blocking the ui thread, like in current implementation.

Headless tool would be interesting also for search/lookup purposes.

Cheers -Terry

[Edward K. Ream]

On Mon, Mar 19, 2012 at 11:28 PM, Ville M. Vainio <viva...@gmail.com> wrote:
> If leoBridge still work, sure!

If leoBridge didn't work all of Leo's unit tests would fail when run
externally. See:
@file leoDynamicTest.py-->main & helpers (leoDynamicTest.py)

EKR