Testing KeepNote 0.7 (include indexed full text search)

[rasmus]

Hey everyone,

I am beginning to test a new version of KeepNote (version 0.7).  It contains fully indexed search, which should be much faster than the manual full text search that I have implemented now.  I have also added an option to use smaller font sizes for the treeview and listview in order to view more notes at once (See "Look and Feel" options).  Please help me squash any bugs that you find in this release by submitting bugs to the issue tracker.

Thanks,
Matt

website: http://keepnote.org
download: http://keepnote.org/keepnote/download-test
issue tracker: http://code.google.com/p/keepnote/issues/list

[Chris]

I'm going to try it out. Is there anything specifically we should try
to to look for? Like any areas you may have modified enough to think
that something could be wrong there more than others?

[Chris]

WOW I just tried this thing and the searching is SO FAST. Awesome job!
I haven't tried it with any big notebooks(downsized all my notebooks)
but this would DEFINITELY be a speed boost to those too, I'm sure.
I'll keep on the look out for bugs too just wanted to say that this
was very good... I'll need to look at the code soon to learn how you
did it.

On Feb 2, 9:08 pm, rasmus <matt.ras...@gmail.com> wrote:

[rasmus]

Good question. Things to look for:

1. Does the full text search work for you? You will need a newer
version of sqlite (>= 3.5.0) than the one that ships with python 2.5.
I use the fts3 module of sqlite now in order to get much faster
search.
2. Do all the basic notebook manipulations work? I have been
refactoring the code that reads and writes notebooks, so make sure you
can rename, move, copy, delete, create new notes, create new
notebooks, etc. I don't think there should be any problems, but maybe
I missed something that some one else will catch.

Thats about it. I have tried to limit my work to just the search to
make sure it is working well before going to other features. Thanks
for all your help.

Matt

[Ton van Vliet]

On Wed, 2 Feb 2011 18:08:26 -0800 (PST), you wrote:

>I am beginning to test a new version of KeepNote (version 0.7). It contains
>fully indexed search, which should be much faster than the manual full text
>search that I have implemented now. I have also added an option to use
>smaller font sizes for the treeview and listview in order to view more notes
>at once (See "Look and Feel" options). Please help me squash any bugs that
>you find in this release by submitting bugs to the issue tracker.

Should this also improve startup times? or is that a 'co-incidental' improvement?

With 0.6.8 I have startup times in the 10 to 20 seconds range (no matter how often I perform a re-index)

With 0.7.0 I have startup times in the 1 to 2 seconds range (except for the first startup, where a re-index was performed automatically)

observation 1:
When I change the application font under preferences | look-and-feel, and press the 'apply' button, that only the list-view gets updated, but not the tree-view. The tree-view is updated only after a restart of the application.

observation 2:
When entering text in the search field, while typing, all nodes with names that contain the entered text appear twice in the dropdown list

Thanks again,
Ton.

[Matt Rasmussen]

Hi Ton,

On Thu, Feb 3, 2011 at 8:35 AM, Ton van Vliet <ton.va...@skynet.be> wrote:
> On Wed, 2 Feb 2011 18:08:26 -0800 (PST), you wrote:
>
>>I am beginning to test a new version of KeepNote (version 0.7).  It contains
>>fully indexed search, which should be much faster than the manual full text
>>search that I have implemented now.  I have also added an option to use
>>smaller font sizes for the treeview and listview in order to view more notes
>>at once (See "Look and Feel" options).  Please help me squash any bugs that
>>you find in this release by submitting bugs to the issue tracker.
>
> Should this also improve startup times? or is that a 'co-incidental' improvement?

I also worked on speeding up notebook loading. I'm glad you see an improvement.

> With 0.6.8 I have startup times in the 10 to 20 seconds range (no matter how often I perform a re-index)
>
> With 0.7.0 I have startup times in the 1 to 2 seconds range (except for the first startup, where a re-index was performed automatically)
>
> observation 1:
> When I change the application font under preferences | look-and-feel, and press the 'apply' button, that only the list-view gets updated, but not the tree-view. The tree-view is updated only after a restart of the application.

I notice this too. I don't think also GTK widgets reload their style
when its changed for some reason. An application restart is needed
for this option to take full effect.

> observation 2:
> When entering text in the search field, while typing, all nodes with names that contain the entered text appear twice in the dropdown list

Oh... Hmm let look into why that happened. Maybe the notebook
upgrading did that (indexed notes twice). Try running "Tools > Update
Notebook Index" to see if that solves the issue. In the meantime I
will work to figure out why that happened.

Thanks for the quick feedback. It really helps.

Matt

> Thanks again,
> Ton.
>
> --
> You received this message because you are subscribed to the Google Groups "keepnote" group.
> To post to this group, send email to keep...@googlegroups.com.
> To unsubscribe from this group, send email to keepnote+u...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/keepnote?hl=en.
>
>

[Jar]

Hello,

> I also worked on speeding up notebook loading.  I'm glad you see an improvement.

Startup times got many times better. Thanks.

> > observation 2:
> > When entering text in the search field, while typing, all nodes with names that contain the entered text appear twice in the dropdown list
>
> Oh... Hmm let look into why that happened.  Maybe the notebook
> upgrading did that (indexed notes twice).  Try running "Tools > Update
> Notebook Index" to see if that solves the issue.  In the meantime I
> will work to figure out why that happened.

The reindex did indeed solve the problem with the double presentation
in de dropdown box.

I have one new point. I can add a mailto link to some text. In earlier
versions the internet explorer is started. In 0.7 the default mail
application (outlook in my case) is loaded with a new message windows.
That is correct but in the background there is also a internet
explorer windows open. That is not necessary.

Thanks for the improvements.

Jeroen

[Ton van Vliet]

Hi Matt,

As Jeroen stated, re-indexing solved the double dropdown list issue.

Re-indexing is a 'major' event now: it takes 3 to 4 minutes to complete on a 385 folders/822 files notebook. Much longer than it used to, but as a compensation startup-time was greatly improved.

Thanks,
Ton.

[Matt Rasmussen]

Hi Ton and Jeroen,

On Thu, Feb 3, 2011 at 12:12 PM, Ton van Vliet <ton.va...@skynet.be> wrote:
> Hi Matt,
>
> As Jeroen stated, re-indexing solved the double dropdown list issue.

I have updated the testing version (in place) so that double indexing
the note titles does not occur when upgrading a notebook. So
hopefully that bug is solved.

> Re-indexing is a 'major' event now: it takes 3 to 4 minutes to complete on a 385 folders/822 files notebook. Much longer than it used to, but as a compensation startup-time was greatly improved.

The re-indexing is much more expensive now since I index the full text
of each note. I working to make re-indexing a very rare event. Every
note is indexed as you add/edit it. Re-indexing is only present in
KeepNote for situations where the notebook somehow got out of sync
with the index (bug or tampering with the notes outside of keepnote).
Let me know if you notice any major slow downs for other tasks.

I have been using KeepNote on a sshfs mount. It is slower, but
usable. I am curious how it works for other people who use network
filesystems (dropbox, NFS, etc).

Thanks for all the feedback. Keep it coming.

Matt

[Tom Syroid]

Matt,

Nice work.

Tested on the following:

Main notebook, Mint Julia 10, repos sit in a dropbox folder. No issues. As everyone notes, much faster startup times. (keepnote-0.7-1.deb)

Running off a USB installation at work (repos on Dropbox on USB; program code on the USB also). MUCH faster startup and execution times. What was slow but useable is now simply useable (given the media).

No issue with any commands. Split my notebooks into two (personal and system_admin), both relatively large. Search times are excellent.

Flawless upgrades on both test cases.

Sorry... "nothing" to report. Except all-around improved responsiveness. :-)

Tom.

[Tom Syroid]

Matt,

Gather this is due to the new indexing engine (correct me if I'm wrong please), and if so,

>> you need to put a cautionary note somewhere that upgrading to 0.7 changes your database structure <<

My keepnote databases are stored in a Dropbox folder,

Shared by all my machines (3 currently in total, plus a floater instance on USB),

Folder(s) updated to new indexing system when first opened with 0.7 instance,

Forgot and tried to open same data with an "older" version of keepnote (0.6.8) on another machine,

and the 0.6.8 instance just spun its wheels.

Not important for me, but I guess the first question someone might ask is "once the databases are updated to new indexing, can you so back somehow?"

/tom

[Matt Rasmussen]

Hi Tom,

Thanks for the feedback, it really helps.

On Wed, Feb 9, 2011 at 12:25 AM, Tom Syroid <tsy...@gmail.com> wrote:
> Matt,
> Gather this is due to the new indexing engine (correct me if I'm wrong
> please), and if so,
>>> you need to put a cautionary note somewhere that upgrading to 0.7 changes
>>> your database structure <<

Good point. I'll make the "update notebook" warning window popup to
let the user know that this is occurring when they first try to open a
notebook.

> My keepnote databases are stored in a Dropbox folder,
> Shared by all my machines (3 currently in total, plus a floater instance on
> USB),
> Folder(s) updated to new indexing system when first opened with 0.7
> instance,
> Forgot and tried to open same data with an "older" version of keepnote
> (0.6.8) on another machine,
> and the 0.6.8 instance just spun its wheels.

I will make a change that will cause 0.6.8 to refuse to open the
notebook (by indicating within the notebook that it has a newer
version).

> Not important for me, but I guess the first question someone might ask is
> "once the databases are updated to new indexing, can you so back somehow?"
> /tom

At this time, I do not support downgrades.

Matt

[Matt Rasmussen]

Hi everyone,

Thanks so much for helping me test KeepNote 0.7. I have fixed two main things

1. preferences likes "vertical/horizontal list view layout" should now
be changeable. I was not correctly saving these preferences before
changing them.
2. Upgrading the index is also considered a notebook version upgrade
now. So this should protect again older versions of keepnote from
opening upgraded notebooks.

Try it out and let me know what you think.

Also sorry for the lack of replies. This week has been busy for me
and I am now catching up on my KeepNote mail.

Matt

[rasmus]

Hi everyone. KeepNote 0.7 is now released! Thanks again to everyone
for your helpful feedback and testing.

Matt

On Feb 21, 11:58 pm, Matt Rasmussen <ras...@mit.edu> wrote:
> Hi everyone,
>
> Thanks so much for helping me test KeepNote 0.7.  I have fixed two main things
>
> 1. preferences likes "vertical/horizontal list view layout" should now
> be changeable.  I was not correctly saving these preferences before
> changing them.
> 2. Upgrading the index is also considered a notebook version upgrade
> now.  So this should protect again older versions of keepnote from
> opening upgraded notebooks.
>
> Try it out and let me know what you think.
>
> Also sorry for the lack of replies.  This week has been busy for me
> and I am now catching up on my KeepNote mail.
>
> Matt
>
> On Thu, Feb 10, 2011 at 10:14 AM, Matt Rasmussen <ras...@mit.edu> wrote:
> > Hi Tom,
>
> > Thanks for the feedback, it really helps.
>

[MarkS.]

Hello,

I noticed something when I brought an old notebook into 0.7, but not
having a photo-graphic memory it took me awhile to identify the
pattern.

At least under windows xp, JPG images disappear. That is, you can
paste them into a page, and they look fine. But when you go back to
the page, your image is gone. So also are the jpeg images that were
part of your original notebook before the update. PNG images however
appear to hang around.

When I look at the directory that supports a KN page, there are no jpg
files -- only png. The jpg images appear to have been converted to
png. However, the converted images do not view in the page node. Only
the images that were originally png's view properly. The images *do*
view if the page is viewed in a web browser. So I'm guessing that
there is something slightly broken in the jpg-to-png conversion. Web
browsers can still show the images, but they're non-renderable on the
KN page.

Has KN always done a jpg-to-png conversion, or is that new?

I'm sorry I didn't participate in the earlier testing. Perhaps next go-
around. ;-)

Mark

[MarkS.]

Further note --

If I take the images that *used* to be jpg, but were converted to png
by KN, and open them in PSP, and then resave them with the same name,
they now appear in KN normally. PSP appears to fix them. So almost
certainly there is some subtle thing wrong with the jpg/png conversion
routine.

Mark

[Matt Rasmussen]

Hi all,

Although I tested KeepNote 0.7 for several weeks, one very annoying
bug slipped through (issue 411: can't change helper applications). It
was a simple fix and I have a solution implemented in testing version
KeepNote 0.7.1. I will probably officially release this version soon.

Thanks Mark for finding the jpg/png issue. I will investigate that as
well. I expect the next couple of releases of keepnote to be small
bug fix polishing releases. I have been refactoring a lot of code
under the hood to make way for new features. That has brought a lot
of speed improvements, but it also means I need to retest everything
again. Thanks for your patience.

test release: http://keepnote.org/keepnote/download-test/
issue 411: http://code.google.com/p/keepnote/issues/detail?id=411

Matt

[Matt Rasmussen]

This sounds like the image is really JPG but are rename to *.png. If
you open a PNG file in a text editor the first few bytes should say
"PNG".

Matt

On Sun, Feb 27, 2011 at 12:58 AM, MarkS. <thro...@yahoo.com> wrote:

[Matt Rasmussen]

I have add this to my issue tracker:
http://code.google.com/p/keepnote/issues/detail?id=415

I think I figured out the problem. You can try my fix in testing version 0.7.1:
http://keepnote.org/keepnote/download-test/

I believe this bug was first introduced in keepnote 0.7.

Matt

[MarkS.]

After the first non-asci byte, it says PNG.

I just realized that almost all the images I started with were jpeg,
but not all had the problem. Here's one:

http://www.sciencenews.org/pictures/banners/mar2010/skyscraper/spark_2_160x600.jpg

I've been working out of the browser Opera. When I switched to FF and
tried the copy/paste, it worked with the images it had failed on in
Opera. So evidently Opera is doing something to the images enroute
that makes the JPG/PNG conversion break. Its weird, because Opera
obviously shouldn't be doing anything to the images.

So I guess its not a high-level issue -- more of an annoyance to Opera
fan's like myself, but probably not something you should spend any
time on.

Thanks,
Mark

[MarkS.]

But ... the png images from Opera that were in there viewed fine
before 0.68 or 0.70 (there was only a few hours between the two so I
don't know which one changed things). So *something* must have changed
in KN. Maybe I'll have to unpack the backups to check ...

Mark

On Feb 27, 12:22 pm, "MarkS." <throa...@yahoo.com> wrote:
> After the first non-asci byte, it says PNG.
>
> I just realized that almost all the images I started with were jpeg,
> but not all had the problem. Here's one:
>

>  http://www.sciencenews.org/pictures/banners/mar2010/skyscraper/spark_...

[MarkS.]

Looks like the change occurred between 0.68 and 0.70. Images visible
in 0.68 became invisible in 0.70.

Mark