AgileWiki search ranking solved!

[William la Forge]

For some time now I have been pondering how to search, keyword search being just one example. The problem is ordering the result set, which ideally should be in reverse chronological order.

Reverse chronological ordering is ideal for searching. First because in many cases, like link and inverse link lists, there is no other order with is reasonable. Second because reverse chronological ordering answers the question, what's new? Answering this question is especially important when there are multiple users. Indeed, the biggest problem with search engines like Google is that they are unable to order their result sets this way without devaluing the search. (They order based on frequency of hits, which in turn is based largely on their ordering, creating a ranking which is not fully meaningful.) Currently there is not a good web2.0 search, except for limited domains like twitter.

The key for me was the implementation of table iterator prefilter which I mentioned earlier. It got me thinking about links. I need to introduce a new type of link whose key/name is the timestamp of when it was created. This new type of link then can be used in relation usage, qualifier usage, and for subject->object and object->subject links. And this is all we need for ordering our search results.

Obvious, of course. At least in retrospect. :-)

[Bill]

My first solution for search result ordering has problems. Since we
are now using the uuid in the prefilter, i.e. doing joins on the uuid,
we can not switch to a timestamp. Now in the case of a keyword or
other qualifier, we can easily just use the journal of the keyword or
qualifier rolon in place of a search. But this is also the start of a
general solution as well.

Consider the following link. How do we order the list of links? The
answer is to display a subset of the intersection of the user's
journal and the journal of the following link. This gives us all the
addFollowing and removeFollowing journal entries made by the user.
This is actually more interesting than just a long list of all the
rolons which a user is following. And a good intersection search can
be fast.

Journaling is generally underrated.

Bill

[laforge49]

Finally some progress on this issue. The new following table displays
the list of rolons being followed in reverse chronological order based
on the date when following was elected. The process is fast and
scalable, but a bit indirect.

When a following link is created, the journal entry for this change is
added to the ark, to the user, to the rolon being followed and to the
following relation. In addition, the timestamp of the journal entry is
saved in the link connecting the user's rolon with the rolon being
followed.

The following table is produced by iterating through the intersection
of the user journal and the following relation rolon. (The
intersection search is very fast--it does NOT use a sequential scan!)
Any journal entries are checked to see if the rolon of any journals
containing that journal entry is being followed by the user--this
becomes a candidate for display. However the link to that rolon must
contain the timestamp of the journal entry, as otherwise duplicates
will be displayed.

That's how far I've progressed. Next I want to include in the table a
link to the journal entry. This way users can see how the results are
ordered, i.e. the timestamp of the journal entry, and by clicking on
the journal entry they can also see who created the link. (Though this
is a bit redundant for following links.)

This then will be a model for other tables.