Issue 36 in xappy: Allow applying multiple caches to an index

[xa...@googlecode.com]

Status: New
Owner: ----

New issue 36 by brunovia...@gmail.com: Allow applying multiple caches to an
index
http://code.google.com/p/xappy/issues/detail?id=36

Right now xappy allows only one cache to be applied, I think we should
allow applying multiple caches to an index. This will allow one to test
different caches strategies.

Attached there is a test and some changes that will allow this. The content
of the file is a sketch that doesn't touch xappy code, the proper support
for this must be done in xappy. I'm sending this first file to get a
feedback if the strategy is ok or not.

Attachments:
test_multi_cache.py 7.7 KB

[xa...@googlecode.com]

Comment #1 on issue 36 by brunovia...@gmail.com: Allow applying multiple

caches to an index
http://code.google.com/p/xappy/issues/detail?id=36

this is a patch against xappy, no unit test failed because of the changes.
I had some failures, but were in tests not related to the changes. I still
must add some tests for removing and incremental updating documents, but I
think this can be reviewed.

Attachments:
multicache.diff 11.6 KB

[xa...@googlecode.com]

Comment #2 on issue 36 by brunovia...@gmail.com: Allow applying multiple

caches to an index
http://code.google.com/p/xappy/issues/detail?id=36

new version of the patch that deal with incremental updates and deletions.

Attachments:
multicache.diff 14.8 KB

[xa...@googlecode.com]

Comment #3 on issue 36 by brunovia...@gmail.com: Allow applying multiple

Attachments:
multicache.diff 14.7 KB

[xa...@googlecode.com]

Comment #4 on issue 36 by brunovia...@gmail.com: Allow applying multiple

caches to an index
http://code.google.com/p/xappy/issues/detail?id=36

a new patch with a test that checks the two code paths related to cache:
when the cache has enough results for a query and when the cache has not
enough results. The first case was not being tested in old patches. For
deletion, I've decided to take a sub-optimal approach: when we have
multiple caches applied, the user must explicity remove the cached items
for each cache. This is simple enough for now, I think.

Attachments:
multicache.diff 16.4 KB

[xa...@googlecode.com]

Updates:
Status: Fixed

Comment #5 on issue 36 by boulton.rj: Allow applying multiple caches to an
index
http://code.google.com/p/xappy/issues/detail?id=36

I've applied this patch with the following modifications:

- I'd much rather use json to store the cache_id -> slot_number_offset
data, instead of cPickle; it's more language independent, easier to debug,
and generally nicer. For this amount of data, I very much doubt there are
any significant performance differences either.

- I've added some doccomments to the new functions in
xappy/cachemanager/xapian_manager.py Generally, all functions should have
a comment explaining their purpose, what type of values should be supplied
as their parameters, and what they return. (This isn't currently true of
all code in xappy, but it's something to aim for.)

- I've changed a bit of whitespace in xappy/indexerconnection.py, and
changed "ATTENTION" to "FIXME", and added a bit more explanation there.

- I've added a (MIT) license header to the test file. I like all files to
have a license header - please confirm that this license header is okay, or
provide me with an updated header (I'll remove the file again if not).

- I've added a copyright header naming you to the files you changed.

[xa...@googlecode.com]

Comment #6 on issue 36 by brunovia...@gmail.com: Allow applying multiple

caches to an index
http://code.google.com/p/xappy/issues/detail?id=36

I'm ok with the license header and with the other changes, thanks!