Query speed

[Nguyen Bach]

Hi gensim's users,

I am experimenting gensim with my corpora. One thing that I am
experiencing right now is the latency of queries.
I have measured query speed which is the time when I send a query and
receive the most closest document ID.
Here are some results with the TfIdf model.
6.2M documents (~62M words): 8 seconds/query
10M documents (~200M words): 20 seconds/query

I think it is very slow. Let say if I want to 100,000 queries, then it
will take me 222 hours to finish it.
An attempt that I have tried is to make a multi-threaded server, for
example 5 threads. Now 222 hours reduce to 44.4 hours, however it is
still very slow in my situation.

I am wondering if someone can share his/her experience on speeding up
the query time.

Thanks,
Nguyen

[William]

Hi Nguyen,
With some additional information, it might be possible to find ways to
speed up your queries. Can you clarify whether ~62M words and ~200M
words refers to unique tokens in your dictionary, or to the total
number of words (including repeats) in all documents in your entire
corpora?

What are the specs for the machine you are using? You can benchmark
your machine by running the command 'python -m gensim.test.simspeed'
and following the directions. That may give you an indication of
whether your machine is a bottleneck, or your code/use of gensim.

Have you installed accelerated BLAS libraries on your system? They can
bring speed benefits.

Are you using the batch query document interface for queries, as
described on this page? http://radimrehurek.com/gensim/similarities/docsim.html
Those are more efficient than one query at a time.

For comparison, I have a ~2M document corpus with 100K unique tokens,
and queries take well under 0.1 second to return on my I7 laptop.

-William

[Radim Řehůřek]

Like William says, try running the simspeed.py script and paste the
results here.

There are several ways to speed things up with similarity queries in
gensim:

1. submit more queries at once (call `my_index[many_queries]` instead
of `my_index[query]`)

2. install optimized BLAS libraries, see http://radimrehurek.com/gensim/distributed.html
(although this doesn't help as much with sparse queries, like in
tfidf)

3. use dimensionality reduction -- some latent document
representation, such as LSA, LDA -- instead of plain tfidf
representation.

I have also forked the FLANN repo on github, which promises better
asymptotic complexity for queries. But it doesn't support cosine
similarity, so plugging it in will be non-trivial.

And finally, if you're using a higher-level interface to querying,
such as the similarity server, I have made some improvements there
recently which bring it closer to the raw MatrixSimilarity performance
(less overhead). I plan to release a new version of gensim+simserver
this weekend.

Best,
Radim

On May 18, 3:27 pm, William <william.b...@gmail.com> wrote:
> Hi Nguyen,
> With some additional information, it might be possible to find ways to
> speed up your queries. Can you clarify whether ~62M words and ~200M
> words refers to unique tokens in your dictionary, or to the total
> number of words (including repeats) in all documents in your entire
> corpora?
>
> What are the specs for the machine you are using? You can benchmark
> your machine by running the command 'python -m gensim.test.simspeed'
> and following the directions. That may give you an indication of
> whether your machine is a bottleneck, or your code/use of gensim.
>
> Have you installed accelerated BLAS libraries on your system? They can
> bring speed benefits.
>
> Are you using the batch query document interface for queries, as

> described on this page?http://radimrehurek.com/gensim/similarities/docsim.html