I have a couple of questions about GenericTypeahead after looked at
the code of cleo.
(1) Looks like GenericTypeahead return all of matched elements. Is
this correct?
If it is true, what if a key(term) are matched with lots of
elements? For example, 1 million. It will comsumes lots of memory and
increase latence.
(2) In the indexDataStore, looks like elements are not sorted by
score. Also, the seach() function has a paramter timeoutMillis.
What if the function search() reutrn earlier because of
timeoutMillis, can GenericTypeahead guarntee return elemetns with
the highest score?
On Apr 9, 9:01 pm, yuan <huapengy...@gmail.com> wrote:
> Hi:
> I have a couple of questions about GenericTypeahead after looked at
> the code of cleo.
> (1) Looks like GenericTypeahead return all of matched elements. Is
> this correct?
> If it is true, what if a key(term) are matched with lots of
> elements? For example, 1 million. It will comsumes lots of memory and
> increase latence.
This really depends on the type of Collector you are using. There are
several different types Collector you can use to return over 1 million
elements or top 10 elements. Choose the collector that suits your
needs.
> (2) In the indexDataStore, looks like elements are not sorted by
> score. Also, the seach() function has a paramter timeoutMillis.
> What if the function search() reutrn earlier because of
> timeoutMillis, can GenericTypeahead guarntee return elemetns with
> the highest score?
For typeahead, as a user is typing more characters, the search scope
is narrowing down quickly. The timeoutMillis is for early termination
to guarantee response time but not full coverage. If your program
really need to find the elements with the highest score, you should
NOT consider using timeout.
We need return the highest score, we also expect low latency. But if
the inverted list is long, it may take long to process the entire
list.
I am not sure whether there are some ways to contorl the maximum lengh
of the inverted list. I see you mentioned cleo partition in the
document. Looks like the partition is based on elementID rather than
prefix?
If the partition is based on elelmentID, the maximum length of the
invered list may not be very long. But a single query will touch all
of partitions. In a distributed system, it means it will touch all of
nodes. I am understanding correctly?
Thanks.
Yuan
On Apr 10, 12:54 am, Jingwei <jingwei...@gmail.com> wrote:
> On Apr 9, 9:01 pm, yuan <huapengy...@gmail.com> wrote:
> > Hi:
> > I have a couple of questions about GenericTypeahead after looked at
> > the code of cleo.
> > (1) Looks like GenericTypeahead return all of matched elements. Is
> > this correct?
> > If it is true, what if a key(term) are matched with lots of
> > elements? For example, 1 million. It will comsumes lots of memory and
> > increase latence.
> This really depends on the type of Collector you are using. There are
> several different types Collector you can use to return over 1 million
> elements or top 10 elements. Choose the collector that suits your
> needs.
> > (2) In the indexDataStore, looks like elements are not sorted by
> > score. Also, the seach() function has a paramter timeoutMillis.
> > What if the function search() reutrn earlier because of
> > timeoutMillis, can GenericTypeahead guarntee return elemetns with
> > the highest score?
> For typeahead, as a user is typing more characters, the search scope
> is narrowing down quickly. The timeoutMillis is for early termination
> to guarantee response time but not full coverage. If your program
> really need to find the elements with the highest score, you should
> NOT consider using timeout.
Partitions will help since requests are evaluated in parallel.
Partition is based on elementID. You may want to experiment with your
data sets to see how many partitions you can hold on one box in order
to have acceptable performance.
Best
Jingwei
On Apr 10, 11:58 am, yuan <huapengy...@gmail.com> wrote:
> We need return the highest score, we also expect low latency. But if
> the inverted list is long, it may take long to process the entire
> list.
> I am not sure whether there are some ways to contorl the maximum lengh
> of the inverted list. I see you mentioned cleo partition in the
> document. Looks like the partition is based on elementID rather than
> prefix?
> If the partition is based on elelmentID, the maximum length of the
> invered list may not be very long. But a single query will touch all
> of partitions. In a distributed system, it means it will touch all of
> nodes. I am understanding correctly?
> Thanks.
> Yuan
> On Apr 10, 12:54 am, Jingwei <jingwei...@gmail.com> wrote:
> > Hi,
> > Please see my reply below.
> > On Apr 9, 9:01 pm, yuan <huapengy...@gmail.com> wrote:
> > > Hi:
> > > I have a couple of questions about GenericTypeahead after looked at
> > > the code of cleo.
> > > (1) Looks like GenericTypeahead return all of matched elements. Is
> > > this correct?
> > > If it is true, what if a key(term) are matched with lots of
> > > elements? For example, 1 million. It will comsumes lots of memory and
> > > increase latence.
> > This really depends on the type of Collector you are using. There are
> > several different types Collector you can use to return over 1 million
> > elements or top 10 elements. Choose the collector that suits your
> > needs.
> > > (2) In the indexDataStore, looks like elements are not sorted by
> > > score. Also, the seach() function has a paramter timeoutMillis.
> > > What if the function search() reutrn earlier because of
> > > timeoutMillis, can GenericTypeahead guarntee return elemetns with
> > > the highest score?
> > For typeahead, as a user is typing more characters, the search scope
> > is narrowing down quickly. The timeoutMillis is for early termination
> > to guarantee response time but not full coverage. If your program
> > really need to find the elements with the highest score, you should
> > NOT consider using timeout.
Hi Yuan. Recently,I'm also working with cleo,but I faced a lot of problems,especially dealing with the big data,If you have time,I would like to ask you some questions.My email address is Altynai...@gamil.com Thanks!
On Monday, September 10, 2012 6:30:28 PM UTC-7, Altynai wrote:
> Hi Yuan. > Recently,I'm also working with cleo,but I faced a lot of > problems,especially dealing with the big data,If you have time,I would like > to ask you some questions.My email address is Altyn...@gamil.com<javascript:> > Thanks!
