There are two threads in my program.
one creates dataEvent and consumes it by calling
zoieSystem.consume(envet), the data is from a file, the logic is like
this :
while( (text= filereader.readLine() )!=null ){
.... create data event with text....
zoieSystem.consume(dataEvent);
zoieSystem.syncWithVersion(10000, version);
version++;
}
the other thread searches the index by calling
zoieSystem.getIndexReaders();
The problem is that there are many small segments in the index
directory ,it seems like that the segments are never merged.
The param mergeFactor is 6 in ZoieMergePolicy .
For realtime indexing scenarios, we want to avoid large segment merges. So instead of optimizing for 1 large segment, zoie optimizes for N balanced segments and thus spread the merge cost over time.
The configuration parameters are number of Large segments and number of Smal segments. You should only be tuning those parameters. I would suggest you leave mergeFactor alone. For us, our production system has been running rather smoothly without changing the default parameters.
On Wed, Sep 28, 2011 at 7:47 PM, Bob <hellstone2...@gmail.com> wrote: > There are two threads in my program. > one creates dataEvent and consumes it by calling > zoieSystem.consume(envet), the data is from a file, the logic is like > this :
> the other thread searches the index by calling > zoieSystem.getIndexReaders();
> The problem is that there are many small segments in the index > directory ,it seems like that the segments are never merged. > The param mergeFactor is 6 in ZoieMergePolicy .
> -- > You received this message because you are subscribed to the Google Groups > "zoie" group. > To post to this group, send email to zoie@googlegroups.com. > To unsubscribe from this group, send email to > zoie+unsubscribe@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/zoie?hl=en.
I think the reason is that the writer.addIndexesNoOptimize()
function is changed in lucene3.3;
In Lucene 3.0 , addIndexesNoOptimize() maybe cause merge. In
Lucene3.3, it does not.
Maybe this is a bug .
On Sep 29, 1:41 pm, John Wang <john.w...@gmail.com> wrote:
> For realtime indexing scenarios, we want to avoid large segment merges.
> So instead of optimizing for 1 large segment, zoie optimizes for N balanced
> segments and thus spread the merge cost over time.
> The configuration parameters are number of Large segments and number of
> Smal segments. You should only be tuning those parameters. I would suggest
> you leave mergeFactor alone. For us, our production system has been running
> rather smoothly without changing the default parameters.
> -John
> On Wed, Sep 28, 2011 at 7:47 PM, Bob <hellstone2...@gmail.com> wrote:
> > There are two threads in my program.
> > one creates dataEvent and consumes it by calling
> > zoieSystem.consume(envet), the data is from a file, the logic is like
> > this :
> > the other thread searches the index by calling
> > zoieSystem.getIndexReaders();
> > The problem is that there are many small segments in the index
> > directory ,it seems like that the segments are never merged.
> > The param mergeFactor is 6 in ZoieMergePolicy .
> > --
> > You received this message because you are subscribed to the Google Groups
> > "zoie" group.
> > To post to this group, send email to zoie@googlegroups.com.
> > To unsubscribe from this group, send email to
> > zoie+unsubscribe@googlegroups.com.
> > For more options, visit this group at
> >http://groups.google.com/group/zoie?hl=en.
> I think the reason is that the writer.addIndexesNoOptimize()
> function is changed in lucene3.3;
> In Lucene 3.0 , addIndexesNoOptimize() maybe cause merge. In
> Lucene3.3, it does not.
> Maybe this is a bug .
> On Sep 29, 1:41 pm, John Wang <john.w...@gmail.com> wrote:
> > Hi Bob:
> > For realtime indexing scenarios, we want to avoid large segment merges.
> > So instead of optimizing for 1 large segment, zoie optimizes for N balanced
> > segments and thus spread the merge cost over time.
> > The configuration parameters are number of Large segments and number of
> > Smal segments. You should only be tuning those parameters. I would suggest
> > you leave mergeFactor alone. For us, our production system has been running
> > rather smoothly without changing the default parameters.
> > -John
> > On Wed, Sep 28, 2011 at 7:47 PM, Bob <hellstone2...@gmail.com> wrote:
> > > There are two threads in my program.
> > > one creates dataEvent and consumes it by calling
> > > zoieSystem.consume(envet), the data is from a file, the logic is like
> > > this :
> > > the other thread searches the index by calling
> > > zoieSystem.getIndexReaders();
> > > The problem is that there are many small segments in the index
> > > directory ,it seems like that the segments are never merged.
> > > The param mergeFactor is 6 in ZoieMergePolicy .
> > > --
> > > You received this message because you are subscribed to the Google Groups
> > > "zoie" group.
> > > To post to this group, send email to zoie@googlegroups.com.
> > > To unsubscribe from this group, send email to
> > > zoie+unsubscribe@googlegroups.com.
> > > For more options, visit this group at
> > >http://groups.google.com/group/zoie?hl=en.
On Thu, Sep 29, 2011 at 12:27 AM, Bob <hellstone2...@gmail.com> wrote: > I think the reason is that the writer.addIndexesNoOptimize() > function is changed in lucene3.3; > In Lucene 3.0 , addIndexesNoOptimize() maybe cause merge. In > Lucene3.3, it does not. > Maybe this is a bug .
> On Sep 29, 1:41 pm, John Wang <john.w...@gmail.com> wrote: > > Hi Bob:
> > For realtime indexing scenarios, we want to avoid large segment > merges. > > So instead of optimizing for 1 large segment, zoie optimizes for N > balanced > > segments and thus spread the merge cost over time.
> > The configuration parameters are number of Large segments and number > of > > Smal segments. You should only be tuning those parameters. I would > suggest > > you leave mergeFactor alone. For us, our production system has been > running > > rather smoothly without changing the default parameters.
> > -John
> > On Wed, Sep 28, 2011 at 7:47 PM, Bob <hellstone2...@gmail.com> wrote: > > > There are two threads in my program. > > > one creates dataEvent and consumes it by calling > > > zoieSystem.consume(envet), the data is from a file, the logic is like > > > this :
> > > the other thread searches the index by calling > > > zoieSystem.getIndexReaders();
> > > The problem is that there are many small segments in the index > > > directory ,it seems like that the segments are never merged. > > > The param mergeFactor is 6 in ZoieMergePolicy .
> > > -- > > > You received this message because you are subscribed to the Google > Groups > > > "zoie" group. > > > To post to this group, send email to zoie@googlegroups.com. > > > To unsubscribe from this group, send email to > > > zoie+unsubscribe@googlegroups.com. > > > For more options, visit this group at > > >http://groups.google.com/group/zoie?hl=en.
> -- > You received this message because you are subscribed to the Google Groups > "zoie" group. > To post to this group, send email to zoie@googlegroups.com. > To unsubscribe from this group, send email to > zoie+unsubscribe@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/zoie?hl=en.
> On Thu, Sep 29, 2011 at 12:27 AM, Bob <hellstone2...@gmail.com> wrote:
>> I think the reason is that the writer.addIndexesNoOptimize() >> function is changed in lucene3.3; >> In Lucene 3.0 , addIndexesNoOptimize() maybe cause merge. In >> Lucene3.3, it does not. >> Maybe this is a bug .
>> On Sep 29, 1:41 pm, John Wang <john.w...@gmail.com> wrote: >> > Hi Bob:
>> > For realtime indexing scenarios, we want to avoid large segment >> merges. >> > So instead of optimizing for 1 large segment, zoie optimizes for N >> balanced >> > segments and thus spread the merge cost over time.
>> > The configuration parameters are number of Large segments and >> number of >> > Smal segments. You should only be tuning those parameters. I would >> suggest >> > you leave mergeFactor alone. For us, our production system has been >> running >> > rather smoothly without changing the default parameters.
>> > -John
>> > On Wed, Sep 28, 2011 at 7:47 PM, Bob <hellstone2...@gmail.com> wrote: >> > > There are two threads in my program. >> > > one creates dataEvent and consumes it by calling >> > > zoieSystem.consume(envet), the data is from a file, the logic is like >> > > this :
>> > > the other thread searches the index by calling >> > > zoieSystem.getIndexReaders();
>> > > The problem is that there are many small segments in the index >> > > directory ,it seems like that the segments are never merged. >> > > The param mergeFactor is 6 in ZoieMergePolicy .
>> > > -- >> > > You received this message because you are subscribed to the Google >> Groups >> > > "zoie" group. >> > > To post to this group, send email to zoie@googlegroups.com. >> > > To unsubscribe from this group, send email to >> > > zoie+unsubscribe@googlegroups.com. >> > > For more options, visit this group at >> > >http://groups.google.com/group/zoie?hl=en.
>> -- >> You received this message because you are subscribed to the Google Groups >> "zoie" group. >> To post to this group, send email to zoie@googlegroups.com. >> To unsubscribe from this group, send email to >> zoie+unsubscribe@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/zoie?hl=en.
