Exception occurred while trying to associate many-many entities using ManyAssociation (ElasticSearchIndexException)
Jaydatt Desai
I was doing some testing with ManyAssociation, where I was adding many(~200) number of entities to this ManyAssociation property. For Index/Query here is ElasticSearch.
So, when I was executing this test case it failed and throws an "ElasticSearchIndexException". This is only occurred when trying to associate entities up to some limit (works perfectly with ~100 entities in this case, but throws exception when we try to associate ~200), and this limit vary upon entity size.
Details are below:
Exception:
org.qi4j.index.elasticsearch.ElasticSearchIndexException: failure in bulk execution:
[178]: index [qi4j_index], type [qi4j_entities], id [869f4cf5-b258-4aa4-9536-c11b336862c5-0], message [IllegalArgumentException[Document contains at least one immense term in field="_all" (whose UTF8 encoding is longer than the max length 32766), all of which were skipped. Please correct the analyzer to not produce such terms. The prefix of the first immense term is: '[38 36 39 66 34 63 66 35 2d 62 32 35 38 2d 34 61 61 34 2d 39 35 33 36 2d 63 31 31 62 33 33]...']]
at org.qi4j.index.elasticsearch.ElasticSearchIndexer$Mixin.notifyChanges(ElasticSearchIndexer.java:150)
at org.qi4j.spi.entitystore.StateChangeNotificationConcern$1.commit(StateChangeNotificationConcern.java:44)
at org.qi4j.spi.entitystore.ConcurrentModificationCheckConcern$ConcurrentCheckingEntityStoreUnitOfWork$1.commit(ConcurrentModificationCheckConcern.java:116)
Code:
@Test
public void testManyAssociation() throws Exception{
UnitOfWork uow = module.newUnitOfWork();
TestEntity testEntity = module.currentUnitOfWork().newEntity(TestEntity.class);
for(int i = 0 ; i<200; i++) {
TestEntity2 testEntity2 = module.currentUnitOfWork().newEntity(TestEntity2.class);
testEntity2.property().set("test");
testEntity.manyAssociation().add(testEntity2);
}
uow.complete();
}
public interface TestEntity
extends EntityComposite
{
@Optional
Property<String> property();
ManyAssociation<TestEntity2> manyAssociation();
}
public interface TestEntity2
extends EntityComposite
{
@Optional
Property<String> property();
@Optional
Property<List<Byte>> binaryProperty();
}
Paul Merlin
Looks like a bug in the ElasticSearch Query engine.
Filled it as https://ops4j1.jira.com/browse/QI-412
Thanks for the test-code.
Will try to fix this asap.
Jaydatt Desai a écrit :
Paul Merlin
Finally found some time to investigate this.
An indexed term in Lucene was capped to 32k.
All ManyAssociations of an entity are stored in a term.
If the UTF-8 JSON encoded term is > 32k, ElasticSearch simply cannot index it.
This has been fixed in Lucene 4.5 (https://issues.apache.org/jira/browse/LUCENE-4583) and ElasticSearch 1.3.x.
The dependency version is fixed in the Qi4j develop branch (worked for 10k associations) and will be included in 2.1. For the time being, simply override the ElasticSearch version used in your project with the latest one available (1.3.4).
Note that there are obviously some other limits going on, not every use-case has been tested. This one was fixed by a simple version upgrade, we'll see if others will need indexing shema changes. Thanks for the report !
Cheers
/Paul
Jaydatt Desai
Paul Merlin
Hi Paul,Thanks a lot for the fix and for updating me, I will take a look on your fix on development branch and I will also change elastic search to the latest one in my project.So, thanks again for the fix...
https://github.com/Qi4j/qi4j-sdk/commit/b8efcda1446afbf0dde033719d6956542bcec9d8
Bharvi Dixit
Niclas Hedhman
--
You received this message because you are subscribed to the Google Groups "qi4j-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to qi4j-dev+u...@googlegroups.com.
To post to this group, send email to qi4j...@googlegroups.com.
Visit this group at http://groups.google.com/group/qi4j-dev.
For more options, visit https://groups.google.com/d/optout.
--