SolrException

samuel pouchin

unread,

Jan 5, 2016, 1:26:22 PM1/5/16

to ddf-users

Hi,

I 'm actually trying to ingest metacards using DDF 2.8.1 and I have the following error :

at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:168)

at org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:69)

at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:51)

...

Caused by: java.lang.IllegalArgumentException: Document contains at least one immense term in field="metadata_txt" (whose UTF8 encoding is longer than the max length 32766), all of which were skipped. Please correct the analyzer to not produce such terms. The prefix of the first immense term is: '[60, 117, 114, 110, 58, 82, 101, 115, 117, 108, 116, 115, 32, 115, 99, 104, 101, 109, 97, 86, 101, 114, 115, 105, 111, 110, 61, 34, 48, 46]...', original message: bytes can be at most 32766 in length; got 44566

at org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:687)

at org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:359)

at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:318)

at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:241)

at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:465)

at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1526)

at org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:240)

at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:164)

... 62 more

Caused by: org.apache.lucene.util.BytesRefHash$MaxBytesLengthExceededException: bytes can be at most 32766 in length; got 44566

at org.apache.lucene.util.BytesRefHash.add(BytesRefHash.java:284)

at org.apache.lucene.index.TermsHashPerField.add(TermsHashPerField.java:151)

at org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:663)

... 69 more

This error doesn't occur with DDF 2.6.1

I have tried many workarounds found on the web but none of them solve this issue.

Do you have something specific to suggest that would solve this issue ?

Thanks in advance

Samuel

Keith Wire

unread,

Jan 5, 2016, 1:32:08 PM1/5/16

to samuel pouchin, ddf-users

Can you attach the metacard you attempting to ingest?

—Keith

--
You received this message because you are subscribed to the Google Groups "ddf-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ddf-users+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

samuel pouchin

unread,

Jan 5, 2016, 2:03:51 PM1/5/16

to ddf-users

Hi,

The error occurs on the attribute descriptor Metacard.METADATA, parameter AttributeType<?> type was set to STRING_TYPE
If using STRING_TYPE, you're metadata lenght must be at most 32766 bytes (UTF 8 encoding restriction). Many similar issues can be find on the web.

I have changed STRING_TYPE to XML_TYPE and now the metacards are well ingested into ddf 2.8.1

Hope this can help in case someone meets the same issue.

Reply all

Reply to author

Forward