Dates in Hbase-Indexer and also in solR (TrieDateField)
102 views
Skip to first unread message
Salou Guillaume
Sep 21, 2015, 9:30:29 AM9/21/15
to HBase Indexer Users
Hi All,
I am new to HBase Indexer. I am working around Hbase-Indexer and Phoenix.
My main problem is about dates in Hbase-Indexer. I don't know how to push dateFormat through solR. I'm able to push date as a string format in solR, but It's not good.
I also try to create my own class like that :
private static final ByteArrayValueMapper DATE_PHOENIX_MAPPER = new AbstractByteValueMapper(TrieDateField.class) {
@Override
protected Object mapInternal(byte[] input) {
String ts = ""+PDataType.TIMESTAMP.toObject(input);
String out = ts.replaceAll("\\s","T").replaceAll("\\..*","Z");
return out.getBytes();
}
};
But It doens't work, I get this error in SolR, I think my mistake deals with TrieDateField.class but I think that's the only way to index date types in Solr.
] Error adding field 'modification_d'='2015-09-21T14:24:07Z' msg=For input string: "2015-09-21T14:24:07Z" at org.apache.solr.update.DocumentBuilder.toDocument(DocumentBuilder.java:178) at org.apache.solr.update.AddUpdateCommand.getLuceneDocument(AddUpdateCommand.java:78) at org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:238) at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:164) at org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:69) at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:51) at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:926) at org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1081) at org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:692) at org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:247) at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:174) at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:99) at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1984) at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:818) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:422) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:211) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.solr.servlet.SolrHadoopAuthenticationFilter$2.doFilter(SolrHadoopAuthenticationFilter.java:288) at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:592) at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationFilter.doFilter(DelegationTokenAuthenticationFilter.java:277) at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:555) at org.apache.solr.servlet.SolrHadoopAuthenticationFilter.doFilter(SolrHadoopAuthenticationFilter.java:293) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.solr.servlet.HostnameFilter.doFilter(HostnameFilter.java:86) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:861) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:620) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.NumberFormatException: For input string: "2015-09-21T14:24:07Z" at sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:1250) at java.lang.Double.parseDouble(Double.java:540) at org.apache.solr.schema.TrieField.createField(TrieField.java:618) at org.apache.solr.schema.TrieField.createFields(TrieField.java:663) at org.apache.solr.update.DocumentBuilder.addField(DocumentBuilder.java:50) at org.apache.solr.update.DocumentBuilder.toDocument(DocumentBuilder.java:125) ... 39 more
Gabriel Reid
Sep 21, 2015, 10:03:57 AM9/21/15
to Salou Guillaume, HBase Indexer Users
Based on that stack trace, it looks like the value is being mapped
correctly to a date string in hbase-indexer, but that the schema isn't
interpreting it as a date.
Could you post the Solr schema configuration that you're using related
to that field? Is there any chance you've got it configured as a
TrieDoubleField instead of a TrieDateField?
- Gabriel
> --
> You received this message because you are subscribed to the Google Groups
> "HBase Indexer Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to hbase-indexer-u...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
correctly to a date string in hbase-indexer, but that the schema isn't
interpreting it as a date.
Could you post the Solr schema configuration that you're using related
to that field? Is there any chance you've got it configured as a
TrieDoubleField instead of a TrieDateField?
- Gabriel
> You received this message because you are subscribed to the Google Groups
> "HBase Indexer Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to hbase-indexer-u...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
Salou Guillaume
Sep 21, 2015, 10:39:50 AM9/21/15
to HBase Indexer Users, g.s...@gmail.com
Hi Gabriel,
Here is the part of my schema dealing with that date :
<field name="modification_d" type="date_solr_phoenix" value="0:MODIFICATION"/>
Thanks
> email to hbase-indexer-user+unsub...@googlegroups.com.
Gabriel Reid
Sep 21, 2015, 10:52:46 AM9/21/15
to Salou Guillaume, HBase Indexer Users
That looks like it's from your indexer configuration, but I wanted to
see your Solr schema configuration. Could you post that?
>> > email to hbase-indexer-u...@googlegroups.com.
see your Solr schema configuration. Could you post that?
> --
> You received this message because you are subscribed to the Google Groups
> "HBase Indexer Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to hbase-indexer-u...@googlegroups.com.
> You received this message because you are subscribed to the Google Groups
> "HBase Indexer Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
Salou Guillaume
Sep 21, 2015, 11:06:16 AM9/21/15
to HBase Indexer Users, g.s...@gmail.com
The schema.xml is unchanged since the core/collection has been created, and it's the default example schema. I thought those files are unused.
I join the file.
>> > email to hbase-indexer-user+unsub...@googlegroups.com.
>> > For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups
> "HBase Indexer Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to hbase-indexer-user+unsub...@googlegroups.com.
Gabriel Reid
Sep 22, 2015, 2:53:58 AM9/22/15
to Salou Guillaume, HBase Indexer Users
The problem is that there is a mismatch between the type of data that
you're creating within hbase-indexer (a date stored in a string) and
the type of data specified for that field in the solr schema (double).
Because your field ends with "_d", it is being picked up by this field
definition in the solr schema:
<dynamicField name="*_d" type="double" indexed="true" stored="true"/>
That dynamic field definition means that all fields ending with "_d"
should be interpreted as a double value.
There is also a TrieDate-based dynamic field defined in the default Solr schema:
<dynamicField name="*_tdt" type="tdate" indexed="true" stored="true"/>
This means that if you change the field name in your indexer
configuration from "modification_d" to "modification_tdt", things will
(hopefully) work as you intended.
- Gabriel
>> >> > email to hbase-indexer-u...@googlegroups.com.
you're creating within hbase-indexer (a date stored in a string) and
the type of data specified for that field in the solr schema (double).
Because your field ends with "_d", it is being picked up by this field
definition in the solr schema:
<dynamicField name="*_d" type="double" indexed="true" stored="true"/>
That dynamic field definition means that all fields ending with "_d"
should be interpreted as a double value.
There is also a TrieDate-based dynamic field defined in the default Solr schema:
<dynamicField name="*_tdt" type="tdate" indexed="true" stored="true"/>
This means that if you change the field name in your indexer
configuration from "modification_d" to "modification_tdt", things will
(hopefully) work as you intended.
- Gabriel
>> >> > For more options, visit https://groups.google.com/d/optout.
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "HBase Indexer Users" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> > an
>> > email to hbase-indexer-u...@googlegroups.com.
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "HBase Indexer Users" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> > an
>> > For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups
> "HBase Indexer Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to hbase-indexer-u...@googlegroups.com.
>
> --
> You received this message because you are subscribed to the Google Groups
> "HBase Indexer Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
Salou Guillaume
Sep 22, 2015, 7:48:35 AM9/22/15
to HBase Indexer Users, g.s...@gmail.com
It works thanks Gabriel !! ;)
>> >> > email to hbase-indexer-user+unsub...@googlegroups.com.
>> >> > For more options, visit https://groups.google.com/d/optout.
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "HBase Indexer Users" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> > an
>> > email to hbase-indexer-user+unsub...@googlegroups.com.
>> > For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups
> "HBase Indexer Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to hbase-indexer-user+unsub...@googlegroups.com.
Murtaza Kanchwala
May 23, 2016, 3:08:03 AM5/23/16
to HBase Indexer Users
Just for Information :
If you need to update your input date in Solr core(as per the Solr DateField Standards), You can do:
{
convertTimestamp {
field : lastUpdatedDate #This is your Solr Field
inputFormats : ["yyyy-MM-dd'T'HH:mm:ss'Z'", "yyyy-MM-dd", "yyyy-MM-dd'T'HH:mm:ss.SSSZ"] #Your HBase Input formats
inputTimezone : America/Los_Angeles
outputFormat : "yyyy-MM-dd'T'HH:mm:ss.SSS'Z'" #Your Solr DateField Format
outputTimezone : UTC
Reply all
Reply to author
Forward
0 new messages