Elasticsearch error: org.elasticsearch.index.query.QueryShardException: failed to create query

Cindy

unread,

Jul 3, 2019, 3:51:46 PM7/3/19

to AtoM Users

Hi,

We're experiencing the following issue:

The index seems still working, but the elasticsearch is producing big logs in /var/log/elasticsearch/elasticsearch.log. Following is the part of errors, I've also attached the error file.

        at org.elasticsearch.index.query.QueryShardContext.toQuery(QueryShardContext.java:329) ~[elasticsearch-5.6.16.jar:5.6.16]
        at org.elasticsearch.index.query.QueryShardContext.toQuery(QueryShardContext.java:312) ~[elasticsearch-5.6.16.jar:5.6.16]
        at org.elasticsearch.search.SearchService.parseSource(SearchService.java:617) ~[elasticsearch-5.6.16.jar:5.6.16]
        at org.elasticsearch.search.SearchService.createContext(SearchService.java:485) ~[elasticsearch-5.6.16.jar:5.6.16]
        at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:461) ~[elasticsearch-5.6.16.jar:5.6.16]
        at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:257) ~[elasticsearch-5.6.16.jar:5.6.16]
        at org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:343) ~[elasticsearch-5.6.16.jar:5.6.16]
        at org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:340) ~[elasticsearch-5.6.16.jar:5.6.16]
        at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) ~[elasticsearch-5.6.16.jar:5.6.16]
        at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:662) [elasticsearch-5.6.16.jar:5.6.16]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:675) [elasticsearch-5.6.16.jar:5.6.16]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.6.16.jar:5.6.16]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_191]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_191]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
Caused by: java.lang.NumberFormatException: For input string: "18934,31976"
        at sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:2043) ~[?:?]
        at sun.misc.FloatingDecimal.parseDouble(FloatingDecimal.java:110) ~[?:?]
        at java.lang.Double.parseDouble(Double.java:538) ~[?:1.8.0_191]
        at org.elasticsearch.index.mapper.NumberFieldMapper$NumberType.hasDecimalPart(NumberFieldMapper.java:890) ~[elasticsearch-5.6.16.jar:5.6.16]
        at org.elasticsearch.index.mapper.NumberFieldMapper$NumberType$6.termQuery(NumberFieldMapper.java:605) ~[elasticsearch-5.6.16.jar:5.6.16]
        at org.elasticsearch.index.mapper.NumberFieldMapper$NumberFieldType.termQuery(NumberFieldMapper.java:958) ~[elasticsearch-5.6.16.jar:5.6.16]
        at org.elasticsearch.index.query.TermQueryBuilder.doToQuery(TermQueryBuilder.java:143) ~[elasticsearch-5.6.16.jar:5.6.16]
        at org.elasticsearch.index.query.AbstractQueryBuilder.toQuery(AbstractQueryBuilder.java:96) ~[elasticsearch-5.6.16.jar:5.6.16]
        at org.elasticsearch.index.query.BoolQueryBuilder.addBooleanClauses(BoolQueryBuilder.java:444) ~[elasticsearch-5.6.16.jar:5.6.16]
        at org.elasticsearch.index.query.BoolQueryBuilder.doToQuery(BoolQueryBuilder.java:418) ~[elasticsearch-5.6.16.jar:5.6.16]
        at org.elasticsearch.index.query.AbstractQueryBuilder.toQuery(AbstractQueryBuilder.java:96) ~[elasticsearch-5.6.16.jar:5.6.16]
        at org.elasticsearch.index.query.QueryShardContext.lambda$toQuery$1(QueryShardContext.java:313) ~[elasticsearch-5.6.16.jar:5.6.16]
        at org.elasticsearch.index.query.QueryShardContext.toQuery(QueryShardContext.java:325) ~[elasticsearch-5.6.16.jar:5.6.16]

We have AtoM 2.5 running on Oracle 7.6 with elasticsearch-5.6.16, Java-1.8.0_191, php-7.2.

The elastic search status is "yellow" and we have repopulated the search index, but still getting the errors.

Any help is appreciated.

Thank you.

elasticsearch_error.txt

Mike Cantelon

unread,

Jul 10, 2019, 4:17:52 PM7/10/19

to AtoM Users

Hi Cindy,

Could you try entering the following commands on your AtoM server from the AtoM root directory, assuming your ElasticSearch index for AtoM is called "atom"?

$ curl -X DELETE "localhost:9200/atom"

$ curl localhost:9200/atom # to confirm delete... result should include text confirming index no longer exists

$ ./symfony search:populate # to rebuild your index

Cheers,

Mike Cantelon,

Artefactual Systems

Cindy

unread,

Jul 10, 2019, 5:35:51 PM7/10/19

to AtoM Users

Hi Mike,

My Elasticsearch index for AtoM is called 'atom'

Elasticsearch server information:

- Host: 127.0.0.1

- port: 9200

- Index name: atom

Do I need backup database before run curl -X DELETE "localhost:9200/atom"?

Thanks,

Dan Gillean

unread,

Jul 10, 2019, 5:51:32 PM7/10/19

to ICA-AtoM Users

Hi Cindy,

While making a database backup is never a bad idea, this command won't touch the database - it will manually delete the search index. I believe Mike is hoping that repopulating it from scratch will resolve the issue you've encountered.

Additionally i should add about the yellow health status: I think it's primarily because we're only using one node in our default AtoM Elasticsearch deployment, and ES is warning you that this could involve data loss if the node goes down. According to the docs, "yellow means that the primary shard is allocated but replicas are not." Since ES is not our primary data store and it's easy to repopulate, I don't think this is critical. See:

Cheers,

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

--
You received this message because you are subscribed to the Google Groups "AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/6a6931ff-624b-4e0c-90e5-6f9c54d2dc0d%40googlegroups.com.

Cindy

unread,

Jul 11, 2019, 5:28:19 PM7/11/19

to ica-ato...@googlegroups.com

Ah, It seems the elasticsearch log is generated when a web crawler( Go-http-client/1.1) is crawling on the website.

/var/log/nginx/access.log:

64.62.202.73 - - [11/Jul/2019:17:11:06 -0400] "GET /index.php/informationobject/browse?names=58728,119356,119369&sort=startDate&topLod=0&view=card HTTP/1.1" 200 6683 "-" "Mozilla/5.0 (compatible; Go-http-client/1.1; +centu...@gmail.com)" "64.62.202.73"

/var/log/elasticsearch/elasticsearch.log:

at org.elasticsearch.index.query.QueryShardContext.toQuery(QueryShardContext.java:329) ~[elasticsearch-5.6.16.jar:5.6.16]

at org.elasticsearch.index.query.QueryShardContext.toQuery(QueryShardContext.java:312) ~[elasticsearch-5.6.16.jar:5.6.16]

at org.elasticsearch.search.SearchService.parseSource(SearchService.java:617) ~[elasticsearch-5.6.16.jar:5.6.16]

at org.elasticsearch.search.SearchService.createContext(SearchService.java:485) ~[elasticsearch-5.6.16.jar:5.6.16]

at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:461) ~[elasticsearch-5.6.16.jar:5.6.16]

at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:257) ~[elasticsearch-5.6.16.jar:5.6.16]

at org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:343) ~[elasticsearch-5.6.16.jar:5.6.16]

at org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:340) ~[elasticsearch-5.6.16.jar:5.6.16]

at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) ~[elasticsearch-5.6.16.jar:5.6.16]

at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:662) [elasticsearch-5.6.16.jar:5.6.16]

at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:675) [elasticsearch-5.6.16.jar:5.6.16]

at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.6.16.jar:5.6.16]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_191]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_191]

at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]

Caused by: java.lang.NumberFormatException: For input string: "58728,119356,119369"

at sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:2043) ~[?:?]

at sun.misc.FloatingDecimal.parseDouble(FloatingDecimal.java:110) ~[?:?]

at java.lang.Double.parseDouble(Double.java:538) ~[?:1.8.0_191]

........

Elastic has announced blocking the Go-http-client/1.1 in Feb 2019( from their web site), see https://discuss.elastic.co/t/blocking-of-go-http-client-1-1-crawler/165948

I've added blocking Go-http-client/1.1 in robots.txt,hopefully it can fix the issue.

Thanks,

To unsubscribe from this group and stop receiving emails from it, send an email to ica-ato...@googlegroups.com.

Dan Gillean

unread,

Jul 11, 2019, 5:58:03 PM7/11/19

to ICA-AtoM Users

Wow, interesting! Thank you for sharing your findings, Cindy. I've not seen a bot trigger that sort of error in AtoM before - hopefully this will help someone else!

As an aside, we are looking into whether it might be possible for us to upgrade AtoM to use ES 7.2 (or later) in the AtoM 2.6 release. I can't promise this will happen right now - Elastic is a "move fast and break things" kind of company, and there are many more breaking changes in ES6 and ES7 that will require significant analysis and code changes in AtoM. However, if we can, we will! It's nice to learn of yet another reason this would be beneficial to the project and our community.

Cheers,

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

Elastic has announced blocking the Go-http-client/1.1 in Feb 2019( for v7.0?), see https://discuss.elastic.co/t/blocking-of-go-http-client-1-1-crawler/165948

To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/0ae86d2c-b864-4729-98e6-116022fbda23%40googlegroups.com.

Orestes Sanchez

unread,

Jul 16, 2019, 9:42:25 AM7/16/19

to AtoM Users

Thank you for sharing the plans!

I am interested on contributing to code development on the upgrade to ElasticSearch 7. I started to play with current 2.6 branch and planning to give a try to upgrade Elastica PHP library to the latest available version. I am not familiar with the code yet, but I expect most of the code will be on Elastica side, if it is not this way, I am dead ;(

I will appreciate any advice on how to join the development team.

Regards.

Orestes.

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/0ae86d2c-b864-4729-98e6-116022fbda23%40googlegroups.com.

José Raddaoui

unread,

Jul 17, 2019, 5:23:10 AM7/17/19

to AtoM Users

Hi Orestes,

I'd be really happy to help you. But this upgrade will require several changes alongside upgrading the Elastica lib. The biggest challenge to upgrade from 5.x to a higher Elasticsearch version is that we need to split our single index with multiple types to multiple indexes with a single type each.

You may find useful information in the pull request created to upgrade from Elasticsearch 1.x to 5.x, but this upgrade will require quite different changes. Don't hesitate to ask me at my personal email if you need more information or help (jraddaoui at artefactual dot com).

Best regards.

Reply all

Reply to author

Forward