gsearch and indexing not updating
105 views
Skip to first unread message
Aaron Collier
Apr 14, 2016, 6:08:16 PM4/14/16
to islandora
Looks like there might be a few topics on this but I didn't see any resolution. I've run through the installation steps a few times now and specifically the GSearch and SOLR (https://wiki.duraspace.org/display/ISLANDORA/milestone+6+-+Installing+Solr+and+GSearch) and I'm always unable to get the indexing to work automatically.
I'm hoping there may be some general advice on something that may be missed in the instructions or something to check specifically that may indicate what's going on.
At this point, I've only been able to manually update the index using /fedoragsearch/rest, but often times that doesn't even work.
Thanks.
Aaron Collier
Apr 14, 2016, 7:40:20 PM4/14/16
to islandora
FYI - I seem to be able to get the solr index updated only if I click the "reload" button on the admin panel of /solr.
It isn't updating [automatically] through the gsearch interface at this point.
Nick Ruest
Apr 15, 2016, 3:08:52 AM4/15/16
to isla...@googlegroups.com
Hi Aaron,
What are your logs saying? catalina.out, fedoragsearch.daily.log
What are you using for your configurations? What's your GSearch and Solr
configuration look like?
It might also be helpful to spin up an Islandora Vagrant[1] machine and
compare paths, files, and configurations.
You can also checkout Islandora Vagrant Base Box, specifically, the
GSearch[2] and Solr[3] install scripts.
-nruest
[1] https://github.com/Islandora-Labs/islandora_vagrant
[2]
https://github.com/Islandora-Labs/islandora_vagrant_base_box/blob/master/scripts/gsearch.sh
[3]
https://github.com/Islandora-Labs/islandora_vagrant_base_box/blob/master/scripts/solr.sh
On 2016-04-14 07:40 PM, Aaron Collier wrote:
> FYI - I seem to be able to get the solr index updated only if I click
> the "reload" button on the admin panel of /solr.
>
> It isn't updating [automatically] through the gsearch interface at this
> point.
>
> On Thursday, April 14, 2016 at 3:08:16 PM UTC-7, Aaron Collier wrote:
>
> Looks like there might be a few topics on this but I didn't see any
> resolution. I've run through the installation steps a few times now
> and specifically the GSearch and SOLR
> (https://wiki.duraspace.org/display/ISLANDORA/milestone+6+-+Installing+Solr+and+GSearch
> <https://wiki.duraspace.org/display/ISLANDORA/milestone+6+-+Installing+Solr+and+GSearch>)
> For more information about using this group, please read our Listserv
> Guidelines: http://islandora.ca/content/welcome-islandora-listserv
> ---
> You received this message because you are subscribed to the Google
> Groups "islandora" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to islandora+...@googlegroups.com
> <mailto:islandora+...@googlegroups.com>.
> Visit this group at https://groups.google.com/group/islandora.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/islandora/9d2772e6-c60a-49a5-b086-d3327e4b07e4%40googlegroups.com
> <https://groups.google.com/d/msgid/islandora/9d2772e6-c60a-49a5-b086-d3327e4b07e4%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.
What are your logs saying? catalina.out, fedoragsearch.daily.log
What are you using for your configurations? What's your GSearch and Solr
configuration look like?
It might also be helpful to spin up an Islandora Vagrant[1] machine and
compare paths, files, and configurations.
You can also checkout Islandora Vagrant Base Box, specifically, the
GSearch[2] and Solr[3] install scripts.
-nruest
[1] https://github.com/Islandora-Labs/islandora_vagrant
[2]
https://github.com/Islandora-Labs/islandora_vagrant_base_box/blob/master/scripts/gsearch.sh
[3]
https://github.com/Islandora-Labs/islandora_vagrant_base_box/blob/master/scripts/solr.sh
On 2016-04-14 07:40 PM, Aaron Collier wrote:
> FYI - I seem to be able to get the solr index updated only if I click
> the "reload" button on the admin panel of /solr.
>
> It isn't updating [automatically] through the gsearch interface at this
> point.
>
> On Thursday, April 14, 2016 at 3:08:16 PM UTC-7, Aaron Collier wrote:
>
> Looks like there might be a few topics on this but I didn't see any
> resolution. I've run through the installation steps a few times now
> and specifically the GSearch and SOLR
> (https://wiki.duraspace.org/display/ISLANDORA/milestone+6+-+Installing+Solr+and+GSearch
> and I'm always unable to get the indexing to work automatically.
>
> I'm hoping there may be some general advice on something that may be
> missed in the instructions or something to check specifically that
> may indicate what's going on.
>
> At this point, I've only been able to manually update the index
> using /fedoragsearch/rest, but often times that doesn't even work.
>
> Thanks.
>
> --
>
> I'm hoping there may be some general advice on something that may be
> missed in the instructions or something to check specifically that
> may indicate what's going on.
>
> At this point, I've only been able to manually update the index
> using /fedoragsearch/rest, but often times that doesn't even work.
>
> Thanks.
>
> For more information about using this group, please read our Listserv
> Guidelines: http://islandora.ca/content/welcome-islandora-listserv
> ---
> You received this message because you are subscribed to the Google
> Groups "islandora" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to islandora+...@googlegroups.com
> <mailto:islandora+...@googlegroups.com>.
> Visit this group at https://groups.google.com/group/islandora.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/islandora/9d2772e6-c60a-49a5-b086-d3327e4b07e4%40googlegroups.com
> <https://groups.google.com/d/msgid/islandora/9d2772e6-c60a-49a5-b086-d3327e4b07e4%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.
Jared Whiklo
Apr 15, 2016, 9:37:24 AM4/15/16
to isla...@googlegroups.com
It sounds like maybe it isn't listening on the correct JMS topic/queue.
You could ensure that the topic/queue defined in the fedora.fcfg matches
the one in GSearch. (I can't remember where)
Alternatively (if you're tired of fighting GSearch) I wrote a camel
route web-app that replaces GSearch.
https://github.com/uml-digitalinitiatives/fc3-camel-indexer
It does what GSearch does, but it's a little clearer about what is going
on. You can use the same XSLTs but you might need to update any xsl:include.
cheers,
jared
On 2016-04-14 6:40 PM, Aaron Collier wrote:
> FYI - I seem to be able to get the solr index updated only if I click
> the "reload" button on the admin panel of /solr.
>
> It isn't updating [automatically] through the gsearch interface at this
> point.
>
> On Thursday, April 14, 2016 at 3:08:16 PM UTC-7, Aaron Collier wrote:
>
> Looks like there might be a few topics on this but I didn't see any
> resolution. I've run through the installation steps a few times now
> and specifically the GSearch and SOLR
> (https://wiki.duraspace.org/display/ISLANDORA/milestone+6+-+Installing+Solr+and+GSearch
> <https://wiki.duraspace.org/display/ISLANDORA/milestone+6+-+Installing+Solr+and+GSearch>)
Jared Whiklo
jwh...@gmail.com
--------------------------------------------------
How about never? Is never good for you?
You could ensure that the topic/queue defined in the fedora.fcfg matches
the one in GSearch. (I can't remember where)
Alternatively (if you're tired of fighting GSearch) I wrote a camel
route web-app that replaces GSearch.
https://github.com/uml-digitalinitiatives/fc3-camel-indexer
It does what GSearch does, but it's a little clearer about what is going
on. You can use the same XSLTs but you might need to update any xsl:include.
cheers,
jared
On 2016-04-14 6:40 PM, Aaron Collier wrote:
> FYI - I seem to be able to get the solr index updated only if I click
> the "reload" button on the admin panel of /solr.
>
> It isn't updating [automatically] through the gsearch interface at this
> point.
>
> On Thursday, April 14, 2016 at 3:08:16 PM UTC-7, Aaron Collier wrote:
>
> Looks like there might be a few topics on this but I didn't see any
> resolution. I've run through the installation steps a few times now
> and specifically the GSearch and SOLR
> (https://wiki.duraspace.org/display/ISLANDORA/milestone+6+-+Installing+Solr+and+GSearch
> and I'm always unable to get the indexing to work automatically.
>
> I'm hoping there may be some general advice on something that may be
> missed in the instructions or something to check specifically that
> may indicate what's going on.
>
> At this point, I've only been able to manually update the index
> using /fedoragsearch/rest, but often times that doesn't even work.
>
> Thanks.
>
>
> I'm hoping there may be some general advice on something that may be
> missed in the instructions or something to check specifically that
> may indicate what's going on.
>
> At this point, I've only been able to manually update the index
> using /fedoragsearch/rest, but often times that doesn't even work.
>
> Thanks.
>
> --
> For more information about using this group, please read our Listserv
> Guidelines: http://islandora.ca/content/welcome-islandora-listserv
> ---
> You received this message because you are subscribed to the Google
> Groups "islandora" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to islandora+...@googlegroups.com
> <mailto:islandora+...@googlegroups.com>.
> Visit this group at https://groups.google.com/group/islandora.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/islandora/9d2772e6-c60a-49a5-b086-d3327e4b07e4%40googlegroups.com
> <https://groups.google.com/d/msgid/islandora/9d2772e6-c60a-49a5-b086-d3327e4b07e4%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.
--
> For more information about using this group, please read our Listserv
> Guidelines: http://islandora.ca/content/welcome-islandora-listserv
> ---
> You received this message because you are subscribed to the Google
> Groups "islandora" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to islandora+...@googlegroups.com
> <mailto:islandora+...@googlegroups.com>.
> Visit this group at https://groups.google.com/group/islandora.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/islandora/9d2772e6-c60a-49a5-b086-d3327e4b07e4%40googlegroups.com
> <https://groups.google.com/d/msgid/islandora/9d2772e6-c60a-49a5-b086-d3327e4b07e4%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.
Jared Whiklo
jwh...@gmail.com
--------------------------------------------------
How about never? Is never good for you?
Aaron Collier
Apr 19, 2016, 4:27:29 PM4/19/16
to islandora
Hey Nick, finally getting back to the logs here. After inserting an item, I see this in fedoragsearch.daily.log:
DEBUG 2016-04-19 13:20:23,930 (Config) insertSystemProperties propertyValue=http://localhost:8080/solr
DEBUG 2016-04-19 13:20:25,860 (GenericOperationsImpl) indexDocExists pid=islandora:8 indexDocExists=false
INFO 2016-04-19 13:20:25,860 (OperationsImpl) IndexDocument=islandora:8 insertTotal=1 updateTotal=0 deleteTotal=0 emptyTotal=0 warnCount=0 docCount=7
DEBUG 2016-04-19 13:20:25,860 (OperationsImpl) closeIndexReader indexName=FgsIndex docCount=6
DEBUG 2016-04-19 13:20:25,860 (Config) insertSystemProperties propertyValue=/usr/local/fedora/solr/collection1/data/index
INFO 2016-04-19 13:20:25,861 (OperationsImpl) updateIndex fromPid indexName=FgsIndex indexDirSpace=110795 docCount=7
DEBUG 2016-04-19 13:20:25,861 (OperationsImpl) resultXml =
<solrUpdateIndex indexName="FgsIndex">
<inserted>islandora:8</inserted>
<counts insertTotal="1" updateTotal="0" deleteTotal="0" emptyTotal="0" docCount="7" warnCount="0"/>
</solrUpdateIndex>
DEBUG 2016-04-19 13:20:25,861 (GTransformer) xsltName=fgsconfigFinal/index/FgsIndex/updateIndexToResultPage
DEBUG 2016-04-19 13:20:25,888 (GTransformer) getTransformer transformer=org.apache.xalan.transformer.TransformerImpl@5730fcad uriResolver=null
DEBUG 2016-04-19 13:20:25,889 (GenericOperationsImpl) resultXml=<?xml version="1.0" encoding="UTF-8"?>
<resultPage operation="updateIndex" action="fromPid" value="islandora:8" repositoryName="FgsRepos" indexNames="" resultPageXslt="" dateTime="Tue Apr 19 13:20:17 PDT 2016">
<updateIndex xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:foxml="info:fedora/fedora-system:def/foxml#" xmlns:zs="http://www.loc.gov/zing/srw/" warnCount="0" docCount="7" deleteTotal="0" updateTotal="0" insertTotal="1" indexName="FgsIndex"/>
</resultPage>
INFO 2016-04-19 13:20:25,889 (UpdateListener) Index updated by notification message, returning:
<?xml version="1.0" encoding="UTF-8"?>
<resultPage operation="updateIndex" action="fromPid" value="islandora:8" repositoryName="FgsRepos" indexNames="" resultPageXslt="" dateTime="Tue Apr 19 13:20:17 PDT 2016">
<updateIndex xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:foxml="info:fedora/fedora-system:def/foxml#" xmlns:zs="http://www.loc.gov/zing/srw/" warnCount="0" docCount="7" deleteTotal="0" updateTotal="0" insertTotal="1" indexName="FgsIndex"/>
</resultPage>
Which seems to indicate the it was pushed over to SOLR properly, but SOLR isn't reflecting it properly until I reload the core from the solr admin interface.
Thanks.
Nick Ruest
Apr 19, 2016, 8:01:31 PM4/19/16
to isla...@googlegroups.com
Do you have an autoCommit setup in your solrconfig.xml?
Something like this:
https://github.com/discoverygarden/basic-solr-config/blob/4.x/conf/solrconfig.xml#L346
-nruest
> > an email to islandora+...@googlegroups.com <javascript:>
> > <mailto:islandora+...@googlegroups.com <javascript:>>.
> <https://groups.google.com/d/msgid/islandora/b47b26ac-2c0b-47f8-b0b2-d79c64237521%40googlegroups.com?utm_medium=email&utm_source=footer>.
Something like this:
https://github.com/discoverygarden/basic-solr-config/blob/4.x/conf/solrconfig.xml#L346
-nruest
> > <mailto:islandora+...@googlegroups.com <javascript:>>.
> > Visit this group at https://groups.google.com/group/islandora
> <https://groups.google.com/group/islandora>.
> > To view this discussion on the web visit
> >
> https://groups.google.com/d/msgid/islandora/9d2772e6-c60a-49a5-b086-d3327e4b07e4%40googlegroups.com
> <https://groups.google.com/d/msgid/islandora/9d2772e6-c60a-49a5-b086-d3327e4b07e4%40googlegroups.com>
>
> >
> <https://groups.google.com/d/msgid/islandora/9d2772e6-c60a-49a5-b086-d3327e4b07e4%40googlegroups.com?utm_medium=email&utm_source=footer
> >
> https://groups.google.com/d/msgid/islandora/9d2772e6-c60a-49a5-b086-d3327e4b07e4%40googlegroups.com
> <https://groups.google.com/d/msgid/islandora/9d2772e6-c60a-49a5-b086-d3327e4b07e4%40googlegroups.com>
>
> >
> <https://groups.google.com/d/msgid/islandora/9d2772e6-c60a-49a5-b086-d3327e4b07e4%40googlegroups.com?utm_medium=email&utm_source=footer>>.
>
> > For more options, visit https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>.
>
> > For more options, visit https://groups.google.com/d/optout
>
> --
> For more information about using this group, please read our Listserv
> Guidelines: http://islandora.ca/content/welcome-islandora-listserv
> ---
> You received this message because you are subscribed to the Google
> Groups "islandora" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to islandora+...@googlegroups.com
> <mailto:islandora+...@googlegroups.com>.
> Visit this group at https://groups.google.com/group/islandora.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/islandora/b47b26ac-2c0b-47f8-b0b2-d79c64237521%40googlegroups.com
> --
> For more information about using this group, please read our Listserv
> Guidelines: http://islandora.ca/content/welcome-islandora-listserv
> ---
> You received this message because you are subscribed to the Google
> Groups "islandora" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to islandora+...@googlegroups.com
> <mailto:islandora+...@googlegroups.com>.
> Visit this group at https://groups.google.com/group/islandora.
> To view this discussion on the web visit
> <https://groups.google.com/d/msgid/islandora/b47b26ac-2c0b-47f8-b0b2-d79c64237521%40googlegroups.com?utm_medium=email&utm_source=footer>.
Aaron Collier
Apr 20, 2016, 12:23:36 PM4/20/16
to islandora
PROGRESS!!
I was sure this was in place but must have been overwritten. It is now working on my dev system. Now to go see if this addresses the entity issue.
Thanks!!
Aaron Collier
Apr 20, 2016, 12:28:01 PM4/20/16
to islandora
FYI - the version i pulled down from: https://github.com/discoverygarden/basic-solr-config.git
is missing the line:
<openSearcher>true</openSearcher>
which appears to be the main issue here.
Aaron Collier
Apr 20, 2016, 5:55:17 PM4/20/16
to islandora
WOOH!
Nick, your tip was a huge help. I was pulling the wrong branch of the config. All is well now!
Thanks thanks thanks!
Beer on me next time I see you.
nick ruest
Apr 20, 2016, 7:07:22 PM4/20/16
to isla...@googlegroups.com
Awesome! Glad to hear!
To unsubscribe from this group and stop receiving emails from it, send an email to islandora+...@googlegroups.com.
Visit this group at https://groups.google.com/group/islandora.
To view this discussion on the web visit https://groups.google.com/d/msgid/islandora/9d2f5c10-f39c-482e-ad79-5d79a33053e5%40googlegroups.com.
Reply all
Reply to author
Forward
0 new messages