FedoraGsearch not indexing submissions

[Peter .]

I am running Fedora 3.6.2 with Solr 3.6.2 (I couldn't get any of the Solr 4.x series to play nice with fgs) and fedoragsearch 2.6. I can ingest items into fedora with no problems. I can visit all the following pages:

localhost:8080/solr
localhost:8080/fedoragsearch
localhost:8080/fedora

When I go to localhost:8080/fedoragsearch and hit the update index button nothing happens. I can confirm that an object is in the respository. I tried stopping solr and removing the index directory and starting it up again - no change. What could be the problem?

Thanks.

[Peter .]

Comparing islandora sandbox with my own setup, I noticed that I have solr search set up for the entire site along with solr for islandora. Is it okay to use both modules or do I need to remove the solr module for drupal and just use the solr module for islandora?

[Peter .]

Is this a policy issue? This is what I have in my permit-apim-to-authenticated.xml file:

<?xml version="1.0" encoding="UTF-8"?>
<Policy xmlns="urn:oasis:names:tc:xacml:1.0:policy"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        PolicyId="permit-apim-to-authenticated"
        RuleCombiningAlgId="urn:oasis:names:tc:xacml:1.0:rule-combining-algorithm:first-applicable">
  <Description>note that other policies may provide exceptions to this broad policy.  This policy assumes api-m users have to be authenticated</Description>
  <Target>
    <Subjects>
      <AnySubject/>
    </Subjects>
    <Resources>
      <AnyResource/>
    </Resources>
    <Actions>
      <Action>
        <ActionMatch MatchId="urn:oasis:names:tc:xacml:1.0:function:string-equal">
          <AttributeValue DataType="http://www.w3.org/2001/XMLSchema#string">urn:fedora:names:fedora:2.1:action:api-m</AttributeValue>
          <ActionAttributeDesignator DataType="http://www.w3.org/2001/XMLSchema#string"
            AttributeId="urn:fedora:names:fedora:2.1:action:api"/>
        </ActionMatch>
      </Action>
    </Actions>
  </Target>
  <Rule RuleId="1" Effect="Permit"/>
</Policy>

[Peter .]

Oops, forgot to add the error I am seing in fedora.log:

org.fcrepo.server.errors.authorization.AuthzDeniedException:

at org.fcrepo.server.security.impl.DefaultPolicyEnforcementPoint.enforce(DefaultPolicyEnforcementPoint.java:143) ~[fcrepo-server-3.6.2.jar:na]

at org.fcrepo.server.security.DefaultAuthorization.enforcePurgeObject(DefaultAuthorization.java:803) ~[fcrepo-server-3.6.2.jar:na]

at org.fcrepo.server.management.DefaultManagement.purgeObject(DefaultManagement.java:362) ~[fcrepo-server-3.6.2.jar:na]

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.6.0_30]

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.6.0_30]

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.6.0_30]

at java.lang.reflect.Method.invoke(Method.java:622) ~[na:1.6.0_30]

at org.fcrepo.server.messaging.NotificationInvocationHandler.invoke(NotificationInvocationHandler.java:76) ~[fcrepo-server-3.6.2.jar:na]

at com.sun.proxy.$Proxy7.purgeObject(Unknown Source) ~[na:na]

at org.fcrepo.server.management.ManagementModule.purgeObject(ManagementModule.java:486) ~[fcrepo-server-3.6.2.jar:na]

at org.fcrepo.server.rest.FedoraObjectsResource.deleteObject(FedoraObjectsResource.java:385) ~[fcrepo-server-3.6.2.jar:na]

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.6.0_30]

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.6.0_30]

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.6.0_30]

at java.lang.reflect.Method.invoke(Method.java:622) ~[na:1.6.0_30]

at org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:180) ~[cxf-api-2.6.2.jar:2.6.2]

at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96) ~[cxf-api-2.6.2.jar:2.6.2]

at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:167) ~[cxf-rt-frontend-jaxrs-2.6.2.jar:2.6.2]

at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:94) ~[cxf-rt-frontend-jaxrs-2.6.2.jar:2.6.2]

at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:58) ~[cxf-api-2.6.2.jar:2.6.2]

at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:94) ~[cxf-api-2.6.2.jar:2.6.2]

at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:262) ~[cxf-api-2.6.2.jar:2.6.2]

at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121) ~[cxf-api-2.6.2.jar:2.6.2]

at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:211) ~[cxf-rt-transports-http-2.6.2.jar:2.6.2]

at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:213) ~[cxf-rt-transports-http-2.6.2.jar:2.6.2]

at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:154) ~[cxf-rt-transports-http-2.6.2.jar:2.6.2]

at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:130) ~[cxf-rt-transports-http-2.6.2.jar:2.6.2]

at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:221) ~[cxf-rt-transports-http-2.6.2.jar:2.6.2]

at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:141) ~[cxf-rt-transports-http-2.6.2.jar:2.6.2]

at javax.servlet.http.HttpServlet.service(HttpServlet.java:637) ~[tomcat6-servlet-2.5-api-6.0.24.jar:na]

at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:197) ~[cxf-rt-transports-http-2.6.2.jar:2.6.2]

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) ~[catalina-6.0.24.jar:na]

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) ~[catalina-6.0.24.jar:na]

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:369) ~[spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE]

at org.fcrepo.server.security.jaas.AuthFilterJAAS.doFilter(AuthFilterJAAS.java:329) ~[fcrepo-security-jaas-3.6.2.jar:na]

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:381) ~[spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE]

at org.springframework.security.web.access.channel.ChannelProcessingFilter.doFilter(ChannelProcessingFilter.java:109) ~[spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE]

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:381) ~[spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE]

at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:168) ~[spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE]

at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237) ~[spring-web-3.0.7.RELEASE.jar:3.0.7.RELEASE]

at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167) ~[spring-web-3.0.7.RELEASE.jar:3.0.7.RELEASE]

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) ~[catalina-6.0.24.jar:na]

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) ~[catalina-6.0.24.jar:na]

at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) ~[catalina-6.0.24.jar:na]

at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) ~[catalina-6.0.24.jar:na]

at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) ~[catalina-6.0.24.jar:na]

at com.googlecode.psiprobe.Tomcat60AgentValve.invoke(Tomcat60AgentValve.java:30) ~[na:na]

at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) ~[catalina-6.0.24.jar:na]

at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) ~[catalina-6.0.24.jar:na]

at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298) ~[catalina-6.0.24.jar:na]

at org.apache.coyote.http11.Http11AprProcessor.process(Http11AprProcessor.java:865) ~[tomcat-coyote-6.0.24.jar:na]

at org.apache.coyote.http11.Http11AprProtocol$Http11ConnectionHandler.process(Http11AprProtocol.java:579) ~[tomcat-coyote-6.0.24.jar:na]

at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1555) ~[tomcat-coyote-6.0.24.jar:na]

at java.lang.Thread.run(Thread.java:701) ~[na:1.6.0_30]

[Peter .]

When I try to browse the index in fedoragsearch, it lists terms from the drupal database and not the fedora database. It looks like solr is indexing the drupal database and not the fedora one. Is there a configuration I missed somewhere to get this working?

[Peter .]

Well I have to say that some serious work needs to be done for the documentation for getting fedoragsearch installed and working correctly. I have gone though the set up in the islandora sandbox and compared that with my configuration and the online documentation for getting things going. While my setup follows the online documentation pretty closely, the sandbox version does not. In fact I don't know how the sandbox version is actually working since its fgsconfig-basic.properties file has incorrect paths in it.

One thing I did notice in that file, is that they are using Lucene instead of Solr. Is that what I need to do in order to get this working? I also noticed that the sandbox does not use permit-apim-to-authenticated.xml file. I am not a pro at xacml files so I am thinking this could be where my real problem lies.

Still hoping someone can shed some light on this issue.

Thanks.

[Peter .]

When I am at localhost:8080/fedoragsearch and on the browseIndex tab and click browse, I get the following error (it still doesn't list fields from the fedora repository in the fieldname dropdown:

Thu Mar 20 10:53:28 PDT 2014 transform /fgsconfigFinal/rest/adminBrowseIndexToHtml.xslt: ; nested exception is: net.sf.saxon.trans.XPathException: org.xml.sax.SAXParseException; Premature end of file.

[Gert Schmeltz Pedersen]

You get Premature end of file because you have some serious error in your setup.

I think you should configure following the documentation page included with the official GSearch download.

For Islandora you will use Solr. Lucene is the index and search library included in Solr.

As I recommended on the fedora list, if you do not get objects indexed as expected:

Start by looking into the indexing of one Fedora object:

1. remove the solr index directory

2. goto http://localhost:8080/fedoragsearch/rest?operation=updateIndex

3. enter the Pid of the Fedora object and press updateIndex fromPid

4. if an index document was inserted then click browseIndex else goto step 7

5. open the Field name select box, select PID, press Browse

6. if the PID appears, click it to search for it, and click getIndexDocument to see its index document

    if the PID does not appear, no index document was generated for the PID, then

7. look into fedoragsearch.log to find out what happened during step 3

    find the part where the generation of the index document takes place

    notice that your indexing stylesheet, .../index/FgsIndex/foxmlToSolr.xslt is the key element

    you probably need to inspect closely what happens when the indexing stylesheet generates the index document from the foxml of your Fedora object

    it is instructive to watch this process by running xsltproc on the stylesheet and the foxml from command line

Let me know how far you get following these steps.

Gert

[Peter .]

After playing around with this for a couple more days, I have found that solr throws errors when I copy over the schema-4.2.0-for-fgs-2.6.xml file into solr/conf/schema.xml. If I revert schema file back to the orginal schema.xml file solr is happy but fedoragsearch isn't. The documentation for this part of the process on the islandora site is poor. I think there are some steps missing or more clarity is needed. Any ideas as to why schema-4.2.0-for-fgs-2.6.xml is causing the issues? There is something in that file that wants solr to load

solr.SpatialRecursivePrefixTreeFieldType which it cant find.

Thanks.

[nick ruest]

Hi Peter-

This has been mentioned a few times before, but I will repeat it again here. The documentation and lack instructions for the most recent versions of GSearch and Solr because they were not officially supported in the 7.x-1.2; the most recent official release of Islandora.

If you would like to continue down the path of the recent versions of GSearch and Solr, I would suggest searching through both Google groups. There are a few threads on this and there are example config files available on Github.

-nruest

I

--
You received this message because you are subscribed to the Google Groups "islandora" group.
To unsubscribe from this group and stop receiving emails from it, send an email to islandora+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Peter .]

Hi Nick,

I am not using the latest versions of solr nor gsearch. I am using solr-3.6.2 and gsearch 2.6 with fedora 3.6.2. I would assume that these versions are supported. Are you telling me they are not (they are the versions I see being used in the sandbox)? I have searched this group and have not found anything to address my issue. It is interesting that you mention that Islandora 7.1.2 doesn't support the latest versions of solr and yet the link in the installation page https://wiki.duraspace.org/display/ISLANDORA712/Installing+Solr+and+GSearch points directly to the latest version. No where on this page does it specify what versions are officially supported and what version of the files should be copied. I would challenge you to follow these instructions as written and see if you get a working installation.

Cheers,
Peter

[Peter Murray]

From one Peter to another, I respectfully submit these points.

The Islandora community is made up primarily of volunteers.  Except for Melissa, our community manager, each of us have day jobs for employers that have us using Islandora.  Speaking only for myself — one of the multitude that has a day job outside of this community — I participate in the community out of an enlightened self-interest: I want to see the software grow and its user base expand.  My time participating in the community’s activities is a hedge against paying support costs to some company’s proprietary black box software that I can neither control nor influence to any large degree.

This is complex software.  It has a lot of moving parts (Drupal, Islandora and its modules, Tomcat, Fedora Commons, FedoraGSearch, SOLR, Djatoka, Open SeaDragon, Apache HTTPD, Tesseract, Kakadu, ImageMagick, FFMPEG, and many others) that provide a powerful base of functionality, but each with its own complexity.  (Personally, I have an intense dislike of SOLR because it and I don’t share a world view, and so I dread any SOLR issue that comes up with our installation.)  Getting things to work takes trial and error, experimentation, examining the logs, reading the documentation, querying your favorite search engine, and consulting with colleagues.  One’s best bet may be to start with the VM where everything is configured to a known, good working state and picking apart the pieces until you understand them.  Speaking for myself again, I feel bad about the complexity of the software and try to make it better by removing complexity and improving the documentation.  I don’t want the barriers of entry to be high, but they are and those barriers can only be lowered so far.

There are those in the community that are willing to engage with clients on a consulting and/or hosted operation basis.  Most of us the rest of us are here, if I may make general assumptions, because we like the software, like the community, and want to see both thrive.  We ask questions when we need something and we answer questions when we know something.

I’m sorry that you are having issues — really, I am, because I want you to join the community and share your knowledge with the next person that joins.  I sense your frustrations, and project my own feelings of single-minded focus when the pieces are not working as advertised and desired.  I would point out — again, with respect — that your expectations for the required “sweat equity” in the project and expectations of the community to respond to multiple, repeated queries and challenges are not in the community norms.  They are also not typical of most open source projects.

To your specific question that started this thread, if you have a directory named in part "schema-4.2.0” I don’t think you can reasonably expect it to work with SOLR 3.6.2.  In particular, it looks like SpatialRecursivePrefixTreeFieldType was something introduced in Solr4 (https://cwiki.apache.org/confluence/display/solr/Spatial+Search).

Peter

--

Peter Murray
Assistant Director, Technology Services Development
LYRASIS
Peter....@lyrasis.org
+1 678-235-2955
800.999.8558 x2955

[Rick Sarvas]

Peter,
Ping me at my work email:

richard.sarvas @ lib.uconn.edu

I'm also working on a series of Fedora Commons/Islandora/Solr issues. Perhaps between the pair of us we can we can get our respective issues resolved and possibly pass some information back to the Islandora folks that can be used to revise the install documentation.


Rick