Fedora Rebuild Failed

[mcf...@grinnell.edu]

Good morning. 

Over the course of many weeks I have been working on upgrading Grinnell's RepositoryX server (my test/development host for Fedora, FGS and Solr) but I keep running into issues with my Fedora data.  Last evening I cleared the $FEDORA_HOME/data directory from RepositoryX and started a fresh rsync of corresponding data from our production environment.  That process completed with an indication that some files could not be copied, but the repository is conveying valid objects forward to my test front-end server.

However, when I try to rebuild either the resource index or the MySQL database I get the following errors back, and this also renders Fedora unable to be restarted...

java.lang.IllegalArgumentException: Malformed token-as-blobId: info:fedora/grinnell:16221/new
java.lang.IllegalArgumentException: Malformed token-as-blobId: info:fedora/grinnell:16221/new
    at org.fcrepo.server.storage.lowlevel.akubra.AkubraLowlevelStorage.getToken(AkubraLowlevelStorage.java:618)
    at org.fcrepo.server.storage.lowlevel.akubra.AkubraLowlevelStorage.access$000(AkubraLowlevelStorage.java:49)
    at org.fcrepo.server.storage.lowlevel.akubra.AkubraLowlevelStorage$ConnectionClosingKeyIterator.next(AkubraLowlevelStorage.java:698)
    at org.fcrepo.server.storage.lowlevel.akubra.AkubraLowlevelStorage$ConnectionClosingKeyIterator.next(AkubraLowlevelStorage.java:676)
    at org.fcrepo.server.utilities.rebuild.Rebuild.run(Rebuild.java:138)
    at org.fcrepo.server.utilities.rebuild.Rebuild.main(Rebuild.java:467)

Object grinnell:16221 is indeed missing.  I believe it was ingested in October 2016, it was later found to be a duplicate of another object so it was deleted in March 2017.  I am no longer sure what means were used to delete it, but I believe the condition leading to this error exists in my production data and was copied to RepositoryX as part of the rsync.

Can anyone suggest what I might do to clear this error and get my test repository back into working condition?

Thanks in advance for any suggestions.

-Mark M.

[McFate, Mark]

Fedora rebuild is still not working but I did manage to get the server restarted.  Now the fedora-rebuild.sh script refuses to run and throws this...

ERROR 09:16:19.898 (Server) Fatal error while starting server
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.fcrepo.server.messaging.Messaging': Invocation of init method failed; nested exception is org.fcrepo.server.errors.ModuleInitializationException: Could not create Transport. Reason: java.io.IOException: Failed to bind to server socket: tcp://localhost:61616 due to: java.net.BindException: Address already in use

Can anyone suggest how to free up localhost:61616, or any remedy for this?

Mark A. McFate
Digital Library Applications Developer
Burling Library, Grinnell College
1111 6th Ave., Grinnell, IA  50112-1690
(641) 269-3674    mcf...@grinnell.edu

---

From: isla...@googlegroups.com <isla...@googlegroups.com> on behalf of mcf...@grinnell.edu <mcf...@grinnell.edu>
Sent: Wednesday, July 26, 2017 8:52 AM
To: islandora
Subject: [islandora] Fedora Rebuild Failed

 

--
For more information about using this group, please read our Listserv Guidelines: http://islandora.ca/content/welcome-islandora-listserv
---
You received this message because you are subscribed to a topic in the Google Groups "islandora" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/islandora/zCPRU8opAAg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to islandora+...@googlegroups.com.
Visit this group at https://groups.google.com/group/islandora.
To view this discussion on the web visit https://groups.google.com/d/msgid/islandora/1d6b91f9-d7b5-48a2-a991-4fed3bccfead%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[dp...@metro.org]

Hi, your ActiveMQ broker got stuck probably. Try shutting down tomcat (not a restart), wait a bit, check the logs,.. then start again. Data directory includes(activemq-data , datastreamStore, fedora-xacml-policies, objectStore, resourceIndex) . All there? Not sure if moving activemq-data from one server to another works as expected, but i guess you can start by checking your permissions on /data? Should be tomcat7 tomcat7 

Any strange entries in catalina.log?

Diego Pino

Metro.org

[McFate, Mark]

Good morning Diego and thanks for the quick reply.

So I rebooted the sever, not just Tomcat, to clear the port conflict and Fedora did restart successfully.  Whenever I restart it now I still get these...

INFO: Closing Spring root WebApplicationContext
Jul 26, 2017 9:34:03 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/fedora] appears to have started a thread named [JotmBatch] but has failed to stop it. This is very likely to create a memory leak.
Jul 26, 2017 9:34:03 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/fedora] appears to have started a thread named [JotmClock] but has failed to stop it. This is very likely to create a memory leak.
Jul 26, 2017 9:34:03 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/fedora] appears to have started a thread named [Write-lock Reaper] but has failed to stop it. This is very likely to create a memory leak.
Jul 26, 2017 9:34:03 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/fedora] appears to have started a thread named [pool-2-thread-1] but has failed to stop it. This is very likely to create a memory leak.
Jul 26, 2017 9:34:03 AM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap
SEVERE: The web application [/fedora] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@623c7200]) and a value of type [org.apache.cxf.BusFactory.BusHolder] (value [org.apache.cxf.BusFactory$BusHolder@6e609062]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.
ERROR 2017-07-26 09:34:04,197 (MessagingClient) Messaging Exception encountered attempting to stop Messaging Client: fedoragsearch0. Exception message: The JMS connection has failed: java.io.EOFException
com.yourmediashelf.fedora.client.messaging.MessagingException: The JMS connection has failed: java.io.EOFException

...but I've been told before that they can be ignored as they are part of shutdown, not startup.

The server is running again, and yes, all of the ../data directories you mentioned are there.  Mine are all owned by fedora:fedora as that is the user:group account that basically owns the Tomcat process.

I'm going to start another build of my resource index but I fear it will throw the same grinnell:16221/new errors as before.  So I really wonder what's behind that issue and how to fix it up?

Thanks

Mark A. McFate
Digital Library Applications Developer
Burling Library, Grinnell College
1111 6th Ave., Grinnell, IA  50112-1690
(641) 269-3674    mcf...@grinnell.edu

---

From: isla...@googlegroups.com <isla...@googlegroups.com> on behalf of dp...@metro.org <dp...@metro.org>
Sent: Wednesday, July 26, 2017 9:29:50 AM
To: islandora
Cc: Ciota, Rebecca; Bauder, Julia; Ellis, David; Wilkins, Michael
Subject: Re: [islandora] Fedora Rebuild Failed

 

To view this discussion on the web visit https://groups.google.com/d/msgid/islandora/40ca19bb-773e-44b2-9243-effab4399434%40googlegroups.com.

[mcf...@grinnell.edu]

I'm back to looking at this again and my Fedora is in a better place (I moved the ActiveMQ message broker to a disk of its own) and working nicely, BUT I still can't rebuild my indicies.  Deleted obejct grinnell:16221 is still causing problems...

Rebuild failed:

java.lang.IllegalArgumentException: Malformed token-as-blobId: info:fedora/grinnell:16221/new
java.lang.IllegalArgumentException: Malformed token-as-blobId: info:fedora/grinnell:16221/new
    at org.fcrepo.server.storage.lowlevel.akubra.AkubraLowlevelStorage.getToken(AkubraLowlevelStorage.java:618)
    at org.fcrepo.server.storage.lowlevel.akubra.AkubraLowlevelStorage.access$000(AkubraLowlevelStorage.java:49)
    at org.fcrepo.server.storage.lowlevel.akubra.AkubraLowlevelStorage$ConnectionClosingKeyIterator.next(AkubraLowlevelStorage.java:698)
    at org.fcrepo.server.storage.lowlevel.akubra.AkubraLowlevelStorage$ConnectionClosingKeyIterator.next(AkubraLowlevelStorage.java:676)
    at org.fcrepo.server.utilities.rebuild.Rebuild.run(Rebuild.java:138)
    at org.fcrepo.server.utilities.rebuild.Rebuild.main(Rebuild.java:467)

I'm still looking for a viable fix for this problem and I'm all ears.

[McFate, Mark]

And I think I have this fixed!  The solution was pretty quick and simple now that my ActiveMQ is running better.  I navigated to /usr/local/fedora/data and did a “find . -name *16221*”.  I had done this before but specified too much of the PID and ‘find' didn’t locate anything.  8^(  This time it worked, so I deleted the file ./objectStore/c0/info%3Afedora%2Fgrinnell%3A16221%2Fnew  and re-launched the rebuild script.  It worked without error!

--
For more information about using this group, please read our Listserv Guidelines: http://islandora.ca/content/welcome-islandora-listserv
---
You received this message because you are subscribed to a topic in the Google Groups "islandora" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/islandora/zCPRU8opAAg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to islandora+...@googlegroups.com.
Visit this group at https://groups.google.com/group/islandora.

To view this discussion on the web visit https://groups.google.com/d/msgid/islandora/a9abab15-1fc9-411e-88f1-f5aa4e13dcef%40googlegroups.com.