We encountered a worrisome problem today after filling up a filesystem and being forced to move our data to another filesystem. After the move and rebuild, we can no longer add datastreams to objects.
I'm running Fedora 3.4, with the default Akubra low-level storage module.
Here's what happened:
1. filesystem /fedora/data filled up, Fedora (and Tomcat) hung.
2. I shut down Fedora, Tomcat, Postgres.
3. I moved the directory /fedora/data/objects to another filesystem, /fedora/db01/objects.
4. I moved the directory /fedora/data/datastreams to another filesystem, /fedora/db01/datastreams.
5. I made symlinks from the original locations to the new locations:
/fedora/data/objects -> /fedora/db01/objects
/fedora/data/datastreams -> /fedora/db01/datastreams
6. I started up Postgres, and ran fedora-rebuild.sh twice, once to rebuild the database and once to rebuild the resource index. The rebuild completed successfully.
7. I then started up Tomcat + Fedora, and I was able to view all the objects.
8. I then did a test ingest of an object, and a stub object was successfully created, with a DC datastream. I verified that I could find the object and the datastream on the filesystem (both through the direct path and through the symlinked path). I verified that permsissions were correct at every directory level, and at the file level.
I then try to add a datastream to the test object, and I receive this error (truncated):
ERROR 2010-10-05 14:51:16.424 [http-9090-1] (WebApplicationImpl) Internal server error
javax.ws.rs.WebApplicationException: org.fcrepo.server.errors.GeneralException: Unable to add or modify object (commit canceled)
[...]
Caused by: java.io.IOException: Rename failed for an unknown reason.
at org.akubraproject.fs.FSBlob.moveTo(FSBlob.java:151) [akubra-fs-0.3.jar:na]
I've tested this with the REST API, through the web admin interface, and through the java GUI admin tool, and the results are the same in all three cases. This error occurs regardless of whether I'm adding a datastream to a newly-created object or attempting to add a datastream to an older object.
This may properly be an Akubra matter, but I thought I would bring it up here, first. Any suggestions or help would be greatly appreciated.
thanks,
-- Scott
Following is the full stack trace:
ERROR 2010-10-05 14:51:16.424 [http-9090-1] (WebApplicationImpl) Internal server error
javax.ws.rs.WebApplicationException: org.fcrepo.server.errors.GeneralException: Unable to add or modify object (commit canceled)
at org.fcrepo.server.rest.BaseRestResource.handleException(BaseRestResource.java:168) [fcrepo-server-3.4.jar:na]
at org.fcrepo.server.rest.DatastreamResource.addOrUpdateDatastream(DatastreamResource.java:583) [fcrepo-server-3.4.jar:na]
at org.fcrepo.server.rest.DatastreamResource.addDatastream(DatastreamResource.java:358) [fcrepo-server-3.4.jar:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [na:1.6.0_21]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) [na:1.6.0_21]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [na:1.6.0_21]
at java.lang.reflect.Method.invoke(Method.java:597) [na:1.6.0_21]
at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:175) [jersey-bundle-1.0.3.1.jar:1.0.3.1]
at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:67) [jersey-bundle-1.0.3.1.jar:1.0.3.1]
at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:163) [jersey-bundle-1.0.3.1.jar:1.0.3.1]
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:111) [jersey-bundle-1.0.3.1.jar:1.0.3.1]
at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:71) [jersey-bundle-1.0.3.1.jar:1.0.3.1]
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:111) [jersey-bundle-1.0.3.1.jar:1.0.3.1]
at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:63) [jersey-bundle-1.0.3.1.jar:1.0.3.1]
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:689) [jersey-bundle-1.0.3.1.jar:1.0.3.1]
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:647) [jersey-bundle-1.0.3.1.jar:1.0.3.1]
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:638) [jersey-bundle-1.0.3.1.jar:1.0.3.1]
at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:309) [jersey-bundle-1.0.3.1.jar:1.0.3.1]
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:425) [jersey-bundle-1.0.3.1.jar:1.0.3.1]
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:590) [jersey-bundle-1.0.3.1.jar:1.0.3.1]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) [servlet-api.jar:na]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) [catalina.jar:6.0.29]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) [catalina.jar:6.0.29]
at org.fcrepo.server.security.servletfilters.FilterRestApiFlash.doFilter(FilterRestApiFlash.java:66) [fcrepo-server-3.4.jar:na]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) [catalina.jar:6.0.29]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) [catalina.jar:6.0.29]
at org.fcrepo.server.security.servletfilters.FilterSetup.doFilter(FilterSetup.java:235) [fcrepo-server-3.4.jar:na]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) [catalina.jar:6.0.29]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) [catalina.jar:6.0.29]
at org.fcrepo.server.security.servletfilters.FilterSetup.doFilter(FilterSetup.java:235) [fcrepo-server-3.4.jar:na]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) [catalina.jar:6.0.29]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) [catalina.jar:6.0.29]
at org.fcrepo.server.security.servletfilters.FilterSetup.doFilter(FilterSetup.java:235) [fcrepo-server-3.4.jar:na]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) [catalina.jar:6.0.29]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) [catalina.jar:6.0.29]
at org.fcrepo.server.security.servletfilters.FilterSetup.doFilter(FilterSetup.java:235) [fcrepo-server-3.4.jar:na]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) [catalina.jar:6.0.29]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) [catalina.jar:6.0.29]
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) [catalina.jar:6.0.29]
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) [catalina.jar:6.0.29]
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:563) [catalina.jar:6.0.29]
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) [catalina.jar:6.0.29]
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) [catalina.jar:6.0.29]
at org.apache.catalina.valves.RequestFilterValve.process(RequestFilterValve.java:269) [catalina.jar:6.0.29]
at org.apache.catalina.valves.RemoteAddrValve.invoke(RemoteAddrValve.java:81) [catalina.jar:6.0.29]
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555) [catalina.jar:6.0.29]
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) [catalina.jar:6.0.29]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298) [catalina.jar:6.0.29]
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:857) [tomcat-coyote.jar:6.0.29]
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588) [tomcat-coyote.jar:6.0.29]
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489) [tomcat-coyote.jar:6.0.29]
at java.lang.Thread.run(Thread.java:619) [na:1.6.0_21]
Caused by: org.fcrepo.server.errors.GeneralException: Unable to add or modify object (commit canceled)
at org.fcrepo.server.storage.DefaultDOManager.doCommit(DefaultDOManager.java:1438) [fcrepo-server-3.4.jar:na]
at org.fcrepo.server.storage.SimpleDOWriter.commit(SimpleDOWriter.java:508) [fcrepo-server-3.4.jar:na]
at org.fcrepo.server.management.DefaultManagement.addDatastream(DefaultManagement.java:572) [fcrepo-server-3.4.jar:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [na:1.6.0_21]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) [na:1.6.0_21]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [na:1.6.0_21]
at java.lang.reflect.Method.invoke(Method.java:597) [na:1.6.0_21]
at org.fcrepo.server.messaging.NotificationInvocationHandler.invoke(NotificationInvocationHandler.java:68) [fcrepo-server-3.4.jar:na]
at $Proxy0.addDatastream(Unknown Source) [na:na]
at org.fcrepo.server.management.ManagementModule.addDatastream(ManagementModule.java:227) [fcrepo-server-3.4.jar:na]
at org.fcrepo.server.rest.DatastreamResource.addOrUpdateDatastream(DatastreamResource.java:467) [fcrepo-server-3.4.jar:na]
... 50 common frames omitted
Caused by: org.fcrepo.common.FaultException: java.io.IOException: Rename failed for an unknown reason.
at org.fcrepo.server.storage.lowlevel.akubra.AkubraLowlevelStorage.rename(AkubraLowlevelStorage.java:363) [fcrepo-server-3.4.jar:na]
at org.fcrepo.server.storage.lowlevel.akubra.AkubraLowlevelStorage.safeOverwrite(AkubraLowlevelStorage.java:306) [fcrepo-server-3.4.jar:na]
at org.fcrepo.server.storage.lowlevel.akubra.AkubraLowlevelStorage.replace(AkubraLowlevelStorage.java:244) [fcrepo-server-3.4.jar:na]
at org.fcrepo.server.storage.lowlevel.akubra.AkubraLowlevelStorage.replaceObject(AkubraLowlevelStorage.java:150) [fcrepo-server-3.4.jar:na]
at org.fcrepo.server.storage.lowlevel.akubra.AkubraLowlevelStorageModule.replaceObject(AkubraLowlevelStorageModule.java:79) [fcrepo-server-3.4.jar:na]
at org.fcrepo.server.storage.DefaultDOManager.doCommit(DefaultDOManager.java:1334) [fcrepo-server-3.4.jar:na]
... 60 common frames omitted
Caused by: java.io.IOException: Rename failed for an unknown reason.
at org.akubraproject.fs.FSBlob.moveTo(FSBlob.java:151) [akubra-fs-0.3.jar:na]
at org.akubraproject.map.IdMappingBlob.moveTo(IdMappingBlob.java:69) [akubra-map-0.3.jar:na]
at org.fcrepo.server.storage.lowlevel.akubra.AkubraLowlevelStorage.rename(AkubraLowlevelStorage.java:361) [fcrepo-server-3.4.jar:na]
... 65 common frames omitted
--
Scott Prater
Library, Instructional, and Research Applications (LIRA)
Division of Information Technology (DoIT)
University of Wisconsin - Madison
pra...@wisc.edu