Batch ingest video error 500

259 views
Skip to first unread message

Lingling Jiang

unread,
Mar 19, 2015, 3:53:06 PM3/19/15
to isla...@googlegroups.com
Hi folks,

I am back on batch ingest headache again. 

This time I tried to ingest video file (.mov file). I tried to convert .mov to .mp4 with ffmpeg command directly and it worked. So the .mov file is good.
My video test directory only contains a .mov file. there is no any metaddata xml file which means there is no encoding issue. 

Commands I used are:
drush -v -d --user=admin --uri=http://ourdomain islandora_batch_scan_preprocess --type=directory --target=/video/testing --namespace=lingling1  --content_models=islandora:sp_videoCModel  --parent=lingling1:collection

drush -v -d --user=admin --uri=http://ourdomain islandora_batch_ingest

The first command is OK. but the second threw the following error message:

03/19/15 15:34:36  Successfully logged into Drupal as admin (uid=1) [1.91 sec, 37.96 MB][bootstrap]


03/19/15 15:34:36  Found command: batch-process (commandfile=core) [1.91 sec, 37.96 MB] [bootstrap]


03/19/15 15:34:36  Calling hook drush_core_batch_process [2.03 sec, 38.74 MB]               [debug]


03/19/15 15:41:10  WD islandora: Failed to ingest object: lingling1:1367code:           [error]


03/19/15 15:41:10  500message:  [395.56 sec, 41.02 MB]


03/19/15 15:41:10  Exception occured: Failed to ingest lingling1:1367. [395.56 sec,     [ok]


03/19/15 15:41:10  41.01 MB]


03/19/15 15:41:10  Returned from hook drush_core_batch_process [395.6 sec, 42.49 MB]        [debug]


Drupal log message only shows:

Failed to ingest object: lingling1:1367
code: 500
message: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">

On Fedora.log I can see some messages around time 03/19/15 15:41:10

uploadService has thrown exception, unwinding no

org.apache.cxf.interceptor.Fault: Could not send Message.


        at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:64) ~[cxf-api-2.7.3.jar:2.7.3]


        at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:271) [cxf-api-2.7.3.jar:2.7.3]


        at org.apache.cxf.interceptor.OutgoingChainInterceptor.handleMessage(OutgoingChainInterceptor.java:77) [cxf-api-2.7.3.jar:2.7.3]


        at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:271) [cxf-api-2.7.3.jar:2.7.3]


        at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121) [cxf-api-2.7.3.jar:2.7.3]


        at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:239) [cxf-rt-transports-http-2.7.3.jar:2.7.3]


        at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:218) [cxf-rt-transports-http-2.7.3.jar:2.7.3]


        at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:198) [cxf-rt-transports-http-2.7.3.jar:2.7.3]


        at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:137) [cxf-rt-transports-http-2.7.3.jar:2.7.3]


        at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:158) [cxf-rt-transports-http-2.7.3.jar:2.7.3]


        at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:243) [cxf-rt-transports-http-2.7.3.jar:2.7.3]


        at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:163) [cxf-rt-transports-http-2.7.3.jar:2.7.3]


        at javax.servlet.http.HttpServlet.service(HttpServlet.java:641) [servlet-api-3.0.jar:na]


        at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:219) [cxf-rt-transports-http-2.7.3.jar:2.7.3]


        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305) [tomcat-catalina-7.0.28.jar:7.0.28]


        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) [tomcat-catalina-7.0.28.jar:7.0.28]


        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:369) [spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE]


        at org.fcrepo.server.security.jaas.AuthFilterJAAS.doFilter(AuthFilterJAAS.java:329) [fcrepo-security-jaas-3.7.1.jar:na]


        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:381) [spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE]


        at org.springframework.security.web.access.channel.ChannelProcessingFilter.doFilter(ChannelProcessingFilter.java:109) [spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE]


        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:381) [spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE]


        at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:168) [spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE]


        at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237) [spring-web-3.0.7.RELEASE.jar:3.0.7.RELEASE]


        at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167) [spring-web-3.0.7.RELEASE.jar:3.0.7.RELEASE]


        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) [tomcat-catalina-7.0.28.jar:7.0.28]


        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) [tomcat-catalina-7.0.28.jar:7.0.28]


        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:225) [tomcat-catalina-7.0.28.jar:7.0.28]


        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123) [tomcat-catalina-7.0.28.jar:7.0.28]


        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472) [tomcat-catalina-7.0.28.jar:7.0.28]


        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168) [tomcat-catalina-7.0.28.jar:7.0.28]


        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98) [tomcat-catalina-7.0.28.jar:7.0.28]


        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:927) [tomcat-catalina-7.0.28.jar:7.0.28]


        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118) [tomcat-catalina-7.0.28.jar:7.0.28]


        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407) [tomcat-catalina-7.0.28.jar:7.0.28]


        at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:200) [tomcat-coyote-7.0.28.jar:7.0.28]


        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:579) [tomcat-coyote-7.0.28.jar:7.0.28]


        at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310) [tomcat-coyote-7.0.28.jar:7.0.28]


        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_76]


        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_76]


        at java.lang.Thread.run(Thread.java:745) [na:1.7.0_76]


Caused by: org.apache.catalina.connector.ClientAbortException: null


        at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:388) ~[tomcat-catalina-7.0.28.jar:7.0.28]


        at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:462) ~[tomcat-coyote-7.0.28.jar:7.0.28]


        at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:334) ~[tomcat-catalina-7.0.28.jar:7.0.28]


        at org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:283) ~[tomcat-catalina-7.0.28.jar:7.0.28]


        at org.apache.catalina.connector.CoyoteOutputStream.close(CoyoteOutputStream.java:108) ~[tomcat-catalina-7.0.28.jar:7.0.28]


        at org.apache.cxf.transport.http.AbstractHTTPDestination$WrappedOutputStream.close(AbstractHTTPDestination.java:726) ~[cxf-rt-transports-http-2.7.3.jar:2.7.3]


        at org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:56) ~[cxf-api-2.7.3.jar:2.7.3]


        at org.apache.cxf.transport.http.AbstractHTTPDestination$BackChannelConduit.close(AbstractHTTPDestination.java:656) ~[cxf-rt-transports-http-2.7.3.jar:2.7.3]


        at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:62) ~[cxf-api-2.7.3.jar:2.7.3]


        ... 39 common frames omitted


Caused by: java.net.SocketException: Broken pipe


        at java.net.SocketOutputStream.socketWrite0(Native Method) ~[na:1.7.0_76]


        at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113) ~[na:1.7.0_76]


        at java.net.SocketOutputStream.write(SocketOutputStream.java:159) ~[na:1.7.0_76]


        at org.apache.coyote.ajp.AjpProcessor.output(AjpProcessor.java:295) [tomcat-coyote-7.0.28.jar:7.0.28]


        at org.apache.coyote.ajp.AbstractAjpProcessor$SocketOutputBuffer.doWrite(AbstractAjpProcessor.java:1131) ~[tomcat-coyote-7.0.28.jar:7.0.28]


        at org.apache.coyote.Response.doWrite(Response.java:504) ~[tomcat-coyote-7.0.28.jar:7.0.28]


        at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:383) ~[tomcat-catalina-7.0.28.jar:7.0.28]


        ... 47 common frames omitted


WARN 2015-03-19 15:41:45.111 [ajp-bio-8009-exec-1847] (PhaseInterceptorChain) Interceptor for {http://www.springframework.org/schema/beans}uploadService has thrown exception, unwinding now


org.apache.cxf.interceptor.Fault: Could not send Message.


        at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:64) ~[cxf-api-2.7.3.jar:2.7.3]


        at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:271) [cxf-api-2.7.3.jar:2.7.3]


        at org.apache.cxf.interceptor.AbstractFaultChainInitiatorObserver.onMessage(AbstractFaultChainInitiatorObserver.java:114) [cxf-api-2.7.3.jar:2.7.3]


        at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:331) [cxf-api-2.7.3.jar:2.7.3]


        at org.apache.cxf.interceptor.OutgoingChainInterceptor.handleMessage(OutgoingChainInterceptor.java:77) [cxf-api-2.7.3.jar:2.7.3]


        at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:271) [cxf-api-2.7.3.jar:2.7.3]


        at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121) [cxf-api-2.7.3.jar:2.7.3]


        at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:239) [cxf-rt-transports-http-2.7.3.jar:2.7.3]


        at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:218) [cxf-rt-transports-http-2.7.3.jar:2.7.3]


        at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:198) [cxf-rt-transports-http-2.7.3.jar:2.7.3]


        at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:137) [cxf-rt-transports-http-2.7.3.jar:2.7.3]


        at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:158) [cxf-rt-transports-http-2.7.3.jar:2.7.3]


        at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:243) [cxf-rt-transports-http-2.7.3.jar:2.7.3]


        at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:163) [cxf-rt-transports-http-2.7.3.jar:2.7.3]


        at javax.servlet.http.HttpServlet.service(HttpServlet.java:641) [servlet-api-3.0.jar:na]


        at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:219) [cxf-rt-transports-http-2.7.3.jar:2.7.3]


        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305) [tomcat-catalina-7.0.28.jar:7.0.28]


        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) [tomcat-catalina-7.0.28.jar:7.0.28]


        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:369) [spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE]


        at org.fcrepo.server.security.jaas.AuthFilterJAAS.doFilter(AuthFilterJAAS.java:329) [fcrepo-security-jaas-3.7.1.jar:na]


        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:381) [spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE]


        at org.springframework.security.web.access.channel.ChannelProcessingFilter.doFilter(ChannelProcessingFilter.java:109) [spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE]


        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:381) [spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE]


        at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:168) [spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE]


        at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237) [spring-web-3.0.7.RELEASE.jar:3.0.7.RELEASE]


        at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167) [spring-web-3.0.7.RELEASE.jar:3.0.7.RELEASE]


        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) [tomcat-catalina-7.0.28.jar:7.0.28]


        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) [tomcat-catalina-7.0.28.jar:7.0.28]


        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:225) [tomcat-catalina-7.0.28.jar:7.0.28]


        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123) [tomcat-catalina-7.0.28.jar:7.0.28]


        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472) [tomcat-catalina-7.0.28.jar:7.0.28]


        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168) [tomcat-catalina-7.0.28.jar:7.0.28]


        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98) [tomcat-catalina-7.0.28.jar:7.0.28]


        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:927) [tomcat-catalina-7.0.28.jar:7.0.28]


        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118) [tomcat-catalina-7.0.28.jar:7.0.28]


        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407) [tomcat-catalina-7.0.28.jar:7.0.28]


        at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:200) [tomcat-coyote-7.0.28.jar:7.0.28]


        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:579) [tomcat-coyote-7.0.28.jar:7.0.28]


        at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310) [tomcat-coyote-7.0.28.jar:7.0.28]


        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_76]


        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_76]


        at java.lang.Thread.run(Thread.java:745) [na:1.7.0_76]


Caused by: org.apache.catalina.connector.ClientAbortException: null


        at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:388) ~[tomcat-catalina-7.0.28.jar:7.0.28]


        at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:462) ~[tomcat-coyote-7.0.28.jar:7.0.28]


        at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:334) ~[tomcat-catalina-7.0.28.jar:7.0.28]


        at org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:283) ~[tomcat-catalina-7.0.28.jar:7.0.28]


        at org.apache.catalina.connector.CoyoteOutputStream.close(CoyoteOutputStream.java:108) ~[tomcat-catalina-7.0.28.jar:7.0.28]


        at org.apache.cxf.transport.http.AbstractHTTPDestination$WrappedOutputStream.close(AbstractHTTPDestination.java:726) ~[cxf-rt-transports-http-2.7.3.jar:2.7.3]


        at org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:56) ~[cxf-api-2.7.3.jar:2.7.3]


        at org.apache.cxf.transport.http.AbstractHTTPDestination$BackChannelConduit.close(AbstractHTTPDestination.java:656) ~[cxf-rt-transports-http-2.7.3.jar:2.7.3]


        at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:62) ~[cxf-api-2.7.3.jar:2.7.3]


        ... 41 common frames omitted


Caused by: java.net.SocketException: Broken pipe


        at java.net.SocketOutputStream.socketWrite0(Native Method) ~[na:1.7.0_76]


        at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113) ~[na:1.7.0_76]


        at java.net.SocketOutputStream.write(SocketOutputStream.java:159) ~[na:1.7.0_76]


        at org.apache.coyote.ajp.AjpProcessor.output(AjpProcessor.java:295) [tomcat-coyote-7.0.28.jar:7.0.28]


        at org.apache.coyote.ajp.AbstractAjpProcessor$SocketOutputBuffer.doWrite(AbstractAjpProcessor.java:1131) ~[tomcat-coyote-7.0.28.jar:7.0.28]


        at org.apache.coyote.Response.doWrite(Response.java:504) ~[tomcat-coyote-7.0.28.jar:7.0.28]


        at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:383) ~[tomcat-catalina-7.0.28.jar:7.0.28]


What is the cause for error 500 under this case?

Thank you!
Lingling









Nick Ruest

unread,
Mar 19, 2015, 3:54:33 PM3/19/15
to isla...@googlegroups.com
How big is the file in question?

Usually you get 500 errors when you've exhausted memory. At least, that
is my case with batch ingest of big old massive video and warc files.

-nruest

On 15-03-19 03:53 PM, Lingling Jiang wrote:
> Hi folks,
>
> I am back on batch ingest headache again.
>
> This time I tried to ingest video file (.mov file). I tried to convert
> .mov to .mp4 with ffmpeg command directly and it worked. So the .mov
> file is good.
> My video test directory only contains a .mov file. there is no any
> metaddata xml file which means there is no encoding issue.
>
> Commands I used are:
> |
> drush -v -d --user=admin --uri=http://ourdomain
> islandora_batch_scan_preprocess --type=directory --target=/video/testing
> --namespace=lingling1 --content_models=islandora:sp_videoCModel
> --parent=lingling1:collection
>
> drush -v -d --user=admin --uri=http://ourdomain islandora_batch_ingest
> |
>
> The first command is OK. but the second threw the following error message:
> |
>
> 03/19/1515:34:36Successfullylogged intoDrupalasadmin
> (uid=1)[1.91sec,37.96MB][bootstrap]
>
>
> 03/19/1515:34:36Foundcommand:batch-process
> (commandfile=core)[1.91sec,37.96MB][bootstrap]
>
>
> 03/19/1515:34:36Callinghook drush_core_batch_process
> [2.03sec,38.74MB][debug]
>
>
> 03/19/1515:41:10 WD islandora:Failedto ingest
> object:lingling1:1367code:*[error]*
>
>
> 03/19/1515:41:10500message:[395.56sec,41.02MB]
>
>
> 03/19/1515:41:10Exceptionoccured:Failedto ingest
> lingling1:1367.[395.56sec,*[ok]*
>
>
> 03/19/1515:41:1041.01MB]
>
>
> 03/19/1515:41:10Returnedfromhook drush_core_batch_process
> [395.6sec,42.49MB][debug]
>
> |
>
> Drupal log message only shows:
>
> Failed to ingest object: lingling1:1367
> code: 500
> message: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
>
> On Fedora.log I can see some messages around time 03/19/1515:41:10
> |
>
> uploadService has thrown exception,unwinding no
>
> org.apache.cxf.interceptor.Fault:Couldnotsend Message.
>
>
> at
> org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:64)~[cxf-api-2.7.3.jar:2.7.3]
> java.net.SocketOutputStream.write(SocketOutputStream.java:159)~[na:1.7.0_76]
>
>
> at
> org.apache.coyote.ajp.AjpProcessor.output(AjpProcessor.java:295)[tomcat-coyote-7.0.28.jar:7.0.28]
>
>
> at
> org.apache.coyote.ajp.AbstractAjpProcessor$SocketOutputBuffer.doWrite(AbstractAjpProcessor.java:1131)~[tomcat-coyote-7.0.28.jar:7.0.28]
>
>
> at
> org.apache.coyote.Response.doWrite(Response.java:504)~[tomcat-coyote-7.0.28.jar:7.0.28]
>
>
> at
> org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:383)~[tomcat-catalina-7.0.28.jar:7.0.28]
>
>
> ...47common frames omitted
>
>
> WARN
> 2015-03-1915:41:45.111[ajp-bio-8009-exec-1847](PhaseInterceptorChain)Interceptorfor{http://www.springframework.org/schema/beans}uploadService
> has thrown exception, unwinding now
>
>
> org.apache.cxf.interceptor.Fault:Couldnotsend Message.
>
>
> at
> org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:64)~[cxf-api-2.7.3.jar:2.7.3]
> java.net.SocketOutputStream.write(SocketOutputStream.java:159)~[na:1.7.0_76]
>
>
> at
> org.apache.coyote.ajp.AjpProcessor.output(AjpProcessor.java:295)[tomcat-coyote-7.0.28.jar:7.0.28]
>
>
> at
> org.apache.coyote.ajp.AbstractAjpProcessor$SocketOutputBuffer.doWrite(AbstractAjpProcessor.java:1131)~[tomcat-coyote-7.0.28.jar:7.0.28]
>
>
> at
> org.apache.coyote.Response.doWrite(Response.java:504)~[tomcat-coyote-7.0.28.jar:7.0.28]
>
>
> at
> org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:383)~[tomcat-catalina-7.0.28.jar:7.0.28]
>
> |
>
> What is the cause for error 500 under this case?
>
> Thank you!
> Lingling
>
>
>
>
>
>
>
>
>
> --
> For more information about using this group, please read our Listserv
> Guidelines: http://islandora.ca/content/welcome-islandora-listserv
> ---
> You received this message because you are subscribed to the Google
> Groups "islandora" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to islandora+...@googlegroups.com
> <mailto:islandora+...@googlegroups.com>.
> Visit this group at http://groups.google.com/group/islandora.
> For more options, visit https://groups.google.com/d/optout.

Lingling Jiang

unread,
Mar 20, 2015, 8:40:48 AM3/20/15
to isla...@googlegroups.com
Hi Nick,

I tried with 3G and 8G and both failed with error 500. My memory is 10G which should be enough. I manually tried ffmpeg command which is supposed to be used by video SP to generate .mp4 file. It took some time for converting but it worked. That is why I think the .mov file is good for ingesting.

I ingested 232M .mov file which didn't throw error 500. 

thanks
Lingling
...

Nick Ruest

unread,
Mar 20, 2015, 9:32:29 AM3/20/15
to isla...@googlegroups.com
What's your JAVA_OPS look like, and how much swap space do you have?
Also, do you have around 3x as much free disk space as the object you're
trying to ingest in your /tmp partition?

-nruest
> 03/19/1515:34:36Successfullylogged intoDrupalasadmin
> (uid=1)[1.91sec,37.96MB][bootstrap]
>
>
> 03/19/1515:34:36Foundcommand:batch-process
> (commandfile=core)[1.91sec,37.96MB][bootstrap]
>
>
> 03/19/1515:34:36Callinghook drush_core_batch_process
> [2.03sec,38.74MB][debug]
>
>
> 03/19/1515:41:10 WD islandora:Failedto ingest
> object:lingling1:1367code:*[error]*
>
>
> 03/19/1515:41:10500message:[395.56sec,41.02MB]
>
>
> 03/19/1515:41:10Exceptionoccured:Failedto ingest
> lingling1:1367.[395.56sec,*[ok]*
>
>
> 03/19/1515:41:1041.01MB]
>
>
> 03/19/1515:41:10Returnedfromhook drush_core_batch_process
> [395.6sec,42.49MB][debug]
>
> |
>
> Drupal log message only shows:
>
> Failed to ingest object: lingling1:1367
> code: 500
> message: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
>
> On Fedora.log I can see some messages around time 03/19/1515:41:10
> |
>
> uploadService has thrown exception,unwinding no
>
> org.apache.cxf.interceptor.Fault:Couldnotsend Message.
>
>
> at
> org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:64)~[cxf-api-2.7.3.jar:2.7.3]
>
>
> at
> org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:271)[cxf-api-2.7.3.jar:2.7.3]
>
>
> at
> org.apache.cxf.interceptor.OutgoingChainInterceptor.handleMessage(OutgoingChainInterceptor.java:77)[cxf-api-2.7.3.jar:2.7.3]
>
>
> at
> org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:271)[cxf-api-2.7.3.jar:2.7.3]
>
>
> at
> org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)[cxf-api-2.7.3.jar:2.7.3]
>
>
> at
> org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:239)[cxf-rt-transports-http-2.7.3.jar:2.7.3]
>
>
> at
> org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:218)[cxf-rt-transports-http-2.7.3.jar:2.7.3]
>
>
> at
> org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.
>
> |
> ...
>

Lingling Jiang

unread,
Mar 27, 2015, 11:44:19 AM3/27/15
to isla...@googlegroups.com
Hi group,

Help!!

I gave up creating local mp4 datastream for large videos now. so It only ingest the .mov file as OBJ datastream, later we can uplaod .mp4 file manually as the file size is much smaller. I now succeeded with 2.3G .mov file. 
Our drupal server has 10G memory. I increased upload_max_filesize = 10G, post_max_size = 10G, no limit on memory_limit in PHP (It's crazy, just for testing). 

So now I tried with 8G video (.mov file). On Drush ingest log I still get this vague 500 error code on Drupal server.

On our dsu-fedora server, Here is related fedora log message. Can you see any clues why I even can’t drush ingest a 8G video file? what is the maximum file size that Drush Ingest can handle? Except the PHP settings I need change, any settings on fedora side I need change?

WARN 2015-03-27 10:37:15.404 [ajp-bio-8009-exec-528] (PhaseInterceptorChain) Interceptor for {http://www.springframework.org/schema/beans}uploadService has thrown exception, unwinding now org.apache.cxf.interceptor.Fault: Could not send Message.
Caused by: org.apache.catalina.connector.ClientAbortException: null
        at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:388) ~[tomcat-catalina-7.0.28.jar:7.0.28]
        at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:462) ~[tomcat-coyote-7.0.28.jar:7.0.28]
        at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:334) ~[tomcat-catalina-7.0.28.jar:7.0.28]
        at org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:283) ~[tomcat-catalina-7.0.28.jar:7.0.28]
        at org.apache.catalina.connector.CoyoteOutputStream.close(CoyoteOutputStream.java:108) ~[tomcat-catalina-7.0.28.jar:7.0.28]
        at org.apache.cxf.transport.http.AbstractHTTPDestination$WrappedOutputStream.close(AbstractHTTPDestination.java:726) ~[cxf-rt-transports-http-2.7.3.jar:2.7.3]
        at org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:56) ~[cxf-api-2.7.3.jar:2.7.3]
        at org.apache.cxf.transport.http.AbstractHTTPDestination$BackChannelConduit.close(AbstractHTTPDestination.java:656) ~[cxf-rt-transports-http-2.7.3.jar:2.7.3]
        at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:62) ~[cxf-api-2.7.3.jar:2.7.3
INFO 2015-03-27 10:47:31.638 [ajp-bio-8009-exec-539] (DefaultManagement) Removed uploaded file '600130' because it expired.
INFO 2015-03-27 10:47:31.664 [ajp-bio-8009-exec-539] (DefaultManagement) Removed uploaded file '600129' because it expired.
INFO 2015-03-27 10:47:31.766 [ajp-bio-8009-exec-539] (DefaultManagement) Removed uploaded file '600128' because it expired.
WARN 2015-03-27 10:53:08.976 [ActiveMQ Transport: tcp:///127.0.0.1:42325@61616] (Transport) Transport Connection to: tcp://127.0.0.1:42325 failed: java.io.EOFException
WARN 2015-03-27 10:57:18.294 [http-bio-8080-exec-10] (BaseRestResource) Authorization failed; unable to fulfill REST API request
org.fcrepo.server.errors.authorization.AuthzDeniedException: 
        at org.fcrepo.server.security.impl.DefaultPolicyEnforcementPoint.enforce(DefaultPolicyEnforcementPoint.java:143) ~[fcrepo-server-3.7.1.jar:na]
        at org.fcrepo.server.security.DefaultAuthorization.enforceGetObjectProfile(DefaultAuthorization.java:1058) ~[fcrepo-server-3.7.1.jar:na]
        at org.fcrepo.server.access.DefaultAccess.getObjectProfile(DefaultAccess.java:617) ~[fcrepo-server-3.7.1.jar:na]
        at org.fcrepo.server.rest.FedoraObjectsResource.getObjectProfile(FedoraObjectsResource.java:351) ~[fcrepo-server-3.7.1.jar:na]
        at sun.reflect.GeneratedMethodAccessor66.invoke(Unknown Source) ~[na:na]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_76]
        at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_76]
        at org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:180) [cxf-api-2.7.3.jar:2.7.3]
        at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96) [cxf-api-2.7.3.jar:2.7.3]
        at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:193) [cxf-rt-frontend-jaxrs-2.7.3.jar:2.7.3]
        at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:102) [cxf-rt-frontend-jaxrs-2.7.3.jar:2.7.3]
        at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:58) [cxf-api-2.7.3.jar:2.7.3]
        at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:94) [cxf-api-2.7.3.jar:2.7.3]
        at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:271) [cxf-api-2.7.3.jar:2.7.3]
        at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121) [cxf-api-2.7.3.jar:2.7.3]
        at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:239) [cxf-rt-transports-http-2.7.3.jar:2.7.3]
        at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:218) [cxf-rt-transports-http-2.7.3.jar:2.7.3]
        at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:163) [cxf-rt-transports-http-2.7.3.jar:2.7.3]
        at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:137) [cxf-rt-transports-http-2.7.3.jar:2.7.3]
        at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:158) [cxf-rt-transports-http-2.7.3.jar:2.7.3]
        at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:243) [cxf-rt-transports-http-2.7.3.jar:2.7.3]
        at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doGet(AbstractHTTPServlet.java:168) [cxf-rt-transports-http-2.7.3.jar:2.7.3]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:621) [servlet-api-3.0.jar:na]
        at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:219) [cxf-rt-transports-http-2.7.3.jar:2.7.3]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305) [tomcat-catalina-7.0.28.jar:7.0.28]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) [tomcat-catalina-7.0.28.jar:7.0.28]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:369) [spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE]
        at org.fcrepo.server.security.jaas.AuthFilterJAAS.doFilter(AuthFilterJAAS.java:329) [fcrepo-security-jaas-3.7.1.jar:na]
org.fcrepo.server.errors.authorization.AuthzDeniedException: 
        at org.fcrepo.server.security.impl.DefaultPolicyEnforcementPoint.enforce(DefaultPolicyEnforcementPoint.java:143) ~[fcrepo-server-3.7.1.jar:na]
        at org.fcrepo.server.security.DefaultAuthorization.enforceGetObjectProfile(DefaultAuthorization.java:1058) ~[fcrepo-server-3.7.1.jar:na]
        at org.fcrepo.server.access.DefaultAccess.getObjectProfile(DefaultAccess.java:617) ~[fcrepo-server-3.7.1.jar:na]
        at org.fcrepo.server.rest.FedoraObjectsResource.getObjectProfile(FedoraObjectsResource.java:351) ~[fcrepo-server-3.7.1.jar:na]
        at sun.reflect.GeneratedMethodAccessor66.invoke(Unknown Source) ~[na:na]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_76]
        at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_76]
        at org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:180) [cxf-api-2.7.3.jar:2.7.3]
        at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96) [cxf-api-2.7.3.jar:2.7.3]
        at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:193) [cxf-rt-frontend-jaxrs-2.7.3.jar:2.7.3]
        at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:102) [cxf-rt-frontend-jaxrs-2.7.3.jar:2.7.3]
        at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:58) [cxf-api-2.7.3.jar:2.7.3]
        at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:94) [cxf-api-2.7.3.jar:2.7.3]
        at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:271) [cxf-api-2.7.3.jar:2.7.3]
        at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121) [cxf-api-2.7.3.jar:2.7.3]
        at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:239) [cxf-rt-transports-http-2.7.3.jar:2.7.3]
        at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:218) [cxf-rt-transports-http-2.7.3.jar:2.7.3]
        at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:163) [cxf-rt-transports-http-2.7.3.jar:2.7.3]
        at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:137) [cxf-rt-transports-http-2.7.3.jar:2.7.3]
        at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:158) [cxf-rt-transports-http-2.7.3.jar:2.7.3]
        at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:243) [cxf-rt-transports-http-2.7.3.jar:2.7.3]
        at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doGet(AbstractHTTPServlet.java:168) [cxf-rt-transports-http-2.7.3.jar:2.7.3]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:621) [servlet-api-3.0.jar:na]
        at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:219) [cxf-rt-transports-http-2.7.3.jar:2.7.3]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305) [tomcat-catalina-7.0.28.jar:7.0.28]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) [tomcat-catalina-7.0.28.jar:7.0.28]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:369) [spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE]
        at org.fcrepo.server.security.jaas.AuthFilterJAAS.doFilter(AuthFilterJAAS.java:329) [fcrepo-security-jaas-3.7.1.jar:na]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:381) [spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE]
        at org.springframework.security.web.access.channel.ChannelProcessingFilter.doFilter(ChannelProces


Thanks a million!!

Lingling Jiang




Phil Redmon

unread,
Mar 27, 2015, 12:01:19 PM3/27/15
to isla...@googlegroups.com
That looks a bit like you're having a permission issue. Can you ingest any videos via drush, or is it only throwing errors for large files?

--
For more information about using this group, please read our Listserv Guidelines: http://islandora.ca/content/welcome-islandora-listserv
---
You received this message because you are subscribed to the Google Groups "islandora" group.
To unsubscribe from this group and stop receiving emails from it, send an email to islandora+...@googlegroups.com.



--
phil

Lingling Jiang

unread,
Mar 27, 2015, 1:13:20 PM3/27/15
to isla...@googlegroups.com
Hi Phil,

I already ingested 232M, 1.1G, 2.3G .mov files successfully in Drush. The first two files are good with creating .mp4 datastream locally as well but the converting is really resource extensive and I could not make it for 2.3G file with that setting. So from 2.3G file I turned off "create .mp4 file locally" option in Video SP configure page in order to have it ingested. 

Meanwhile I keep drush ingesting large images and books every day but I have to make sure the size of each batch does not go over around 9G which I think is because of the memory size of our Drupal server. 
Please note, our fedora server is a separate server. Please correct me if I am wrong: Does fedora rest api have any limit on file size if it is posted from another server? I have this question because I know some of instances in the community actually have everything (Fedora/Islandora/Solr/GSearch etc.) on the same server which may not have the same issue when communicating as localhost.

Best,
Lingling

On Thursday, March 19, 2015 at 3:53:06 PM UTC-4, Lingling Jiang wrote:
...

Diego Pino

unread,
Mar 30, 2015, 12:48:02 PM3/30/15
to isla...@googlegroups.com
Hi, not an expert on this topic, but how large is your java temp directory / mount point (tomcat)? And your servers virtual memory/swap? At some point at the past i had a similar problem with a server with a very small tmp mount point and also virtually no swap. I debugged everything to an auth problem(that was the info i got from logs), but after days of no luck playing around with authN, etc., i found it was memory/disk space related after all! No google info on this, so it was a little tricky.

If swap is Ok, check for this java option/default value. I think you can override this using something like this. 
-Djava.io.tmpdir=/yourjavatemp

Maybe, just remotely maybe, your issue can be related to this.

Good luck

Diego

Lingling Jiang

unread,
Mar 31, 2015, 4:11:46 PM3/31/15
to isla...@googlegroups.com
Thanks Diego! I will try it ... I do need good luck...


On Thursday, March 19, 2015 at 3:53:06 PM UTC-4, Lingling Jiang wrote:
...

Lingling Jiang

unread,
Apr 14, 2015, 11:44:56 AM4/14/15
to isla...@googlegroups.com
Hi folks,

This is an update about ingesting large files (>4G) in Islandora which hopefully can be helpful for future improvement in islandora batch ingest framework. 

Basically I have no luck on ingesting files > 4G by using dursh commands with "--type=directory" even we allocated more resources  on both servers. 

So I switched my path to work on fedora server directly to test whether our fedora installation is not capable of handling large files (Million thanks to Donald Moses and Paul Pound at UPEI who shed some light on this approach). I tried to work on fedora server HTTP api directly with curl script to ingest large files to the repo. It totally depends on the system resources on the server itself, mainly the free disk space for our fedora installation directory as the uploaded file is kept there during ingesting. So far the largest file succeeded is over 30G. I only have about 50G space available so to be cautious I probably won't go over 30G at the moment. but I am confident it can ingest files larger than that if you have enough disk space to hold the uploaded file temporarily without impacting the overall server performance. 

It looks like I am doing some stress testing on Fedora which turns out to be good. I am not sure why Islanodra batch ingest framework seems not be able to handling large files. Our stack is a little special as we have separated drupal server and fedora server. Will this issue go away when everything is on the same server?

I also want to thank all other folks in community in trying to help me on the issue!

Cheers,
Lingling


On Thursday, March 19, 2015 at 3:53:06 PM UTC-4, Lingling Jiang wrote:
...

Diego Pino

unread,
Apr 14, 2015, 5:07:20 PM4/14/15
to isla...@googlegroups.com
Hi,
What was the largest file you could upload via drush? Also, what php version are you using? There was a bug reported long time ago, but fixed only on 2013 about php setting max post size to 2gbytes (always),

If you are on php 5.3.x (if not just forget this)
So, can you re-try using 
post_max_size = 0
upload_max_filesize = 0

on you php ini file just as test(not intending to work like this in production environment).

The other option would be to do some changes on how tuque manages the post request (HttpConnection::postRequest), we could implement a chunked upload (it's possible using curl) to manage large files, or ,upload first to fedora, get the file id back and then create the datastream using this id as file reference…

By the way, is your fedora under http or https accessible? 

Cheers

Diego

Lingling Jiang

unread,
Apr 15, 2015, 9:14:20 AM4/15/15
to isla...@googlegroups.com
Hi Diego,

Our PHP version is 5.4.x.  I increased those php limits to 8G for a 5G file but still got 500 error. Our fedora server can be accessed by both http and https. 

Cheers,
Lingling

Diego Pino

unread,
Apr 15, 2015, 12:58:06 PM4/15/15
to isla...@googlegroups.com
Sorry for the constant questions without answer, but is Islandora currently accessing Fedora via SSL or plain http? What i'm trying to do here is to mimic your problem and put some debug code in the post function to see if i can replicate(and hopefully fix) your problem, and also understand what are the limits we are encountering to at least document this.

Cheers,

Diego

Lingling Jiang

unread,
Apr 16, 2015, 9:32:06 AM4/16/15
to isla...@googlegroups.com
Hi Diego,

Our islandora is accessing fedora via plain http. When we were monitoring the ingesting process on fedora server, we can see the large file being transferred from Drupal server. It suddenly threw error like 'uploadService has thrown exception, unwinding' when the transferring was about to complete (like hitting to 5.2G when we tried with a 5.2G file). 

Another interesting thing is that I did another experiment this morning: I used curl to ingest a 8G video file to our fedora server from our Drupal server directly. It worked and no error reported!! I didn't change any php settings on our Drupal server and fedora server, simply issued curl commands to post the large file to fedora http api. 

Hope my input can help...

Best,
Lingling

Diego Pino

unread,
Apr 17, 2015, 9:03:31 AM4/17/15
to isla...@googlegroups.com
Hi, this is great help Lingling. If direct curl access (-X POST) is working, then we can discard your fedora server as a possible problem, it's on the php/drupal/islandora side. I will do some tests locally and give some php curl options a try.

Thanks!

Diego

patrick....@commonmediainc.com

unread,
Jan 12, 2018, 4:47:53 PM1/12/18
to islandora
Hi Lingling,

Can you help me understand this approach? Can you point me to any instructions or resources? "So I switched my path to work on fedora server directly to test whether our fedora installation is not capable of handling large files (Million thanks to Donald Moses and Paul Pound at UPEI who shed some light on this approach). I tried to work on fedora server HTTP api directly with curl script to ingest large files to the repo." 

I'm finding the Fedora documentation impenetrable. Has someone documented this approach for islandora users?

Thanks, Pat

dp...@metro.org

unread,
Jan 15, 2018, 11:02:37 PM1/15/18
to islandora
Hey patrick,


Try something like this (mind dump really but should work, test on some test machine first!) and replace all the things in italic with your stuff

curl -vv -i -X POST --upload-file myhugevideo.mp4 -u fedoraAdmin:yourpass "http://127.0.0.1:8080/fedora/objects/yourtest:pid/datastreams/OBJ?dsLabel=Huge%Video&versionable=true&controlGroup=M&mimeType=video/mp4"

Note: the object must exist. Also, if you have an OBJ ds already you can swap the POST for a PUT. Finally, add the checksum parameters if you want to be sure that you actually managed to upload all without any corruption.

Hope this helps, feel free to correct me (please) if i missed something

Best

Diego Pino
Metro.org

patrick....@commonmediainc.com

unread,
Jan 16, 2018, 9:02:01 AM1/16/18
to islandora
That's exactly what I needed. Thanks Diego, you rock!

dp...@metro.org

unread,
Jan 17, 2018, 8:21:33 AM1/17/18
to islandora
You are very welcome.Let us know if you had/had no issues dealing with huge files =)
Reply all
Reply to author
Forward
0 new messages