[Fedora-commons-users] Large datastream ingest issue: Bad request; unable to fulfill REST API request java.lang.NumberFormatException: For input string: "2326355355"

[West, Graeme]

Hello (again) Fedora people :)

I'm planning a repository migration which will involve storing large video files in Fedora. As such I've been asking around as to the potential issues and gotchas involved.

I understand that Fedora 3.3 contains a bug related to datastream checksumming, which I would welcome any more information about. However, I encountered another problem while ingesting a 2.2GB file into Fedora as a Managed Content datastream via the Flash/Flex admin interface on my local machine.

The full error traceback is below, but the gist of it is:

> WARN 2010-05-24 15:28:59.282 [http-8080-2] (DatastreamResource) Bad request; unable to fulfill REST API request
> java.lang.NumberFormatException: For input string: "2326355355"


"2326355355" happens to be, more or less, the size of the file involved (my Mac reports that the file is 2,326,355,123 bytes, but I suppose Fedora may also be counting the XML of the datastream too).

I therefore wondered whether there's an issue simply with Tomcat's memory allocation or whether this is something more fundamental. It could be an offshoot of the checksumming bug, though the problem also occurs when I specify that the datastream should not be checksummed in any way.

I'm running Mac OS X 10.6.3, and using the latest Apple Java JDK on a Core 2 Duo (64 bit) machine with 2GB RAM. The Java process seems to be running in 64-bit mode.

Thanks in advance for any advice.

Regards,


Graeme West

Digital Repository Developer
Glasgow Caledonian University
graem...@gcu.ac.uk



> WARN 2010-05-24 15:28:59.282 [http-8080-2] (DatastreamResource) Bad request; unable to fulfill REST API request
> java.lang.NumberFormatException: For input string: "2326355355"
> at java.lang.NumberFormatException.forInputString(NumberFormatException.java:48)
> at java.lang.Integer.parseInt(Integer.java:461)
> at java.lang.Integer.parseInt(Integer.java:499)
> at fedora.server.rest.RestUtil.getRequestContent(RestUtil.java:107)
> at fedora.server.rest.DatastreamResource.addOrUpdateDatastream(DatastreamResource.java:372)
> at fedora.server.rest.DatastreamResource.addDatastream(DatastreamResource.java:321)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at com.sun.jersey.server.impl.model.method.dispatch.EntityParamDispatchProvider$ResponseOutInvoker._dispatch(EntityParamDispatchProvider.java:157)
> at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:67)
> at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:124)
> at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:111)
> at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:71)
> at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:111)
> at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:63)
> at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:555)
> at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:514)
> at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:505)
> at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:359)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> at fedora.server.security.servletfilters.FilterRestApiFlash.doFilter(FilterRestApiFlash.java:65)
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> at fedora.server.security.servletfilters.FilterSetup.doFilter(FilterSetup.java:234)
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> at fedora.server.security.servletfilters.FilterSetup.doFilter(FilterSetup.java:234)
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> at fedora.server.security.servletfilters.FilterSetup.doFilter(FilterSetup.java:234)
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> at fedora.server.security.servletfilters.FilterSetup.doFilter(FilterSetup.java:234)
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> at fedora.server.security.servletfilters.FilterSetup.doFilter(FilterSetup.java:234)
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
> at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
> at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:525)
> at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
> at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
> at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
> at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
> at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:849)
> at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
> at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:454)
> at java.lang.Thread.run(Thread.java:637)

Glasgow Caledonian University is a registered Scottish charity, number SC021474

Winner: Times Higher Education's Widening Participation Initiative of the Year 2009 and Herald Society's Education Initiative of the Year 2009
http://www.gcu.ac.uk/newsevents/news/bycategory/theuniversity/1/name,6219,en.html

[Jason Nugent]

Hi Graeme,

I suspect that the problem lies here:

>
>> WARN 2010-05-24 15:28:59.282 [http-8080-2] (DatastreamResource) Bad request; unable to fulfill REST API request
>> java.lang.NumberFormatException: For input string: "2326355355"
>> at java.lang.NumberFormatException.forInputString(NumberFormatException.java:48)
>> at java.lang.Integer.parseInt(Integer.java:461)

The maximum size of an integer in Java is 2^31-1, or 2147483648. That
is a smaller value than the size here. To confirm, I wrote a short Java
app that attempted to convert a String to an int using
Integer.parseInt() and encountered the same error if I made the number
too large. I am including the code here:

public class Test {


public static void main(String[] args) {

try {
int i = Integer.parseInt("3922823822822");
System.out.println(i);
} catch (NumberFormatException e) {
System.out.println("Exception caught: " + e.toString());
}
}
}


compile with "javac Test.java" and then run with "java Test". If you
make the number in the parseInt call nice and small, it works without
problem.

I believe the fix here is to use a long datatype instead of an int, in
the Fedora code. This may impact database storage requirements and
other methods in the code that depend on being handed an integer,
though, and those would need to be looked at as well.

Jason

--
Jason Nugent
Systems Programmer/Database Developer
Electronic Text Centre
University of New Brunswick
jnu...@unb.ca
(506) 447 3177

[Steve Bayliss]

Hi Graeme

Jason's correct, this should be long rather than int, otherwise we have a
2GB file size restriction. I have raised
http://www.fedora-commons.org/jira/browse/FCREPO-704 for this.

Regarding the bug relating to datastream checksumming, I think this is
probably the issue in question:
http://www.fedora-commons.org/jira/browse/FCREPO-696

Regards
Steve

----------------------------------------------------------------------------
--

_______________________________________________
Fedora-commons-users mailing list
Fedora-com...@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/fedora-commons-users

[West, Graeme]

Hi Steve, Jason,
Thanks to both for the very helpful answers. They certainly explain the odd behaviour I was seeing.

Steve has suggested a quick fix which can be applied to the trunk version:

> org.fcrepo.server.rest.RestUtil.
> Line 108 - change Integer.ParseInt to Long.parseLong
> Line 119 - change the type of size from int to long
> line 138 - change the return type of method getSize() from int to long

I will have a go at making a patch but I'd be grateful if others in the community would consider looking at this issue too. Hopefully the necessary fixes will make it into 3.4 :)

Thanks, and keep up the great work,

Graeme

> Email has been scanned for viruses by Altman Technologies' email management service - www.altman.co.uk/emailsystems