[fcrepo-user] Getting a 'I/O Error: Stream closed" error on heavy load with REST API

[Ludovic Deravet]

Hello,

We are having a strange issue (Stream closed) when working for the REST API of Fedora:

2011-03-07 23:23:15,864 ERROR [DISSEMINATION] (http-127.0.0.1-8080-4) Exception while processing dissemination service request: uri=/resource/709d8fa2-592a-4d3e-a86d-c1279741a41c;headers=[host=localhost:8080, user-agent=Mozilla/5.0 (Windows; U; Windows NT 6.1; fr; rv:1.9.2.6) Gecko/20100625 Firefox/3.6.6, accept-language=fr,fr-fr;q=0.8,en-us;q=0.5,en;q=0.3, accept-encoding=gzip,deflate, accept-charset=ISO-8859-1,utf-8;q=0.7,*;q=0.7, keep-alive=115, connection=keep-alive, cookie=GUEST_LANGUAGE_ID=en_US; COOKIE_SUPPORT=true; LOGIN=746573744074656e666f7263652e636f6d; SCREEN_NAME=6f6672492f4353617541493d, accept=application/xml;notice=tree, cache-control=max-age=0]. [ps 'cellar' - psid '709d8fa2-592a-4d3e-a86d-c1279741a41c']: I/O error: Stream closed; nested exception is java.io.IOException: Stream closed

org.springframework.web.client.ResourceAccessException: I/O error: Stream closed; nested exception is java.io.IOException: Stream closed

                at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:453)

                at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:401)

                at org.springframework.web.client.RestTemplate.getForObject(RestTemplate.java:199)

The Springframework class RestTemplate is call by our code and looks like this:

restTemplate.getForObject(fedoraUrl, String.class, urlVariables);

It runs fine under normal load but start to appears randomly under heavy load. Someone has already got such a problem and can give me some hints about how to solve it? Thanks.

Kind regards

--
Ludovic

[Scott Prater]

Ludovic,

What version of Fedora are you on? It looks like you may have run into
this bug:

https://jira.duraspace.org/browse/FCREPO-774

... which was fixed in version 3.4.2.

-- Scott


Ludovic Deravet wrote:
> Hello,
>
> We are having a strange issue (*Stream closed*) when working for the REST

> ------------------------------------------------------------------------
>
> ------------------------------------------------------------------------------
> What You Don't Know About Data Connectivity CAN Hurt You
> This paper provides an overview of data connectivity, details
> its effect on application quality, and explores various alternative
> solutions. http://p.sf.net/sfu/progress-d2d
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Fedora-commons-users mailing list
> Fedora-com...@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/fedora-commons-users


--
Scott Prater
Library, Instructional, and Research Applications (LIRA)
Division of Information Technology (DoIT)
University of Wisconsin - Madison
pra...@wisc.edu

[Ludovic Deravet]

Hello Scott,

We have migrated to 3.4.2 a week ago so we are getting this problem on this release.

Kind regards,

Ludovic

Ludovic

[Steve Bayliss]

Hi Ludovic

 

It would be great if you're able to reproduce this and supply the Fedora server log file so we can try and track down exactly where this is happening in the code.

 

Regards

Steve

[Asger Askov Blekinge]

Hi

I was the guy that solved the previous bugs. Give me the log files, and I will solve this one for you.

Regards

[Ludovic Deravet]

Hello,

Thank you for your quick replies and sorry for being late to respond to them but I wanted to perform various intensive tests before coming back with eventual log files.

In our Java application, we are using the HTTPClient library to generate HTTP calls on the Fedora's REST API (setup using SpringFramework 3+) that has the following default configuration:

41   DEBUG org.apache.commons.httpclient.params.DefaultHttpParams  - Set parameter http.useragent = Jakarta Commons-HttpClient/3.1

43   DEBUG org.apache.commons.httpclient.params.DefaultHttpParams  - Set parameter http.protocol.version = HTTP/1.1

44   DEBUG org.apache.commons.httpclient.params.DefaultHttpParams  - Set parameter http.connection-manager.class = class org.apache.commons.httpclient.SimpleHttpConnectionManager

44   DEBUG org.apache.commons.httpclient.params.DefaultHttpParams  - Set parameter http.protocol.cookie-policy = default

44   DEBUG org.apache.commons.httpclient.params.DefaultHttpParams  - Set parameter http.protocol.element-charset = US-ASCII

44   DEBUG org.apache.commons.httpclient.params.DefaultHttpParams  - Set parameter http.protocol.content-charset = ISO-8859-1

(the class highlighted in red was the cause of the problem in multi-threads context).

In this default configuration, the HTTPConnection instance was shared among all threads and was generating the following kind of errors (connection is not open, attempted read on closed stream, no Content-Type found) when the application is under heavy load.

Indeed, in our situation, we are ingesting hundred of METS envelopes (and so creating thousands of digital objects with access to datastreams through the Fedora's REST API) in few minutes. It was only in that case that the connections to the Fedora server were failing randomly.

So in summary, with Fedora 3.4.2, everything works fine when using MultiThreadedHttpConnectionManager (of HttpClient) when calling the REST API.

Sorry for the misleading message that might have raised doubts about the possible errors under heavy load. 

Thanks again for your support.

Kind regards,

Ludovic

------------------------------------------------------------------------------
Colocation vs. Managed Hosting
A question and answer guide to determining the best fit
for your organization - today and in the future.
http://p.sf.net/sfu/internap-sfd2d

_______________________________________________
Fedora-commons-users mailing list
Fedora-com...@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/fedora-commons-users

--
Ludovic

[Steve Bayliss]

Hi Ludovic

 

No problems, thanks for letting us know; and glad to see that you are putting the server under heavy load with some tests, do let us know if you encounter any other problems.