SocketTimeoutException: Read timed out error

[Sandeep Sahu]

Good morning Everyone,

We are in the process of upgrading the i2b2 software. We ran the upgrade scripts for the database and installed the v1.8 build on a new machine. We are getting 'SocketTimeoutException: Read timed out' in the server logs. The timeout error comes after 60 seconds. Please note this error comes with our database only and not with the i2b2 demo database created with scripts from the i2b2 website.

 

Please let us know how we can increase the relevant timeout value or resolve this issue. Also, please point us to other timeout configuration for the i2b2 application.

 

Below is the stacktrace from logs:

i2b2-docker-build-server-1     | 16:29:22,046 INFO  [org.apache.axis2.transport.http.HTTPSender] (default task-1) Unable to send to url[http://localhost:9090/i2b2/services/PMService/getServices]: java.net.SocketTimeoutException: Read timed out i2b2-docker-build-server-1     | at java.net.SocketInputStream.socketRead0(Native Method) i2b2-docker-build-server-1     | at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) i2b2-docker-build-server-1     | at java.net.SocketInputStream.read(SocketInputStream.java:171) i2b2-docker-build-server-1     | at java.net.SocketInputStream.read(SocketInputStream.java:141) i2b2-docker-build-server-1     | at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137) i2b2-docker-build-server-1     | at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153) i2b2-docker-build-server-1     | at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280) i2b2-docker-build-server-1     | at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) i2b2-docker-build-server-1     | at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) i2b2-docker-build-server-1     | at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259) i2b2-docker-build-server-1     | at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163) i2b2-docker-build-server-1     | at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157) i2b2-docker-build-server-1     | at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273) i2b2-docker-build-server-1     | at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125) i2b2-docker-build-server-1     | at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) i2b2-docker-build-server-1     | at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) i2b2-docker-build-server-1     | at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) i2b2-docker-build-server-1     | at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) i2b2-docker-build-server-1     | at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) i2b2-docker-build-server-1     | at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) i2b2-docker-build-server-1     | at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) i2b2-docker-build-server-1     | at org.apache.axis2.transport.http.impl.httpclient4.RequestImpl.execute(RequestImpl.java:210) i2b2-docker-build-server-1     | at org.apache.axis2.transport.http.HTTPSender.send(HTTPSender.java:193) i2b2-docker-build-server-1     | at org.apache.axis2.transport.http.AbstractHTTPTransportSender.writeMessageWithCommons(AbstractHTTPTransportSender.java:385) i2b2-docker-build-server-1     | at org.apache.axis2.transport.http.AbstractHTTPTransportSender.invoke(AbstractHTTPTransportSender.java:213) i2b2-docker-build-server-1     | at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:431) i2b2-docker-build-server-1     | at org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java:399) i2b2-docker-build-server-1     | at org.apache.axis2.description.OutInAxisOperationClient.executeImpl(OutInAxisOperation.java:225) i2b2-docker-build-server-1     | at org.apache.axis2.client.OperationClient.execute(OperationClient.java:150) i2b2-docker-build-server-1     | at org.apache.axis2.client.ServiceClient.sendReceive(ServiceClient.java:533) i2b2-docker-build-server-1     | at org.apache.axis2.client.ServiceClient.sendReceive(ServiceClient.java:509) i2b2-docker-build-server-1     | at edu.harvard.i2b2.common.util.axis2.ServiceClient.sendREST(ServiceClient.java:110) i2b2-docker-build-server-1     | at edu.harvard.i2b2.common.util.axis2.ServiceClient.sendREST(ServiceClient.java:68) i2b2-docker-build-server-1     | at edu.harvard.i2b2.pm.ws.PMServiceDriver.getRoles(PMServiceDriver.java:88) i2b2-docker-build-server-1     | at edu.harvard.i2b2.ontology.delegate.RequestHandler.getRoleInfo(RequestHandler.java:76) i2b2-docker-build-server-1     | at edu.harvard.i2b2.ontology.delegate.GetTermInfoHandler.<init>(GetTermInfoHandler.java:50) i2b2-docker-build-server-1     | at edu.harvard.i2b2.ontology.ws.OntologyService.getTermInfo(OntologyService.java:383) i2b2-docker-build-server-1     | at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) i2b2-docker-build-server-1     | at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

Thank you.

Best,
Sandeep

[Jeff Klann]

Hi, could you provide some more information?

* It looks like you are using a Docker implementation of i2b2. Could you provide some information on this and where you obtained the image? We have not released an official 1.8.0 Docker image.

* You say it works with the scripts created from the i2b2 website. What scripts were used to create this non-working i2b2?

You might simply have the cell URL or port configured incorrectly (Cell URL Properties) but it is difficult to be sure without more information.

Thanks,

Jeff

[Sandeep Sahu]

Thank you Jeff for the response. I have provided more information about our environment below in the email. Please let us know if you need more information.

We got the 'SocketTimeoutException: Read timed out' error even in a regular (non-docker) environment. The error log is at the end of this email.

We have an existing v1.6 i2b2 installation. Upgrade scripts for the database from GitHub repo (https://github.com/i2b2/i2b2-data/tree/master/edu.harvard.i2b2.data/Release_1-8/Upgrade) were run on the existing database. Binary distribution for the i2b2 application software (https://www.i2b2.org/software/download.html?d=458) was installed on a new Linux server.

Also, please note that the SocketTimeoutException only comes for certain operations like running a query in i2b2. We can log in to the application and browse the Ontology hierarchy. Also, following is the response when I tried to access the URL http://localhost:9090/i2b2/services/OntologyService/getTermInfo in a browser:

<soapenv:Reason xmlns:soapenv="http://www.w3.org/2003/05/soap-envelope">

<soapenv:Text xml:lang="en-US">Umarshaller error: Error during unmarshall <nsk6ANG:getTermInfo xmlns:nsk6ANG="http://ws.ontology.i2b2.harvard.edu"/></soapenv:Text>

</soapenv:Reason>

Moreover, operation OntologyService/getTermInfo is shown in the http://localhost:9090/i2b2/services/listServices.

Error in the server log:

2024-01-11 16:41:27,409 INFO  [org.apache.axis2.transport.http.HTTPSender] (Thread-156) Unable to send to url[http://localhost:9090/i2b2/services/OntologyService/getTermInfo]: java.net.SocketTimeoutException: Read timed out

        at java.net.SocketInputStream.socketRead0(Native Method)

        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)

        at java.net.SocketInputStream.read(SocketInputStream.java:171)

        at java.net.SocketInputStream.read(SocketInputStream.java:141)

        at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)

        at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)

        at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280)

        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)

        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)

        at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)

        at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)

        at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157)

        at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)

        at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)

        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)

        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)

        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)

        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)

        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)

        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72)

        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)

        at org.apache.axis2.transport.http.impl.httpclient4.RequestImpl.execute(RequestImpl.java:210)

        at org.apache.axis2.transport.http.HTTPSender.send(HTTPSender.java:193)

        at org.apache.axis2.transport.http.AbstractHTTPTransportSender.writeMessageWithCommons(AbstractHTTPTransportSender.java:385)

        at org.apache.axis2.transport.http.AbstractHTTPTransportSender.invoke(AbstractHTTPTransportSender.java:213)

        at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:431)

        at org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java:399)

        at org.apache.axis2.description.OutInAxisOperationClient.executeImpl(OutInAxisOperation.java:225)

        at org.apache.axis2.client.OperationClient.execute(OperationClient.java:150)

        at org.apache.axis2.client.ServiceClient.sendReceive(ServiceClient.java:533)

        at org.apache.axis2.client.ServiceClient.sendReceive(ServiceClient.java:509)

        at edu.harvard.i2b2.common.util.axis2.ServiceClient.sendREST(ServiceClient.java:110)

        at edu.harvard.i2b2.crc.delegate.ontology.CallOntologyUtil.callOntology(CallOntologyUtil.java:88)

Best,
Sandeep

[Sandeep Sahu]

Hi Jeff,

We debugged the error and found from the stacktrace that the exception is coming from the sendREST() method of ServiceClient class. To resolve the exception, we are considering setting a higher value timeout in this method. Please provide your thoughts on this approach.

Thanks,
Sandeep

[Md.Saber Hossain]

Hi Sandeep,

If API endpoints are accessible via a browser but not from the services, you are probably using docker bridge network. In this case, browsers can access the API endpoint by using 'localhost'. Containers within the same bridge network should use the 'container_name' instead of using 'localhost'.

Browser: http://localhost:9090/i2b2/services/listServices.

Wildfly: http://<container_name>:9090/i2b2/services/listServices.

Review your PM_CELL_DATA table's URL column and replace 'localhost:port' with 'container_name:port'.

Thanks,

Saber

--
You received this message because you are subscribed to the Google Groups "i2b2 Install Help" group.
To unsubscribe from this group and stop receiving emails from it, send an email to i2b2-install-h...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/i2b2-install-help/aae4da5b-ce4a-4548-a325-8a2622577ac0n%40googlegroups.com.

[Sandeep Sahu]

Thank you Saber for the reply. 

This issue is not limited to the docker environment. We are getting the same error in a regular (non-docker) environment, as mentioned in one of my earlier emails. Also, this issue does not show up with the i2b2 demodata database setup using the database scripts from github. We had an existing v1.6 i2b2 database containing our organization data that was upgraded to v1.8. The issue comes only with our org database; not with i2b2 demodata database.

Best,

Sandeep

[Md.Saber Hossain]

Hi Sandeep, All I wanted to say, the cell urls stored in the i2b2_data_config.json and PM_CELL_DATA table are not correct.

Here are the steps I would follow to fix the issue:

 

Scenario 1:

let's assume your web client can communicate with the wildfly server via localhost and you do not have reverse proxy setup in your apache server,

- Check on which port the wildfly server is listening to (default is 8080). Based on your server log, the wildfly server should listen on port 9090.

Now,

- Revise the "urlCellPM" value in the i2b2_data_config.json file in the webclient. It should be set "urlCellPM": "http://127.0.0.1:9090/i2b2/services/PMService/"

- Revise the "URL" column value for all cells in the PM_CELL_DATA table. All should have ip and port set to e.g."http://127.0.0.1:9090/i2b2/services/QueryToolService/"

Scenario 2:

If you are using a proxy server,
- Add a virtual host in you apache server

<VirtualHost 127.0.0.1:80>

ProxyRequests Off

ProxyPreserveHost Off

<Location /i2b2/services/>

Require ip 127.0.0.1

ProxyPass http://localhost:9090/i2b2/services/

</Location>

</VirtualHost>

- Revise the "urlCellPM" value in the i2b2_data_config.json file in the webclient. It should be set "urlCellPM": "http://127.0.0.1/i2b2/services/PMService/"

- Revise the "URL"  value for all cells in the PM_CELL_DATA table. All should have ip and port set to "http://127.0.0.1/i2b2/services/QueryToolService/"

Good Luck,
Saber

To view this discussion on the web visit https://groups.google.com/d/msgid/i2b2-install-help/43b40fdd-2d06-4451-8241-a14554f10393n%40googlegroups.com.

[Sandeep Sahu]

Hi Saber,

We had tried both solutions/scenarios mentioned in your email but are still getting the same SocketTimeoutException error. Just so you know, we could login to the application and browse the hierarchy/ontology. The error comes after we create and run a query. 

Thanks,
Sandeep

[Sandeep Sahu]

Hi Jeff and Saber,

We are happy to report that this issue of SocketTimeoutException has been resolved. We compared the relevant i2b2 server code between v1.6 and v1.8 and found that the following lines that explicitly set the timeout were present in v1.6 but not in v1.8. Adding them to the sendREST() method of the class ServiceClient fixed the issue. We are now able to execute the query and see the results.

options.setProperty(HTTPConstants.SO_TIMEOUT,new Integer(125000));

options.setProperty(HTTPConstants.CONNECTION_TIMEOUT,new Integer(125000));

We are wondering if the above lines of code could be added to the i2b2 repository.

Thanks,

Sandeep