Problem with 'Too many open files'

86 views
Skip to first unread message

boo...@gmail.com

unread,
Jun 6, 2018, 8:00:20 AM6/6/18
to Dataverse Users Community
We are running version 4.8.6. for a week now and this morning we had a problem that was new to us: 

org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http://localhost:8983/solr java.net.SocketException java.net.SocketException: Too many open files ]]

And Solr request are done for almost everything in the UI, therefore our Dataverse wasn't operational anymore. I had to restart glassfish to get things working. 

Somewhere in the code the sockets are not being closed. 
Anyone experienced similar problems with 4.8.6?

Don Sizemore

unread,
Jun 6, 2018, 8:34:05 AM6/6/18
to dataverse...@googlegroups.com
Hello,

On the Glassfish side: Odum has submitted two pull requests to address file descriptor leaks in Dataverse 4.8.6 (and have a patched warfile we're happy to share with you).

and

Your problems may be solr-specific; there is updated documentation on setting ulimit values at

I hope this helps,
Donald


--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.
To post to this group, send email to dataverse-community@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/9903d7a6-0749-4908-9e00-81f94a8032c6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

boo...@gmail.com

unread,
Jun 7, 2018, 4:36:03 AM6/7/18
to Dataverse Users Community
Thanks, 

The file descriptor leaks are a very likely cause, because after upgrading I did a 'reExportAll' and these leaks are in the export code. 
Because we run a fork of I should incorporate these fixes in our code, or just wait for the next upgrade and incorporate it then.  

Op woensdag 6 juni 2018 14:34:05 UTC+2 schreef Donald Sizemore II:
Hello,

On the Glassfish side: Odum has submitted two pull requests to address file descriptor leaks in Dataverse 4.8.6 (and have a patched warfile we're happy to share with you).

and

Your problems may be solr-specific; there is updated documentation on setting ulimit values at

I hope this helps,
Donald

On Wed, Jun 6, 2018 at 8:00 AM, <boo...@gmail.com> wrote:
We are running version 4.8.6. for a week now and this morning we had a problem that was new to us: 

org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http://localhost:8983/solr java.net.SocketException java.net.SocketException: Too many open files ]]

And Solr request are done for almost everything in the UI, therefore our Dataverse wasn't operational anymore. I had to restart glassfish to get things working. 

Somewhere in the code the sockets are not being closed. 
Anyone experienced similar problems with 4.8.6?

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.
To post to this group, send email to dataverse...@googlegroups.com.

Philip Durbin

unread,
Jun 7, 2018, 7:03:38 AM6/7/18
to dataverse...@googlegroups.com
I don't mean to steal anyone's thunder but since Don is on this thread I'd feel remiss if I didn't mention that Dataverse 4.9 is out and he's already playing around with it: http://irclog.iq.harvard.edu/dataverse/2018-06-06#i_68444

For what it's worth, it sounds to me like a problem specific to Solr and I hope the ulimit settings mentioned above would help but please note that Dataverse 4.9 requires a much newer version of Solr (7.3.1 instead of 4.6.0): http://guides.dataverse.org/en/4.9/installation/prerequisites.html#solr


On Thu, Jun 7, 2018 at 4:36 AM, <boo...@gmail.com> wrote:
Thanks, 

The file descriptor leaks are a very likely cause, because after upgrading I did a 'reExportAll' and these leaks are in the export code. 
Because we run a fork of I should incorporate these fixes in our code, or just wait for the next upgrade and incorporate it then.  

Op woensdag 6 juni 2018 14:34:05 UTC+2 schreef Donald Sizemore II:
Hello,

On the Glassfish side: Odum has submitted two pull requests to address file descriptor leaks in Dataverse 4.8.6 (and have a patched warfile we're happy to share with you).

and

Your problems may be solr-specific; there is updated documentation on setting ulimit values at

I hope this helps,
Donald

On Wed, Jun 6, 2018 at 8:00 AM, <boo...@gmail.com> wrote:
We are running version 4.8.6. for a week now and this morning we had a problem that was new to us: 

org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http://localhost:8983/solr java.net.SocketException java.net.SocketException: Too many open files ]]

And Solr request are done for almost everything in the UI, therefore our Dataverse wasn't operational anymore. I had to restart glassfish to get things working. 

Somewhere in the code the sockets are not being closed. 
Anyone experienced similar problems with 4.8.6?

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsubscribe...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.
To post to this group, send email to dataverse-community@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/8935c5c6-53e0-41e9-ba2b-17cbf90f106d%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--

Don Sizemore

unread,
Jun 7, 2018, 9:44:53 AM6/7/18
to dataverse...@googlegroups.com
"The file descriptor leaks are a very likely cause, because after upgrading I did a 'reExportAll' and these leaks are in the export code. 
Because we run a fork of I should incorporate these fixes in our code, or just wait for the next upgrade and incorporate it then."

The fixes were fairly surgical; incorporating them into a patched warfile of your fork might be well worth the trouble.

Note that we didn't get them all but in production Odum's patched warfile seems to top out around 12,000 open file descriptors. Nearly all are named export_*.

The greater fix is to up your ulimit values for glassfish and solr, then kernel security releases should clear out your descriptors often enough =)

I hope this helps,
Don

Reply all
Reply to author
Forward
0 new messages