Too many open files on MacOS

[mih...@gmail.com]

Hi,

I use cassandra-unit for integration tests and I encountered the following problem:

If I have more than 10 tests then only 10 first test work and then I have 'Too many open files" exception and the rest of the tests fail.

For linux increasing number of open files (ulimit -n 15000) helps but I cannot make it work for MacOS. I followed instructions from http://apple.stackexchange.com/questions/33715/too-many-open-files and set max open files to 1000000 (ulimit -S -n returns it) but still I get "Too many open files" exception.

On Windows I haven't noticed such problems - there is other problem - cassandra-unit waits only 10 seconds (hardcoded) for cassandra daemon to start and for Windows it seems to be not enough (at least in my case I had to increase it to 15 seconds)

Has anyone had similar problem with this "Too many open files" exception on MacOS and knows a solution?

Steps which cause exception:

1. create JUnit test class

2. add CassandraCQLUnit @Rule

3. create several test methods (in my case I created 12 tests and 10 of them pass and two of them fail) - these tests do nothing (just assert true) - it probably differs for different machine settings so if 12 tests work for you try 20 or more tests to see if there are no exceptions.

4. run all tests (for me 10 first test pass and the rest fail due to "Too many open files" exception

example test Class:
...
public class CassandraTest {

    @Rule
    public CassandraCQLUnit cassandraCQLUnit = new CassandraCQLUnit(new ClassPathCQLDataSet("cql/dataset1.cql","billing_int"));

    @Test
    public void test1() {

             assert true;
    }
    
    @Test
    public void test2() {
             assert true;
    }
    ...

    ...
    @Test
    public void test12() {
             assert true;
    }
}

Kind regards,
Michal

[Jérémy SEVELLEC]

Hi, 

Thanks for this interesting feedback. 

Unfortunately, it's the first time i hear about this problem, 

Could you open an issue on github so it will be possible to track and follow this problem?

Regards,

Jérémy

--
Vous recevez ce message, car vous êtes abonné au groupe Google Groupes "cassandra-unit-users".
Pour vous désabonner de ce groupe et ne plus recevoir d'e-mails le concernant, envoyez un e-mail à l'adresse cassandra-unit-u...@googlegroups.com.
Pour obtenir davantage d'options, consultez la page https://groups.google.com/d/optout.

--
Jérémy

[Vladimir KUPTSOV]

Hi, I've just reproduce this issue on Ubuntu.
When I tried to create a keyspace from a cql-file with multiple tables creations, alters, inserts and so on, I got an exception "too many open files".
I've check
$ ulimit -Hn
4096
And I run the test again and started to check open files count for java processes
$ ps aux | grep java | grep -v grep | awk '{print $2":"}{system ("sudo ls -l /proc/"$2"/fd | wc -l")}'
11723:
568
$ ps aux | grep java | grep -v grep | awk '{print $2":"}{system ("sudo ls -l /proc/"$2"/fd | wc -l")}'
11723:
3962

After that I received an error "com.datastax.driver.core.exceptions.DriverInternalError: An unexpected error occurred server side on /127.0.0.1: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.io.FileNotFoundException: target/embeddedCassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-jb-41-Data.db (too many open files)"

As described here
http://www.datastax.com/documentation/cassandra/2.0/cassandra/install/installRecommendSettings.html Casandra really needs top open a lot of file's descriptors.

So, I've create file
$ cat /etc/security/limits.d/test_cassandra.conf
*  -  memlock  unlimited
*  -  nofile   100000

relogin and check limits
$ ulimit -Hn
100000

And run test again. Everything works fine.
But I think it isn't convinient solution.
Do you know how to reduce a count of opened files for Cassandra in the embedded mode?

Pour vous désabonner de ce groupe et ne plus recevoir d'e-mails le concernant, envoyez un e-mail à l'adresse cassandra-unit-users+unsub...@googlegroups.com.

Pour obtenir davantage d'options, consultez la page https://groups.google.com/d/optout.

--
Jérémy

[dan.ande...@gmail.com]

I am seeing a similar issue.

One developer machine is running into this "too many open files" problem with Cassandra in the unit test environment. The machine is running OS X Mavericks. Cassandra is running embedded with version 2.0.10.

On all other machines (OSX and Linux), the number of open files for Cassandra is around 500. But, this machine has 30K + open files for Cassandra alone..

Any ideas on what could be going on?

Thanks!

[Nate McCall]

Anyone experiencing this, it would be helpful to grab the output of
'lsof -i -P' in conjunction with at least a 'kill -3' on the jvm
process, at best the output from 'jvmtop' or 'jstack.'

In short any additional information on the what is being held open and
by what threads.

Another thought - with what version of JNA (or not using it?) are
folks seeing this?

>>>> cassandra-unit-u...@googlegroups.com.

>>>> Pour obtenir davantage d'options, consultez la page
>>>> https://groups.google.com/d/optout.
>>>
>>>
>>>
>>>
>>> --
>>> Jérémy
>

> --
> Vous recevez ce message, car vous êtes abonné au groupe Google Groupes
> "cassandra-unit-users".
> Pour vous désabonner de ce groupe et ne plus recevoir d'e-mails le
> concernant, envoyez un e-mail à l'adresse

> cassandra-unit-u...@googlegroups.com.

[dan.ande...@gmail.com]

Hi Nate,

I think I found the culprit. A new maven dependency was added to my build that created a runtime issue. The exception hit during compaction and open files were not closed. Since the compaction failed, another task was scheduled, running into the same problem.. repeat.

I can buy the argument that this is user error, but it feels like Cassandra should be a bit more defensive with open files in this background task. Should I file a ticket?

2014-10-06--07-05-20-466 [CompactionExecutor:10] ERROR org.apache.cassandra.service.CassandraDaemon - Exception in thread Thread[CompactionExecutor:10,1,main]

java.lang.NoSuchMethodError: com.google.common.util.concurrent.RateLimiter.acquire(I)V

at org.apache.cassandra.io.compress.CompressedThrottledReader.reBuffer(CompressedThrottledReader.java:40) ~[cassandra-all-2.0.10.jar:2.0.10]

at org.apache.cassandra.io.util.RandomAccessReader.seek(RandomAccessReader.java:283) ~[cassandra-all-2.0.10.jar:2.0.10]

at org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.computeNext(SSTableScanner.java:262) ~[cassandra-all-2.0.10.jar:2.0.10]

at org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.computeNext(SSTableScanner.java:203) ~[cassandra-all-2.0.10.jar:2.0.10]

at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) ~[guava-18.0.jar:na]

at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) ~[guava-18.0.jar:na]

at org.apache.cassandra.io.sstable.SSTableScanner.hasNext(SSTableScanner.java:183) ~[cassandra-all-2.0.10.jar:2.0.10]

at org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:144) ~[cassandra-all-2.0.10.jar:2.0.10]

at org.apache.cassandra.utils.MergeIterator$ManyToOne.<init>(MergeIterator.java:87) ~[cassandra-all-2.0.10.jar:2.0.10]

at org.apache.cassandra.utils.MergeIterator.get(MergeIterator.java:46) ~[cassandra-all-2.0.10.jar:2.0.10]

at org.apache.cassandra.db.compaction.CompactionIterable.iterator(CompactionIterable.java:47) ~[cassandra-all-2.0.10.jar:2.0.10]

at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:129) ~[cassandra-all-2.0.10.jar:2.0.10]

at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) ~[cassandra-all-2.0.10.jar:2.0.10]

at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[cassandra-all-2.0.10.jar:2.0.10]

at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60) ~[cassandra-all-2.0.10.jar:2.0.10]

at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59) ~[cassandra-all-2.0.10.jar:2.0.10]

at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:198) ~[cassandra-all-2.0.10.jar:2.0.10]

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.7.0_45]

at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_45]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_45]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_45]

at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]

Thanks!

>>>> cassandra-unit-users+unsub...@googlegroups.com.

>>>> Pour obtenir davantage d'options, consultez la page
>>>> https://groups.google.com/d/optout.
>>>
>>>
>>>
>>>
>>> --
>>> Jérémy
>
> --
> Vous recevez ce message, car vous êtes abonné au groupe Google Groupes
> "cassandra-unit-users".
> Pour vous désabonner de ce groupe et ne plus recevoir d'e-mails le
> concernant, envoyez un e-mail à l'adresse

> cassandra-unit-users+unsub...@googlegroups.com.

[Nate McCall]

Yeah - I think do open an issue explaining with the details,
particularly admitting this is not a likely run-time case.

They are dealing with all the IO and compression errors that might
crop up, but I agree that a general cow-catcher is a safe bet when
file handles are open in the context of that seek method
(https://github.com/apache/cassandra/blob/cassandra-2.1/src/java/org/apache/cassandra/io/util/RandomAccessReader.java#L271-L284).

I would suggest knocking out the patch as well so it's there if they
decide to do it.

Actually, In looking at that code a bit more, that hierarchy is a
little wonky in how the exceptions are created inconsistently by
subclasses. You could make a go at cleaning that up somewhat (adding a
throws clause to reBuffer, maybe?). Up to you.

Thanks for replying with the solution either way.

>> >>>> cassandra-unit-u...@googlegroups.com.

>> >>>> Pour obtenir davantage d'options, consultez la page
>> >>>> https://groups.google.com/d/optout.
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> Jérémy
>> >
>> > --
>> > Vous recevez ce message, car vous êtes abonné au groupe Google Groupes
>> > "cassandra-unit-users".
>> > Pour vous désabonner de ce groupe et ne plus recevoir d'e-mails le
>> > concernant, envoyez un e-mail à l'adresse

>> > cassandra-unit-u...@googlegroups.com.

>> > Pour obtenir davantage d'options, consultez la page
>> > https://groups.google.com/d/optout.
>
> --

> Vous recevez ce message, car vous êtes abonné au groupe Google Groupes
> "cassandra-unit-users".
> Pour vous désabonner de ce groupe et ne plus recevoir d'e-mails le
> concernant, envoyez un e-mail à l'adresse

> cassandra-unit-u...@googlegroups.com.

[dan.ande...@gmail.com]

I will open a ticket and see if there is something I can patch (still getting up to speed with c* internals). Thanks for the help Nate.

For anybody that runs into this issue, my problem was a little more substantive than conflicting libraries. Specifically, I ran into https://issues.apache.org/jira/browse/CASSANDRA-6639 using Cassandra 2.0.10. When another developer added a dependency that pulled in guava 16+, the problem started.

Solutions: Upgrade to 2.1.0 or use an external cassandra instance (like ccm) or use guava 15 in your project.

Thanks,

Dan

>> >>>> cassandra-unit-users+unsub...@googlegroups.com.

>> >>>> Pour obtenir davantage d'options, consultez la page
>> >>>> https://groups.google.com/d/optout.
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> Jérémy
>> >
>> > --
>> > Vous recevez ce message, car vous êtes abonné au groupe Google Groupes
>> > "cassandra-unit-users".
>> > Pour vous désabonner de ce groupe et ne plus recevoir d'e-mails le
>> > concernant, envoyez un e-mail à l'adresse

>> > cassandra-unit-users+unsub...@googlegroups.com.

>> > Pour obtenir davantage d'options, consultez la page
>> > https://groups.google.com/d/optout.
>
> --
> Vous recevez ce message, car vous êtes abonné au groupe Google Groupes
> "cassandra-unit-users".
> Pour vous désabonner de ce groupe et ne plus recevoir d'e-mails le
> concernant, envoyez un e-mail à l'adresse

> cassandra-unit-users+unsub...@googlegroups.com.

[david carter]

Hey,

I'm also seeing this issue. None of the solutions posted seem to help, I'm using cassandra 2.1.0 and cassandra unit 2.0.2.1 (I have tried various versions of both) and after a few tests running I get the 'too many files open' error. 

@dan, which maven dependency was added that was causing the conflict for you?

Did a ticket get created in the end regarding this? Anyone else find any other solutions?