Stopping embedded cassandra service

[Matthias Keller]

Hi

I've just read
http://prettyprint.me/2010/02/14/running-cassandra-as-an-embedded-service/
and got it to work nicely, except that for some unknown reason, stopping
the server always takes ages - or to be more precise: exactly 60 seconds...
Is there some configuration value somewhere I could tweak to change
that? I don't know what cassandra is doing after I did a
cassandraDaemon.deactivate(); but it hangs for exactly 60 seconds in
CassandraDaemon#stopServer() in the server.join() call...
I plan to integrate the embedded cassandra into my integration test to
test some interfaces I wrote, but waiting 60 seconds for cassandra to
vanish is just way too long...

thanks for your input

Matt

[Matthias Keller]

Ah I think I found it, I didn't shutdown my hector cluster instance
which seems to have kept the cassandra instance alive with one of its
helper threads...
Calling HFactory.shutdownCluster(cluster); plus
cluster.getConnectionManager().shutdown(); (because the
cluster.getName() vs cluster.getClusterName() bugfix isn't released yet
:-( )

Matt

[Patricio Echagüe]

you need to fork a JVM per test class.

Look at Hector pom for that.

<build>

  <plugins>

      <!-- Tests execution -->

      <plugin>

        <groupId>org.apache.maven.plugins</groupId>

        <artifactId>maven-surefire-plugin</artifactId>

        <configuration>

          <forkMode>always</forkMode>

          <!-- <argLine>-Xmx512M -Xms512M</argLine> -->

        </configuration>

      </plugin>

      ....

  </plugins>

</build>

[Jonathan Shook]

Hector devs,
Can these non-app threads be set as daemon threads? Does this cause
any other issues?
It might simplify some things.

[Patricio Echagüe]

Are you talking about Cassandra threads ?

[Jonathan Shook]

Actually, I was just referring to the thread pool threads.
"pool-1-thread-1"

2011/3/16 Patricio Echagüe <patr...@gmail.com>:

[Patricio Echagüe]

one of the hector threads is the retry mechanisms. I still don't know if you refer to it or Cassandra threads (which are quite a few).

Could you please narrow down what threads those are?

2011/3/16 Jonathan Shook <jsh...@gmail.com>

[Matthias Keller]

Hmm well I was too early with that... While the
cassandraDaemon#stopServer() now returns immediately, its threads still
live along:

org.testng.remote.RemoteTestNG at localhost:2725
Thread [main] (Running)
Thread [ReaderThread] (Running)
Daemon Thread [FileWatchdog] (Running)
Thread [FlushSorter:1] (Running)
Thread [MemtablePostFlusher:1] (Running)
Thread [FlushWriter:1] (Running)
Thread [ScheduledTasks:1] (Running)
Daemon Thread [EXPIRING-MAP-TIMER-1] (Running)
Thread [COMMIT-LOG-WRITER] (Running)
Thread [PERIODIC-COMMIT-LOG-SYNCER] (Running)
Thread [CompactionExecutor:1] (Running)
Thread [SSTABLE-DELETER] (Running)
Thread [ACCEPT-/127.0.0.1] (Running)
Thread [MutationStage:2] (Running)
Thread [MutationStage:8] (Running)
Thread [MutationStage:1] (Running)
Thread [MutationStage:6] (Running)
Thread [MutationStage:4] (Running)
Thread [MutationStage:5] (Running)
Thread [MutationStage:7] (Running)
Thread [InternalResponseStage:1] (Running)
Thread [StreamStage:1] (Running)
Thread [ReadStage:25] (Running)
Thread [MiscStage:1] (Running)
Thread [MutationStage:10] (Running)
Thread [ReadStage:29] (Running)
Thread [ReadStage:31] (Running)
Thread [AntiEntropyStage:1] (Running)
Thread [MutationStage:3] (Running)
Thread [MutationStage:17] (Running)
Thread [MutationStage:15] (Running)
Thread [MutationStage:19] (Running)
Thread [MutationStage:21] (Running)
Thread [ReadStage:27] (Running)
Thread [ReadStage:23] (Running)
Thread [ReadStage:21] (Running)
Thread [ReadStage:19] (Running)
Thread [ReadStage:15] (Running)
Thread [ReadStage:17] (Running)
Thread [RequestResponseStage:1] (Running)
Thread [ReadStage:13] (Running)
Thread [ReadStage:11] (Running)
Thread [ReadStage:1] (Running)
Thread [ReadStage:9] (Running)
Thread [ReadStage:3] (Running)
Thread [MutationStage:29] (Running)
Thread [ReadStage:7] (Running)
Thread [ReadStage:5] (Running)
Thread [MutationStage:31] (Running)
Thread [MutationStage:27] (Running)
Thread [MutationStage:25] (Running)
Thread [MutationStage:23] (Running)
Thread [MutationStage:13] (Running)
Thread [MutationStage:11] (Running)
Thread [MutationStage:9] (Running)
Thread [ReadStage:4] (Running)
Thread [ReadStage:30] (Running)
Thread [ReadStage:28] (Running)
Thread [MutationStage:14] (Running)
Thread [ReadStage:18] (Running)
Thread [ReadStage:12] (Running)
Thread [ReadStage:14] (Running)
Thread [ReadStage:32] (Running)
Thread [MutationStage:16] (Running)
Thread [MutationStage:20] (Running)
Thread [MutationStage:18] (Running)
Thread [ReadStage:20] (Running)
Thread [ReadStage:22] (Running)
Thread [ReadStage:26] (Running)
Thread [ReadStage:24] (Running)
Thread [MutationStage:32] (Running)
Thread [ReadStage:6] (Running)
Thread [ReadStage:2] (Running)
Thread [ReadStage:8] (Running)
Thread [MutationStage:24] (Running)
Thread [MutationStage:28] (Running)
Thread [ReadStage:10] (Running)
Thread [ReadStage:16] (Running)
Thread [MutationStage:26] (Running)
Thread [MutationStage:12] (Running)
Thread [ReadRepairStage:2] (Running)
Thread [MutationStage:22] (Running)
Thread [MutationStage:30] (Running)
Thread [InternalResponseStage:2] (Running)
Thread [RequestResponseStage:2] (Running)
Thread [ReadRepairStage:1] (Running)
Thread [MigrationStage:1] (Running)
Thread [GossipStage:1] (Running)
Thread [HintedHandoff:1] (Running)
Daemon Thread
[perf4j-async-stats-appender-sink-hector_CoalescingStatistics] (Running)

How can I kill cassandra completely?

Thanks

Matt

[Patricio Echagüe]

For a JVM per test class.

[Jonathan Shook]

I should have started another thread thread. This thread thread is not
the right thread thread for my thread question. thread.

But since I'm here, I'm specifically talking about hector client-side
threads. This is different than the other issue which is referring to
embedded Cassandra worker threads.

Here is my scenario:
1) Initialize a hector cluster with HFactory.createCluster(...)
2) Add some defs using cluster.addKeyspace(...)
3) do nothing else.

At this point, my app should exit, no? When I look to see why it
didn't, there are only 2 non-deamon threads left, and one of them is
managed by the VM.
The other is a pooled thread, presumably from the Hector thread pool.
It was named "pool-1-thread-1".
I'm using the latest release from Maven.

2011/3/16 Patricio Echagüe <patr...@gmail.com>:

[Nate McCall]

You have to explicitly call cluster.shutdown() or System.exit().
Otherwise the executor for CassandraHostRetryService will block.

[Patricio Echagüe]

1) what is the JVM thread's name?

2) did you try with HFactory.shutdownCluster(cluster) ?

3) what is the value of CassandraHostConfigurator#AutoDiscoverHosts and CassandraHostConfigurator#RetryDownedHosts?

[Jonathan Shook]

The name of the thread was: pool-1-thread-1

[Jonathan Shook]

This explains why it didn't work intuitively. Now I have the right
framing to ask the question.

Is it possible to make this more intuitive, by making the
CassandraHostRetryService executor thread a deamon thread?
Or, does this create other problems that I'm not thinking of?

I was expecting that once my app thread has ended, that all auxiliary
threads would automatically be reaped and my app would exit normally.
I can add the necessary shutdown calls, but it was just an unexpected
twist that I thought might be improved upon.

[Nate McCall]

It used to register with the jvm via Runtime#addShutdownHook for just
this reason, but this leads to classloader leakage in containers which
support re-deployment (ie. Tomcat).

This change is only in master and 0.7.0 tip - it has not been released yet.

[Matthias Keller]

Hi

Sorry I didn't have time to look further into it last week.
My 'problem' is that even if I use a test with a separate JVM, I'd really like to cleanup AFTER the test right away which is not possible since at the end of the test, the cassandra service is still running, thus all files are locked and cannot be deleted.
I really don't understand why it shouldn't be possible to stop an instance programmatically without terminating the JVM :-/

Thanks

Matt

On 2011-03-16 19:37, Patricio Echagüe wrote:

you need to fork a JVM per test class.

[Nate McCall]

Ran's article, though very informative, is over a year old which is a
very long time in this community.

Have a look at how the unit tests in the current Apache Cassandra
source distribution.
In the test/unit/ directory see:
o.a.c.SchemaLoader
o.a.c.EmbeddedServer
o.a.c.CleanupHelper

and how they are used from the internal test cases and ant. Some of
our test infrastructure has already moved over this way, but we will
be doing more work here going forward integrating with the
cassandra-maven-plugin as well:
http://mojo.codehaus.org/cassandra-maven-plugin/

In general, I think you would find it easier to call truncate between
test cases as opposed to spinning up and tearing down a bunch of
thread pools and multiple adding/removing multiple file directories
and their contents.

On Mon, Mar 21, 2011 at 4:03 AM, Matthias Keller