Top_n_hashtags were not appearing. Correct way to stop cluster/adapter?

[scotavoosh]

While running the sample twittertopiccount app, I ran into the problem
that the top_n_hashtags file was not appearing. First issue was
configuring the app to pass through a proxy. That can be done by
adding these lines of code to the connectAndRead method in
TwitterFeedListener before opening the URLConnection:

System.setProperty("https.proxyHost","your.proxy.com");
System.setProperty("https.proxyPort","8080");

The problem still persisted. The log files showed that events were
being dispatched but were not being consumed. Also in the log file was
a line that said:

2011-12-08 15:07:29,544 io.s4.comm.file.StaticTaskManager INFO
(StaticTaskManager.java:185) Process taken up by another process
lockFile:/Users/user/Development/s4/build/s4-image/scripts/../s4-core/
lock/s4s4node-0
2011-12-08 15:07:29,544 io.s4.comm.file.StaticTaskManager INFO
(StaticTaskManager.java:102) processAvailable:false

I found the solution was to remove the lock files in /Users/user/
Development/s4/build/s4-image/scripts/../s4-core/lock/s4s4node-0

These need to be removed after killing the running processes for the
cluster, client-adapter, and feed listener.

What is the correct way to stop the cluster and adapter so that these
lock files are released?

[Matthieu Morel]

Hi,

I'm answering on the s4-...@incubator.apache.org mailing list, since
this is the mailing list we are now using.

On 12/9/11 9:05 PM, scotavoosh wrote:
> While running the sample twittertopiccount app, I ran into the problem
> that the top_n_hashtags file was not appearing. First issue was
> configuring the app to pass through a proxy. That can be done by
> adding these lines of code to the connectAndRead method in
> TwitterFeedListener before opening the URLConnection:
>
> System.setProperty("https.proxyHost","your.proxy.com");
> System.setProperty("https.proxyPort","8080");

I assume this was specific to your network configuration. But that will
be useful to others with similar network configs, thanks!

>
> The problem still persisted. The log files showed that events were
> being dispatched but were not being consumed. Also in the log file was
> a line that said:
>
> 2011-12-08 15:07:29,544 io.s4.comm.file.StaticTaskManager INFO
> (StaticTaskManager.java:185) Process taken up by another process
> lockFile:/Users/user/Development/s4/build/s4-image/scripts/../s4-core/
> lock/s4s4node-0
> 2011-12-08 15:07:29,544 io.s4.comm.file.StaticTaskManager INFO
> (StaticTaskManager.java:102) processAvailable:false
>
> I found the solution was to remove the lock files in /Users/user/
> Development/s4/build/s4-image/scripts/../s4-core/lock/s4s4node-0
>
> These need to be removed after killing the running processes for the
> cluster, client-adapter, and feed listener.
>
> What is the correct way to stop the cluster and adapter so that these
> lock files are released?

There are currently 2 ways to control assignation of logical tasks to S4
nodes:
- through the file system, by creating "lock" files
- through Zookeeper, by using ephemeral nodes

S4 nodes delete owned lock files upon exit using the File.deleteOnExit
method (
http://docs.oracle.com/javase/6/docs/api/java/io/File.html#deleteOnExit() ).
Unfortunately, this won't work well if you kill the S4 node process:
lock files will remain.

Therefore you should either:
- clean-up lock files at start-up through a script or an initialization
routine
- use the Zookeeper task assignation mechanism (
http://docs.s4.io/tutorials/zookeeper.html )

By the way, future versions of S4 (from 0.5) will only use Zookeeper for
task assignation.

Hope this helps,

Matthieu