Help with Skyline: Setting up Analyzer, Queuing Horizon Elements, Getting Stuff to show up on Web UI

[Karan Lala]

Hi There!

I wanted to work on a side project in python, and I came across the mirage branch of the Skyline project - it seemed pretty cool! I am trying to actually work with it now, but I'm running into trouble in the following areas: Setting up Analyzer, Queing up Horizon Elements, Getting Stuff to show up on Web UI

*Background info: Working on a Mac*

So, my questions:

1. Setting up Analyzer

    So, I have everything set up (I think), and I go to run the seed data. After I run it, I got to check all of the log files and to see if Redis has any data. Here is what I get in analyzer.log

     

Traceback (most recent call last):

  File "/Users/klala/skyline/bin/../src/analyzer/analyzer-agent.py", line 70, in <module>

    daemon_runner.do_action()

  File "/Library/Python/2.7/site-packages/daemon/runner.py", line 189, in do_action

    func(self)

  File "/Library/Python/2.7/site-packages/daemon/runner.py", line 124, in _start

    self.daemon_context.open()

  File "/Library/Python/2.7/site-packages/daemon/daemon.py", line 346, in open

    self.pidfile.__enter__()

  File "/Library/Python/2.7/site-packages/lockfile/__init__.py", line 238, in __enter__

    self.acquire()

  File "/Library/Python/2.7/site-packages/daemon/pidfile.py", line 42, in acquire

    super(TimeoutPIDLockFile, self).acquire(timeout, *args, **kwargs)

  File "/Library/Python/2.7/site-packages/lockfile/pidlockfile.py", line 88, in acquire

    self.path)

lockfile.LockTimeout: Timeout waiting to acquire lock for /var/run/skyline/analyzer.pid

 2015-06-29 14:28:27 :: 72722 :: WARNING: skyline is set for more cores than needed.

2015-06-29 14:28:27 :: 72722 :: seconds to run    :: 0.49

2015-06-29 14:28:27 :: 72722 :: total metrics     :: 1

2015-06-29 14:28:27 :: 72722 :: total analyzed    :: 1

2015-06-29 14:28:27 :: 72722 :: total anomalies   :: 0

2015-06-29 14:28:27 :: 72722 :: exception stats   :: {}

2015-06-29 14:28:27 :: 72722 :: anomaly breakdown :: {}

2015-06-29 14:28:27 :: 72722 :: sleeping due to low run time...

    Now, here I know that analyzer.log isn't what it should be, since it should have more than 1 metric because I am using the large data.json file. Also, there's an error. How can I go about fixing this?

2. Queuing Horizon Elements

  * I first tried to do this project on AWS, and I didn't have this issue there. I want to replicate what I was able to do on AWS onto my Mac*

  When I look at my horizon.log file, I see:

Traceback (most recent call last):

  File "/Users/klala/skyline/bin/../src/horizon/horizon-agent.py", line 89, in <module>

    daemon_runner.do_action()

  File "/Library/Python/2.7/site-packages/daemon/runner.py", line 189, in do_action

    func(self)

  File "/Library/Python/2.7/site-packages/daemon/runner.py", line 124, in _start

    self.daemon_context.open()

  File "/Library/Python/2.7/site-packages/daemon/daemon.py", line 346, in open

    self.pidfile.__enter__()

  File "/Library/Python/2.7/site-packages/lockfile/__init__.py", line 238, in __enter__

    self.acquire()

  File "/Library/Python/2.7/site-packages/daemon/pidfile.py", line 42, in acquire

    super(TimeoutPIDLockFile, self).acquire(timeout, *args, **kwargs)

  File "/Library/Python/2.7/site-packages/lockfile/pidlockfile.py", line 88, in acquire

    self.path)

lockfile.LockTimeout: Timeout waiting to acquire lock for /var/run/skyline/horizon.pid

2015-06-29 15:08:38 :: 72738 :: worker queue is empty and timed out

How do I fix this?

3. Getting stuff to show up on Web UI

When I run the web UI, it is blank. How do I get my seed data to show up on it? (I tried reducing DURATION to 300 milliseconds to see if something showed up, but it didn't)

I really don't know what I am doing and I would  appreciate any help :)

Thanks!

  

[Karan Lala]

Also, my Horizon.log looks like this:

2015-06-29 16:08:39 :: 78059 :: metrics. keyspace is 1

2015-06-29 16:08:39 :: 78059 :: blocked 0 times

2015-06-29 16:08:39 :: 78059 :: euthanized 0 geriatric keys

2015-06-29 16:08:39 :: 78059 :: sleeping due to low run time...

2015-06-29 16:08:49 :: 78067 :: operated on metrics. in 0.161651 seconds

2015-06-29 16:08:49 :: 78067 :: metrics. keyspace is 1

2015-06-29 16:08:49 :: 78067 :: blocked 0 times

2015-06-29 16:08:49 :: 78067 :: euthanized 0 geriatric keys

2015-06-29 16:08:49 :: 78067 :: sleeping due to low run time...

2015-06-29 16:08:52 :: 76735 :: worker queue is empty and timed out

despite the fact that I ran the seed data on it.  How do I fix this? 

Thank you!

[Dan K]

Hi Karan,

Did you ever find a resolution to this? I have identical behavior at the moment, and it's gotten to be quite frustrating.

[Karan Lala]

Hey Dan,

Check if the /var/run/skyline directory exists. If that directory exists and you are getting that error in your log file, feel free to email me!

[Dan K]

Karan,

All is well, I had a similar issue (#1, and #2) that was being caused by a crappy startup script I wrote.

As for #3, the seed_data.py never displayed inside of the webapp for me. I ended up writing a custom script in Java and it finally all looks good.

Dan

[Karan Lala]

Awesome! 

Yeah, the seed data never displayed for me either. Would you mind sharing the Java script that you wrote?

Thanks :)

[Dan K]

The script is a port of a previously posted seed_data variation, with some minor adjustments to get it all converted to Java. 

Unfortunately, because I'm working on this for my employer, I can't share the script in any substantial way. However, what I can do is give some general guidance / point to the right code for someone to do it themselves. I'm telling you though, it's a direct port of the above link :)

As a little note: if it matters at all, I send it all via udp @ port 2025

Code to correctly encode a data point in messagePack:

ByteArrayOutputStream out = new ByteArrayOutputStream();
MessagePacker messagepacker = MessagePack.newDefaultPacker(out);
double[] message = new double[2]; //formatted as Key/Value pair. message[0] = epoch seconds, [1] = data value

messagepacker
 .packArrayHeader(2)
      .packString(metricName)    
      .packArrayHeader(2) 
           .packDouble(message[0]) 
           .packDouble(message[1])
 .close();

 byte[] buf = out.toByteArray(); 
 out.reset();

[earthgecko]

Hi Dan

I am sure your use case is pretty "close" topology wise, however just pointing out that if it is related to metrics, UDP is not necessarily the best option, especially if the feed is remote to the skyline host.  I'd tell a udp joke but I am not sure you would get it :)

For anything other than local LAN (or maybe localhost) I would suggest that adding a listener on TCP as per https://github.com/etsy/skyline/wiki/Getting-Data-Into-Skyline and have java connect via tcp rather than udp.
Anomaly detection is hard enough, not sure that adding udp into the variables is going to be helpful, metrics and udp :)

G