analyzer-agent runtime warning in logs

[Kurt Abersold]

Greetings,

I am deploying Skyline in a testing environment at my workplace. I used bflad/chef-skyline Chef cookbook to deploy. Everything seems to be working, but I am not sure if skyline-analyzer it doing its job, as the log only contains one entry from when the application is started. Analyzer-agent is using a lot of CPU, so it appears to be doing something, but without logs, I am not confident it's functioning properly. The error appears to be related to pandas. I am running pandas==0.15.0 from pip.

Contents of /var/log/skyline/analyzer.log

started with pid 21378

/usr/local/lib/python2.7/dist-packages/pandas/core/ops.py:562: RuntimeWarning: invalid value encountered in less

  result = getattr(x, name)(y)

[Abe Stanway]

Hmm, that doesn't sound good. What data in Redis?

--
You received this message because you are subscribed to the Google Groups "skyline-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to skyline-dev...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Abe Stanway

abe.is

[Kurt Abersold]

Ok, so it looks like it may be an issue with redis. I am running Redis server v=2.6.17 sha=00000000:0 malloc=jemalloc-3.2.0 bits=64. Should I 

be running 2.8.x instead? The output from redis-cli monitor seems off, but I really don't know much about redis. I am pretty confident I have carbon-relay configured properly, and the data is displaying in graphite. My relay configuration is pretty much the same as displayed at: https://github.com/etsy/skyline/wiki/Getting-Data-Into-Skyline.

A snippet from redis-cli monitor:

kurtabersold@skyline:~$ redis-cli monitor

OK

1415818892.783328 [0 unix:/tmp/redis.sock] "PING"

1415818892.792504 [0 unix:/tmp/redis.sock] "SMEMBERS" "metrics.unique_metrics"

1415818892.793033 [0 unix:/tmp/redis.sock] "WATCH" "metrics.VoIP.xmsprobe1_mycompany_com.empPacketInfoTotalSIPpackets"

1415818892.793279 [0 unix:/tmp/redis.sock] "GET" "metrics.VoIP.xmsprobe1_mycompany_com.empPacketInfoTotalSIPpackets"

1415818892.794284 [0 unix:/tmp/redis.sock] "MULTI"

1415818892.794521 [0 unix:/tmp/redis.sock] "SMEMBERS" "mini.unique_metrics"

1415818892.795378 [0 unix:/tmp/redis.sock] "SET" "metrics.VoIP.xmsprobe1_mycompany_com.empPacketInfoTotalSIPpackets" "\x92\xcbA\xd5\x18\xb6\x94@\x00\x00\xcb@\x9f\xdc\x00\x00\x00\x00\x00\x92\xcbA\xd5\x18\xb6\xa3\x80\x00\x00\xcb@\x9f\xdc\x00\x00\x00\x00\x00\x92\xcbA\xd5\x18\xb6\xb2@\x00\x00\xcb@\x9c\x8c\x00\x00\x00\x00\x00\x92\xcbA\xd5\x18\xb6\xc1\x80\x00\x00\xcb@\x9a\xe0\x00\x00\x00\x00\x00\x92\xcbA\xd5\x18\xb6\xd0\x80\x00\x00\xcb@\x98\xd8\x00\x00\x00\x00\x00\x92\xcbA\xd5\x18\xb6\xdf@\x00\x00\xcb@\x9f\b\x00\x00\x00\x00\x00\x92\xcbA\xd5\x18\xb6\xee@\x00\x00\xcb@\xa2l\x00\x00\x00\x00\x00\x92\xcbA\xd5\x18\xb6\xfd\x80\x00\x00\xcb@\x96\xa4\x00\x00\x00\x00\x00\x92\xcbA\xd5\x18\xb7\x0c\x80\x00\x00\xcb@\x9d\x9c\x00\x00\x00\x00\x00\x92\xcbA\xd5\x18\xb7\x1b@\x00\x00\xcb@\xa1`\x00\x00\x00\x00\x00\x92\xcbA\xd5\x18\xb7*\x80\x00\x00\xcb@\xa0\x94\x00\x00\x00\x00\x00\x92\xcbA\xd5\x18\xb79@\x00\x00\xcb@\xa2\x0e\x00\x00\x00\x00\x00\x92\xcbA\xd5\x18\xb7H@\x00\x00\xcb@\x9cl\x00\x00\x00\x00\x00\x92\xcbA\xd5\x18\xb7W\x80\x00\x00\xcb@\x9f\xfc\x00\x00\x00\x00\x00\x92\xcbA\xd5\x18\xb7f\x80\x00\x00\xcb@\xa1B\x00\x00\x00\x00\x00\x92\xcbA\xd5\x18\xb7u@\x00\x00\xcb@\x9e\xe0\x00\x00\x00\x00\x00\x92\xcbA\xd5\x18\xb7\x84@\x00\x00\xcb@\xa1&\x00\x00\x00\x00\x00\x92\xcbA\xd5\x18\xb7\x93\x80\x00\x00\xcb@\x9e\xe4\x00\x00\x00\x00\x00\x92\xcbA\xd5\x18\xb7\xa2\x80\x00\x00\xcb@\xa1\b\x00\x00\x00\x00\x00\x92\xcbA\xd5\x18\xb7\xb1@\x00\x00\xcb@\x9c\xbc\x00\x00\x00\x00\x00\x92\xcbA\xd5\x18\xb7\xc0\x80\x00\x00\xcb@\xa5\x9a\x00\x00\x00\x00\x00\x92\xcbA\xd5\x18\xb7\xcf\x80\x00\x00\xcb@\xa2\b\x00\x00\x00\x00\x00\x92\xcbA\xd5\x18\xb7\xde@\x00\x00\xcb@\xa4H\x00\x00\x00\x00\x00\x92\xcbA\xd5\x18\xb7\xed@\x00\x00\xcb@\xa2t\x00\x00\x00\x00\x00\x92\xcbA\xd5\x18\xb7\xfc\x80\x00\x00\xcb@\xa2\x1a\x00\x00\x00\x00\x00\x92\xcbA\xd5\x18\xb8\x0b\x80\x00\x00\xcb@\xa3.\x00\x00\x00\x00\x00\x92\xcbA\xd5\x18\xb8\x1a@\x00\x00\xcb@\xa5P\x00\x00\x0

[Abe Stanway]

No, that seems ok. You're seeing the messagepack in the Redis key.

Is that the full logs? No line numbers or tracebacks or anything? It's probably got to do with Pandas.

[Kurt Abersold]

Yes, that is the full log. The log truncates when the daemon is restarted. I will try removing/re-installing pandas. Any known issues with different versions?

[Abe Stanway]

Couldn't rightly say :/

There should be a traceback that gets printed - everything that happens in the analyzer is wrapped in a block that catches and prints exceptions. Can you try tailing the log and seeing if you can catch it before it truncates?

[Kurt Abersold]

While following the log, I stopped then started the service. I still get the same runtime warning, with no traceback. I think everything in my config is ok, but out of curiosity, what is CARBON_PORT supposed to point to? The default of 2003 is the port my carbon-relay is listening on for plain text (not pickle). Should I switch that to 2103, which is the port carbon-cache:a is listening on?

kabersold@kale:/var/log/skyline$ tail -f analyzer.log 

started with pid 1561

/usr/local/lib/python2.7/dist-packages/pandas/core/ops.py:562: RuntimeWarning: invalid value encountered in less

  result = getattr(x, name)(y)

tail: analyzer.log: file truncated

started with pid 1634

2014-11-12 12:36:28 :: 1634 :: starting skyline analyzer

:562: RuntimeWarning: invalid value encountered in less

  result = getattr(x, name)(y)

^C

kabersold@kale:/var/log/skyline$

[Abe Stanway]

CARBON_PORT is if you to send Skyline internal metrics to Graphite.

I'll look into Pandas - will try installing that version and seeing if that's the issue.

[Kurt Abersold]

I fixed the issue by installing Skyline on Ubuntu 14.04 rather than 12.04.

[Michał Łowicki]

I'm having the same issue:

  result = getattr(x, name)(y)

/var/skyline/local/lib/python2.7/site-packages/pandas/core/ops.py:565: RuntimeWarning: invalid value encountered in less

  result = getattr(x, name)(y)

/var/skyline/local/lib/python2.7/site-packages/pandas/core/ops.py:565: RuntimeWarning: invalid value encountered in less

  result = getattr(x, name)(y)

with pandas 0.15.2 installed.

[Jason Ramapuram]

I am still having this issue as well. pandas version: 0.16.0 on ubuntu 14.04

[Lex Herbert]

I was experiencing this issue under RHEL 6.4. Was able to work around by downgrading pandas to 0.14.1.

[Adithya Benny]

I too started getting "/usr/lib64/python2.6/site-packages/pandas/core/ops.py:566: RuntimeWarning: invalid value encountered in less

  result = getattr(x, name)(y)" out of nowhere; it was fine in the beginning.

I'm running Skyline on CentOS 6.6. As pip didn't provide any version of pandas except 0.16.1, I'd to download 0.14.1 from their website and run 'sudo python setup.py install'.

Now, I've started getting "ux-x86_64.egg/pandas/stats/moments.py:526: RuntimeWarning: invalid value encountered in sqrt

  result = np.sqrt(x)" and some other errors from the statsmodel package (I don't have the backtrace since the logs clear when the Analyzer dameon restarts). I don't seem to have a dependable way of reproducing these errors either. On the bright side, the errors aren't as frequent as before.

Guess I'll have to comb through the python packages used by algorithms.py ; that plus the fact that there aren't any commits to their repository in the last year is rather disconcerting. Shame.. the project looked so promising.

[Tim Hawkins]

Yep another "me too" 

Running on fedora 22 

[thawkins@timdev04 bin]$ cat /var/log/skyline/analyzer.log

started with pid 15936

/usr/lib64/python2.7/site-packages/pandas/core/ops.py:566: RuntimeWarning: inval

id value encountered in less

  result = getattr(x, name)(y)

/usr/lib64/python2.7/site-packages/pandas/core/ops.py:566: RuntimeWarning: inval

id value encountered in less

  result = getattr(x, name)(y)

......... more of same ....... 

/usr/lib64/python2.7/site-packages/pandas/core/ops.py:566: RuntimeWarning: inval

id value encountered in less

  result = getattr(x, name)(y)

ARNING: skyline is set for more cores than needed.

2015-07-17 13:49:14 :: 15936 :: seconds to run    :: 0.05

2015-07-17 13:49:14 :: 15936 :: total metrics     :: 1

2015-07-17 13:49:14 :: 15936 :: total analyzed    :: 1

2015-07-17 13:49:14 :: 15936 :: total anomalies   :: 0

2015-07-17 13:49:14 :: 15936 :: exception stats   :: {}

2015-07-17 13:49:14 :: 15936 :: anomaly breakdown :: {}

2015-07-17 13:49:14 :: 15936 :: sleeping due to low run time...

2015-07-17 13:49:24 :: 15936 :: WARNING: skyline is set for more cores than need

ed.

....... more of same ..... 

[thawkins@timdev04 bin]$ sudo dnf info python-pandas

Last metadata expiration check performed 2:34:01 ago on Fri Jul 17 11:21:11 2015.

Installed Packages

Name        : python-pandas

Arch        : x86_64

Epoch       : 0

Version     : 0.16.2

Release     : 1.fc22

Size        : 27 M

Repo        : @System

From repo   : updates

Summary     : Python library providing high-performance data analysis tools

URL         : http://pandas.pydata.org/

License     : BSD

Description : pandas is an open source, BSD-licensed library providing

            : high-performance, easy-to-use data structures and data

            : analysis tools for the Python programming language.

[thawkins@timdev04 bin]$