returner_sentry problem

[schlag]

Hi,

I am trying to hook up a sentry returner and have the following pillar settings for a number of my minions:

raven:

  servers:

    - https://obfus...@app.getsentry.com/obfuscated

  project: Saltstack

  tags:

    - host

I've created a team and project for this at getsentry called "Saltstack".  

So far no events have come through.  I wasn't quite following what the public_key or secret_key were for in the documentation (http://docs.saltstack.com/ref/returners/all/salt.returners.sentry_return.html) because I couldn't find anything related at getsentry.com.

I've tried testing this returner by using this: salt \* test.ping --return sentry_return I see the output, but nothing arrives at getsentry.com. Any ideas? Am curious how others tie into it.

Thanks in advance

[schlag]

meant to add that i've installed raven via pip (raven==4.0.3)

debian wheezy

salt --versions-report

           Salt: 0.17.4

         Python: 2.6.6 (r266:84292, Dec 26 2010, 22:31:48)

         Jinja2: 2.5.5

       M2Crypto: 0.20.1

 msgpack-python: 0.1.10

   msgpack-pure: Not Installed

       pycrypto: 2.1.0

         PyYAML: 3.09

          PyZMQ: 13.1.0

            ZMQ: 3.2.3

 

 

[Tim Messier]

The public_key and secret_key come from your sentry project. On getsentry.com go to your Saltstack project settings page and click on API Keys. It provides a url in the format of:

  https://<public_key>:<secret_key>@app.sentry.com/<project>

You can set the servers field to just point to https://app.getsentry.com. The keys are used by raven to authenticate and send messages to your specific sentry project.

My pillar is something like this:

raven:
  servers:
    - https://app.getsentry.com
  public_key: <key from API Keys page>
  secret_key: <key from API Keys page>
  project: <project id number>
  tags:
    - host
    - os
    - cpuarch

--
You received this message because you are subscribed to the Google Groups "Salt-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to salt-users+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[schlag]

thanks for that info!  

 so here is what i have in pillar now (keys & project id are obfuscated):

raven:

  servers:

    - https://app.getsentry.com

  public_key: pubkey

  secret_key: secretkey

  project:    project-id

  tags:

    - host

    - os

    - cpuarch

i also came across the sentry logging handler here http://docs.saltstack.com/ref/configuration/logging/handlers/salt.log.handlers.sentry_mod.html and set the following up in /etc/salt/master.d/sentry.conf:

sentry_handler:

  dsn:        https://pubkey:secr...@app.getsentry.com/project-id

  project:    project-id

  public_key: pubkey

  secret_key: secretkey

  log_level:  info

i've double & triple checked the keys and project-id.  when i restart salt-master after adding the sentry config, i get this error:

HTTPError: HTTP Error 400: BAD REQUEST

[ERROR   ] Unable to reach Sentry log server: HTTP Error 400: BAD REQUEST (url: https://app.getsentry.com/api/project-id/store/, body: Missing required attribute in authentication header: sentry_secret)

Traceback (most recent call last):

  File "/usr/local/lib/python2.6/dist-packages/raven/transport/threaded.py", line 104, in send_sync

    super(ThreadedHTTPTransport, self).send(data, headers)

  File "/usr/local/lib/python2.6/dist-packages/raven/transport/http.py", line 41, in send

    response = urlopen(req, data, self.timeout).read()

  File "/usr/lib/python2.6/urllib2.py", line 126, in urlopen

    return _opener.open(url, data, timeout)

  File "/usr/lib/python2.6/urllib2.py", line 397, in open

    response = meth(req, response)

  File "/usr/lib/python2.6/urllib2.py", line 510, in http_response

    'http', request, response, code, msg, hdrs)

  File "/usr/lib/python2.6/urllib2.py", line 435, in error

    return self._call_chain(*args)

  File "/usr/lib/python2.6/urllib2.py", line 369, in _call_chain

    result = func(*args)

  File "/usr/lib/python2.6/urllib2.py", line 518, in http_error_default

    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)

In getsentry, I chose Django/python as the "platform" and tried "Other" as well to see if that helped, but to no avail.

I tried simplifying the sentry_handler config in /etc/salt/master.d/sentry.conf, by making it look like this:

sentry_handler:

  dsn:        https://pubkey:secr...@app.getsentry.com/project-id

and now get this warning after restarting salt-master:

Starting salt master control daemon: salt-master/usr/local/lib/python2.6/dist-packages/raven/base.py:149: DeprecationWarning: Manually configured connections are deprecated. Switch to a DSN.

  warnings.warn('Manually configured connections are deprecated. Switch to a DSN.', DeprecationWarning)

So it seems to not be picked up, or maybe the error is just worded badly.  I am using a DSN in the handler config.  Either way, I'm still not receiving any events at getsentry.com.  

[Tim Messier]

Hmm..first did the change for the minion’s pillar data work? Does salt \* test.ping --return sentry work now? (Also note, --return sentry not --return sentry_return)

As for the DeprecationWarning, that is coming from raven (and is just a warning, I am using raven==3.5.2 and don’t get that warning, probably because it isn’t marked as deprecated in that version).

When you switched to just specifying just the dsn, did you also stop specifying the log_level? Because by default the log level is error, so you may not have had any errors for sentry to report.

[schlag]

yeah, so i've tried a few things.  yeah, the first raven settings in pillar didnt work.  and yes, i tried removing log_level when i switched to dsn.  thanks for the info on the deprecated warning and '--return sentry'.  i must have seen '--return sentry_return' somewhere online.

just to recap, my pillar settings are as follows (my project, & keys are obfuscated):

raven:

  servers:

    - https://app.getsentry.com

  project:     12345

  public_key:  deadbeefdeadbeefdeadbeefdeadbeef

  secret_key: beefdeadbeefdeadbeefdeadbeefdead

  tags:

    - os

    - master

    - saltversion

    - cpuarch

verified that the minions see it with 'salt-call pillar.get raven'

eg:

salt-call pillar.get raven

local:

    ----------

    project:

        12345

    public_key:

        deadbeefdeadbeefdeadbeefdeadbeef

    secret_key:

        beefdeadbeefdeadbeefdeadbeefdead

    servers:

        - https://app.getsentry.com

    tags:

        - os

        - master

        - saltversion

        - cpuarch

tested it with a 'salt \* test.ping --return sentry' on the master, still nothing in getsentry.  here is some debug output:

--snip

salt \* test.ping --return sentry -l debug

[DEBUG   ] Reading configuration from /etc/salt/master

[DEBUG   ] Including configuration from '/etc/salt/master.d/halite.conf'

[DEBUG   ] Reading configuration from /etc/salt/master.d/halite.conf

[DEBUG   ] Including configuration from '/etc/salt/master.d/schedule.conf'

[DEBUG   ] Reading configuration from /etc/salt/master.d/schedule.conf

[DEBUG   ] Including configuration from '/etc/salt/master.d/sentry.conf'

[DEBUG   ] Reading configuration from /etc/salt/master.d/sentry.conf

[DEBUG   ] Missing configuration file: /root/.salt

[DEBUG   ] loading log_handlers in ['/var/cache/salt/master/extmods/log_handlers', '/usr/lib/python2.6/dist-packages/salt/log/handlers']

[DEBUG   ] Skipping /var/cache/salt/master/extmods/log_handlers, it is not a directory

[DEBUG   ] Loaded sentry_mod as virtual sentry

[INFO    ] Processing `log_handlers.sentry`

/usr/local/lib/python2.6/dist-packages/raven/base.py:149: DeprecationWarning: Manually configured connections are deprecated. Switch to a DSN.

  warnings.warn('Manually configured connections are deprecated. Switch to a DSN.', DeprecationWarning)

[DEBUG   ] Adding the 'sentry' provided logging handler: <raven.handlers.logging.SentryHandler object at 0x26e0450>

[DEBUG   ] Configuration file path: /etc/salt/master

[DEBUG   ] Reading configuration from /etc/salt/master

[DEBUG   ] Including configuration from '/etc/salt/master.d/halite.conf'

[DEBUG   ] Reading configuration from /etc/salt/master.d/halite.conf

[DEBUG   ] Including configuration from '/etc/salt/master.d/schedule.conf'

[DEBUG   ] Reading configuration from /etc/salt/master.d/schedule.conf

[DEBUG   ] Including configuration from '/etc/salt/master.d/sentry.conf'

[DEBUG   ] Reading configuration from /etc/salt/master.d/sentry.conf

[DEBUG   ] Missing configuration file: /root/.salt

[DEBUG   ] LocalClientEvent PUB socket URI: ipc:///var/run/salt/master/master_event_pub.ipc

[DEBUG   ] LocalClientEvent PULL socket URI: ipc:///var/run/salt/master/master_event_pull.ipc

[DEBUG   ] loading output in ['/var/cache/salt/master/extmods/output', '/usr/lib/python2.6/dist-packages/salt/output']

[DEBUG   ] Skipping /var/cache/salt/master/extmods/output, it is not a directory

[DEBUG   ] Loaded no_out as virtual quiet

[DEBUG   ] Loaded json_out as virtual json

[DEBUG   ] Loaded yaml_out as virtual yaml

[DEBUG   ] Loaded pprint_out as virtual pprint

minion1:

    True

[DEBUG   ] loading output in ['/var/cache/salt/master/extmods/output', '/usr/lib/python2.6/dist-packages/salt/output']

[DEBUG   ] Skipping /var/cache/salt/master/extmods/output, it is not a directory

[DEBUG   ] Loaded no_out as virtual quiet

[DEBUG   ] Loaded json_out as virtual json

[DEBUG   ] Loaded yaml_out as virtual yaml

[DEBUG   ] Loaded pprint_out as virtual pprint

--snip

i do at least see this in the salt-master log after setting that output to debug:

--snip

2014-01-30 14:12:03,444 [salt.master                                 ][DEBUG   ] Published command details {'tgt_type': 'glob', 'jid': '20140130141203442417', 'tgt': 'ln*', 'ret': 'sentry', 'user': 'sudo_user1, 'arg': [], 'fun': 'test.ping'}

--snip

but I don't see any traffic going towards app.getsentry.com when i run the test.ping.

we have a pretty hefty enterprise plan and are under our daily usage.  i'll contact them still though to see if they're seeing anything on their end.

one thing of note is that if i enable sentry_handler with a dsn and i specify log_level: info (so that i can actually see stuff), i start seeing these errors:

--snip

[ERROR   ] Unable to reach Sentry log server: HTTP Error 400: BAD REQUEST (url: https://app.getsentry.com/api/12345/store/, body: Missing required attribute in authentication header: sentry_secret)

Traceback (most recent call last):

  File "/usr/local/lib/python2.6/dist-packages/raven/transport/threaded.py", line 104, in send_sync

    super(ThreadedHTTPTransport, self).send(data, headers)

  File "/usr/local/lib/python2.6/dist-packages/raven/transport/http.py", line 41, in send

    response = urlopen(req, data, self.timeout).read()

  File "/usr/lib/python2.6/urllib2.py", line 126, in urlopen

    return _opener.open(url, data, timeout)

  File "/usr/lib/python2.6/urllib2.py", line 397, in open

    response = meth(req, response)

  File "/usr/lib/python2.6/urllib2.py", line 510, in http_response

    'http', request, response, code, msg, hdrs)

  File "/usr/lib/python2.6/urllib2.py", line 435, in error

    return self._call_chain(*args)

  File "/usr/lib/python2.6/urllib2.py", line 369, in _call_chain

    result = func(*args)

  File "/usr/lib/python2.6/urllib2.py", line 518, in http_error_default

    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)

HTTPError: HTTP Error 400: BAD REQUEST

Sentry is attempting to send 3 pending error messages

Waiting up to 10 seconds

Press Ctrl-C to quit

--snip

i upgraded my master and all minions to 0.17.5

salt --versions-report

           Salt: 0.17.5

         Python: 2.6.6 (r266:84292, Dec 26 2010, 22:31:48)

         Jinja2: 2.5.5

       M2Crypto: 0.20.1

 msgpack-python: 0.1.10

   msgpack-pure: Not Installed

       pycrypto: 2.1.0

         PyYAML: 3.09

          PyZMQ: 13.1.0

            ZMQ: 3.2.3

my whole point of this exercise is to try to do the following:

 1) get salt warnings & errors into getsentry (things that are normally just logged to syslog locally) <-- i presume sentry_handler manages this

 2) get salt events (state executions & their results, module executions, etc..) into getsentry  <-- i was thinking that a returner would help here, but even if i get this part working, i'm not sure what is required to get that data pushed into sentry w/o having to use '--return sentry' every time.    is my approach even right here?

thanks again!

[Tim Messier]

I did some more testing, I installed some newer version of raven on some hosts (since that seems to be the only difference between your setup and mine), and it seems that salt is failing with raven>3.5. I would file a bug about this and while you want for that to be fixed, you could try installing raven==3.5.2 (that is the version that is working for me).

As for 2: if you are running commands via the cli, I think you will always need to specifiy --return sentry, I don’t see an option to set a default returner (feature request?). You do seem to be able to specify a returner using the salt scheduler if that is of any help.

--

[schlag]

pip installing raven==3.5.2 did the trick!   thank you.  i'll look into the scheduler for specifying returners.  very much appreciated!