How to enable webhook when object create/update/delete

[caixing yang]

Hi all,
When I enable webhook via Extras>webhooks for object changed, but it seems it doesn't work, I don't know if any extra configure needed, so if anyone else also has the same issue, thanks very much.

[Brian Candler]

Have you set WEBHOOKS_ENABLED in configuration.py?

Note: it's mentioned here, but the config example given is out of date.

[Brian Candler]

The other thing you need to check is whether your rqworker process is running (*).

You can check whether the webhook request has been enqueued like this:

# redis-cli keys "*"

1) "rq:job:3337ef96-a2f8-42ac-9484-789b852df9de"

2) "rq:job:b7b69bbe-277b-43bd-8cd8-61d1fe29b8cd"

3) "rq:job:c44ff53f-afa7-4c91-863e-40783d91b710"

4) "rq:queues"

5) "rq:queue:default"

If you see the enqueued jobs then the problem is most likely with rqworker not picking them up.

HTH,

Brian.

(*) I use a systemd unit for this.  I run netbox as user "netbox", but you might want to run it as "www-data" instead.

[Unit]

Description=Netbox - rqworker webhook processing

After=network-online.target

Wants=network-online.target

Conflicts=shutdown.target

[Service]

User=netbox

ExecStart=/usr/bin/python3 /opt/netbox/netbox/manage.py rqworker

WorkingDirectory=/opt/netbox/netbox

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-user.target

[Brian Candler]

One other possible problem: you may find the version of the python redis module you have is out of date. django-rq requires redis >= 3.0.0, but due to a long-standing problem in pip, it doesn't automatically install the correct version in certain circumstances.

You can easily find out whether this affects you simply by typing "pip3 check".  Details are in #3460.

[caixing yang]

Hi Brian, 

    Thanks a lot for your help, I summarized the possible reason as you mentioned, hope you can give some some advices, thanks in advance. 

1. Set WEBHOOKS_ENABLED in configuration.py.

REDIS = {

    'HOST': 'localhost',

    'PORT': 6379,

    'PASSWORD': '',

    'DATABASE': 0,

    'DEFAULT_TIMEOUT': 300,

    'SSL': False,

}

WEBHOOKS_ENABLED = True

2.rqworker process

root@netbox:~# redis-cli keys "*"

1) "rq:finished:default"

2) "rq:worker:b55ec2727c1c4715a64e829902ea4934"

3) "rq:workers"

4) "rq:queues"

5) "rq:workers:default"

3.django-rq and redis version

root@netbox:~# redis-server -v

Redis server v=4.0.9 sha=00000000:0 malloc=jemalloc-3.6.0 bits=64 build=9435c3c2879311f3

root@netbox:/opt/netbox# cat requirements.txt 

Django>=2.2,<2.3

django-cacheops==4.1

django-cors-headers==3.0.2

django-debug-toolbar==2.0

django-filter==2.1.0

django-mptt==0.9.1

django-prometheus==1.0.15

django-rq==2.1.0

django-tables2==2.0.6

django-taggit==1.1.0

django-taggit-serializer==0.1.7

django-timezone-field==3.0

djangorestframework==3.9.4

drf-yasg[validation]==1.16.0

graphviz==0.10.1

Jinja2==2.10.1

Markdown==2.6.11

netaddr==0.7.19

Pillow==6.0.0

psycopg2-binary==2.8.3

py-gfm==0.1.4

pycryptodome==3.8.2

在 2019年9月9日星期一 UTC+8上午3:04:43，Brian Candler写道：

[Brian Candler]

You need to check the installed version of the "redis" python library, by running these commands:

pip3 list | grep redis

pip3 check

What does it show?

The output of redis-cli suggests that a worker has been created, but no messages have been queued.  Have you created a webhook in the GUI, and is it enabled?  You can check by running the following database query:

sudo su - postgres -c 'psql netbox'

select name,payload_url,enabled from extras_webhook;

\q

I also notice you wrote this:

REDIS = {

    'HOST': 'localhost',

    'PORT': 6379,

    'PASSWORD': '',

    'DATABASE': 0,

    'DEFAULT_TIMEOUT': 300,

    'SSL': False,

}

That's wrong, because you're missing 'CACHE_DATABASE': 1.  Are you definitely running netbox 2.6?  Can you check that your configuration.py doesn't contain two separate instances of the REDIS setting, with one overriding the other?  And check for any other missing settings which are in configuration.example.py which should be in configuration.py

If you're running something older than 2.6, then please upgrade.

When you make changes in the GUI, are they reflected as Object Changes in the database? Navigate to Organization > Changelog.  The same mechanism which writes the changelogs is also responsible for triggering the webhooks.

HTH,

Brian.

[caixing yang]

Hi Brian, 

   Of course, I run at the latest v2.6.3, and I think my configure is correct, as below. the configuration.py setting is same with configuration.example.py, because it's copied from that, I also set CACHE_DATABASE': 1, but still doesn't work.

I already enable webhooks in GUI, and when object changes, it will reflected as changelog, database is always good.

But I found from official documents, it says need the webhooks post request data, I don't know if I need the request data, and where/how to configure it.( https://netbox.readthedocs.io/en/stable/additional-features/webhooks/)

1.

root@netbox:~# pip3 list | grep redis

DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.

redis (3.3.8)

root@netbox:~# 

root@netbox:~# pip3 check

No broken requirements found.

2.

root@netbox:~# sudo su - postgres -c 'psql netbox'

psql (10.10 (Ubuntu 10.10-0ubuntu0.18.04.1))

Type "help" for help.

netbox=# select name,payload_url,enabled from extras_webhook;

   name   |                                                    payload_url                                                     | enabled 

----------+--------------------------------------------------------------------------------------------------------------------+---------

 dingding | https://oapi.dingtalk.com/robot/send?access_token=f583ecc614febb8d8f48b952626e3f59d840ff299f7ba32ce1233db834671bf2 | t

(1 row)

netbox=# \q

root@netbox:~# 

3.

在 2019年9月10日星期二 UTC+8上午1:16:45，Brian Candler写道：

[Brian Candler]

On Tuesday, 10 September 2019 03:01:06 UTC+1, caixing yang wrote:

But I found from official documents, it says need the webhooks post request data, I don't know if I need the request data, and where/how to configure it.( https://netbox.readthedocs.io/en/stable/additional-features/webhooks/)

The webhook makes a POST to the given URL, and the content is not configurable.  There are examples in that page of the data which is posted on "created" and "deleted" events; an "updated" event is the same as "created".

The next place you should look is in your web browser: go to http://netbox.yourdomain.com/admin/webhook-backend-status/

This is the django_rq status page.  What do you see there?  If you see a number of failed jobs > 0 then click on the number, it will list the jobs and you can click on each one for details.  If workers is 1, but everything else is 0, then no webhook events are being queued.  If the page isn't displayed at all, it means that WEBHOOKS_ENABLED is False and you need to find out why.

Apart from that, I can't guess what's wrong on your system.  All I can say is, it works fine for me, and something is different on your system.  If I were in front of your system, I would be checking your configuration.py very carefully, e.g. by doing

cd /opt/netbox/netbox/netbox

diff -u configuration.example.py configuration.py

I would be checking what userid netbox is running under, and checking pip3 packages *as that user*, e.g.

ps auxwww | grep netbox

sudo su - www-data -s /bin/bash

pip3 list | grep redis

pip3 list | grep django

exit

And I would be checking whether there are any changes in your source code from v2.6.3 (e.g. by "git status" and "git diff v2.6.3" if it's a git checkout).  And after that, I'd be inserting debugging into the python at the point where enqueue_webhooks is called.

[caixing yang]

Hi Brian, 

 I check webhook backend, it shows jobs finished, so maybe webhook works fined, and dingtalk webhook do not support the Post data, and so on, I don't know, I will check it,  thanks very much. 

1.

2.

root@netbox:~# sudo su - www-data -s /bin/bash

www-data@netbox:~$ pip3 list | grep redis

DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.

redis (3.3.8)

www-data@netbox:~$ pip3 list | grep django

DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.

django-cacheops (4.1)

django-cors-headers (3.0.2)

django-debug-toolbar (2.0)

django-filter (2.1.0)

django-js-asset (1.2.2)

django-mptt (0.9.1)

django-prometheus (1.0.15)

django-rq (2.1.0)

django-tables2 (2.0.6)

django-taggit (1.1.0)

django-taggit-serializer (0.1.7)

django-timezone-field (3.0)

djangorestframework (3.9.4)

www-data@netbox:~$ exit

logout

3.root@netbox:/opt/netbox/netbox/netbox# diff -u configuration.example.py configuration.py

--- configuration.example.py    2019-06-24 07:19:07.568362868 +0000

+++ configuration.py    2019-09-10 07:33:36.148926376 +0000

@@ -8,13 +8,13 @@

 # access to the server via any other hostnames. The first FQDN in the list will be treated as the preferred name.

 #

 # Example: ALLOWED_HOSTS = ['netbox.example.com', 'netbox.internal.local']

-ALLOWED_HOSTS = []

+ALLOWED_HOSTS = ['172.20.0.35']

 

 # PostgreSQL database configuration.

 DATABASE = {

     'NAME': 'netbox',         # Database name

-    'USER': '',               # PostgreSQL username

-    'PASSWORD': '',           # PostgreSQL password

+    'USER': 'netbox',               # PostgreSQL username

+    'PASSWORD': 'WHFk7hpW',           # PostgreSQL password

     'HOST': 'localhost',      # Database server

     'PORT': '',               # Database port (leave blank for default)

 }

@@ -23,18 +23,7 @@

 # For optimal security, SECRET_KEY should be at least 50 characters in length and contain a mix of letters, numbers, and

 # symbols. NetBox will not run without this defined. For more information, see

 # https://docs.djangoproject.com/en/dev/ref/settings/#std:setting-SECRET_KEY

-SECRET_KEY = ''

-

-# Redis database settings. The Redis database is used for caching and background processing such as webhooks

-REDIS = {

-    'HOST': 'localhost',

-    'PORT': 6379,

-    'PASSWORD': '',

-    'DATABASE': 0,

-    'CACHE_DATABASE': 1,

-    'DEFAULT_TIMEOUT': 300,

-    'SSL': False,

-}

+SECRET_KEY = 'WHFk7hpW'

 

 

 #########################

@@ -61,9 +50,6 @@

 # BASE_PATH = 'netbox/'

 BASE_PATH = ''

 

-# Cache timeout in seconds. Set to 0 to dissable caching. Defaults to 900 (15 minutes)

-CACHE_TIMEOUT = 900

-

 # Maximum number of days to retain logged changes. Set to 0 to retain changes indefinitely. (Default: 90)

 CHANGELOG_RETENTION = 90

 

@@ -72,7 +58,7 @@

 # CORS_ORIGIN_REGEX_WHITELIST. For more information, see https://github.com/ottoyiu/django-cors-headers

 CORS_ORIGIN_ALLOW_ALL = False

 CORS_ORIGIN_WHITELIST = [

-    # 'https://hostname.example.com',

+    # 'hostname.example.com',

 ]

 CORS_ORIGIN_REGEX_WHITELIST = [

     # r'^(https?://)?(\w+\.)?example\.com$',

@@ -97,14 +83,6 @@

 # (all prefixes and IP addresses not assigned to a VRF), set ENFORCE_GLOBAL_UNIQUE to True.

 ENFORCE_GLOBAL_UNIQUE = False

 

-# Exempt certain models from the enforcement of view permissions. Models listed here will be viewable by all users and

-# by anonymous users. List models in the form `<app>.<model>`. Add '*' to this list to exempt all models.

-EXEMPT_VIEW_PERMISSIONS = [

-    # 'dcim.site',

-    # 'dcim.region',

-    # 'ipam.prefix',

-]

-

 # Enable custom logging. Please see the Django documentation for detailed guidance on configuring custom logs:

 #   https://docs.djangoproject.com/en/1.11/topics/logging/

 LOGGING = {}

@@ -129,12 +107,9 @@

 # the default value of this setting is derived from the installed location.

 # MEDIA_ROOT = '/opt/netbox/netbox/media'

 

-# Expose Prometheus monitoring metrics at the HTTP endpoint '/metrics'

-METRICS_ENABLED = False

-

 # Credentials that NetBox will uses to authenticate to devices when connecting via NAPALM.

-NAPALM_USERNAME = ''

-NAPALM_PASSWORD = ''

+NAPALM_USERNAME = 'tapadmin'

+NAPALM_PASSWORD = 'WHFk7hpW'

 

 # NAPALM timeout (in seconds). (Default: 30)

 NAPALM_TIMEOUT = 30

@@ -150,6 +125,17 @@

 # prefer IPv4 instead.

 PREFER_IPV4 = False

 

+# Redis database settings (optional). A Redis database is required only if the webhooks backend is enabled.

+REDIS = {

+    'HOST': 'localhost',

+    'PORT': 6379,

+    'PASSWORD': '',

+    'DATABASE': 0,

+    'CACHE_DATABASE': 1,

+    'DEFAULT_TIMEOUT': 300,

+    'SSL': False,

+}

+

 # The file path where custom reports will be stored. A trailing slash is not needed. Note that the default value of

 # this setting is derived from the installed location.

 # REPORTS_ROOT = '/opt/netbox/netbox/reports'

@@ -160,11 +146,11 @@

 SESSION_FILE_PATH = None

 

 # Time zone (default: UTC)

-TIME_ZONE = 'UTC'

+TIME_ZONE = 'Asia/Shanghai'

 

 # The webhooks backend is disabled by default. Set this to True to enable it. Note that this requires a Redis

 # database be configured and accessible by NetBox.

-WEBHOOKS_ENABLED = False

+WEBHOOKS_ENABLED = True

 

 # Date/time formatting. See the following link for supported formats:

 # https://docs.djangoproject.com/en/dev/ref/templates/builtins/#date

在 2019年9月10日星期二 UTC+8下午3:03:31，Brian Candler写道：

[caixing yang]

Hi Brian, 

   And BTW, my dingtalk webhook need msgtype is text to post data, so do you know how to grab the Args data content, and sent it to dingtalk, thanks . 

在 2019年9月10日星期二 UTC+8下午3:42:24，caixing yang写道：

[Brian Candler]

As I said before: the POST content from Netbox webhooks is not customizable.

You would need to write your own webhook server, which accepts an incoming HTTP POST, reformats the data, and POSTs it onward to a different URL.