API for dcim/devices takes too long!

366 views
Skip to first unread message

Ashith Bolar

unread,
Apr 27, 2020, 6:34:19 PM4/27/20
to NetBox
We have about 35,000 devices. We have set the MAX_PAGE_SIZE to 500, since that's the highest value for which requests consistently come back with an http status code of 200. Anything higher times out to either 502 or 504.

Here are the times for various lower limits. They seem to take way longer than usual.

$ time curl -s -H "Content-Type: application/json" -H "Accept: application/json; indent=4" http://netbox.domain.com/api/dcim/devices/?limit=50 > /dev/null

real    0m6.874s
user    0m0.008s
sys     0m0.012s

$ time curl -s -H "Content-Type: application/json" -H "Accept: application/json; indent=4" http://netbox.domain.com/api/dcim/devices/?limit=100 > /dev/null

real    0m7.383s
user    0m0.012s
sys     0m0.008s

$ time curl -s -H "Content-Type: application/json" -H "Accept: application/json; indent=4" http://netbox.domain.com/api/dcim/devices/?limit=200 > /dev/null

real    0m13.565s
user    0m0.020s
sys     0m0.000s

$ time curl -s -H "Content-Type: application/json" -H "Accept: application/json; indent=4" http://netbox.domain.com/api/dcim/devices/?limit=500 > /dev/null

real    0m37.734s
user    0m0.012s
sys     0m0.012s

By way of pulling 500 devices per page, it takes about 30-45 minutes to pull all 30,000 records.

Brian Candler

unread,
Apr 28, 2020, 4:08:02 AM4/28/20
to NetBox
What Netbox version?

Are you using config contexts?  Does it make a difference to add &exclude=config_context to the URL? (Remember to wrap the whole URL in quotes if using curl)

AFAICS this parameter is not in the documentation, but it's mentioned in the release notes for v2.6.0:

Ashith Bolar

unread,
Apr 28, 2020, 12:21:06 PM4/28/20
to NetBox
Thanks for the response!
 
What Netbox version?
2.7.12

I am new to netbox. Did not know it existed. With the exclude=config_context, the queries most definitely seem much quicker. I am getting a results back in 2-5 seconds instead of 40 seconds. Now I guess the question is how to set/add this parameter in my pynetbox call? nb.dcim.devices.all/get

Brian Candler

unread,
Apr 28, 2020, 1:33:15 PM4/28/20
to NetBox
>>> nb.dcim.devices.all()[0].config_context
{'snmp_mibs': ['if_mib_secret']}
>>> nb.dcim.devices.filter(exclude="config_context")[0].config_context
<class 'pynetbox.core.response.JsonField'>

I can see from tcpdump the second one is generating:
GET /api/dcim/devices/?exclude=config_context
which is what you want.

I can't see how to do it for a single device:

>>> nb.dcim.devices.get(29, exclude="config_context").config_context
{'snmp_mibs': ['if_mib_secret']}

just generates
GET /api/dcim/devices/29/

Tomoc Field

unread,
Apr 28, 2020, 6:57:26 PM4/28/20
to Brian Candler, NetBox
This was actually really helpful for me. Smaller install, but dropped request time for all devices from 58s to 11s

--
You received this message because you are subscribed to the Google Groups "NetBox" group.
To unsubscribe from this group and stop receiving emails from it, send an email to netbox-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/netbox-discuss/91522f81-aeba-4bc8-b159-174ba0312b43%40googlegroups.com.

Ashith Bolar

unread,
Apr 29, 2020, 12:09:14 AM4/29/20
to NetBox
That worked! Thank you, Brian!
Reply all
Reply to author
Forward
0 new messages