API for dcim/devices takes too long!

[Ashith Bolar]

We have about 35,000 devices. We have set the MAX_PAGE_SIZE to 500, since that's the highest value for which requests consistently come back with an http status code of 200. Anything higher times out to either 502 or 504.

Here are the times for various lower limits. They seem to take way longer than usual.

$ time curl -s -H "Content-Type: application/json" -H "Accept: application/json; indent=4" http://netbox.domain.com/api/dcim/devices/?limit=50 > /dev/null

real    0m6.874s

user    0m0.008s

sys     0m0.012s

$ time curl -s -H "Content-Type: application/json" -H "Accept: application/json; indent=4" http://netbox.domain.com/api/dcim/devices/?limit=100 > /dev/null

real    0m7.383s

user    0m0.012s

sys     0m0.008s

$ time curl -s -H "Content-Type: application/json" -H "Accept: application/json; indent=4" http://netbox.domain.com/api/dcim/devices/?limit=200 > /dev/null

real    0m13.565s

user    0m0.020s

sys     0m0.000s

$ time curl -s -H "Content-Type: application/json" -H "Accept: application/json; indent=4" http://netbox.domain.com/api/dcim/devices/?limit=500 > /dev/null

real    0m37.734s

user    0m0.012s

sys     0m0.012s

By way of pulling 500 devices per page, it takes about 30-45 minutes to pull all 30,000 records.

[Brian Candler]

What Netbox version?

Are you using config contexts?  Does it make a difference to add &exclude=config_context to the URL? (Remember to wrap the whole URL in quotes if using curl)

AFAICS this parameter is not in the documentation, but it's mentioned in the release notes for v2.6.0:

https://github.com/netbox-community/netbox/releases/tag/v2.6.0

[Ashith Bolar]

Thanks for the response!

 

What Netbox version?

2.7.12

I am new to netbox. Did not know it existed. With the exclude=config_context, the queries most definitely seem much quicker. I am getting a results back in 2-5 seconds instead of 40 seconds. Now I guess the question is how to set/add this parameter in my pynetbox call? nb.dcim.devices.all/get

[Brian Candler]

>>> nb.dcim.devices.all()[0].config_context

{'snmp_mibs': ['if_mib_secret']}

>>> nb.dcim.devices.filter(exclude="config_context")[0].config_context

<class 'pynetbox.core.response.JsonField'>

I can see from tcpdump the second one is generating:

GET /api/dcim/devices/?exclude=config_context

which is what you want.

I can't see how to do it for a single device:

>>> nb.dcim.devices.get(29, exclude="config_context").config_context

{'snmp_mibs': ['if_mib_secret']}

just generates

GET /api/dcim/devices/29/

[Tomoc Field]

This was actually really helpful for me. Smaller install, but dropped request time for all devices from 58s to 11s

--
You received this message because you are subscribed to the Google Groups "NetBox" group.
To unsubscribe from this group and stop receiving emails from it, send an email to netbox-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/netbox-discuss/91522f81-aeba-4bc8-b159-174ba0312b43%40googlegroups.com.

[Ashith Bolar]

That worked! Thank you, Brian!