Establishing a maximum limit for API pagination

[Jeremy Stretch]

The NetBox API uses Django REST Framework's LimitOffsetPagination mechanism to paginate results returned by an API call. By default, NetBox will only return the first 50 objects in a query, but this limit is configurable per request. For example:

GET /api/dcim/devices/ - Returns devices 1-50
GET /api/dcim/devices/?offset=50 - Returns devices 51-100

GET /api/dcim/devices/?limit=100 - Returns devices 1-100
GET /api/dcim/devices/?limit=100&offset=100 - Returns devices 101-200

Issue #1176 was opened recently asking whether there's a way to disable pagination. While pagination cannot be explicitly disabled (e.g. by setting limit=0), there is no maximum limit to the number of objects that can be returned per request. So, pagination can be effectively disabled for a request by passing limit=9999999.

While this behavior may be desired in some cases, it also leaves open a potential avenue for very efficient denial of service attacks. A very small number of (potentially unauthenticated) requests can severely degrade performance by forcing NetBox to, for instance, retrieve tens of thousands of interface objects at once.

I'm curious what people think about hard-coding a maximum limit into the API pagination logic; perhaps 10,000. Any limit greater than this cut-off passed by a request would be ignored and the default number of objects returned. Thoughts?

Jeremy

[Keenan Tims]

Maybe add a permission or flag for 'API users' that disables
pagination or allows unlimited result limits? I think not providing a
way to get unpaginated results from API calls has the potential to
create a lot of friction for easily whacking together custom reports
and the like by bolting together existing libraries. Given the
constrained scope of NetBox, that's going to be almost inevitable for
a lot of users to integrate with other tools like DNS, config
management, etc. If that solution is not suitable, my preference would
be to leave it unlimited; I see the (possibly unexpected and resulting
in unnoticed missing data) complexity it adds to consuming the API
worse than the risk of DDoS for something that will be a firewalled
and authenticated internal service in any sane deployment.

Keenan

Quoting 'Jeremy Stretch' via NetBox <netbox-...@googlegroups.com>:

> The NetBox API uses Django REST Framework's LimitOffsetPagination

> <http://www.django-rest-framework.org/api-guide/pagination/#limitoffsetpagination>

> mechanism to paginate results returned by an API call. By default, NetBox
> will only return the first 50 objects in a query, but this limit is
> configurable per request. For example:
>
> GET /api/dcim/devices/ - Returns devices 1-50
> GET /api/dcim/devices/?offset=50 - Returns devices 51-100
>
> GET /api/dcim/devices/?limit=100 - Returns devices 1-100
> GET /api/dcim/devices/?limit=100&offset=100 - Returns devices 101-200
>

> Issue #1176 <https://github.com/digitalocean/netbox/issues/1176> was opened

> recently asking whether there's a way to disable pagination. While
> pagination cannot be explicitly disabled (e.g. by setting limit=0), there
> is no maximum limit to the number of objects that can be returned per
> request. So, pagination can be effectively disabled for a request by
> passing limit=9999999.
>
> While this behavior may be desired in some cases, it also leaves open a
> potential avenue for very efficient denial of service attacks. A very small
> number of (potentially unauthenticated) requests can severely degrade
> performance by forcing NetBox to, for instance, retrieve tens of thousands
> of interface objects at once.
>
> I'm curious what people think about hard-coding a maximum limit into the
> API pagination logic; perhaps 10,000. Any limit greater than this cut-off
> passed by a request would be ignored and the default number of objects
> returned. Thoughts?
>
> Jeremy
>

> --
> You received this message because you are subscribed to the Google
> Groups "NetBox" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to netbox-discus...@googlegroups.com.
> To post to this group, send email to netbox-...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/netbox-discuss/CADvjYzohxXWEr0u-T72vE1gSXVk%2B7wt7XcvNG_KeHXJehP2ECA%40mail.gmail.com.
> For more options, visit https://groups.google.com/d/optout.

[Jeremy Stretch]

Maybe a better approach would be to implement it as a configuration setting that a user can adjust as they see fit. We could default the limit to something relatively low, perhaps 1000, and the admin can adjust this or disable it entirely in configuration.py.

On Tue, May 16, 2017 at 5:51 PM, Keenan Tims <kt...@gotroot.ca> wrote:

Maybe add a permission or flag for 'API users' that disables pagination or allows unlimited result limits? I think not providing a way to get unpaginated results from API calls has the potential to create a lot of friction for easily whacking together custom reports and the like by bolting together existing libraries. Given the constrained scope of NetBox, that's going to be almost inevitable for a lot of users to integrate with other tools like DNS, config management, etc. If that solution is not suitable, my preference would be to leave it unlimited; I see the (possibly unexpected and resulting in unnoticed missing data) complexity it adds to consuming the API worse than the risk of DDoS for something that will be a firewalled and authenticated internal service in any sane deployment.

Keenan

To unsubscribe from this group and stop receiving emails from it, send an email to netbox-discuss+unsubscribe@googlegroups.com.

To post to this group, send email to netbox-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/netbox-discuss/CADvjYzohxXWEr0u-T72vE1gSXVk%2B7wt7XcvNG_KeHXJehP2ECA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "NetBox" group.

To unsubscribe from this group and stop receiving emails from it, send an email to netbox-discuss+unsubscribe@googlegroups.com.

To post to this group, send email to netbox-...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/netbox-discuss/20170516215138.Horde.IUMcnGNZ9P3JebmRygEDqoJ%40zero.gotroot.ca.