Missing IP addresses in API call

[Bobola Oke]

Hello,

Running netbox version 2.10.8

I upgraded to v2.10.8 from v2.10.4 to see if the issue persists and looks like it does:

IP addresses created are not returned using the API calls.

I noticed that if I filter with a Parent Prefix of /24, the seeming missing IP addresses are returned as expected.

The tested netbox installation has over 49k IP address entries. If all the records are requested via API, the most recent IP addresses are not returned. 

The count in the API is correct, the length of the list ['results'] is less than the count.

Linux netbox-demo 4.15.0-140-generic #144-Ubuntu SMP Fri Mar 19 14:12:35 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
root@netbox-demo:/opt/netbox# python3
Python 3.6.9 (default, Jan 26 2021, 15:33:00)
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> print(sys.version)
3.6.9 (default, Jan 26 2021, 15:33:00)
[GCC 8.4.0]
>>> 

Please let me know if you need more information and if there are further tests I need to perform on my end to clarify the issue.

Thanks

[Brian Candler]

> The count in the API is correct, the length of the list ['results'] is less than the count.

What's the exact length of the 'results' list you get?  It sounds to me like you are hitting pagination.

https://netbox.readthedocs.io/en/stable/rest-api/overview/#pagination

[Bobola Oke]

Hi Brian,

Quickly: Alot of your [previous] discussions on this mailing list have been an excellent resource to help me find fixes and best practices for netbox. Thank you!

I guess it really does sound like pagination from the way I had described it but it is not. 

I should have added that I am requesting 'next' result values and updating the list until 'next' is null - if that makes sense..

The difference in the length of ['results'] and 'count' is the number of newly created IP addresses.

Basically, only the newer IP addresses that are created are not being returned.

Further tests I did showed it could be a LXC related issue:

The original netbox is running in a container and I create a copy of the original to run dev on.

Turns out that the original netbox returns the complete ipaddr values (count and length is the same) after new IP addresses are created.

The copy does not exhibit the same (expected) behavior.

I am probably missing some knowledge on the internal workings of netbox/LXC which is causing this perceived strangeness... but on the slight chance it could develop into a bug on the netbox side, I will leave this message in the public domain.

If you or anyone else have any theories on this, I'd be happy to read it.

Thanks again,

--
You received this message because you are subscribed to the Google Groups "NetBox" group.
To unsubscribe from this group and stop receiving emails from it, send an email to netbox-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/netbox-discuss/31bfdb8e-ca7a-4d94-8726-81955c3f6954n%40googlegroups.com.

[Brian Candler]

Could it be caching?  Have you tried waiting 15 minutes, or explicitly invalidating the cache, after creating new entries?

cd /opt/netbox

source venv/bin/activate

python3 netbox/manage.py invalidate all

If that's the problem, it would still be a bug in Netbox - hence it would be useful if you post the exact API calls that you're using to create IPs and to query them.

I run Netbox under lxd, and it shouldn't make any difference.

[Bobola Oke]

Hi Brian,

I suspect it could be caching.

The invalidate all script did not seem to improve the situation.

In summary the issue is this:

1. Copy existing netbox container (with approx 50k IP addresses) as netbox-demo 
2. Create IP address in netbox-demo
3. Query for all IP addresses in netbox-demo returns all values except for newly created IP address
4. Query for the specific IP address returns correctly
5. python3 netbox/manage.py invalidate all seems to make no difference
6. Issue not present on original netbox container.

Thanks again for your pointers. Since it looks like this issue only occurs on the container copy, it may not be a *real* issue after all.

I guess could be interesting for people who make LXC copies as a dev environment

Of course, if I find any new info/behavior, I will update this thread.

Regards,

--

You received this message because you are subscribed to the Google Groups "NetBox" group.
To unsubscribe from this group and stop receiving emails from it, send an email to netbox-discus...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/netbox-discuss/00d8c279-96e7-4b87-88f7-0c1cd2cac1dan%40googlegroups.com.

[Brian Candler]

It's definitely nothing to do with lxc.

I suggest your next step should be to do raw SQL queries to see if the ipam_ipaddress rows have actually appeared in the database or not.  It could be something strange like your database clone is read-only - although I would have hoped that errors on insertion to the database would be returned as errors in the API calls.

[Bobola Oke]

Brian,

The IP addresses appear in the Postgres DB when queried directly. 

Also remember that I can successfully request these IP addresses when using filters. So it has it stored somehow.

It is just somehow excluded when I request the entire IP addresses.

Not really sure how to query the Redis Cache. But there are some keys in DB 1

Also, not sure if there is a quick way to run netbox without Redis to exclude the errors on the cache:

On the configuration.py, setting the REDIS var to an empty dictionary 'REDIS = {}' and the netbox does not start.

REDIS configuration snippet below:

#REDIS = {}

REDIS = {
    'tasks': {
        'HOST': 'localhost',
        'PORT': 6379,
        'PASSWORD': '',
        'DATABASE': 0,
        'DEFAULT_TIMEOUT': 300,
        'SSL': False,
    },
    'caching': {
        'HOST': 'localhost',
        'PORT': 6379,
        'PASSWORD': '',
        'DATABASE': 1,
        'DEFAULT_TIMEOUT': 300,
        'SSL': False,
    }
}

Thanks,

On Mon, Mar 29, 2021 at 10:11 AM Brian Candler <b.ca...@pobox.com> wrote:

It's definitely nothing to do with lxc.

I suggest your next step should be to do raw SQL queries to see if the ipam_ipaddress rows have actually appeared in the database or not.  It could be something strange like your database clone is read-only - although I would have hoped that errors on insertion to the database would be returned as errors in the API calls.

--

You received this message because you are subscribed to the Google Groups "NetBox" group.
To unsubscribe from this group and stop receiving emails from it, send an email to netbox-discus...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/netbox-discuss/d343c3f2-428e-46b4-9561-a8d7980e6bb8n%40googlegroups.com.

[Bobola Oke]

Hi again Brian,

Your first response was actually the right one. 

Also your follow up response on posting the exact calls to query netbox helped me find the issue -  It was indeed pagination related..

I have a clone netbox but only updated the IP and DNS name for accessing the clone box. Everything works fantastic except when requesting pagination data.

What I missed was I needed to update configuration.py and the nginx configuration files to reflect the new hostname.

Because of my setup, NOT doing this meant the pagination URL passed to me was that of the original server like below:

ORIGINAL REQ URL: http://netbox-demo.example.com/api/ipam/ip-addresses/?limit=400

NEXT VALUE URL: http://netbox.example.com/api/ipam/ip-addresses/?limit=400&offset=400

So of course I do not see the new values created on the demo server!

WIth your help, it only took about three days to finally figure out. :D... could have easily been over a month or probably never if I did not discuss with someone!

Thank you for your help.

Regards,

[Brian Candler]

Great, glad it's sorted now.  Netbox constructs absolute URLs using the hostname in the Host: or X-Forwarded-Host: header in the incoming request, so normally you'd get the proxy to pass that on unchanged, as per the sample configs:

    location / {

        proxy_pass http://127.0.0.1:8001;

        proxy_set_header X-Forwarded-Host $http_host;

        proxy_set_header X-Real-IP $remote_addr;

        proxy_set_header X-Forwarded-Proto $scheme;

    }

Setting a different API key on the demo server might be a good idea too.