Incorrect Padding exception 2.9.8

[Jaime Botello]

Hi, I'm trying to troubleshoot the following exception that started today in our production netbox system

<class 'binascii.Error'>
Incorrect padding Python version: 3.7.9 NetBox version: 2.9.8

Any pointers would be appreciated

thanks

[Brian Candler]

Please say what action triggers this error.

Please also show the full backtrace - you can get this by setting DEBUG=True in configuration.py, and/or setting ADMINS and EMAIL so that an exception is automatically mailed out.

[Sam Morris]

I have seen this when changing SECRET_KEY without either clearing my cookies or deleting all sessions from the database (I can't remember which one). It looked to me like the new value of SECRET_KEY caused validation of either cookie or session (can't remember which) data to fail. And as part of a fallback code path in handling that exception, some value that was not a valid Base64 string was being base64-decoded, resulting in this exception being thrown.

Sorry I don't have all the details to hand but this was easy to reproduce by using netbox for a bit, stopping netbox, then changing SECRET_KEY, starting netbox again and then finally trying to access netbox again.

On Wednesday, 25 November 2020 at 05:29:52 UTC Jaime Botello wrote:

[Brian Candler]

SECRET_KEY is not something that's intended to be changed frequently.  As you said, it's used to encrypt or sign cookies, so any existing cookies will be invalid if you change it - and AFAIK Django doesn't provide any built-in mechanism for smooth rotation, although apparently there's a third-party module which permits it:

https://medium.com/@bayraktar.eralp/changing-rotating-django-secret-key-without-logging-users-out-804a29d3ea65
https://stackoverflow.com/questions/15170637/effects-of-changing-djangos-secret-key

Having said that, it shouldn't crash.

I can confirm that adding an 'X' to the end of my SECRET_KEY and then restarting Netbox causes the same problem.  The backtrace shows whether the exception is raised within Django itself or within some Netbox code.  I get:

Internal Server Error: /

Error at /

Incorrect padding

Request Method: GET

Request URL: http://netbox.example.net/

Django Version: 3.1

Python Executable: /opt/netbox/venv/bin/python3

Python Version: 3.6.9

Python Path: ['/opt/netbox/netbox', '/opt/netbox/venv/bin', '/usr/lib/python36.zip', '/usr/lib/python3.6', '/usr/lib/python3.6/lib-dynload', '/opt/netbox/venv/lib/python3.6/site-packages']

Server time: Thu, 26 Nov 2020 07:58:45 +0000

Installed Applications:

['django.contrib.admin',

 'django.contrib.auth',

 'django.contrib.contenttypes',

 'django.contrib.sessions',

 'django.contrib.messages',

 'django.contrib.staticfiles',

 'django.contrib.humanize',

 'cacheops',

 'corsheaders',

 'debug_toolbar',

 'django_filters',

 'django_tables2',

 'django_prometheus',

 'mptt',

 'rest_framework',

 'taggit',

 'timezone_field',

 'circuits',

 'dcim',

 'ipam',

 'extras',

 'secrets',

 'tenancy',

 'users',

 'utilities',

 'virtualization',

 'django_rq',

 'drf_yasg']

Installed Middleware:

['debug_toolbar.middleware.DebugToolbarMiddleware',

 'django_prometheus.middleware.PrometheusBeforeMiddleware',

 'corsheaders.middleware.CorsMiddleware',

 'django.contrib.sessions.middleware.SessionMiddleware',

 'django.middleware.common.CommonMiddleware',

 'django.middleware.csrf.CsrfViewMiddleware',

 'django.contrib.auth.middleware.AuthenticationMiddleware',

 'django.contrib.messages.middleware.MessageMiddleware',

 'django.middleware.clickjacking.XFrameOptionsMiddleware',

 'django.middleware.security.SecurityMiddleware',

 'utilities.middleware.ExceptionHandlingMiddleware',

 'utilities.middleware.RemoteUserMiddleware',

 'utilities.middleware.LoginRequiredMiddleware',

 'utilities.middleware.APIVersionMiddleware',

 'extras.middleware.ObjectChangeMiddleware',

 'django_prometheus.middleware.PrometheusAfterMiddleware']

Traceback (most recent call last):

  File "/opt/netbox/venv/lib/python3.6/site-packages/django/contrib/sessions/backends/base.py", line 215, in _get_session

    return self._session_cache

During handling of the above exception ('SessionStore' object has no attribute '_session_cache'), another exception occurred:

  File "/opt/netbox/venv/lib/python3.6/site-packages/django/contrib/sessions/backends/base.py", line 118, in decode

    return signing.loads(session_data, salt=self.key_salt, serializer=self.serializer)

  File "/opt/netbox/venv/lib/python3.6/site-packages/django/core/signing.py", line 135, in loads

    base64d = TimestampSigner(key, salt=salt).unsign(s, max_age=max_age).encode()

  File "/opt/netbox/venv/lib/python3.6/site-packages/django/core/signing.py", line 201, in unsign

    result = super().unsign(value)

  File "/opt/netbox/venv/lib/python3.6/site-packages/django/core/signing.py", line 184, in unsign

    raise BadSignature('Signature "%s" does not match' % sig)

During handling of the above exception (Signature "zSKiK_CFV8srWdAp7Rz0LOQjd-SWiuwtnvgzRkIceU4" does not match), another exception occurred:

  File "/opt/netbox/venv/lib/python3.6/site-packages/django/core/handlers/exception.py", line 47, in inner

    response = get_response(request)

  File "/opt/netbox/venv/lib/python3.6/site-packages/django/core/handlers/base.py", line 179, in _get_response

    response = wrapped_callback(request, *callback_args, **callback_kwargs)

  File "/opt/netbox/venv/lib/python3.6/site-packages/django/views/generic/base.py", line 73, in view

    return self.dispatch(request, *args, **kwargs)

  File "/opt/netbox/venv/lib/python3.6/site-packages/django/views/generic/base.py", line 101, in dispatch

    return handler(request, *args, **kwargs)

  File "/opt/netbox/netbox/netbox/views.py", line 193, in get

    connected_consoleports = ConsolePort.objects.restrict(request.user, 'view').filter(

  File "/opt/netbox/venv/lib/python3.6/site-packages/django/db/models/manager.py", line 85, in manager_method

    return getattr(self.get_queryset(), name)(*args, **kwargs)

  File "/opt/netbox/netbox/utilities/querysets.py", line 22, in restrict

    if user.is_superuser or permission_is_exempt(permission_required):

  File "/opt/netbox/venv/lib/python3.6/site-packages/django/utils/functional.py", line 240, in inner

    self._setup()

  File "/opt/netbox/venv/lib/python3.6/site-packages/django/utils/functional.py", line 376, in _setup

    self._wrapped = self._setupfunc()

  File "/opt/netbox/venv/lib/python3.6/site-packages/django/contrib/auth/middleware.py", line 23, in <lambda>

    request.user = SimpleLazyObject(lambda: get_user(request))

  File "/opt/netbox/venv/lib/python3.6/site-packages/django/contrib/auth/middleware.py", line 11, in get_user

    request._cached_user = auth.get_user(request)

  File "/opt/netbox/venv/lib/python3.6/site-packages/django/contrib/auth/__init__.py", line 174, in get_user

    user_id = _get_user_session_key(request)

  File "/opt/netbox/venv/lib/python3.6/site-packages/django/contrib/auth/__init__.py", line 58, in _get_user_session_key

    return get_user_model()._meta.pk.to_python(request.session[SESSION_KEY])

  File "/opt/netbox/venv/lib/python3.6/site-packages/django/contrib/sessions/backends/base.py", line 65, in __getitem__

    return self._session[key]

  File "/opt/netbox/venv/lib/python3.6/site-packages/django/contrib/sessions/backends/base.py", line 220, in _get_session

    self._session_cache = self.load()

  File "/opt/netbox/venv/lib/python3.6/site-packages/django/contrib/sessions/backends/db.py", line 44, in load

    return self.decode(s.session_data) if s else {}

  File "/opt/netbox/venv/lib/python3.6/site-packages/django/contrib/sessions/backends/base.py", line 122, in decode

    return self._legacy_decode(session_data)

  File "/opt/netbox/venv/lib/python3.6/site-packages/django/contrib/sessions/backends/base.py", line 126, in _legacy_decode

    encoded_data = base64.b64decode(session_data.encode('ascii'))

  File "/usr/lib/python3.6/base64.py", line 87, in b64decode

    return binascii.a2b_base64(s)

Exception Type: Error at /

Exception Value: Incorrect padding

Request information:

USER: [unable to retrieve the current user]

GET: No GET data

POST: No POST data

FILES: No FILES data

Clearly there's an attempt to raise a BadSignature error there, but for some reason it's getting masked.

I think this is a bug in Django itself:

/opt/netbox/venv/lib/python3.6/site-packages/django/contrib/sessions/backends/base.py

line 126, in function _legacy_decode I think the base64.b64decode() should be inside the "try", not outside.

--- /opt/netbox/venv/lib/python3.6/site-packages/django/contrib/sessions/backends/base.py.orig 2020-11-26 08:19:00.345368064 +0000

+++ /opt/netbox/venv/lib/python3.6/site-packages/django/contrib/sessions/backends/base.py 2020-11-26 08:19:32.957434179 +0000

@@ -123,8 +123,9 @@

     def _legacy_decode(self, session_data):

         # RemovedInDjango40Warning: pre-Django 3.1 format will be invalid.

-        encoded_data = base64.b64decode(session_data.encode('ascii'))

         try:

+            # could produce binascii.Error

+            encoded_data = base64.b64decode(session_data.encode('ascii'))

             # could produce ValueError if there is no ':'

             hash, serialized = encoded_data.split(b':', 1)

             expected_hash = self._hash(serialized)

With this change, I see that the user is treated as not logged in if the SECRET_KEY is wrong (and when corrected, they are logged in again).  OTOH, I'm not sure it should really have gotten this far if the signature validation had failed.

If you want to raise this with the Django project feel free.  I don't have the time to make a standalone non-Netbox replication of this problem.

Regards,

Brian.