login error even with SESSION_FILE_PATH on hot standby instance -- tries to update last_login

[Kartik Subbarao]

I'm running netbox 2.5.9 on Ubuntu 16.04 with postgresql 9.5 set up as a hot standby replica. I have SESSION_FILE_PATH set to /var/run/netbox and have verified that netbox creates session files successfuly. When I try to login, I get the following exception:

<class 'django.db.utils.InternalError'>

cannot execute UPDATE in a read-only transaction

Below is the full stack trace. The key line that jumps out to me is this one:

File "/usr/local/lib/python3.5/dist-packages/django/contrib/auth/models.py", line 20, in update_last_login
    user.save(update_fields=['last_login'])

It looks like django is trying to update the last_login field in the database, which leads to the read-only error. Has anyone else run into this problem? Any suggestions would be greatly appreciated.

Thanks,

    -Kartik

=====

Internal Server Error: /login/
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/django/db/backends/utils.py", line 85, in _execute
    return self.cursor.execute(sql, params)
psycopg2.InternalError: cannot execute UPDATE in a read-only transaction


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/django/core/handlers/exception.py", line 34, in inner
    response = get_response(request)
  File "/usr/local/lib/python3.5/dist-packages/django/core/handlers/base.py", line 126, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/usr/local/lib/python3.5/dist-packages/django/core/handlers/base.py", line 124, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/local/lib/python3.5/dist-packages/django/views/generic/base.py", line 68, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/django/utils/decorators.py", line 45, in _wrapper
    return bound_method(*args, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/django/views/decorators/debug.py", line 76, in sensitive_post_parameters_wrapper
    return view(request, *args, **kwargs)
  File "/opt/netbox/netbox/users/views.py", line 29, in dispatch
    return super().dispatch(*args, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/django/views/generic/base.py", line 88, in dispatch
    return handler(request, *args, **kwargs)
  File "/opt/netbox/netbox/users/views.py", line 48, in post
    auth_login(request, form.get_user())
  File "/usr/local/lib/python3.5/dist-packages/django/contrib/auth/__init__.py", line 132, in login
    user_logged_in.send(sender=user.__class__, request=request, user=user)
  File "/usr/local/lib/python3.5/dist-packages/django/dispatch/dispatcher.py", line 175, in send
    for receiver in self._live_receivers(sender)
  File "/usr/local/lib/python3.5/dist-packages/django/dispatch/dispatcher.py", line 175, in <listcomp>
    for receiver in self._live_receivers(sender)
  File "/usr/local/lib/python3.5/dist-packages/django/contrib/auth/models.py", line 20, in update_last_login
    user.save(update_fields=['last_login'])
  File "/usr/local/lib/python3.5/dist-packages/django/contrib/auth/base_user.py", line 73, in save
    super().save(*args, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/django/db/models/base.py", line 718, in save
    force_update=force_update, update_fields=update_fields)
  File "/usr/local/lib/python3.5/dist-packages/django/db/models/base.py", line 748, in save_base
    updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
  File "/usr/local/lib/python3.5/dist-packages/django/db/models/base.py", line 812, in _save_table
    forced_update)
  File "/usr/local/lib/python3.5/dist-packages/django/db/models/base.py", line 861, in _do_update
    return filtered._update(values) > 0
  File "/usr/local/lib/python3.5/dist-packages/django/db/models/query.py", line 712, in _update
    return query.get_compiler(self.db).execute_sql(CURSOR)
  File "/usr/local/lib/python3.5/dist-packages/django/db/models/sql/compiler.py", line 1383, in execute_sql
    cursor = super().execute_sql(result_type)
  File "/usr/local/lib/python3.5/dist-packages/django/db/models/sql/compiler.py", line 1065, in execute_sql
    cursor.execute(sql, params)
  File "/usr/local/lib/python3.5/dist-packages/django/db/backends/utils.py", line 68, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/usr/local/lib/python3.5/dist-packages/django/db/backends/utils.py", line 77, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/usr/local/lib/python3.5/dist-packages/django/db/backends/utils.py", line 85, in _execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.5/dist-packages/django/db/utils.py", line 89, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/usr/local/lib/python3.5/dist-packages/django/db/backends/utils.py", line 85, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.InternalError: cannot execute UPDATE in a read-only transaction

Not Found: /favicon.ico

[rossm...@gmail.com]

Yes, I tried to do exactly the same thing and had exactly the same experience. I got around the issue my simply build a second netbox server and performing daily DB restores to it from the primary. So long as you upgrade both on the same day you are gold.

See here. https://forum.level1techs.com/t/netbox-ipam-dcim-guide/132435/3 section IV

[Kartik Subbarao]

Thanks for the datapoint Ross. I'm wondering what the difference is between our environments and the netbox developers' environments. I'm hoping one of them can chime in with some suggestions.

Regards,

    -Kartik

--
You received this message because you are subscribed to a topic in the Google Groups "NetBox" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/netbox-discuss/1tYF9d-wRl8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to netbox-discus...@googlegroups.com.
To post to this group, send email to netbox-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/netbox-discuss/699ded9d-b197-4f9f-ada3-32c642c4baa9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Brian Candler]

In a replicated database setup, one side is the master and the other is the replica.  You have to point the application to the master side, otherwise it won't be able to write.  In this case, as you've seen, Netbox can't update metadata about last login.

If you want to avoid Netbox writing this metadata, then you could raise a feature request for a way to disable the last login writes.  But of course, this will destroy the validity of the access audit trail, since people will be able to login to the replica Netbox instance and query it, without you having any record of them having done so.

So the normal approach would be to have users login to the master, and replicate to another instance which isn't used.  If the master fails, you'd promote the replica to master and then get people to login to the other instance.

[Kartik Subbarao]

Thanks for the explanation Brian. I still think I'm missing something though. The 2.5.0 release announcement has the following:

#2426 - Introduced SESSION_FILE_PATH configuration setting for authentication without write access to database

This, along with the description of SESSION_FILE_PATH in the netbox docs ( https://netbox.readthedocs.io/en/stable/configuration/optional-settings/#session_file_path ), seem to strongly imply that Jeremy's intent in adding this feature was to allow users to login to replica instances without having to point to the master database. I don't understand how SESSION_FILE_PATH fixes issue #2426 if it doesn't enable this. Jeremy, if you can chime in with your thoughts that'd be great :-)

For my current use case, an audit trail beyond what the apache access log already provides isn't important. The netbox enhancement that I'd actually prefer would be to enable cookie-based sessions (setting SESSION_ENGINE to "django.contrib.sessions.backends.signed_cookies"). That way, users could point to a netbox.example.com VIP in their browser. If the master crashed and keepalived flipped the VIP over to the replica, user sessions could transparently continue for read operations. Jeremy hinted at the possibility of exposing SESSION_ENGINE in the #2426 discussion. If this is practical to implement, I think it could be helpful for many users who want to deploy netbox in various ways.

Regards,

    -Kartik

--
You received this message because you are subscribed to a topic in the Google Groups "NetBox" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/netbox-discuss/1tYF9d-wRl8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to netbox-discus...@googlegroups.com.
To post to this group, send email to netbox-...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/netbox-discuss/73886a46-be1a-4e7a-9c57-0340fb1a9832%40googlegroups.com.

[Brian Candler]

Ah I see - I wasn't aware of that change.

It looks like update_last_login() is part of django contrib auth and I don't see an obvious way to disable it.

[Kartik Subbarao]

I'm not too familiar with django, but the following link looks promising:

https://stackoverflow.com/questions/49025407/in-django-1-11-how-to-allow-users-to-login-on-a-read-only-database

It refers to a python module called django-no-last-login which calls user_logged_in.disconnect(update_last_login) to disable the update:

https://github.com/MSA-Argentina/django-no-last-login/blob/master/nolastlogin/models.py#L11

On 4/6/2019 12:18 PM, Brian Candler wrote:

Ah I see - I wasn't aware of that change.

It looks like update_last_login() is part of django contrib auth and I don't see an obvious way to disable it.

--
You received this message because you are subscribed to a topic in the Google Groups "NetBox" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/netbox-discuss/1tYF9d-wRl8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to netbox-discus...@googlegroups.com.
To post to this group, send email to netbox-...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/netbox-discuss/6c89bb4b-2afe-409f-bf46-b28dda5d1798%40googlegroups.com.

[Kartik Subbarao]

Just as a followup -- I've updated the #2426 discussion and also created #3050 for the enhancement request to add cookie-based sessions:

https://github.com/digitalocean/netbox/issues/2426#issuecomment-480830172

https://github.com/digitalocean/netbox/issues/3050

Let me know if there's any more information that would be helpful on this topic.

Thanks,

    -Kartik