External Authentication - REMOTE_USER header

[Josh Miller]

Has anyone tried using an external authentication scheme that inserts the REMOTE_USER header?

I'm trying to use Kerberos to authenticate users while still using the built-in identity database and authorization controls. I've got Apache accepting Kerberos service tickets and it properly inserts the REMOTE_USER header with the user's User Principle Name (UPN), and have configured a user account in the built-in identity database named exactly the same as the user's UPN, being sure to match case. 

I followed the instructions in the Django documentation[1] and configured it to use the PersistentRemoteUserMiddleware middle-ware to maintain session persistence, because I configured Apache to only request authentication to /login/ and send a 401 response. Also, I added RemoteUserBackend in AUTHENTICATION_BACKENDS and kept the ModelBackend in there because I still wanted to allow form-based authentication for an administrator account. The end result is that the application behaves the same regardless of the presence or absence of the REMOTE_USER header, and prompts with form-based authentication.

Any thoughts? I appreciate any suggestions or assistance offered.

Josh

1. https://docs.djangoproject.com/en/1.11/howto/auth-remote-user/
   

[Shuichiro MAKIGAKI]

We need some small patches to enable remote_user auth in Netbox, but it works for me now.

In many cases, REMOTE_USER key in request header becomes HTTP_REMOTE_USER in Django middleware [1].

In addition to changing middleware and backends in configration.py, we have to add new custom middleware python file to use HTTP_REMOTE_USER instead of REMOTE_USER. (Follow the documents around “Warning” in the same link [1])

I added netbox/netbox/middleware.py before running django server.

1. https://docs.djangoproject.com/en/1.11/howto/auth-remote-user/#configuration

Regards,

Makkie

2017/09/21 6:35、Josh Miller <cont...@gmail.com>のメール:

--
You received this message because you are subscribed to the Google Groups "NetBox" group.
To unsubscribe from this group and stop receiving emails from it, send an email to netbox-discus...@googlegroups.com.
To post to this group, send email to netbox-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/netbox-discuss/4995713c-811e-4696-8b23-be4d543bd93f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Brian Candler]

> We need some small patches to enable remote_user auth in Netbox, but it works for me now.

Can you share your patches? I am very interested in protecting Netbox with mod_auth_openidc to limit access to a specific G-Suite domain, and in that case, I'd like to use the HTTP_OIDC_CLAIM_EMAIL header (or another OIDC claim) to identify the user in Netbox to avoid a second level of login.

Regards,

Brian. 

[Brian Candler]

In fact, mod_auth_openidc can set the REMOTE_USER to whatever is required:

https://github.com/pingidentity/mod_auth_openidc/blob/master/auth_openidc.conf#L362

So any patch which makes REMOTE_USER work for Netbox behind an Apache reverse proxy should do the job.

[Josh Miller]

I got it to work last weekend.

It seems REMOTE_USER isn't really intended to be passed as an HTTP header, but as an environment variable. I don't understand yet how Apache is supposed to place the header into an environment variable in WSGI environment when it only communicates with it over a tcp port.

HTTP headers with underscores are removed by the frontend webserver, Apache or NGINX, for security reasons. When HTTP headers are placed into the WSGI environment all dashes converted to underscores and prefixed with "HTTP_".

I decided to subclass RemoteUserMiddleware to manually configure the HTTP header 'HTTP_KERBEROS_USER'. I then configured Apache to convert the REMOTE_USER environment variable to 'KERBEROS-USER' HTTP header. WSGI will then convert the 'KERBEROS-USER' header into the 'HTTP_KERBEROS_USER' environment variable. I also have Apache strip the header on client requests for security.

By default, Django's RemoteUserMiddleware will create remote users that are active. I've started work on subclassing RemoteUserBackend to set these new users to inactive after creation. Later I plan to extract user details from Active Directory over LDAP on user creation.

[Brian Candler]

Thanks for that.  I wasn't aware that remote users were created automatically; I can't see this described at

https://docs.djangoproject.com/en/1.11/howto/auth-remote-user/

But that's basically what I want: new users added as regular users, so I can manage permissions on them using the local database.  Auto-populating things like first name, last name and E-mail from OIDC claims would a bit more of a challenge that is probably not worth the effort :-)

[Josh Miller]

The auto-populating from LDAP is really not that difficult. Python has modules to bind to LDAP and reference accounts by User Principal Name (UPN), which is what Apache provides from Kerberos authentication.

In the RemoteUserBackend there's a 'configure_user' method that is called after user account creation, and it's currently empty and simply returns the user unmodified. I plan on placing code in there to perform LDAP queries to auto-populate the appropriate fields. I'm not sure if this something that should sync on each login or if syncing email or name changes can be handed off to other modules.

[Josh Miller]

After some research I think I have a handle on how Apache is supposed to pass environment variables, specifically REMOTE_USER, to Django. 

Normally Apache doesn't require the use of ProxyPass (mod_proxy) and gunicorn to support WSGI applications, and cannot pass environment variables this way. The more direct approach is to use the WSGIScriptAlias Apache configuration command to run wsgi.py directly from Apache, and optionally run mod_wsgi as a separate daemon process (which is recommended). Apache then calls the function definition "application" and passes a dictionary containing variables from the Apache environment, as well as whatever is inherited from fork(). From my understanding, using the REMOTE_USER environment variable is ostensibly more secure than trusting an HTTP header for remote user authentication, because it's impossible to spoof an environment variable in an HTTP request header.

After some troubleshooting, I was finally able to get Netbox to work without using gunicorn and ProxyPass, and only with WSGI settings. As a result, I no longer need to subclass RemoteUserMiddleware to set a custom HTTP header; Django will use the REMOTE_USER environment variable.

[Josh Miller]

Brian,

I reread your post and realized I mistakenly thought you were talking about LDAP syncing. I now realize you were wanting to sync user details from OIDC claims, but thought it might not be worth the effort. Just for fun I decided to look into how difficult it would be.

There doesn't seem to be a way to pass environment variables from Apache to Netbox while using ProxyPass (mod_proxy) and Gunicorn. So the alternative to environment variables is for Apache to set HTTP headers to pass parameters into the Netbox environment. One has to take care to unset these headers in requests from end users, otherwise it could be possible to spoof the headers used in authentication. I prefer to use environment variables and changed my environment to use mod_wsgi and not ProxyPass(mod_proxy) and Gunicorn to avoid the possibility of spoofed HTTP headers.

HTTP headers passed into the WSGI environment, either by Gunicorn or mod_wsgi, are converted to upper case and are prefixed with "HTTP_" to avoid the possibility of spoofing or overwriting environment variables. So the HTTP header "Authorization" becomes "HTTP_AUTHORIZATION", and "REMOTE_USER" becomes "HTTP_REMOTE_USER" (although I couldn't get this one to work for some reason).

From my reading of the mod_auth_openidc documentation(1), UserInfo claims retrieved by Apache can be configured to pass UserInfo claims as individual headers/variables (prefixed with "OIDC_CLAIM_", or "HTTP_OIDC_CLAIM_" if passed as HTTP headers), a JSON object passed as the "OIDC_userinfo_json" header/environment variable, or signed/encrypted JWT. The second option, JSON, seems to be the easiest to use.

From there I would subclass RemoteUserBackend and name it OidcUserInfoSyncRemoteUserBackend, and override the authentication method and create a new private method "oidc_userinfo_sync". I'd still use the "REMOTE_USER" variable to retrieve the User Model from the Django user management backend, but I would include a call to the newly created "_oidc_userinfo_sync" method to sync user details from the UserInfo claims.

With all of this in mind I came up with the following. Keep in mind I haven't tested this "solution" so your mileage my vary. It's more than likely I'm forgetting something. If you decide to try it out, let me know how it goes.

1. https://github.com/zmartzone/mod_auth_openidc/blob/master/auth_openidc.conf

#Apache configuration

OIDCRemoteUserClaim email # Keep in mind some of the OIDC Providers (OP) don't guarantee the uniqueness of the email claim

RequestHeader set REMOTE_USER expr=%{REMOTE_USER}
OIDCPassUserInfoAs json
OIDCPassClaimsAs headers #I would prefer 'environment', but 'headers' should work with the existing ProxyPass configuration.

#netbox/netbox/settings.py file

AUTHENTICATION_BACKENDS = [
    'utilities.backends.OidcUserInfoSyncRemoteUserBackend', # ADD
    'django.contrib.auth.backends.ModelBackend',
]

MIDDLEWARE = (
    'debug_toolbar.middleware.DebugToolbarMiddleware',
    'corsheaders.middleware.CorsMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.common.CommonMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.contrib.auth.middleware.CustomHeaderRemoteUserMiddleware', # ADD. make sure this is after AuthenticationMiddleware and before SessionAuthenticationMiddleware
    'django.contrib.auth.middleware.SessionAuthenticationMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',
    'django.middleware.clickjacking.XFrameOptionsMiddleware',
    'django.middleware.security.SecurityMiddleware',
    'utilities.middleware.LoginRequiredMiddleware',
    'utilities.middleware.APIVersionMiddleware',
)

# You'll have to subclass RemoteUserMiddleware to change the header from 'REMOTE_USER' to 'HTTP_REMOTE_USER' if you want to keep the ProxyPass configuration and Gunicorn. Place this in netbox/utilities/middleware.py

class CustomHeaderRemoteUserMiddleware(RemoteUserMiddleware):
    header = "HTTP_REMOTE_USER"

#copy into the new netbox/utilities/backends.py file

import json
from django.contrib.auth.backends import RemoteUserBackend
from django.contrib.auth import get_user_model

UserModel = get_user_model()

class OidcUserInfoSyncRemoteUserBackend(RemoteUserBackend):
 
    #HTTP header
    USERINFO_JSON_HEADER = 'HTTP_OIDC_USERINFO_JSON'

    # Environment variable
    #USERINFO_JSON_HEADER = 'OIDC_userinfo_json'

    def _oidc_userinfo_sync(self, request, user):
        try:
            USERINFO_DICT = json.loads(getattr(request.META, USERINFO_JSON_HEADER, ''))
        except ValueError:
            USERINFO_DICT = ''

        if user and USERINFO_DICT:
            email = getattr(USERINFO_DICT, 'email', '')
            given_name = getattr(USERINFO_DICT, 'given_name', '')
            family_name = getattr(USERINFO_DICT, 'family_name', '')

            user.firstname, user.lastname, user.email = given_name, family_name, email
            user.save()

    def authenticate(self, request, remote_user):
        """
        The username passed as ``remote_user`` is considered trusted.  This
        method simply returns the ``User`` object with the given username,
        creating a new ``User`` object if ``create_unknown_user`` is ``True``.

        Returns None if ``create_unknown_user`` is ``False`` and a ``User``
        object with the given username is not found in the database.
        """
        if not remote_user:
            return
        user = None
        username = self.clean_username(remote_user)

        # Note that this could be accomplished in one try-except clause, but
        # instead we use get_or_create when creating unknown users since it has
        # built-in safeguards for multiple threads.
        if self.create_unknown_user:
            user, created = UserModel._default_manager.get_or_create(**{
                UserModel.USERNAME_FIELD: username
            })
            if created:
                user = self.configure_user(user)
        else:
            try:
                user = UserModel._default_manager.get_by_natural_key(username)
            except UserModel.DoesNotExist:
                pass
        #Sync user model to any claims present
        self._oidc_userinfo_sync(request, user)
        return user if self.user_can_authenticate(user) else None

[Josh Miller]

Forgot to return the updated user model:

    def _oidc_userinfo_sync(self, request, user):
        [...]
        return user

    def authenticate(self, request, remote_user):
        [...]
        user = self._oidc_userinfo_sync(request, user)

[Brian Candler]

Wonderfully detailed examples, thank you!  Integrating this is not top of my pile at the moment, but this will be very useful when I get around to it.

And an interesting example of how difficult real-world integration of apps into centralised authentication can be :-)

Cheers,

Brian.

[Brian Candler]

I finally got round to deploying this, and I had to make a few changes.

1. Change

   'django.contrib.auth.middleware.CustomHeaderRemoteUserMiddleware', # ADD. make sure this is after AuthenticationMiddleware and before SessionAuthenticationMiddleware

to

   'utilities.middleware.CustomHeaderRemoteUserMiddleware', # ADD. make sure this is after AuthenticationMiddleware and before SessionAuthenticationMiddleware

2. Add the missing import of RemoteUserMiddleware superclass:

from django.contrib.auth.middleware import RemoteUserMiddleware

class CustomHeaderRemoteUserMiddleware(RemoteUserMiddleware):
    header = "HTTP_REMOTE_USER"

3. To avoid a NameError I had to move this part out of the class and up to the top level:

    #HTTP header
    USERINFO_JSON_HEADER = 'HTTP_OIDC_USERINFO_JSON'

    # Environment variable
    #USERINFO_JSON_HEADER = 'OIDC_userinfo_json'

4. I got "configure_user() missing 1 required positional argument: 'user'" error on first login.  It needs 'request' as the first argument

5. request.META and USERINFO_DICT are dicts, not objects, so you need foo.get(...) instead of getattr(foo, ...)

6. User model attributes are first_name / last_name, not firstname / lastname

I ended up with this:

import json

from django.contrib.auth.backends import RemoteUserBackend

from django.contrib.auth import get_user_model

UserModel = get_user_model()

#HTTP header

USERINFO_JSON_HEADER = 'HTTP_OIDC_USERINFO_JSON'

# Environment variable

#USERINFO_JSON_HEADER = 'OIDC_userinfo_json'

class OidcUserInfoSyncRemoteUserBackend(RemoteUserBackend):

    def _oidc_userinfo_sync(self, request, user):

        try:

            USERINFO_DICT = json.loads(request.META.get(USERINFO_JSON_HEADER, ''))

        except ValueError:

            USERINFO_DICT = {}

        if user and USERINFO_DICT:

            user.email = USERINFO_DICT.get('email')

            user.first_name = USERINFO_DICT.get('given_name')

            user.last_name = USERINFO_DICT.get('family_name')

            user.save()

        return user

    def authenticate(self, request, remote_user):

        """

        The username passed as ``remote_user`` is considered trusted.  This

        method simply returns the ``User`` object with the given username,

        creating a new ``User`` object if ``create_unknown_user`` is ``True``.

        Returns None if ``create_unknown_user`` is ``False`` and a ``User``

        object with the given username is not found in the database.

        """

        if not remote_user:

            return

        user = None

        username = self.clean_username(remote_user)

        # Note that this could be accomplished in one try-except clause, but

        # instead we use get_or_create when creating unknown users since it has

        # built-in safeguards for multiple threads.

        if self.create_unknown_user:

            user, created = UserModel._default_manager.get_or_create(**{

                UserModel.USERNAME_FIELD: username

            })

            if created:

                user = self.configure_user(request, user)

        else:

            try:

                user = UserModel._default_manager.get_by_natural_key(username)

            except UserModel.DoesNotExist:

                pass

        #Sync user model to any claims present

        user = self._oidc_userinfo_sync(request, user)

        return user if self.user_can_authenticate(user) else None

Now it seems to work nicely - many thanks!!!

After doing a manual database update to apply is_staff and is_superuser to the first user, I can administer everything else.

There are some other things which would be nice to tidy up: e.g. I'd like to remove the "Log out" button from the user menu (since it's not possible to log out).  Plus, it's a bit painful that I've now forked netbox, but hopefully this work can be picked up in #2328

Cheers,

Brian.

[Brian Candler]

Aside: I do wonder whether it would be better to proxy using a protocol which explicitly carries environment variables - e.g. fastcgi, scgi or uwsgi.  One option would be Apache2 + mod_proxy_uwsgi at the frontend, and uwsgi instead of gunicorn at the backend.  However I have never tried that sort of stack.

[Brian Candler]

A fix to my fix is required.  For users who are missing given_name / family_name in the OIDC provider, you get an exception:

null value in column "first_name" violates not-null constraint

You need to add default values of empty string:

        if user and USERINFO_DICT:
            user.email = USERINFO_DICT.get('email', '')
            user.first_name = USERINFO_DICT.get('given_name', '')
            user.last_name = USERINFO_DICT.get('family_name', '')
            user.save()

[Joshua Miller]

Wow, I completely forgot about this. That's cool that you were able to get it to work.

I think you were getting NameError with the header variable because it was looking in the 'global' namespace; I should have specified the instance namespace by prefixing the variable with "self." Not that it matters, but it looks cleaner to me to keep it all together.

class OidcUserInfoSyncRemoteUserBackend(RemoteUserBackend):

    USERINFO_JSON_HEADER = 'HTTP_OIDC_USERINFO_JSON'

    def _oidc_userinfo_sync(self, request, user):

       try:

           USERINFO_DICT = json.loads(request.META.get(self.USERINFO_JSON_HEADER, ''))

       except ValueError:

           USERINFO_DICT = {}

Did you find anything about including authorization information inside the JSON object?

[Brian Candler]

On Monday, 3 June 2019 21:06:43 UTC+1, Joshua Miller wrote:

Did you find anything about including authorization information inside the JSON object?

You mean e.g. group memberships?

I haven't looked at it for a long time, but I seem to remember it varies quite a lot between OIDC providers.  e.g. Azure Active Directory can give a list of group UUIDs, but it seems Google does not:

https://stackoverflow.com/questions/35292406/to-add-google-group-as-claims-into-a-token

https://developers.google.com/identity/protocols/OpenIDConnect

I note also from the latter I should probably be using "sub" for the username, not the email (although that isn't very pretty to display in the menu bar)

[Joshua Miller]

Yes. What identity provider are you using, and how are the group membership claims transformed to fit in the JWT?

[Brian Candler]

I am using Google, and according to that link I posted there is no way to get group membership claims in the JWT - it says you need to invoke a separate List call to the Google Directory API.

Right now I'm happy enough maintaining group memberships locally in Netbox.

Actually, for users who are not a member of *any* group, they still have read access to the database.  I believe view perms aren't being enforced until at least 2.6:

https://github.com/digitalocean/netbox/issues/323

(I do have LOGIN_REQUIRED=True, but of course, these users *are* actually logged in and have been automatically created as local users.  It means that all users of our org get read access to Netbox)

[Brian Candler]

Minor note, in case it helps anyone else - authentication broke when I updated from netbox 2.5.13 to 2.6.1.

The problem, once I'd found it, was simple enough. netbox/netbox/settings.py has now gained its own AUTHENTICATION_BACKENDS section:

AUTHENTICATION_BACKENDS = [

    'utilities.auth_backends.ViewExemptModelBackend',

]

and this was overriding my own AUTHENTICATION_BACKENDS which I had put higher up in the file (and merged back in with git stash / git stash apply).  I had to remove my own one, and update the netbox-supplied one to:

AUTHENTICATION_BACKENDS = [

    'utilities.backends.OidcUserInfoSyncRemoteUserBackend', # ADD

    'utilities.auth_backends.ViewExemptModelBackend',

]

Cheers,

Brian.