AWX 21.0.1 Docker Deployment+HAProxy | SAML/SSO Login 'RelayState' Error

[Donny Ta]

Hi everyone,

Attached is the error that I'm getting after I DUO authenticate (two-factor authenticate) to my IDP. Seems like everything is good but AWX doesn't know how to handle the response after the DUO? I'm not sure what is going on, to be honest. I am still new with the whole SAML/SSO authentication piece.

Anyways, the setup is a bit odd. I have the AWX containers serving the webpage out port 8043, but I use an HAProxy container to then proxy that to port 443 so that users can just hit the FQDN of the app (this also helps with documentation as the previous version that we had running was hosted on port 443).

I think it may be that HAProxy is not forwarding all necessary responses to the AWX containers? I'm not sure though.

Below is my haproxy config.

global
    stats socket /tmp/admin.sock
    stats timeout 30s

defaults
    log     global
    mode    http
    option  httplog
    option  dontlognull
    timeout connect 5000
    timeout client  50000
    timeout server  50000
    timeout tunnel        3600s
    timeout http-keep-alive  1s
    timeout http-request    15s
    timeout queue           30s
    timeout tarpit          60s
    default-server inter 3s rise 2 fall 3
    option forwardfor

cache web_cache
    total-max-size 4095
    max-object-size 10000
    max-age 30

frontend localnodes
    bind *:8013
    mode http
    default_backend nodes

frontend localnodes_ssl
    bind *:443
    bind *:8043
    mode tcp
    default_backend nodes_ssl

backend nodes
    mode http
    balance roundrobin
    option forwardfor
    option http-pretend-keepalive
    http-request cache-use web_cache
    http-response cache-store web_cache
    http-request set-header X-Forwarded-Port %[dst_port]
    http-request add-header X-Forwarded-Proto https if { ssl_fc }
    option httpchk HEAD / HTTP/1.1\r\nHost:localhost
        server tools_awx_1 tools_awx_1:8013 check
        server tools_awx_2 tools_awx_2:8013 check

backend nodes_ssl
    mode tcp
    balance roundrobin
        server tools_awx_1 tools_awx_1:8043 maxconn 10000 weight 10 cookie tools_awx_1 check
        server tools_awx_2 tools_awx_2:8043 maxconn 10000 weight 10 cookie tools_awx_2 check

listen stats
    bind *:1936
    stats enable
    stats uri /

Any help would be much appreciated! Thank you!!

Donny

[AWX Project]

From the screen shot it looks like you are making it all the way back to awx, and the url /sso/complete/saml/ is the last leg in the handshake before you are authenticated.

My gut is saying that there is something in the response coming from DUO that AWX isn't handeling properly.

It could be a configuration issue on the AWX SAML adapter or something in the SAML response itself.

Chrome and Fire Fox have extensions which can be installed to help trace the SAML request/responses going back and forth as part of the authentication process. 

For example https://chrome.google.com/webstore/detail/saml-chrome-panel/paijfdbeoenhembfhkhllainmocckace

Can you install one of those extensions and capture the SAML response coming to AWX from DUO and share as much of that as possible?
Text would be preferred over an image in case we need to try and run commands on it to replicate the issue?
If you need to redact information maybe use a jinja type syntax like {{ my_user_name }}/{{ my_first_name }}.

The full trace back would also be helpful at the top of the grey section there is a link that says "Switch to copy and paste view".
From what I see I'm leaning towards it being an issue with it trying to process one of the fields coming back from DUO.

If that is the case, your SSO configuration would also be useful.

You can grab that from the URL /api/v2/settings/saml/ but values in there will likely also need to be redacted.

[Donny Ta]

Hi,

Apologies for the late response, I was caught up on other work.

Anyways, here's the 'Switch to copy and paste view':
Environment:


Request Method: POST
Request URL: https://awx.disteng.net.tfayd.com/sso/complete/saml/

Django Version: 3.2.12
Python Version: 3.9.10
Installed Applications:
['django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.messages',
 'django.contrib.sessions',
 'django.contrib.sites',
 'django.contrib.staticfiles',
 'oauth2_provider',
 'rest_framework',
 'django_extensions',
 'channels',
 'polymorphic',
 'taggit',
 'social_django',
 'django_guid',
 'corsheaders',
 'awx.conf',
 'awx.main',
 'awx.api',
 'awx.ui',
 'awx.sso',
 'solo',
 'rest_framework_swagger',
 'debug_toolbar']
Installed Middleware:
['debug_toolbar.middleware.DebugToolbarMiddleware',
 'django_guid.middleware.guid_middleware',
 'awx.main.middleware.TimingMiddleware',
 'django.contrib.sessions.middleware.SessionMiddleware',
 'awx.main.middleware.MigrationRanCheckMiddleware',
 'corsheaders.middleware.CorsMiddleware',
 'django.middleware.locale.LocaleMiddleware',
 'django.middleware.common.CommonMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'awx.main.middleware.DisableLocalAuthMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware',
 'awx.sso.middleware.SocialAuthMiddleware',
 'crum.CurrentRequestUserMiddleware',
 'awx.main.middleware.URLModificationMiddleware',
 'awx.main.middleware.SessionTimeoutMiddleware']



Traceback (most recent call last):
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/datastructures.py", line 76, in __getitem__
    list_ = super().__getitem__(key)

During handling of the above exception ('RelayState'), another exception occurred:
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/handlers/base.py", line 181, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/lib64/python3.9/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/views/decorators/cache.py", line 44, in _wrapped_view_func
    response = view_func(request, *args, **kwargs)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
    return view_func(*args, **kwargs)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/social_django/utils.py", line 46, in wrapper
    return func(request, backend, *args, **kwargs)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/social_django/views.py", line 31, in complete
    return do_complete(request.backend, _do_login, user=request.user,
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/social_core/actions.py", line 45, in do_complete
    user = backend.complete(user=user, *args, **kwargs)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/social_core/backends/base.py", line 40, in complete
    return self.auth_complete(*args, **kwargs)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/social_core/backends/saml.py", line 313, in auth_complete
    idp_name = self.strategy.request_data()['RelayState']
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/datastructures.py", line 78, in __getitem__
    raise MultiValueDictKeyError(key)

Exception Type: MultiValueDictKeyError at /sso/complete/saml/
Exception Value: 'RelayState'

As for the SAML response, I cannot share that because it has a lot of confidential information in it. I can see that the response is a '500' internal server error, though.

I hope that helps.

Thank you,

Donny

[AWX Project]

Apologies for the break in our response. Have you been able to resolve the SAML issues you were having?

-The AWX Team

[Donny Ta]

I was not able to get this working. I am still having the same issue

[AWX Project]

The stack trace looks like its all django.core and no AWX code so this should be fun to figure out!

In the last part of the message it's complaining about a MultiValueDictKeyError in regards to RelayState so lets start there.

RelayState is a parameter that gets passed back and forth as part of the SAML login process to help us keep track of which service provider we are talking to.

In the initial call to sso/login/saml as the Location in the 302 response header pointing you to your SAML provider should have a parameter in the URL like &RelayState=.

That should be generated from the provider name in the SAML adapter in AWX.

After you go through the SSO process, RelayState should come back as a parameter to the sso/complete/saml endpoint next to the encoded SAMLResponse parameter.

And we do capture and stuff the RelayState into the database as part of the login process so maybe our issue is related to that?

Can you look at your calls to sso/login/saml and sso/complete/saml and let us know what your RelayState looks like for each request?

-The AWX Team