InResponseTo not set when state loaded from session.

[Martin Fraser]

This is my first time using SimpleSAMLPHP but I've hit a wall and can't work out how to fix this problem. I've attached the IdP log with the timestamps referred to in the report below but I don't have access to the SP log. If it would help I can get a copy of them.

I have consistently been unable to get authentication working but on one attempt while communicating with the developer of the SP it just worked. So I investigated and found the following. I'm using the drupalauth:External auth source for this, which is why there's reference to Drupal.

It worked when I was already logged into Drupal and attempted to login to the SP using the link supplied @ 16:40:28 InResponseTo is present.

If I close the SP tab and use the link again it fails. @ 16:40:52 logs report "Loading State" InResponseTo is missing.

If I then delete the cookie and local storage data for SP, it still fails thought I'm logged into Drupal. @ 16:41:32 logs report "Loading State" InResponseTo is missing.

If I delete the SP cookie again (recreated from previous failed attempt) and the cookies for SimpleSAML, it still fails and I'm still in Drupal. @ 16:42:31 logs report "Loading State" InResponseTo is missing.

If I then delete the session for Drupal as well as everything deleted previously, then follow the link, it redirects to the Drupal login page indicating I'm no longer logged into Drupal, as expected, then fails to login after I enter correct credentials in Drupal. After failure. I am logged into Drupal. @ 16:43:12 logs report "Saved State" and then there is a four second delay before @ 16:43:16 log reports "Loading State" InResponseTo is missing from samlp:Response.

If I then delete the session for Drupal as well as everything deleted previously, manually login to Drupal, then follow the link below, it works! @ 16:44:44 This time, and the first successful attempt, there is no mention of "Saved State" since the system is not loading authentication data from the SAML IdP session.

Looks to me like there's some sort of mismatch between the SAML session and the Drupal session. When these are created fresh we get an InReponseTo attribute, but when existing sessions are checked, this doesn't happen even though the login credentials in the XML attributes being sent back are always the same and correct.

I've been stumped by this for a while and even resorted to looking at the source code to try and figure out what's happening with the InResponseTo header. The report I'm getting from the SP developer is that their system is complaining that the InResponseTo is missing, which makes sense to me.

If there's anything else you need to help with this, let me know. I've attached the log with the timestamps mentioned above.

Thanks in advance for any help you can give.

[Jaime Perez Crespo]

Hi Martin,

The Drupal and the SimpleSAMLphp sessions are independent and actually MUST NOT depend on each other. If it only works fine when you already have a valid session with Drupal, it looks like authenticating in Drupal is breaking your flow somehow. Besides, a missing InResponseTo indicates IdP-initiated login, which is something that I can see in the logs indeed. You can’t get an InResponseTo field when authentication is initiated at the IdP.

In any case, this sounds like an issue with sessions. If both Drupal and SimpleSAMLphp are using the same session backend, make sure that their session names are different.

--
Jaime Pérez
UNINETT / Feide

jaime...@uninett.no
jaime...@protonmail.com
9A08 EA20 E062 70B4 616B 43E3 562A FE3A 6293 62C2

"Two roads diverged in a wood, and I, I took the one less traveled by, and that has made all the difference."
- Robert Frost

[Martin Fraser]

Hi Jamie.

Thanks very much for the reply. In my investigation I did spot that the missing InResponseTo was due to IdP initiated authentication but in each of my tests I'm following a link on the SP end, which initiates the flow, so I think it should be SP initiated every time, not just when the sessions are clean. Is there something else I can check related to this?

The SP site has a session cookie and I can see two cookies on the IdP end (my end) one is called SimpleSAMLSession and the other is a Drupal generated session name, SSESS6a1917889ed4cf7f0da838dd5ae99839, which is supposed to be unique, as it is in this case.

So, I was looking down the route of the initiator of the flow but can't see anything wrong.

Cheers again for the response, always good to have something else to check.

[Peter Schober]

* Martin Fraser <martind...@gmail.com> [2017-02-01 15:21]:

> Thanks very much for the reply. In my investigation I did spot that the
> missing InResponseTo was due to IdP initiated authentication but in each of
> my tests I'm following a link on the SP end, which initiates the flow, so I
> think it should be SP initiated every time, not just when the sessions are
> clean. Is there something else I can check related to this?

* SP-initiated: SAML 2.0 Authentication Request is sent to the IDP.
* IDP-initiated: Something proprietary is sent to the IDP. As there's
no SAML message to reference in the latter case InResponseTo will
have to remain emtpy.

If there's a SAML 2.0 authentication request then it's SP-initated (as
far as the IDP is concerned). No reason to guess.
-peter

[Martin Fraser]

Thanks for clarifying, that was my understanding. Reviewing the attached log file I can see six <samlp:AuthnRequest> coming in as well as six lines which read:

SAML2.0 - IdP.SSOService: incoming authentication

This to me indicates that all six of my test requests as described above are SP initiated, as expected. So the question remains, why is there no InResponseTo attribute in the response when all six of the requests from SP have an ID attribute?

[Jaime Perez Crespo]

Hi Martin,

On 2 Feb 2017, at 01:00 AM, Martin Fraser <martind...@gmail.com> wrote:
> Thanks for clarifying, that was my understanding. Reviewing the attached log file I can see six <samlp:AuthnRequest> coming in as well as six lines which read:
> SAML2.0 - IdP.SSOService: incoming authentication

Counting authentication requests and specific log lines is not enough. You should be able to follow the log from the authentication request to the authentication response to verify that what’s happening is indeed an SP-initiated authentication. If you do that, you will see it’s not the case:

—8<—
Here you get the first AuthnRequest with ID "_48FCA075-DD82-4BAB-8425-BC4894478298":

> Jan 27 16:40:28 simplesamlphp INFO [db5aebdb83] SAML2.0 - IdP.SSOService: Accessing SAML 2.0 IdP endpoint SSOService
> Jan 27 16:40:28 simplesamlphp DEBUG [db5aebdb83] Received message:
> Jan 27 16:40:28 simplesamlphp DEBUG [db5aebdb83] <samlp:AuthnRequest xmlns:samlp="urn:oasis:names:tc:SAML:2.0:protocol" xmlns:saml="urn:oasis:names:tc:SAML:2.0:assertion" Version="2.0" ID="_48FCA075-DD82-4BAB-8425-BC4894478298" IssueInstant="2017-01-27T16:40:27.825Z" Destination="https://staging.entrepreneurialscotland.com/simplesaml/saml2/idp/SSOService.php">
> Jan 27 16:40:28 simplesamlphp DEBUG [db5aebdb83] <saml:Issuer>https://app.staging.goodpractice.net</saml:Issuer>
> Jan 27 16:40:28 simplesamlphp DEBUG [db5aebdb83] <samlp:NameIDPolicy AllowCreate="true"/>
> Jan 27 16:40:28 simplesamlphp DEBUG [db5aebdb83] </samlp:AuthnRequest>
> Jan 27 16:40:28 simplesamlphp INFO [db5aebdb83] SAML2.0 - IdP.SSOService: incoming authentication request: 'https://app.staging.goodpractice.net’

Here, however, something went wrong and you didn’t continue SSO. Instead, you get yet another AuthnRequest with ID "_8545DE70-01EF-4484-976B-B706D8773991”. Note also that the “track ID” (the code in brackets right after the log level) changes:

> Jan 27 16:40:52 simplesamlphp DEBUG [101cbb7a76] Session: 'drupal-userpass' not valid because we are not authenticated.
> Jan 27 16:40:52 simplesamlphp DEBUG [101cbb7a76] Saved state: '_94c513cdb4dac3930fd823d13485954814937872c6:https://staging.entrepreneurialscotland.com/simplesaml/saml2/idp/SSOService.php?spentityid=https%3A%2F%2Fapp.staging.goodpractice.net&cookieTime=1485535252'
> Jan 27 16:40:52 simplesamlphp WARNING [101cbb7a76] The class or interface 'SimpleSAML_Module' is now using namespaces, please use 'SimpleSAML\Module'.
> Jan 27 16:40:52 simplesamlphp INFO [101cbb7a76] SAML2.0 - IdP.SSOService: Accessing SAML 2.0 IdP endpoint SSOService
> Jan 27 16:40:52 simplesamlphp DEBUG [101cbb7a76] Received message:
> Jan 27 16:40:52 simplesamlphp DEBUG [101cbb7a76] <samlp:AuthnRequest xmlns:samlp="urn:oasis:names:tc:SAML:2.0:protocol" xmlns:saml="urn:oasis:names:tc:SAML:2.0:assertion" Version="2.0" ID="_8545DE70-01EF-4484-976B-B706D8773991" IssueInstant="2017-01-27T16:40:51.348Z" Destination="https://staging.entrepreneurialscotland.com/simplesaml/saml2/idp/SSOService.php">
> Jan 27 16:40:52 simplesamlphp DEBUG [101cbb7a76] <saml:Issuer>https://app.staging.goodpractice.net</saml:Issuer>
> Jan 27 16:40:52 simplesamlphp DEBUG [101cbb7a76] <samlp:NameIDPolicy AllowCreate="true"/>
> Jan 27 16:40:52 simplesamlphp DEBUG [101cbb7a76] </samlp:AuthnRequest>
> Jan 27 16:40:52 simplesamlphp INFO [101cbb7a76] SAML2.0 - IdP.SSOService: incoming authentication request: 'https://app.staging.goodpractice.net’

Here, SimpleSAMLphp tries to load the state with ID “_94c513cdb4dac3930fd823d13485954814937872c6”. The “track ID” changes again. Note that part of the state is the SSO URL of your IdP with a parameter indicating the entity ID of the service that requested authentication. As it reads from the documentation, this is the URL for IdP-initiated authentication:

https://simplesamlphp.org/docs/stable/simplesamlphp-idp#section_11

> Jan 27 16:40:53 simplesamlphp DEBUG [d9b47223e8] Loading state: '_94c513cdb4dac3930fd823d13485954814937872c6:https://staging.entrepreneurialscotland.com/simplesaml/saml2/idp/SSOService.php?spentityid=https%3A%2F%2Fapp.staging.goodpractice.net&cookieTime=1485535252'

Finally, you get a log line clearly stating that you are starting IdP-initiated authentication. Next, authentication succeeds and a SAMLResponse is sent back to the SP, WITHOUT the InResponseTo attribute, as expected from an IdP-initiated authentication. Note that the “track ID” changes again.

> Jan 27 16:40:53 simplesamlphp INFO [1afdf257bc] SAML2.0 - IdP.SSOService: IdP initiated authentication: 'https://app.staging.goodpractice.net’

—>8—

> This to me indicates that all six of my test requests as described above are SP initiated, as expected.

As we have just seen, no, it’s not like that. You don’t even need to perform the same analysis I just did, it’s even easier. If you search for SAML responses, there are 6 of them. Now, you can search for “IdP initiated authentication” messages, and you will see that there are indeed 4 of them:

> Jan 27 16:40:53 simplesamlphp INFO [1afdf257bc] SAML2.0 - IdP.SSOService: IdP initiated authentication: 'https://app.staging.goodpractice.net'
> Jan 27 16:41:33 simplesamlphp INFO [83fe050075] SAML2.0 - IdP.SSOService: IdP initiated authentication: 'https://app.staging.goodpractice.net'
> Jan 27 16:42:33 simplesamlphp INFO [15607e1c63] SAML2.0 - IdP.SSOService: IdP initiated authentication: 'https://app.staging.goodpractice.net'
> Jan 27 16:43:16 simplesamlphp INFO [6149e9a67f] SAML2.0 - IdP.SSOService: IdP initiated authentication: 'https://app.staging.goodpractice.net'

> So the question remains, why is there no InResponseTo attribute in the response when all six of the requests from SP have an ID attribute?

Because 4 out of 6 requests are not replied to, and instead, an IdP-initiated authentication is performed, where no InResponseTo attribute can be available.

Now, once we know why there’s no InResponseTo attribute in 4 out of 6 SAML responses, what we need to find out is why 4 of 6 requests are failing and SimpleSAMLphp is resorting to IdP-initiated authentication. In the example I just used, this happened right after a debug message saying it was trying to load the sate with ID “_94c513cdb4dac3930fd823d13485954814937872c6”. SimpleSAMLphp redirects to the IdP-initiated authentication when it fails to load the state, meaning at that point the state was missing for some reason.

Additionally, I noted in the example every time the "track ID” changed. I did this because the "track ID” should NEVER change, unless the session is lost. The "track ID” is tied to the session, and the session is kept ALWAYS, surviving authentication and logout, so the "track ID” must survive also. However, you can see lots of changes of the "track ID” in your log, which makes sense if you are removing the session cookies, but not in the middle of an authentication flow, as we have seen it happening.

This indicates that you are having issues with your sessions. Given that you are using external authentication with its own session management, it looks pretty clear that something in the external authentication mechanism is breaking the SimpleSAMLphp session. I would recommend you to install memcache and change your SimpleSAMLphp configuration to use it as the session backend. If your problems disappear, my diagnose will be confirmed.

[Martin Fraser]

Jamie, that was a comprehensive and clear reply, thank you so much.

I guess my lack of experience with Simple SAML shows, I should have spotted the IdP initiated messages in the logs there.

You sir are a genius. I'm on shared hosting so had to wait while they set up memcache for me. With a simple change to the config.php file I can report that this is now working. Thanks again.

If I can ask one more thing though. This is set up on a staging server but adding memcache caused some down time. Do you have any pointers that might allow me to get this working with normal php sessions, without memcache, and still allow Drupal and Simple SAML to co-exist? It's my understanding of sessions in PHP that they are shared across the domain for a given site and session cookie. I made sure that both services were using different session cookies so I would have expected there to be different sessions, I guess I was wrong.

If they are set to use exactly the same cookie name, will the data inside that session just be shared and left intact rather than changed as I've seen here? I appreciate you may not know much about Drupal but any suggestions will be very welcome.

Of course, if you don't know of anything off hand then again I thank you for taking the time to help me and dig me out of a complex problem.

Cheers.

[Jaime Perez Crespo]

Hi Martin,

On 3 Feb 2017, at 00:11 AM, Martin Fraser <martind...@gmail.com> wrote:
> Jamie, that was a comprehensive and clear reply, thank you so much.

It’s Jaime actually, Spanish name ;-)

> I guess my lack of experience with Simple SAML shows, I should have spotted the IdP initiated messages in the logs there.
>
> You sir are a genius.

Oh, I wish I was!

> I'm on shared hosting so had to wait while they set up memcache for me. With a simple change to the config.php file I can report that this is now working. Thanks again.

Good to hear!

> If I can ask one more thing though. This is set up on a staging server but adding memcache caused some down time. Do you have any pointers that might allow me to get this working with normal php sessions, without memcache, and still allow Drupal and Simple SAML to co-exist?

I understand you are using this module, from the fact that you are using the “drupalauth:External” auth source, right?

https://github.com/Sanchiz/drupalauth

In that case, the only way you could use the PHP session handler in PHP is that they fix what’s wrong in the module:

https://github.com/Sanchiz/drupalauth/issues/35

They are loading Drupal’s environment without taking any care of SimpleSAMLphp’s environment first, effectively smashing one with the other. That’s why it doesn’t work.

By they way, they are quite explicit about this limitation:

—8<—
> • Configure SimpleSAMLphp to use something other than phpsession for session storage, e.g., SQL or memcache (See: store.type in simplesamlphp/config/config.php).

—>8—

> It's my understanding of sessions in PHP that they are shared across the domain for a given site and session cookie.

The sessions themselves are stored in the server, and retrieved based on the session ID contained in the session cookie. When and where the cookie is available depends on the options used when it is set (domain, path, etc).

> I made sure that both services were using different session cookies so I would have expected there to be different sessions, I guess I was wrong.

This is an inherent limitation of the PHP session handler. It doesn’t allow having two different sessions open at the same time, so even though it is possible to have multiple sessions at the same time, you need to close one before opening another. With this in mind, any application using PHP sessions needs to be aware of the limitation and make sure to be as less invasive as possible wrt other sessions that might coexist. If the application is not taking this into account (as it seems the case of Drupal), other applications will suffer the consequences regardless of how careful they are themselves.

> If they are set to use exactly the same cookie name, will the data inside that session just be shared and left intact rather than changed as I've seen here? I appreciate you may not know much about Drupal but any suggestions will be very welcome.

I don’t know much about Drupal indeed, so I can’t tell. However, even if it is left intact (you could try to configure both the same way and see what happens), it’s a bad idea because you never know what could the other be doing with the session (now or in the future). Furthermore, it’s a bad idea from a security perspective, since even if SSP is perfectly fine and there is no way to mess with the session for an attacker, sharing the session with Drupal could open for attackers to exploit any possible weakness in Drupal to modify SimpleSAMLphp’s session and gather unauthorized access.

> Of course, if you don't know of anything off hand then again I thank you for taking the time to help me and dig me out of a complex problem.

No problem!

[Martin Fraser]

Sorry for the name typo ;)

Thanks for the clarification, that all makes perfect sense. I've left Simple SAML using memcache and things are working as expected. I never spotted the note in the module but your explanation clears that up. I'll leave things as they are, time being what it is, but may patch the Drupal module to see if I can work round this issue.

Again, thanks for the help, you're a life saver.