Unsafe URL with %3f URL rewritten without UnsafeAllow3F

[ron...@googlemail.com]

Hi

I have a university using Blackboard, Xerte and Tsugi LTI 1.3 integration which until this week has been working fine. It seems that following an Apache update to 2.4.61 they are getting the following error: Unsafe URL with %3f URL rewritten without UnsafeAllow3F, referer: https ://blackboard.blah_blah

 

A quick search revealed discussions like this: https://stackoverflow.com/questions/78729429/403-forbidden-when-url-contains-get-with-encoded-question-mark-unsafeallow3f

e.g. 403 Forbidden when URL contains GET with encoded question mark / UnsafeAllow3F

 

The odd thing is I have the same Xerte + Tsugi install integrated with a Moodle install and that is still working fine from the same server/Apache version etc.

 

I’ve tried adding UnsafeAllow3F to the .htaccess in the Tsugi parent folder and also to the .htaccess in Tsugi/lti but this doesn’t seem to make any difference.

 

Any suggestions where to look next and what to change? And why this differs with Blackboard vs Moodle?

 

Thanks

Ron

 

[Chuck Severance]

Ron,

There is a trend in web hosting frameworks that does not allow certain url encoding patterns to pass.  In Sakai, we saw this first with Perusal.  I fixed it six months ago in Sakai:

https://sakaiproject.atlassian.net/browse/SAK-49209

https://sakaiproject.atlassian.net/browse/SAK-49345

It took two fixes because my first fix had a ChatGPT introduced flaw that I later fixed.  I tried to use base-62 instead of base-64 encoding (long story).

The problem was when I was producing a Content Item return URL in Sakai, I was adding a Sakai internal url to that url as a parameter.  The first url was url encoded and then when it was added to the return url it ended up being "double url encoded” - some part of the Persual hosting environment (load balancer, nginx, apache, node - who knows) rejected *any* url that had the double url encodings and there was nothing that could be done on their end because “security”.

So the solution in the LMS (Sakai) was to switch the encoding of the internal sakai url parameter on the content item return url to use a modified base-64 encoding rather than double URL encoding.  Of course in Sakai, I had to decode the base-64 bit at the right moments.  But once this was done - things were perfect and flawless.

The double URL encoding worked flawlessly for 10 years until it became a “security problem” and prohibited at the lowest level in web frameworks.  The reason it works in Moodle is because they probably avoided the url parameter on a url and the reason you still work with Sakai is that I fixed it.

So, long story short - Blackboard is going to get in more and more trouble until they fix this.  It is not just a Apache thing - it is a web thing that is sneaking in everywhere.  Most tools will not hack their Apache to be “less secure” - so they will just tell Blackboard to bugger off.

I am 100% happy to talk to Blackboard and help - I have code in Sakai that they could imitate - and a ton of pain about bad / wrong approaches that will save them a load of misery.

On the other hand they won’t fix this instantly.  Perhaps you can convince then to get started but it will take a while.

In the interim unless you can convince Apache to be less “secure” to make it so your stuff works, you might want to back-level Apache until Blackboard fixes this.

But their world will start crashing down in general if they don’t fix this - so they can’t just ignore this.

Let us know if you figure out how to convince Apache to relax a bit.  Or what version you go back to to be “safe”.  I have a lot of Tsugi servers and don’t generally expect that a new Apache will break things.  And most of my testing is with Sakai which does not make this mistake.

Happy to help in any way.

/Chuck

-- 
You received this message because you are subscribed to the Google Groups "Tsugi Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tsugi-dev+...@apereo.org.
To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/tsugi-dev/01c001dad9d6%2429555d60%247c001820%24%40googlemail.com.

[Chuck Severance]

Hi all,

Ron’s replies are not making it to the list - so I have forward this to the list.  And yes Ron, feel free to use whatever I send to the list any way you like.

/Chuck

From: RonM <ron...@googlemail.com> 

Sent: Monday, July 22, 2024 11:36 AM
To: Tsugi Developers <tsug...@apereo.org>
Cc: cs...@umich.edu <cs...@umich.edu>; RonM <ron...@googlemail.com>
Subject: Re: [tsugi-dev] Unsafe URL with %3f URL rewritten without UnsafeAllow3F

 

Anyway following your response and with help from Tom Reijnders and some further error tracing we have found a short term solution by adding UnsafeAllow3F to the oidc_login and keyset rules. So Apache is now relaxed enough to get all the LTI projects working again but clearly this is not the long term fix.

I’ve summarised all this and included a link to your full response and offer of help below and will update if/when I hear anything further.

Many thanks

Ron

From: ronm123
Sent: Friday, July 19, 2024 5:03 PM
To: 'Tsugi Developers' tsug...@apereo.org
Subject: RE: [tsugi-dev] Unsafe URL with %3f URL rewritten without UnsafeAllow3F

Hi Dr Chuck

Thanks for the detailed response.

This is a bit of a can of worms and my initial priority was to try to help the university get their existing LTI links working again.

It seems likely that it was the Apache upgrade to 2.4.61 that resulted in their LTI links no longer working but the university are also saying that all their other LTI resources from other tools are still working. I don’t have any details about those and of course they are coming from other sources and may not be using the same Apache version or not using Apache and/or Tsugi anyway. As you say it’s not just an Apache thing anyway but given at the moment it’s only happening with the Xerte/Tsugi integration it’s likely to be difficult to get a change by Blackboard – certainly in the short term.

The university do have a snapshot from before the Apache update but restoring that would likely lose changes to Xerte projects that users have made since. I guess we could backup the current Xerte (incl Tsugi), restore the snapshot to undo the Apache change and then restore the more recent Xerte backup but that would mean quite a bit of disruption and not ideal.

I’ve tried adding [UnsafeAllow3F] to some of the rules but that isn’t resolving the issue so may be missing something. Indeed looking at the error log I’m seeing %3A and %2F but not %3F so probably need to read up if there are other flags that needed adding.

Is it ok to share your response with the university IT team?

Thanks

Ron

[Charles Severance]

Maybe not all my accounts are members - sheesh.

/Chuck

Begin forwarded message:

From: <ron...@googlemail.com>

Subject: RE: Fwd: [tsugi-dev] Unsafe URL with %3f URL rewritten without UnsafeAllow3F

Date: July 22, 2024 at 11:31:16 AM EDT

To: "'Charles Severance'" <drc...@gmail.com>, <tsug...@apereo.org>

Hi Dr Chuck

I’m replying to all but as you know my replies don’t seem to be reaching the list. Then again if I look at https://groups.google.com/a/apereo.org/g/tsugi-dev/c/xAMaqCNpP_w I don’t see your reply below either? I’ve also BCC’d Tom.

 

A few comments…

 

In this particular integration we aren’t using the store so the two rules we changed by adding the UnsafeAllow3F flag were:

RewriteRule ^oidc_login/(.*)$ oidc_login.php?guid=$1 [UnsafeAllow3F,L,QSA]

RewriteRule ^keyset/(.*)$ keyset.php?issuer_guid=$1 [UnsafeAllow3F,L,QSA]

 

This is also an older version of Tsugi (not sure what version) but in fact we planned to configure LTI with their test server and update Tsugi at the same time until this issue on production took priority.

 

Looking at the current config and your example below they are very similar and certainly with no ?=blah but longer strings e.g.

LTI 1.3 OpenID Connect Endpoint: 

…/tsugi/lti/oidc_login/longstring which is the LTI 1.3 Unique Issuer GUID no ? or %3f

 

LTI 1.3 Tool Redirect Endpoint:

…/tsugi/lti/oidc_launch 

 

LTI 1.3 Tool Keyset URL:

,,,/tsugi/lti/keyset/longstring which is the LTI 1.3 Unique Issuer GUID no ? or %3f

 

LTI Content Item / Deep Link Endpoint:

…/tsugi/lti/store/

 

We added a trace to the Apache error log and were able to see more detail and that it was those two rules that needed the flag and the basic error is Unsafe URL with %3f URL rewritten without UnsafeAllow3F, referer: https://blackboard.uwe.ac.uk/

 

Obviously there’s a ? in the above rewrite rules but we didn’t try with those rewrite rules disabled altogether.

 

Thanks

Ron

 

 

From: Charles Severance <drc...@gmail.com> 
Sent: Monday, July 22, 2024 2:02 PM
To: tsug...@apereo.org
Cc: RonM <ron...@googlemail.com>
Subject: Re: Fwd: [tsugi-dev] Unsafe URL with %3f URL rewritten without UnsafeAllow3F

 

Ron,

I assume you mean patching .htaccess rules below.  Upon more investigation, this looks like this is not the same thing as the double url encoding in content item.  It looks like if there is a Rewrite rule that has a '?' in the rule and there is a %3F elsewhere in the URL, things go awry.

Searching the Tsugi code base for .htaccess files that also have a '?' we find:

./lti/store/.htaccess:    RewriteRule ^sakai-config/(.*)$ sakai-config.php?guid=$1 [L,QSA]
./lti/store/.htaccess:    RewriteRule ^ims-config/(.*)$ ims-config.php?guid=$1 [L,QSA]
./lti/.htaccess:    RewriteRule ^oidc_login/(.*)$ oidc_login.php?guid=$1 [L,QSA]
./lti/.htaccess:    RewriteRule ^keyset/(.*)$ keyset.php?issuer_guid=$1 [L,QSA]

This looks to me like some "sort of" legacy backwards compatiblity as I was changing the the patterns for passing in URLs.  The modern pattern for these urls is as follows - there is no query parameters at all.  It was over 2 years ago that I stopped using query parameters on these URLs.

LTI 1.3 OpenID Connect Endpoint: 

http://localhost:8888/py4e/tsugi/lti/oidc_login/7 

LTI 1.3 Tool Redirect Endpoint:

http://localhost:8888/py4e/tsugi/lti/oidc_launch 

LTI 1.3 Tool Keyset URL:

http://localhost:8888/py4e/tsugi/lti/keyset 

LTI Content Item / Deep Link Endpoint:

http://localhost:8888/py4e/tsugi/lti/store/ 

I think the fix is to change all this code to just parse the positional parameters directly without the .htaccess remapping and then remove the rules in the .htaccess completely.

But I don't want to break anything.  So - before I start fixing stuff - could you let me know the urls that are being used for oidc_login and keyset requests for this particular Blackboard instance?  In particular, do those URLs include ? or %3F ??

Thanks.

/Chuck

-- 

You received this message because you are subscribed to the Google Groups "Tsugi Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tsugi-dev+...@apereo.org.

To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/tsugi-dev/4B6F865B-B743-4CDB-8E69-3F5C3780AF77%40gmail.com.

[Charles Severance]

Ron,

That is exactly the detail I needed.  I will work up a fix on the Tsugi side.   I will test on my ends to make sure I don’t see any regressions.  Then I will put it in a branch so you can test the best you can.

I don’t think that it will work with the rules disabled - so I would not try that except in testing.  I think I am using the rewrite rules to avoid parsing url parameters (lazy).  I will make it so either the key value or positional parameter work without the rewrite rules - then we can delete the rules.

Thanks.

/Chuck

From: <ron...@googlemail.com>

Subject: RE: Fwd: [tsugi-dev] Unsafe URL with %3f URL rewritten without UnsafeAllow3F

Date: July 22, 2024 at 11:31:16 AM EDT

To: "'Charles Severance'" <drc...@gmail.com>, <tsug...@apereo.org>

[ron...@googlemail.com]

Hi Dr Chuck

I’m replying to all but as you know my replies don’t seem to be reaching the list. Then again if I look at https://groups.google.com/a/apereo.org/g/tsugi-dev/c/xAMaqCNpP_w I don’t see your reply below either? I’ve also BCC’d Tom.

 

A few comments…

 

In this particular integration we aren’t using the store so the two rules we changed by adding the UnsafeAllow3F flag were:

RewriteRule ^oidc_login/(.*)$ oidc_login.php?guid=$1 [UnsafeAllow3F,L,QSA]

RewriteRule ^keyset/(.*)$ keyset.php?issuer_guid=$1 [UnsafeAllow3F,L,QSA]

 

This is also an older version of Tsugi (not sure what version) but in fact we planned to configure LTI with their test server and update Tsugi at the same time until this issue on production took priority.

 

Looking at the current config and your example below they are very similar and certainly with no ?=blah but longer strings e.g.

LTI 1.3 OpenID Connect Endpoint:

…/tsugi/lti/oidc_login/longstring which is the LTI 1.3 Unique Issuer GUID no ? or %3f

 

LTI 1.3 Tool Redirect Endpoint:

…/tsugi/lti/oidc_launch

 

LTI 1.3 Tool Keyset URL:

,,,/tsugi/lti/keyset/longstring which is the LTI 1.3 Unique Issuer GUID no ? or %3f

 

LTI Content Item / Deep Link Endpoint:

…/tsugi/lti/store/

 

We added a trace to the Apache error log and were able to see more detail and that it was those two rules that needed the flag and the basic error is Unsafe URL with %3f URL rewritten without UnsafeAllow3F, referer: https://blackboard.uwe.ac.uk/

 

Obviously there’s a ? in the above rewrite rules but we didn’t try with those rewrite rules disabled altogether.

 

Thanks

Ron

 

 

From: Charles Severance <drc...@gmail.com>
Sent: Monday, July 22, 2024 2:02 PM
To: tsug...@apereo.org
Cc: RonM <ron...@googlemail.com>
Subject: Re: Fwd: [tsugi-dev] Unsafe URL with %3f URL rewritten without UnsafeAllow3F

 

Ron,

I assume you mean patching .htaccess rules below.  Upon more investigation, this looks like this is not the same thing as the double url encoding in content item.  It looks like if there is a Rewrite rule that has a '?' in the rule and there is a %3F elsewhere in the URL, things go awry.

Searching the Tsugi code base for .htaccess files that also have a '?' we find:

./lti/store/.htaccess:    RewriteRule ^sakai-config/(.*)$ sakai-config.php?guid=$1 [L,QSA]
./lti/store/.htaccess:    RewriteRule ^ims-config/(.*)$ ims-config.php?guid=$1 [L,QSA]
./lti/.htaccess:    RewriteRule ^oidc_login/(.*)$ oidc_login.php?guid=$1 [L,QSA]
./lti/.htaccess:    RewriteRule ^keyset/(.*)$ keyset.php?issuer_guid=$1 [L,QSA]

This looks to me like some "sort of" legacy backwards compatiblity as I was changing the the patterns for passing in URLs.  The modern pattern for these urls is as follows - there is no query parameters at all.  It was over 2 years ago that I stopped using query parameters on these URLs.

LTI 1.3 OpenID Connect Endpoint: 

http://localhost:8888/py4e/tsugi/lti/oidc_login/7 

LTI 1.3 Tool Redirect Endpoint:

http://localhost:8888/py4e/tsugi/lti/oidc_launch 

LTI 1.3 Tool Keyset URL:

http://localhost:8888/py4e/tsugi/lti/keyset 

LTI Content Item / Deep Link Endpoint:

http://localhost:8888/py4e/tsugi/lti/store/ 

I think the fix is to change all this code to just parse the positional parameters directly without the .htaccess remapping and then remove the rules in the .htaccess completely.

But I don't want to break anything.  So - before I start fixing stuff - could you let me know the urls that are being used for oidc_login and keyset requests for this particular Blackboard instance?  In particular, do those URLs include ? or %3F ??

Thanks.

/Chuck

On 7/22/24 8:19 AM, Chuck Severance wrote:

--

[Charles Severance]

Ron,

I assume you mean patching .htaccess rules below.  Upon more investigation, this looks like this is not the same thing as the double url encoding in content item.  It looks like if there is a Rewrite rule that has a '?' in the rule and there is a %3F elsewhere in the URL, things go awry.

Searching the Tsugi code base for .htaccess files that also have a '?' we find:

./lti/store/.htaccess:    RewriteRule ^sakai-config/(.*)$ sakai-config.php?guid=$1 [L,QSA]
./lti/store/.htaccess:    RewriteRule ^ims-config/(.*)$ ims-config.php?guid=$1 [L,QSA]
./lti/.htaccess:    RewriteRule ^oidc_login/(.*)$ oidc_login.php?guid=$1 [L,QSA]
./lti/.htaccess:    RewriteRule ^keyset/(.*)$ keyset.php?issuer_guid=$1 [L,QSA]

This looks to me like some "sort of" legacy backwards compatiblity as I was changing the the patterns for passing in URLs.  The modern pattern for these urls is as follows - there is no query parameters at all.  It was over 2 years ago that I stopped using query parameters on these URLs.

LTI 1.3 OpenID Connect Endpoint: 
http://localhost:8888/py4e/tsugi/lti/oidc_login/7 

LTI 1.3 Tool Redirect Endpoint:
http://localhost:8888/py4e/tsugi/lti/oidc_launch 

LTI 1.3 Tool Keyset URL:
http://localhost:8888/py4e/tsugi/lti/keyset 

LTI Content Item / Deep Link Endpoint:
http://localhost:8888/py4e/tsugi/lti/store/ 

I think the fix is to change all this code to just parse the positional parameters directly without the .htaccess remapping and then remove the rules in the .htaccess completely.

But I don't want to break anything.  So - before I start fixing stuff - could you let me know the urls that are being used for oidc_login and keyset requests for this particular Blackboard instance?  In particular, do those URLs include ? or %3F ??

Thanks.

/Chuck

On 7/22/24 8:19 AM, Chuck Severance wrote:

--

[RonM]

My replies via email do not seem to be reaching the list so for this reply I'm using the web interface instead...

Hi again Dr Chuck

Apologies but I replied to your reply on Friday evening but have only just been alerted by Tom that for some reason that reply didn’t reach the list (see below). So I’m going to try again here and will check the list/group after sending.

Anyway following your response and with help from Tom Reijnders and some further error tracing we have found a short term solution by adding UnsafeAllow3F to the oidc_login and keyset rules. So Apache is now relaxed enough to get all the LTI projects working again but clearly this is not the long term fix.

I’ve summarised all this and included a link to your full response and offer of help below and will update if/when I hear anything further.

Many thanks

Ron

From: ronm123
Sent: Friday, July 19, 2024 5:03 PM
To: 'Tsugi Developers' tsug...@apereo.org
Subject: RE: [tsugi-dev] Unsafe URL with %3f URL rewritten without UnsafeAllow3F

Hi Dr Chuck

Thanks for the detailed response.

This is a bit of a can of worms and my initial priority was to try to help the university get their existing LTI links working again.

It seems likely that it was the Apache upgrade to 2.4.61 that resulted in their LTI links no longer working but the university are also saying that all their other LTI resources from other tools are still working. I don’t have any details about those and of course they are coming from other sources and may not be using the same Apache version or not using Apache and/or Tsugi anyway. As you say it’s not just an Apache thing anyway but given at the moment it’s only happening with the Xerte/Tsugi integration it’s likely to be difficult to get a change by Blackboard – certainly in the short term.

The university do have a snapshot from before the Apache update but restoring that would likely lose changes to Xerte projects that users have made since. I guess we could backup the current Xerte (incl Tsugi), restore the snapshot to undo the Apache change and then restore the more recent Xerte backup but that would mean quite a bit of disruption and not ideal.

I’ve tried adding [UnsafeAllow3F] to some of the rules but that isn’t resolving the issue so may be missing something. Indeed looking at the error log I’m seeing %3A and %2F but not %3F so probably need to read up if there are other flags that needed adding.

Is it ok to share your response with the university IT team?

Thanks

Ron

[ron...@googlemail.com]

Hi Dr Chuck

Thanks for the detailed response.

This is a bit of a can of worms and my initial priority was to try to help the university get their existing LTI links working again.

It seems likely that it was the Apache upgrade to 2.4.61 that resulted in their LTI links no longer working but the university are also saying that all their other LTI resources from other tools are still working. I don’t have any details about those and of course they are coming from other sources and may not be using the same Apache version or not using Apache and/or Tsugi anyway. As you say it’s not just an Apache thing anyway but given at the moment it’s only happening with the Xerte/Tsugi integration it’s likely to be difficult to get a change by Blackboard – certainly in the short term.

The university do have a snapshot from before the Apache update but restoring that would likely lose changes to Xerte projects that users have made since. I guess we could backup the current Xerte (incl Tsugi), restore the snapshot to undo the Apache change and then restore the more recent Xerte backup but that would mean quite a bit of disruption and not ideal.

 

I’ve tried adding [UnsafeAllow3F] to some of the rules but that isn’t resolving the issue so may be missing something. Indeed looking at the error log I’m seeing %3A and %2F but not %3F so probably need to read up if there are other flags that needed adding.

 

Is it ok to share your response with the university IT team?

 

Thanks

Ron

 

 

From: Chuck Severance <cs...@umich.edu>

Sent: Friday, July 19, 2024 3:36 PM
To: Tsugi Developers <tsug...@apereo.org>

[ron...@googlemail.com]

Hi again Dr Chuck

Apologies but I replied to your reply on Friday evening but have only just been alerted by Tom that for some reason that reply didn’t reach the list (see below). So I’m going to try again here and will check the list/group after sending.

 

Anyway following your response and with help from Tom Reijnders and some further error tracing we have found a short term solution by adding UnsafeAllow3F to the oidc_login and keyset rules. So Apache is now relaxed enough to get all the LTI projects working again but clearly this is not the long term fix.

 

I’ve summarised all this and included a link to your full response and offer of help below and will update if/when I hear anything further.

 

Many thanks

Ron

 

 

From: ronm123
Sent: Friday, July 19, 2024 5:03 PM
To: 'Tsugi Developers' tsug...@apereo.org
Subject: RE: [tsugi-dev] Unsafe URL with %3f URL rewritten without UnsafeAllow3F

 

Hi Dr Chuck

Thanks for the detailed response.

This is a bit of a can of worms and my initial priority was to try to help the university get their existing LTI links working again.

It seems likely that it was the Apache upgrade to 2.4.61 that resulted in their LTI links no longer working but the university are also saying that all their other LTI resources from other tools are still working. I don’t have any details about those and of course they are coming from other sources and may not be using the same Apache version or not using Apache and/or Tsugi anyway. As you say it’s not just an Apache thing anyway but given at the moment it’s only happening with the Xerte/Tsugi integration it’s likely to be difficult to get a change by Blackboard – certainly in the short term.

The university do have a snapshot from before the Apache update but restoring that would likely lose changes to Xerte projects that users have made since. I guess we could backup the current Xerte (incl Tsugi), restore the snapshot to undo the Apache change and then restore the more recent Xerte backup but that would mean quite a bit of disruption and not ideal.

 

I’ve tried adding [UnsafeAllow3F] to some of the rules but that isn’t resolving the issue so may be missing something. Indeed looking at the error log I’m seeing %3A and %2F but not %3F so probably need to read up if there are other flags that needed adding.

 

Is it ok to share your response with the university IT team?

 

Thanks

Ron

 

From: Chuck Severance

Sent: Friday, July 19, 2024 3:36 PM
To: Tsugi Developers <tsug...@apereo.org>