5063 ERROR_SECURITY_VIOLATION from one address, but not another

[imnt...@gmail.com]

I have PWM set up, and have two DNS entries pointing to it, password.nghs.com and pwm.nghs.com. If I go to password.nghs.com, the site comes up, but if I go to pwm.nghs.com, I get:
A security violation has occurred. Please try again later. { 5063 ERROR_SECURITY_VIOLATION (current network address '66.249.66.141' has changed from original network address '66.249.66.137') }

password.nghs.com is a legacy address, and was changed to a CNAME pointing at pwm.nghs.com.

Does anyone have any idea why this is happening? Or, more importantly, how to stop it?

Thanks,
Jameson

[imnt...@gmail.com]

On Friday, March 18, 2016 at 11:08:36 AM UTC-4, imnt...@gmail.com wrote:
> I have PWM set up, and have two DNS entries pointing to it, password.nghs.com and pwm.nghs.com. If I go to password.nghs.com, the site comes up, but if I go to pwm.nghs.com, I get:
> A security violation has occurred. Please try again later. { 5063 ERROR_SECURITY_VIOLATION (current network address '66.249.66.141' has changed from original network address '66.249.66.137') }
>

Interestingly, I also don't see the error if I access it via the IP address. Only the pwm.nghs.com address which is the address we've published to users.

[imnt...@gmail.com]

Also, if I go to pwm.nghs.com/pwm/public, using the Sign-in button from there will redirect me to a sign-in page that works.

[Jason Rivard]

It's your src IP address that is changing.  Probably because for one of the urls your using a different proxy/gateway then the other.

See:

http://www.pwm-project.org/pwm/public/reference/settings.jsp#setting_key_network.allowMultiIPSession

[Jameson]

The IPs listed in the error are actually Google web crawling bots. Why would they be able to cause the error to show up for me who is connecting from an internal private IP? Thanks.

[Jameson]

Btw, I created a robots.txt, and that seems to have taken care of the issue for now.

[Jonathan Cauthorn]

I'm getting this same error and here are the circumstances surrounding it:

Error:

2016-05-12T23:28:12Z, ERROR, filter.RequestInitializationFilter, {96979} 5063 ERROR_SECURITY_VIOLATION (current network address '10.43.10.253' has changed from original network address '10.43.10.254') [10.43.10.254]
2016-05-12T23:28:12Z, ERROR, http.PwmResponse, {96979} 5063 ERROR_SECURITY_VIOLATION (current network address '10.43.10.253' has changed from original network address '10.43.10.254') [10.43.10.254]
2016-05-12T23:28:12Z, WARN , http.PwmHttpResponseWrapper, attempt to write cookie 'SESSION' after response is committed
2016-05-12T23:34:01Z, INFO , stats.StatisticsManager, published anonymous statistics to https://pwm-cloud.appspot.com/rest/pwm/statistics
2016-05-12T23:36:12Z, ERROR, filter.RequestInitializationFilter, {96987} 5063 ERROR_SECURITY_VIOLATION (current network address '10.43.10.254' has changed from original network address '10.43.10.253') [10.43.10.253]
2016-05-12T23:36:12Z, ERROR, http.PwmResponse, {96987} 5063 ERROR_SECURITY_VIOLATION (current network address '10.43.10.254' has changed from original network address '10.43.10.253') [10.43.10.253]
2016-05-12T23:36:12Z, WARN , http.PwmHttpResponseWrapper, attempt to write cookie 'SESSION' after response is committed

Note - I the first 3 messages repeat, the INFO message only shows once.

Situation:

I'm only getting this error immediately after I set a test pwm user in the pwm configuration interface and then clicked test configuration button. At this point I get an error in the web interface and this series of messages in the log. When I delete my test pwm user from the configuration and run the Test Configuration button again the ldap shows good and the error goes away, but with no pwm test user.

This works for all testing otherwise, but I'd rather have a pwm test user for it to periodically test itself.

I also get this error message when the server starts about the localdb:

2016-05-11T17:10:29Z, ERROR, pwm.PwmEnvironment, unable to write contents of application lock file: The process cannot access the file because another process has locked a portion of the file
2016-05-11T17:10:29Z, INFO , pwm.PwmApplication, initializing, application mode=RUNNING, applicationPath=C:\Program Files\Apache Software Foundation\Tomcat 7.0\webapps\pwm\WEB-INF, pwmEnvironment.getConfig()File=C:\Program Files\Apache Software Foundation\Tomcat 7.0\webapps\pwm\WEB-INF\PwmConfiguration.xml
2016-05-11T17:10:31Z, INFO , localdb.LocalDBFactory, LocalDB open in 938ms, db size: 47.45 MB at C:\Program Files\Apache Software Foundation\Tomcat 7.0\webapps\pwm\WEB-INF\LocalDB, 80.31 GB free

This does not seem to impact the operation otherwise.

There are 2 PWM servers behind a load balancer. I have sticky sessions set so a session should not be bouncing back and forth between the two.
Version: PWM v1.8.0-SNAPSHOT b13038137 r9f802607617def42e749d1392a5389d93d
Running on Windows 2012 R2 JDK 8.0 66 (64 bit) Tomcat 7.0.67

If you can point me in the right direction I'll try something different.

[Jameson]

I would try to verify that traffic coming from the load balancer to the PWM servers doesn't have more than one mapped IP it can come from. If nothing else, have you tried disabling the check for originating IP in PWM? 

[Jonathan Cauthorn]

Thanks for the suggestion - I'll put on windump (like tcpdump) and check, and check with our load balancer guy.

Do you know what setting the check for IP is in PWM? Maybe it's under security I'll have to go check.

Thanks,

+Jonathan

--

+Jonathan

[Jameson]

Look at Settings -> Security -> Web Security -> Allow Roaming Source Network Address

[Jonathan Cauthorn]

General resolution:

It looks like the issue was that the A10 load balancer was set to use Source NAT (snat). This serves the function of allowing devices behind the load balancer to use other services behind the load balancer. (Just ask your load balancer guy.)

While I don't understand exactly why it was performing this way, we have a work around.

We simply go straight from the load balancer to the PWM servers directly, without using SNAT.
Apparently the SNAT was changing the source when PWM was doing a stickyRedirectTest as per this indicator:

https://pwm.{no}.edu/pwm/?stickyRedirectTest=P0MtG06BmgXMosi81E65uAxC5qnschou15531ce5447
 
And because the load balancer switched the source NAT at that time it failed the test, and generated the 5063 ERROR_SECURITY_VIOLATION message. This is basically detecting what looks like a man in the middle attack, which would be bad if someone was doing that in front of a password management system.

Hope that helps save you the days of time it took us to troubleshoot it. :)
+Jonathan

Our work around was to simply disable the SNAT or go directly from the load balancer to the PWM server.