bad json service definition breaks cas 5.0.x?

[Baron Fujimoto]

So this happened. We are using CAS 5.0.3.1 with JSON service registry
files. We inadvertantly created a service registration with an error in
the regex for the serviceID (improperly escaped "\\." as "\."). Although
we didn't notice it at the time, this was logged with the error/warnings

ERROR [org.apereo.cas.services.AbstractResourceBasedServiceRegistryDao] - <Error reading configuration file Bad_serviceID-20170328162739.json>
java.lang.IllegalArgumentException: org.hjson.ParseException: Expected valid escape sequence at 3:30
...
WARN [org.apereo.cas.services.AbstractResourceBasedServiceRegistryDao] - <Could not load service definition from file /home/cas/cas/services/Bad_serviceID-20170328162739.json>
WARN [org.apereo.cas.services.AbstractResourceBasedServiceRegistryDao] - <1 errors encountered when loading service definitions. New definitions are not loaded until errors are corrected>

But CAS continued on working in general. However, some time later, the CAS
service was restarted after an OS reboot. Upon restart, the same errors
are noted, but there was crucial difference

INFO [org.apereo.cas.services.DefaultServicesManagerImpl] - <Loaded 0 services from JsonServiceRegistryDao.>

None of the other valid service definitions were loaded, and thus our CAS
was effectively broken for everyone by this one bad definition.

Is there a way to configure CAS to just ignore the bad definitions rather
than just fail completely in situations like this? While we probably should
have caught this sooner, it definitely turned out to be a time bomb for
us. Is there a good reason for this behavior that we're missing?

Aloha,
-baron
--
Baron Fujimoto <ba...@hawaii.edu> :: UH Information Technology Services
minutas cantorum, minutas balorum, minutas carboratum desendus pantorum

[Bryan Wooten]

Ok, If you are manually editing the JSON service registry or using any tool (home grown or provided by Apereo) you MUST carefully validate the final JSON file syntax.

I never make a change during normal hours.

We are HA with 4 CAS servers. I make the change on one server. (They are all standalone no rsync or cross mounts for any config file) I then restart that one server and verify is comes back up and starts servicing tickets. (All CAS servers are behind a Citirix Netscalar Load Balancer). So there is no outage.

After I confirm the change has the correct JSON syntax I then push the change to the other servers.

Owning SSO is terrifying.  We can never have an outage for 60k users and 500 servers. Never. Or my life gets bad.

So yeah, it is manual and I take great care and someday I hope to put in place an automated system.

Until then I live with a little stress.....

--
- CAS gitter chatroom: https://gitter.im/apereo/cas
- CAS mailing list guidelines: https://apereo.github.io/cas/Mailing-Lists.html
- CAS documentation website: https://apereo.github.io/cas
- CAS project website: https://github.com/apereo/cas
---
You received this message because you are subscribed to the Google Groups "CAS Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cas-user+unsubscribe@apereo.org.
To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/cas-user/20170502002914.mfo3dcbcul3oo5k5%40combobulate.mgt.hawaii.edu.

[Baron Fujimoto]

Yes, but one of the nice things about CAS was that we could drop new
service definitions into a running instance and have them picked up
automatically. Anything that requires us to actually restart our service
get us involved in a formal change management process and restricts us to
weekend early mornings. :(

The thing that seems inconsistent to me in this case, is that the bad
service definition doesn't tank an already running instance. The faulty
definition just doesn't get incorporated into the running config. This
seems much preferable to breaking everything should the system be
restarted for some reason, IMO.

So, I guess I understand the way it is, but must it be that way? Should
it be that way?

>> email to cas-user+u...@apereo.org.

>> To view this discussion on the web visit https://groups.google.com/a/
>> apereo.org/d/msgid/cas-user/20170502002914.mfo3dcbcul3oo5k5%
>> 40combobulate.mgt.hawaii.edu.
>>
>

>--
>- CAS gitter chatroom: https://gitter.im/apereo/cas
>- CAS mailing list guidelines: https://apereo.github.io/cas/Mailing-Lists.html
>- CAS documentation website: https://apereo.github.io/cas
>- CAS project website: https://github.com/apereo/cas
>---
>You received this message because you are subscribed to the Google Groups "CAS Community" group.

>To unsubscribe from this group and stop receiving emails from it, send an email to cas-user+u...@apereo.org.
>To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/cas-user/CAG9x2GVK0zMcb2Swb0KhKK6n%3D4pG%3DiZFnu8aRLtdxb%2BtJN7%2B3Q%40mail.gmail.com.

[Bryan Wooten]

You are not wrong.

We restart CAS once a week. We are currently running 3.6.x, it has a memory leak with load over time. We do 400k logins per day. (CPU goes to 100% doing nothing but garbage collection)

The weekly restart mitigates / solves (?) this issue,

But if in the interval we messed up the JSON Service Registry then the CAS server is not working.

So yes perhaps an enhancement request for better notification of a bad Service Registry re-load may be warranted. 

I think a proper log4.xml setting in conjunction with monitoring like Solarwinds/Orion or even ELK would solve this issue with a timely notification to the correct people.

So many moving pieces. We do the best we can to support our organization.

>> email to cas-user+unsubscribe@apereo.org.

>> To view this discussion on the web visit https://groups.google.com/a/
>> apereo.org/d/msgid/cas-user/20170502002914.mfo3dcbcul3oo5k5%
>> 40combobulate.mgt.hawaii.edu.
>>
>
>--
>- CAS gitter chatroom: https://gitter.im/apereo/cas
>- CAS mailing list guidelines: https://apereo.github.io/cas/Mailing-Lists.html
>- CAS documentation website: https://apereo.github.io/cas
>- CAS project website: https://github.com/apereo/cas
>---
>You received this message because you are subscribed to the Google Groups "CAS Community" group.

>To unsubscribe from this group and stop receiving emails from it, send an email to cas-user+unsubscribe@apereo.org.

>To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/cas-user/CAG9x2GVK0zMcb2Swb0KhKK6n%3D4pG%3DiZFnu8aRLtdxb%2BtJN7%2B3Q%40mail.gmail.com.

--
Baron Fujimoto <ba...@hawaii.edu> :: UH Information Technology Services
minutas cantorum, minutas balorum, minutas carboratum desendus pantorum

--
- CAS gitter chatroom: https://gitter.im/apereo/cas
- CAS mailing list guidelines: https://apereo.github.io/cas/Mailing-Lists.html
- CAS documentation website: https://apereo.github.io/cas
- CAS project website: https://github.com/apereo/cas
---
You received this message because you are subscribed to the Google Groups "CAS Community" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cas-user+unsubscribe@apereo.org.
To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/cas-user/20170502234855.3tewdzzz6buc4mjh%40combobulate.mgt.hawaii.edu.

[Waldbieser, Carl]

There were 2 things that I always did when modifying the service registry in CAS v3.x.

1) After making a change, I ran a short python script that would validate the well formedness of the JSON and doe some other simple checks.
2) I would put the service URL in my browser to see what our login page looked like (we display different UI elements based on the service).

#1 took care of most issues related to typos.
#2 was also a good sanity check for new services-- if I botched it somehow, I would our "please contact the help desk" page instead of the service page.

In CAS v5, the service definitions are now broken out, so I may need to revisit a validation script of some kind.

Thanks,
Carl Waldbieser
ITS Identity Management
Lafayette College

> >> email to cas-user+u...@apereo.org.

> >> To view this discussion on the web visit https://groups.google.com/a/
> >> apereo.org/d/msgid/cas-user/20170502002914.mfo3dcbcul3oo5k5%
> >> 40combobulate.mgt.hawaii.edu.
> >>
> >
> >--
> >- CAS gitter chatroom: https://gitter.im/apereo/cas

> >- CAS mailing list guidelines: https://apereo.github.io/cas/
> Mailing-Lists.html
> >- CAS documentation website: https://apereo.github.io/cas
> >- CAS project website: https://github.com/apereo/cas
> >---
> >You received this message because you are subscribed to the Google Groups
> "CAS Community" group.
> >To unsubscribe from this group and stop receiving emails from it, send an

> email to cas-user+u...@apereo.org.

> >To view this discussion on the web visit https://groups.google.com/a/

> apereo.org/d/msgid/cas-user/CAG9x2GVK0zMcb2Swb0KhKK6n%
> 3D4pG%3DiZFnu8aRLtdxb%2BtJN7%2B3Q%40mail.gmail.com.

>
> --
> Baron Fujimoto <ba...@hawaii.edu> :: UH Information Technology Services
> minutas cantorum, minutas balorum, minutas carboratum desendus pantorum
>
> --
> - CAS gitter chatroom: https://gitter.im/apereo/cas
> - CAS mailing list guidelines: https://apereo.github.io/cas/
> Mailing-Lists.html
> - CAS documentation website: https://apereo.github.io/cas
> - CAS project website: https://github.com/apereo/cas
> ---
> You received this message because you are subscribed to the Google Groups
> "CAS Community" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to cas-user+u...@apereo.org.

> To view this discussion on the web visit https://groups.google.com/a/

> apereo.org/d/msgid/cas-user/20170502234855.3tewdzzz6buc4mjh%

> 40combobulate.mgt.hawaii.edu.
>

--
- CAS gitter chatroom: https://gitter.im/apereo/cas
- CAS mailing list guidelines: https://apereo.github.io/cas/Mailing-Lists.html
- CAS documentation website: https://apereo.github.io/cas
- CAS project website: https://github.com/apereo/cas
---
You received this message because you are subscribed to the Google Groups "CAS Community" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cas-user+u...@apereo.org.
To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/cas-user/CAG9x2GXfierNrnA3yFd6dsx35XOdQOreCHyApa_ysJ4WA4ooGw%40mail.gmail.com.