CAS 6.1.6 inotify instances skyrocketing with Groovy files in SAML service.

[William Jojo]

Been running 6.1.6 for about 2 weeks. No issues - until I added SAML support. This morning I noticed CAS no longer working. Checked log and found:

From log:

2020-06-01 09:05:32,086 INFO [org.apereo.cas.util.io.PathWatcherService] - <Watching directory at [/etc/cas/saml]>

2020-06-01 09:05:32,086 ERROR [org.apereo.cas.services.ReturnMappedAttributeReleasePolicy] - <User limit of inotify instances reached or too many open files>

java.io.IOException: User limit of inotify instances reached or too many open files

at sun.nio.fs.LinuxWatchService.<init>(LinuxWatchService.java:64) ~[?:?]

at sun.nio.fs.LinuxFileSystem.newWatchService(LinuxFileSystem.java:47) ~[?:?]

at org.apereo.cas.util.io.PathWatcherService.<init>(PathWatcherService.java:62) ~[cas-server-core-util-api-6.1.6.jar:6.1.6]

at org.apereo.cas.util.io.PathWatcherService.<init>(PathWatcherService.java:40) ~[cas-server-core-util-api-6.1.6.jar:6.1.6]

at org.apereo.cas.util.io.FileWatcherService.<init>(FileWatcherService.java:26) ~[cas-server-core-util-api-6.1.6.jar:6.1.6]

at org.apereo.cas.util.scripting.WatchableGroovyScriptResource.<init>(WatchableGroovyScriptResource.java:31) ~[cas-server-core-util-api-6.1.6.jar:6.1.6]

Thought this was odd since never had this problem with any other area of CAS watch areas. Did some digging and seems this is NOT an issue UNTIL I added the groovy files to a SAML service.

The portion of the JSON is as follows:

      memberOf:

      [

java.util.ArrayList

[

file:/etc/cas/saml/memberOf.groovy

]

      ]

      eduPersonPrimaryAffiliation:

      [

        java.util.ArrayList

        [

                file:/etc/cas/saml/eduPersonPrimaryAffiliation.groovy

        ]

      ]

Now look at this output:

root@casdev-master:~# while (( 1 == 1 )); do date; lsof | grep inotify | grep 31744 | wc -l; sleep 120; done

Mon Jun  1 11:28:05 EDT 2020

178

Mon Jun  1 11:30:05 EDT 2020

178

Mon Jun  1 11:32:06 EDT 2020

178

Mon Jun  1 11:34:06 EDT 2020

178

Mon Jun  1 11:36:07 EDT 2020

178

Mon Jun  1 11:38:08 EDT 2020

178

Mon Jun  1 11:40:08 EDT 2020

1872

Mon Jun  1 11:42:09 EDT 2020

2500

Mon Jun  1 11:44:10 EDT 2020

3192

Mon Jun  1 11:46:11 EDT 2020

3948

Mon Jun  1 11:48:12 EDT 2020

4768

Mon Jun  1 11:50:13 EDT 2020

5652

Mon Jun  1 11:52:14 EDT 2020

6600

There are 178 inotify watches consistently UNTIL I edit the service file and allow the Groovy files to be used. Then it just goes out of control. There were this many entries for each:

root@casdev-master:~# lsof | grep inotify | grep 31744 | grep edu | wc -l

1200

root@casdev-master:~# lsof | grep inotify | grep 31744 | grep member | wc -l

1104

It seems too be increasing by hundreds of entries per TID in a very brief period of time and it also seems to be affecting other inotify counts as a result. Any thoughts on why this would suddenly go out of control when adding Groovy files to the service?

Thank you!

Bill

[William Jojo]

Well, I was able to stop the hemorrhaging. Have been watching it for about an hour and the inotify list remains constant.

By using an inline Groovy script, CAS no longer needed to setup a watcher service for the Groovy scripts. However, I lost the ability to log debug info and compound statements seem to not be allowed in the inline model. Fortunately my code was not so horribly complex that I was able to work it out. For example:

AD groups are horribly disfigured (read in DN form) from the LDAP query, so we rewrite them like so:

      memberOf:

      [

        java.util.ArrayList

        [

          groovy { def groups = attributes['memberOf']; def result = []; for ( cn in groups )  result.add( ( cn =~ /CN=([^,]+),/)[0][1] ) ;  return result; }

        ]

      ]

If anyone has an idea on where to begin looking for the cause of this issue, I am quite happy to help determine why this flies out of control.

Also, if anyone has info on the syntax limitations of inline Groovy scripts and how to still do logging in the inline script that would be very helpful for debugging.

Thank you!

Bill

[Morning Star]

Hi William,

We are also facing the same issue like you.  Could you please help us?

[joj...@gmail.com]

Morning Star,

Which CAS and OS versions are you using?

The only solution for us was to in-line them. We lost logging, but since no one else spoke up about the issue and I had no time to further research it myself, I moved on since in-line worked.

Let me try it on my test system and see if 6.2 or 6.3 still have the problem. It is possible that it is already fixed.

Bill

Sent from a device.

On Mar 4, 2021, at 12:15 AM, Morning Star <anusu...@gmail.com> wrote:

Hi William,

[William Jojo]

Morning Star,

The problem persists on 6.2.8 - which is the latest one I have available to test at the moment. Seems to only be increasing the inotify list with each use of the service that uses the external Groovy file (meaning not inline). Maybe an improvement, but it has been some time since I dug into this issue.

Bill

[Tom O'Neill]

Hi All,

 

Good timing – I think I just ran into this issue this morning running a Groovy authentication policy but most of our services are CAS.

I’m going to keep an eye on it and will let you know what I find.

 

CAS 6.3.0 on Java 11

 

Thanks,

Tom

 

From: cas-...@apereo.org <cas-...@apereo.org> On Behalf Of William Jojo
Sent: Monday, March 8, 2021 1:46 PM
To: Morning Star <anusu...@gmail.com>
Cc: CAS Community <cas-...@apereo.org>
Subject: [EXT] [cas-user] Re: CAS 6.1.6 inotify instances skyrocketing with Groovy files in SAML service.

 

CAUTION: This email originated from outside of SIG. Exercise caution when opening attachments or clicking links, especially from unknown senders.

[EXT-STAMP-ADDED]

--
- Website: https://apereo.github.io/cas
- Gitter Chatroom: https://gitter.im/apereo/cas
- List Guidelines: https://goo.gl/1VRrw7
- Contributions: https://goo.gl/mh7qDG
---
You received this message because you are subscribed to the Google Groups "CAS Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cas-user+u...@apereo.org.
To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/cas-user/CAOGev1jGXZNNoTTod%2B04s5pELEyX20%3D78oSQyNGsLK5MoOxu1Q%40mail.gmail.com.