Resource manager change listener performance and tuning

[Brad Wood]

In the latest version of my Undertow-powered tool, I've added support for change listeners in my custom resource manager.  I really like this feature as it allows my welcome file cache and servlet path cache to be cleared automatically.  

Since my release, I've had a couple users complain about the performance of the XNIO watcher when they have a large number of files in the base path for the resource manager.  One user had a CMS with a large amount of static files-- around 100,000 files and they saw a delay of about 60 seconds while starting up the XNIO watcher.  Another user had 280,000 files in their base path and their server wouldn't even start, throwing the following error

java.lang.OutOfMemoryError: Java heap space
at java.base/java.io.UnixFileSystem.resolve(Unknown Source)
at java.base/java.io.File.<init>(Unknown Source)
at java.base/java.io.File.listFiles(Unknown Source)
at org.xnio.nio.WatchServiceFileSystemWatcher.doScan(WatchServiceFileSystemWatcher.java:237)
at org.xnio.nio.WatchServiceFileSystemWatcher.watchPath(WatchServiceFileSystemWatcher.java:182)
...

I really like the watcher and I'd like to leave it on by default, but I'm curious how other people handle this.  It's a low level sort of feature that my users aren't really going to understand or care about and they want the server tool to "just work" in all cases, lol.

• Is there a reliable way to detect when there will be too many files for the watcher to perform well and disable it programmatically?
• Would it work to try and start the watcher asynchronously, or does the registration of the listener need to be a blocking operation?
• Is there a way to tune the XNIO watchers such as ignoring certain subfolders like node_modules, etc?
• Is there a way to have the watcher fail more gracefully than a java heap space error when there are a large number of files?
• Have other people run into this, or is the change listener really not a super common feature?

Thanks!

~Brad

Developer Advocate

Ortus Solutions, Corp 

E-mail: br...@coldbox.org

ColdBox Platform: http://www.coldbox.org 

Blog: http://www.codersrevolution.com

[Stuart Douglas]

This is enabled in WildFly for exploded deployments. Basically we just recommend that users turn it off if they have too many files, but it does not seem to be a huge issue for most people.

All of your suggestions above are workable, I think you could do most of them by creating your own org.xnio.FileSystemWatcher implementation that decides which paths to actually watch, starts lazily etc and then registers concrete listeners accordingly.

Stuart

--
You received this message because you are subscribed to the Google Groups "Undertow Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to undertow-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/undertow-dev/CALbQ1o%3D0va7k17bSG%2B87M68ZetDpg%3D49Q4h5-vC6Sk-qJR6Nww%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.