How to enable DEBUG logs for Xnat Container Service plugin code (logback + remote debug)

[Richard Guo]

Dear XNAT Discussion Group,

Environment
• XNAT running on Kubernetes (namespace: ais-xnat), plugin: Container Service
• Logs present: containers.log (+ rotated files)
• Problem: I added SLF4J statements in Container Service–related classes (e.g., log.debug("...")), but I cannot see any DEBUG lines in containers.log or other container-service logs. VS Code remote debugging has also not attached successfully yet.

What I have tried

1. Verified code uses SLF4J (org.slf4j.LoggerFactory) and the package is under org.nrg.containers....

2. Enabled loggers in $XNAT_HOME/config/logback.xml and restarted Tomcat/XNAT:
   • org.nrg.containers at DEBUG
   • org.nrg.containers.api at DEBUG
   • org.nrg.containers.services at DEBUG
   • org.nrg.containers.kubernetes at DEBUG

3. Confirmed appender routing: the same appender used for containers.log should capture these categories.

4. Reproduced the workflow (cache-recall / dcm2niix launch), but DEBUG lines did not appear.

5. Searched with grep -rni DEBUG containers.log* and monitored with tail -f containers.log.

Questions

1. Are there additional logger categories I should enable for Container Service (or command resolution) to surface DEBUG output? For example, org.nrg.framework, org.nrg.xnatx.plugins, or others used by the plugin?

2. Is there a recommended, up-to-date sample logback.xml snippet for enabling full Container Service DEBUG in XNAT 1.9.x?

3. For remote debugging on Kubernetes: is the approach below correct?
   • Set JAVA_TOOL_OPTIONS="-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005" on the xnat-web container
   • Expose containerPort 5005 and kubectl port-forward svc/xnat-web-debug 5005:5005
   • Attach VS Code on localhost:5005 with sources matching the deployed JAR

4. Does the Container Service plugin route some logs through a different category or level (e.g., INFO only) during orchestration steps, which could explain why log.debug lines do not appear?

5. If there is a known interaction with shaded logging bindings in the plugin JAR, what is the recommended fix?

If helpful, I can share:
• A minimal code snippet with the log.debug call and its package/class name
• My current logback.xml relevant sections (logger + appender)
• The exact Container Service version and XNAT version

Many thanks for any guidance or a canonical logging/debug configuration.

Kind regards,
Chenxi (Richard) Guo

[Rick Herrick]

I think the issue is probably that $XNAT_HOME/config/logback.xml doesn't really mean anything to XNAT. The config folder isn't on XNAT's classpath at all. The plugins folder is, which is how plugin jars are added to the run-time space, but that's limited to jar files, e.g. you can't just add resources or class files to that folder and have them picked up.

You can try a couple different things:

• Extract the containers-logback.xml from your plugin jar, update that, then add it back to the jar file (this requires a restart of Tomcat)
• If you're on a later version, I think this was in 1.9.2 but not 100% sure, we added a logging configuration panel to the XNAT site administration page, which lets you set the current logging level of each particular appender live on the server without a restart

I can't say for sure about debugging into XNAT on Kubernetes, but that looks correct to me. The one thing you may need to change is "address=*:5005". That "*:" is necessary in later versions of Java but I'm not sure it works properly with Java 8. Or then again it might! All of our standard configurations under Java 8 omit that so that it's just "address=5005".

Re #4, no, the logging configuration remains the same throughout the lifecycle. The only time any log levels will change once initialized is through the logging configuration panel.

Rick Herrick 

Senior Software Developer

ri...@xnatworks.io

https://xnatworks.io | Find us on LinkedIn! 

--
You received this message because you are subscribed to the Google Groups "xnat_discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to xnat_discussi...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/xnat_discussion/a3e2dbbe-b970-4b04-9545-b1703180cf53n%40googlegroups.com.

[John Flavin]

If you’re already building the container service jar yourself, it would likely be easiest to directly edit the container service's logging configuration file. Which, for reference, is this: https://bitbucket.org/xnatdev/container-service/src/master/src/main/resources/META-INF/resources/containers-logback.xml

You'll see a few different logging configurations there.

• The main one captures everything from org.nrg.containers into containers.log
• ...except for command resolution in org.nrg.containers.services.impl.CommandResolutionServiceImpl, which is quite verbose and gets logged separately into the file containers-services-commandresolution.log.
• The kubernetes client logs from io.kubernetes.client get routed to containers-kubernetes.log
• We still have a logging configuration for the older docker client org.mandas but haven't added one for the new docker client com.github.dockerjava. We should probably fix that.

John Flavin

To view this discussion visit https://groups.google.com/d/msgid/xnat_discussion/CAFykhbC_MMbN31TLh2gB8igdMjOqQYO2reyAoC2JpXiut5UqKQ%40mail.gmail.com.

[Richard Guo]

Hi John and Rick,

I’ve updated the Container Service logging config as follows:

• File: containers-logback.xml

• Changes: raised org.nrg.containers to DEBUG and kept subsystem routing (CommandResolutionServiceImpl, Kubernetes client) as is.
  
  Do I need to rebuild the plugin again?
  
  Chenxi Guo

To view this discussion visit https://groups.google.com/d/msgid/xnat_discussion/CADwXCSq5XLhcLt57GJJ4QxpDa8OaQJGgp-CbUbrHUzPbApDdQg%40mail.gmail.com.

[John Flavin]

You do need to get that edited file into the plugin jar. One way to do that is to rebuild the jar, yes, and that is probably the easiest-to-understand way. Rick mentioned a different technique, which is to simply copy that edited file to the right location in the jar. If you’re comfortable with the jar CLI tool, you could give that a try.

John Flavin

To view this discussion visit https://groups.google.com/d/msgid/xnat_discussion/CAJD6T44ff3F4RO9j3tHybT4h79aQYtF4iqYmHkC_rmjVGJ92ww%40mail.gmail.com.

[Richard Guo]

Hi John,

I’ve rebuilt the JAR file, moved it to the correct directory, and restarted the pod. However, the log output still looks like this:
2025-10-12 12:48:39,027 [http-nio-8080-exec-5] ERROR org.nrg.containers.api.DockerControlApi - Hub status check created exception.
java.lang.RuntimeException: java.io.IOException: com.sun.jna.LastErrorException: [2] No such file or directory
    at com.github.dockerjava.okhttp.UnixSocketFactory.createSocket(UnixSocketFactory.java:23)
    ...

As you can see, there’s no DEBUG information appearing in the logs — only the same error messages as before.

Could you please let me know if there’s another configuration or environment variable I should check to enable debug-level logging?

Best regards,
Chenxi

To view this discussion visit https://groups.google.com/d/msgid/xnat_discussion/CADwXCSpaco_hPEHr8AbbGDTg3yX1FdEQQGkkLMiLDU%2B%2ByD%2BQfA%40mail.gmail.com.

[Richard Guo]

Dear John,

I figured it out, thank you so much for your kind guidance~

Kind regards

Chenxi 

On Sat, 11 Oct 2025 at 02:04, John Flavin <fla...@gmail.com> wrote:

To view this discussion visit https://groups.google.com/d/msgid/xnat_discussion/CADwXCSpaco_hPEHr8AbbGDTg3yX1FdEQQGkkLMiLDU%2B%2ByD%2BQfA%40mail.gmail.com.