Cannot add remote host with Container Service 3.6.0

[Paul Smith]

Hi,

I'm testing upgrading our XNAT instances to 1.9.0 but I'm running into an issue with the latest version of the Container Service.

If I try to add a host to the Container Service, an IllegalArgumentException is raised (see configuration.log and xapi.log). I get the same error if I use the REST API or the web UI (see attached images for the config I'm setting in the web UI).

I'm using:

Rocky Linux 9.4

openjdk version "1.8.0_432"

tomcat 9.0.82

Docker version 27.3.1, build ce12230

postgres 12.21

XNAT 1.9.0

Container Service Plugin 3.6.0

Batch launch plugin 0.7.0 (and the pipeline engine is not installed)

I have a separate server for the Container Service, and am running it in Docker mode (swarm is not enabled). Any ideas where I might be going wrong?

Thanks,

Paul

[John Flavin]

That's interesting. I haven't seen that error before.

I think the reason this is happening is that we switched the underlying Java library CS uses for Docker API communication from dmandalidis/docker-client to docker-java/docker-java. It seems the way they handle TLS configuration is different.

Reading through their Getting Started and Contributing docs, I would guess that you need to use the tls:// scheme rather than https://. You will also likely need to set the Certificate Path if you aren't doing that already.

If we're lucky, that will be enough to get you on your way. My worry is that this still won't be enough. The docker-java client may need CS to set the DOCKER_TLS_VERIFY setting to tell it that you want to use TLS/HTTPS, but we are currently not doing that. (And if you could do me a favor and not hit the "Blame" button on that code to check who wrote it but forgot to set this configuration option, and also who left the prescient comment TODO are there other things that need to be configured here?, I'd appreciate it.) It would be a very simple matter to add that, but it would require a new release be cut, which could take some time.

In short: change https:// to tls://, ensure you’re providing the path to the certificates, and hope the docker-java client is nice to us.

John Flavin

--
You received this message because you are subscribed to the Google Groups "xnat_discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to xnat_discussi...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/xnat_discussion/8b62feb9-1c81-40f9-91ed-512a82b1f216n%40googlegroups.com.

[Paul Smith]

Thanks for getting back to me so quickly.

Using tcp:// does help - I don't get the 'Unsupported protocol scheme' error anymore. But as you feared, I think the  DOCKER_TLS_VERIFY   flag needs to be set in the CS to enable connecting via HTTPS (or the CS needs to set it via  configBuilder.withDockerTlsVerify(verify) )

If I disable TLS on my docker host, then I can connect to it okay using the container service. But if I enable it TLS on my host, then I get 'Connection refused' errors in the configuration logs (see attached). And if I try to ping the server when TLS is enabled on the host (using the endpoint  /xapi/docker/server/ping ) then I get the error:

com.github.dockerjava.api.exception.BadRequestException: Status 400: Client sent an HTTP request to an HTTPS server.

 

A couple of things I'm confused about:

1. The docker-java docs suggest I should be able to set  DOCKER_TLS_VERIFY   as an environment variable on the client, and that this will be picked up. But it seems to be ignored for some reason.

2. The docker-java docs also suggest that  DOCKER_TLS_VERIFY  toggles whether to use HTTP or HTTPS, but it's perhaps worded slightly confusingly. I think the behaviour of `DOCKER_TLS_VERIFY`  is:

- if it's undefined, TLS is not used

- if it's defined and set to false (or 0), TLS is used but the certificates are not verified

- if it's defined and set to true (or 1), TLS is used and the certificates are verified

To handle these three situations, would the Container Service also need to take an addition parameter when configuring a remote host to control whether certs are verified or not?

Thanks,

Paul 

P. S. It seems the blame button is broken for me so we'll never know the culprit

[Will Horton]

Please correct any part of this if it doesn't seem suitable: 

My inclination from a usability perspective would be to leave DOCKER_TLS_VERIFY unset unless the tls:// protocol appears in the host path input. Once tls:// is read, a checkbox or radio button could appear beneath the certificate path input, asking the admin user to specify whether they want that certificate to be verified or not. 

We could also look for other protocol strings such as tcp: and https: as needed and support the same behavior. 

[John Flavin]

1. The docker-java docs suggest I should be able to set  DOCKER_TLS_VERIFY   as an environment variable on the client, and that this will be picked up. But it seems to be ignored for some reason.

You could try adding it as a system property when starting XNAT, something along the lines of  -DDOCKER_TLS_VERIFY=1 (Though I don't remember the best place to add this to the tomcat args. Maybe someone else can chime in with help there.)

2. The docker-java docs also suggest that  DOCKER_TLS_VERIFY  toggles whether to use HTTP or HTTPS, but it's perhaps worded slightly confusingly. I think the behaviour of `DOCKER_TLS_VERIFY`  is:

- if it's undefined, TLS is not used

- if it's defined and set to false (or 0), TLS is used but the certificates are not verified

- if it's defined and set to true (or 1), TLS is used and the certificates are verified

To handle these three situations, would the Container Service also need to take an addition parameter when configuring a remote host to control whether certs are verified or not?

The behavior I had in mind is that if the Certificate Path is provided, we would set DOCKER_TLS_VERIFY to true (or the equivalent through the docker-client Java API), otherwise we leave it unset. But as you pointed out that would not cover all three cases, only true and unset. To cover the explicit false case, maybe we could use this logic:

• If the host scheme is not tcp://, leave DOCKER_TLS_VERIFY unset.
• If the scheme is tcp:// but cert path is unset, set DOCKER_TLS_VERIFY to false.
• If the scheme is tcp:// and the cert path is set, set DOCKER_TLS_VERIFY to true.

That seems to me that it covers all the options and tracks with the user's intent without requiring an explicit configuration setting. But it would need to be documented. (...and implemented.)

John Flavin

To view this discussion visit https://groups.google.com/d/msgid/xnat_discussion/cb491c93-56cd-4a11-8c40-2dbd25bb7116n%40googlegroups.com.

[Will Horton]

That seems like a more elegant solution than the one I suggested. I'll include that in our ticket. 

[Paul Smith]

> The behavior I had in mind is that if the Certificate Path is provided, we would set DOCKER_TLS_VERIFY to true (or the equivalent through the docker-client Java API), otherwise we leave it unset. But as you pointed out that would not cover all three cases, only true and unset. To cover the explicit false case, maybe we could use this logic:

• If the host scheme is not tcp://, leave DOCKER_TLS_VERIFY unset.
• If the scheme is tcp:// but cert path is unset, set DOCKER_TLS_VERIFY to false.
• If the scheme is tcp:// and the cert path is set, set DOCKER_TLS_VERIFY to true.

That seems to me that it covers all the options and tracks with the user's intent without requiring an explicit configuration setting. But it would need to be documented. (...and implemented.)

Based on this issue, I think the only supported scheme is tcp:// - using http://, https://, or tls:// will raise the 'Unsupported protocol scheme' error.

And for the second case (if the scheme is tcp:// but cert path is unset, set DOCKER_TLS_VERIFY to false), I think this would be slightly incorrect behaviour. If DOCKER_TLS_VERIFY is false , then the cert path will still need to be defined - the certs will be used to connect using TLS, but they will not be verified with root authorities (which is useful if you have self-signed certs).

So I think Will's idea of adding a checkbox to set the flag makes sense. And perhaps the `/xapi/docker/server` endpoint could take an optional `tls_verify` parameter that behaves in the same way as DOCKER_TLS_VERIFY?

Thanks,

Paul

You received this message because you are subscribed to a topic in the Google Groups "xnat_discussion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/xnat_discussion/h1ODNrTIQnY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to xnat_discussi...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/xnat_discussion/CADwXCSrsQAAOOx%3Dr_Z2HWS0jjJWyD6APq_ugsayiWkAMAqXJ0g%40mail.gmail.com.

[Daniel Beasley]

Hi all,

I updated to 1.9.2, with CS 3.7. In setenv I set  -DDOCKER_TLS_VERIFY=1 and changed my host from https to tcp with certificates. All good until I run a container - I get  authConfig not specified.   However I can run in plain docker mode, not swarm. TI have an XNAT running XNAT 1.8.10, as it did before, and works with swarm. The only difference is switching https to tcp. We have a private docker registry that has not changed, and can do everything until I run  the container.

Any ideas?

Many thanks,

Dan

[kel...@wustl.edu]

Hi Dan, 
I saw that 'authConfig' error during an upgrade. In my case it was related to the url of our private Docker registry. The new Docker client seemed not like http/https included in the registry url. I changed 'https://ourserver.wustl.edu' to 'ourserver.wustl.edu' and that fixed it.

-Matt

[Daniel Beasley]

Many thanks, it worked!