HTTPS access with browser works fine, with python causes SSL error

[a.m....@gmail.com]

Hi,

  My XNAT uses HTTPS instead of the default HTTP (to stop IT from shutting down the host). The configuration is via docker-compose, I have made changes in the "server" sections of nginx.conf for the nginx docker (I have attached the modified file).

  The system works fine for access with the browser; I can do everything I want: make projects, assign users, etc. and always using the HTTPS protocol.

  But when I want to upload a large number of NIfTI images, I prefer to use Python scripts, via the XNAT package [https://xnat.readthedocs.io]. When I connect this way, I get the following error (see below).

  I first thought it was something on the XNAT host side, because the command 'curl' gives an error as well: "curl: (60) Peer's Certificate issuer is not recognised." The certificate was given to me by my IT department, and comes from GEANT (I don't know what that is)

  But then on the same local computer where Python XNAT and curl give an error, I can just access and modify XNAT projects via the browser interface. So some of my local software can just access the XNAT host, suggesting there may be a local solution.

   

  Does anyone know what causes this (and how to solve it)? Many thanks!

===================

error message in python

===================

$ python3
Python 3.8.10 (default, Mar 15 2022, 12:22:08)  
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import xnat;
>>> xnat.connect ( host='https://rng-xnat.ecloud.vumc.nl' )   
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
TypeError: connect() got an unexpected keyword argument 'host'
>>> xnat.connect ( 'https://rng-xnat.ecloud.vumc.nl' )      
Traceback (most recent call last):
 File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 665, in urlopen
   httplib_response = self._make_request(
 File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 376, in _make_request
   self._validate_conn(conn)
 File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 996, in _validate_conn
   conn.connect()
 File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 366, in connect
   self.sock = ssl_wrap_socket(
 File "/usr/lib/python3/dist-packages/urllib3/util/ssl_.py", line 370, in ssl_wrap_socket
   return context.wrap_socket(sock, server_hostname=server_hostname)
 File "/usr/lib/python3.8/ssl.py", line 500, in wrap_socket
   return self.sslsocket_class._create(
 File "/usr/lib/python3.8/ssl.py", line 1040, in _create
   self.do_handshake()
 File "/usr/lib/python3.8/ssl.py", line 1309, in do_handshake
   self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1131)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
 File "/usr/lib/python3/dist-packages/requests/adapters.py", line 439, in send
   resp = conn.urlopen(
 File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 719, in urlopen
   retries = retries.increment(
 File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 436, in increment
   raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='rng-xnat.ecloud.vumc.nl', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1131)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
 File "/usr/local/lib/python3.8/dist-packages/xnat/__init__.py", line 515, in connect
   redirect_check_response = requests_session.get(server)
 File "/usr/lib/python3/dist-packages/requests/sessions.py", line 546, in get
   return self.request('GET', url, **kwargs)
 File "/usr/lib/python3/dist-packages/requests/sessions.py", line 533, in request
   resp = self.send(prep, **send_kwargs)
 File "/usr/lib/python3/dist-packages/requests/sessions.py", line 646, in send
   r = adapter.send(request, **kwargs)
 File "/usr/lib/python3/dist-packages/requests/adapters.py", line 514, in send
   raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='rng-xnat.ecloud.vumc.nl', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1131)')))

[Herrick, Rick]

This is very similar to this recent topic on the discussion group. The messages are different, but I suspect that’s just because that one is coming from Electron and yours is coming from the requests Python library and that it’s the same root cause. One thing for certain is that it’s not XNAT itself because it never gets involved in handling SSL. Check out the topic above and this stackoverflow post for possible diagnostics and solutions.

 

The thing I find weird about this is how Electron and Python don’t get weird about the certificate chain for most transactions but then break on certain calls. Maybe it’s keyed to certain content types?

 

From: xnat_di...@googlegroups.com <xnat_di...@googlegroups.com> on behalf of a.m....@gmail.com <a.m....@gmail.com>
Date: Friday, July 8, 2022 at 9:34 AM
To: xnat_discussion <xnat_di...@googlegroups.com>
Subject: [XNAT Discussion] HTTPS access with browser works fine, with python causes SSL error

* External Email - Caution *

--
You received this message because you are subscribed to the Google Groups "xnat_discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to xnat_discussi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/xnat_discussion/5c8dd594-4b54-4c2e-b98b-89a609539f47n%40googlegroups.com.

 

---

The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.

[a.m....@gmail.com]

Thanks Rick it is (a bit of) a relief that it's not XNAT. If I'm honest, I think that I use quite an exotic certificate (not by choice) and this is OK for the browser, but not for Python and curl. No idea why, but I would love to hear from someone who's solved it. As you say it may be outside the scope of this group.

For now I think I have found a solution, at least for the package [https://xnat.readthedocs.io] I'm using. Hoping that it will help others:

1. In this package, the a connection is made with the xnat class's member function connect: [https://xnat.readthedocs.io/en/latest/xnat.html#module-xnat]

     for me, this works as described with HTTP without further options, but not for HTTPS and my 'special' certificate.

2. This function has a boolean parameter verify, and setting this to False skips the certificate check

     that's not the way it should be of course, and many warnings are issued.

Other than the extensive screen output because of all the warnings (which may be controlled by the logging level), this is what I was looking for -- until there is a way to just use the certificates in python.