DSpace 7.2 behind a F5 internal firewall with wildcard certificate

[Sean Carte]

I'm starting to think that it might not be possible to do this, but I'm hoping someone will tell me how very wrong I am.

I have a server that is in our institution's DC, which is behind an F5 firewall and there is a wildcard Sectigo certificate installed.

My dspace local.cfg has the following settings:

dspace.server.url = https://crdb.dut.ac.za/server
dspace.ui.url = https://crdb.dut.ac.za

I requested a certificate and key from the administrator and have this in /etc/apache2/sites-enabled/ssl.conf

SSLCertificateFile /etc/certs/crdb.dut.ac.za/cert.pem
SSLCertificateKeyFile /etc/certs/crdb.dut.ac.za/privkey.pem

SSLCertificateChainFile /etc/certs/crdb.dut.ac.za/chain.pem

Include /etc/certs/options-ssl-apache.conf

    # Proxy all HTTPS requests to "/server" from Apache to Tomcat via AJP connector
    ProxyPass /server ajp://localhost:8009/server
    ProxyPassReverse /server ajp://localhost:8009/server

    # Proxy all HTTPS requests from Apache to PM2 on port 4000
        ProxyPass / http://localhost:4000/
        ProxyPassReverse / http://localhost:4000/

config.prod.yml

ui:
  ssl: false
  host: localhost
  port: 4000
  nameSpace: /
rest:
  ssl: true
  host: crdb.dut.ac.za
  port: 443
  nameSpace: /server

When I run yarn test:rest:

...Testing connection to REST API at https://crdb.dut.ac.za/server/api...

ERROR connecting to REST API
Error: unable to verify the first certificate

As expected, if I run node ./dist/server/main.js, I get the dreaded 'No _links section' error:

GET / 500 1356.467 ms - 231171
No _links section found at https://crdb.dut.ac.za/server/api

Am I wasting my time with this, or has anybody managed to get this working in a similar situation with a proxy firewall handing out a wildcard certificate?

Sean

[John Bain]

come in the dspace slack and maybe I can help you 

[Thiago Henrique Carvalho da Costa]

I have the same problem, did anyone manage to solve it?

Ícone "Verificada pela comunidade"

[Sean Carte]

Hi Thiago

No, I'm still stuck with this problem.

Sean

--
All messages to this mailing list should adhere to the Code of Conduct: https://www.lyrasis.org/about/Pages/Code-of-Conduct.aspx
---
You received this message because you are subscribed to the Google Groups "DSpace Technical Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dspace-tech...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dspace-tech/33a6bd12-384e-477f-a165-6dedf18b3d98n%40googlegroups.com.

[Plate, Michael]

Hi Sean,

your certificate chain is broken:

openssl s_client -connect crdb.dut.ac.za:443

CONNECTED(00000003)
depth=0 C = ZA, postalCode = 4001, ST = KwaZulu-Natal, L = Durban, street = Overport, street = 7 Ritson Road, O = Durban University of Technology, OU = ITSS, CN = *.dut.ac.za
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 C = ZA, postalCode = 4001, ST = KwaZulu-Natal, L = Durban, street = Overport, street = 7 Ritson Road, O = Durban University of Technology, OU = ITSS, CN = *.dut.ac.za
verify error:num=21:unable to verify the first certificate
verify return:1
depth=0 C = ZA, postalCode = 4001, ST = KwaZulu-Natal, L = Durban, street = Overport, street = 7 Ritson Road, O = Durban University of Technology, OU = ITSS, CN = *.dut.ac.za
verify return:1
---
Certificate chain
0 s:C = ZA, postalCode = 4001, ST = KwaZulu-Natal, L = Durban, street = Overport, street = 7 Ritson Road, O = Durban University of Technology, OU = ITSS, CN = *.dut.ac.za
i:C = GB, ST = Greater Manchester, L = Salford, O = Sectigo Limited, CN = Sectigo RSA Organization Validation Secure Server CA
---
[…]

browsers accept this, other programs are more picky about chain order .
If you cant't get around it, try letsencrypt and install certbot (its in debian packages, no need for snap)


Michael

________________________________________
Von: dspac...@googlegroups.com <dspac...@googlegroups.com> im Auftrag von Sean Carte <sean....@gmail.com>
Gesendet: Donnerstag, 7. Juli 2022 07:54
An: Thiago Henrique Carvalho da Costa
Cc: DSpace Technical Support
Betreff: Re: [dspace-tech] Re: DSpace 7.2 behind a F5 internal firewall with wildcard certificate

[…]

[Sean Carte]

Thanks, Michael. That's useful. I'll follow up with our IT department.

Sean

--
All messages to this mailing list should adhere to the Code of Conduct: https://www.lyrasis.org/about/Pages/Code-of-Conduct.aspx
---
You received this message because you are subscribed to the Google Groups "DSpace Technical Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dspace-tech...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/dspace-tech/d00aa2e38fde4d2b8d28b164d724ce99%40bibliothek.uni-kassel.de.

[Sean Carte]

Ah, but wait, I remembered the chain of events that led to me installing the cert, whose chain is broken:

The F5 firewall seems to provide certification through its wildcard certificate. So if you visit our current DSpace-CRIS 5 repository at https://openscholar.dut.ac.za/  and check the connection security for that site, you will see that it is verified by Sectigo Ltd. However, on that server, I'm using a self-signed certificate. (It used to be LetsEncrypt before the F5.)

/etc/apache2/sites-enabled/default-ssl.conf

                SSLCertificateFile      /etc/ssl/certs/apache-selfsigned.crt
                SSLCertificateKeyFile /etc/ssl/private/apache-selfsigned.key

That didn't work for DSpace 7 (I forget the exact error, but I suspect it was the verification error). So I requested the certificate from the IT admin, and installed that.

But it seems as though that doesn't even get seen by openssl s_client ...

For comparison, if I run

openssl s_client -connect openscholar.dut.ac.za:443

I get a similar error: Verification error: unable to verify the first certificate.

I'm really out of my depth here and not sure who or where to seek help. All I know is that I can get this working unless it's behind the F5. But then, in that case, I'm using LetsEncrypt.

Sean

[Mark H. Wood]

On Fri, Jul 08, 2022 at 11:24:37AM +0200, Sean Carte wrote:
> Ah, but wait, I remembered the chain of events that led to me installing
> the cert, whose chain is broken:
>
> The F5 firewall seems to provide certification through its wildcard
> certificate. So if you visit our current DSpace-CRIS 5 repository at
> https://openscholar.dut.ac.za/ and check the connection security for that
> site, you will see that it is verified by Sectigo Ltd. However, on that
> server, I'm using a self-signed certificate. (It used to be LetsEncrypt
> before the F5.)
>
> /etc/apache2/sites-enabled/default-ssl.conf
> SSLCertificateFile /etc/ssl/certs/apache-selfsigned.crt
> SSLCertificateKeyFile /etc/ssl/private/apache-selfsigned.key
>
> That didn't work for DSpace 7 (I forget the exact error, but I suspect it
> was the verification error). So I requested the certificate from the IT
> admin, and installed that.
>
> But it seems as though that doesn't even get seen by openssl s_client ...
>
> For comparison, if I run
> openssl s_client -connect openscholar.dut.ac.za:443
>
> I get a similar error: Verification error: unable to verify the first
> certificate.
>
> I'm really out of my depth here and not sure who or where to seek help. All
> I know is that I can get this working unless it's behind the F5. But then,
> in that case, I'm using LetsEncrypt.

Your wildcard certificate for *.dut.ac.za is signed by "C = GB, ST =

Greater Manchester, L = Salford, O = Sectigo Limited, CN = Sectigo RSA

Organization Validation Secure Server CA" but that cert. is not in your
client's trust store. It's not in the trusted certificates bundle on
my workstation, either. Google has a lot of hits on "sectigo root
certificate not trusted" which may shed some light. You may need help
from Sectigo customer support. Or there may be some reason why that
particular cert. is no longer trusted.

Or it may be an intermediate authority whose cert. should be sent out
with the server cert. to complete the trust chain to the root. In
that case, you may need to get a copy of that cert. and install it in
the F5's trust store (once you're sure that *you* trust it).

A client must be able to construct a path from the certificate
presented by the site, through that cert.'s issuer cert. ("i:" in the
certificate chain) via *its* issuer, etc. until it reaches a root
certificate that it already trusts.

--
Mark H. Wood
Lead Technology Analyst

University Library
Indiana University - Purdue University Indianapolis
755 W. Michigan Street
Indianapolis, IN 46202
317-274-0749
www.ulib.iupui.edu

[Michael Plate]

Hi Sean,

Am 08.07.22 um 11:24 schrieb Sean Carte:

> Ah, but wait, I remembered the chain of events that led to me installing
> the cert, whose chain is broken:
>
> The F5 firewall seems to provide certification through its wildcard
> certificate. So if you visit our current DSpace-CRIS 5 repository at

> https://openscholar.dut.ac.za/ <https://openscholar.dut.ac.za/>  and

> check the connection security for that site, you will see that it is
> verified by Sectigo Ltd. However, on that server, I'm using a
> self-signed certificate. (It used to be LetsEncrypt before the F5.)

[…]

I'm not sure if I can follow you…we only have IP-based FWs, the
encryption is done always on the host.
You have a self-signed cert and the F5 decrypts inside - like a browser
/ proxy- and encrypts again with another cert to outside ?

Michael

[Mohammad S. AlMutairi]

If your apache SSL configuration is correct you can force the NodeJS and the starting of the service later to access crdb.dut.ac.za directly using the server local IP instead of going thru the Public-IP/F5. You can do it by adding an entry into /etc/hosts for the server private IP address. The other way of overcoming this issue is by skipping the /etc/hosts file entry and doing the steps you see below:

A) echo "export NODE_EXTRA_CA_CERTS=/etc/certs/crdb.dut.ac.za/cert.pem" >> /etc/environment

B) source /etc/environment

C) yarn test:rest

Good luck

[Sean Carte]

Thank you, Mohammad! I added an entry to /etc/hosts and it does appear to work.

(I also tried your /etc/environment suggestion, but still got the 'unable to verify the first certificate' error.)

Thanks again, I really appreciate your help.

Sean

To view this discussion on the web visit https://groups.google.com/d/msgid/dspace-tech/15623b2a-d4c2-49f9-bb8a-88e21b67cc51n%40googlegroups.com.

[Sean Carte]

Sorry, Michael, if I understood it myself, I might be able to explain it better.

But thanks for your help in resolving this.

Sean

--
All messages to this mailing list should adhere to the Code of Conduct: https://www.lyrasis.org/about/Pages/Code-of-Conduct.aspx
---
You received this message because you are subscribed to the Google Groups "DSpace Technical Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dspace-tech...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/dspace-tech/550e7ada-9d67-00bd-8eb7-0b24fb0dc867%40bibliothek.uni-kassel.de.

[Sean Carte]

Thanks for the input, Mark. Unfortunately, I am not in a position to suggest changes to the F5's configuration. Fortunately, Mohammad's workaround worked.

Sean

--
All messages to this mailing list should adhere to the Code of Conduct: https://www.lyrasis.org/about/Pages/Code-of-Conduct.aspx
---
You received this message because you are subscribed to the Google Groups "DSpace Technical Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dspace-tech...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/dspace-tech/Ysguzgx1FU%2BATiUl%40IUPUI.Edu.

[Mohammad S. AlMutairi]

On Saturday, July 9, 2022 at 10:08:20 AM UTC+3 sean....@gmail.com wrote:

Thank you, Mohammad! I added an entry to /etc/hosts and it does appear to work.

(I also tried your /etc/environment suggestion, but still got the 'unable to verify the first certificate' error.)

The attached bundle file should do it or if you prefer you can use /etc/certs/crdb.dut.ac.za/chain.pem instead if the certificate installed on the F5 appliance is identical to the certificate you had configured in the webserver.

A) unset NODE_EXTRA_CA_CERTS

B) sed -i '/^export NODE_EXTRA_CA_CERTS=/d' /etc/environment

C) echo "export NODE_EXTRA_CA_CERTS=/etc/certs/crdb.dut.ac.za/chain.pem" >> /etc/environment

D) source /etc/environment

E) yarn test:rest

 

Thanks again, I really appreciate your help.

Glad it worked for you and it will always be my pleasure to help.

[hb wooley]

I hope this might help someone, this is how I got past the "unable to verify the first certificate" error. This was completed on RedHat server with a wildcard certificate.

Connect to your server and output the certificate as "logcertfile".
openssl s_client -connect <example.com>:443 -servername <example.com> | tee logcertfile

Grep for the issuer:
openssl x509 -in logcertfile -noout -text | grep -i "issuer"

Use the above issuer to get the issuer certificate. Use the below to get the issuercertificate and output it as intermediate.crt
curl --output intermediate.crt http://<issuer>/<some_certificate>.crt

Locate your Apache/HTTPD ssl.conf file look for "SSLCertificateFile", use this certificate for the below steps. Mine was in the PEM format, so I changed the format from CRT to PEM
openssl x509 -inform DER -in intermediate.crt -out intermediate.pem -text

Just to be safe, make a copy of the original certificate.
cp /etc/pki/tls/certs/<example.com>.pem <example.com>.pubcert.with.chain.pem

Append the intermediate.pem to your SSL certificate (SSLCertificateFile):
intermediate.pem >> /etc/pki/tls/certs/<example.com>.pubcert.with.chain.pem
NOTE: this certificate will need to have all of your trusted certificates (chain) added to it.

Modify your ssl.conf file to point to the new certificate:
SSLCertificateFile /etc/pki/tls/certs/<example.com>.pubcert.with.chain.pem
check then restart Apache/HTTPD
apachectl configtest
apachectl graceful

Add NODE_EXTRA_CA_CERTS to dspace’s .bash_profile:
export NODE_EXTRA_CA_CERTS='/etc/pki/tls/certs/<example.com>.pubcert.with.chain.pem'
source ~/.bash_profile
or logout the back in again

Now you should be able run:
cd /dspace-angular-dspace-7.3/
yarn test:rest

RESPONSE: 200
Checking JSON returned for validity...
        "dspaceVersion" = DSpace 7.2.2-SNAPSHOT
        "dspaceUI" = http://localhost:4000
        "dspaceServer" = https://<example.com>/server
        "dspaceServer" property matches UI's "rest" config? true
        Does "/api" endpoint have HAL links ("_links" section)? true
Done in 3.75s.

For testing you can use:
cd /dspace-ui-deploy
NODE_EXTRA_CA_CERTS='/etc/pki/tls/certs/<example.com>.pubcert.with.chain.pem' node ./dist/server/main.js

To run in production:
cd /dspace-ui-deploy
Edit and add "NODE_EXTRA_CA_CERTS" to dspace-ui.json
{
    "apps": [
        {
           "name": "dspace-ui",
           "cwd": "/dspace-ui-deploy",
           "script": "dist/server/main.js",
           "env": {
              "NODE_ENV": "production",
              "NODE_EXTRA_CA_CERTS": "/etc/pki/tls/certs/<example.com>.pubcert.with.chain.pem"
           }
        }
    ]
}
pm2 start dspace-ui.json


I hope I didn't leave anything out, I did this a couple weeks back.

[Mohammad S. AlMutairi]

@hb Setting the NODE_EXTRA_CA_CERTS variable do work but it requires extra steps as you posted in the instructions. Try the /etc/host file entry instead. It's the easiest and the cleanest approach of all known ways to tackle this issue or similar ones.