DSpace 7.6.1 Configuration Issues with Apache HTTPD and SSL

[Arta Seyedian]

DSpace 7.6.1 Configuration Issues with Apache HTTPD and SSL

Hello everyone,

I’m currently setting up a DSpace 7.6.1 instance for an internally-hosted and accessed metadata database and have encountered several issues that I’m struggling to resolve.

Below, I have my relevant config files listed out. But first, I will address the issue I’m encountering.

The results of yarn test:rest are exactly what you would expect from a working setup:

[dspace@pedsdspace01 dspace-angular-dspace-7.6.1]$ yarn test:rest
yarn run v1.22.22
$ ts-node --project ./tsconfig.ts-node.json scripts/test-rest.ts
Building production app config
Overriding app config with /home/dspace/dspace-angular-dspace-7.6.1/config/config.yml
Overriding app config with /home/dspace/dspace-angular-dspace-7.6.1/config/config.prod.yml
...Testing connection to REST API at https://pedsdspace01.research.chop.edu/server/api...

(node:2078877) Warning: Setting the NODE_TLS_REJECT_UNAUTHORIZED environment variable to '0' makes TLS connections and HTTPS requests insecure by disabling certificate verification.
(Use `node --trace-warnings ...` to show where the warning was created)
RESPONSE: 200 200 

Checking JSON returned for validity...
    "dspaceVersion" = DSpace 7.6.1
    "dspaceUI" = https://pedsdspace01.research.chop.edu
    "dspaceServer" = https://pedsdspace01.research.chop.edu/server
    "dspaceServer" property matches UI's "rest" config? true
    Does "/api" endpoint have HAL links ("_links" section)? true
Done in 2.11s.

You might have noticed a warning message about NODE_TLS_REJECT_UNAUTHORIZED. That’s because I set NODE_TLS_REJECT_UNAUTHORIZED to 0 in my ~/.bashrc. I was still encountering problems when I would just set NODE_EXTRA_CA_CERTS. These are the environmental variables I have set w/r/t Node in my ~/.bashrc:

export NODE_EXTRA_CA_CERTS="/etc/pki/tls/certs/pedsdspace01.research.chop.edu.crt"
export NODE_TLS_REJECT_UNAUTHORIZED=0
export NODE_OPTIONS="--max-old-space-size=4096"

I am trying to test my setup through yarn start:dev. My config details are below, but for now, it should be useful to know that ui.ssl: false and rest.ssl: true.

When I forward port 4000 to my machine and go to http://localhost:4000, I get DSpace’s 500 page, which, believe it or not, is a huge achievement:

Visiting the URL (which can only be accessed internally) via its URL https://pedsdspace01.research.chop.edu/ does not produce the same thing:

I am told “Invalid Host header.” Nothing really illuminating in the DevTools.

The backend works fine. I am totally able to access https://pedsdspace01.research.chop.edu/server/#/server/api and see The HAL Browser:

So the issue seems to be with connecting the frontend to the backend.

I have valid certifications issued by my IT department:

/etc/pki/tls/certs/pedsdspace01.research.chop.edu.crt
/etc/pki/tls/private/pedsdspace01.research.chop.edu.pem

Environment Setup

• Backend: DSpace REST API running on Tomcat with HTTP on port 8080 and AJP on port 8009.
• Frontend: DSpace Angular UI running on Node.js with HTTP on port 4000.
• Proxy: Apache HTTPD acting as a reverse proxy, handling SSL termination and forwarding requests to Tomcat and the Angular UI.

Configuration Files

1. config.dev.yml

ui:
  ssl: false
  host: localhost
  port: 4000
  nameSpace: /
  rateLimiter:
    windowMs: 60000 
    max: 500 
  useProxies: true

rest:
  ssl: true
  host: pedsdspace01.research.chop.edu
  port: 443
  nameSpace: /server

2. config.prod.yml

ui:
  ssl: false
  host: pedsdspace01.research.chop.edu
  port: 443
  nameSpace: /
  basePath: /
  rateLimiter:
    windowMs: 60000
    max: 500
  useProxies: true

rest:
  ssl: true
  host: pedsdspace01.research.chop.edu
  port: 443
  nameSpace: /server

3. local.cfg

dspace.ui.url = https://pedsdspace01.research.chop.edu
dspace.server.url = https://pedsdspace01.research.chop.edu/server

solr.server = http://localhost:8983/solr

db.url = jdbc:postgresql://localhost:5432/dspace
db.driver = org.postgresql.Driver
db.dialect = org.hibernate.dialect.PostgreSQL94Dialect
db.username = dspace
db.password = dspace
db.schema = public

4. server.xml

<Connector port="8080"  
                minSpareThreads="25"
                enableLookups="false"
                redirectPort="8443"
                connectionTimeout="20000"
                disableUploadTimeout="true"
                URIEncoding="UTF-8"/>

<Connector 
           protocol="AJP/1.3" 
           port="8009" 
           redirectPort="8443" 
           URIEncoding="UTF-8" 
           secretRequired="false" />

Here, I inserted secretRequired because I noticed the same type of error in my catalina.err file as in this StackOverflow post.

6. ssl.conf

Listen 443 https

<VirtualHost *:443>
    ServerName pedsdspace01.research.chop.edu

    # Add your desired log settings
    LogLevel trace6
    ErrorLog /var/log/httpd/pedsdspace01.research.chop.edu.error.log
    CustomLog /var/log/httpd/pedsdspace01.research.chop.edu.access.log combined
    # SSL logging for requests
    CustomLog logs/ssl_request_log "%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b"

    # Ensure the correct headers and host settings
    ProxyPreserveHost On
    RequestHeader set X-Forwarded-Proto https

    # SSL Configuration
    SSLEngine on
    SSLCertificateFile /etc/pki/tls/certs/pedsdspace01.research.chop.edu.crt
    SSLCertificateKeyFile /etc/pki/tls/private/pedsdspace01.research.chop.edu.pem

    # Proxy requests to the Tomcat server (backend)
    ProxyPass /server ajp://localhost:8009/server
    ProxyPassReverse /server ajp://localhost:8009/server

    # Proxy requests to the Angular UI server (frontend)
    ProxyPass / http://localhost:4000/
    ProxyPassReverse / http://localhost:4000/

</VirtualHost>

[DSpace Technical Support]

Hello,

Based on the information you shared, I believe your "ui" settings in your config.prod.yml may be incorrect.

When using Apache HTTPD as a proxy, typically your config.prod.yml will have UI settings that use "localhost" like this:

ui: 

  ssl: false 

  host: localhost 

  port: 400 

  nameSpace: /

The reason is that the User Interface itself should actually be running on http:/localhost:4000/ . That way Apache HTTPD can proxy requests to localhost:4000 whenever someone accesses your site via your public URL.  That is configured via these settings (which you already have in your Apache ssl.conf)

ProxyPass / http://localhost:4000/ 

ProxyPassReverse / http://localhost:4000/

NOTE specifically that Apache is trying to proxy requests to http://localhost:4000/.  That's correct, but it also means your "ui" settings in your config.prod.yml should run from http://localhost:4000/ in order to receive those requests via Apache.

Everything else in your configuration looks correct at a glance.  The "rest" settings of your config.prod.yml look correct, as they *require* using the full/public URL of the REST API.  The backend settings in local.cfg also look correct, and they seem to be proven to work because your "yarn rest:test" command is running successfully.

Hopefully it's just the "ui" settings that are wrong in your config.prod.yml and fixing them will solve the problem.  If you are still having issues, you may want to look closer at our Troubleshooting Guide to see if you can find more detailed error messages in the UI or in the backend logs, and also review the "Common Installation Issues" section of Install Guide to see if you are seeing any of the errors listed there.

Tim

[Arta Seyedian]

Forgive me, that was a typo. My UI is pointing towards my localhost. Not sure why it says otherwise in the text I copied and pasted.

ui: ssl: false host: localhost port: 443 nameSpace: / basePath: / rateLimiter: windowMs: 60000 max: 500 useProxies: true rest: ssl: true host: pedsdspace01.research.chop.edu port: 443 nameSpace: /server

Note that I get the same result whether I use config.prod.yml or config.dev.yml, and that I have been testing my setup with yarn start:dev.

I suspect that it might have something to do with using RHEL; some of the instructions around installing Apache HTTPD are not fully compatible with RHEL so I had to install it with sudo yum install httpd - I have also confirmed that the appropriate modules are installed and are being loaded by default.

sudo yum install httpd sudo systemctl enable httpd sudo systemctl start httpd

Others have suggested that I edit angular.json. I tried inserting this snippet:

"architect": { "serve": { "options": { "allowedHosts": ["pedsdspace01.research.chop.edu"] } } }

And was given the “cookies consent” form and nothing else. I rolled back this change because I felt as though there was something else I was doing wrong, and that I shouldn’t have to edit angular.json to get it to work.

Someone else has responded to me separately and suggested the following change within angular.json as well:

Search the section with "serve": { … "options": {
and add after

"port": 4000, "disableHostCheck": true

Though I have yet to experiment with it.

My colleague is also working on this issue and reports that he’s now getting the frontend to appear but is experiencing some CORS-related issue that is preventing the backend from working. I know there’s information about CORS-related issues in the troubleshooting guide but I haven’t had time to get back to this problem since making this post.

Anyway, thank you for your response and recommendation. I will continue tinkering with this problem and see where we end up. Worst-case-scenario, we will just try to use a Docker container for the frontend and see where that leads us.

All the best,
Arta

​

[Arta Seyedian]

I got it to work. I feel incredible. The sheer sense of accomplishment as I got the dev frontend to work.

A caveat is the fact that I have overridden a lot of the authentication checks. So it is working despite the fact that Apache doesn’t seem to like the SSL cert/key:

[Mon May 20 13:37:17.864496 2024] [ssl:info] [pid 3509986:tid 3510173] (70014)End of file found: [client 10.14.54.192:54627] AH01992: SSL library error 1 reading data [Mon May 20 13:37:17.864532 2024] [ssl:info] [pid 3509986:tid 3510173] SSL Library Error: error:0A000126:SSL routines::unexpected eof while reading [Mon May 20 13:37:17.864591 2024] [ssl:debug] [pid 3509986:tid 3510173] ssl_engine_io.c(1477): (70014)End of file found: [client 10.14.54.192:54627] AH02007: SSL handshake interrupted by system [Hint: Stop button pressed in browser?!]

Nonetheless, this is a step in the right direction.

The changes I had made that tipped it into the right direction for yarn start:dev are:

• SSLCertificateFile was originally pedsdspace01.research.chop.edu.crt, but I had switched it to the cert key pedsdspace01.research.chop.edu.pem because I was getting that EOF error above and I had noticed that it contained both the certificate and the private key. I changed it back.

SSLCertificateFile /etc/pki/tls/certs/pedsdspace01.research.chop.edu.crt

• Fixed the typo here, set it to SSL:

dspace.ui.url = https://pedsdspace01.research.chop.edu

and removing the following item from server.xml (not sure why I thought it was a good idea to put it in there in the first place):

# Redirect HTTP to HTTPS <VirtualHost *:80> ServerName pedsdspace01.research.chop.edu RewriteEngine On # RewriteCond %{HTTPS} off # RewriteRule ^(.*)$ https://%{HTTP_HOST}%{REQUEST_URI} [L,R=301] </VirtualHost>

but more broadly:

• angular.json having its authentication options disabled, specifically allowedHosts and disableHostCheck:

... "serve": { "builder": "@angular-builders/custom-webpack:dev-server", "options": { "allowedHosts": [ "pedsdspace01.research.chop.edu" ], "browserTarget": "dspace-angular:build", "port": 4000, "disableHostCheck": true }, ...

• Standard config.dev.yml settings for frontend:

ui: ssl: false host: localhost port: 4000 nameSpace: / rateLimiter: windowMs: 60000 max: 500 useProxies: true rest: ssl: true host: pedsdspace01.research.chop.edu port: 443 nameSpace: /server

• Standard production local.cfg:

dspace.server.url = https://pedsdspace01.research.chop.edu/server # dspace.ui.url=http://10.30.11.25:4000 dspace.ui.url = https://pedsdspace01.research.chop.edu

• Standard server.xml. Not going to show it because it’s the same as what you would get if you follow the directions, but has an extra secretRequired = "false" argument for the AJP connector.

Also not entirely sure how much this post had to do with it, but I called ./bin/dspace database migrate force even though the database migration was already all successful. I arrived there based on this thread where OP had the same inscrutable Java error as I did in dspace.log:

java.lang.NullPointerException: Cannot invoke "Object.getClass()" because "modelObject" is null

This might be a totally fragile implementation but I am willing to take the W.

Thank you you all,
Arta

​

[Alex Fraser (Staff)]

Hi Arta,

 

I found the UI configuration a bit… finicky. As your configuration also implies, we have the Apache web server, UI and Tomcat on the same (RHEL 8) server. I have included some of the relevant (templated) server configuration we are using at the bottom of my reply in case it is of use. At the moment we only have a non-production environment but the concept is that the server only runs the UI in production mode, with (custom theme) development being done by running it locally in dev mode, which I note required http://localhost:4000 to be added to rest.cors.allowed-origins in local.cfg in order for the backend to be available to client-side UI code. The UI, incorporating our custom theme, is then built and packaged to be subsequently deployed to the server.

 

I think the Apache web server “unexpected eof” error is caused by the client (the UI?) closing the connection to the web server after receiving the certificate, eg after certificate verification failed. With a certificate signed by a well-known public CA, this can generally be resolved by ensuring intermediate certificates are also sent by the server (which is necessary for maximum client compatibility). For Apache web server >= 2.4.8, the recommended way to do this is for the file referenced by SSLCertificateFile to contain the end-entity certificate followed by the intermediate certificates – possibly what you achieved by switching from the .crt to the .pem file.

 

I can’t immediately see why the HTTP to HTTPS redirect would be a problem, nor why you need the changes you described to angular.json (the only change we made is to add a theme, per the documentation); I think you may be able to undo these changes without ill-effect.

 

Kind regards,

Alex

 

Apache web server, under /etc/httpd/conf.d:

 

ServerTokens Prod

TraceEnable off

 

<VirtualHost _default_:80>

         ServerName {{ server_name }}

         RedirectPermanent / https://{{ server_name }}/

</VirtualHost>

 

<VirtualHost _default_:443>

         ServerName {{ server_name }}

 

         Protocols h2 http/1.1

 

         ErrorLog logs/ssl_error_log

         TransferLog logs/ssl_access_log

         LogLevel warn

 

         SSLEngine on

         SSLHonorCipherOrder on

         SSLCipherSuite PROFILE=SYSTEM

 

         SSLCertificateFile {{ httpd_certificate_file }}

         SSLCertificateKeyFile {{ httpd_certificate_key_file }}

 

         BrowserMatch "MSIE [2-5]" \

                 nokeepalive ssl-unclean-shutdown \

                 downgrade-1.0 force-response-1.0

 

         Header set Strict-Transport-Security max-age=31536000

         Header unset X-Powered-By

 

         ProxyPreserveHost On

         RequestHeader set X-Forwarded-Proto https

 

         ProxyPass        /server ajp://127.0.0.1:{{ dspace_backend_port }}/server retry=0

         ProxyPassReverse /server ajp://127.0.0.1:{{ dspace_backend_port }}/server

 

         ProxyPass        / http://127.0.0.1:{{ dspace_ui_port }}/ retry=0

         ProxyPassReverse / http://127.0.0.1:{{ dspace_ui_port }}/

</VirtualHost>

 

dspace-ui-deploy/config/config.prod.yml:

 

ui:

ssl: false

  host: 127.0.0.1

  port: {{ dspace_ui_port }}

  nameSpace: /

[...]

rest:

  ssl: true

  host: {{ server_name }}

  port: 443

  nameSpace: /server

 

Apache Tomcat server.xml:

 

[...]

    <Connector protocol="AJP/1.3"

               address="127.0.0.1"

               port="{{ dspace_backend_port }}"

               redirectPort="8443"

               maxParameterCount="1000"

               secretRequired="false"

               />

[...]

--
All messages to this mailing list should adhere to the Code of Conduct: https://www.lyrasis.org/about/Pages/Code-of-Conduct.aspx
---
You received this message because you are subscribed to the Google Groups "DSpace Technical Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dspace-tech...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dspace-tech/d1e7d0d3-1859-4422-be42-59924b83a6c8n%40googlegroups.com.

[Plate, Michael]

Hi,

did you try to figure out the correct SSL config by using openssl, not the browser ? While browsers tend to be lax, Java is absolutely picky on that.
A good example is the DSpace Demo site:

openssl s_client -connect demo.dspace.org:443

this is configured OK. If you get an error because of missing intermediate certs or wrong order of the certs this can be a reason for problems.

CU

Michael
________________________________________
Von: dspac...@googlegroups.com <dspac...@googlegroups.com> im Auftrag von Arta Seyedian <arta.s...@gmail.com>
Gesendet: Montag, 20. Mai 2024 20:28
An: DSpace Technical Support
Betreff: [Extern] [dspace-tech] Re: DSpace 7.6.1 Configuration Issues with Apache HTTPD and SSL