Intermittent "Error fetching items" on 7.2

[Night Librarian]

Greetings,

On DSPace 7.2 in Ubuntu 20.04, backend and frontend work fine, but now and then I stop being able to browse.  When I try to browse by author, title, subject, I have an "Error fetching items" message.  A few minutes later, everything starts working again, until the next hiccup. When I look at the Dev Tools, the console shows a few lines centered around "Failed to load resource: net::ERR_CONNECTION_REFUSED" (see screenshot). 

pm2 log shows a few blocks that look like this:

ERROR Error: undefined doesn't contain the link browses
    at MapSubscriber.project (/dspace-angular/dist/server/main.js:3:6347448)
    at MapSubscriber._next (/dspace-angular/dist/server/main.js:3:5287042)
    at MapSubscriber.next (/dspace-angular/dist/server/main.js:3:1110558)
    at MapSubscriber._next (/dspace-angular/dist/server/main.js:3:5287148)
    at MapSubscriber.next (/dspace-angular/dist/server/main.js:3:1110558)
    at TakeSubscriber._next (/dspace-angular/dist/server/main.js:3:1863095)
    at TakeSubscriber.next (/dspace-angular/dist/server/main.js:3:1110558)
    at FilterSubscriber._next (/dspace-angular/dist/server/main.js:3:5775291)
    at FilterSubscriber.next (/dspace-angular/dist/server/main.js:3:1110558)
    at MapSubscriber._next (/dspace-angular/dist/server/main.js:3:5287148)
ERROR Error: undefined doesn't contain the link sites
    at MapSubscriber.project (/dspace-angular/dist/server/main.js:3:6347448)
    at MapSubscriber._next (/dspace-angular/dist/server/main.js:3:5287042)
    at MapSubscriber.next (/dspace-angular/dist/server/main.js:3:1110558)
    at MapSubscriber._next (/dspace-angular/dist/server/main.js:3:5287148)
    at MapSubscriber.next (/dspace-angular/dist/server/main.js:3:1110558)
    at TakeSubscriber._next (/dspace-angular/dist/server/main.js:3:1863095)
    at TakeSubscriber.next (/dspace-angular/dist/server/main.js:3:1110558)
    at FilterSubscriber._next (/dspace-angular/dist/server/main.js:3:5775291)
    at FilterSubscriber.next (/dspace-angular/dist/server/main.js:3:1110558)
    at MapSubscriber._next (/dspace-angular/dist/server/main.js:3:5287148)
ERROR TypeError: authorizations.filter is not a function
    at SwitchMapSubscriber.project (/dspace-angular/dist/server/main.js:3:242131)
    at SwitchMapSubscriber._next (/dspace-angular/dist/server/main.js:3:4054280)
    at SwitchMapSubscriber.next (/dspace-angular/dist/server/main.js:3:1110558)
    at CatchSubscriber.notifyNext (/dspace-angular/dist/server/main.js:3:8026230)
    at SimpleInnerSubscriber._next (/dspace-angular/dist/server/main.js:3:8025925)
    at SimpleInnerSubscriber.next (/dspace-angular/dist/server/main.js:3:1110558)
    at Observable_Observable._subscribe (/dspace-angular/dist/server/main.js:3:5467843)
    at Observable_Observable._trySubscribe (/dspace-angular/dist/server/main.js:3:1764443)
    at Observable_Observable.subscribe (/dspace-angular/dist/server/main.js:3:1764225)
    at innerSubscribe (/dspace-angular/dist/server/main.js:3:8026540)

Don't see anything in other logs.  Is there something else I should check?

Many thanks in advance!

[Night Librarian]

To add to this, it looks like the problem occurs when I access the frontend URI from other computers on the network.  First, when I click on any "Browse by" or on list of communities, or on any item, I see "Error fetching item" like this:

When I press F5 or click reload in the browser, the page flashes/blinks, then loads the content like this:

When I am accessing the site on Chrome or Firefox from within the virtual machine itself, the content loads right away, without the need to reload.

[Night Librarian]

Turns out that I have a bigger problem than just having to reload pages.  When I restart Ubuntu, Tomcat, HAL, SOLR work fine in the browser, but trying to access DSpace Frontend gives me error 503 "Service Unavailable".  

/dspace-angular/config/config.prod.yml

# Angular Universal server settings
# NOTE: these must be 'synced' with the 'dspace.ui.url' setting in your backend's local.cfg.
ui:
  ssl: true
  host: localhost
  port: 443
  # NOTE: Space is capitalized because 'namespace' is a reserved string in TypeScript
  nameSpace: /
<...>

# The REST API server settings
# NOTE: these must be 'synced' with the 'dspace.server.url' setting in your backend's local.cfg.
rest:
  ssl: true
  host: mycompany.ca
  port: 443
  # NOTE: Space is capitalized because 'namespace' is a reserved string in TypeScript
  nameSpace: /server
--------------------------------------------------------------------------------------
/dspace-angular/dspace-ui.json
{
    "apps": [
        {
           "name": "dspace-ui",
           "cwd": "/dspace-angular",
           "script": "dist/server/main.js",
           "env": {
            "NODE_ENV": "production",
        "DSPACE_REST_SSL": "true",
        "DSPACE_REST_HOST": "mycompany.ca",
        "DSPACE_REST_PORT": "443",
        "DSPACE_REST_NAMESPACE": "/server"
           }
        }
    ]
}
---------------------------------------------------------------------------------------
/dspace/config/local.cfg

##########################
# SERVER CONFIGURATION   #
##########################

# DSpace installation directory.
dspace.dir=/dspace

# URL of DSpace backend ('server' webapp). Include port number etc.
dspace.server.url = https://mycompany.ca/server

# URL of DSpace frontend (Angular UI). Include port number etc.
dspace.ui.url = https://mycompany.ca

# Name of the site
dspace.name = My repository

<...>

# Solr server/webapp.
solr.server = http://localhost:8983/solr
---------------------------------------------------------------------------------------
/etc/apache2/sites-available

<VirtualHost *:443>

#ServerName www.example.com

ServerAdmin webmaster@localhost
DocumentRoot /var/www/html

<...>

ErrorLog ${APACHE_LOG_DIR}/error.log
CustomLog ${APACHE_LOG_DIR}/access.log combined

<...>

SSLEngine On

SSLCertificateFile  /etc/apache2/ssl/certs/wildcard_mycompany_ca.crt
SSLCertificateKeyFile  /etc/apache2/ssl/private/wildcard_mycompany_ca.key
SSLCertificateChainFile /etc/apache2/ssl/certs/DigiCertCA.crt

# Proxy all HTTPS requests to "/server" from Apache to Tomcat via AJP connector
ProxyPass /server ajp://localhost:8009/server
ProxyPassReverse /server ajp://localhost:8009/server
 
<...>

# Proxy all HTTPS requests from Apache to PM2 on port 4000
# NOTE that this proxy URL must match the "ui" settings in your config.prod.yml
ProxyPass / http://localhost:4000/
ProxyPassReverse / http://localhost:4000/


</VirtualHost>
---------------------------------------------------------------------------------------
/etc/tomcat9/server.xml

<!-- Define a non-SSL/TLS HTTP/1.1 Connector on port 8080 -->
    <Connector port="8080" protocol="HTTP/1.1"
           minSpareThreads="25"
           enableLookups="false"
               connectionTimeout="20000"
               redirectPort="8443"
               disableUploadTimeout="true"
           URIEncoding="UTF-8"/>

<!-- Define an AJP 1.3 Connector on port 8009 -->
    <Connector protocol="AJP/1.3"
               address="::"
               port="8009"
               redirectPort="8443"
           tomcatAuthentication="false"
               enableLookups="false"
               secretRequired="false"
           URIEncoding="UTF-8" /> 

---------------------------------------------------------------------------------------

Dev tools say: "GET  https://mycompany.ca/     [HTTP/1.1 503 Service Unavailable 41ms]".  Network tab has this:

websocket?url=https%3A%2F%2Fmycompany.ca%2F&nocache=1662874542940:


Request URL: wss://me.kis.v2.scr.kaspersky-labs.com/7D8B79A2-8974-4D7B-A76A-F4F29624C06BgRTf2XFqPinpC4nCnUthd68wxfoCLgSlMs5di1UJi-DmJ1Mp5c5zD2Ik7JrtlYhTxPivLp8WuTF9axm3UQhQxw/websocket?url=https%3A%2F%2Fmycompany.ca%2F&nocache=1662874542940
Request Method: GET
Status Code: 101
Cache-Control: no-store, no-cache, must-revalidate, max-age=0
Connection: Upgrade
Content-Length: 0
Content-Type: text/html; charset=utf-8
Expires: Mon, 04 Dec 1999 21:29:02 GMT
Pragma: no-cache
Sec-WebSocket-Accept: H3VEl0aWr9UdeTXDpVhdALMVJ44=
Upgrade: websocket
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9,es;q=0.8,ru;q=0.7,fr;q=0.6
Cache-Control: no-cache
Connection: Upgrade
Host: me.kis.v2.scr.kaspersky-labs.com
Origin: https://mycompany.ca
Pragma: no-cache
Sec-WebSocket-Extensions: permessage-deflate; client_max_window_bits
Sec-WebSocket-Key: ifpMhcbVqO+kSgKTkpLtGg==
Sec-WebSocket-Version: 13
Upgrade: websocket
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36 Edg/105.0.1343.27

mycompany.ca:

Request URL: https://mycompany.ca/
Request Method: GET
Status Code: 503 Service Unavailable
Remote Address: [IP address]:443
Referrer Policy: strict-origin-when-cross-origin
Connection: close
Content-Length: 601
Content-Type: text/html; charset=iso-8859-1
Date: Sun, 11 Sep 2022 05:35:43 GMT
Server: Apache/2.4.41 (Ubuntu)
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9,es;q=0.8,ru;q=0.7,fr;q=0.6
Cache-Control: max-age=0
Connection: keep-alive
Cookie: _<...>
Host: mycompany.ca
sec-ch-ua: "Microsoft Edge";v="105", " Not;A Brand";v="99", "Chromium";v="105"
sec-ch-ua-mobile: ?0
sec-ch-ua-platform: "Windows"
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: none
Sec-Fetch-User: ?1
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36 Edg/105.0.1343.27

main.js?attr=<...> :

Request URL: https://me.kis.v2.scr.kaspersky-labs.com/FD126C42-EBFA-4E12-B309-BB3FDD723AC1/main.js?attr=gRTf2XFqPinpC4nCnUthd1fRNl647BHuclZhtDL4ZyXEGIYwu_bR72lP9tAT4fKr
Request Method: GET
Status Code: 200
Remote Address: 185.85.13.154:443
Referrer Policy: strict-origin-when-cross-origin
cache-control: no-store, no-cache, must-revalidate, max-age=0
content-length: 112656
content-type: application/x-javascript;charset=UTF-8
expires: Mon, 04 Dec 1999 21:29:02 GMT
pragma: no-cache
:authority: me.kis.v2.scr.kaspersky-labs.com
:method: GET
:path: /FD126C42-EBFA-4E12-B309-BB3FDD723AC1/main.js?attr=gRTf2XFqPinpC4nCnUthd1fRNl647BHuclZhtDL4ZyXEGIYwu_bR72lP9tAT4fKr
:scheme: https
accept: */*
accept-encoding: gzip, deflate, br
accept-language: en-US,en;q=0.9,es;q=0.8,ru;q=0.7,fr;q=0.6
referer: https://mycompany.ca/
sec-ch-ua: "Microsoft Edge";v="105", " Not;A Brand";v="99", "Chromium";v="105"
sec-ch-ua-mobile: ?0
sec-ch-ua-platform: "Windows"
sec-fetch-dest: script
sec-fetch-mode: no-cors
sec-fetch-site: cross-site
user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36 Edg/105.0.1343.27

---------------------------------------------------------------------------------------

Finally, pm2 logs complain about service key/certificate not found at ./config/ssl/key.pem, but I have a valid certificate from Digicert that has .crt and .key files in apache2 directory and the browser shows my site as secure.

Service key not found at ./config/ssl/key.pem
Certificate not found at ./config/ssl/key.pem
Disabling certificate validation and proceeding with a self-signed certificate. If this is a production server, it is recommended that you configure a valid certificate instead.
Error: listen EADDRINUSE: address already in use 127.0.0.1:443
    at Server.setupListenHandle [as _listen2] (node:net:1432:16)
    at listenInCluster (node:net:1480:12)
    at GetAddrInfoReqWrap.doListen [as callback] (node:net:1629:7)
    at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:111:8) {
  code: 'EADDRINUSE',
  errno: -98,
  syscall: 'listen',
  address: '127.0.0.1',
  port: 443
}

I am sorry for the lengthy post.  I read the installation and troubleshooting instructions and various posts in this group, and then tried to get as much info as I could. All suggestions will be very much appreciated!

[Tim Donohue]

Hi,

One thing jumps out at me as possibly being a problem...these settings:

ui:
  ssl: true
  host: localhost
  port: 443

Those appear to say your UI is running at https://localhost/

However, later on in your proxy you have...

# Proxy all HTTPS requests from Apache to PM2 on port 4000
# NOTE that this proxy URL must match the "ui" settings in your config.prod.yml
ProxyPass / http://localhost:4000/
ProxyPassReverse / http://localhost:4000/

Notice how that is proxying to a DIFFERENT URL!?

I suspect you need your settings in config.*.yml to be:

ui:

  ssl: false

  host: localhost

  port: 4000

That will ensure the UI initially starts up on port 4000 (i.e. http://localhost:4000).  That way your proxy can automatically redirect users to http://localhost:4000 whenever they access your site from the public URL (configured in your proxy settings)   (If you reread the installation instructions you'll notice we have notes in step 5 (Configuration) that say that the above configuration is "a common setup for when you want to use Apache or Nginx to handle HTTPS and proxy requests to Node..."

Tim

[Night Librarian]

Thank you, Tim!

You were right.  I read the instructions more than once, but confused myself with the proxy part... :)   Updating config.prod.yml with your suggestion is working! 

ui:
  ssl: false
  host: localhost
  port: 4000

As an added bonus, my problem with the page continuously spinning while using "browse by date" is gone now and it works fine.

I have one problem remaining - starting the UI.  Every time I reboot Ubuntu, I can't access the UI in the browser.  I manually run one of these two commands and the site comes up, but if I stop one of them, I get errors after trying any of the two commands again.

sudo node ./dist/server/main.js

or

sudo pm2 start dspace-ui.json

So, I reboot Ubuntu and can repeat the process.  If I understand correctly, I can use pm2 to start automatically, so I tried to set this up using this page as a guide (How to Enable PM2 to Auto Start Node.js App at System Boot (tecmint.com)):

pm2 startup

systemctl status pm2-root.service
pm2 start dspace-ui.json
sudo pm2 saves

That way, afer reboot the pm2 starts and the UI loads, but I have an "error loading top communities" message where the communities should be displayed.  The say goes for other pages.

Once I "pm2 unstartup", I'm back to having to launch pm2 manually, but at least the site now works great.  Just wondering what I could do to automate the startup.