Blank page 500 server error

190 views
Skip to first unread message

BrzI Channel

unread,
Nov 13, 2023, 2:54:38 AM11/13/23
to AtoM Users
Hi folks,

we had a major power outage. After the recovery process the VM server is back up and running but the Application just produces a blank page.

nginx error log has a lot of these:
2023/11/12 22:36:16 [error] 1530#1530: *12074 FastCGI sent in stderr: "PHP message: The "i18n" object does not exist in the current context" while reading response header from upstream, client: xx.xx.xxx.xx, server: _, request: "GET /index.php/informationobject/browse?creators=433&sortDir=desc&collection=151641&sort=identifier&subjects=152323&view=table&sf_culture=fr&topLod=0&media=print HTTP/1.1", upstream: "fastcgi://unix:/run/php7.2-fpm.atom.sock:", host: "atom.xxxx.xxxx.cxx"

I also tried enabling debug mode. After editing index.php I still get " You are not allowed to access this file. Check qubit_dev.php for more information. " when accessing from a browser running on the same server.

I believe I am running ver. 2.6

I have restarted PHP-FPM, Nginx, memcached and job scheduler

Any tips on what else I should be looking at to restore the functionality ?

Thanks

Dan Gillean

unread,
Nov 13, 2023, 8:40:18 AM11/13/23
to ica-ato...@googlegroups.com
Hi there, 

You can check your version from the command line with: 
AtoM full release versions include two numbers - the application version, and the database schema version. We try to only include changes that require updates to the database schema in major releases, and we include the expected schema version number for each release in the release notes. This can be helpful when upgrading and/or loading a backup after an outage or similar - if the schema version of your installation does not match the expected version, then it's likely that a step was skipped during the upgrade / recovery process. Generally, this is either dropping and recreating the database BEFORE loading your backup, or forgetting to run the upgrade task AFTER loading the sqldump. 

For 2.6.x releases, the expected schema version should be 184. 

I would recommend that you try running some of AtoM's common maintenance tasks, as these may help resolve issues caused by a long-running process (such as an import, move, deletion, etc) timing out or being interrupted when the power went down. For example, try running the nested set build, the slug regeneration, and repopulating the search index. Restarting PHP-FPM and clearing the application cache are also often helpful. There are details and links to all these tasks on our Troubleshooting page: 
If that doesn't work, then I would suggest that you try to gather further information. First, we have a SQL query in the Troubleshooting docs that can help look for common forms of data corruption in information objects, as well as recommend some ways to fix them. In general, we always recommend making a backup before using SQL - then see: 
Additionally, if repopulating the search index does not work as expected, you can check its status with: 
Let us know how it goes with the above! 

Finally: for Debug mode, you will likely need to edit a configuration file and then restart PHP-FPM. See: 
Cheers, 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056
@accesstomemory
he / him


--
You received this message because you are subscribed to the Google Groups "AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/a9b148e7-2b77-4fd3-acba-12467aa7f1bdn%40googlegroups.com.

BrzI Channel

unread,
Nov 13, 2023, 3:47:10 PM11/13/23
to AtoM Users
Running 2.6.4 v184.
I did edit the qubit_dev.php file and then restarted php-fpm but still got error message in browser even when accessing site in local browser.

I will go through the steps one more time and report back.

Thanks Dan

BrzI Channel

unread,
Nov 13, 2023, 4:25:44 PM11/13/23
to AtoM Users
Here is my htop. Looks OK to me with the possible exception that elasticsearch is using 47% of memory. Don't know if that matters.

  1  [|||                                                                                                                                                    1.3%]   Tasks: 107, 204 thr; 1 running
  2  [||                                                                                                                                                     0.7%]   Load average: 0.01 0.13 0.14
  Mem[|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||     2.99G/4.84G]   Uptime: 12:57:48
  Swp[|                                                                                                                                                3.50M/976M]

  PID USER      PRI  NI  VIRT   RES   SHR S CPU% MEM%   TIME+  Command
 8978 admin    20   0 35676  6844  3816 R  0.7  0.1  0:00.97 htop
 8706 elasticse  20   0 5043M 2343M 45948 S  0.0 47.3  0:16.94 /usr/bin/java -Xms2g -Xmx2g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -server -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Djdk.io.permissionsUseCanonicalPath=true -D
 1892 mysql      20   0 1741M  420M 37272 S  0.0  8.5  1:19.98 /usr/sbin/mysqld
 8584 www-data   20   0  437M 15984 10700 S  0.0  0.3  0:00.04 php-fpm: pool atom
 1521 www-data   20   0  145M 12504  7584 S  0.0  0.2  2:36.26 nginx: worker process
 1954 lightdm    20   0  616M 32060 25896 S  0.0  0.6  0:51.02 /usr/sbin/slick-greeter
 1437 root       20   0  435M 23516 18676 S  0.0  0.5  0:02.48 php-fpm: master process (/etc/php/7.2/fpm/php-fpm.conf)
 1761 mysql      20   0 1741M  420M 37272 S  0.0  8.5  2:20.24 /usr/sbin/mysqld
 1420 memcache   20   0  414M  2876  2364 S  0.0  0.1  0:06.98 /usr/bin/memcached -m 64 -p 11211 -u memcache -l 127.0.0.1 -P /var/run/memcached/memcached.pid
 2046 mysql      20   0 1741M  420M 37272 S  0.0  8.5  0:00.87 /usr/sbin/mysqld
 8758 elasticse  20   0 5043M 2343M 45948 S  0.0 47.3  0:02.27 /usr/bin/java -Xms2g -Xmx2g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -server -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Djdk.io.permissionsUseCanonicalPath=true -D
 1836 mysql      20   0 1741M  420M 37272 S  0.0  8.5  0:01.15 /usr/sbin/mysqld
 1838 mysql      20   0 1741M  420M 37272 S  0.0  8.5  0:01.09 /usr/sbin/mysqld
  742 root        0 -20  151M  7564  6320 S  0.0  0.1  0:32.66 /usr/bin/vmtoolsd
 8760 elasticse  20   0 5043M 2343M 45948 S  0.0 47.3  0:00.26 /usr/bin/java -Xms2g -Xmx2g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -server -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Djdk.io.permissionsUseCanonicalPath=true -D
 8757 elasticse  20   0 5043M 2343M 45948 S  0.0 47.3  0:06.56 /usr/bin/java -Xms2g -Xmx2g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -server -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Djdk.io.permissionsUseCanonicalPath=true -D
 8602 www-data   20   0  437M 15992 10704 S  0.0  0.3  0:00.02 php-fpm: pool atom
 8767 elasticse  20   0 5043M 2343M 45948 S  0.0 47.3  0:00.29 /usr/bin/java -Xms2g -Xmx2g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -server -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Djdk.io.permissionsUseCanonicalPath=true -D
 2182 www-data   20   0  270M 32956 16784 S  0.0  0.6  0:10.45 /usr/bin/php7.2 -d memory_limit=-1 -d error_reporting=E_ALL symfony jobs:worker
 1891 mysql      20   0 1741M  420M 37272 S  0.0  8.5  0:07.47 /usr/sbin/mysqld
 1354 gearman    20   0  487M  6572  4608 S  0.0  0.1  0:01.76 /usr/sbin/gearmand --pid-file=/run/gearman/gearmand.pid --listen=localhost --daemon --log-file=/var/log/gearman-job-server/gearmand.log
 1889 mysql      20   0 1741M  420M 37272 S  0.0  8.5  0:07.07 /usr/sbin/mysqld
 8813 elasticse  20   0 5043M 2343M 45948 S  0.0 47.3  0:00.06 /usr/bin/java -Xms2g -Xmx2g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -server -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Djdk.io.permissionsUseCanonicalPath=true -D
 8779 elasticse  20   0 5043M 2343M 45948 S  0.0 47.3  0:00.05 /usr/bin/java -Xms2g -Xmx2g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -server -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Djdk.io.permissionsUseCanonicalPath=true -D
 2069 mysql      20   0 1741M  420M 37272 S  0.0  8.5  0:11.97 /usr/sbin/mysqld
 8965 admin    20   0  107M  5784  4772 S  0.0  0.1  0:00.02 sshd: admin@pts/1
 2051 mysql      20   0 1741M  420M 37272 S  0.0  8.5  0:02.50 /usr/sbin/mysqld
 1704 root       20   0  624M 46828 32572 S  0.0  0.9  0:18.41 /usr/lib/xorg/Xorg -core :0 -seat seat0 -auth /var/run/lightdm/root/:0 -nolisten tcp vt7 -novtswitch
 1334 gearman    20   0  487M  6572  4608 S  0.0  0.1  0:07.13 /usr/sbin/gearmand --pid-file=/run/gearman/gearmand.pid --listen=localhost --daemon --log-file=/var/log/gearman-job-server/gearmand.log
 2047 mysql      20   0 1741M  420M 37272 S  0.0  8.5  0:00.89 /usr/sbin/mysqld
 8766 elasticse  20   0 5043M 2343M 45948 S  0.0 47.3  0:00.06 /usr/bin/java -Xms2g -Xmx2g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -server -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Djdk.io.permissionsUseCanonicalPath=true -D
 2366 rtkit      20   0  179M  2784  2516 S  0.0  0.1  0:00.21 /usr/lib/rtkit/rtkit-daemon
 8771 elasticse  20   0 5043M 2343M 45948 S  0.0 47.3  0:00.03 /usr/bin/java -Xms2g -Xmx2g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -server -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Djdk.io.permissionsUseCanonicalPath=true -D
 8795 elasticse  20   0 5043M 2343M 45948 S  0.0 47.3  0:00.09 /usr/bin/java -Xms2g -Xmx2g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -server -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Djdk.io.permissionsUseCanonicalPath=true -D
 7467 www-data   20   0  437M 15152  9864 S  0.0  0.3  0:00.06 php-fpm: pool atom
 8595 www-data   20   0  437M 15988 10700 S  0.0  0.3  0:00.02 php-fpm: pool atom
 1499 memcache   20   0  414M  2876  2364 S  0.0  0.1  0:03.43 /usr/bin/memcached -m 64 -p 11211 -u memcache -l 127.0.0.1 -P /var/run/memcached/memcached.pid
 7503 www-data   20   0  437M 15136  9852 S  0.0  0.3  0:00.04 php-fpm: pool atom
 8979 elasticse  20   0 5043M 2343M 45948 S  0.0 47.3  0:00.03 /usr/bin/java -Xms2g -Xmx2g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -server -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Djdk.io.permissionsUseCanonicalPath=true -D
 1839 mysql      20   0 1741M  420M 37272 S  0.0  8.5  0:01.11 /usr/sbin/mysqld
 8752 elasticse  20   0 5043M 2343M 45948 S  0.0 47.3  0:00.10 /usr/bin/java -Xms2g -Xmx2g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -server -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Djdk.io.permissionsUseCanonicalPath=true -D
 8803 elasticse  20   0 5043M 2343M 45948 S  0.0 47.3  0:00.06 /usr/bin/java -Xms2g -Xmx2g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -server -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Djdk.io.permissionsUseCanonicalPath=true -D
 1084 root       20   0  469M 14488 11924 S  0.0  0.3  0:00.60 /usr/sbin/NetworkManager --no-daemon
 1890 mysql      20   0 1741M  420M 37272 S  0.0  8.5  0:06.82 /usr/sbin/mysqld
 8804 elasticse  20   0 5043M 2343M 45948 S  0.0 47.3  0:00.09 /usr/bin/java -Xms2g -Xmx2g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -server -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Djdk.io.permissionsUseCanonicalPath=true -D
 1372 roo
Message has been deleted

BrzI Channel

unread,
Nov 14, 2023, 6:13:33 PM11/14/23
to AtoM Users
Here is what I did (in sequence). No errors anywhere.

Rebuild nested set
Regenerate slugs
Repopulate search index
Restart PHP-FPM
Reload nginx
Restart memcached
Clear cache
Restart job scheduler
Clear jobs
Reboot
SQL-search index status (required sudo)
SQL-no missing slugs or object IDs


nginx logs now show a lot of this :
2023/11/14 14:49:42 [error] 1589#1589: *4623 FastCGI sent in stderr: "PHP message: PHP Parse error:  syntax error, unexpected 'false' (T_STRING), expecting ',' or ')' in /usr/share/nginx/atom/index.php on line 5" while reading response header from upstream, client: 111.225.149.55, server: _, request: "GET /index.php/informationobject/browse?levels=227&limit=25&names=15869&page=3&sort=lastUpdated&topLod=0 HTTP/1.1", upstream: "fastcgi://unix:/run/php7.2-fpm.atom.sock:", host: "atom.xxx.xxx.xxx"

Added IP address in atom.conf for debugging - denied access both on server and specified addtional IP address
Edited index.php to 'true' for qubit
Going to https://%siteURL%/qubit_dev.php/informationobject/browse - denied access both on server and specified addtional IP address

Dan Gillean

unread,
Nov 15, 2023, 9:55:46 AM11/15/23
to ica-ato...@googlegroups.com
Hi again, 

One of our developers reminded me that the original error message about the i18n object is one we have seen in the forum before. You can try searching that in the forum yourself, but I will try to summarize some of the suggestions based on previous experiences. 

Generally, we have found that this tends to be one of the following: 
  1. Incorrect permissions
  2. Missing or empty configuration files 
  3. Typos or other errors in the PHP pool
  4. Insufficient memory
For 1: try resetting the permissions with: 
  • sudo chown -R www-data:www-data /usr/share/nginx/atom
For 2: Here is a thread in the forum about this: 
(Note that he lists a different error message in his post, but scroll up in the thread and you will see it's all related to the same originating issue)

For 3: For a 2.6 installation, be sure to review /etc/php/7.2/fpm/pool.d/atom.conf against the sample pool configuration in the documentation. You can also make sure that PHP-FPM is running as expected with:
  • sudo php7.2-fpm --test
Additionally, you can try resetting the fail counter and then restarting the job scheduler's worker, to ensure it is not holding onto the available memory: 
  • sudo systemctl reset-failed atom-worker
  • sudo systemctl restart atom-worker
  • sudo systemctl status atom-worker
For 4: If possible, try increasing the memory of your installation. In our Technical Requirements, we recommend the following for a small-to-medium sized production instance:

Processor: 2 vCPUs @ 2.3GHz
Memory: 7GB
Disk space (processing): 50GB at a minimum for AtoM’s core stack plus more storage would be required for supporting any substantial number of digital objects.


You might also try increasing some of the PHP execution limits. See: 

Finally, regarding the other issue you mention: our developer does not recall previous seeing syntax error messages in index.php. Is it possible that you caused it inadvertently when doing this?

Added IP address in atom.conf for debugging - denied access both on server and specified addtional IP address
Edited index.php to 'true' for qubit

I would double-check the file again, and restart PHP-FPM again after any changes. 

Hope some of this helps! 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056
@accesstomemory
he / him


Message has been deleted

BrzI Channel

unread,
Nov 15, 2023, 1:35:28 PM11/15/23
to AtoM Users
Went through all of those:

Only this:

My atom.conf only has these tweaks applied:
php_admin_value[max_execution_time] = 300
php_admin_value[post_max_size] = 11G
php_admin_value[upload_max_filesize] = 10G
Otherwise it matches the original

BrzI Channel

unread,
Nov 15, 2023, 1:39:34 PM11/15/23
to AtoM Users
Here are my now different nginx error logs...:

2023/11/15 10:37:04 [error] 1510#1510: *3520 FastCGI sent in stderr: "PHP message: PHP Parse error:  syntax error, unexpected 'true' (T_STRING), expecting ',' or ')' in /usr/share/nginx/atom/index.php on line 5" while reading response header from upstream, client: 52.233.106.84, server: _, request: "GET /index.php/informationobject/browse?sf_culture=fr&sort=referenceCode&view=table&sortDir=desc&creators=433&subjects=152314&topLod=0 HTTP/1.1", upstream: "fastcgi://unix:/run/php7.2-fpm.atom.sock:", host: "atom.xx.xx.xx"
2023/11/15 10:37:04 [error] 1510#1510: *3531 FastCGI sent in stderr: "PHP message: PHP Parse error:  syntax error, unexpected 'true' (T_STRING), expecting ',' or ')' in /usr/share/nginx/atom/index.php on line 5" while reading response header from upstream, client: 23.22.35.162, server: _, request: "GET /index.php/informationobject/browse?collection=601&creators=487&names=82004&onlyMedia=1&sf_culture=fr&sort=alphabetic&sortDir=asc&topLod=0 HTTP/1.1", upstream: "fastcgi://unix:/run/php7.2-fpm.atom.sock:", host: "atom.xx.xx.xx"
2023/11/15 10:37:04 [error] 1510#1510: *3520 FastCGI sent in stderr: "PHP message: PHP Parse error:  syntax error, unexpected 'true' (T_STRING), expecting ',' or ')' in /usr/share/nginx/atom/index.php on line 5" while reading response header from upstream, client: 52.233.106.84, server: _, request: "GET /index.php/informationobject/browse?sortDir=desc&collection=151641&sort=lastUpdated&sf_culture=en&view=table&subjects=152314&mediatypes=136&topLod=0 HTTP/1.1", upstream: "fastcgi://unix:/run/php7.2-fpm.atom.sock:", host: "atom.xx.xx.xx"
Message has been deleted

BrzI Channel

unread,
Nov 15, 2023, 1:51:22 PM11/15/23
to AtoM Users
and my index.php file content:

<?php

require_once(dirname(__FILE__).'/config/ProjectConfiguration.class.php');

$configuration = ProjectConfiguration::getApplicationConfiguration('qubit', 'prod' true);
sfContext::createInstance($configuration)->dispatch();

José Raddaoui

unread,
Nov 15, 2023, 1:58:07 PM11/15/23
to AtoM Users
Hi,

It's missing a comma between 'prod' and true in line 5:


Best regards,
Radda.
Message has been deleted

BrzI Channel

unread,
Nov 15, 2023, 2:34:31 PM11/15/23
to AtoM Users
Thank you. Between that typo and all the other tasks - that resolved the issue. Application is now back up.

Issue closed.
Reply all
Reply to author
Forward
0 new messages