atom_worker process exits docker container with code 255 when exporting CSV

279 views
Skip to first unread message

jesus.s...@gmail.com

unread,
Jul 13, 2020, 4:07:03 AM7/13/20
to AtoM Users
Hello,

We are running atom 2.5.1 on docker, and when trying to CSV export a not so large (2000 rows) collection, the atom_worker process exits with code 255. The job never ends and keeps cycling, so I finish it through the console:

This is the docker log:

[atomadminpre@svatomi ~]$ docker-compose exec atom php symfony jobs:worker
Warning: Zend Engine 3.0 is no longer supported by Dynatrace PHP Agent. Module will not be loaded.
2020-07-13 00:41:33 > New ability: arFindingAidJob
2020-07-13 00:41:33 > New ability: arInheritRightsJob
2020-07-13 00:41:33 > New ability: arCalculateDescendantDatesJob
2020-07-13 00:41:33 > New ability: arObjectMoveJob
2020-07-13 00:41:33 > New ability: arInformationObjectCsvExportJob
2020-07-13 00:41:33 > New ability: qtSwordPluginWorker
2020-07-13 00:41:33 > New ability: arUpdatePublicationStatusJob
2020-07-13 00:41:33 > New ability: arFileImportJob
2020-07-13 00:41:33 > New ability: arInformationObjectXmlExportJob
2020-07-13 00:41:33 > New ability: arXmlExportSingleFileJob
2020-07-13 00:41:33 > New ability: arGenerateReportJob
2020-07-13 00:41:33 > New ability: arActorCsvExportJob
2020-07-13 00:41:33 > New ability: arActorXmlExportJob
2020-07-13 00:41:33 > New ability: arRepositoryCsvExportJob
2020-07-13 00:41:33 > New ability: arUpdateEsIoDocumentsJob
2020-07-13 00:41:33 > Running worker...
2020-07-13 00:41:33 > PID 6600


[atomadminpre@svatomi ~]$ docker-compose logs -f atom_worker
Attaching to docker_atom_worker_4, docker_atom_worker_2, docker_atom_worker_3, docker_atom_worker_1
atom_worker_1    | 2020-07-13 00:48:39 > Job started.
atom_worker_1    | 2020-07-13 00:48:39 > Starting export to /tmp/663d875c7223311f1559fde2774acde0.
atom_worker_3    | 2020-07-13 00:48:46 > Job started.
atom_worker_3    | 2020-07-13 00:48:46 > Starting export to /tmp/663d875c7223311f1559fde2774acde0.
atom_worker_4    | 2020-07-13 00:48:52 > Job started.
atom_worker_4    | 2020-07-13 00:48:52 > Starting export to /tmp/663d875c7223311f1559fde2774acde0.
atom_worker_2    | 2020-07-13 00:48:59 > Job started.
atom_worker_2    | 2020-07-13 00:48:59 > Starting export to /tmp/663d875c7223311f1559fde2774acde0.
atom_worker_2    | Warning: Zend Engine 3.0 is no longer supported by Dynatrace PHP Agent. Module will not be loaded.
atom_worker_4    | Warning: Zend Engine 3.0 is no longer supported by Dynatrace PHP Agent. Module will not be loaded.
atom_worker_3    | Warning: Zend Engine 3.0 is no longer supported by Dynatrace PHP Agent. Module will not be loaded.
atom_worker_1    | Warning: Zend Engine 3.0 is no longer supported by Dynatrace PHP Agent. Module will not be loaded.
atom_worker_1    | Warning: Zend Engine 3.0 is no longer supported by Dynatrace PHP Agent. Module will not be loaded.
atom_worker_3    | Warning: Zend Engine 3.0 is no longer supported by Dynatrace PHP Agent. Module will not be loaded.
atom_worker_4    | Warning: Zend Engine 3.0 is no longer supported by Dynatrace PHP Agent. Module will not be loaded.
atom_worker_2    | Warning: Zend Engine 3.0 is no longer supported by Dynatrace PHP Agent. Module will not be loaded.
atom_worker_3    | 2020-07-13 00:49:48 > New ability: arFindingAidJob
atom_worker_3    | 2020-07-13 00:49:48 > New ability: arInheritRightsJob
atom_worker_3    | 2020-07-13 00:49:48 > New ability: arCalculateDescendantDatesJob
atom_worker_3    | 2020-07-13 00:49:48 > New ability: arObjectMoveJob
atom_worker_3    | 2020-07-13 00:49:48 > New ability: arInformationObjectCsvExportJob
atom_worker_2    | 2020-07-13 00:49:48 > New ability: arFindingAidJob
atom_worker_2    | 2020-07-13 00:49:48 > New ability: arInheritRightsJob
atom_worker_2    | 2020-07-13 00:49:48 > New ability: arCalculateDescendantDatesJob
atom_worker_2    | 2020-07-13 00:49:48 > New ability: arObjectMoveJob
atom_worker_2    | 2020-07-13 00:49:48 > New ability: arInformationObjectCsvExportJob
atom_worker_4    | 2020-07-13 00:49:48 > New ability: arFindingAidJob
atom_worker_4    | 2020-07-13 00:49:48 > New ability: arInheritRightsJob
atom_worker_4    | 2020-07-13 00:49:48 > New ability: arCalculateDescendantDatesJob
atom_worker_4    | 2020-07-13 00:49:48 > New ability: arObjectMoveJob
atom_worker_4    | 2020-07-13 00:49:48 > New ability: arInformationObjectCsvExportJob
atom_worker_1    | 2020-07-13 00:49:48 > New ability: arFindingAidJob
atom_worker_1    | 2020-07-13 00:49:48 > New ability: arInheritRightsJob
atom_worker_1    | 2020-07-13 00:49:48 > New ability: arCalculateDescendantDatesJob
atom_worker_1    | 2020-07-13 00:49:48 > New ability: arObjectMoveJob
atom_worker_1    | 2020-07-13 00:49:48 > New ability: arInformationObjectCsvExportJob
atom_worker_1    | 2020-07-13 00:49:48 > Ability not defined: qtSwordPluginWorker. Please ensure the job is in the lib/task/job directory or that the plugin is enabled.
atom_worker_1    | 2020-07-13 00:49:48 > New ability: arUpdatePublicationStatusJob
atom_worker_1    | 2020-07-13 00:49:48 > New ability: arFileImportJob
atom_worker_1    | 2020-07-13 00:49:48 > New ability: arInformationObjectXmlExportJob
atom_worker_1    | 2020-07-13 00:49:48 > New ability: arXmlExportSingleFileJob
atom_worker_1    | 2020-07-13 00:49:48 > New ability: arGenerateReportJob
atom_worker_1    | 2020-07-13 00:49:48 > New ability: arActorCsvExportJob
atom_worker_1    | 2020-07-13 00:49:48 > New ability: arActorXmlExportJob
atom_worker_1    | 2020-07-13 00:49:48 > New ability: arRepositoryCsvExportJob
atom_worker_1    | 2020-07-13 00:49:48 > New ability: arUpdateEsIoDocumentsJob
atom_worker_1    | 2020-07-13 00:49:48 > Running worker...
atom_worker_1    | 2020-07-13 00:49:48 > PID 14
atom_worker_4    | 2020-07-13 00:49:48 > Ability not defined: qtSwordPluginWorker. Please ensure the job is in the lib/task/job directory or that the plugin is enabled.
atom_worker_4    | 2020-07-13 00:49:48 > New ability: arUpdatePublicationStatusJob
atom_worker_4    | 2020-07-13 00:49:48 > New ability: arFileImportJob
atom_worker_4    | 2020-07-13 00:49:48 > New ability: arInformationObjectXmlExportJob
atom_worker_4    | 2020-07-13 00:49:48 > New ability: arXmlExportSingleFileJob
atom_worker_4    | 2020-07-13 00:49:48 > New ability: arGenerateReportJob
atom_worker_4    | 2020-07-13 00:49:48 > New ability: arActorCsvExportJob
atom_worker_4    | 2020-07-13 00:49:48 > New ability: arActorXmlExportJob
atom_worker_4    | 2020-07-13 00:49:48 > New ability: arRepositoryCsvExportJob
atom_worker_4    | 2020-07-13 00:49:48 > New ability: arUpdateEsIoDocumentsJob
atom_worker_4    | 2020-07-13 00:49:48 > Running worker...
atom_worker_4    | 2020-07-13 00:49:48 > PID 14
atom_worker_3    | 2020-07-13 00:49:48 > Ability not defined: qtSwordPluginWorker. Please ensure the job is in the lib/task/job directory or that the plugin is enabled.
atom_worker_3    | 2020-07-13 00:49:48 > New ability: arUpdatePublicationStatusJob
atom_worker_3    | 2020-07-13 00:49:48 > New ability: arFileImportJob
atom_worker_3    | 2020-07-13 00:49:48 > New ability: arInformationObjectXmlExportJob
atom_worker_3    | 2020-07-13 00:49:48 > New ability: arXmlExportSingleFileJob
atom_worker_3    | 2020-07-13 00:49:48 > New ability: arGenerateReportJob
atom_worker_2    | 2020-07-13 00:49:48 > Ability not defined: qtSwordPluginWorker. Please ensure the job is in the lib/task/job directory or that the plugin is enabled.
atom_worker_2    | 2020-07-13 00:49:48 > New ability: arUpdatePublicationStatusJob
atom_worker_2    | 2020-07-13 00:49:48 > New ability: arFileImportJob
atom_worker_2    | 2020-07-13 00:49:48 > New ability: arInformationObjectXmlExportJob
atom_worker_2    | 2020-07-13 00:49:48 > New ability: arXmlExportSingleFileJob
atom_worker_2    | 2020-07-13 00:49:48 > New ability: arGenerateReportJob
atom_worker_2    | 2020-07-13 00:49:48 > New ability: arActorCsvExportJob
atom_worker_2    | 2020-07-13 00:49:48 > New ability: arActorXmlExportJob
atom_worker_2    | 2020-07-13 00:49:48 > New ability: arRepositoryCsvExportJob
atom_worker_2    | 2020-07-13 00:49:48 > New ability: arUpdateEsIoDocumentsJob
atom_worker_2    | 2020-07-13 00:49:48 > Running worker...
atom_worker_2    | 2020-07-13 00:49:48 > PID 14
atom_worker_3    | 2020-07-13 00:49:48 > New ability: arActorCsvExportJob
atom_worker_3    | 2020-07-13 00:49:48 > New ability: arActorXmlExportJob
atom_worker_3    | 2020-07-13 00:49:48 > New ability: arRepositoryCsvExportJob
atom_worker_3    | 2020-07-13 00:49:48 > New ability: arUpdateEsIoDocumentsJob
atom_worker_3    | 2020-07-13 00:49:48 > Running worker...
atom_worker_3    | 2020-07-13 00:49:48 > PID 14
atom_worker_1    | 2020-07-13 00:49:48 > Job started.
atom_worker_1    | 2020-07-13 00:49:48 > Starting export to /tmp/663d875c7223311f1559fde2774acde0.
atom_worker_2    | 2020-07-13 00:49:54 > Job started.
atom_worker_2    | 2020-07-13 00:49:54 > Starting export to /tmp/663d875c7223311f1559fde2774acde0.
docker_atom_worker_1 exited with code 255
atom_worker_3    | 2020-07-13 00:50:01 > Job started.
atom_worker_3    | 2020-07-13 00:50:01 > Starting export to /tmp/663d875c7223311f1559fde2774acde0.
docker_atom_worker_2 exited with code 255
atom_worker_4    | 2020-07-13 00:50:09 > Job started.
atom_worker_4    | 2020-07-13 00:50:09 > Starting export to /tmp/663d875c7223311f1559fde2774acde0.
docker_atom_worker_3 exited with code 255
docker_atom_worker_4 exited with code 255
atom_worker_3    | Warning: Zend Engine 3.0 is no longer supported by Dynatrace PHP Agent. Module will not be loaded.
atom_worker_2    | Warning: Zend Engine 3.0 is no longer supported by Dynatrace PHP Agent. Module will not be loaded.
atom_worker_1    | Warning: Zend Engine 3.0 is no longer supported by Dynatrace PHP Agent. Module will not be loaded.
^CERROR: Aborting.

[atomadminpre@svatomi ~]$ docker-compose exec atom php symfony jobs:worker
Warning: Zend Engine 3.0 is no longer supported by Dynatrace PHP Agent. Module will not be loaded.
2020-07-13 00:50:52 > New ability: arFindingAidJob
2020-07-13 00:50:52 > New ability: arInheritRightsJob
2020-07-13 00:50:52 > New ability: arCalculateDescendantDatesJob
2020-07-13 00:50:52 > New ability: arObjectMoveJob
2020-07-13 00:50:52 > New ability: arInformationObjectCsvExportJob
2020-07-13 00:50:52 > Ability not defined: qtSwordPluginWorker. Please ensure the job is in the lib/task/job directory or that the plugin is enabled.
2020-07-13 00:50:52 > New ability: arUpdatePublicationStatusJob
2020-07-13 00:50:52 > New ability: arFileImportJob
2020-07-13 00:50:52 > New ability: arInformationObjectXmlExportJob
2020-07-13 00:50:52 > New ability: arXmlExportSingleFileJob
2020-07-13 00:50:52 > New ability: arGenerateReportJob
2020-07-13 00:50:52 > New ability: arActorCsvExportJob
2020-07-13 00:50:52 > New ability: arActorXmlExportJob
2020-07-13 00:50:52 > New ability: arRepositoryCsvExportJob
2020-07-13 00:50:52 > New ability: arUpdateEsIoDocumentsJob
2020-07-13 00:50:52 > Running worker...
2020-07-13 00:50:52 > PID 6737
2020-07-13 00:50:54 > Job started.
2020-07-13 00:50:54 > Starting export to /tmp/663d875c7223311f1559fde2774acde0.
[atomadminpre@svatomi ~]$ docker-compose exec atom php symfony jobs:clear
Warning: Zend Engine 3.0 is no longer supported by Dynatrace PHP Agent. Module will not be loaded.
WARNING: AtoM reports there are jobs currently running. It is *highly* recommended you make sure there aren't any jobs actually running.

                                                                  
  Are you SURE you want to clear all jobs in the database? (y/N)  
                                                                  
y
>> jobs:clear All jobs cleared successfully!
[atomadminpre@svatomi ~]$

Any idea how to further debug the problem?

Best,
Jesus

José Raddaoui

unread,
Jul 14, 2020, 7:17:14 AM7/14/20
to AtoM Users
Hi Jesus,

My first impression is that "Warning: Zend Engine 3.0 is no longer supported by Dynatrace PHP Agent. Module will not be loaded." is more than a warning. I have not seen this before but Zend is used for the permissions module in AtoM, which is used in both the GUI and worker permission checks. Is your environment using the default Docker Compose setup? Could it be that the AtoM worker Docker service/container is using a custom Dockerfile or a different base image?

Best regards,
Radda.

jesus.s...@gmail.com

unread,
Jul 14, 2020, 10:31:21 AM7/14/20
to AtoM Users
Hi José.

Dockerfile and base image are default for stable/2.5.x
It works flawlessly when exporting a shorter sample, let's say 200 rows...

José Raddaoui

unread,
Jul 14, 2020, 2:34:42 PM7/14/20
to AtoM Users
Hi Jesus,

Could it be memory related? You should be able to increase the limit used by the workers in https://github.com/artefactual/atom/blob/stable/2.5.x/docker/bootstrap.php#L57 and you'll need to recreate the containers to reflect the change.

Bests.

jesus san miguel

unread,
Jul 15, 2020, 5:11:02 AM7/15/20
to ica-ato...@googlegroups.com
Yes, you were right. Just out of curiosity I deployed the containers
to a different hypervisor and could see the memory exhaustion log
message (see below on CoreOS).
Increased the worker memory to 4GB and finally the job was able to complete:

2020-07-15 01:33:53 > Job started.
2020-07-15 01:33:53 > Starting export to /tmp/2afbf272648d2e02e9c2224a395e8554.
PHP Fatal error: Allowed memory size of 536870912 bytes exhausted
(tried to allocate 20480 bytes) in
/atom/src/lib/model/om/BaseObject.php on line 559
PHP Fatal error: Allowed memory size of 536870912 bytes exhausted
(tried to allocate 20480 bytes) in
/atom/src/vendor/symfony/lib/autoload/sfSimpleAutoload.class.php on
line 165

// Let's increase atom_worker_1 memory from 521M to 4G //

2020-07-15 02:02:23 > New ability: arFindingAidJob
2020-07-15 02:02:23 > New ability: arInheritRightsJob
2020-07-15 02:02:23 > New ability: arCalculateDescendantDatesJob
2020-07-15 02:02:23 > New ability: arObjectMoveJob
2020-07-15 02:02:23 > New ability: arInformationObjectCsvExportJob
2020-07-15 02:02:23 > Ability not defined: qtSwordPluginWorker. Please
ensure the job is in the lib/task/job directory or that the plugin is
enabled.
2020-07-15 02:02:23 > New ability: arUpdatePublicationStatusJob
2020-07-15 02:02:23 > New ability: arFileImportJob
2020-07-15 02:02:23 > New ability: arInformationObjectXmlExportJob
2020-07-15 02:02:23 > New ability: arXmlExportSingleFileJob
2020-07-15 02:02:23 > New ability: arGenerateReportJob
2020-07-15 02:02:23 > New ability: arActorCsvExportJob
2020-07-15 02:02:23 > New ability: arActorXmlExportJob
2020-07-15 02:02:23 > New ability: arRepositoryCsvExportJob
2020-07-15 02:02:23 > New ability: arUpdateEsIoDocumentsJob
2020-07-15 02:02:23 > Running worker...
2020-07-15 02:02:23 > PID 15
2020-07-15 02:05:44 > Job started.
2020-07-15 02:05:44 > Starting export to /tmp/546daf598da8989fd213180ed616a2b4.
2020-07-15 02:07:08 > Exported 1 descriptions.
2020-07-15 02:07:08 > Creating ZIP file
/atom/src/downloads/jobs/970627414218ccff3497cb7a784288f5.zip.
2020-07-15 02:07:08 > Export and archiving complete.
2020-07-15 02:07:08 > Job finished.

You rock man!
> --
> You received this message because you are subscribed to a topic in the Google Groups "AtoM Users" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/ica-atom-users/H3JZArwVYig/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to ica-atom-user...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/dab5aa21-46e2-4104-9fae-741b39d1b7d0n%40googlegroups.com.

José Raddaoui

unread,
Jul 15, 2020, 2:26:28 PM7/15/20
to AtoM Users
Glad that made it work, Jesus! Bests!

jesus san miguel

unread,
Aug 12, 2020, 12:35:57 PM8/12/20
to ica-ato...@googlegroups.com
The machine running docker on RH7 still has problems not related to
memory limits though:

Increasing Docker verbosity I get this log:

atom_worker_1 | 2020-08-12 17:59:54 > New ability: arFindingAidJob
atom_worker_1 | 2020-08-12 17:59:54 > New ability: arInheritRightsJob
atom_worker_1 | 2020-08-12 17:59:54 > New ability:
arUpdateEsActorRelationsJob
atom_worker_1 | 2020-08-12 17:59:54 > New ability:
arCalculateDescendantDatesJob
gearmand_1 | INFO 2020-08-12 15:59:54.000000 [ main ]
Accepted connection from 172.18.0.8:33058
atom_worker_1 | 2020-08-12 17:59:54 > New ability: arObjectMoveJob
atom_worker_1 | 2020-08-12 17:59:54 > New ability:
arInformationObjectCsvExportJob
atom_worker_1 | 2020-08-12 17:59:54 > New ability: qtSwordPluginWorker
atom_worker_1 | 2020-08-12 17:59:54 > New ability:
arUpdatePublicationStatusJob
atom_worker_1 | 2020-08-12 17:59:54 > New ability: arFileImportJob
atom_worker_1 | 2020-08-12 17:59:54 > New ability:
arInformationObjectXmlExportJob
atom_worker_1 | 2020-08-12 17:59:54 > New ability: arXmlExportSingleFileJob
atom_worker_1 | 2020-08-12 17:59:54 > New ability: arGenerateReportJob
atom_worker_1 | 2020-08-12 17:59:54 > New ability: arActorCsvExportJob
atom_worker_1 | 2020-08-12 17:59:54 > New ability: arActorXmlExportJob
atom_worker_1 | 2020-08-12 17:59:54 > New ability: arRepositoryCsvExportJob
atom_worker_1 | 2020-08-12 17:59:54 > New ability: arUpdateEsIoDocumentsJob
atom_worker_1 | 2020-08-12 17:59:54 > New ability:
arPhysicalObjectCsvHoldingsReportJob
atom_worker_1 | 2020-08-12 17:59:54 > Running worker...
atom_worker_1 | 2020-08-12 17:59:54 > PID 26
atom_worker_1 | 2020-08-12 17:59:54 > Job started.
atom_worker_1 | 2020-08-12 17:59:54 > Starting export to
/tmp/095a5a0985c190486cbb17fceaee64e3.
atom_1 | 172.18.0.7 - 12/Aug/2020:16:00:00 +0000 "GET /index.php" 200
nginx_1 | 10.223.23.23 - - [12/Aug/2020:16:00:01 +0000] "GET
/ HTTP/1.1" 200 3634 "-" "Go-http-client/1.1" "-"
gearmand_1 | INFO 2020-08-12 16:00:00.000000 [ 1 ] Peer
connection has called close()
gearmand_1 | INFO 2020-08-12 16:00:00.000000 [ 1 ]
Disconnected 172.18.0.8:33058
gearmand_1 | INFO 2020-08-12 16:00:00.000000 [ 1 ] Gear
connection disconnected: -:-
docker_atom_worker_1 exited with code 255

Any idea?
> To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/b4baff8d-7dc1-4a22-9f7a-88b7eb5a5b3cn%40googlegroups.com.

José Raddaoui

unread,
Aug 12, 2020, 1:22:55 PM8/12/20
to AtoM Users
Hi Jesus,

The AtoM worker needs to be restarted after the upgrade. Maybe you only need to run "docker-compose up -d" again after you imported and upgraded the database. Or maybe the Gearman server container didn't start properly.

You could check the status of the AtoM worker and the Gearman Server containers with:

docker-compose ps

And check the logs if the restart fails with:

docker-compose logs -f atom_worker gearmand

Best regards,
Radda. 

Reply all
Reply to author
Forward
0 new messages