arUpdateEsIoDocumentsJob Spamming System

[Tom Misilo]

Hello,

We have noticed since upgrading to 2.10.1 that the job arUpdateEsIoDocumentsJob keeps getting queued over and over again. Causing millions of "jobs" to be logged. For example since Friday we have had 11 million new jobs created .

It doesn't seem like it's stalled as I am seeing jobs that are being completed, just that more and more keep getting added. 

I did log an issue https://github.com/artefactual/atom/issues/2301 but wanted to check here as well in case others have seen this and and have a solution.

Thanks

Tom

[Anvit Srivastav]

Hi Tom, 

There wasn't have anything that changed in 2.10.1 in terms of document updates. Have you tried clearing the jobs completely - https://www.accesstomemory.org/en/docs/2.10/admin-manual/maintenance/asynchronous-jobs/#troubleshooting-jobs-that-will-not-complete and restarting the atom-worker and gearman, since this could happen if the worker itself is stuck.

[Tom Misilo]

Yes, I have done that and it keeps creating new jobs and processing them. That is why we were only down to 11M. We can do this again, but it does take a few hours to delete all the jobs!

Are there any other things we could check? 

This is what I see in the gearmand container

```

gearmand-1  |  NOTICE 2026-02-20 14:44:29.000000 [  proc ] accepted,a778d6ef985a1f0e80caabccbda99674-arUpdateEsIoDocumentsJob,c4102a31f60cb8c192c1cd333cd12861,0 -> libgearman-server/server.cc:321
gearmand-1  |    INFO 2026-02-20 14:44:29.000000 [     2 ] Gear connection disconnected: -:-
gearmand-1  |    INFO 2026-02-20 14:44:29.000000 [     3 ] Peer connection has called close()
gearmand-1  |    INFO 2026-02-20 14:44:29.000000 [     3 ] Disconnected 172.18.0.6:48362
gearmand-1  |    INFO 2026-02-20 14:44:29.000000 [     3 ] Gear connection disconnected: -:-
gearmand-1  |    INFO 2026-02-20 14:44:29.000000 [  main ] Accepted connection from 172.18.0.6:48370
gearmand-1  |    INFO 2026-02-20 14:44:29.000000 [     1 ] Peer connection has called close()
gearmand-1  |    INFO 2026-02-20 14:44:29.000000 [     1 ] Disconnected 172.18.0.6:48370
gearmand-1  |    INFO 2026-02-20 14:44:29.000000 [     1 ] Gear connection disconnected: -:-
gearmand-1  |    INFO 2026-02-20 14:44:29.000000 [  main ] Accepted connection from 172.18.0.6:48386
gearmand-1  |    INFO 2026-02-20 14:44:29.000000 [  main ] Accepted connection from 172.18.0.6:48392
gearmand-1  |    INFO 2026-02-20 14:44:29.000000 [     3 ] Peer connection has called close()
gearmand-1  |    INFO 2026-02-20 14:44:29.000000 [     3 ] Disconnected 172.18.0.6:48392
gearmand-1  |  NOTICE 2026-02-20 14:44:29.000000 [  proc ] accepted,a778d6ef985a1f0e80caabccbda99674-arUpdateEsIoDocumentsJob,d481deed40a6d58efa28f9cc7bdcc911,0 -> libgearman-server/server.cc:321
gearmand-1  |    INFO 2026-02-20 14:44:29.000000 [     3 ] Gear connection disconnected: -:-
gearmand-1  |    INFO 2026-02-20 14:44:29.000000 [     4 ] Peer connection has called close()
gearmand-1  |    INFO 2026-02-20 14:44:29.000000 [     4 ] Disconnected 172.18.0.6:48386
gearmand-1  |    INFO 2026-02-20 14:44:29.000000 [     4 ] Gear connection disconnected: -:-
gearmand-1  |    INFO 2026-02-20 14:44:30.000000 [  main ] Accepted connection from 172.18.0.6:48408
gearmand-1  |    INFO 2026-02-20 14:44:30.000000 [     2 ] Peer connection has called close()
gearmand-1  |    INFO 2026-02-20 14:44:30.000000 [     2 ] Disconnected 172.18.0.6:48408
gearmand-1  |    INFO 2026-02-20 14:44:30.000000 [     2 ] Gear connection disconnected: -:-
gearmand-1  |    INFO 2026-02-20 14:44:30.000000 [  main ] Accepted connection from 172.18.0.6:48422
gearmand-1  |    INFO 2026-02-20 14:44:30.000000 [  main ] Accepted connection from 172.18.0.6:48434
^Cgearmand-1  |    INFO 2026-02-20 14:44:30.000000 [     4 ] Peer connection has called close()
gearmand-1  |    INFO 2026-02-20 14:44:30.000000 [     4 ] Disconnected 172.18.0.6:48434
gearmand-1  |  NOTICE 2026-02-20 14:44:30.000000 [  proc ] accepted,a778d6ef985a1f0e80caabccbda99674-arUpdateEsIoDocumentsJob,ebaa198aa621a8f01a43caf1030e1451,0 -> libgearman-server/server.cc:321
gearmand-1  |    INFO 2026-02-20 14:44:30.000000 [     4 ] Gear connection disconnected: -:-
gearmand-1  |    INFO 2026-02-20 14:44:30.000000 [     1 ] Peer connection has called close()
gearmand-1  |    INFO 2026-02-20 14:44:30.000000 [     1 ] Disconnected 172.18.0.6:48422
gearmand-1  |    INFO 2026-02-20 14:44:30.000000 [     1 ] Gear connection disconnected: -:-
gearmand-1  |    INFO 2026-02-20 14:44:30.000000 [  main ] Accepted connection from 172.18.0.6:48442
gearmand-1  |    INFO 2026-02-20 14:44:30.000000 [     3 ] Peer connection has called close()
gearmand-1  |    INFO 2026-02-20 14:44:30.000000 [     3 ] Disconnected 172.18.0.6:48442
gearmand-1  |    INFO 2026-02-20 14:44:30.000000 [     3 ] Gear connection disconnected: -:-
gearmand-1  |    INFO 2026-02-20 14:44:30.000000 [  main ] Accepted connection from 172.18.0.6:48454

```

app_worker appears to be processing jobs

```

app_worker-1  | 2026-02-20 06:57:30 > Parameters: {"ioIds":[1498730],"updateIos":false,"updateDescendants":true,"objectId":1498730,"name":"arUpdateEsIoDocumentsJob","id":"1779489"}
app_worker-1  | 2026-02-20 06:57:30 > Updating description(s) related to Academic Deans Council files.
app_worker-1  | 2026-02-20 06:57:30 > Updating descendants of 1 description(s).
app_worker-1  | 2026-02-20 06:57:30 > Updating descendant of 1 description(s).
app_worker-1  | 2026-02-20 06:57:30 > Job finished.
app_worker-1  | 2026-02-20 06:57:30 > Jobs completed: 1025
app_worker-1  | 2026-02-20 06:57:30 > Job started.
app_worker-1  | 2026-02-20 06:57:30 > Parameters: {"ioIds":[1498730],"updateIos":false,"updateDescendants":true,"objectId":1498730,"name":"arUpdateEsIoDocumentsJob","id":"1779490"}
app_worker-1  | 2026-02-20 06:57:30 > Updating description(s) related to Academic Deans Council files.
app_worker-1  | 2026-02-20 06:57:30 > Updating descendants of 1 description(s).
app_worker-1  | 2026-02-20 06:57:30 > Updating descendant of 1 description(s).
app_worker-1  | 2026-02-20 06:57:30 > Job finished.
app_worker-1  | 2026-02-20 06:57:30 > Jobs completed: 1026
app_worker-1  | 2026-02-20 06:57:30 > Job started.
app_worker-1  | 2026-02-20 06:57:30 > Parameters: {"ioIds":[1498730],"updateIos":false,"updateDescendants":true,"objectId":1498730,"name":"arUpdateEsIoDocumentsJob","id":"1779491"}
app_worker-1  | 2026-02-20 06:57:30 > Updating description(s) related to Academic Deans Council files.
app_worker-1  | 2026-02-20 06:57:30 > Updating descendants of 1 description(s).
app_worker-1  | 2026-02-20 06:57:30 > Updating descendant of 1 description(s).
app_worker-1  | 2026-02-20 06:57:30 > Job finished.
app_worker-1  | 2026-02-20 06:57:30 > Jobs completed: 1027
app_worker-1  | 2026-02-20 06:57:30 > Job started.
app_worker-1  | 2026-02-20 06:57:31 > Parameters: {"ioIds":[1498730],"updateIos":false,"updateDescendants":true,"objectId":1498730,"name":"arUpdateEsIoDocumentsJob","id":"1779492"}
app_worker-1  | 2026-02-20 06:57:31 > Updating description(s) related to Academic Deans Council files.
app_worker-1  | 2026-02-20 06:57:31 > Updating descendants of 1 description(s).
app_worker-1  | 2026-02-20 06:57:31 > Updating descendant of 1 description(s).
app_worker-1  | 2026-02-20 06:57:31 > Job finished.
app_worker-1  | 2026-02-20 06:57:31 > Jobs completed: 1028
app_worker-1  | 2026-02-20 06:57:31 > Job started.
app_worker-1  | 2026-02-20 06:57:31 > Parameters: {"ioIds":[1498730],"updateIos":false,"updateDescendants":true,"objectId":1498730,"name":"arUpdateEsIoDocumentsJob","id":"1779493"}
app_worker-1  | 2026-02-20 06:57:31 > Updating description(s) related to Academic Deans Council files.
app_worker-1  | 2026-02-20 06:57:31 > Updating descendants of 1 description(s).
app_worker-1  | 2026-02-20 06:57:31 > Updating descendant of 1 description(s).
app_worker-1  | 2026-02-20 06:57:31 > Job finished.
app_worker-1  | 2026-02-20 06:57:31 > Jobs completed: 1029
app_worker-1  | 2026-02-20 06:57:31 > Job started.
app_worker-1  | 2026-02-20 06:57:31 > Parameters: {"ioIds":[1498730],"updateIos":false,"updateDescendants":true,"objectId":1498730,"name":"arUpdateEsIoDocumentsJob","id":"1779494"}
app_worker-1  | 2026-02-20 06:57:31 > Updating description(s) related to Academic Deans Council files.
app_worker-1  | 2026-02-20 06:57:31 > Updating descendants of 1 description(s).
^Capp_worker-1  | 2026-02-20 06:57:31 > Updating descendant of 1 description(s).
app_worker-1  | 2026-02-20 06:57:31 > Job finished.

```

Thanks!

On Mon, Feb 23, 2026 at 1:51 PM 'Anvit Srivastav' via AtoM Users <ica-ato...@googlegroups.com> wrote:

Hi Tom, 

There wasn't have anything that changed in 2.10.1 in terms of document updates. Have you tried clearing the jobs completely - https://www.accesstomemory.org/en/docs/2.10/admin-manual/maintenance/asynchronous-jobs/#troubleshooting-jobs-that-will-not-complete and restarting the atom-worker and gearman, since this could happen if the worker itself is stuck.

--
You received this message because you are subscribed to a topic in the Google Groups "AtoM Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ica-atom-users/RSIW21mGZ5M/unsubscribe.
To unsubscribe from this group and all its topics, send an email to ica-atom-user...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/ica-atom-users/8c15bf37-70cd-4b60-b273-395dec8efc51n%40googlegroups.com.

[Anvit Srivastav]

It is hard for me to guess what's causing this, because this suggests actual new jobs are being created and finishing constantly, and typically something needs to happen for a new job to be created. You can get an alert for an existing job if the atom worker is restarted before it is finished, but this seems to be different since they're actually finishing.

You don't have a CLI task running that's possibly being triggered automatically do you? And hopefully the system isn't out of memory?

One other thing to try is backing up the db, and using that db on a local environment to see if you can manage to recreate that issue, which would help confirm that it's a server issue.

To view this discussion visit https://groups.google.com/d/msgid/ica-atom-users/CACZhg%3DG-boXYVn-FfbOVvpsHrhk41AfhNiO30KEAAsW9V2T7bg%40mail.gmail.com.

--

Anvit Srivastav (He/him)

AtoM Maintainer
Artefactual Systems
604-527-2056

@archivematica / @accesstomemory

I am a settler on the unceded homelands of the xwməθkwəy̓əm (Musqueam), Skwxwú7mesh (Squamish), and səl̓ílwətaʔɬ/Selilwitulh (Tsleil-Waututh) Peoples. I am committed to being in solidarity with Indigenous Sovereignties on their own terms.

[Tom Misilo]

I don't see any OOM messages. That's the weird thing it is doing the same thing in both our production and test environment.

What we are doing is

"I am basically adding a child level to the collection and then it tells me it is updating the descendents, and so I go check the jobs page, and within a few moments the job is "done" and I go back to the finding aid and the child (Series) is not there."

[Tom Misilo]

Video of what we are trying: https://www.youtube.com/watch?v=Tkq8vEM0ZzU

[Tom Misilo]

So it looks like we can add a new child record through Add New (doing it through Edit, seems to be broken?).

We upgraded from 2.8.2 so not sure if something between 2.10.1 and that version might give some hints.

Thanks!

Tom

[Tom Misilo]

Anvit,

Is there a way to see what database migrations happened between v2.8.2 and v2.10.1 in case we need to downgrade our system so we could possibly revert those pieces but keep the existing data that might have changed since we upgraded.

Thanks,

Tom

[Johan Pieterse]

Hi

If you are technical sort of yes.  The migration files shows the code of what commands/sql command is executed. 

But did you not do a backup before upgrade?

Johan Pieterse
082 337-1406

You received this message because you are subscribed to the Google Groups "AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-user...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/ica-atom-users/db7383f9-b6fb-4bda-8801-c7b414b93ff6n%40googlegroups.com.

[Tom Misilo]

We did a back up, but it's been a couple of weeks since we upgraded (accessions changed, etc...) so there has been data changing.

To view this discussion visit https://groups.google.com/d/msgid/ica-atom-users/CAA4s_cDP1-%2B4Sa-TfcsYqucVkHutkoYsQn8d7M1oqXBW438S-A%40mail.gmail.com.

[Johan Pieterse]

There are really two parts to this: finding what schema changes happened between versions, and then the practical approach to comparing and potentially reverting. Let me address both.

1. Finding the migration changes between v2.8.2 and v2.10.1

With any new AtoM release that includes changes to the database, a schema migration script is included that makes the necessary changes to conform to the expected schema. During the upgrade process, these migration scripts are executed when the upgrade task is run (AtoM) — specifically via php symfony tools:upgrade-sql.

The migration scripts live in the AtoM source code under lib/task/migrate/ (the arUpgradeSqlTask class). Each migration is a numbered PHP class that runs sequentially. You can inspect the actual SQL changes by looking at the AtoM GitHub repo and comparing the two branches:

Browse lib/task/migrate/migrations/ between the stable/2.8.x and stable/2.10.x branches

Any migration class numbered higher than what existed in 2.8.2 represents a schema change introduced after that release

Artefactual has consistently avoided including database schema migrations in point releases (GitHub) , meaning v2.8.0 → v2.8.2 likely had zero schema changes. The schema differences will be concentrated in the 2.9.0 and 2.10.0 major releases.

2. Comparing two MySQL instances directly

This is the more practical approach — rather than trying to trace individual migrations, just diff the schemas directly. Several methods work well:

Option A: mysqldump schema-only comparison (simplest, no extra tools)

# Dump schema only from your v2.10.1 instance

mysqldump -u root -d atom_v210 > /tmp/schema_v210.sql

# Dump schema only from a v2.8.2 reference instance  

mysqldump -u root -d atom_v282 > /tmp/schema_v282.sql

# Diff them

diff /tmp/schema_v282.sql /tmp/schema_v210.sql > /tmp/schema_changes.diff

The -d flag dumps structure only (no data). The diff output will show you every column addition, table creation, index change, etc. It's noisy because of auto-increment values and ordering, but it's comprehensive.

Option B: mysql-schema-diff (generates executable ALTER statements)

mysql-schema-diff compares the data structures of two MySQL databases and returns the differences as a sequence of MySQL commands (User Guide) that can transform one schema into the other. This is extremely useful for your use case because the output is directly usable for reverting:

# Install on Ubuntu

sudo apt-get install libmysql-diff-perl

# Compare: this generates SQL to transform v2.10 BACK to v2.8

mysql-schema-diff --host=localhost --user=root \

  --password='' db:atom_v210 db:atom_v282 > /tmp/downgrade.sql

The output gives you the exact ALTER TABLE, DROP TABLE, DROP COLUMN statements needed. Obviously review carefully before applying.

Option C: DBDiff (PHP-based, does schema AND data)

DBDiff compares two databases, local or remote, and produces a migration file of the differences automatically, including up and down SQL (DBDiff) . Since it's PHP-based, it runs natively in your AtoM environment:

composer global require dbdiff/dbdiff

dbdiff server1.atom_v282:server2.atom_v210 --type=schema

3. Practical downgrade strategy

Here's what I'd actually recommend for a safe approach:

Before anything, take a full backup of your current v2.10.1 database and files:

mysqldump -u root atom > /tmp/atom_v210_full_backup_$(date +%Y%m%d).sql

Set up a reference v2.8.2 instance — spin up a clean AtoM 2.8.2 installation (even a temporary VM) with a fresh database. This gives you a known-good v2.8.2 schema to compare against.

Run the schema comparison using one of the methods above. Focus on identifying:

New tables added in 2.10 (these can likely just be dropped)

New columns added to existing tables (these may contain data you want to preserve)

Changed column types or constraints (these are the trickiest to revert)

Index changes

The hard part: finding aids, cached XML, and digital objects are not stored in the database — these are found in the uploads and downloads directories (AtoM) . So your data preservation concern is mainly about the relational data in MySQL, plus those file directories.

Important caveat: AtoM doesn't officially support downgrades. The tools:upgrade-sql migrations are one-directional — there's no built-in tools:downgrade-sql. Even if you revert the schema, the v2.8.2 codebase may expect data in formats that the 2.10.x migrations transformed. For example, if a migration moved data from one table to another or changed how values are encoded, just reverting the schema won't undo those data transformations.

The safest downgrade path is usually: restore a pre-upgrade database backup (you did take one before upgrading, right?), then re-import only the new/changed records created since the upgrade. This preserves data integrity far better than trying to reverse-engineer migration rollbacks.

Hope that helps — and yes, always take backups before major version upgrades. That pre-upgrade snapshot is worth its weight in gold in situations like this.

Johan Pieterse
082 337-1406

To view this discussion visit https://groups.google.com/d/msgid/ica-atom-users/CACZhg%3DGM2VcTi3Q95wJqM_vZe8GAdwuAvx7qd-bfEJN7J3JTRw%40mail.gmail.com.

[Tom Misilo]

Thank you for this.

We had ` jobs:worker --application=qubit --env=prod` and it seems like it was the "--application=qubit --env=prod" being there that broke it.

Removing the application and env flags seems to make everything work.

Thanks again!

[Anvit Srivastav]

I'm glad that ended up being resolved! The application, env, and also the connection flags are for use by symfony and should not be used since they're set automatically.

I'm curious to know where you were setting the flags? Was it from your atom-worker.service file or were you creating a new worker manually?