arUpdateEsIoDocumentsJob Spamming System

[Tom Misilo]

Hello,

We have noticed since upgrading to 2.10.1 that the job arUpdateEsIoDocumentsJob keeps getting queued over and over again. Causing millions of "jobs" to be logged. For example since Friday we have had 11 million new jobs created .

It doesn't seem like it's stalled as I am seeing jobs that are being completed, just that more and more keep getting added. 

I did log an issue https://github.com/artefactual/atom/issues/2301 but wanted to check here as well in case others have seen this and and have a solution.

Thanks

Tom

[Anvit Srivastav]

Hi Tom, 

There wasn't have anything that changed in 2.10.1 in terms of document updates. Have you tried clearing the jobs completely - https://www.accesstomemory.org/en/docs/2.10/admin-manual/maintenance/asynchronous-jobs/#troubleshooting-jobs-that-will-not-complete and restarting the atom-worker and gearman, since this could happen if the worker itself is stuck.

[Tom Misilo]

Yes, I have done that and it keeps creating new jobs and processing them. That is why we were only down to 11M. We can do this again, but it does take a few hours to delete all the jobs!

Are there any other things we could check? 

This is what I see in the gearmand container

```

gearmand-1  |  NOTICE 2026-02-20 14:44:29.000000 [  proc ] accepted,a778d6ef985a1f0e80caabccbda99674-arUpdateEsIoDocumentsJob,c4102a31f60cb8c192c1cd333cd12861,0 -> libgearman-server/server.cc:321
gearmand-1  |    INFO 2026-02-20 14:44:29.000000 [     2 ] Gear connection disconnected: -:-
gearmand-1  |    INFO 2026-02-20 14:44:29.000000 [     3 ] Peer connection has called close()
gearmand-1  |    INFO 2026-02-20 14:44:29.000000 [     3 ] Disconnected 172.18.0.6:48362
gearmand-1  |    INFO 2026-02-20 14:44:29.000000 [     3 ] Gear connection disconnected: -:-
gearmand-1  |    INFO 2026-02-20 14:44:29.000000 [  main ] Accepted connection from 172.18.0.6:48370
gearmand-1  |    INFO 2026-02-20 14:44:29.000000 [     1 ] Peer connection has called close()
gearmand-1  |    INFO 2026-02-20 14:44:29.000000 [     1 ] Disconnected 172.18.0.6:48370
gearmand-1  |    INFO 2026-02-20 14:44:29.000000 [     1 ] Gear connection disconnected: -:-
gearmand-1  |    INFO 2026-02-20 14:44:29.000000 [  main ] Accepted connection from 172.18.0.6:48386
gearmand-1  |    INFO 2026-02-20 14:44:29.000000 [  main ] Accepted connection from 172.18.0.6:48392
gearmand-1  |    INFO 2026-02-20 14:44:29.000000 [     3 ] Peer connection has called close()
gearmand-1  |    INFO 2026-02-20 14:44:29.000000 [     3 ] Disconnected 172.18.0.6:48392
gearmand-1  |  NOTICE 2026-02-20 14:44:29.000000 [  proc ] accepted,a778d6ef985a1f0e80caabccbda99674-arUpdateEsIoDocumentsJob,d481deed40a6d58efa28f9cc7bdcc911,0 -> libgearman-server/server.cc:321
gearmand-1  |    INFO 2026-02-20 14:44:29.000000 [     3 ] Gear connection disconnected: -:-
gearmand-1  |    INFO 2026-02-20 14:44:29.000000 [     4 ] Peer connection has called close()
gearmand-1  |    INFO 2026-02-20 14:44:29.000000 [     4 ] Disconnected 172.18.0.6:48386
gearmand-1  |    INFO 2026-02-20 14:44:29.000000 [     4 ] Gear connection disconnected: -:-
gearmand-1  |    INFO 2026-02-20 14:44:30.000000 [  main ] Accepted connection from 172.18.0.6:48408
gearmand-1  |    INFO 2026-02-20 14:44:30.000000 [     2 ] Peer connection has called close()
gearmand-1  |    INFO 2026-02-20 14:44:30.000000 [     2 ] Disconnected 172.18.0.6:48408
gearmand-1  |    INFO 2026-02-20 14:44:30.000000 [     2 ] Gear connection disconnected: -:-
gearmand-1  |    INFO 2026-02-20 14:44:30.000000 [  main ] Accepted connection from 172.18.0.6:48422
gearmand-1  |    INFO 2026-02-20 14:44:30.000000 [  main ] Accepted connection from 172.18.0.6:48434
^Cgearmand-1  |    INFO 2026-02-20 14:44:30.000000 [     4 ] Peer connection has called close()
gearmand-1  |    INFO 2026-02-20 14:44:30.000000 [     4 ] Disconnected 172.18.0.6:48434
gearmand-1  |  NOTICE 2026-02-20 14:44:30.000000 [  proc ] accepted,a778d6ef985a1f0e80caabccbda99674-arUpdateEsIoDocumentsJob,ebaa198aa621a8f01a43caf1030e1451,0 -> libgearman-server/server.cc:321
gearmand-1  |    INFO 2026-02-20 14:44:30.000000 [     4 ] Gear connection disconnected: -:-
gearmand-1  |    INFO 2026-02-20 14:44:30.000000 [     1 ] Peer connection has called close()
gearmand-1  |    INFO 2026-02-20 14:44:30.000000 [     1 ] Disconnected 172.18.0.6:48422
gearmand-1  |    INFO 2026-02-20 14:44:30.000000 [     1 ] Gear connection disconnected: -:-
gearmand-1  |    INFO 2026-02-20 14:44:30.000000 [  main ] Accepted connection from 172.18.0.6:48442
gearmand-1  |    INFO 2026-02-20 14:44:30.000000 [     3 ] Peer connection has called close()
gearmand-1  |    INFO 2026-02-20 14:44:30.000000 [     3 ] Disconnected 172.18.0.6:48442
gearmand-1  |    INFO 2026-02-20 14:44:30.000000 [     3 ] Gear connection disconnected: -:-
gearmand-1  |    INFO 2026-02-20 14:44:30.000000 [  main ] Accepted connection from 172.18.0.6:48454

```

app_worker appears to be processing jobs

```

app_worker-1  | 2026-02-20 06:57:30 > Parameters: {"ioIds":[1498730],"updateIos":false,"updateDescendants":true,"objectId":1498730,"name":"arUpdateEsIoDocumentsJob","id":"1779489"}
app_worker-1  | 2026-02-20 06:57:30 > Updating description(s) related to Academic Deans Council files.
app_worker-1  | 2026-02-20 06:57:30 > Updating descendants of 1 description(s).
app_worker-1  | 2026-02-20 06:57:30 > Updating descendant of 1 description(s).
app_worker-1  | 2026-02-20 06:57:30 > Job finished.
app_worker-1  | 2026-02-20 06:57:30 > Jobs completed: 1025
app_worker-1  | 2026-02-20 06:57:30 > Job started.
app_worker-1  | 2026-02-20 06:57:30 > Parameters: {"ioIds":[1498730],"updateIos":false,"updateDescendants":true,"objectId":1498730,"name":"arUpdateEsIoDocumentsJob","id":"1779490"}
app_worker-1  | 2026-02-20 06:57:30 > Updating description(s) related to Academic Deans Council files.
app_worker-1  | 2026-02-20 06:57:30 > Updating descendants of 1 description(s).
app_worker-1  | 2026-02-20 06:57:30 > Updating descendant of 1 description(s).
app_worker-1  | 2026-02-20 06:57:30 > Job finished.
app_worker-1  | 2026-02-20 06:57:30 > Jobs completed: 1026
app_worker-1  | 2026-02-20 06:57:30 > Job started.
app_worker-1  | 2026-02-20 06:57:30 > Parameters: {"ioIds":[1498730],"updateIos":false,"updateDescendants":true,"objectId":1498730,"name":"arUpdateEsIoDocumentsJob","id":"1779491"}
app_worker-1  | 2026-02-20 06:57:30 > Updating description(s) related to Academic Deans Council files.
app_worker-1  | 2026-02-20 06:57:30 > Updating descendants of 1 description(s).
app_worker-1  | 2026-02-20 06:57:30 > Updating descendant of 1 description(s).
app_worker-1  | 2026-02-20 06:57:30 > Job finished.
app_worker-1  | 2026-02-20 06:57:30 > Jobs completed: 1027
app_worker-1  | 2026-02-20 06:57:30 > Job started.
app_worker-1  | 2026-02-20 06:57:31 > Parameters: {"ioIds":[1498730],"updateIos":false,"updateDescendants":true,"objectId":1498730,"name":"arUpdateEsIoDocumentsJob","id":"1779492"}
app_worker-1  | 2026-02-20 06:57:31 > Updating description(s) related to Academic Deans Council files.
app_worker-1  | 2026-02-20 06:57:31 > Updating descendants of 1 description(s).
app_worker-1  | 2026-02-20 06:57:31 > Updating descendant of 1 description(s).
app_worker-1  | 2026-02-20 06:57:31 > Job finished.
app_worker-1  | 2026-02-20 06:57:31 > Jobs completed: 1028
app_worker-1  | 2026-02-20 06:57:31 > Job started.
app_worker-1  | 2026-02-20 06:57:31 > Parameters: {"ioIds":[1498730],"updateIos":false,"updateDescendants":true,"objectId":1498730,"name":"arUpdateEsIoDocumentsJob","id":"1779493"}
app_worker-1  | 2026-02-20 06:57:31 > Updating description(s) related to Academic Deans Council files.
app_worker-1  | 2026-02-20 06:57:31 > Updating descendants of 1 description(s).
app_worker-1  | 2026-02-20 06:57:31 > Updating descendant of 1 description(s).
app_worker-1  | 2026-02-20 06:57:31 > Job finished.
app_worker-1  | 2026-02-20 06:57:31 > Jobs completed: 1029
app_worker-1  | 2026-02-20 06:57:31 > Job started.
app_worker-1  | 2026-02-20 06:57:31 > Parameters: {"ioIds":[1498730],"updateIos":false,"updateDescendants":true,"objectId":1498730,"name":"arUpdateEsIoDocumentsJob","id":"1779494"}
app_worker-1  | 2026-02-20 06:57:31 > Updating description(s) related to Academic Deans Council files.
app_worker-1  | 2026-02-20 06:57:31 > Updating descendants of 1 description(s).
^Capp_worker-1  | 2026-02-20 06:57:31 > Updating descendant of 1 description(s).
app_worker-1  | 2026-02-20 06:57:31 > Job finished.

```

Thanks!

On Mon, Feb 23, 2026 at 1:51 PM 'Anvit Srivastav' via AtoM Users <ica-ato...@googlegroups.com> wrote:

Hi Tom, 

There wasn't have anything that changed in 2.10.1 in terms of document updates. Have you tried clearing the jobs completely - https://www.accesstomemory.org/en/docs/2.10/admin-manual/maintenance/asynchronous-jobs/#troubleshooting-jobs-that-will-not-complete and restarting the atom-worker and gearman, since this could happen if the worker itself is stuck.

--
You received this message because you are subscribed to a topic in the Google Groups "AtoM Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ica-atom-users/RSIW21mGZ5M/unsubscribe.
To unsubscribe from this group and all its topics, send an email to ica-atom-user...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/ica-atom-users/8c15bf37-70cd-4b60-b273-395dec8efc51n%40googlegroups.com.

[Anvit Srivastav]

It is hard for me to guess what's causing this, because this suggests actual new jobs are being created and finishing constantly, and typically something needs to happen for a new job to be created. You can get an alert for an existing job if the atom worker is restarted before it is finished, but this seems to be different since they're actually finishing.

You don't have a CLI task running that's possibly being triggered automatically do you? And hopefully the system isn't out of memory?

One other thing to try is backing up the db, and using that db on a local environment to see if you can manage to recreate that issue, which would help confirm that it's a server issue.

To view this discussion visit https://groups.google.com/d/msgid/ica-atom-users/CACZhg%3DG-boXYVn-FfbOVvpsHrhk41AfhNiO30KEAAsW9V2T7bg%40mail.gmail.com.

--

Anvit Srivastav (He/him)

AtoM Maintainer
Artefactual Systems
604-527-2056

@archivematica / @accesstomemory

I am a settler on the unceded homelands of the xwməθkwəy̓əm (Musqueam), Skwxwú7mesh (Squamish), and səl̓ílwətaʔɬ/Selilwitulh (Tsleil-Waututh) Peoples. I am committed to being in solidarity with Indigenous Sovereignties on their own terms.

[Tom Misilo]

I don't see any OOM messages. That's the weird thing it is doing the same thing in both our production and test environment.

What we are doing is

"I am basically adding a child level to the collection and then it tells me it is updating the descendents, and so I go check the jobs page, and within a few moments the job is "done" and I go back to the finding aid and the child (Series) is not there."

[Tom Misilo]

Video of what we are trying: https://www.youtube.com/watch?v=Tkq8vEM0ZzU

[Tom Misilo]

So it looks like we can add a new child record through Add New (doing it through Edit, seems to be broken?).

We upgraded from 2.8.2 so not sure if something between 2.10.1 and that version might give some hints.

Thanks!

Tom