jobs stuck in terminated state

[Jon Schewe]

This past weekend a number of my jobs are stuck in a terminated state. The job log shows that the backup completed successfully and the after job script executed as it should, however when I check running jobs I see the jobs showing up as:

 18561 Virtual  XXXXX.2025-03-29_00.01.01_54 has terminated 

And it's preventing other jobs on the same client and storage from executing.

Here's the end of the log from one of the jobs. The warning about "TLS-PSK" is expected, that's from the after job script and is just how the python library works.

 2025-03-31 06:32:00 bareos-sd JobId 18561: Elapsed time=17:57:00, Transfer rate=115.3 M Bytes/second
 2025-03-31 06:32:00 bareos-sd JobId 18561: Releasing device "LTO-9_drive1" (/dev/tape/by-id/scsi-35000e111ca01f0d3-nst).
 2025-03-31 06:32:04 bareos-sd JobId 18561: Releasing device "onsite-file3" (/mnt/onsite-file).
 2025-03-31 06:32:04 bareos-dir JobId 18561: Insert of attributes batch table with 589764 entries start
 2025-03-31 06:32:17 bareos-dir JobId 18561: Insert of attributes batch table done
 2025-03-31 06:32:17 bareos-dir JobId 18561: Joblevel was set to joblevel of first consolidated job: Full
 2025-03-31 06:32:18 bareos-dir JobId 18561: Bareos bareos-dir 24.0.2~pre11.1b367c590 (27Feb25):
  Build OS:               Red Hat Enterprise Linux release 9.5 (Plow)
  JobId:                  18561
  Job:                    XXXXX.2025-03-29_00.01.01_54
  Backup Level:           Virtual Full
  Client:                 "XXXX" 24.0.3~pre0.54685a85d (27Mar25) Red Hat Enterprise Linux release 8.10 (Ootpa),redhat
  FileSet:                "XXXX" 2024-10-29 21:00:00
  Pool:                   "offsite-LTO-9" (From Job Pool's NextPool resource)
  Catalog:                "MyCatalog" (From Client resource)
  Storage:                "LTO-9" (From Storage from Pool's NextPool resource)
  Scheduled time:         29-Mar-2025 00:01:01
  Start time:             29-Mar-2025 18:00:01
  End time:               29-Mar-2025 18:02:45
  Elapsed time:           2 mins 44 secs
  Priority:               10
  Allow Mixed Priority:   yes
  SD Files Written:       1,389,765
  SD Bytes Written:       7,456,741,957,914 (7.456 TB)
  Rate:                   45467938.8 KB/s
  Volume name(s):         ANJ641L9
  Volume Session Id:      987
  Volume Session Time:    1742768420
  Last Volume Bytes:      21,615,942,833,152 (21.61 TB)
  SD Errors:              0
  SD termination status:  OK
  Accurate:               yes
  Bareos binary info:     Bareos community build (UNSUPPORTED): Get professional support from https://www.bareos.com
  Job triggered by:       Scheduler
  Termination:            Backup OK

 2025-03-31 06:32:18 bareos-dir JobId 18561: shell command: run AfterJob "/var/lib/bbn/bareos-scripts/after-offsite 'artifacts.bbn.com-offsite' artifacts.bbn.com-offsite.2025-03-29_00.01.01_54 ANJ641L9"
 2025-03-31 06:32:19 bareos-dir JobId 18561: AfterJob: WARNING:root:socket error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate cha
 2025-03-31 06:32:19 bareos-dir JobId 18561: AfterJob: in (_ssl.c:1028)
 2025-03-31 06:32:19 bareos-dir JobId 18561: AfterJob: WARNING:root:Failed to connect via TLS-PSK. Trying plain connection.
 2025-03-31 06:32:19 bareos-dir JobId 18561: AfterJob: INFO:root:Authentication: b'OK: bareos-dir Version: 24.0.2~pre11.1b367c590 (27 February 2025)'
 2025-03-31 06:32:19 bareos-dir JobId 18561: console command: run AfterJob "update jobid=18561 jobtype=A"

[Jon Schewe]

One change I did make before the weekend was to set "Allow Mixed Priority = yes" on all jobs so that long running archive jobs don't prevent regular backups from completing. Could this have something to do with my jobs being stuck in this terminated, but not quite terminated state?