Hi Federico,
-how many instances of the Matcher service are you running?
We have only one Matcher, though Optimizers are 6.
- do you have other extensions in WMS? E.g in TaskQueueDB?
We have several modules in WMS (e.g. Agent to control our test job scheme (named "Scouting"))
Direct extensions to vanilla components are SiteDirector (PilotWrapper) and TaskQueueDB.
TaskQueueDB extenstion adds some new functions (e.g. update priority)
- which DIRAC version is this?
v7r0p56
- can you get to know the percentage of jobs being rescheduled?
Roughly 2% for Done and 9% for Failed jobs (RescheduleCounter>0 in JobDB)
- is the logging above full? I think there are more lines to be shown
I chopped since it is a bit longer and may be confusing.
Here is the rest.
JobPath Checking Scouting Running 2021-09-11 23:57:47
JobSanity Checking InputData Running 2021-09-11 23:57:47
Scouting Checking JobSanity Running 2021-09-11 23:57:47
JobManager Received Job Rescheduled Unknown 2021-09-11 23:57:47
InputData Checking JobScheduling Unknown 2021-09-11 23:57:49
JobScheduling Checking JobScheduling On Hold: after rescheduling 2 2021-09-11 23:57:49
JobScheduling Waiting Pilot Agent Submission Unknown 2021-09-12 00:02:59
Matcher Matched Assigned Unknown 2021-09-12 00:03:05
JobWrapper Rescheduled Input Sandbox Download Unknown 2021-09-12 00:38:19
JobWrapper Rescheduled Input Sandbox Download Failed Input Sandbox Download 2021-09-12 00:38:19
JobWrapper Running Failed Downloading InputSandbox Failed Input Sandbox Download 2021-09-12 00:38:19
JobA...@DIRAC.IITG.in Matched Job Received by Agent Failed Input Sandbox Download 2021-09-12 00:43:41
JobA...@DIRAC.IITG.in Matched Installing Software Failed Input Sandbox Download 2021-09-12 01:15:14
JobA...@DIRAC.IITG.in Matched Submitted To CE Failed Input Sandbox Download 2021-09-12 01:25:28
JobWrapper Running Job Initialization Failed Input Sandbox Download 2021-09-12 02:36:21
JobWrapper Running Downloading InputSandbox Failed Input Sandbox Download 2021-09-12 03:36:28
JobManager Received Job Rescheduled Unknown 2021-09-12 04:56:09
JobPath Checking Scouting Unknown 2021-09-12 04:56:09
Scouting Checking JobSanity Unknown 2021-09-12 04:56:09
JobSanity Checking InputData Unknown 2021-09-12 04:56:09
JobScheduling Checking InputData On Hold: after rescheduling 3 2021-09-12 04:56:11
InputData Checking JobScheduling On Hold: after rescheduling 3 2021-09-12 04:56:11
JobScheduling Waiting Pilot Agent Submission Unknown 2021-09-12 05:06:34
JobWrapper Running Failed Downloading InputSandbox Unknown 2021-09-12 05:06:40
JobWrapper Rescheduled Input Sandbox Download Unknown 2021-09-12 05:06:40
JobWrapper Rescheduled Input Sandbox Download Failed Input Sandbox Download 2021-09-12 05:06:40
Matcher Matched Assigned Failed Input Sandbox Download 2021-09-12 05:06:55
JobA...@DIRAC.Shandong.cn Matched Job Received by Agent Failed Input Sandbox Download 2021-09-12 05:06:55
JobA...@DIRAC.Shandong.cn Matched Installing Software Failed Input Sandbox Download 2021-09-12 05:06:57
JobA...@DIRAC.Shandong.cn Matched Submitted To CE Failed Input Sandbox Download 2021-09-12 05:07:02
JobWrapper Running Job Initialization Failed Input Sandbox Download 2021-09-12 05:07:08
JobWrapper Running Downloading InputSandbox Failed Input Sandbox Download 2021-09-12 05:07:10
JobWrapper Running Input Data Resolution Failed Input Sandbox Download 2021-09-12 05:07:16
JobWrapper Running Application Failed Input Sandbox Download 2021-09-12 05:07:23
Job_205833717 Running Application Executing RunScriptStep1 2021-09-12 05:07:27
basf2 Running Application Running 2021-09-12 05:07:39
gbasf2 Running Application Preparing to upload 2021-09-12 08:25:32
gbasf2 Running Application Uploading 2021-09-12 08:25:42
gbasf2 Running Application Done 2021-09-12 08:25:53
Job_205833717 Running Application Done 2021-09-12 08:25:55
JobWrapper Completed Application Finished Successfully Done 2021-09-12 08:26:06
JobWrapper Completed Uploading Output Sandbox Done 2021-09-12 08:26:07
JobWrapper Done Execution Complete Done 2021-09-12 08:26:09
JobWrapper Completed Output Sandbox Uploaded Done 2021-09-12 08:26:09
gbasf2 Completed Output Sandbox Uploaded Preparing to upload 2021-09-12 13:13:01
gbasf2 Completed Output Sandbox Uploaded Uploading 2021-09-12 13:13:32
gbasf2 Completed Output Sandbox Uploaded Error: existing Replica 2021-09-12 13:39:57
gbasf2 Completed Output Sandbox Uploaded Error: uploading failed 2021-09-12 13:39:57
gbasf2 Completed Output Sandbox Uploaded Error: existing Replica 2021-09-12 14:03:05
Job_205833717 Completed Output Sandbox Uploaded basf2helper.py Exited With Status 254 2021-09-12 14:03:06
Job_205833717 Completed Output Sandbox Uploaded RuntimeError("basf2helper.py Exited With Status 254",) 2021-09-12 14:03:07
JobWrapper Completed Application Finished With Errors RuntimeError("basf2helper.py Exited With Status 254",) 2021-09-12 14:03:16
JobWrapper Failed Application Finished With Errors RuntimeError("basf2helper.py Exited With Status 254",) 2021-09-12 14:03:18