/v1/scheduler/jobs/summary and UI showing multiple instances of the same job

[Mark Selby]

We are having an issue where the UI, which is fueled by the jobs/summary API call is show multiple running instances of a single job when we know this is (1) not enabled for the jobs and (2) not showing up in Mesos as running tasks.

I have gone through some of code to see how this can happen but I have not been able to pinpoint what is going on.

We have failed over the Chronos scheduler a couple of time to a failover node.

of the 273 jobs that we have configured we have a small subset which show this false behavior

Any insights on what is going on would be much appreciated.

We are running 3.0.2 as we had upgrade to mesos 1.x before 2.4 was patched for this. 

      "state": "20 running",

      "state": "6 running",

      "state": "20 running",

      "state": "18 running",

      "state": "6 running",

      "state": "18 running",

    {

      "name": "LUIGI-monthly-pings",

      "status": "success",

      "state": "6 running",

      "schedule": "R/2018-08-24T05:30:00.000Z/PT6H",

      "parents": [],

      "disabled": false

    },

    {

      "name": "ingestion-health-check",

      "status": "success",

      "state": "idle",

      "schedule": "R/2018-08-24T12:00:00.000Z/PT24H",

      "parents": [],

      "disabled": false

    },

    {

      "name": "LUIGI-Universal_Player",

      "status": "success",

      "state": "20 running",

      "schedule": "R/2018-08-23T21:00:00.000Z/PT1H",

      "parents": [],

      "disabled": false

    },