vcjob's status is still pending even though all tasks completed

[Alton Patrick]

We have seen this a few times now in our use of Volcano. We create a vcjob with a single task. This spawns one pod. The pod finishes (successfully). But the vcjob's status remains stuck at "pending." After restarting the Volcano controller, the vcjob's status gets updated to "completed" as expected.

This happens sporadically. If there is a pattern, we haven't identified it yet, except it seems more likely to happen when there are more vcjobs running on our cluster.

Has anyone else seen this? Any suggestions for fixing it or debugging?

Thanks.

-Alton