Another way to reproduce this issue is by using an activeDeadlineSeconds to kill the pod after some time. The pod would fail with DeadlineExceeded but will not be deleted. The pipeline will hang until a manual operation is taken.