Help Needed: Dataproc Batches and Composer - Job Status Signaling

[Damian Przekop]

Hi everyone,

We're working on a Proof of Concept (PoC) where we're using Composer to orchestrate Dataproc Batches for feature engineering tasks.

We're facing an issue with correctly signaling the job status in Composer. Specifically, the status of the Dataproc Batch job isn't being accurately reflected in Composer during its execution and after it finishes.

We're trying to figure out if this is an inherent limitation or a misconfiguration in our environment.

Has anyone else experienced this? We'd appreciate any insights or suggestions on how to resolve this.

Thanks in advance!