TFX CLI as part of a CI/CD pipeline

[Sascha Heyer]

Hi TFX Team and Community, 

using the TFX CLI as part of a CI/CD pipeline.
https://www.tensorflow.org/tfx/guide/cli

Every time a new version of the pipeline is released we initiate a build to update the Kubeflow pipeline. 

Using tfx pipeline upgrade works well as long as there is already an existing pipeline. 

For the very first build, before the pipeline is there it will fail. 
We can use tfx pipeline create for the first time but it would involve calling the command manually. 

To my questions:

1. Better approaches to deploy a TFX pipeline to Kubeflow in a automatic fashion?
2. Is there an option like "overwrite" for the pipeline create command. Or a fallback for tfx pipeline update in case the pipeline is not exiting?

Thanks a lot
Best regards
Sascha

[Dhruvesh Talati]

@Zhitao Li  to help with this 

--
You received this message because you are subscribed to the Google Groups "TensorFlow Extended (TFX)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tfx+uns...@tensorflow.org.
To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/tfx/d0832ade-0978-465c-942c-e8e0a6e2da04n%40tensorflow.org.

--

Cheers,

Dhruvesh Talati, on behalf of TFX::OSS team

[Sascha Heyer]

Or is the recommended approach to use the Kubeflow Pipelines SDK?

[Jiyong Jung]

Thank you for using the TFX CLI.

There is no option like 'overwrite' at this moment and 'update' will fail if the pipeline doesn't exist yet.
Currently, the only possible way would be 'list' pipelines and check whether it already exists.
(We might be able to add 'get' to make it more usable.)

To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/tfx/d89e8d8e-2ba9-4269-840b-673c3fcbfa3en%40tensorflow.org.

[Sascha Heyer]

@Jiyong thanks a lot for the feedback. 
Conditional steps in the build pipeline are only possible with workarounds.

Better approaches to deploy a TFX pipeline to Kubeflow in an automatic fashion?
Can you provide some of your experience?

[Sascha Heyer]

From the implementation, I can see it is anyway checked if the pipeline already exists. 
Can't we just incorporate this into the command itself. 

https://github.com/tensorflow/tfx/blob/1afb72cc939c6f20cc2d01c44a446f5cd331fc27/tfx/tools/cli/handler/kubeflow_handler.py#L61

Jiyoung, what do you think, adding an official feature request on Github probably the best way to track that request?

[Sascha Heyer]

Appendix:
Found out _check_pipeline_existence just checks if the pipeline file already exists locally.
Too bad though that is the functionality I was looking for. 

[Jiyong Jung]

Thank you for your investigation.

Filed a FR: https://github.com/tensorflow/tfx/issues/3386

I'll implement that but I'm not sure what would be a more desirable syntax yet.

I would be grateful if you could give me some feedback on it (possibly on GitHub).

[Sascha Heyer]

👍