Is there any way of triggering DataPrep based flow from AppEngine?

[Chauhan B]

Hi,

 

I am new to Google DataPrep, and for one of my implementation I am trying to use Google DataPrep.

 

I have to create a UI using App Maker and I have to run a pipeline for performing intermediate jobs [data cleansing & transformation] and pushing final data to destination. I have used Google DataPrep for performing intermediate data cleansing and transformation but I am not able to make out how to run/execute my dataprep based flows on button click event from my AppMaker based UI?

 

What is the way of calling any dataprep pipeline on every button click from UI? Is it possible or not?

 

Any quick reference would be of great help.

 

Regards,

Bhupesh

[Ali T (Cloud Platform Support)]

Hi,

When you launch a Dataprep job, it actually creates a Dataflow template. That being said, once the Dataprep job is launched at least once, you can run it as a Dataflow template. Thus, from your app, when the button is clicked, you could launch the job using the Dataflow template REST API. 

The template created from the Dataprep job will be located in GCS. To locate it, in the Dataprep UI, click on the jobs tab. When hovering over the historical jobs, on the far right side you will have a “view results” button. On this page, on the right hand side, there will be a “Job Summary” section which indicates the GCS location of the created Dataflow template.

Lastly, prior to doing so, I would suggest going over the known limitations when running a Dataflow job from a Dataprep created template.