Terraform Jenkins airflow bigquery

[Rajesh Rajesh]

Hi ,

I am trying a simple component where once file arrives in gcs trigger dataflow job insert into big query 

What I want is how to wire everything together using terraform  github ( for source control ) and wire devops . Can somebody guide me step by step?

Idea is terraform should take care end to end  if I have to take it to multiple envs 

[Rajesh Rajesh]

Team any help appreciated 

[Felipe Bergallo Corral]

Sorry, I'm reading through this and I'm not entirely sure what it is that you want to do? You mention Terraform, GCS triggers, a Dataflow job, an insert into BigQuery, Github, Wire Devops, Jenkins and Airflow...

I understand that you want to use these... but to do what? It seems like a lot of systems with overlapping functions, for example:
 -  Dataflow and Airflow do very similar things. Google Dataflow is actually an alternative to Apache Airflow.
 -  GCS trigger can interact with any API because it's used for Cloud Functions, so if your intent is just to insert data into BigQuery you may not need any kind of Dataflow outside of the creation of said Cloud Function (with the appropriate trigger)
 -  Is the insert into BigQuery the result of a Dataflow job or are you inserting part of the contents of a file on GCS into BigQuery?
 -  I presume that your inclusion of Jenkins in the title refers to Jenkins.io, how is that a part of your question? You don't mention it in the body.

Looking forward to your reply.

[Rajesh Rajesh]

Any thoughts on my updated points 

On Wed, Jan 5, 2022 at 11:32 AM Rajesh Rajesh <mrajes...@gmail.com> wrote:

 -  Is the insert into BigQuery the result of a Dataflow job or are you inserting part of the contents of a file on GCS into BigQuery?

        Planning to do Transformations and Clean up data so want to leverage Data Flow , rather than simple Cloud Functions to insert all data to Big Query 

 -  I presume that your inclusion of Jenkins in the title refers to Jenkins.io, how is that a part of your question? You don't mention it in the body.

      This is bit of lacking in my understanding when it comes to do actual Deployment , My idea was  instead of Cloud Build , will leverage Jenkins for CI/CD including Test Automation 

 -  Dataflow and Airflow do very similar things. Google Dataflow is actually an alternative to Apache Airflow.

      Well Here is the bigger problem I have  . I am trying to Deploy everything  via  Terraform,, so Dataflow is transformatiomns and leading  to BQ  but then say for some data cleanups or  Dq checks i want to spin   Python  code or do Big Query Transformations of the  staging data etc . So want to combine this 

 

--
You received this message because you are subscribed to the Google Groups "Google Cloud Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-d...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-dev/f197ce9c-2f00-4a96-9cc2-09d178264379n%40googlegroups.com.