Trigger Pipeline with S3 File Change

[J B]

We are trying to start a build in GoCD when a file in S3 is added or updated.

The pipelines require a material, which is tied to a source control repository. These files are too large for source code and are using S3 as a way to manage this.

Is there a way to solve this? Is a material required for the pipeline to trigger? A timer can be used to start the pipeline, but how can it be configured to run a shell script to fetch the file from S3 as the first step, because the rest of the build cannot continue without it? If the pipeline requires a material, and that material must be in a repository, will the pipeline run without it and continue to execute stages/jobs/tasks?

Thank you!

[Ashwanth Kumar]

Your material for the pipeline can be the repository where you've your Shell scripts required for the build. It's a good thing to keep those separate and use GoCD to just trigger them. 

For triggering pipeline with S3 File Change, I see you've the following choices

1. You can write your own SCM Plugin. (Shameless plug - Take a look at this repo for an example of SCM Plugin). 
2. If your GoCD is on AWS, you can setup Event Notifications on S3 to trigger a Lambda function whenever a new file is added (using ObjectCreated Event) and use the Schedule Endpoint of the pipeline to trigger it from the Lambda function. 

You can choose either of the two approaches depending on how comfortable your organization / team is with either GoCD or AWS Stack. 

--
You received this message because you are subscribed to the Google Groups "go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/a5527548-8231-4311-9d73-8cff15d603fa%40googlegroups.com.

--

Ashwanth Kumar / ashwanthkumar.in