Using Multiple Repositories in a Pipeline

[Jaz Chana]

Hi,

I have multiple projects that are all compiled, tested and built in the same way. In summary I want to be able to build and test all of these projects in the same way and not block any of the projects from each other when doing this. For example, if I make a change to project A and B and check-in at the same time I want to test and build them on go server at the same time.

An easy way to do this is to create a pipeline for each of the applications, e.g. Pipeline A, Pipeline B, each with its own material (svn repo). However there are 15 applications and I don't want to have to repeat this configuration in 15 Pipelines. So I thought an easier way to do this would be to create a template and then 15 pipelines that use that template. So the bulk of the configuration is all in one template. However I'll still have 15 pipelines and there will be some configurations required in each.

Is there a way to achieve the above using a single pipeline but with multiple materials? 

The other reason I want to do this is because I want to be able to deploy to same bucket in s3 (useing the go-cd s3 plugin). For example I want projectA.war and projectB.war to be published to go-artifacts/pipeline/published/artifacts/1.1.

Any help would be appreciated.

Cheers

Jaz

[Mike Leary]

+1

[Radoslav Minchev]

It's an interesting scenario, but if all these are really different projects I would prefer to have better control on them by having a "build" pipeline per project. I don't want if project A fails to affect the rest of the projects.

my 2 cents.

-rado

[Jaz Chana]

I the way I was thinking it could work is, I could check-in project A and while thats building check-in project B and have two separate instances of the pipeline running, for each project. This way they would then not be dependant on each other and a failure in one instance would not result in a failure to build another instance. 

--
You received this message because you are subscribed to a topic in the Google Groups "go-cd" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/go-cd/2ETiT-zgr0w/unsubscribe.
To unsubscribe from this group and all its topics, send an email to go-cd+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Radoslav Minchev]

yeah, that's an option, I still think that a better separation of concerns is always a good choice when a project(s) grow. 

Best,

Rado

[Jaz Chana]

I think you are right and normally that's exactly what I would do. What's different here is that we have a series of 'micro-services' that have identical project structures and build lifecycles. A change in one will result in a change in the the others,

For now I think I'll just bite the bullet and create the 15 individual build pipelines.

[Rajesh Waran]

Is there a way to get a generic pipeline that can take material url as parameter. Since we have microservices and each is linked to a repository and we have one pipeline per repository.

ended up many. 

DevOps worried about the maintenance.

Any suggestion ?

[Magnus Lyckå]

Den måndag 7 mars 2016 kl. 15:12:38 UTC+1 skrev Rajesh Waran:

Is there a way to get a generic pipeline that can take material url as parameter. Since we have microservices and each is linked to a repository and we have one pipeline per repository.

ended up many. 

DevOps worried about the maintenance.

Any suggestion ?

Make a pipeline template which captures the stages/jobs/tasks performed.

Make a pipeline for each micro service. The pipeline refers to the template, and lists material.
It might also need to use parameters to pass data to the templates.

We've built a tool called gocdpb, hosted at https://github.com/magnus-lycka/gocd-pipeline-builder
which creates pipelines based on a git repo it's in. This tool is still in beta, so there are some rough
edges...

To install this tool:

    sudo pip2 install gocdpb

Then you can make a pattern file for it looking something like this:

[
  {
    "environment": "my_go_environment",
    "create-a-pipeline": {
      "group": "{{ group }}",
      "pipeline": {
        "name": "{{ repo_name }}",
        "template": "my_pipeline_template",
        "parameters": [],
        "materials": [
          {
            "type": "git",
            "attributes": {
              "url": "{{ repo_url }}",
              "branch": "master"
            }
          }
        ],
        "stages": null
      }
    }
  },
  {
    "add-downstream-dependencies": [
      {
        "name": "downstream_pipeline",
        "task": {
          "type": "fetch",
          "attributes": {
            "run_if": [
              "passed"
            ],
            "pipeline": "{{ repo_name }}",
            "stage": "defaultStage",
            "job": "defaultJob",
            "is_source_a_file": true,
            "source": "mybuildartifact.jar"
          }
        }
      }
    ]
  }
]

This json file uses Jinja2 templating (The {{ variable }} stuff). Assuming that you're in a git repo, it will figure out repo_url, and repo_name will default to the basename of the current directory. I.e. you just need to hand it the name of the pipeline group, and how to connect to the go-server. Assuming you make the json available at http://someserver/gopattern.json, and you log in as 'go' at http://goserver, you can run:

    gocdpb -j http://someserver/gopattern.json -D group=mygroup -C username=go -P password

Besides the README at https://github.com/magnus-lycka/gocd-pipeline-builder there are examples of use in
https://github.com/magnus-lycka/gocd-pipeline-builder/tree/master/src/texttest

[Kadu Barral]

There are any news about this in the last 6 years?

Same case here, we are working with golang microservices and pipelines are about the same GENERATE PROTO (with make) > BUILD GO BINARY (make) > BUILD DOCKER IMAGE > UNIT TEST (make) > DEPLOY (kustomize)

There are any way to use the same pipeline with different materials?

Each microservice is one repo in github.

Thanks

[Sriram Narayanan]

On Wed, Apr 7, 2021 at 12:22 AM Kadu Barral <kadub...@gmail.com> wrote:

There are any news about this in the last 6 years?

Same case here, we are working with golang microservices and pipelines are about the same GENERATE PROTO (with make) > BUILD GO BINARY (make) > BUILD DOCKER IMAGE > UNIT TEST (make) > DEPLOY (kustomize)

There are any way to use the same pipeline with different materials?

Each microservice is one repo in github.

I have used Templates for this - where we create a template with the flow that you describe, and then create pipelines per template with materials per pipeline.

I've used the XML configuration approach.

-- Ram

 

You received this message because you are subscribed to the Google Groups "go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/977a45ed-cfa9-4851-88e0-6286b3fd02a1n%40googlegroups.com.