Multibranch pipelines

[jkon...@soldevelo.com]

Hi devs,

While working on a multibranch pipeline for referencedata (OLMIS-3498), I've encountered a couple of issues and wanted to ask, what are your thoughts on:

1) Publishing images - should we add a "-build#" suffix to every build, or just the ones outside of master branch?

2) Supporting previous releases in contract/performance tests: 

- should we support only the latest release, or all the previous releases?

- where should we store a list of component versions that a release has? My first throught was to have a couple of files similar to settings.env

- how to determine which release a built image belongs to? let's say we have built a patch release for referencedata - 9.0.1 and we have an existing release which has referencedata 9.0.0. 

3) I'm not sure which steps should be executed inside this new multi-branch pipeline job and which ones should be triggered by it. At first I throught it would be nice to have all the steps (build, test, publish, sonar analysis, contract, performance and functional tests, deploy) included in the Jenkinsfile, but after building a service we also run contract/performance tests for a couple other services (e.g. referencedata and fulfillment contract tests after building referencedata). It would also throttle other builds.

- should we parametrize existing builds and embed them into this new job, or should the job handle all the steps at its own?

- we only run performance and e2e tests once a day. I imagine it would be better to keep it this way, maybe run them immediately for images built on branches other than master?

- does it make sense to deploy images built for previous releases?

4) The Pipeline plugin has some limitations:

- It doesn't support docker-compose. I had to use 'sh' step instead and provide a path to docker-compose and wonder if it's alright 

sh '/usr/local/bin/docker-compose -f docker-compose.builder.yml build image'

- I had to install "Pipeline Utility Steps" plugin so that we can load gradle.properties and determine service's version. The EnvInject plugin doesn't fully support pipelines.

- Additionally, it would be nice to have meaningful ids for our credentials. For example, the .env file is referenced like so:

withCredentials([file(credentialsId: '8da5ba56-8ebb-4a6a-bdb5-43c9d0efb120', variable: 'ENV_FILE')])

Unfortunately, I'm unable to edit them, I think we will have to re-create these credentials.

Thank you,

Jakub

[Josh Zamor]

Hi Jakub,

Good questions.  Here’s some thoughts:

1) On adding a “-build#” suffix to every build, I think we should, so long as we also tag this image (not rebuild) without the build number when it passes tests.  In this way if 10.2.1-SNAPSHOT is being worked on in a release branch, then we’ll have an image under test such as 10.2.1-SNAPSHOT-build1124 and an image tagged 10.2.1-SNAPSHOT which is the last successful build of the release.  This lets us continue to have the latest of a release, as well as a specific build for use in testing.  We’ll need to remember to clean up the docker images with build numbers once they’re no longer needed - when and however (success/failure) the pipeline ends.

2) What we want is to be able to hand the contract tests a list of specific images to test with.  In the current ref distro, that file which lists the service versions is the “.env” file.  For previous releases where the version was in the docker-compose (and one that’s more robust anyway), the command `docker-compose config` can tell us the image tags that a specific release used.  Again our goal for this ticket is to primarily tell the contract tests which images to run a particular test against.  In a pipeline, I think this is equivalent as saying:

- Run every image as the version that’s in ref-distro (the head for the in-progress work).

- EXCEPT for the image that we just built previously in the pipeline.  For that service run the image tagged with the build number.

3) That’s a good question.  Ideally we have as much coded into the Jenkinsfile and checked into git as possible alongside the project CI is supposed to build.  This I think includes running tests that are beyond the scope of just a single service - if a stock management contract tests fails because of a commit in referencedata, then the commit in referencedata should be considered a breaking commit and fail the referencedata pipeline.  For performance and E2E, well keep those out of the service’s Jenkinsfile.  I’m not sure I understand your question on deploy with previous releases or parametrize existing builds, could you elaborate?

4) 

- Using sh is okay.  However I think it’s slightly improved if you add docker-compose to the PATH:  https://stackoverflow.com/questions/43237051/how-to-set-path-in-jenkins-declarative-pipeline

- Even better though I think adding a shared library which adds this to the path of every jenkinsfile is even better:  https://jenkins.io/doc/book/pipeline/shared-libraries/

Thoughts?  It would help if the Jenkinsfile lands in git, that might help us collaborate on the right steps.

Best,

Josh

 

SolDevelo Sp. z o.o. [LLC] / www.soldevelo.com
Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland
Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41

--
You received this message because you are subscribed to the Google Groups "OpenLMIS Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openlmis-dev...@googlegroups.com.
To post to this group, send email to openlm...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/openlmis-dev/2195ef7b-76f2-4dd7-ba37-d0d6c0e400dd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Sebastian Brudziński]

On 2) - with the number of (breaking) changes to our endpoints in all our services, I'd be hesitant to just use the latest ref-distro image versions and assume this will work for with the built patch images in most cases. Ideally, our API would be more stable, but I don't think we are there yet. My feeling is that we would very quickly run into a problem with failing contract test because there were breaking changes introduced to one or more services during current development.

Perhaps a simple ability to override this default behavior of using latest ref-distro versions would be sufficient? I know this would require to manually click the button to run the job and specify the versions, but this would provide a working alternative for automated patch builds that break due to the contract test failures caused by breaking API changes.

Any thoughts?

Sebastian.

To view this discussion on the web visit https://groups.google.com/d/msgid/openlmis-dev/DB88D06F-94A2-44D4-92F7-2D6366B5D556%40villagereach.org.

For more options, visit https://groups.google.com/d/optout.

--

Sebastian Brudziński
Senior Software Developer / Team Leader
sbrud...@soldevelo.com

[Jakub Kondrat]

Hi Josh and Sebastian,

thanks for your feedback!

1) It seems that we don't need to push these images with -build# suffix. We can just tag the image and use it in sonar analysis and contract tests. If they pass, we could push the actual tag (e.x. 10.0.0-SNAPSHOT). Is there any case where we would want to publish "1.0.0-SNAPSHOT-build174"?

2) I agree that it would be nice to at least have the option to manually specify component versions.

3) I have added both referencedata and stockmanagement contract tests to the pipeline.
By parametrizing existing builds I meant that we would have the possibility to manually specify component versions (performance- or functional- tests) as job parameters. Currently these builds run automatically, once a day. I think we could run them manually when needed and provide a version of the image that was built + matching component versions. Is there any other way we could (performance) test the built image against a known release?

I have opened a review with the Jenkinsfile and pushed it to the repo, please feel free to join.

Best,
Jakub

To view this discussion on the web visit https://groups.google.com/d/msgid/openlmis-dev/71f45415-ff46-e25c-6d22-681504f42cf7%40soldevelo.com.

For more options, visit https://groups.google.com/d/optout.

--

Jakub Kondrat
Software Developer
jkon...@soldevelo.com