Multi-platform binaries in a single job
34 views
Skip to first unread message
Marcus MacWilliam
Oct 3, 2016, 12:08:58 PM10/3/16
to Jenkins Users
Current system
To better understand my requirements, I need to first give an overview of our current CI build system,
and why it is too cumbersome.
Our company has its own legacy build environments for our core products, which has worked for over
30 years, and we do not have the resources to re-write the mechanism. It is based 'roughly' on make,
but uses our own set of shell and python scripts, to setup the environment for an individual platform.
Our current CI builds for our artefacts are spread over 4 separate jobs:
1. Configuration - a very simple project containing a single resource file, used by the other three jobs.
The file defines the flavour and the base packs of the software. For us a pack defines a set of modules,
what flavour they are in, what version to use, and some build options. The CI job just puts this file into
a ZIP file artefact, and posts to our Nexus repository.
Note: run on a Linux node.
2. Linux binaries - a Maven project which uses the ZIP file above, sets up the development environment,
inside the workspace/target directory, pulls all source code for the modules from git, then builds the Linux
binaries.
Note: run on a Linux node.
3. Windows binaries - a Maven project which uses the ZIP file above, sets up the development environment,
inside the workspace\target directory, pulls all source code for the modules from git, then builds the Windows
binaries.
Note: run on a Windows node.
4. Java binaries - a Maven project which uses the ZIP file above, sets up the development environment,
inside the workspace/target directory, pulls all source code for the modules from git, then builds the Java
binaries.
Note: run on a Linux node.
All 4 jobs, have their own script repository, 1 for each of the 4 CI build jobs.
Jobs 2-4 above, all start by doing identical setup:
i. ZIP file unpacked to workspace/target
ii. retrieve clone of the git repository for the development environment setup
iii. set location (environment variables) of root, data root, Java root and binaries, relative to the workspace\target
iv. run configuration (python) scripts, to create all required directories for the above
v. invoke environment setup script, which determines the compiler, and other 3rd party tools
vi. invoke generate-environment script, which uses the configuration file, and generates files required by the build tools
vii. invoke setup-environment script, which invokes the scripts generated by the previous stage
viii. invoke specific command to build binaries
ix. invoke specific commands to assemble artefacts
x. deploy artefacts to Nexus repository
Steps 1-7 above, take the majority of the time, and get the shell into the correct state so that 8-10 will deploy
the correct artefact to our Nexus repository.
Required System
What I would like to do is the following: create a single build repository of scripts, and a single
CI project, with multiple nodes defined, to build the artefacts.
At this time I would imagine I need to use a shared workspace, and some type of multi-configuration
plug-in. I am completely new to Jenkins, so do not know how to set this up, and would appreciate any
pointers on how to do this.
Our development environment already has a mechanism built in for multiple platforms to share
the same source code, but build binaries in platform dependent locations. So this would be safe for us.
When the CI job is invoked it will do the following:
Note: Do not know yet whether this would be a Maven project, which invokes the scripts, or a Freestyle
project to invoke the scripts.
Master Node (start)
1. Clean the old workspace (I think there is a plug-in which adds this as an option to the job configuration)
2. Retrieve the GiT repository from BitBucket (using the plug-in)
3. Sets up a shared workspace\target which does steps ii. to vi.
- instead of unpacking a ZIP file, this repository would have the configuration file
ii. retrieve clone of the git repository for the development environment setup
iii. set location (environment variables) of root, data root, Java root and binaries, relative to the workspace\target
iv. run configuration (python) scripts, to create all required directories for the above
v. invoke environment setup script, which determines the compiler, and other 3rd party tools
vi. invoke generate-environment script, which uses the configuration file, and generates files required by the build tools
Linux Node (parallel)
1. Using the shared workspace from the Master node
vii. invoke setup-environment script, which invokes the scripts generated by the previous stage
viii. invoke specific command to build binaries (Linux and Java)
ix. invoke specific commands to assemble artefacts
Windows Node (parallel)
1. Using the shared workspace from the Master node
vii. invoke setup-environment script, which invokes the scripts generated by the previous stage
viii. invoke specific command to build binaries (Linux and Java)
ix. invoke specific commands to assemble artefacts
Master Node (complete)
1. Publish all of the artefacts from the 2 build processes above, to our Nexus repository.
Note: This will only run if the 2 parallel nodes for binaries complete with no errors.
Any help on setting up the CI job, or GiT repository to enable us to do this, would be greatly
appreciated. I may need specific help with what plug-ins to install for Jenkins, and how to setup
the parameters for these plug-ins, in the CI job.
Note: It is also possible that the best approach to achieve this would be a pipeline, however I am
less familiar with this approach to Continuous Integration.
Yours thankfully,
Marcus MacWilliam, Principal Software Engineer,
1Spatial Ltd, Cambridge, UK.
To better understand my requirements, I need to first give an overview of our current CI build system,
and why it is too cumbersome.
Our company has its own legacy build environments for our core products, which has worked for over
30 years, and we do not have the resources to re-write the mechanism. It is based 'roughly' on make,
but uses our own set of shell and python scripts, to setup the environment for an individual platform.
Our current CI builds for our artefacts are spread over 4 separate jobs:
1. Configuration - a very simple project containing a single resource file, used by the other three jobs.
The file defines the flavour and the base packs of the software. For us a pack defines a set of modules,
what flavour they are in, what version to use, and some build options. The CI job just puts this file into
a ZIP file artefact, and posts to our Nexus repository.
Note: run on a Linux node.
2. Linux binaries - a Maven project which uses the ZIP file above, sets up the development environment,
inside the workspace/target directory, pulls all source code for the modules from git, then builds the Linux
binaries.
Note: run on a Linux node.
3. Windows binaries - a Maven project which uses the ZIP file above, sets up the development environment,
inside the workspace\target directory, pulls all source code for the modules from git, then builds the Windows
binaries.
Note: run on a Windows node.
4. Java binaries - a Maven project which uses the ZIP file above, sets up the development environment,
inside the workspace/target directory, pulls all source code for the modules from git, then builds the Java
binaries.
Note: run on a Linux node.
All 4 jobs, have their own script repository, 1 for each of the 4 CI build jobs.
Jobs 2-4 above, all start by doing identical setup:
i. ZIP file unpacked to workspace/target
ii. retrieve clone of the git repository for the development environment setup
iii. set location (environment variables) of root, data root, Java root and binaries, relative to the workspace\target
iv. run configuration (python) scripts, to create all required directories for the above
v. invoke environment setup script, which determines the compiler, and other 3rd party tools
vi. invoke generate-environment script, which uses the configuration file, and generates files required by the build tools
vii. invoke setup-environment script, which invokes the scripts generated by the previous stage
viii. invoke specific command to build binaries
ix. invoke specific commands to assemble artefacts
x. deploy artefacts to Nexus repository
Steps 1-7 above, take the majority of the time, and get the shell into the correct state so that 8-10 will deploy
the correct artefact to our Nexus repository.
Required System
What I would like to do is the following: create a single build repository of scripts, and a single
CI project, with multiple nodes defined, to build the artefacts.
At this time I would imagine I need to use a shared workspace, and some type of multi-configuration
plug-in. I am completely new to Jenkins, so do not know how to set this up, and would appreciate any
pointers on how to do this.
Our development environment already has a mechanism built in for multiple platforms to share
the same source code, but build binaries in platform dependent locations. So this would be safe for us.
When the CI job is invoked it will do the following:
Note: Do not know yet whether this would be a Maven project, which invokes the scripts, or a Freestyle
project to invoke the scripts.
Master Node (start)
1. Clean the old workspace (I think there is a plug-in which adds this as an option to the job configuration)
2. Retrieve the GiT repository from BitBucket (using the plug-in)
3. Sets up a shared workspace\target which does steps ii. to vi.
- instead of unpacking a ZIP file, this repository would have the configuration file
ii. retrieve clone of the git repository for the development environment setup
iii. set location (environment variables) of root, data root, Java root and binaries, relative to the workspace\target
iv. run configuration (python) scripts, to create all required directories for the above
v. invoke environment setup script, which determines the compiler, and other 3rd party tools
vi. invoke generate-environment script, which uses the configuration file, and generates files required by the build tools
Linux Node (parallel)
1. Using the shared workspace from the Master node
vii. invoke setup-environment script, which invokes the scripts generated by the previous stage
viii. invoke specific command to build binaries (Linux and Java)
ix. invoke specific commands to assemble artefacts
Windows Node (parallel)
1. Using the shared workspace from the Master node
vii. invoke setup-environment script, which invokes the scripts generated by the previous stage
viii. invoke specific command to build binaries (Linux and Java)
ix. invoke specific commands to assemble artefacts
Master Node (complete)
1. Publish all of the artefacts from the 2 build processes above, to our Nexus repository.
Note: This will only run if the 2 parallel nodes for binaries complete with no errors.
Any help on setting up the CI job, or GiT repository to enable us to do this, would be greatly
appreciated. I may need specific help with what plug-ins to install for Jenkins, and how to setup
the parameters for these plug-ins, in the CI job.
Note: It is also possible that the best approach to achieve this would be a pipeline, however I am
less familiar with this approach to Continuous Integration.
Yours thankfully,
Marcus MacWilliam, Principal Software Engineer,
1Spatial Ltd, Cambridge, UK.
Daniel Beck
Oct 3, 2016, 3:55:05 PM10/3/16
to Jenkins Users
> On 03.10.2016, at 18:08, Marcus MacWilliam <marcus.m...@gmail.com> wrote:
>
> I am completely new to Jenkins, so do not know how to set this up, and would appreciate any
> pointers on how to do this.
More: https://jenkins.io/doc/pipeline/
Reply all
Reply to author
Forward
0 new messages