Install default java apps - possible?

[jer...@saasindustries.com]

So I have managed to get my IaC to setup all the namespaces. For example:

app

 - app-testing

 - app-staging

 - app-production

dev

 - dev-testing

 - dev-staging

And I have things like "management" being installed to the "app" namespace. And for each namespace intended for environments, I have default runtimes like "messaging" being installed into there. It is working quite nicely and I can automate everything as far as runtimes go. Awesomesauce!

Now with that out of the way, I would like to be able to install the base application through IaC.

So for example, if I have 30 microservices, can I install those remotely as well?

So the goal would be to have the configuration ship with the application just like how the runtimes do, except with the java apps as an example, I need it to be put into gogs, jenkins and etc so it can start the build process and setup the environments. I already have the environment namespaces setup, so I wonder if it is possible to get my apps in there as well without having to do it manually through the UI. I figure there is a main entry point when you load something from git, so is that exposed to be able to accomplish the same with automation?

If not, this would be an awesome improvement to be able to orchestrate a complete application using IaC. This is especially useful when migrating from one cluster to the next when you want to resize node VMs and a bunch of other things. It would really help to make things predictable, reliable and with high quality.

[James Strachan]

On 24 November 2016 at 07:59, <jer...@saasindustries.com> wrote:

So I have managed to get my IaC to setup all the namespaces. For example:

app

 - app-testing

 - app-staging

 - app-production

dev

 - dev-testing

 - dev-staging

And I have things like "management" being installed to the "app" namespace. And for each namespace intended for environments, I have default runtimes like "messaging" being installed into there. It is working quite nicely and I can automate everything as far as runtimes go. Awesomesauce!

yay!

 

Now with that out of the way, I would like to be able to install the base application through IaC.

So for example, if I have 30 microservices, can I install those remotely as well?

Wouldn't a CD pipeline do that for each version of each microservice? e.g. create a microservice in some team (app, dev or foo) and then have the pipeline perform the testing & roll out to staging + prod?

I guess a question is "where are the 30 microservices" :) It depends where the are as to how best to install them - more on this later...

So the goal would be to have the configuration ship with the application just like how the runtimes do, except with the java apps as an example, I need it to be put into gogs, jenkins and etc so it can start the build process and setup the environments. I already have the environment namespaces setup, so I wonder if it is possible to get my apps in there as well without having to do it manually through the UI. I figure there is a main entry point when you load something from git, so is that exposed to be able to accomplish the same with automation?

So given a team - say "app" - you just need to create/import the microservice into the development namespace for that team.  (e.g. in the app namespace). You can do that via the fabric8 console; or via the "mvn fabric8:import" maven goal if you have a maven project.

On OpenShift an 'app' is a BuildConfig resource (i.e. its a way to build stuff); on Kubernetes its just a Jenkins Job definition; then its the Jenkinsfile which defines how to perform the build, how to test and move it through staging & production. To map the logical environments (test, staging, prod) to physical namespaces (and/or separate kubernetes clusters) we use the fabric8-environments ConfigMap in the development namespace to do the mapping. This lets us have reuable pipelines that know about dev/test/staging/prod but don't need to be editted for each team which uses different physical namespaces.

 

If not, this would be an awesome improvement to be able to orchestrate a complete application using IaC. This is especially useful when migrating from one cluster to the next when you want to resize node VMs and a bunch of other things. It would really help to make things predictable, reliable and with high quality.

So another option for importing projects into a development namespace - (or another way of saying that is - importing jobs into a teams jenkins master) - is to use jenkins plugins.

There's some docs here on discovering jobs:

http://fabric8.io/guide/cdelivery.html#discovering-jenkins-jobs

right now the mulitbranch plugin only works for a single repo at a time so is only useful for branches on a single microservice. The github organisation plugin only works for github & works at the organisation level. So if you are using github for the source code of your Jenkinsfile and microservices; you can import a github organisation per team and it'll auto-create all the Jenkins jobs (and hence apps, pipelines and releases). i.e. it'll import all the microservices for a given organisation/team in one go.

The missing thing is supporting other git providers.

As soon as we've got SSO working with gitlab we'll probably switch over to use that by default; using an Organisation in gitlab for each 'team' in fabric8.

http://fabric8.io/guide/terminology.html

Once we're there we could look to create a gitlab organisation plugin for jenkins - which works like the github one but using gitlab. Then we could auto-import all the projects in an organisation in gitlab? (We could even support auto-creating teams in kubernetes/openshift for each organisation in gitlab too?)

Then we can automate creating the entire kubernetes cluster from scatch - assuming a populated gitlab. But then if you're starting from scratch and you run gitlab on kubernetes; you'd have an empty gitlab; so this strategy would only work if you've persistence in your gitlab - or gitlab is external (like github is).

--

James
-------
Red Hat

Twitter: @jstrachan
Email: james.s...@gmail.com

Blog: https://medium.com/@jstrachan/

fabric8: http://fabric8.io/

open source microservices platform

[Jeremy Wilson]

Wouldn't a CD pipeline do that for each version of each microservice? e.g. create a microservice in some team (app, dev or foo) and then have the pipeline perform the testing & roll out to staging + prod?
I guess a question is "where are the 30 microservices" :) It depends where the are as to how best to install them - more on this later..

The use case I am using fabric8 for, is to create a platform which has a pre-built SaaS infrastructure in it. So things which are common among all SaaS applications like, user management, licensing & entitlement, self service, reporting and etc. So the expectation is, that I will provision a fully working SaaS application to one of our Joint Ventures fully deployed. I am trying to minimize manual steps by taking advantage of IaC and other automation. This process will be repeated dozens of times each year. Additionally, if I want to create a second cluster to replace the first cluster, I would like to have this as automated as possible to gain repeatable and predictable results. From disaster recovery all the way to business continuity.

We do plan on using Gitlab as the on-premise solution for all our code repositories. There will be a central location, and we will just automate the transfer of these components from the central location, into the new cluster's Gitlab instance.

But rather than manually using the UI to add every component, I would like to automate the process.I expect that I will have hundreds of services which will need to be provisioned when the platform hits maturity. Doing this manually for each join venture and dozens and dozens of times each year will be a pain, and we will have to rely too much on domain knowledge and documentation. The preference is obviously to automate this.

But even if I did not have the unique requirements for repeating the process over and over again, I would still want it for disaster recovery, swapping clusters for more powerful ones and etc to minimize the labor , domain knowledge and etc.

Basically, our organization is going to propagating Fabric8 into dozens of start up companies every year :)

So given a team - say "app" - you just need to create/import the microservice into the development namespace for that team.  (e.g. in the app namespace). You can do that via the fabric8 console; or via the "mvn fabric8:import" maven goal if you have a maven project.
On OpenShift an 'app' is a BuildConfig resource (i.e. its a way to build stuff); on Kubernetes its just a Jenkins Job definition; then its the Jenkinsfile which defines how to perform the build, how to test and move it through staging & production. To map the logical environments (test, staging, prod) to physical namespaces (and/or separate kubernetes clusters) we use the fabric8-environments ConfigMap in the development namespace to do the mapping. This lets us have reuable pipelines that know about dev/test/staging/prod but don't need to be editted for each team which uses different physical namespaces.

Yeah, so figure the configuration would ship with the application, I might only have to minor tweak a few things per repository in the config. But if I have hundreds of services that make up an application, I would automate transferring from the central repos into the new Join Ventures on premise Gitlab. From there, I want to load up all the repositories through automation to get them building and etc without having to manually do each one in the UI. Configuration in the repo would have the pipeline defined and everything else. Any config value that need modification, I can bulk do those in the automation before it is injected into the new repository destination.

I am in Google, and will eventually also support AWS, but I have not plans on going into OpenShift at this time.

I will look more into the mvn fabric8:import - so if that would allow me to accomplish the part of getting all of the repositories built inside the cluster instance through an automation tool, that would be great.

Thank you for the context James, I will look into your points a bit further and see where I get.

Thank you,

Jeremy Wilson
Technology
SaaS Industries
 jer...@saasindustries.com
 (480) 370-8663
 https://www.linkedin.com/in/jeremywilson
 https://twitter.com/antifragileer

Join the #YesPHX start-up community at https://yesphx.typeform.com/to/FMZoEr

Two great books to help you create a badass engineering organization
[1] Lean-Agile Software Development: Achieving Enterprise Agility
[2] The Art of Scalability

Confidentiality Statement: All incoming and outgoing email is monitored and archived. This e-mail message, and all attachments, may contain legally privileged or confidential information intended solely for the use of the individual, agent, or entity named in the e-mail. If you are not a designated recipient, you may not review, copy or distribute this message. If you receive this in error, please notify the sender by reply e-mail and delete this message. This email may be produced at the request of regulators or in connection with civil litigation. 

--
You received this message because you are subscribed to the Google Groups "fabric8" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fabric8+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.