Custom Distribution Service [GSoC 2020] - Hosting proposal

[Sladyn Nunes]

Hi all,

This post is regarding the hosting of the custom distribution service, considering we are reaching the conclusion of Google Summer of Code 2020, we would like to have this service hosted on a domain like jenkins.io/custom-distribution-service as was proposed in the project idea and the summer of code proposal. 

I would like to know the initial thoughts on this proposal. 

Design Doc link : https://docs.google.com/document/d/1-ujWVJ2a5VYkUF6UA7m4bEpSDxmb3mJZhCbmoKO716U/edit?usp=sharing
Github Link : https://github.com/jenkinsci/custom-distribution-service
Issue Link : https://github.com/jenkinsci/custom-distribution-service/issues/93

[Gavin Mogan]

A subdomain will be easier that taking it into the main site. The main site will need a proxy and cache exemptions and all that.

I see its a spring boot app. Does it need any other resources like a db or file system or anything? Is the docker image publishable? If so its pretty simple to setup a helm chart. Especially if you can update the docker file to be multi stage and do all the work then you can just call 

https://github.com/jenkins-infra/pipeline-library#builddockerimage in your jenkinsfile

Assuming no objections by infra people, and there's you can either copy the wiki exporter chart, or I can make one.

Gavin

--
You received this message because you are subscribed to the Google Groups "Jenkins Infrastructure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkins-infr...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/jenkins-infra/CAC-LeqvA8ccfa94GKQt5%2BKL84jGVgAEtd5mZP6EWhOP4oYmDeA%40mail.gmail.com.

[Tim Jacomb]

Agree with Gavin.

Definitely go with a sub domain.

You should be able to test your chart on minikube, see the readme of plugin site in the charts repo

Thanks

Tim

To view this discussion on the web, visit https://groups.google.com/d/msgid/jenkins-infra/CAG%3D_DuuUpTguaU1_XF998a21y6fcjYA4rUjB5eyJ9tfTRC2S4A%40mail.gmail.com.

[Olblak]

Hi

Definitely prefer a subdomain it's easier to maintain as it doesn't put dependency on the main website.

Once your have a helm chart for it, you can deploy your application on the kubernetes cluster

Step 1: Open Jira ticket on issues.jenkins-ci.org so we can keep track of the work involved here on the INFRA project

Step2: Define the subdomain you want to use and open a PR here, once it's done I'll create the DNS record

Step3: Add your helm chart to jenkins-infra/charts

To add a chart you have to open en pull request with

• Your custom helm chart under this directory, if you have a clear idea of what you need we can redirect you to the closest helm chart that you can use as an example
  
• Define a helmfile for your application here, to describe how you want to deploy your helm chart
  
• Reference your helmfile from this file to specifically request to deploy your appliication on our production k8s cluster
  

If you have any questions feel free to reach out to Tim, Gavin or me

Cheers

To view this discussion on the web, visit https://groups.google.com/d/msgid/jenkins-infra/CAH-3BifjcVfEDUJeNoST7ngMgBpD9UFB8De0L7q3hg1tM9LCTw%40mail.gmail.com.

[Sladyn Nunes]

Hi,
Thanks for all the help gentlemen, I have opened a ticket here to track the progress. https://issues.jenkins-ci.org/browse/INFRA-2700. 

> Does it need any other resources like a db or file system or anything? Is the docker image publishable? If so its pretty simple to setup a helm chart. Especially if you can update the docker file to be multi stage and do all the work then you can just call  https://github.com/jenkins-infra/pipeline-library#builddockerimage in your jenkinsfile

No it does not need a DB. I am not too sure if the docker image is publishable but the docker-compose works perfectly, is that indication that its publishable ? Any specific things I need to look for ?

> Your custom helm chart under this directory, if you have a clear idea of what you need we can redirect you to the closest helm chart that you can use as an example

Things I need, 
a) Front-end Deployment - It is a react project.
b) Back-end Deployment - It is a spring-boot service 

[Gavin Mogan]

Does the springboot app serve the react pages? or do you need 2 different services?

I would start by publishing the image to your own namespace on hub.docker.com just so we can see it working. If you do that, It looks like there's two docker images, which makes this a little tricker.

The front end looks like a CRA thing, so it should be able to be statically compiled, and just served up with nginx.

The back end needs its own container.

To simplify things, at least short term. I would recommend making sure the spring boot app can serve content from a directory. (https://spring.io/blog/2013/12/19/serving-static-web-content-with-spring-boot looks simple, but classpath always confuses me). And create an infra ticket to get a new docker repo created in jenkins4eval and jenkins-infra.

Then I can submit a PR that:

* Move the existing dockerfiles so docker-compose works

* Moves the dockerfile into a multistage one, so everything is packed into a single image

* Update the jenkinsfile to use the above image that builds and publishes dockerfiles based on release tags

* Create you a helm chart for deployment.

That's assuming you don't want to do that stuff yourself. But i'm happy to do it, and answer any questions on the PR.

Gavin

To view this discussion on the web, visit https://groups.google.com/d/msgid/jenkins-infra/72c335b9-eb2c-4387-806f-a9f56ff96008n%40googlegroups.com.

[Olblak]

Thanks Gavin for the follow up.

Just to add some information

No it does not need a DB. I am not too sure if the docker image is publishable but the docker-compose works perfectly, is that indication that its publishable ? Any specific things I need to look for ?

The rule is, is your container contains any secrets like an API token or whatever? Then It's definitely a bad idea to publish it publicly. Same if your containers contains licensed code or a database that forbid you to share without violating a license.

Otherwise in any other situation we just publish the docker image on Dockerhub, everything related to infrastructure is available in the jenkinsciinfra organization.

Cheers

To view this discussion on the web, visit https://groups.google.com/d/msgid/jenkins-infra/CAG%3D_DusCLZA_cF%3Dax%3DU_VX3iWOSBN80Vxy-NSk5MyHUXkXZf%3DA%40mail.gmail.com.

[Sladyn Nunes]

Hi all, 

> Then I can submit a PR that:

    * Move the existing dockerfiles so docker-compose works

   * Moves the dockerfile into a multistage one, so everything is packed into a single image

   * Update the jenkinsfile to use the above image that builds and publishes dockerfiles based on release tags

   * Create you a helm chart for deployment.

Thanks Gavin for the help, this would help me a lot.

> To simplify things, at least short term. I would recommend making sure the spring boot app can serve content from a directory. (https://spring.io/blog/2013/12/19/serving-static-web-content-with-spring-boot looks simple, but classpath always confuses me)

I read this article and I guess Spring Boot comes with a pre-configured implementation of ResourceHttpRequestHandler to facilitate serving static resources, this is by default enables in a spring boot starter pack. So I guess it does. Is there any way I can confirm it.Thanks

>  And create an infra ticket to get a new docker repo created in jenkins4eval and jenkins-infra.

https://issues.jenkins-ci.org/browse/INFRA-2704: Created a ticket for the docker repo hosting

> The rule is, is your container contains any secrets like an API token or whatever? Then It's definitely a bad idea to publish it publicly. Same if your containers contains licensed code or a database that forbid you to share without violating a license.

Yeah it does not violate any of the above,so its safe for public publishing.

[Gavin Mogan]

> So I guess it does. Is there any way I can confirm it.Thanks

Put some files in the public directory (like foobar.txt), then access your service http://server/foobar.txt

if the file contents get returned, then we are golden, just make sure I'm aware of that path, and I'll try to do the rest this weekend

Gavin

To view this discussion on the web, visit https://groups.google.com/d/msgid/jenkins-infra/a6d22b12-3c1c-4fb1-b32c-d4be62f34397n%40googlegroups.com.

[Olblak]

>  And create an infra ticket to get a new docker repo created in jenkins4eval and jenkins-infra.

https://issues.jenkins-ci.org/browse/INFRA-2704: Created a ticket for the docker repo hosting

It would be nice to have the name you want to use for the docker image :)

To view this discussion on the web, visit https://groups.google.com/d/msgid/jenkins-infra/CAG%3D_DuvjLuCYP3kLavwtqutij%3DaE_ca39h_vokqKK%3D11z5n7bQ%40mail.gmail.com.

[Sladyn Nunes]

> Put some files in the public directory (like foobar.txt), then access your service http://server/foobar.txt

if the file contents get returned, then we are golden, just make sure I'm aware of that path, and I'll try to do the rest this weekend

Yeah I did this and it works like a charm.

[Gavin Mogan]

> just make sure i'm aware of the path

Is it /app/public? (looking at /app in https://github.com/jenkinsci/custom-distribution-service/blob/master/Dockerfile#L28)

To view this discussion on the web, visit https://groups.google.com/d/msgid/jenkins-infra/470e8a09-5b97-4b3a-a9ef-6b80bd53847bn%40googlegroups.com.

[Sladyn Nunes]

Yeah all files from the public directory are being served and since the WORKDIR is /app/ that should be the path.

[Gavin Mogan]

> It would be nice to have the name you want to use for the docker image :)

So as Olblak mentioned a couple days ago, still need to know the docker image name you want created.

https://issues.jenkins-ci.org/browse/INFRA-2704

I do have a working production ready docker image ready, just need to know the name (above), and a few PRs to land in infra to make this easy and simple

[Gavin Mogan]

Once https://github.com/jenkinsci/custom-distribution-service/pull/131 and https://github.com/jenkins-infra/charts/pull/393 are both merged, we should be able to have docker images built.

latest will always be latest master commit

tagname should be releases

Once that is done, i'll work on a helm chart for deployment.

You received this message because you are subscribed to a topic in the Google Groups "Jenkins Infrastructure" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkins-infra/v3UJfiFte8w/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkins-infr...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/jenkins-infra/b3bdda26-8388-42da-8476-2262cb290bb9n%40googlegroups.com.

[Sladyn Nunes]

Thanks Gavin, do let me know if you need anything else for this to go through. Aiming to include it in my phase 3 report if all goes well. 
Thanks again. 

[Gavin Mogan]

I've been working on the infra to make docker images easily

I more or less finished it today, just polishing up the final docker glue.

That being said sometime since we merged and now, your docker image doesn't build

docker build -f Dockerfile.infra .

gives me

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-pmd-plugin:3.13.0:pmd (pmd) on project custom-distribution-service: Execution pmd of goal org.apache.maven.plugins:maven-pmd-plugin:3.13.0:pmd failed: org.apache.maven.reporting.MavenReportException: Could not find resource '/config/pmd.xml'. -> [Help 1]

i'm thinking you want ./config/pmd.xml (relative to current directory) not /config/pmd.xml (root)

Gavin

To view this discussion on the web, visit https://groups.google.com/d/msgid/jenkins-infra/cc922fd0-4fe2-47e8-94cc-87d3d32f6b0en%40googlegroups.com.

[Sladyn Nunes]

Hi,

This has now been fixed right ? 

[Gavin Mogan]

Yea I submitted a PR that got it fixed.

just FYI, its been asked like a bunch of times now, but olblak's original email:

Step 1: Open Jira ticket on issues.jenkins-ci.org so we can keep track of the work involved here on the INFRA project

Step 2: Define the subdomain you want to use and open a PR here, once it's done I'll create the DNS record

there's no domain name requested or setup yet.

To view this discussion on the web, visit https://groups.google.com/d/msgid/jenkins-infra/42a65273-42f9-47f3-a5a0-b31dde565fe3n%40googlegroups.com.

[Sladyn Nunes]

Step 1: This step was done quite a while ago : https://issues.jenkins-ci.org/browse/INFRA-2700
Step 2 : I created a PR here https://github.com/jenkins-infra/azure/pull/154

[Sladyn Nunes]

Thanks for all the help on this thread, I really appreciate it.
This PR has been merged https://github.com/jenkins-infra/azure/pull/154
Are there any more action items I need to complete ? 
Thanks.

[Gavin Mogan]

I still would like a tagged release but I can work around it.

Is there any env variables I need to be aware of or set? Even if they are defaulted right it might be worth me doing one so it's easy to see how to change/add them later.

Gavin

To view this discussion on the web, visit https://groups.google.com/d/msgid/jenkins-infra/d49954d9-9c67-4b90-93ea-ee2bbcd88d87n%40googlegroups.com.

[Sladyn Nunes]

I guess this is the only one https://github.com/jenkinsci/custom-distribution-service/blob/master/frontend/.env .

For this one we already have added the required env variables https://github.com/jenkinsci/custom-distribution-service/pull/88

[Gavin Mogan]

https://customize.jenkins.io/ is running though the javascript is not loading, hopefully https://github.com/jenkinsci/custom-distribution-service/pull/150 will fix the problem.

The last steps for me are to setup the auto PR generator on new releases, but I'm waiting for a github tag + release to happen before i set that up.

To view this discussion on the web, visit https://groups.google.com/d/msgid/jenkins-infra/d5c13ea7-5939-4727-b06e-ea5ee3af19c6n%40googlegroups.com.