External Artifact Storage

[Carlos Sanchez]

Hi there, 

With the goal to make Jenkins more "cloud-friendly" or "cloud-native" I have created a JEP draft to enable the storage of artifacts outside the Jenkins home, ready for review

https://github.com/carlossg/jep/tree/jep-submission/jep/0000

TL;DR

Adding an ArtifactManager implementation (with required core changes) that transparently stores artifacts in a external blob store / artifact repository. First implementation using AWS S3 

Please share your feedback

Thanks!

[Jesse Glick]

+1

[Oleg Nenashev]

Looks good to me **as a DRAFT**.
Please propose a pull request so that JEP editors can review it.

We also need somebody to be selected as BDFL Delegate if KK is not available to review it.

BR, Oleg


On Friday, April 13, 2018 at 4:25:38 AM UTC+2, Jesse Glick wrote:

+1

[Carlos Sanchez]

PR submitted at https://github.com/jenkinsci/jep/pull/83

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/0851cc7d-d3cc-4da9-8a76-b841817e38da%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Baptiste Mathus]

Posted a small comment/request for clarification on the merged PR, but otherwise LGTM.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CALHFn6P6vQxzVXmLLQnff7CFeOTYbC6aF7OmM-KVBM%2BXwGPbjg%40mail.gmail.com.

[R. Tyler Croy]

(replies inline)

On Fri, 13 Apr 2018, Carlos Sanchez wrote:

> PR submitted at https://github.com/jenkinsci/jep/pull/83

Related to this work is this pull request which implements extensions to
VirtualFile, which I would consider very key to this work:
https://github.com/jenkinsci/jenkins/pull/3302

With that pull request there have been some questions raised about how the JEP
Workflow interacts with the implementation workflow, and merging of code into
plugin and core repositories. I wanted to share the expectations I had in mind
when bringing over parts of PEP into JEP-1.

The workflow for a Standards JEP is generally going to be Draft->Accepted[0]->Final[1]

JEP-1 is somewhat intentionally vague on how these map to merges of pull
requests and APIs. My original thoughts around this were that this would be a
good thing and allow us the most flexibility in writing code alongside
discussing design and APIs, depending on the repository and design. I recognize
how this ambiguity also leaves room for confusion.


Working backwards, Final is the most clear and obvious in my opinion. A JEP
marked as Final means designs and APIs are merged, finalized, and considered
"supported" for whatever our hand-wavey-API-support-policy is (i.e. public
APIs) in whatever project they're being merged.

The requirements for implementation around Accepted[0] much more loosely
defined as:

"The proposed implementation, if applicable, must be solid, must not complicate
Jenkins unduly, and must be the same license as the component the proposal is
meant to added to"

In the case of Jenkins Essentials, we're merging code regardless of Accepted,
or even Draft JEP status, because obviously, that's most expedient given the
state of that project.

In the case of Jenkins plugins and core which are already being used, I
understand that is not the case. However, I think it would be a failure for JEP
if we are reluctant to merge code and make _use_ of it before the "Accepted" or
"Final" states are reached.

To a certain extent I believe that no API or design can be safely considered
Final without real world experimental or beta usage. Hiding something behind a
feature flag, or a @Beta annotation, for core or plugins is (IMHO) a really
good thing to strive for even in the Draft stage to get _real_ usage of designs
in the hands of testers and users. All the while, still communicating our
intention that these are *beta* (not in the Google sense).

The quote "No plan survives contact with the enemy", and my favorite Mike Tyson
derivative "Everybody has a plan until they get punched in the mouth" come to
mind :)


With regards to the pull request for Jenkins core above, my recommendation for
the folks working on core is to merge and ship designed, documented, and
clearly marked Beta APIs which do not unduly complicate Jenkins, and of course
can be safely released and exercised in weekly and LTS releases. I think
plugins should be encouraged to follow similar guidelines at their
maintainers' discretion.


Optimizing for safely trying new things in released versions of code is IMHO a
very good thing.



Cheers



[0] https://github.com/jenkinsci/jep/tree/master/jep/1#accepted
[1] https://github.com/jenkinsci/jep/tree/master/jep/1#finalizing-a-jep

[Kohsuke Kawaguchi]

This is a great effort that tackles a part of a broad problem space for large scale / cloud native Jenkins users. My hats off to Carlos, Jesse, and Oleg for driving these efforts. I've assigned Jesse as BDFL Delegate to this JEP.

For the future, a few nit pickings. I feel that the write-up of this JEP mixes the core part and S3 specific part in one piece, which muddies water a bit. For example, it reads "Core APIs already existed ..., but lacked the crucial capability ... to provide a satisfactory S3 implementation," which makes it sound like this change was needed to make S3 work, but really AFAIK the core changes are aiming at all cloud blob stores, not just S3. I also know Carlos and Jesse are focused on doing S3 as companion implementation of this core change, but I also believe we welcome (and in fact encourage) other people to do other cloud implementations, like Azure, and there's nothing in the core changes that tie us to S3.

I also think the amount of coding effort that went into this is well past the criteria that I expect for "Accepted", if that's what led to Tyler's commentary below, but I'm probably missing some context.

So overall, for future JEPs in this space, it'd be good to get a JEP up a little sooner so that we can overlap the time it takes for the actual development and the consensus building.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/20180420163747.GF1836%40grape.lasagna.io.

For more options, visit https://groups.google.com/d/optout.

--

Kohsuke Kawaguchi

[Liam Newman]

Re: Accepted status: 

I have a PR 76 pending merge that (among other things) clarifies the meaning of "Accepted" status.  

One of the key points is that review and acceptance should not be taken lightly.

By marking a JEP as "Accepted" the Reviewer indicates they believe that the JEP has 

clear scope, design completeness, community consensus, and (if applicable) 

in-progress implementation.  Without all of these a JEP cannot be accepted. 

For this reason, it is not unusual for JEPs to remain in "Draft" state even after 

they have strong community support and progressing implementation. They 

must still pass the other criteria, such as scoping and design completeness."

I hope this PR answers people's questions.  If it doesn't, I'd like to hear from them how to improve it.

Thanks,

-L.