Alternative directory implementation of ArtifactManager?

[Staffan Forsell]

Hello,

I have been looking for a way moving the artifacts out of the builds directory and have been looking at the ArtifactManager. 

Is there a reason that I can't find an implementation of this that just uses a different local directory on the master?

Would should an plugin be hard to implement and how would one start?

Is there any obvious reasons that just creating a system property that switches the artifacts to a different dir would not work?

Our use-case is that we create a lot of artifacts (zips/vms etc) that we want to save for testing. We do this by changing the whole buildsDir to a glusterfs. This gives us massive amounts of space but the less than ideal performance with many small files, so we would like to separate the rest of the jobs and build files from the artifacts.

Best regards Staffan

[Markus Winter]

Implementation is quite easy I would say.
You need an ArtifactManager (look at the StandardArtifactManager, you can probably just copy and replace the getArtifactsDir() method)
And an ArtifactManagerFactory where you can configure the root path for the artifacts

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/00cbdc00-ed36-4a4d-9187-4af486e44822%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Daniel Beck]

> On 30. Nov 2018, at 10:40, Staffan Forsell <sta...@gmail.com> wrote:
>
> Is there any obvious reasons that just creating a system property that switches the artifacts to a different dir would not work?

Given how many regressions the custom builds dir introduced due to moving jobs between folders, renaming them, etc., and all those actions newly having to be mirrored in a different structure, I'm not surprised nobody bothered -- just having custom builds dir is good enough in many cases. Would not recommend attempting it with using job names for the lookup.

Would look at artifact-manager-s3 as a sort of template for custom implementation for artifact storage.

[Staffan Forsell]

artifact-manager-s3 seems to contain examples of most of the things needed. 

I did not quite understand...do you recommend against trying to do this or something else? If we would implement the same tests and use the same key as artifact-manager-s3, I would thing it would not be unreasonable work?

I figure if there is infrastructure now for this to work on s3 or azure, then just mapping it to a different directory should be pretty simple in my mind.

Another way of doing this would be to just invest in lots of SSD storage for our jenkins servers... Might be cheaper if this would prove to be very complicated.

[Jesse Glick]

On Fri, Nov 30, 2018 at 8:59 AM Staffan Forsell <sta...@gmail.com> wrote:
> Another way of doing this would be to just invest in lots of SSD storage for our jenkins servers.

Or just changing your projects to deploy artifacts to a proper
repository manager. The `archiveArtifacts` step and its kin is
intended only as a simple built-in way to retain smallish files if you
have not yet bothered to set up a real system.

[Staffan Forsell]

Well, we have lots of jobs, both freestyle and different styles of the new pipeline, not all for sw that's currently in active development but we still need to know we can build these.
We would rather invest our time in something that's useful for all jobs without changing all those old job definitions.
Also everything that we keep for a while on Jenkins as an artifact is not necessarily something that we would deploy to artifactory. Coupling the artifact retention to the number of builds in Jenkins is often easier than having a "works for all cleanup policy" in artifactory.

[Jesse Glick]

On Fri, Nov 30, 2018 at 8:59 AM Staffan Forsell <sta...@gmail.com> wrote:

> artifact-manager-s3 seems to contain examples of most of the things needed.

Too complicated for your purpose. Creating a plugin with a lightly
edited clone of `StandardArtifactManager` would suffice. (Use the
Maven archetype for a plugin with global configuration, so the user
can pick the location in the GUI or using configuration-as-code.)

[Staffan Forsell]

I started and got the basic function and configuration working.

How are moves and renames handled? E.g. if a admin moves a job to a folder? How does the ArtifactManager handle this?

I was thinking of saving the redirected artifact build dir in the build.xml in order to be able to keep the artifacts working in case of a move.

The alternative is to move the artifacts so they match the new build dir structure in the alternative location, but I don't know where I would do this...

I found StashAwareArtifactManager.copyAllArtifactsAndStashes() from the s3-plugn implementation, but this implies handling stashes also which is not in the scope for my ArtifactManager.

Is there a correct way of handling artifacts when jobs are moved?

/S

[Jesse Glick]

On Tue, Dec 4, 2018 at 3:48 AM Staffan Forsell <sta...@gmail.com> wrote:
> How are moves and renames handled? E.g. if a admin moves a job to a folder? How does the ArtifactManager handle this?

For `StandardArtifactManager`, that would be implicit in the job
directory itself moving. For others, including `artifact-manager-s3`,
this is not handled automatically. `JCloudsArtifactManager.key` could
perhaps be made non-`transient` to track the originally stored
location, or there could be an `ItemListener.onLocationChanged` to
physically relocate the blob(s), or the `unique-id` plugin could be
relied on for another key which persists across moves. For now we
decided to keep things simple and transparent—there is a predictable
blob location based on what you see in Jenkins—and just punt on the
use case of historical builds preceding a move/rename.

(On a related note, by default `artifact-manager-s3` declines to
remove artifacts after build deletion in Jenkins, on the grounds that
AWS retention policies and Glacier migration are likely better, and we
want the Jenkins master to require minimal IAM permissions: a
compromised master should not be able to destroy or alter cloud data.)

[Staffan Forsell]

Well for us such retention policys does not apply and the time to keep artifacts is better handled though jenkins since it created them, than by other time based policies. We typically keep a number of artifacts per job while some builds are kept forever, typically release builds for traceability.

Well now I got most of it working but I discovered in my unit tests that deleting the job does not remove the artifacts, but removing the build does.

It seems that removing the job only recursively removed the job directory, see hudson.model.Job#delete.

To me this seems like a bug, it makes it harder to clean clean artifacts for a custom ArtifactsManager.

I managed to get around it using ItemListener.onDelete() to call run.deleteArtifacts() but I kind of think that this should be jenkins own responsibility while deleting a job.

Should I file an issue or do you have an other opinion?

[Jesse Glick]

On Wed, Dec 5, 2018 at 4:33 PM Staffan Forsell <sta...@gmail.com> wrote:
> I managed to get around it using ItemListener.onDelete() to call run.deleteArtifacts() but I kind of think that this should be jenkins own responsibility while deleting a job.

Perhaps. `Job.delete` could call `ArtifactManager.delete` on every
`Run`. But this would be inefficient for `StandardArtifactManager`
(much slower than just deleting the whole job directory), and useless
overhead for `JCloudsArtifactManager` (which would typically not
delete anything). For now I would just be inclined to leave the
`ItemListener` workaround as is.