Artifact storage reduction project

[Mark Waite]

JFrog sponsors the Jenkins artifact repository at https://repo.jenkins-ci.org.  They've asked us to reduce our storage usage from the current 9+ TB to fit within their hosting guidelines of less than 5 TB of artifacts per sponsored open source project.

Darin Pope has agreed to lead a project to reduce our artifact storage requirements.  He will work with Damien Duportal as needed to plan the project, perform experiments, run test brownouts, and resolve issues.

Darin will need administrator access to the artifact repository while he's working on this project.

Any concerns from the security team or from others if Darin is given administrator access to https://repo.jenkins-ci.org while working on the project?

Mark Waite

[Mark Waite]

Darin suggested that he first be given write access to the incrementals tree so that he can create the cleanup script that removes older incremental artifacts.  Wadeck and Damien is that OK with you?

Mark Waite

[wfoll...@cloudbees.com]

Hello Mark,

From my PoV, this access can be granted for this project. It has to be temporary as Darin is not part of the infra team. I would prefer for the next such occurrence, to have the infra team working on it, to reduce the need to grant permission to other people.

The permissions are pretty powerful in terms of potential of damage, so limiting them to only few people with the need is required.

Wadeck

[Damien Duportal]

It's a +1 for me if it is a temporary access as stated by Wadeck in the above message.

The rationale is that the infra team is busy on other tasks with the same level of priority (ci.jenkins.io migration to AWS + FOSDEM) hence Mark's proposal to have temporary help.

Darin has the skillset and rigorous behavior required for such accesses, and we'll work together on the actions.


@Mark, I'll meet with Darin today, but I believe that the most visible usage (both storage and bandwidth) is the Atlassian public mirror and it shall be the priority. We'll see the requirements around incrementals cleanup as well though so we don't loose any opportunity to show advances to JFrog.
We'll report in public issues to give everyone visibility.

Damien

[Damien Duportal]

Quick addendum as my previous message is unclear:

• Temporary access to only the incremental repo. for Darin is a good idea to limit the impact of extended permissions (e.g. the risk of a human mistake removing things). It's safer for a first step!
• The low "impact" I mentioned was around the storage and bandwidth metrics decrease if only focusing on the "incremental" purge. It might help to retrieve a bit of storage: it's a good thing and should provide quick feedback to JFrog. But it is a low imapct.
• Most of the effort should be spent on tuning the Atlassian public mirror as all of our metrics shows it's huge (both on storage and bandwidth).