Proposal: AWS Secrets Manager Credentials Provider plugin

235 views
Skip to first unread message

Chris Kilding

unread,
Apr 4, 2019, 10:26:48 AM4/4/19
to jenkin...@googlegroups.com
Hello,

My company has a need to load CI secrets (e.g. API tokens and machine user passwords) from AWS Secrets Manager. We used to do this with hacky Shell scripts but wanted something better. We looked around for a Jenkins Credentials Provider plugin to do this, but we could not find one, so we’ve written one.

We’re happy to share this with the community under an MIT license (or something similar) - indeed this would be the easiest route for other teams across our company to adopt it in their Jenkins installations.

We’re interested in feedback for where this proposed plugin could or should live, and - if it sounds relevant to your interests - what kind of features you’d like to see it support.

Its current features are:

- Read-only view of Secrets Manager Secrets.
- Credential metadata caching (duration: 5 minutes).
- Jenkins Configuration as Code support.

Its optional configuration parameters are:

- Filter AWS Secrets by tag
- Customise AWS endpoint configuration (useful for local testing)

We’re initially thinking it should be a Github repo under the ‘jenkinsci’ or ‘aws’ organisations, with our own engineers added to that repo as external collaborators. (These would seem to be the most natural homes for the plugin.) But we’re open to other suggestions :)

Regards,

Chris Kilding

Jesse Glick

unread,
Apr 4, 2019, 11:30:02 AM4/4/19
to Jenkins Dev
On Thu, Apr 4, 2019 at 10:26 AM Chris Kilding
<chris+...@chriskilding.com> wrote:
> We looked around for a Jenkins Credentials Provider plugin to do this, but we could not find one, so we’ve written one.

Great! This has been missing for a while.

> what kind of features you’d like to see it support

Without knowing much about this AWS feature, I would say that at a
high level it should work similarly to the
`kubernetes-credentials-provider` plugin which make K8s `Secret`s
available as Jenkins credentials. That plugin uses annotations to
allow Jenkins-specific bindings to be made without needing any
particular configuration on the Jenkins side, making it easier to tear
down and recreate a Jenkins service.

> We’re initially thinking it should be a Github repo under the ‘jenkinsci’ or ‘aws’ organisations

It would make sense under @jenkinsci I think.
https://wiki.jenkins.io/display/JENKINS/Hosting+Plugins

Regarding relationships to existing Jenkins plugins, you can integrate
with `aws-global-configuration` if some Jenkins configuration is
necessary. Check whether there is some overlap with
`aws-parameter-store`. I believe it is independent of
`aws-credentials`, and likely no integration would be needed since for
this use case you would normally just use an EC2 instance account if I
get it right.

Daniel Beck

unread,
Apr 4, 2019, 11:51:05 AM4/4/19
to jenkin...@googlegroups.com
Can't wait to check this out. Thanks for publishing it!

> On 4. Apr 2019, at 16:21, Chris Kilding <chris+...@chriskilding.com> wrote:
>
> We’re initially thinking it should be a Github repo under the ‘jenkinsci’ or ‘aws’ organisations, with our own engineers added to that repo as external collaborators. (These would seem to be the most natural homes for the plugin.) But we’re open to other suggestions :)

We require that plugins distributed by the Jenkins project be hosted in the jenkinsci organization (with some exceptions grandfathered in). So if you want to make it as easy as possible to install with the built-in plugin manager, I would recommend you host in jenkinsci.

Docs for this (these largely superseded the wiki): https://jenkins.io/doc/developer/publishing/requesting-hosting/

Chris Kilding

unread,
Apr 4, 2019, 12:10:31 PM4/4/19
to jenkin...@googlegroups.com
Hi,

That sounds good - we’re happy to host in jenkinsci and make the plugin available through the built-in Jenkins plugin manager.

As Jesse suggested we did indeed base our plugin on the Kubernetes credentials provider.

Would we be able to do it all in a single Github repo inside the jenkinsci org? Or would we have to run a separate upstream repo in our own Github account, with a fork under jenkinsci? (We’d prefer to go with the first option and cut out the middleman, if that makes sense.)
-- 
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Jesse Glick

unread,
Apr 4, 2019, 1:53:45 PM4/4/19
to Jenkins Dev
On Thu, Apr 4, 2019 at 12:10 PM Chris Kilding
<chris+...@chriskilding.com> wrote:
> Would we be able to do it all in a single Github repo inside the jenkinsci org?

Yes.

> a separate upstream repo in our own Github account, with a fork under jenkinsci?

We explicitly discourage having this fork structure as it makes it
harder to track which repository is considered authoritative. The
hosting process involves you offering a repo in your own org which
@jenkinsci forks, but as soon as that is done you should delete your
fork.

Stephen Connolly

unread,
Apr 8, 2019, 3:00:22 PM4/8/19
to jenkin...@googlegroups.com
It would be great to get any feedback on the docs I left for writing such things: 


For more options, visit https://groups.google.com/d/optout.
--
Sent from my phone

Chris Kilding

unread,
Apr 9, 2019, 6:57:07 AM4/9/19
to jenkin...@googlegroups.com
Hi Stephen,

When I started building the plugin my basis was a combination of your guide and the Kubernetes Credentials Provider.

The following sections of your guide were particularly valuable right from the start of development:

- The caching strategy guide for remote credentials providers (cache metadata for N minutes, retrieve secret data live).
- “Implementing a new Credentials type” (when and how to extend the standard types properly, when to make a new type).

I’m finding the guide is becoming more understandable (and I can get more out of it) now I’ve learned the basic architecture of Jenkins plugins. In the beginning it’s easier to work off an existing plugin’s code, but now I can read more of your guide and see implementation mistakes that need to be corrected.

Much like any other technical manual, its utility for the reader increases with the reader’s knowledge of the subject. Many thanks for writing it :)

Chris

Chris Kilding

unread,
Apr 9, 2019, 7:13:10 AM4/9/19
to jenkin...@googlegroups.com
Another thing worth mentioning is that I had to make two patches to Moto (and by proxy Localstack) to enable proper integration testing of the AWS Credentials Provider.

I will be able to open source the provider after the following PRs to Moto are merged, and after Localstack bumps its Moto dependency to include them:

- `list-secrets` support (previously Moto only supported retrieval with DescribeSecret): https://github.com/spulec/moto/pull/2143
- `delete-secret` support: https://github.com/spulec/moto/pull/2145

Stephen Connolly

unread,
Apr 9, 2019, 8:41:56 AM4/9/19
to jenkin...@googlegroups.com
On Tue, 9 Apr 2019 at 11:57, Chris Kilding <chris+...@chriskilding.com> wrote:
Hi Stephen,

When I started building the plugin my basis was a combination of your guide and the Kubernetes Credentials Provider.

The following sections of your guide were particularly valuable right from the start of development:

- The caching strategy guide for remote credentials providers (cache metadata for N minutes, retrieve secret data live).
- “Implementing a new Credentials type” (when and how to extend the standard types properly, when to make a new type).

I’m finding the guide is becoming more understandable (and I can get more out of it) now I’ve learned the basic architecture of Jenkins plugins. In the beginning it’s easier to work off an existing plugin’s code, but now I can read more of your guide and see implementation mistakes that need to be corrected.

Much like any other technical manual, its utility for the reader increases with the reader’s knowledge of the subject. Many thanks for writing it :)

Glad to hear. I wasn't happy moving my focus away from Jenkins plugins without leaving guides to the community for the stuff I believe to be important (credentials and scm-api). I am glad you have found (and are still finding) utility in the content. Feel free to create PRs with any suggested improvements as I am paying attention... just not as frequently ;-)

- Stephen
 

Chris Kilding

unread,
Apr 23, 2019, 12:37:49 PM4/23/19
to jenkin...@googlegroups.com
Hi Stephen,

I did have one question that’s not covered by the guide...

The guide says that a credential that fetches its secret value from a remote service must throw an IOException if there is a data-related problem (eg 404 not found), and an InterruptedException if the lookup times out.

However the guide also says that if there is already a de-facto standard credential type, one’s own credential Provider should use that.

In my case I need to extend BaseStandardCredentials / implement StringCredentials, and look up the AWS Secret Value within getSecret(). However the StringCredentials interface does not allow IOException or InterruptedException to be thrown by implementations.

The guide even acknowledges that this was a design flaw in the standard credentials types... but unfortunately it doesn’t say what to do instead, to work around this.

So, should an implementation of StringCredentials#getSecret()...

- Throw an unchecked exception?
- Return null?
- Something else?

Chris

Chris Kilding

unread,
May 7, 2019, 9:19:06 AM5/7/19
to jenkin...@googlegroups.com
Hello again,

I solved my previous question when I found that Jenkins would sensibly handle a CredentialsUnavailableException thrown from the getSecret() implementation. This is just a suitably named unchecked exception that’s provided by the credentials plugin.

Chris

Chris Kilding

unread,
May 10, 2019, 9:08:31 AM5/10/19
to jenkin...@googlegroups.com
The AWS Secrets Manager Credentials Provider plugin is now up on Github at https://github.com/chriskilding/aws-secrets-manager-credentials-provider-plugin

I need to do some initial exploratory testing and bug fixing on the plugin before I can release any official builds of the compiled .hpi file. But if you’d like to try it, you can check it out, install dependencies, and run ‘mvn clean verify’ to build it (see the README).

Once the plugin is ready, I’ll get the show on the road to migrate it to @jenkinsci and make it official.

Chris

Jeff Pearce

unread,
May 12, 2019, 9:56:23 AM5/12/19
to Jenkins Developers
I think you forgot to include some resources - when I build it can't find Messages.*. I can work around for my testing though
To unsubscribe from this group and stop receiving emails from it, send an email to jenkin...@googlegroups.com.


--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkin...@googlegroups.com.
--
Sent from my phone


--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkin...@googlegroups.com.


--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkin...@googlegroups.com.


--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkin...@googlegroups.com.


--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkin...@googlegroups.com.


--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkin...@googlegroups.com.

Jeff Pearce

unread,
May 12, 2019, 9:56:24 AM5/12/19
to Jenkins Developers
This is an awesome plugin. I have a couple of people at my company interested in testing it - let me know if there are any specific areas that need focus.

I'd suggest getting hosting on jenkinio soon. That would allow you to release alpha and beta versions of the plugin. I'm happy to help if you haven't gone through the process before

Best
Jeff


On Friday, May 10, 2019 at 6:08:31 AM UTC-7, Chris Kilding wrote:
To unsubscribe from this group and stop receiving emails from it, send an email to jenkin...@googlegroups.com.


--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkin...@googlegroups.com.
--
Sent from my phone


--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkin...@googlegroups.com.


--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkin...@googlegroups.com.


--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkin...@googlegroups.com.


--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkin...@googlegroups.com.


--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkin...@googlegroups.com.

Chris Kilding

unread,
May 12, 2019, 7:08:28 PM5/12/19
to jenkin...@googlegroups.com
Hi Jeff,

Yep, the compile error appears to be because the Maven localize plugin is not run when our IDEs compile the project, so the Messages class is never generated from the translation files. I’ll see about a fix tomorrow. The workaround in the meantime is to run ‘mvn localize:generate’ manually and then resume compilation with the IDE.

For testing I’d appreciate your team’s input on aspects like:

- Checking plugin stability over time - can it run for long periods without memory leaks or crashing?
- Finding edge cases related to AWS secrets lifecycle management (eg rapid soft deletion or undeletion of secrets).
- Finding edge cases related to AWS API calls (the AWS SDK does retry with backoff and jitter automatically, but are the defaults good for our use case - or do we need to tweak the retry / timeout / delay parameters?)
- Documentation - is the README easy enough for a first-time user to understand?

Regards,

Chris
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

Chris Kilding

unread,
May 13, 2019, 9:17:13 AM5/13/19
to jenkin...@googlegroups.com
Due to the interest in the plugin I’ve now opened a hosting request to transfer it to @jenkinsci at https://issues.jenkins-ci.org/browse/HOSTING-763

Chris

Chris Kilding

unread,
May 14, 2019, 6:03:17 AM5/14/19
to jenkin...@googlegroups.com

Jeff - could you delete and re-create your fork, so that the @jenkinsci repo is the authoritative root of the repository graph?

Regards,

Chris

Chris Kilding

unread,
May 16, 2019, 1:55:37 PM5/16/19
to jenkin...@googlegroups.com
The plugin now has a passing CI build on its master branch.

The next step would probably be to release an alpha version to the update site. I’ve seen the instructions for doing this manually, but I’m wondering if Jenkins has an automated pipeline to release plugins?

Chris

Slide

unread,
May 16, 2019, 2:15:53 PM5/16/19
to jenkin...@googlegroups.com
There is not currently an automated way to do this, the manual way is the only way.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/2dcbd23a-8a73-415c-baa8-092ef495e877%40www.fastmail.com.

For more options, visit https://groups.google.com/d/optout.


--
Website: http://earl-of-code.com

Chris Kilding

unread,
May 17, 2019, 8:19:43 AM5/17/19
to jenkin...@googlegroups.com
Alrighty, the manual way it is.

I’m following the steps to trim down the plugin POM in the release manual. However I’ve noticed the following quirks:

- When I inherit the parent POM’s groupId, I get the older “org.jenkins-ci.plugins” rather than the newer “io.jenkins.plugins”. Is this a problem, or can I stick with the older groupId? (It seems preferable for the plugin to inherit settings wherever possible.)
- When I took out the repositories and pluginRepositories sections, the plugin could not compile. Is there a fix for this, or should I keep a local copy of those sections in my POM?
- If I update to parent POM v3.36+ then Findbugs is replaced by Spotbugs. Will the Jenkins Findbugs report publisher (which all plugin builds have access to) understand Spotbugs reports?

Regards,

Chris

Joseph P

unread,
May 17, 2019, 9:55:10 AM5/17/19
to Jenkins Developers
- If I update to parent POM v3.36+ then Findbugs is replaced by Spotbugs. Will the Jenkins Findbugs report publisher (which all plugin builds have access to) understand Spotbugs reports?

Yes, SpotBugs is configured to work as a replacement and Jenkins CI will continue to fail.

Latest that is the experience many repos who have switched have noticed, see this PR: https://github.com/jenkinsci/credentials-plugin/pull/114
Which has a commit directed for "fix spotbugs"

On Friday, May 17, 2019 at 2:19:43 PM UTC+2, Chris Kilding wrote:
Alrighty, the manual way it is.

I’m following the steps to trim down the plugin POM in the release manual. However I’ve noticed the following quirks:

- When I inherit the parent POM’s groupId, I get the older “org.jenkins-ci.plugins” rather than the newer “io.jenkins.plugins”. Is this a problem, or can I stick with the older groupId? (It seems preferable for the plugin to inherit settings wherever possible.)
- When I took out the repositories and pluginRepositories sections, the plugin could not compile. Is there a fix for this, or should I keep a local copy of those sections in my POM?
- If I update to parent POM v3.36+ then Findbugs is replaced by Spotbugs. Will the Jenkins Findbugs report publisher (which all plugin builds have access to) understand Spotbugs reports?

Regards,

Chris

On Thu, 16 May 2019, at 7:15 PM, Slide wrote:
There is not currently an automated way to do this, the manual way is the only way.



--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkin...@googlegroups.com.


--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkin...@googlegroups.com.


--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkin...@googlegroups.com.


--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkin...@googlegroups.com.


--


--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkin...@googlegroups.com.

Chris Kilding

unread,
May 20, 2019, 7:56:25 AM5/20/19
to jenkin...@googlegroups.com
Jeff - I’m close to getting the plugin released via the Jenkins Incrementals update site (the new repository for automated plugin deployments). Will that be sufficient for your engineers to test with, or will you need a traditional alpha/beta release to the Experimental update site?

I should be able to release to Incrementals as soon as I’ve received upload permissions: https://github.com/jenkins-infra/repository-permissions-updater/pull/1137

Chris
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

Jeff Pearce

unread,
May 20, 2019, 1:19:09 PM5/20/19
to jenkin...@googlegroups.com

Hi Chris

 

No need to publish an Alpha version for me – I’m happy to build and install manually

 

Best

Jeff

 

From: <jenkin...@googlegroups.com> on behalf of Chris Kilding <chris+...@chriskilding.com>
Reply-To: "jenkin...@googlegroups.com" <jenkin...@googlegroups.com>
Date: Monday, May 20, 2019 at 4:56 AM
To: "jenkin...@googlegroups.com" <jenkin...@googlegroups.com>
Subject: Re: Proposal: AWS Secrets Manager Credentials Provider plugin

 

Notice: This email is from an external sender.

 

Chris Kilding

unread,
May 21, 2019, 5:59:33 AM5/21/19
to jenkin...@googlegroups.com
I’ll push the plugin through to Incrementals anyway as it validates the archive as part of the process, which is a useful automated check to have.

Unfortunately I’m seeing extremely slow upload speeds to the Incrementals Azure Function API, and ‘400 Bad Request’ errors, sometimes with no accompanying error message. (I’ve posted to the list separately about this as well.) The most it’s told me is that the archive might be invalid, but it doesn’t say why.

I haven’t seen the source of this Azure Function around - it might be private - so would someone be able to help me diagnose this?

Chris

Chris Kilding

unread,
May 23, 2019, 11:30:11 AM5/23/19
to jenkin...@googlegroups.com
Found the source of the incrementals-publisher function in the end. But I think the latest code in its master branch hasn’t been deployed yet - I see the old opaque error message in my build logs, whereas I should be seeing the newer more descriptive error message.

Could someone who is familiar with the incrementals-publisher indicate whether the deployed version of the function is commit d7e6b06 or newer?


Chris

Chris Kilding

unread,
May 29, 2019, 8:08:06 AM5/29/19
to jenkin...@googlegroups.com
I’m pleased to announce that version 0.0.1-alpha+1 of the AWS Secrets Manager Credentials Provider is now available on the Jenkins experimental update site.

If you would like to test drive the plugin, you’ll need to add the experimental update site to your Jenkins server and then install the plugin from it. Here are the instructions: https://jenkins.io/doc/developer/publishing/releasing-experimental-updates/#using-the-experimental-update-center

Or you can install the plugin on Jenkins Evergreen in the usual way, as it is published to the incrementals update site.

Or you can build the .hpi from source (https://github.com/jenkinsci/aws-secrets-manager-credentials-provider-plugin) and upload it to Jenkins, as before.

Please report any bugs you find to https://issues.jenkins-ci.org. Or if you have any other feedback then please reply to this message.

Chris



PS It seemed that either my initial alpha release or merging to master unblocked deployments to the incrementals repo, but I’m not sure which one did it.

Chris Kilding

unread,
Jun 26, 2019, 6:41:37 PM6/26/19
to jenkin...@googlegroups.com
The first stable version of the plugin (0.0.1) is now available in the main Jenkins update center.


And if you have any feedback, feature requests, or bug reports, just reply below - or file a Jira ticket :)

Chris
Reply all
Reply to author
Forward
0 new messages