Plugin to hide/mask/obfuscate regex in all build output?

[Jason Antman]

Hello,

We're looking to hide anything matching a list of regular expressions (specifically AWS access/secret keys at the moment) in the output of ALL builds. The closest thing I've been able to find is the Mask Passwords plugin, but that only handles specifically-defined strings (which are defined in the plugin config, which seems to be even *less* secure).

Is anyone aware of a plugin to do this? If not, does anyone have thoughts on me trying to add this in a pull request against Mask Passwords? Essentially the features I'd want are:

1. Masking of arbitrary regular expressions, i.e. I want '(?<![A-Za-z0-9/+=])[A-Za-z0-9/+=]{40}(?![A-Za-z0-9/+=])' to be masked/obfuscated anywhere it appears in build output.
2. Option to enable it globally on all jobs (i.e. force this for every build).

Thanks,
Jason Antman

[Jesse Glick]

On Tue, Jul 26, 2016 at 10:14 AM, Jason Antman <ja...@jasonantman.com> wrote:
> The closest thing I've been able to find is the Mask Passwords
> plugin, but that only handles specifically-defined strings (which are
> defined in the plugin config, which seems to be even *less* secure).

The Credentials Binding plugin does automatic masking for Pipeline
builds. Analogous support for freestyle builds is in progress.

[Jason Antman]

Jesse,

My concern isn't simply masking credentials. It's masking strings (in this case AWS secret keys) however they get into or appear in a build, period. i.e. if someone runs `cat ~/.aws/credentials` or for that matter makes an API call to generate credentials inside a job, we want it masked. We're also planning on transitioning to a service which will dynamically generate temporary credentials for builds; when we do, we still want them masked but won't have the literal string available before build time.

I'm working on a fork of the Mask Passwords plugin that supports masking user-defined regexes from the build output. Assuming I can get it working (my Java is really rusty), I'll open a pull request for it.

Thanks,

Jason

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-dev/zJH7-UvAz6g/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr0%3D-qTGnsFZEgUBs42wPrBqK50vxb2%2Ba-k%2BhO4-0CNeHg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Jesse Glick]

On Tue, Jul 26, 2016 at 8:38 PM, Jason Antman <ja...@jasonantman.com> wrote:
> if someone runs `cat ~/.aws/credentials` or for that matter makes an
> API call to generate credentials inside a job, we want it masked.

Indeed this would require a new feature.

> I'm working on a fork of the Mask Passwords plugin that supports masking
> user-defined regexes from the build output.

Or you could just create a separate plugin, since I suspect you would
be creating a separate `BuildWrapper` anyway.

(Consider making it a `SimpleBuildWrapper`—then you get Pipeline
compatibility for free.)

[Jason Antman]

If you think it's a separate plugin, I can give that a try.

Same BuildWrapper. Mask Passwords uses regexes internally already; it regex-escapes the passwords and then concatenates them into a big or'ed regex. So my code just adds another type to the configuration (user-specified regex) and appends those to the masking regex.

In terms of a separate plugin... I could probably use Mask Passwords as a starting point and have something working in a day or so... but it's almost certain that I won't be able to maintain this long-term, hence the hope that it could just be a feature I could contribute...

-Jason

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-dev/zJH7-UvAz6g/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr2K-SDSgovK0hcy2s%2BSA3CQe%2B-F4Z9rv_9FYtYPgWyipw%40mail.gmail.com.

[Jesse Glick]

On Wed, Jul 27, 2016 at 5:46 AM, Jason Antman <ja...@jasonantman.com> wrote:
> If you think it's a separate plugin, I can give that a try.

No strong opinion either way.

[Jason Antman]

Ok.

So I think I've got the code done and tested for the arbitrary regex masking, either globally or per-job: https://github.com/manheim/mask-passwords-plugin/tree/mask-regex

The other ask at my company is some way to **force** this for EVERY job, period. I understand that's a feature most people wouldn't want, and I might well keep that code in an internal branch only, rather than a pull request I submit. However... I'm wondering if there's any way to accomplish this? My initial instinct was hoping that there was some hook I could use when a Job is created or modified, but I couldn't find one. I then looked into extending the built-in Job class, but from what I read, that doesn't seem to be possible. If you have any tips (either how to do it, or that it's completely impossible and I should stop researching) it would be greatly appreciated.

-Jason

PS - The backstory on the crazy-sounding ask above: We've had keys to our AWS accounts compromised a number of times in the past year. Luckily, no major impact, but management and security people are up in arms. The goal is to prevent disclosure of AWS credentials to anyone who's not specifically authorized for them. And the #1 place they turn up is in build output - whether echoed from the environment or hard-coded somewhere, or retrieved dynamically. For the latter, and for the occasional person who hard-codes their personal credentials in a git repo, there's no way to mask an individual secret; we want to mask anything that matches the pattern published by Amazon.

Unfortunately, while we're in the very slow process of fixing this, we have a few Jenkins instances with a *lot* of autonomous development teams working in them, and everyone manages their own jobs. So some jobs are manually configured in the UI, some are pushed via the API from outside tooling, and some are Pipelines built via Job DSL, where the pipeline is re-configured from DSL on every Git commit, and therefore every commit updates the jobs from the canonical Groovy.

So essentially, I'm down to 3 options that I can think of:

(1) Find some way to hook into the save/update or load of Jobs, and (based on a global config setting) add Mask Passwords to every job if it's not already there.

(2) Find some way to hook into either the running of jobs or the creation of the models, and (based on a global config setting) inject the Mask Passwords build wrapper if it's not already there.

(3) Setup a Job that runs a Groovy script **every minute** and attempts to add the Mask Passwords build wrapper to **every job** in the system (which would probably take longer than a minute for some of our instances), and simply accept that anything using Job DSL and regenerating the pipeline on every commit will probably intermittently miss it.

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-dev/zJH7-UvAz6g/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr1Ogr7dKs%2BhwHGnBbDoq2gyO%3DMSZ6MSbCECk5Jog2-Jww%40mail.gmail.com.

[Jesse Glick]

On Fri, Jul 29, 2016 at 7:19 AM, Jason Antman <ja...@jasonantman.com> wrote:
> some way to **force** this for EVERY job,
> period. I understand that's a feature most people wouldn't want, and I might
> well keep that code in an internal branch only, rather than a pull request I
> submit. However... I'm wondering if there's any way to accomplish this?

IIRC there is an extension point for this purpose.

[Jason Antman]

Yeah, I found the extension point - ConsoleLogFliter.

I've opened a pull request for my changes - https://github.com/jenkinsci/mask-passwords-plugin/pull/6

[Jason Antman]

Jesse, or any Jenkins devs,

Is there anyone I can ping directly about this pull request? I've never contributed to a Jenkins plugin before... any idea what I can expect in terms of upstream response to this?

Thanks,
Jason

[Baptiste Mathus]

You can normally ping the maintainer, but in general s/he should already have been notified... If there's one... If not, which is not always clear, well then you can take over that maintenance yourself.

To get review, you can try pinging the jenkinsci/code-reviewers team.

Cheers

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/a9d3ee1f-0ce6-4158-b340-ea918a3967b5%40googlegroups.com.