Help troubleshooting Jenkins hashicorp-vault-plugin?

3,804 views
Skip to first unread message

chris...@nike.com

unread,
Jul 6, 2017, 6:02:42 PM7/6/17
to Vault
We're trying to deploy Vault support in our Jenkins instance and I'm running into issues with Jenkins connecting to our Vault instance. I'm fairly new to Vault but have been using Jenkins for a while. I'm hoping someone can suggest some things to look at (or other places to ask)...

As a PoC, we have a job that just reads a secret from our Vault instance and then runs a shell command to export all of the environment variables. I've tried with both regular token authentication and AppRole-based authentication with the same results.

When we run the job, we get the following error:
Started by user admin
Building in workspace /var/lib/jenkins/jobs/test/workspace
FATAL: could not read from vault: Vault responded with HTTP status code: 403 at path: secret/jenkins
com.bettercloud.vault.VaultException: Vault responded with HTTP status code: 403
	at com.bettercloud.vault.api.Logical.read(Logical.java:64)
	at com.datapipe.jenkins.vault.VaultAccessor.read(VaultAccessor.java:31)
Caused: com.datapipe.jenkins.vault.exception.VaultPluginException: could not read from vault: Vault responded with HTTP status code: 403 at path: secret/jenkins
	at com.datapipe.jenkins.vault.VaultAccessor.read(VaultAccessor.java:33)
	at com.datapipe.jenkins.vault.VaultBuildWrapper.provideEnvironmentVariablesFromVault(VaultBuildWrapper.java:131)
	at com.datapipe.jenkins.vault.VaultBuildWrapper.setUp(VaultBuildWrapper.java:91)
	at jenkins.tasks.SimpleBuildWrapper.setUp(SimpleBuildWrapper.java:146)
	at hudson.model.Build$BuildExecution.doRun(Build.java:157)
	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:490)
	at hudson.model.Run.execute(Run.java:1735)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
	at hudson.model.ResourceController.execute(ResourceController.java:97)
	at hudson.model.Executor.run(Executor.java:415)
Finished: FAILURE


If I change the Jenkins job to just be a shell command to curl the API directly, it's successful:
Started by user admin
Building in workspace /var/lib/jenkins/jobs/test/workspace
[workspace] $ /bin/sh -xe /tmp/jenkins7754296936506828979.sh
+ curl -H X-Vault-Token: 98e0ef2d-5212-0dce-744b-a3977761aa88 -X GET http://1.2.3.4:8200/v1/secret/jenkins/vault_test_variable
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100   196  100   196    0     0  87111      0 --:--:-- --:--:-- --:--:-- 98000
{"request_id":"69391266-7206-3fd0-bd25-2e02e42edad0","lease_id":"","renewable":false,"lease_duration":2764800,"data":{"value":"ThisIsTheCLIVariable"},"wrap_info":null,"warnings":null,"auth":null}
Finished: SUCCESS


That would indicate to me that the connection between Jenkins and Vault is working and that the token is good and the permissions for it are valid.

The Jenkins setup for Vault is:
Vault Secret Path: secret/jenkins
environment variable: VAULT_TEST_VARIABLE
key name: vault_test_variable

Credentials configuration:
Kind: Vault Token
Scope: Global
Token: 69391266-7206-3fd0-bd25-2e02e42edad0
ID: 1944d7ae-3254-40a9-881d-37ee2e0d01d7
Description: test

I've reproduced this on both our existing 'unclean' Jenkins server and on a brand new Vagrant-based Jenkins server (built from https://github.com/geerlingguy/ansible-vagrant-examples/tree/master/jenkins). Versions on the 'blank' server:
Jenkins 2.68
BouncyCastle API: 2.16.1
Credentials Plugin: 2.1.14
Folders Plugin: 6.0.4
HashiCorp Vault Program: 2.1.0
Pipeline: SCM Step: 2.6
Pipeline: Step API: 2.12
Structs Plugin: 1.9


Jason Antman

unread,
Jul 6, 2017, 6:22:34 PM7/6/17
to Vault
Chris,

Can you post (minus any sensitive data) both the job configuration XML for the job you're trying, as well as the Vault Plugin configuration XML ( it should be at $JENKINS_HOME/com.datapipe.jenkins.vault.configuration.GlobalVaultConfiguration.xml ). We're using the Vault Plugin in quite a few places as we're rather heavy Jenkins users, but we also do all of our configuration via groovy/JobDSL/Jenkinsfile, and we're doing some "special" stuff with how we get a token into it. Not all of my experience might be applicable, but I have found the Vault Plugin to be rather unintuitive.

The other thing I'll recommend right away is to get rid of the randomly generated Credential ID, and use a meaningful name in that field - it'll be really helpful in the long run.

Aside from posting configs, have you looked at Vault's audit logs? Does the Vault server think it's getting an unknown token, or a policy error, or what? Unfortunately, unless this has changed since the last time I looked at the Vault Jenkins Plugin source code (a few months ago), there's no logging sent to the Jenkins log at all, so you can't even get the actual 403 message on the Jenkins end.

-Jason

chris...@nike.com

unread,
Jul 6, 2017, 6:44:45 PM7/6/17
to Vault
We dont have a com.datapipe.jenkins.vault.configuration.GlobalVaultConfiguration.xml - I assume because on the bare server, I didnt set up global credentials, just job-based ones. On our main 'dirty' server, I did try with global Vault configs and failed the same way.

Here's the Vault audit log - Is it doing some weird encryption on the token that's not bein decrypted? We want to use 777c5c5d-1aea-6d2a-1c81-7e2f64add324:
{"time":"2017-07-06T22:38:49Z","type":"request","auth":{"client_token":"","accessor":"","display_name":"","policies":null,"metadata":null},"request":{"id":"da9f50d0-03a3-a3a7-11be-bff03d611c45","operation":"read","client_token":"hmac-sha256:d8ecf8341f665ee6b6eb6e4b7d93fce649a2c27cd6f72ae671e8148f0dde930c","path":"secret/jenkins","data":null,"remote_address":"5.6.7.8","wrap_ttl":0},"error":"permission denied"}
{"time":"2017-07-06T22:38:49Z","type":"response","error":"1 error(s) occurred:\n\n* permission denied","auth":{"client_token":"","accessor":"","display_name":"","policies":null,"metadata":null},"request":{"id":"da9f50d0-03a3-a3a7-11be-bff03d611c45","operation":"read","client_token":"hmac-sha256:d8ecf8341f665ee6b6eb6e4b7d93fce649a2c27cd6f72ae671e8148f0dde930c","path":"secret/jenkins","data":null,"remote_address":"5.6.7.8","wrap_ttl":0},"response":{"data":{"error":"hmac-sha256:f229374491d17e573263711d8abacd2419da9df0f75ff97fae994cbba4d9bd89"}}}
 

Here's credentials.xml:
<?xml version='1.0' encoding='UTF-8'?>
<com.cloudbees.plugins.credentials.SystemCredentialsProvider plugin="crede...@2.1.14">
  <domainCredentialsMap class="hudson.util.CopyOnWriteMap$Hash">
    <entry>
      <com.cloudbees.plugins.credentials.domains.Domain>
        <specifications/>
      </com.cloudbees.plugins.credentials.domains.Domain>
      <java.util.concurrent.CopyOnWriteArrayList>
        <com.datapipe.jenkins.vault.credentials.VaultTokenCredential plugin="hashicorp-v...@2.1.0">
          <scope>GLOBAL</scope>
          <id>1944d7ae-3254-40a9-881d-37ee2e0d01d7</id>
          <description>test</description>
          <token>{AQAAABAAAAAwHHwAD6q7KSZ7ylGXQU+yj6KDdvBo60pZzhf/jman+61CPL+0MmCAsvecI+AIHVo+fcv5VFNnFDbtWp6oz9u7mA==}</token>
        </com.datapipe.jenkins.vault.credentials.VaultTokenCredential>
      </java.util.concurrent.CopyOnWriteArrayList>
    </entry>
  </domainCredentialsMap>
</com.cloudbees.plugins.credentials.SystemCredentialsProvider>

Here's the job XML:
<?xml version='1.0' encoding='UTF-8'?>
<project>
  <actions/>
  <description></description>
  <keepDependencies>false</keepDependencies>
  <properties/>
  <scm class="hudson.scm.NullSCM"/>
  <canRoam>true</canRoam>
  <disabled>false</disabled>
  <blockBuildWhenDownstreamBuilding>false</blockBuildWhenDownstreamBuilding>
  <blockBuildWhenUpstreamBuilding>false</blockBuildWhenUpstreamBuilding>
  <triggers/>
  <concurrentBuild>false</concurrentBuild>
  <builders>
    <hudson.tasks.Shell>
      <command>curl -H &apos;X-Vault-Token: 98e0ef2d-5212-0dce-744b-a3977761aa88&apos; -X GET http://1.2.3.4:8200/v1/secret/jenkins/vault_test_variable</command>
    </hudson.tasks.Shell>
  </builders>
  <publishers/>
  <buildWrappers>
    <com.datapipe.jenkins.vault.VaultBuildWrapper plugin="hashicorp-v...@2.1.0">
      <configuration>
        <vaultUrl>http://1.2.3.4:8200</vaultUrl>
        <vaultCredentialId>1944d7ae-3254-40a9-881d-37ee2e0d01d7</vaultCredentialId>
      </configuration>
      <vaultSecrets>
        <com.datapipe.jenkins.vault.model.VaultSecret>
          <path>secret/jenkins</path>
          <secretValues>
            <com.datapipe.jenkins.vault.model.VaultSecretValue>
              <envVar>VAULT_TEST_VARIABLE</envVar>
              <vaultKey>vault_test_variable</vaultKey>
            </com.datapipe.jenkins.vault.model.VaultSecretValue>
          </secretValues>
        </com.datapipe.jenkins.vault.model.VaultSecret>
      </vaultSecrets>
      <valuesToMask/>
      <vaultAccessor/>
    </com.datapipe.jenkins.vault.VaultBuildWrapper>
  </buildWrappers>

Jason Antman

unread,
Jul 6, 2017, 6:54:12 PM7/6/17
to Vault
Unless you specifically set the audit backend to log sensitive information in the clear, by default it will only log HMAC-SHA256 hashes of anything sensitive, including the token/accessor; see https://www.vaultproject.io/docs/audit/index.html

If my memory serves me correctly, an audit log entry that includes:
"auth":{"client_token":"","accessor":"","display_name":"","policies":null,"metadata":null}
but a HMAC-SHA256'ed client_token in the request, implies that the client used a token that was either invalid or expired, and therefore the 403 is because the client wasn't properly authenticated.

First thing, I'd check that the token you've configured for Jenkins is correct (i.e. you gave Jenkins the token not the accessor, there wasn't a copy/paste error, and it hasn't expired yet).

I can't see anything obviously wrong with the configs. The only thing I can see there as a possible cause for the problem, is the token configured in that credential being either wrong or expired...

-Jason

On Thu, Jul 6, 2017 at 6:44 PM, <chris...@nike.com> wrote:
We dont have a com.datapipe.jenkins.vault.configuration.GlobalVaultConfiguration.xml - I assume because on the bare server, I didnt set up global credentials, just job-based ones. On our main 'dirty' server, I did try with global Vault configs and failed the same way.

Here's the Vault audit log - Is it doing some weird encryption on the token that's not bein decrypted? We want to use 777c5c5d-1aea-6d2a-1c81-7e2f64add324:
{"time":"2017-07-06T22:38:49Z","type":"request","auth":{"client_token":"","accessor":"","display_name":"","policies":null,"metadata":null},"request":{"id":"da9f50d0-03a3-a3a7-11be-bff03d611c45","operation":"read","client_token":"hmac-sha256:d8ecf8341f665ee6b6eb6e4b7d93fce649a2c27cd6f72ae671e8148f0dde930c","path":"secret/jenkins","data":null,"remote_address":"5.6.7.8","wrap_ttl":0},"error":"permission denied"}
{"time":"2017-07-06T22:38:49Z","type":"response","error":"1 error(s) occurred:\n\n* permission denied","auth":{"client_token":"","accessor":"","display_name":"","policies":null,"metadata":null},"request":{"id":"da9f50d0-03a3-a3a7-11be-bff03d611c45","operation":"read","client_token":"hmac-sha256:d8ecf8341f665ee6b6eb6e4b7d93fce649a2c27cd6f72ae671e8148f0dde930c","path":"secret/jenkins","data":null,"remote_address":"5.6.7.8","wrap_ttl":0},"response":{"data":{"error":"hmac-sha256:f229374491d17e573263711d8abacd2419da9df0f75ff97fae994cbba4d9bd89"}}}
 

Here's credentials.xml:
<?xml version='1.0' encoding='UTF-8'?>
<com.cloudbees.plugins.credentials.SystemCredentialsProvider plugin="crede...@2.1.14">
  <domainCredentialsMap class="hudson.util.CopyOnWriteMap$Hash">
    <entry>
      <com.cloudbees.plugins.credentials.domains.Domain>
        <specifications/>
      </com.cloudbees.plugins.credentials.domains.Domain>
      <java.util.concurrent.CopyOnWriteArrayList>
        <com.datapipe.jenkins.vault.credentials.VaultTokenCredential plugin="hashicorp-vault-plu...@2.1.0">
    <com.datapipe.jenkins.vault.VaultBuildWrapper plugin="hashicorp-vault-plu...@2.1.0">

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/vault/issues
IRC: #vault-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Vault" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/88860306-d664-430b-b560-7e555043aa46%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

chris...@nike.com

unread,
Jul 6, 2017, 7:05:31 PM7/6/17
to Vault
Hmm. The curl command with that token still works. Does the calling method matter for expiration/auth purposes (IE - Curl is allowed while the java call is not)?
To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+...@googlegroups.com.

Jason Antman

unread,
Jul 6, 2017, 7:15:16 PM7/6/17
to Vault
That certainly shouldn't be the case, the protocol is pretty simple. The only thing I would think that could cause that would be an implementation bug in the Java client, and I'm running version 2.1.0 of the plugin as well, so I'm pretty sure that's not it.

Um... apologies for not catching this before...
{"request_id":"69391266-7206-3fd0-bd25-2e02e42edad0","lease_id":"","renewable":false,"lease_duration":2764800,"data":{"value":"ThisIsTheCLIVariable"},"wrap_info":null,"warnings":null,"auth":null}
Is getting a different secret from your job configuration. I believe that's likely the (or a) problem. The curl command you're running (the one I pasted part of above) is getting a secret from /secret/jenkins/vault_test_variable, which has a single key in it, "value", with a value of "ThisIsTheCLIVariable".

The configuration for your Jenkins job, however, 

       <com.datapipe.jenkins.vault.model.VaultSecret>
          <path>secret/jenkins</path>
          <secretValues>
            <com.datapipe.jenkins.vault.model.VaultSecretValue>
              <envVar>VAULT_TEST_VARIABLE</envVar>
              <vaultKey>vault_test_variable</vaultKey>
            </com.datapipe.jenkins.vault.model.VaultSecretValue>
          </secretValues>
        </com.datapipe.jenkins.vault.model.VaultSecret>

Is trying to read a key named "vault_test_variable" from a secret at secret/jenkins, which doesn't exist.

It's important to know that Vault's secret backend can store multiple key/value pairs in one secret path. It appears that your Jenkins configuration has confused keys with path components. I believe that if you changed the VaultSecret configured on the Jenkins job to have the same path as the one you're CURLing (secret/jenkins/vault_test_variable) and set that one Secret Value to use the "value" key, that might do it.

-Jason


On Thu, Jul 6, 2017 at 7:05 PM, <chris...@nike.com> wrote:
Hmm. The curl command with that token still works. Does the calling method matter for expiration/auth purposes (IE - Curl is allowed while the java call is not)?

On Thursday, July 6, 2017 at 3:54:12 PM UTC-7, Jason Antman wrote:
Unless you specifically set the audit backend to log sensitive information in the clear, by default it will only log HMAC-SHA256 hashes of anything sensitive, including the token/accessor; see https://www.vaultproject.io/docs/audit/index.html

If my memory serves me correctly, an audit log entry that includes:
"auth":{"client_token":"","accessor":"","display_name":"","policies":null,"metadata":null}
but a HMAC-SHA256'ed client_token in the request, implies that the client used a token that was either invalid or expired, and therefore the 403 is because the client wasn't properly authenticated.

First thing, I'd check that the token you've configured for Jenkins is correct (i.e. you gave Jenkins the token not the accessor, there wasn't a copy/paste error, and it hasn't expired yet).

I can't see anything obviously wrong with the configs. The only thing I can see there as a possible cause for the problem, is the token configured in that credential being either wrong or expired...

-Jason
On Thu, Jul 6, 2017 at 6:44 PM, <chris...@nike.com> wrote:
We dont have a com.datapipe.jenkins.vault.configuration.GlobalVaultConfiguration.xml - I assume because on the bare server, I didnt set up global credentials, just job-based ones. On our main 'dirty' server, I did try with global Vault configs and failed the same way.

Here's the Vault audit log - Is it doing some weird encryption on the token that's not bein decrypted? We want to use 777c5c5d-1aea-6d2a-1c81-7e2f64add324:
{"time":"2017-07-06T22:38:49Z","type":"request","auth":{"client_token":"","accessor":"","display_name":"","policies":null,"metadata":null},"request":{"id":"da9f50d0-03a3-a3a7-11be-bff03d611c45","operation":"read","client_token":"hmac-sha256:d8ecf8341f665ee6b6eb6e4b7d93fce649a2c27cd6f72ae671e8148f0dde930c","path":"secret/jenkins","data":null,"remote_address":"5.6.7.8","wrap_ttl":0},"error":"permission denied"}
{"time":"2017-07-06T22:38:49Z","type":"response","error":"1 error(s) occurred:\n\n* permission denied","auth":{"client_token":"","accessor":"","display_name":"","policies":null,"metadata":null},"request":{"id":"da9f50d0-03a3-a3a7-11be-bff03d611c45","operation":"read","client_token":"hmac-sha256:d8ecf8341f665ee6b6eb6e4b7d93fce649a2c27cd6f72ae671e8148f0dde930c","path":"secret/jenkins","data":null,"remote_address":"5.6.7.8","wrap_ttl":0},"response":{"data":{"error":"hmac-sha256:f229374491d17e573263711d8abacd2419da9df0f75ff97fae994cbba4d9bd89"}}}
 

Here's credentials.xml:
<?xml version='1.0' encoding='UTF-8'?>
<com.cloudbees.plugins.credentials.SystemCredentialsProvider plugin="crede...@2.1.14">
  <domainCredentialsMap class="hudson.util.CopyOnWriteMap$Hash">
    <entry>
      <com.cloudbees.plugins.credentials.domains.Domain>
        <specifications/>
      </com.cloudbees.plugins.credentials.domains.Domain>
      <java.util.concurrent.CopyOnWriteArrayList>
        <com.datapipe.jenkins.vault.credentials.VaultTokenCredential plugin="hashicorp-vault-plugin@2.1.0">
    <com.datapipe.jenkins.vault.VaultBuildWrapper plugin="hashicorp-vault-plugin@2.1.0">
To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/a0946e59-ccdd-45e8-971f-50b57b47fb79%40googlegroups.com.

chris...@nike.com

unread,
Jul 7, 2017, 11:30:20 AM7/7/17
to Vault
AHA! :facepalm: You're exactly right. It was my mis-understanding of the syntax to insert keys with (I'll have to find the blog post I gleaned that from and comment on it). 
I thought the syntax was "vault write secret/<folder>/<keyname> value=<variable>".
Should be "vault write secret/<folder> <keyname>=<value>".

Fixing the Jenkins job to look for the path "secret/jenkins/vault_test_variable" and the key to "value" worked without throwing an error.

Thanks mucho for catching that! I *really* appreciate the help, gang.
        <com.datapipe.jenkins.vault.credentials.VaultTokenCredential plugin="hashicorp-vault-plu...@2.1.0">
    <com.datapipe.jenkins.vault.VaultBuildWrapper plugin="hashicorp-vault-plu...@2.1.0">

Jason Antman

unread,
Jul 7, 2017, 1:02:59 PM7/7/17
to Vault
Just to clarify a bit here for you...

On Fri, Jul 7, 2017 at 11:30 AM, <chris...@nike.com> wrote:
AHA! :facepalm: You're exactly right. It was my mis-understanding of the syntax to insert keys with (I'll have to find the blog post I gleaned that from and comment on it). 
I thought the syntax was "vault write secret/<folder>/<keyname> value=<variable>".
Should be "vault write secret/<folder> <keyname>=<value>".


The use of "folder" here isn't really optimal, and it's only made worse by the Vault CLI output for vault list, which uses the word "Key" in a confusing way.

I think the thing to keep in mind is that vault's secret backend paths are like "files", which can have contents made up of multiple key/value pairs. The path (secret/folder/something) is one distinct path, analagous to a file, and can store multiple key/value pairs "inside" it.

i.e., we can a single key ("value") to the secret/foo/bar path


$ vault write secret/foo/bar value=baz
Success! Data written to: secret/foo/bar
$ vault read secret/foo/bar
Key                     Value
---                     -----
refresh_interval        768h0m0s
value                   baz 
$ vault list secret/foo
Keys
----
bar
but we can also write multiple key/value pairs to that path as well, which are retrieved as one unit:

$ vault write secret/foo/bar value=baz other=blam extra=quux
Success! Data written to: secret/foo/bar
$ vault read secret/foo/bar                                  
Key                     Value
---                     -----
refresh_interval        768h0m0s
extra                   quux
other                   blam
value                   baz
$ vault list secret/foo
Keys
----
bar
Reply all
Reply to author
Forward
0 new messages