[JIRA] (JENKINS-52646) KV v2 changes the API path to secrets & the response format

0 views
Skip to first unread message

mbarr@mbarr.net (JIRA)

unread,
Jul 19, 2018, 12:28:01 AM7/19/18
to jenkinsc...@googlegroups.com
Matthew Barr created an issue
 
Jenkins / Bug JENKINS-52646
KV v2 changes the API path to secrets & the response format
Issue Type: Bug Bug
Assignee: Peter Tierno
Components: hashicorp-vault-plugin
Created: 2018-07-19 04:27
Environment: Using Jenkins 2.133, plugin 2.1.1, vault 0.10.3 in dev mode, w/ kv version 2 for /secrets/, all on linux.
Priority: Major Major
Reporter: Matthew Barr

It looks like .10 changed the API for kv. https://www.vaultproject.io/api/secret/kv/kv-v2.html

it now requires /secret/data/path, instead of /secret/path.  The response is also altered, so that you get 

{{"data": {"data":

{"foo": "bar" }

}}}

vs{{}}

"data": {
    "foo": "bar"
  }

 

This causes 404 errors if you don't manually put the /data/ into the path, and doesn't allow the key to be found, so you don't get the actual values into the build steps.

 

{{}}

{{}}
Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v7.10.1#710002-sha1:6efc396)

mewadadinesh75@gmail.com (JIRA)

unread,
Jul 19, 2018, 1:35:02 AM7/19/18
to jenkinsc...@googlegroups.com
dinesh mewada updated an issue
 
Jenkins / Epic JENKINS-52646
Change By: dinesh mewada
Issue Type: Bug Epic
Priority: Major Critical

mbarr@mbarr.net (JIRA)

unread,
Jul 19, 2018, 10:49:02 AM7/19/18
to jenkinsc...@googlegroups.com
Matthew Barr updated an issue
Change By: Matthew Barr
It looks like .10 changed the API for kv. [https://www.vaultproject.io/api/secret/kv/kv-v2.html]

it now requires /secret/data/path, instead of /secret/path.  The response is also altered, so that you get 

{{"data": {"data":

{"foo": "bar"
   }
}}}

vs \ {{}}
{code:java}
"data": {
    "foo": "bar"
  }
{code}

 

This causes 404 errors if you don't manually put the /data/ into the path, and doesn't allow the key to be found, so you don't get the actual values into the build steps.

 


{{}}

{{}}

vinogradovia@gmail.com (JIRA)

unread,
Sep 5, 2018, 6:49:02 AM9/5/18
to jenkinsc...@googlegroups.com
Ivan Vinogradov updated an issue
Change By: Ivan Vinogradov
It looks like .10 changed the API for kv. [https://www.vaultproject.io/api/secret/kv/kv-v2.html]

it now requires /secret/data/path, instead of /secret/path.  The response is also altered, so that you get 

  
{ { code:java}
 "data": {
"data":

{
"foo": "bar"
}
 } 
{code } }}

vs\{{}}
{code:java}
"data": {
    "foo": "bar"
  }
{code}
 

This causes 404 errors if you don't manually put the /data/ into the path, and doesn't allow the key to be found, so you don't get the actual values into the build steps.

 
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

john@johnmccabe.net (JIRA)

unread,
Apr 26, 2019, 6:10:05 AM4/26/19
to jenkinsc...@googlegroups.com
John McCabe commented on Epic JENKINS-52646
 
Re: KV v2 changes the API path to secrets & the response format

Hi Peter Tierno, are there any plans on how/when kv2 will be supported, I ran into this myself when running against a dev vault (which defaults to kv2 for /secret).

I notice that there are already some references to being able to choose the kv engine version in the code but not in a released version of the plugin.

john@johnmccabe.net (JIRA)

unread,
Apr 26, 2019, 8:16:02 AM4/26/19
to jenkinsc...@googlegroups.com

Just tried the latest build from master and it looking pretty good.

sgardner661@gmail.com (JIRA)

unread,
May 31, 2019, 4:38:03 PM5/31/19
to jenkinsc...@googlegroups.com

Latest build from master of the plugin caused my Jenkins builds to fail when the job tries logging in via AppRole to get the client token:

FATAL: could not log in into vault
com.bettercloud.vault.VaultException: Vault responded with HTTP status code: 400
	at com.bettercloud.vault.api.Auth.loginByAppRole(Auth.java:409)
	at com.datapipe.jenkins.vault.credentials.VaultAppRoleCredential.getToken(VaultAppRoleCredential.java:50)

Line 50 in VaultAppRoleCredential.java was updated to use the path string variable from the path field specified for the AppRole credentials instead being hardcoded as "approle", there seems to be a formatting error with the path string causing it to fail. Here's the change:
[https://github.com/jenkinsci/hashicorp-vault-plugin/commit/c2c141cbf24b548c266a9134bd0f8df5a4800339

]

-            token = vault.auth().loginByAppRole("approle", roleId, Secret.toString(secretId)).getAuthClientToken();            

+            token = vault.auth().loginByAppRole(path, roleId, Secret.toString(secretId)).getAuthClientToken();

[

https://github.com/jenkinsci/hashicorp-vault-plugin/commit/c2c141cbf24b548c266a9134bd0f8df5a4800339]As a workaround I updated the file to format the path string: 
    public String getPath() {        
-       return path;        
+       return String.format("%s", path);    
    }

Jobs are now able to successfully login via AppRole and get the client token. Am able to pull secrets from both KV v1 and v2 engines. KV v2 no longer needs "data" in the path. Here's my personal repo with the changes:
[https://github.com/stewcee/hashicorp-vault-plugin

]I'm not well-versed in Java and couldn't figure out the proper format for the string when the path field is null, so the path field cannot be null for the AppRole credentials.

sgardner661@gmail.com (JIRA)

unread,
May 31, 2019, 4:40:02 PM5/31/19
to jenkinsc...@googlegroups.com
Steven Gardner edited a comment on Epic JENKINS-52646
Latest build from master of the plugin caused my Jenkins builds to fail when the job tries logging in via AppRole to get the client token:
{code:java}

FATAL: could not log in into vault
com.bettercloud.vault.VaultException: Vault responded with HTTP status code: 400
at com.bettercloud.vault.api.Auth.loginByAppRole(Auth.java:409)
at com.datapipe.jenkins.vault.credentials.VaultAppRoleCredential.getToken(VaultAppRoleCredential.java:50){code}

Line 50 in VaultAppRoleCredential.java was updated to use the path string variable from the path field specified for the AppRole credentials instead being hardcoded as "approle", there seems to be a formatting error with the path string causing it to fail. Here's the change:
[https://github.com/jenkinsci/hashicorp-vault-plugin/commit/c2c141cbf24b548c266a9134bd0f8df5a4800339


 ]
{code:java}

-            token = vault.auth().loginByAppRole("approle", roleId, Secret.toString(secretId)).getAuthClientToken();            

+            token = vault.auth().loginByAppRole(path, roleId, Secret.toString(secretId)).getAuthClientToken(); {code}
 [
|https://github.com/jenkinsci/hashicorp-vault-plugin/commit/c2c141cbf24b548c266a9134bd0f8df5a4800339] As a workaround I updated the file to format the path string:
{code:java}

    public String getPath() {        
-       return path;        
+       return String.format("%s", path);    
    } {code}


Jobs are now able to successfully login via AppRole and get the client token. Am able to pull secrets from both KV v1 and v2 engines. KV v2 no longer needs "data" in the path. Here's my personal repo with the changes:
[https://github.com/stewcee/hashicorp-vault-plugin

]I'm not well-versed in Java and couldn't figure out the proper format for the string when the path field is null, so the path field cannot be null for the AppRole credentials.

sgardner661@gmail.com (JIRA)

unread,
May 31, 2019, 4:41:03 PM5/31/19
to jenkinsc...@googlegroups.com
Steven Gardner edited a comment on Epic JENKINS-52646
Latest build from master of the plugin caused my Jenkins builds to fail when the job tries logging in via AppRole to get the client token:
{code:java}
FATAL: could not log in into vault
com.bettercloud.vault.VaultException: Vault responded with HTTP status code: 400
at com.bettercloud.vault.api.Auth.loginByAppRole(Auth.java:409)
at com.datapipe.jenkins.vault.credentials.VaultAppRoleCredential.getToken(VaultAppRoleCredential.java:50){code}
Line 50 in VaultAppRoleCredential.java was updated to use the path string variable from the path field specified for the AppRole credentials instead being hardcoded as "approle", there seems to be a formatting error with the path string causing it to fail. Here's the change:
[https://github.com/jenkinsci/hashicorp-vault-plugin/commit/c2c141cbf24b548c266a9134bd0f8df5a4800339]
{code:java}
-            token = vault.auth().loginByAppRole("approle", roleId, Secret.toString(secretId)).getAuthClientToken();            

+            token = vault.auth().loginByAppRole(path, roleId, Secret.toString(secretId)).getAuthClientToken(); {code}

As a workaround I updated the file to format the path string:
{code:java}
    public String getPath() {        
-       return path;        
+       return String.format("%s", path);    
    } {code}

Jobs are now able to successfully login via AppRole and get the client token. Am able to pull secrets from both KV v1 and v2 engines. KV v2 no longer needs "data" in the path. Here's my personal repo with the changes:
[https://github.com/stewcee/hashicorp-vault-plugin
]

] I'm not well-versed in Java and couldn't figure out the proper format for the string when the path field is null, so the path field cannot be null for the AppRole credentials.

anton@mostovoy.net (JIRA)

unread,
Jun 3, 2019, 6:25:04 PM6/3/19
to jenkinsc...@googlegroups.com

i ran into the same issue with the latest build from master, and i was able to work around by hitting the save button on the Vault Credential modal.  Is the "path" value being stored differently now?

anton@mostovoy.net (JIRA)

unread,
Jun 3, 2019, 6:26:03 PM6/3/19
to jenkinsc...@googlegroups.com
Anton Mostovoy edited a comment on Epic JENKINS-52646
i ran into the same issue as Steven Gardener with the latest build from master, and i was able to work around by hitting the save button on the Vault Credential modal.  Is the "path" value being stored differently now?

anton@mostovoy.net (JIRA)

unread,
Jun 3, 2019, 6:26:04 PM6/3/19
to jenkinsc...@googlegroups.com
Anton Mostovoy edited a comment on Epic JENKINS-52646
i ran into the same issue as Steven Gardener Gardner with the latest build from master, and i was able to work around by hitting the save button on the Vault Credential modal.  Is the "path" value being stored differently now?

abubadabu@gmail.com (JIRA)

unread,
Jun 21, 2019, 6:12:03 PM6/21/19
to jenkinsc...@googlegroups.com

Any hint on how to retrive the data['data']['data']['username'] for a v2 reply?

 

KV 1.0

curl --header "X-Vault-Token: xxx" https://127.0.0.1/v1/kv1/foobar

{
  "request_id":"...",
  "lease_id":"",
  "renewable":false,
  "lease_duration":2764800,
  "data": {
    "password":"xxx",
    "username":"xxx"
  },
  "wrap_info":null,
  "warnings":null,
  "auth":null
}
wrap([$class: 'VaultBuildWrapper', vaultSecrets: [[path: 'kv1/foobar', secretValues: [
  [envVar: 'username', vaultKey: 'username'],
  [envVar: 'password', vaultKey: 'password']
]]]])
{
  sh('echo $username')
  sh('echo $password')
}

// --> echos username / passwor

KV 2.0

curl --header "X-Vault-Token: xxx" https://127.0.0.1/v1/kv2/data/foobar

{
  "request_id":"...",
  "lease_id":"",
  "renewable":false,
  "lease_duration":0,
  "data": {
    "data": {
      "password":"xxx",
      "username":"xxx"
    },
    "metadata": {
      "created_time":"...",
      "deletion_time":"",
      "destroyed":false,
      "version":1
    }
  },
  "wrap_info": null,
  "warnings": null,
  "auth": null
}
wrap([$class: 'VaultBuildWrapper', vaultSecrets: [[path: 'kv2/data/foobar', secretValues: [
  [envVar: 'username', vaultKey: 'username'],
  [envVar: 'password', vaultKey: 'password']
]]]])
{
  sh('echo $username')
  sh('echo $password')
}

// --> empty echo

abubadabu@gmail.com (JIRA)

unread,
Jun 21, 2019, 6:26:02 PM6/21/19
to jenkinsc...@googlegroups.com
Abubadabu edited a comment on Epic JENKINS-52646
Any hint on how to retrive the data['data']['data']['username'] for a v2 reply?

 

KV 1.0

curl --header "X-Vault-Token: xxx" https://127.0.0.1/v1/kv1/foobar

{
  "request_id":"...",
  "lease_id":"",
  "renewable":false,
  "lease_duration":2764800,
  "data": {
    "password":"xxx",
    "username":"xxx"
  },
  "wrap_info":null,
  "warnings":null,
  "auth":null
}{code}
{code:java}

wrap([$class: 'VaultBuildWrapper', vaultSecrets: [[path: 'kv1/foobar', secretValues: [
  [envVar: 'username', vaultKey: 'username'],
  [envVar: 'password', vaultKey: 'password']
]]]])
{
  sh('echo $username')
  sh('echo $password')
}

// --> echos username / passwor{code}
KV 2.0
[https://www.vaultproject.io/api/secret/kv/kv-v2.html#read-secret-version]
 {code:java}

curl --header "X-Vault-Token: xxx" https://127.0.0.1/v1/kv2/data/foobar

{
  "request_id":"...",
  "lease_id":"",
  "renewable":false,
  "lease_duration":0,
  "data": {
    "data": {
      "password":"xxx",
      "username":"xxx"
    },
    "metadata": {
      "created_time":"...",
      "deletion_time":"",
      "destroyed":false,
      "version":1
    }
  },
  "wrap_info": null,
  "warnings": null,
  "auth": null
}{code}
{code:java}

wrap([$class: 'VaultBuildWrapper', vaultSecrets: [[path: 'kv2/data/foobar', secretValues: [
  [envVar: 'username', vaultKey: 'username'],
  [envVar: 'password', vaultKey: 'password']
]]]])
{
  sh('echo $username')
  sh('echo $password')
}

// --> empty echo{code}

abubadabu@gmail.com (JIRA)

unread,
Jun 21, 2019, 7:11:02 PM6/21/19
to jenkinsc...@googlegroups.com

Ok...looks like it's already fixed...but not released. Any Peter Tierno any plan for a push to release?

rnganeye@mts.ru (JIRA)

unread,
Jul 15, 2019, 5:38:03 AM7/15/19
to jenkinsc...@googlegroups.com

Please merge, we need this feature.

rnganeye@mts.ru (JIRA)

unread,
Jul 15, 2019, 5:38:05 AM7/15/19
to jenkinsc...@googlegroups.com
Rafis Ganeyev edited a comment on Epic JENKINS-52646
Please [~ptierno], please merge, we need this feature.

bksaville@gmail.com (JIRA)

unread,
Jul 16, 2019, 3:55:03 PM7/16/19
to jenkinsc...@googlegroups.com

Second that - we have many users switching to v2 format and we currently cannot work with them.

bksaville@gmail.com (JIRA)

unread,
Aug 16, 2019, 3:33:02 PM8/16/19
to jenkinsc...@googlegroups.com

They released a new version last week that works as long as you specify the engineVersion parameter on the VaultSecret class.

josephp90@gmail.com (JIRA)

unread,
Aug 20, 2019, 1:49:02 AM8/20/19
to jenkinsc...@googlegroups.com
Joseph Petersen resolved as Fixed
 
Reply all
Reply to author
Forward
0 new messages