New auth plugin = me banging my head against a wall!

[randal cobb]

Hello, all!

I'm attempting to write my first Auth plugin for Vault.  In my organization, we have some sort of integration with our domain roles and AWS roles, that has been hidden in a black box by my organization's security team, so I can't really use the built-in AWS credentials plugin already installed in Vault by default (or I wouldn't even attempt this).  So, with the blessing of my org's security team, I do have permission to leverage the microservices they created for generating AWS auth ID and SECRET keys.  So, I am writing a plugin that leverages those services so I can do token-based generation of AWS keys for automation purposes.

Ok, with that wee-bit of background out there.  My problem is (and I think I've only seen one real post on this from googling, but it didn't really help me).   ok, I have my plugin stubbed out, building, and "enabling" in a DEV Vault instance.  However, I'm getting a warning when I enable the plugin in Vault as shown here:

2018-05-23T17:58:47.633-0700 [DEBUG] auth.plugin.auth_plugin_dac366d1.myauth-plugin: starting plugin: metadata=true path=/Users/me/go/src/gitlab.mycompany.com/sse/vault-mynewvault-plugin/myauth-plugin args=[/Users/me/go/src/gitlab.mycompany.com/sse/vault-mynewvault-plugin/myauth-plugin]
2018-05-23T17:58:47.636-0700 [DEBUG] auth.plugin.auth_plugin_dac366d1.myauth-plugin: waiting for RPC address: metadata=true path=/Users/me/go/src/gitlab.mycompany.com/sse/vault-mynewvault-plugin/myauth-plugin
2018-05-23T17:58:47.653-0700 [DEBUG] auth.plugin.auth_plugin_dac366d1.myauth-plugin.vmyauth-plugin: plugin address: metadata=true address=/var/folders/lc/mtty42rj4ddcdzsm68zbzmpwdklbbr/T/plugin177935374 network=unix timestamp=2018-05-23T17:58:47.653-0700
2018-05-23T17:58:47.656-0700 [WARN ] auth.plugin.auth_plugin_dac366d1.myauth-plugin: error closing client during Kill: metadata=true err="rpc error: code = Canceled desc = grpc: the client connection is closing"
2018-05-23T17:58:47.658-0700 [DEBUG] auth.plugin.auth_plugin_dac366d1.myauth-plugin: plugin process exited: metadata=true path=/Users/me/go/src/gitlab.mycompany.com/sse/vault-mynewvault-plugin/myauth-plugin
2018-05-23T17:58:47.658-0700 [INFO ] core: enabled credential backend: path=myauth type=plugin

** Line 4 - highlighted above is a concern.

However, given that warning in the Vault debug output, I still get a success message back from the Vault client telling me the plugin has been enabled.  So far, it seems to be OK until I try to USE the new plugin.  The plugin is configured to accept either a UserId/Password combination, or a previously issued Vault token for a valid UserId/Password combination generated during an Okta login.  However, no matter what or how many of the valid login credentials into the plugin:

vault write -f auth/myauth/login username=foo@mycompany.com password=bar

-or-

vault write -f auth/myauth/login token=<somevalidtoken>

-or even-

vault write -f auth/myauth/login invalid=invalid

I get an error returned by the Vault client:

Error writing data to auth/myauth/login: Error making API request.


URL: PUT http://localhost:8200/v1/auth/myauth/login
Code: 500. Errors:


* plugin exited before we could connect

And an error sent to the Vault server's Debug logs:

2018-05-23T17:59:27.184-0700 [DEBUG] auth.plugin.auth_plugin_dac366d1.myauth-plugin: starting plugin: path=/Users/me/go/src/gitlab.mycompany.com/sse/vault-mynewvault-plugin/myauth-plugin args=[/Users/me/go/src/gitlab.mycompany.com/sse/vault-amynewvault-plugin/myauth-plugin]
2018-05-23T17:59:27.187-0700 [DEBUG] auth.plugin.auth_plugin_dac366d1.myauth-plugin: waiting for RPC address: path=/Users/me/go/src/gitlab.mycompany.com/sse/vault-mynewvault-plugin/myauth-plugin
2018-05-23T17:59:27.227-0700 [ERROR] auth.plugin.auth_plugin_dac366d1.myauth-plugin.vmyauth-plugin: plugin tls init: error="error during token unwrap request: Error making API request.
 
URL: PUT http://127.0.0.1:8200/v1/sys/wrapping/unwrap
Code: 403. Errors:
 
* permission denied" timestamp=2018-05-23T17:59:27.227-0700
2018-05-23T17:59:27.229-0700 [DEBUG] auth.plugin.auth_plugin_dac366d1.myauth-plugin: plugin process exited: path=/Users/me/go/src/gitlab.mycompany.com/sse/vault-mynewvault-plugin/myauth-plugin

Any suggestions on where I need to look to help resolve this?   Oh, and FWIW, this is on a local DEV instance running Vault 0.10.1 on a Mac OS-X 10.13.2 and the plugin was built with Go go1.10.1 darwin/amd4, if that matters any.

Thanks in advance!

Randy

[bkas...@hashicorp.com]

Hi Randy, 

How are you running the vault server? Was it run with: vault server -dev or was there a config file? If there was a config could you provide it here?

Also that first issue (that is highlighted) is normal, Vault will run the plugin initially to get some various bits of information from it then terminate it.

Best,

Brian

[randal cobb]

Brian,

Thanks for the quick reply!   I'm running vault in dev mode, but also using a config file (mostly to tell Vault where to find plugins).  Here are both the launch command, and .hcl file:

vault server -log-level=debug -dev -dev-root-token-id="root" -config=./vault.hcl

and the vault.hcl file:

plugin_directory = "/Users/me/go/src/gitlab.mycompany.com/sse/vault-anewvault-plugin"

storage "inmem" {

  redirect_addr = "http://10.11.214.61:8200"

}

listener "tcp" {

  address = "10.11.214.61:8200"

  tls_disable = true

}

I added the "inmem" storage tag only to try adding the "redirect_addr" flag that I saw resolved this issue for someone else; something about it needed a callback address so the plugin knows where to call vault at instead of assuming localhost, only it made no difference and I haven't cleared out the config file, yet.

Cheers!

Randy

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/vault/issues
IRC: #vault-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Vault" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/9295cc49-fa3e-496e-b033-1c1f0658da87%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[bkas...@hashicorp.com]

Based on the error you're seeing:

2018-05-23T17:59:27.227-0700 [ERROR] auth.plugin.auth_plugin_dac366d1.myauth-plugin.vmyauth-plugin: plugin tls init: error="error during token unwrap request: Error making API request.
 
URL: PUT http://127.0.0.1:8200/v1/sys/wrapping/unwrap
Code: 403. Errors:
 
* permission denied" timestamp=2018-05-23T17:59:27.227-0700
2018-05-23T17:59:27.229-0700 [DEBUG] auth.plugin.auth_plugin_dac366d1.myauth-plugin: plugin process exited: path=/Users/me/go/src/gitlab.mycompany.com/sse/vault-mynewvault-plugin/myauth-plugin

It doesn't look like it's correctly making the call to the IP you specified. Could you post the block of info that is output at the top of the vault logs when the server is started?

[randal cobb]

Well, it looks like it's listening on both localhost AND my network IP.

```

vault server -log-level=debug -dev -dev-root-token-id="root" -config=./vault.hcl

==> Vault server configuration:

             Api Address: http://10.11.214.61:8200

                     Cgo: disabled

         Cluster Address: https://10.11.214.61:8201

              Listener 1: tcp (addr: "127.0.0.1:8200", cluster address: "127.0.0.1:8201", tls: "disabled")

              Listener 2: tcp (addr: "10.11.214.61:8200", cluster address: "10.11.214.61:8201", tls: "disabled")

               Log Level: debug

                   Mlock: supported: false, enabled: false

                 Storage: inmem

                 Version: Vault v0.10.1

             Version Sha: 756fdc4587350daf1c65b93647b2cc31a6f119cd

WARNING! dev mode is enabled! In this mode, Vault runs entirely in-memory
```

To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/fc364090-c7d7-46e8-b912-c403be3ff47c%40googlegroups.com.

[bkas...@hashicorp.com]

Could you remove all the configuration other than the plugin_directory and report back if that fixes the issue?

[randal cobb]

Sorry for the delay in replying; I've been in meetings all morning.

Reducing the vault.hcl to ONLY contain the single line of "plugin_location=.." results in the exact same error when trying to use the login endpoint:

... plugin tls init: error="error during token unwrap request: Error making API request.

URL: PUT http://127.0.0.1:8200/v1/sys/wrapping/unwrap

Code: 403. Errors:

* permission denied" timestamp=2018-05-25T11:13:45.249-0700 ...

To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/0c576dcd-74fb-4094-a620-ee7c57af5621%40googlegroups.com.

[Becca Petrin]

Hi Randal,

I'm going to take a stab at this as I recently wrote my first plugin, albeit a secrets plugin and not an auth plugin.

This is unconventional, but I started by writing the plugin on a branch of the main Vault repo as though it was going to be directly included in Vault. Once I had it up and running and working there, only then did I move it over to being a plugin, which helped me isolate where things were and weren't working.

When I did move it over to being a separate plugin, I used a config with only the plugin directory, just like you're doing, so that seems good. I also added and enabled the plugin like so:

# "command" is the name of the binary here

$ vault write sys/plugins/catalog/ad \
    sha_256=bdd1fc640c48f7d7c3d132b75ee1167cf8c3e47a7d89cc5582f8af863e415df8 \
    command="vault-plugin-secrets-ad"
   
$ vault secrets enable -plugin-name='ad' plugin

I'm sure you've looked at other plugin repos, and I'm sure there are better examples out there, but here's the one that I did in case it happens to help: https://github.com/hashicorp/vault-plugin-secrets-ad.

I don't know if any of that so far is helpful to you but I thought I'd share just in case I can help you keep moving. I know that Brian is away today so you probably won't hear more until next week.

-B

[randal cobb]

Becca,

Thanks for the assistance, I'll definitely take a look and see where my plugin may have diverged from the norm.   I'm certain I've followed the same procedure (I've done it enough that I've created a small script to do it for me) to register and enable my plugin; with the only real differences being how its enabled within Vault (secrets vs. auth plugin).   I suspect, at this point, that it has to be something in my code, but I've basically cloned the example plugin code, added two additional fields and renamed the existing one, then built and deployed.  

At this point, there hasn't been much effort expended to make it work for my needs, so I may just do exactly what you did and start by building it right within the codebase, then separate it later.  I'll give my brain a rest over the weekend and see what happens on Tuesday.

Thanks, again, for the help/suggestion!

To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/26018564-a624-41b4-9fc1-12bddea90692%40googlegroups.com.

[bkas...@hashicorp.com]

Hi Randal,

I was able to successfully run this plugin https://github.com/hashicorp/vault-auth-plugin-example with the latest version of vault. Did you make any changes to the main file of your plugin?

Best,

Brian

[jo...@eximchain.com]

Hey Randal, you ever get around to fixing this?  Running into the same error, not sure what's going on.

[jo...@eximchain.com]

In case somebody else runs into this, the underlying error wasn't related to that message.  Try running the server with log-level set to DEBUG.  For me, that revealed the actual SegFault which was making the plugin process panic, leading to it shutting down and Vault saying the connection was dying.  That SegFault ended up going back to a nil pointer error.

Good luck to anyone else who finds their way down this rabbit hole!

[Jack Kleeman]

I think this is caused by running the vault server with `VAULT_TOKEN` in your environment. This leads the plugin to receive this token, and attempt to use it when it does the unwrap request. In my case, the token was invalid. This fits with Randal saying he was running in dev mode, as was I.

[Steven Callister]

I just wanted to chime in and say I also hit this same exact problem while in dev mode. My terminal had an existing vault token that was different than the dev token.

Should this get reported as a bug, or is this working as intended?

[Brian Kassouf]

I actually think this was fixed fairly recently in: https://github.com/hashicorp/vault/pull/8058

May want to try again with a binary built off of vault master! 

Best,

Brian

To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/16f5e830-0bf6-4686-97f1-7357fbd72652%40googlegroups.com.

[Steven Callister]

Oh nice, thanks! Way to go Hashicorp! :)

To unsubscribe from this group and stop receiving emails from it, send an email to vault...@googlegroups.com.