Custom Vault plugin exiting on windows with strconv.ParseInt: parsing "": invalid syntax error

Alexander Rykalin

unread,

Apr 9, 2019, 12:01:17 PM4/9/19

to Vault

Hi all, I'm developing a plugin https://github.com/Venafi/vault-pki-backend-venafi , it's doing mostly the same as PKI plugin do, but getting certificates from Venafi Platform. This plugin have windows version, but when I'm trying to enable it on windows I'm getting this error:

C:\Users\ar\Documents\vault>.\vault write -f sys/plugins/catalog/venafi-pki-backend sha_256="20464ec079a373f897a7a9b7d8dc6ffad2aacb90dbf078232f84c0d180041ae6" command="vault_pki_backend_venafi.exe"
Error writing data to sys/plugins/catalog/venafi-pki-backend: Error making API request.


URL: PUT http://127.0.0.1:8200/v1/sys/plugins/catalog/venafi-pki-backend
Code: 500. Errors:


* 1 error occurred:
        * rpc error: code = Unknown desc = timeout waiting for connection info






C:\Users\ar\Documents\vault>                                                                                                                ↑

In the server logs I see:

2019-04-09T18:56:31.250+0300 [DEBUG] rollback: attempting rollback: path=secret/
2019-04-09T18:56:58.435+0300 [WARN]  received plugin exited before we could connect attempting as db plugin, attempting as auth/secret plugin
2019-04-09 18:56:58.581346 I | [ERR] plugin: plugin acceptAndServe error: strconv.ParseInt: parsing "": invalid syntax
2019-04-09T18:57:31.250+0300 [DEBUG] rollback: attempting rollback: path=auth/token/

Vault version:

C:\Users\ar\Documents\vault>vault version
Vault v1.1.0 ('36aa8c8dd1936e10ebd7a4c1d412ae0e6f7900bd')

Go version 1.12.3

Integration tests which are using vault.NewTestCluster passing fine.

Where I can look to find the problem?

Thanks,

Alex

Becca Petrin

unread,

Apr 9, 2019, 12:04:55 PM4/9/19

to Vault

Hi Alexander,

Part of the problem is with vault write -f sys/plugins/catalog/venafi-pki-backend. Try vault write -f sys/plugins/catalog/database/venafi-pki-backend.

Some of the failure output is that Vault is trying to figure out what type of plugin it is and it can't. I've noticed that seems to generally happen with database plugins, as I'm presently developing one as well.

Hopefully that's all you need but let us know if there are more problems! Also, what version of Vault you're on if so. Thanks!

-Becca

Becca Petrin

unread,

Apr 9, 2019, 12:09:13 PM4/9/19

to Vault

Also, I should add, my answer presumes you're developing a database plugin. Perhaps you're developing an "auth" or "secret" plugin, I'm not certain. The full docs are here: https://www.vaultproject.io/api/system/plugins-catalog.html

Alexander Rykalin

unread,

Apr 9, 2019, 12:54:58 PM4/9/19

to Vault

Thank you. This helped with writing a plugin to plugin catalog, but now I'm getting same error when trying to enable it:

C:\Users\ar\Documents\vault>vault secrets enable -path=venafi-pki -plugin-name=vault-pki-backend-venafi plugin
Error enabling: Error making API request.

URL: POST http://127.0.0.1:8200/v1/sys/mounts/venafi-pki
Code: 400. Errors:


* rpc error: code = Unknown desc = timeout waiting for connection info

in server logs:

2019-04-09T19:52:38.399+0300 [DEBUG] rollback: attempting rollback: path=secret/
2019-04-09T19:52:38.399+0300 [DEBUG] rollback: attempting rollback: path=cubbyhole/
2019-04-09T19:52:48.230+0300 [DEBUG] secrets.vault-pki-backend-venafi.vault-pki-backend-venafi_13fe848f.vault-pki-backend-venafi: starting plugin: metadata=true path=C:\Users\ar\Documents\vault\bin\go_build_github_com_Venafi_vault_pki_backend_venafi.exe args=[C:\Users\alexander.rykalin\Documents\vault\bin\go_build_github_com_Venafi_vault_pki_backend_venafi.exe]
2019-04-09T19:52:48.239+0300 [DEBUG] secrets.vault-pki-backend-venafi.vault-pki-backend-venafi_13fe848f.vault-pki-backend-venafi: plugin started: metadata=true path=C:\Users\ar\Documents\vault\bin\go_build_github_com_Venafi_vault_pki_backend_venafi.exe pid=17596
2019-04-09T19:52:48.239+0300 [DEBUG] secrets.vault-pki-backend-venafi.vault-pki-backend-venafi_13fe848f.vault-pki-backend-venafi: waiting for RPC address: metadata=true path=C:\Users\ar\Documents\vault\bin\go_build_github_com_Venafi_vault_pki_backend_venafi.exe
2019-04-09T19:52:48.306+0300 [DEBUG] secrets.vault-pki-backend-venafi.vault-pki-backend-venafi_13fe848f.vault-pki-backend-venafi.go_build_github_com_Venafi_vault_pki_backend_venafi.exe: 2019/04/09 19:52:48 Starting plugin: metadata=true
2019-04-09T19:52:48.307+0300 [DEBUG] secrets.vault-pki-backend-venafi.vault-pki-backend-venafi_13fe848f.vault-pki-backend-venafi: using plugin: metadata=true version=4
2019-04-09T19:52:48.307+0300 [DEBUG] secrets.vault-pki-backend-venafi.vault-pki-backend-venafi_13fe848f.vault-pki-backend-venafi.go_build_github_com_Venafi_vault_pki_backend_venafi.exe: plugin address: metadata=true address=127.0.0.1:10002 network=tcp timestamp=2019-04-09T19:52:48.307+0300
2019-04-09T19:52:48.311+0300 [TRACE] secrets.vault-pki-backend-venafi.vault-pki-backend-venafi_13fe848f.vault-pki-backend-venafi: setup: transport=gRPC status=started
2019-04-09 19:52:48.311328 I | [ERR] plugin: plugin acceptAndServe error: strconv.ParseInt: parsing "": invalid syntax
2019-04-09T19:52:53.313+0300 [TRACE] secrets.vault-pki-backend-venafi.vault-pki-backend-venafi_13fe848f.vault-pki-backend-venafi: setup: transport=gRPC status=finished err="rpc error: code = Unknown desc = timeout waiting for connection info" took=5.0024786s
2019-04-09T19:52:53.313+0300 [ERROR] secrets.system.system_a5208107: mount failed: path=venafi-pki/ error="rpc error: code = Unknown desc = timeout waiting for connection info"
2019-04-09T19:53:38.400+0300 [DEBUG] rollback: attempting rollback: path=auth/token/
2019-04-09T19:53:38.400+0300 [DEBUG] rollback: attempting rollback: path=sys/

Need to mention that this plugin works perfectly well on Linux.

вторник, 9 апреля 2019 г., 19:04:55 UTC+3 пользователь Becca Petrin написал:

Becca Petrin

unread,

Apr 9, 2019, 1:47:58 PM4/9/19

to Vault

Interesting! If I had to guess, the error is originating from here: https://github.com/hashicorp/vault/blob/master/vendor/github.com/hashicorp/go-plugin/grpc_broker.go#L313. Tracing that back, it's still not evident to me why you'd get that error, especially on one OS vs. the other.

Would you be willing to share your Vault version, as well as the contents of your Vault config? With all sensitive data redacted, of course.

Becca Petrin

unread,

Apr 9, 2019, 1:49:02 PM4/9/19

to Vault

Oh I see you've posted your Vault version, that's helpful. Just the config additionally, perhaps, then.

Alexander Rykalin

unread,

Apr 10, 2019, 3:01:59 AM4/10/19

to Vault

Config is very simple:

plugin_directory = "bin"

I'm launching Vault in dev mode

vault server -log-level=debug -dev -config=vault-config.hcl

вторник, 9 апреля 2019 г., 20:49:02 UTC+3 пользователь Becca Petrin написал:

Becca Petrin

unread,

Apr 10, 2019, 12:20:51 PM4/10/19

to Vault

It may be that the api_addr needs to be set. https://www.vaultproject.io/docs/configuration/#api_addr. Does that solve it?

Alexander Rykalin

unread,

Apr 11, 2019, 5:57:58 AM4/11/19

to Vault

No, unfortunately not :(

Here is full vault log, maybe it will help: https://pastebin.com/Zn29L1dh

Config:

plugin_directory = "C:/Users/domain.user/Documents/vault/bin"
api_addr = "http://127.0.0.1:8200"

Commands log:

C:\Users\domain.user\Documents\vault>.\vault write -f sys/plugins/catalog/secret/vault-pki-backend-venafi sha_256="20464ec079a373f
Success! Data written to: sys/plugins/catalog/secret/vault-pki-backend-venafi

C:\Users\domain.user\Documents\vault>vault secrets enable -path=venafi-pki -plugin-name=vault-pki-backend-venafi plugin

Error enabling: Error making API request.

URL: POST http://127.0.0.1:8200/v1/sys/mounts/venafi-pki
Code: 400. Errors:

* rpc error: code = Unknown desc = timeout waiting for connection info

C:\Users\domain.user\Documents\vault>

среда, 10 апреля 2019 г., 19:20:51 UTC+3 пользователь Becca Petrin написал:

Becca Petrin

unread,

Apr 11, 2019, 11:53:13 AM4/11/19

to Vault

Hmmm, you may be hitting the issue here: https://github.com/hashicorp/go-plugin/pull/97

Would you try ensuring that dependency is updated to past that fix?

-B

Alexander Rykalin

unread,

Apr 12, 2019, 7:25:32 AM4/12/19

to Vault

Hi, I updated plugin to the latest version but unfortunately it didn't helped :( Error remains the same:

2019-04-12T14:23:47.075+0300 [DEBUG] secrets.vault-pki-backend-venafi.vault-pki-backend-venafi_83e6e933.vault-pki-backend-venafi: starting


 plugin: metadata=true path=C:\Users\ar\Documents\vault\bin\go_build_github_com_Venafi_vault_pki_backend_venafi.exe args=[C

:\Users\ar\Documents\vault\bin\go_build_github_com_Venafi_vault_pki_backend_venafi.exe]

2019-04-12T14:23:47.083+0300 [DEBUG] secrets.vault-pki-backend-venafi.vault-pki-backend-venafi_83e6e933.vault-pki-backend-venafi: plugin s

tarted: metadata=true path=C:\Users\ar\Documents\vault\bin\go_build_github_com_Venafi_vault_pki_backend_venafi.exe pid=5524



2019-04-12T14:23:47.083+0300 [DEBUG] secrets.vault-pki-backend-venafi.vault-pki-backend-venafi_83e6e933.vault-pki-backend-venafi: waiting


for RPC address: metadata=true path=C:\Users\ar\Documents\vault\bin\go_build_github_com_Venafi_vault_pki_backend_venafi.exe

2019-04-12T14:23:47.154+0300 [DEBUG] secrets.vault-pki-backend-venafi.vault-pki-backend-venafi_83e6e933.vault-pki-backend-venafi: using pl

ugin: metadata=true version=4

2019-04-12T14:23:47.154+0300 [DEBUG] secrets.vault-pki-backend-venafi.vault-pki-backend-venafi_83e6e933.vault-pki-backend-venafi.go_build_

github_com_Venafi_vault_pki_backend_venafi.exe: plugin address: metadata=true address=127.0.0.1:10002 network=tcp timestamp=2019-04-12T14:

23:47.112+0300

2019-04-12T14:23:47.154+0300 [TRACE] secrets.vault-pki-backend-venafi.vault-pki-backend-venafi_83e6e933.vault-pki-backend-venafi: setup: t

ransport=gRPC status=started

2019-04-12 14:23:47.157400 I | [ERR] plugin: plugin acceptAndServe error: strconv.ParseInt: parsing "": invalid syntax

2019-04-12T14:23:52.161+0300 [TRACE] secrets.vault-pki-backend-venafi.vault-pki-backend-venafi_83e6e933.vault-pki-backend-venafi: setup: t

ransport=gRPC status=finished err="rpc error: code = Unknown desc = timeout waiting for connection info" took=5.0072557s

2019-04-12T14:23:52.161+0300 [ERROR] secrets.system.system_84e204b1: mount failed: path=venafi-pki/ error="rpc error: code = Unknown desc


= timeout waiting for connection info"

2019-04-12T14:24:03.169+0300 [DEBUG] rollback: attempting rollback: path=auth/token/

2019-04-12T14:24:03.169+0300 [DEBUG] rollback: attempting rollback: path=identity/

2019-04-12T14:24:03.169+0300 [DEBUG] rollback: attempting rollback: path=sys/

2019-04-12T14:24:03.169+0300 [DEBUG] rollback: attempting rollback: path=cubbyhole/

2019-04-12T14:24:03.169+0300 [DEBUG] rollback: attempting rollback: path=secret/

четверг, 11 апреля 2019 г., 18:53:13 UTC+3 пользователь Becca Petrin написал:

Becca Petrin

unread,

Apr 12, 2019, 12:50:20 PM4/12/19

to Vault

Hi Alexander,

What are the contents of your vault-config.hcl?

One thing I notice is that from the config snippets you've posted, Vault is expected to be running on localhost at port 8200. However, I do also see some log output noting the plugin address as being at port 10002. I'm wondering if that may be the issue.

Also, what version of Windows are you on?

-B

Alexander Rykalin

unread,

Apr 15, 2019, 9:21:50 AM4/15/19

to Vault

It is full config I used on vault:

plugin_directory = "C:/Users/domain.user/Documents/vault/bin"
api_addr = "http://127.0.0.1:8200"

I'm using it for the development, so it really small. I'm testing on Windows 10

Regarding listener. As I understand, according to Hashicorp's go-plugin system vault is starting plugins on different address than Vault iteself - https://github.com/hashicorp/go-plugin/blob/master/server.go#L357-L360 On Linux it is unix socets and on windows it is tcp socket. Not sure if it is a problem, maybe I should try to launch other Vault plugins on windows.

пятница, 12 апреля 2019 г., 19:50:20 UTC+3 пользователь Becca Petrin написал:

Alexander Rykalin

unread,

Apr 15, 2019, 9:38:11 AM4/15/19

to Vault

Yes, you're right, the problem was with the listener. I investigated the code of go-plugin. To start plugin listener on Windows it needs two variables to be set

	minPort, err := strconv.ParseInt(os.Getenv("PLUGIN_MIN_PORT"), 10, 32)
	if err != nil {
		return nil, err
	}

	maxPort, err := strconv.ParseInt(os.Getenv("PLUGIN_MAX_PORT"), 10, 32)
	if err != nil {
		return nil, err
	}

https://github.com/hashicorp/go-plugin/blob/master/server.go#L365-L385

понедельник, 15 апреля 2019 г., 16:21:50 UTC+3 пользователь Alexander Rykalin написал:

Becca Petrin

unread,

Apr 15, 2019, 12:48:03 PM4/15/19

to Vault

Interesting! Good find! I was unaware of that.

I'll take a look as well and add that to our docs. Thanks for posting the solution.

-Becca

Becca Petrin

unread,

Apr 15, 2019, 12:58:40 PM4/15/19

to Vault

I went ahead and opened an issue here if you want to follow along: https://github.com/hashicorp/vault/issues/6587

Reply all

Reply to author

Forward