vault ssh backend problem

[Esai]

I installed the latest vault and ssh-helper but things don't seem to work.

All of the hostnames are fqdn's resolving to the external ip, the vault and helper run in the same vm.
The vault is unsealed, the tutorial(https://www.vaultproject.io/docs/secrets/ssh/index.html) commands have all been executed. PAM is clearly configured, the error comes from the application.

root@vault:~# vault version
  Vault v0.6.5 ('5d8d702f33b5fd965cbe8d6d0728295de813a196')
root@vault:~# /usr/local/bin/vault-ssh-helper -version
  vault-ssh-helper v0.1.3

Verify says everything is ok:
root@vault:~# /usr/local/bin/vault-ssh-helper -verify-only -config=/etc/vault-ssh-helper.d/config.hcl
  2017/03/02 12:48:04 [INFO] using SSH mount point: ssh
  2017/03/02 12:48:04 [INFO] vault-ssh-helper verification successful!


Using sshpass I get:
# vault ssh -role otp_key_role root@vault
  Failed to establish SSH connection: "exit status 5"

Without it I can enter the OTP myself:
root@vault:~# vault ssh -role otp_key_role root@vault
  OTP for the session is d247980b-0371-c318-0e5d-aaa9aaa6d549
  [Note: Install 'sshpass' to automate typing in OTP]
  Password:
  Password:
  Password:
  Error while running ssh command: "exit status 255"

At first in the log I saw :
  2017/03/02 12:28:46 [INFO] using SSH mount point: ssh
  2017/03/02 12:28:46 [ERROR]: uuid is improperly formatted



Then I tried an older version of vault-ssh-helper, v0.1.2 and I am getting:
  2017/03/02 12:39:04 [INFO] using SSH mount point: ssh
  2017/03/02 12:39:04 [ERROR]: Error making API request.
 
  URL: PUT https://vault:8200/v1/ssh/verify
  Code: 400. Errors:
 
  * OTP not found


Config files:
Vault:
backend "file" {
  path = "/mnt/storage-pool/"
}

listener "tcp" {
  address = "0.0.0.0:8200"
  tls_disable = 0
  tls_cert_file = "/etc/letsencrypt/live/vault/fullchain.pem"
  tls_key_file = "/etc/letsencrypt/live/vault/privkey.pem"
}

Helper:
vault_addr = "https://vault:8200"
ssh_mount_point = "ssh"
ca_path = "/etc/ssl/certs/"
tls_skip_verify = false
allowed_roles = "*"


Manually generating and requesting:

root@vault:~# vault write ssh/creds/otp_key_role ip=a.a.a.a
Key                Value
---                -----
lease_id           ssh/creds/otp_key_role/ffa183a5-7b65-5d22-0358-ae43ad45c20b
lease_duration     768h0m0s
lease_renewable    false
ip                 a.a.a.a
key                752f4886-6a56-a033-48e7-26b47f62cf8f
key_type           otp
port               22
username           root

root@vault:~# curl -s -X PUT -H 'Content-Type: application/json' -d @otp.json https://vault:8200/v1/ssh/verify
{"errors":["OTP not found"]}


What should I do? The vault side doesn't throw any errors but the OTP isn't accessible from API.

[Vishal Nayak]

Hi Esai,

Please do not downgrade vault-ssh-helper. Earlier version had a
security problem which has been fixed in the latest version.

If the machine is public facing, I guess `uuid is improperly
formatted` is coming from failed login attempts from the outside
world.

`OTP not found` is very strange. I might be wrong but it is not a bad
idea to check if the OTP is generated and queried from the same Vault.
I see that you are using CLI for OTP generation and curl for querying
the OTP. What is your VAULT_ADDR set to? Can you do both using either
the CLI or the curl?

For querying the OTP from CLI, use this `vault write ssh/verify otp=<key>`

Regards,
Vishal

> --
> This mailing list is governed under the HashiCorp Community Guidelines -
> https://www.hashicorp.com/community-guidelines.html. Behavior in violation
> of those guidelines may result in your removal from this mailing list.
>
> GitHub Issues: https://github.com/hashicorp/vault/issues
> IRC: #vault-tool on Freenode
> ---
> You received this message because you are subscribed to the Google Groups
> "Vault" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to vault-tool+...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/vault-tool/bf103eaa-bc45-4b3e-af3a-c7abd66c83e7%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



--
vn

[Esai]

I upgraded back to 0.1.3

root@vault:~# vault write ssh/creds/otp_key_role ip=a.a.a.a
Key             Value
---             -----

lease_id        ssh/creds/otp_key_role/f3bed9bb-5841-3899-6708-237fc18b61dd

lease_duration  768h0m0s
lease_renewable false
ip              a.a.a.a

key             2b0488ff-afe4-32a3-0400-dfb08e06d6f0

key_type        otp
port            22
username        root

root@vault:~# vault write ssh/verify otp=2b0488ff-afe4-32a3-0400-dfb08e06d6f0
Key             Value
---             -----
ip              a.a.a.a
role_name       otp_key_role

username        root

root@vault:~# curl -s -X PUT -H 'Content-Type: application/json' -d @otp.json https://vault:8200/v1/ssh/verify
{"errors":["OTP not found"]}

root@vault:~# ssh ro...@a.a.a.a
Password:
/usr/local/bin/vault-ssh-helper failed: exit code 1


No, the attempts are certainly not from anyone else. I can correlate the log message to my own try.
Also there are 2 types of messages: "uuid is improperly formatted" is the one I'm generating and "uuid string is wrong length" is someone else's try.

The uuid format message was added in the latest version, which is why I tried downgrading.

Vault address is manually set to https://fqdn:8200/.

[Vishal Nayak]

Hi Esai,

The fact that `vault write ssh/verify
otp=2b0488ff-afe4-32a3-0400-dfb08e06d6f0` is returning the role name
and IP asserts that the server is working properly.

Any subsequent requests to verify the OTP will fail because it can
only be verified once, hence the name OTP. Post verification of OTP,
failing to curl and failing to login is understandable. What happens
if you curl before `vault write ssh/verify`? What is the value of
`VAULT_ADDR`?

Both "uuid is improperly formatted" and "uuid string is wrong length"
can be the result of someone else's try; they both fail to qualify to
be the format of the UUID the library expects.

Regards,
Vishal

> https://groups.google.com/d/msgid/vault-tool/86e1fc9f-7e2e-47a9-8fa5-9e6b79ea1db9%40googlegroups.com.

[Esai]

Curl gets the same response before verify as well, you can see that at the end of the first post.
I tried multiple times, I understand how OTP is implemeted in this context.

It seems as the API isn't responding properly to curl.

Are there any other requests I could make that would shed some light on this mess?

I have set the env variable as:
VAULT_ADDR=https://vault.mydomain.tld:8200/

My server has gotten suprisingly little traffic. I have also enabled rate limiting, which keeps requests to a minimum.
Around 80% of the auth requests are from me, I can see log lines appearing as I try to authenticate, they correlate exactly.

[Vishal Nayak]

Hi Esai,

CLI does nothing different than the curl does. It hits the same API.

I am still inclined to believe that curl is not querying the right Vault.

After generating the OTP, please check if you get a successful
response with this:
curl -XPOST -H "X-Vault-Token:<token>"
http://vault.mydomain.tld:8200/v1/ssh/verify -d '{"otp":"<key>"}'

Regards,
Vishal

> https://groups.google.com/d/msgid/vault-tool/665eebf4-6929-47a3-892e-6b76f4623bf2%40googlegroups.com.

[Jeff Mitchell]

Hi Esai,

I'd be interested in knowing if it works as Vishal suggested to try in his previous email, because my suspicion is that there is an issue with reading in your JSON file...either its formatting, or perhaps there's extra whitespace in there around the otp that you don't realize, or something of the sort.

Best,

Jeff

> email to vault-tool+unsubscribe@googlegroups.com.

> To view this discussion on the web visit
> https://groups.google.com/d/msgid/vault-tool/665eebf4-6929-47a3-892e-6b76f4623bf2%40googlegroups.com.
>
> For more options, visit https://groups.google.com/d/optout.



--
vn

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.

GitHub Issues: https://github.com/hashicorp/vault/issues
IRC: #vault-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Vault" group.

To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/CAFy0tpyB7sEPNOuh_6tKpgmkyp8Y1jc5c70aBFSm9arG8JFUWA%40mail.gmail.com.

[Esai]

root@vault:~# vault write ssh/creds/otp_key_role ip=a.a.a.a
Key             Value
---             -----

lease_id        ssh/creds/otp_key_role/c275913a-b44c-4e1d-4e2d-f46e9da7f451

lease_duration  768h0m0s
lease_renewable false
ip              a.a.a.a

key             a2714263-2557-6364-5688-042722a44708

key_type        otp
port            22
username        root

root@vault:~# curl -XPOST -H "X-Vault-Token:70b42c80-22b6-ba04-889c-854357f2248c"
curl: no URL specified!
curl: try 'curl --help' or 'curl --manual' for more information
root@vault:~# curl -XPOST -H "X-Vault-Token:70b42c80-22b6-ba04-889c-854357f2248c" https://vault.mydomain.tld:8200/v1/ssh/verify -d '{"otp":"a2714263-2557-6364-5688-042722a44708"}'                                     
{"request_id":"ba518083-443a-ac35-12b5-e85bf371df77","lease_id":"","renewable":false,"lease_duration":0,"data":{"ip":"a.a.a.a","role_name":"otp_key_role","username":"root"},"wrap_info":null,"warnings":null,"auth":null}

root@vault:~# vault write ssh/creds/otp_key_role ip=a.a.a.a
Key             Value
---             -----

lease_id        ssh/creds/otp_key_role/4a87842a-faf1-578d-4b57-8c733983072b

lease_duration  768h0m0s
lease_renewable false
ip              a.a.a.a

key             eae0681a-3ea0-2b4d-02f3-33c8d74e9bb2

key_type        otp
port            22
username        root

root@vault:~# curl -XPOST https://vault.mydomain.tld:8200/v1/ssh/verify -d '{"otp":"a2714263-2557-6364-5688-042722a44708"}'                                                                                         
{"errors":["OTP not found"]}


So this error appears without authentication, it seems. Maybe the ssh-helper also needs some kind of auth mechanism? I can't see anything in the github readme.
There are no extra whitespaces, I did the process through again, and my configs are in the first post, they contain no formating errors that I can notice.

[Vishal Nayak]

Hi Esai,

"/ssh/verify" is an unauthenticated endpoint and it does not require a
Vault token to be supplied.

I was able to reproduce this issue. Try providing a wrong token there,
and it would still work!

I am investigating this.

Regards,
Vishal

> https://groups.google.com/d/msgid/vault-tool/09946c1c-02d2-4c85-af88-d712edd04618%40googlegroups.com.

[Vishal Nayak]

Hi Esai,

Apparently I made the same mistake you did while reproducing it. Using
the wrong OTP for verification.

root@vault:~# vault write ssh/creds/otp_key_role ip=a.a.a.a
Key Value
--- -----
lease_id ssh/creds/otp_key_role/4a87842a-faf1-578d-4b57-8c733983072b
lease_duration 768h0m0s
lease_renewable false
ip a.a.a.a
key eae0681a-3ea0-2b4d-02f3-33c8d74e9bb2
key_type otp
port 22
username root

root@vault:~# curl -XPOST
https://vault.mydomain.tld:8200/v1/ssh/verify -d
'{"otp":"a2714263-2557-6364-5688-042722a44708"}'
{"errors":["OTP not found"]}

You should have used "eae0681a-3ea0-2b4d-02f3-33c8d74e9bb2".

Regards,
Vishal

--
vn

[Esai]

Really sorry for that, that was my mistake. Expected my mouse to put the right thing in the buffer.
Now it worked:

root@vault:~# curl -XPOST https://vault.mydomain.tld:8200/v1/ssh/verify -d '{"otp":"9627ebda-7e92-cbc4-a8eb-d7e188858c69"}'
{"request_id":"95af9d43-9edc-3dea-8570-bd98bb77c131","lease_id":"","renewable":false,"lease_duration":0,"data":{"ip":"a.a.a.a","role_name":"otp_key_role","username":"root"},"wrap_info":null,"warnings":null,"auth":null}

But the ssh login still doesn't work(with a new key):
root@vault:~# date && ssh vault
Fri Mar  3 11:23:25 EET 2017

Password:
/usr/local/bin/vault-ssh-helper failed: exit code 1

And a log entry is generated:
*** Fri Mar  3 11:23:27 2017
2017/03/03 11:23:27 [INFO] using SSH mount point: ssh
2017/03/03 11:23:27 [ERROR]: uuid is improperly formatted

Also tried with high verbosity on ssh:
http://sprunge.us/JYON


Is there a way to make the logging more verbose on the helper? I would like to see what kind of a request it generates.
Maybe the pam auth is wrong somehow and garbles the input?

My Debian Jessie /etc/pam.d/sshd has these two lines in the beginning:
auth requisite pam_exec.so expose_authtok debug log=/var/log/vault.log /usr/local/bin/vault-ssh-helper -config=/etc/vault-ssh-helper.d/config.hcl
auth optional pam_unix.so debug not_set_pass use_first_pass nodelay

Turning on debugging on pam modules didn't help, there was nothing interesting I could see.

Mar  3 11:23:27 vault sshd[14073]: pam_exec(sshd:auth): send password to child
Mar  3 11:23:27 vault sshd[14074]: pam_exec(sshd:auth): Calling /usr/local/bin/vault-ssh-helper ...
Mar  3 11:23:27 vault sshd[14073]: pam_exec(sshd:auth): /usr/local/bin/vault-ssh-helper failed: exit code 1

What I did try was this:
root@vault:~# echo "7e917382-b570-4316-8d2c-849e09496d40" | /usr/local/bin/vault-ssh-helper -config /etc/vault-ssh-helper.d/config.hcl; echo "Returned: $?"
2017/03/03 11:55:30 [INFO] using SSH mount point: ssh
2017/03/03 11:55:30 [ERROR]: uuid string is wrong length
Returned: 1

That's strange to me, the length is correct, I expected to see "improperly formatted".
I'm probably missing something.

[Vishal Nayak]

Hi Esai,

`echo` appends a `\n'. Try `echo -n "otp"`.

At this point I would do the following:
1) Change vault_addr in the config file: "https://vault:8200" ->
"https://vault.mydomain.tld:8200"
2) Along with enabling the new PAM config lines, make sure that any
previous authentication method in PAM was disabled; like "#@include
common-auth" (differs with platform). Its possible that previous
authentication mechanism is still active, is consuming the OTP and
echoing its output to stdin which is turning out to be a wrong OTP for
Vault auth.

Regards,
Vishal

> https://groups.google.com/d/msgid/vault-tool/3aae14d6-66fc-4bc8-99ed-ccd91ab7510d%40googlegroups.com.

[Esai]

Removing the new line makes things better.

strace shows this:
root@vault:~# echo -n "e2faa192-35ec-8a7c-c4d9-b03d3aeb7b8b" | strace /usr/local/bin/vault-ssh-helper -config /etc/vault-ssh-helper.d/config.hcl | less
...
futex(0xc42002a910, FUTEX_WAKE, 1)      = 1
clock_gettime(CLOCK_MONOTONIC, {151646, 117555529}) = 0
futex(0x859398, FUTEX_WAIT, 0, {9, 989895597}2017/03/03 12:52:32 [ERROR]: Error making API request.

URL: PUT https://vault.mydomain.tld:8200/v1/ssh/verify

Code: 400. Errors:

* OTP not found

 <unfinished ...>
+++ exited with 1 +++


vault_addr is "https://vault.mydomain.tld:8200"

The pam auth lines are the first lines of the file. common auth is included after these lines.
I also tried cleaning the whole file and using just these two lines, it changed nothing.

[Vishal Nayak]

Hi Esai,

I would keep the common-auth commented out.

Wrt the API failure, have a valid OTP in the server. One thing we have
verified so far is that the API works. You shouldn't be seeing "OTP
not found" anymore.

Regards,
Vishal

> https://groups.google.com/d/msgid/vault-tool/c5fee4dd-d475-4b85-906f-3e89c39388e7%40googlegroups.com.

[Esai]

Managed to work it out finally!

Went over everything again and noticed that I had set PermitRootLogin without-password, which is my standard option everywhere.
Setting that to yes in /etc/ssh/sshd_config and restarting my ssh service I was finally able to log in with the OTP.

You should add this to the documentation :)

Thank you all very much for all of the assistance, it has been a very nice experience.

[Vishal Nayak]

Hi Esai,

Glad that it worked out for you!

Regards,
Vishal

> https://groups.google.com/d/msgid/vault-tool/95d2147a-d524-4de7-b2ff-3e7dbaaf607d%40googlegroups.com.

[da...@devopskata.com]

I struggled with this problem as well, but the root cause was different: 

The user I was trying to use to SSH in DID NOT exist on the SSH server.  I created that user with adduser and it worked.