AWS Login failure - https://ec2.us-east-1.amazonaws.com/: dial tcp 54.239.28.176:443: i/o timeout"

[Elizabeth]

Hi,

    Using a single instance Vault deployment (0.10.1) in a test environment, which we have been using for 2 weeks now. All of a sudden on aws ec2 login, this error shows up - 

{"errors":["failed to verify instance ID: error fetching description for instance ID \"i-0260558d472cea6e4\": RequestError: send request failed\ncaused by: Post https://ec2.us-east-1.amazonaws.com/: dial tcp 54.239.28.176:443: i/o timeout"]}

On looking into the issue further, I found that it occurs due to an etcd key not found error when Vault looks for the path below in etcd following the aws ec2 login request.

/v2/keys/vault-data/auth/4a5990ce-ded9-96a3-4f7e-83f1a52ca705/config/certificate/?quorum=false&recursive=false&sorted=true

I find that in etcd the path until /v2/keys/vault-data/auth/4a5990ce-ded9-96a3-4f7e-83f1a52ca705/ exists and then the folders

role/

whitelist/

But the path config/ does not exist.   Also in this env, tls is disabled.. It is a single instance of Vault - so no HA.

What is Vault attempting to do here? Any ideas on what may have happened to cause this error?  How can Vault recover from this?

thanks,

Elizabeth

[Joel Thompson]

Hi Elizabeth,

Vault needs to reach out to the AWS APIs to validate the instance that is logging in. The error message you posted indicates that Vault is timing out when trying to connect to AWS. Did you change network connectivity for your Vault instance (e.g., change security group rules to not allow Vault to talk to the internet)?

The key not found error is unrelated; Vault is merely checking to see if a config value had been set.

--Joel

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/vault/issues
IRC: #vault-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Vault" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/a386eeca-0660-42f3-b969-ca5ad49a2a67%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Shanthi Koikkara]

Hi Joel, Let me check that...was focusing on etcd error...

thanks,

Shanthi

You received this message because you are subscribed to a topic in the Google Groups "Vault" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/vault-tool/CbBgI-IGpTk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to vault-tool+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/CAOXnK5RDaD5cyKKL8yHLpnxtGZqShDU_EASp1ohgB4UjLan6bQ%40mail.gmail.com.

[Elizabeth]

TO close this - we believe it is a CoreOS auto-update that caused this issue ....

thanks,

Elizabeth

--Joel

To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/a386eeca-0660-42f3-b969-ca5ad49a2a67%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/vault/issues
IRC: #vault-tool on Freenode
---
You received this message because you are subscribed to a topic in the Google Groups "Vault" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/vault-tool/CbBgI-IGpTk/unsubscribe.

To unsubscribe from this group and all its topics, send an email to vault-tool+unsubscribe@googlegroups.com.