TLS Handshake Timeouts on S3 backend

[Nick Tkach]

Suddenly (honest, I'm not aware of anything changed my end), every time I try to

terraform init

anywhere that's set for our s3 remote state backend I'm getting TLS handshake timeouts. 

This is on a Macbook Pro OS X 10.12.6 if that makes any difference.

C02PL3M7FVH6:network ntkach$ TF_LOG=DEBUG terraform init
2018/09/12 15:12:08 [INFO] Terraform version: 0.11.8
2018/09/12 15:12:08 [INFO] Go runtime version: go1.10.3





Initializing the backend...

2018/09/12 15:12:08 [INFO] Building AWS region structure
2018/09/12 15:12:08 [INFO] Building AWS auth structure
2018/09/12 15:12:08 [INFO] Setting AWS metadata API timeout to 100ms
2018/09/12 15:12:08 [INFO] Ignoring AWS metadata API endpoint at default location as it doesn't return any instance-id
2018/09/12 15:12:08 [INFO] AWS Auth provider used: "SharedCredentialsProvider"
2018/09/12 15:12:08 [INFO] Initializing DeviceFarm SDK connection
2018/09/12 15:12:08 [DEBUG] [aws-sdk-go] DEBUG: Request sts/GetCallerIdentity Details:
---[ REQUEST POST-SIGN ]-----------------------------


<SNIP POST FOR NOW>

-----------------------------------------------------

2018/09/12 15:12:11 [ERR] Checkpoint error: Get https://checkpoint-api.hashicorp.com/v1/check/terraform?arch=amd64&os=darwin&signature=b7b3f5ff-1c85-f0ac-0039-db25c74e4e49&version=0.11.8: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
2018/09/12 15:12:19 [DEBUG] [aws-sdk-go] DEBUG: Send Request sts/GetCallerIdentity failed, not retrying, error RequestError: send request failed
caused by: Post https://sts.amazonaws.com/: net/http: TLS handshake timeout

2018/09/12 15:12:19 [DEBUG] plugin: waiting for all plugin processes to complete...
Error configuring the backend "s3": RequestError: send request failed
caused by: Post https://sts.amazonaws.com/: net/http: TLS handshake timeout

Please update the configuration in your Terraform files to fix this error.
If you'd like to update the configuration interactively without storing
the values in your configuration, run "terraform init".

I'm confident I've remove the .aws credentials as an issue (tried two different sets of keys with the same permissions) and that it's not something about the files in that specific directory (I can have just the provider and the backend config and it still happens).  Anyone getting this recently? (last 3 or 4 days)

[Nick Tkach]

Seems like it's broader than just this.  I get the same result even just trying to get a different version of the aws provider

local$ terraform init

Initializing provider plugins...
- Checking for available provider plugins on https://releases.hashicorp.com...

Error installing provider "aws": Get https://releases.hashicorp.com/terraform-provider-aws/: net/http: TLS handshake timeout.

Terraform analyses the configuration and state and automatically downloads
plugins for the providers used. However, when attempting to download this
plugin an unexpected error occured.

This may be caused if for some reason Terraform is unable to reach the
plugin repository. The repository may be unreachable if access is blocked
by a firewall.

If automatic installation is not possible or desirable in your environment,
you may alternatively manually install plugins by downloading a suitable
distribution package and placing the plugin's executable file in the
following directory:
    terraform.d/plugins/darwin_amd64

[t.pe...@treeptik.fr]

I still have the same issue and I don't know where to search :

% terraform init

2019/02/02 20:18:55 [INFO] Terraform version: 0.11.11  ac4fff416318bf0915a0ab80e062a99ef3724334

2019/02/02 20:18:55 [INFO] Go runtime version: go1.11.1

2019/02/02 20:18:55 [INFO] CLI args: []string{"/usr/local/bin/terraform", "init"}

2019/02/02 20:18:55 [DEBUG] Attempting to open CLI config file: /Users/thomas/.terraformrc

2019/02/02 20:18:55 [DEBUG] File doesn't exist, but doesn't need to. Ignoring.

2019/02/02 20:18:55 [INFO] CLI command args: []string{"init"}

2019/02/02 20:18:55 [DEBUG] command: loading backend config file: /Users/thomas/Dev/Tests/test-cci

Initializing the backend...

2019/02/02 20:18:55 [DEBUG] command: no data state file found for backend config

2019/02/02 20:18:55 [DEBUG] New state was assigned lineage "19aedc0a-6cf4-c2c9-a907-f711a143c6cb"

2019/02/02 20:18:55 [INFO] Building AWS region structure

2019/02/02 20:18:55 [INFO] Building AWS auth structure

2019/02/02 20:18:55 [INFO] Setting AWS metadata API timeout to 100ms

2019/02/02 20:18:56 [INFO] Ignoring AWS metadata API endpoint at default location as it doesn't return any instance-id

2019/02/02 20:18:56 [INFO] AWS Auth provider used: "EnvProvider"

2019/02/02 20:18:56 [INFO] Initializing DeviceFarm SDK connection

2019/02/02 20:18:56 [DEBUG] [aws-sdk-go] DEBUG: Request sts/GetCallerIdentity Details:

---[ REQUEST POST-SIGN ]-----------------------------

POST / HTTP/1.1

Host: sts.amazonaws.com

User-Agent: aws-sdk-go/1.14.31 (go1.11.1; darwin; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.11.11

Content-Length: 43

Authorization: AWS4-HMAC-SHA256 Credential=AKIAJNJRAUHE7INM3QEA/20190202/us-east-1/sts/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-date, Signature=1bbd05b3e6b0cb749b8531759a76ec278d3c42118fde5b8db6a9343737c50214

Content-Type: application/x-www-form-urlencoded; charset=utf-8

X-Amz-Date: 20190202T191856Z

Accept-Encoding: gzip

Action=GetCallerIdentity&Version=2011-06-15

-----------------------------------------------------

2019/02/02 20:19:06 [DEBUG] [aws-sdk-go] DEBUG: Send Request sts/GetCallerIdentity failed, not retrying, error RequestError: send request failed

caused by: Post https://sts.amazonaws.com/: net/http: TLS handshake timeout

2019/02/02 20:19:06 [DEBUG] plugin: waiting for all plugin processes to complete...

Error configuring the backend "s3": RequestError: send request failed

caused by: Post https://sts.amazonaws.com/: net/http: TLS handshake timeout

Please update the configuration in your Terraform files to fix this error

then run this command again.

Here is my versions :

% terraform --version

2019/02/02 20:18:28 [INFO] Terraform version: 0.11.11  ac4fff416318bf0915a0ab80e062a99ef3724334

2019/02/02 20:18:28 [INFO] Go runtime version: go1.11.1

2019/02/02 20:18:28 [INFO] CLI args: []string{"/usr/local/bin/terraform", "--version"}

2019/02/02 20:18:28 [DEBUG] Attempting to open CLI config file: /Users/thomas/.terraformrc

2019/02/02 20:18:28 [DEBUG] File doesn't exist, but doesn't need to. Ignoring.

2019/02/02 20:18:28 [INFO] CLI command args: []string{"version", "--version"}

Terraform v0.11.11

2019/02/02 20:18:28 [DEBUG] checking for provider in "."

2019/02/02 20:18:28 [DEBUG] checking for provider in "/usr/local/bin"

2019/02/02 20:18:28 [DEBUG] checking for provider in ".terraform/plugins/darwin_amd64"

2019/02/02 20:18:28 [DEBUG] checking for provider in "/Users/thomas/.terraform.d/plugins"

2019/02/02 20:18:28 [DEBUG] found provider "terraform-provider-aws_v1.56.0_x4"

2019/02/02 20:18:28 [DEBUG] found provider "terraform-provider-aws_v1.57.0_x4"

2019/02/02 20:18:28 [DEBUG] found provider "terraform-provider-azurerm_v1.18.0_x4"

2019/02/02 20:18:28 [DEBUG] found provider "terraform-provider-local_v1.1.0_x4"

2019/02/02 20:18:28 [DEBUG] found provider "terraform-provider-random_v2.0.0_x4"

2019/02/02 20:18:28 [DEBUG] found provider "terraform-provider-template_v1.0.0_x4"

2019/02/02 20:18:28 [DEBUG] found valid plugin: "local", "1.1.0", "/Users/thomas/.terraform.d/plugins/terraform-provider-local_v1.1.0_x4"

2019/02/02 20:18:28 [DEBUG] found valid plugin: "random", "2.0.0", "/Users/thomas/.terraform.d/plugins/terraform-provider-random_v2.0.0_x4"

2019/02/02 20:18:28 [DEBUG] found valid plugin: "template", "1.0.0", "/Users/thomas/.terraform.d/plugins/terraform-provider-template_v1.0.0_x4"

2019/02/02 20:18:28 [DEBUG] found valid plugin: "aws", "1.56.0", "/Users/thomas/.terraform.d/plugins/terraform-provider-aws_v1.56.0_x4"

2019/02/02 20:18:28 [DEBUG] found valid plugin: "aws", "1.57.0", "/Users/thomas/.terraform.d/plugins/terraform-provider-aws_v1.57.0_x4"

2019/02/02 20:18:28 [DEBUG] found valid plugin: "azurerm", "1.18.0", "/Users/thomas/.terraform.d/plugins/terraform-provider-azurerm_v1.18.0_x4"

2019/02/02 20:18:29 [DEBUG] plugin: waiting for all plugin processes to complete...

+ provider.aws v1.57.0

[Nick Tkach]

I have yet to find any particular answer.  I can only vaguely guess that it's somehow something about an sdk or go version and the OS x version.  I saw mention somewhere of someone getting a result like that and upgrading to the next version of OS X fixed it.  Not really an option for those of us on a company machine.

Cet e-mail (y compris tout fichier joint) peut contenir des informations confidentielles et/ou protégées. Toute utilisation ou diffusion non autorisée de ce message, entièrement ou partiellement, est strictement interdit. Si vous n’êtes pas le destinataire (ou si vous avez reçu ce message par erreur), merci de le notifier immédiatement à l’expéditeur et de le supprimer.

This e-mail (including any attachments) may contain confidential and/or privileged information. Any unauthorized use or dissemination of this message in whole or in part is strictly prohibited. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail.

N'imprimez que si nécessaire ! Print out this e-mail only if you need to !

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/terraform/issues
IRC: #terraform-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Terraform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to terraform-too...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/terraform-tool/0c7c473a-25f8-458f-bb30-a15fe885db99%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[t.pe...@treeptik.fr]

Tanks for your reply.
I am already on the latest versions of macos, terraform, aws-provider.
The go runtime used by terraform seems to be good (1.11.1).

I suspect a problem with TLS or certificates but only in terminal because i Can access URLs that appear in logs (Check point, sts) with safari.

I tried multiple versions of Terraform from 0.10.8 to 0.12.0-alpha, installed manually of by brew but it’s always thé same

[t.pe...@treeptik.fr]

It works with using sudo !!!

% sudo -E terraform init

Password:

Initializing the backend...

Do you want to copy existing state to the new backend?

  Pre-existing state was found while migrating the previous "local" backend to the

  newly configured "s3" backend. No existing state was found in the newly

  configured "s3" backend. Do you want to copy this state to the new "s3"

  backend? Enter "yes" to copy and "no" to start with an empty state.

  Enter a value: yes

Successfully configured the backend "s3"! Terraform will automatically

use this backend unless the backend configuration changes.

Initializing provider plugins...

The following providers do not have any version constraints in configuration,

so the latest version was installed.

To prevent automatic upgrades to new major versions that may contain breaking

changes, it is recommended to add version = "..." constraints to the

corresponding provider blocks in configuration, with the constraint strings

suggested below.

* provider.aws: version = "~> 1.57"

* provider.local: version = "~> 1.1"

* provider.template: version = "~> 1.0"

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see

any changes that are required for your infrastructure. All Terraform commands

should now work.

If you ever set or change modules or backend configuration for Terraform,

rerun this command to reinitialize your working directory. If you forget, other

commands will detect it and remind you to do so if necessary.

[James Osbourn]

I recall having some similar errors in the past and I think each time it was related to a misconfiguration. I had the region set incorrectly.mor I had failed o use -reconfigure and/or -upgrade on my unit.  If you could share your backend config we could check the code.

Thanks

To view this discussion on the web visit https://groups.google.com/d/msgid/terraform-tool/864425e5-5fb7-4e88-b5ec-fc9ea7e9f894%40googlegroups.com.

[t.pe...@treeptik.fr]

The same code works fine on Ubuntu, and now works fine with "sudo -E" on my mac.

Here is the code 

terraform {

backend "s3" {

encrypt = true

bucket = "bucket-test"

region = "eu-west-1"

key = "terraform/terraform.tfstate"

dynamodb_table = "terraform-state-lock-dynamo"

}

}

# terraform state file setup

# create an S3 bucket to store the state file in

resource "aws_s3_bucket" "terraform-state-storage-s3" {

bucket = "bucket-test"

versioning {

enabled = true

}

# lifecycle {

# prevent_destroy = true

# }

tags {

Name = "S3 Remote Terraform State Store"

Author = "Thomas Perelle"

Environment = "test"

}

}

# create a dynamodb table for locking the state file

resource "aws_dynamodb_table" "dynamodb-terraform-state-lock" {

name = "terraform-state-lock-dynamo"

hash_key = "LockID"

read_capacity = 20

write_capacity = 20

attribute {

name = "LockID"

type = "S"

}

tags {

Name = "DynamoDB Terraform State Lock Table"

Author = "Thomas Perelle"

Environment = "test"

}

}

[James Osbourn]

If this works with auto this would imply to me that permissions may be incorrect on the .terraform directory where you are running in it. Audi can read/write. But you can not. Have you tried removing and running in it again. Or running from a different location?

To view this discussion on the web visit https://groups.google.com/d/msgid/terraform-tool/86b4991e-c9ca-4fe3-81a3-722c01a4680e%40googlegroups.com.

[t.pe...@treeptik.fr]

I have full access on the .terraform and its content. 

I think the problem is not on file access but on SSL/TLS exchange with remote endpoint : AWS STS, Terraform checkpoint, etc. 

But I actually don't know why.