TCP connection to SSH ip/port failed - running packer behind a corporate proxy

[Claudiu Sirbu]

Hi,

I'm trying to run packer on a debian9 system from behind a proxy.
I'm able to ssh into ec2 instances using the following setting in the .ssh/config:

Host *
    port 22
    proxycommand socat - PROXY:<PROXY-IP>:%h:%p,proxyport=<PROXY-PORT>

My system has http_proxy already configured, but packer is not using it: (replaced actual values)

HTTP_PROXY=http://PROXY-IP:PROXY-PORT
FTP_PROXY=http://PROXY-IP:PROXY-PORT
https_proxy=http://PROXY-IP:PROXY-PORT
http_proxy=http://PROXY-IP:PROXY-PORT
ALL_PROXY=http://PROXY-IP:PROXY-PORT
HTTPS_PROXY=http://PROXY-IP:PROXY-PORT
all_proxy=http://PROXY-IP:PROXY-PORT
PROXY_HOST=http://PROXY-IP:PROXY-PORT
ftp_proxy=http://PROXY-IP:PROXY-PORT

Here is the log(relevant part of it) i got from packer:

2017/10/25 12:58:15 [INFO] Packer version: 1.1.1
2017/10/25 12:58:15 Packer Target OS/Arch: linux amd64
2017/10/25 12:58:15 Built with Go Version: go1.9
....
2017/10/25 13:18:10 ui: [1;32m==> amazon-ebs: Adding tags to source instance [0m
2017/10/25 13:18:10 ui: [0;32m    amazon-ebs: Adding tag: "Name": "Packer Builder" [0m
2017/10/25 13:18:11 ui: [0;32m    amazon-ebs: Instance ID: i-0fcec4993bb5bdbea [0m
2017/10/25 13:18:11 ui: [1;32m==> amazon-ebs: Waiting for instance (i-0fcec4993bb5bdbea) to become ready... [0m
2017/10/25 13:18:11 packer: 2017/10/25 13:18:11 Waiting for state to become: running
2017/10/25 13:18:11 packer: 2017/10/25 13:18:11 Using 2s as polling delay (change with AWS_POLL_DELAY_SECONDS)
2017/10/25 13:18:11 packer: 2017/10/25 13:18:11 Allowing 300s to complete (change with AWS_TIMEOUT_SECONDS)
2017/10/25 13:18:39 ui: [0;32m    amazon-ebs: Public DNS: ec2-18-194-126-228.eu-central-1.compute.amazonaws.com [0m
2017/10/25 13:18:39 ui: [0;32m    amazon-ebs: Public IP: 18.194.126.228 [0m
2017/10/25 13:18:39 ui: [0;32m    amazon-ebs: Private IP: 172.31.3.15 [0m
2017/10/25 13:18:39 ui: ask: [1;32m==> amazon-ebs: Pausing after run of step 'StepRunSourceInstance'. Press enter to continue. [0m
2017/10/25 13:18:42 packer: 2017/10/25 13:18:42 [INFO] Not using winrm communicator, skipping get password...
2017/10/25 13:18:42 ui: ask: [1;32m==> amazon-ebs: Pausing after run of step 'StepGetPassword'. Press enter to continue. [0m
2017/10/25 13:18:43 packer: 2017/10/25 13:18:43 [INFO] Waiting for SSH, up to timeout: 5m0s
2017/10/25 13:18:43 ui: [1;32m==> amazon-ebs: Waiting for SSH to become available... [0m
2017/10/25 13:18:58 packer: 2017/10/25 13:18:58 [DEBUG] TCP connection to SSH ip/port failed: dial tcp 18.194.126.228:22: i/o timeout
2017/10/25 13:19:18 packer: 2017/10/25 13:19:18 [DEBUG] TCP connection to SSH ip/port failed: dial tcp 18.194.126.228:22: i/o timeout
2017/10/25 13:19:38 packer: 2017/10/25 13:19:38 [DEBUG] TCP connection to SSH ip/port failed: dial tcp 18.194.126.228:22: i/o timeout
2017/10/25 13:19:58 packer: 2017/10/25 13:19:58 [DEBUG] TCP connection to SSH ip/port failed: dial tcp 18.194.126.228:22: i/o timeout
2017/10/25 13:20:18 packer: 2017/10/25 13:20:18 [DEBUG] TCP connection to SSH ip/port failed: dial tcp 18.194.126.228:22: i/o timeout
2017/10/25 13:20:38 packer: 2017/10/25 13:20:38 [DEBUG] TCP connection to SSH ip/port failed: dial tcp 18.194.126.228:22: i/o timeout
2017/10/25 13:20:58 packer: 2017/10/25 13:20:58 [DEBUG] TCP connection to SSH ip/port failed: dial tcp 18.194.126.228:22: i/o timeout
2017/10/25 13:21:18 packer: 2017/10/25 13:21:18 [DEBUG] TCP connection to SSH ip/port failed: dial tcp 18.194.126.228:22: i/o timeout
2017/10/25 13:21:38 packer: 2017/10/25 13:21:38 [DEBUG] TCP connection to SSH ip/port failed: dial tcp 18.194.126.228:22: i/o timeout
2017/10/25 13:21:58 packer: 2017/10/25 13:21:58 [DEBUG] TCP connection to SSH ip/port failed: dial tcp 18.194.126.228:22: i/o timeout
2017/10/25 13:22:18 packer: 2017/10/25 13:22:18 [DEBUG] TCP connection to SSH ip/port failed: dial tcp 18.194.126.228:22: i/o timeout
2017/10/25 13:22:38 packer: 2017/10/25 13:22:38 [DEBUG] TCP connection to SSH ip/port failed: dial tcp 18.194.126.228:22: i/o timeout
2017/10/25 13:22:58 packer: 2017/10/25 13:22:58 [DEBUG] TCP connection to SSH ip/port failed: dial tcp 18.194.126.228:22: i/o timeout
2017/10/25 13:23:06 packer: 2017/10/25 13:23:06 [DEBUG] TCP connection to SSH ip/port failed: dial tcp 18.194.126.228:22: getsockopt: no route to host
2017/10/25 13:23:26 packer: 2017/10/25 13:23:26 [DEBUG] TCP connection to SSH ip/port failed: dial tcp 18.194.126.228:22: i/o timeout
2017/10/25 13:23:43 ui error: [1;31m==> amazon-ebs: Timeout waiting for SSH. [0m
2017/10/25 13:23:43 ui: ask: [1;32m==> amazon-ebs: Pausing before cleanup of step 'StepGetPassword'. Press enter to continue. [0m
2017/10/25 13:23:46 packer: 2017/10/25 13:23:46 [DEBUG] TCP connection to SSH ip/port failed: dial tcp 18.194.126.228:22: i/o timeout
2017/10/25 13:23:46 packer: 2017/10/25 13:23:46 [DEBUG] SSH wait cancelled. Exiting loop.

Proof that ssh works:

$ ssh-add ec2_amazon-ebs.pem
Identity added: ec2_amazon-ebs.pem (ec2_amazon-ebs.pem)
$ ssh -l ubuntu 18.194.126.228
Warning: Permanently added '18.194.126.228' (ECDSA) to the list of known hosts.
Welcome to Ubuntu 16.04.3 LTS (GNU/Linux 4.4.0-1038-aws x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

  Get cloud support with Ubuntu Advantage Cloud Guest:
    http://www.ubuntu.com/business/services/cloud

0 packages can be updated.
0 updates are security updates.



The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.

Any idea on how to tell packer that it should ssh using the proxy?


Thanks in advance,
Claudiu

[Rickard von Essen]

Packer doesn't run openssh, it uses Go native ssh library which doesn't read that config file.

There is a PR for adding Socks5 proxy support here https://github.com/hashicorp/packer/pull/5439

Would that help? 

...

[Claudiu Sirbu]

Thanks for the quick answer. We are not having a Socks5 Proxy.
I guess I will have to run packer on an EC2 instance in the future. :-)

[Alexander Lehmann]

It should be possible to run a socks5 listener locally and make it connect to the CONNECT proxy like socat does from stdin, though you may have to write the program yourself (should be easy with e.g. netty)

[Sumit Mathur]

We are running Packer on RHEL and have same issue where we are able to connect to EC2 instances through proxy but packer is failing with error "Build 'amazon-ebs' errored: Timeout waiting for SSH."

How we can use packer to ssh to EC2 instance from datacenter server to AWS EC2 instance for backing an AMI.

Flow is like this 

Server --> Proxy --> Bastion --> EC2 instance 

-Sumit

[Sumit Mathur]

Does anyone got workaround on this?

On Wednesday, October 25, 2017 at 5:54:12 PM UTC+5:30, Claudiu Sirbu wrote:

[Claudiu Sirbu]

Hi,

As stated above, packer doesn't work with proxy. A Socks5 implementation will come, but for me is not that useful.

The workaround would be to install packer on an ec2 machine and create the ami's from there.

regards,

claudiu