instance fail to mount EBS volume

[Jingqi]

I suddenly failed to have my data volume mounted to instance (p2.xlarge or p2.16xlarge) with following error:

Waiting for instance  i-03fd833c03d2999a6 to initialize...
[ubu...@54.211.84.206] run: ls /data
Traceback (most recent call last):
  File "/Users/dreyfusslab/CloudSoftwareTools/cryoem-cloud-tools/aws//launch_AWS_instance.py", line 594, in <module>
    AttachMountEBSVol(instanceID,params['volume'],PublicIP,keyPath,params)
  File "/Users/dreyfusslab/CloudSoftwareTools/cryoem-cloud-tools/aws//launch_AWS_instance.py", line 522, in AttachMountEBSVol
    dir_exists=exec_remote_cmd('ls /%s' %(params['dirname']))
  File "/Users/dreyfusslab/CloudSoftwareTools/cryoem-cloud-tools/aws//launch_AWS_instance.py", line 549, in exec_remote_cmd
    return run(cmd)
  File "/Users/dreyfusslab/Library/Python/2.7/lib/python/site-packages/fabric/network.py", line 687, in host_prompting_wrapper
    return func(*args, **kwargs)
  File "/Users/dreyfusslab/Library/Python/2.7/lib/python/site-packages/fabric/operations.py", line 1090, in run
    shell_escape=shell_escape, capture_buffer_size=capture_buffer_size,
  File "/Users/dreyfusslab/Library/Python/2.7/lib/python/site-packages/fabric/operations.py", line 930, in _run_command
    channel=default_channel(), command=wrapped_command, pty=pty,
  File "/Users/dreyfusslab/Library/Python/2.7/lib/python/site-packages/fabric/state.py", line 435, in default_channel
    chan = _open_session()
  File "/Users/dreyfusslab/Library/Python/2.7/lib/python/site-packages/fabric/state.py", line 420, in _open_session
    session = transport.open_session(timeout=env.timeout)
  File "build/bdist.macosx-10.10-intel/egg/paramiko/transport.py", line 732, in open_session
  File "build/bdist.macosx-10.10-intel/egg/paramiko/transport.py", line 860, in open_channel
paramiko.ssh_exception.SSHException: Timeout openning channel.


Any suggestion?

Jingqi

[Ali Khan]

I have suddenly started having the same error myself when I run the command:

$awslaunch --instance=t2.micro --spot 0.1 --availZone=us-east-1b --volume=vol-xxxxxxxxxxx

Michael, do you have any suggestions?

[Michael Cianfrocco]

Hi Ali & Jingqi, 

I am not having this issue - do you think something changed on your computer? The errors you are getting are related to the software program 'fabric' - perhaps try uninstalled and reinstalling? 

$ pip uninstall fabric
$ pip install fabric

Mike

[Ali Khan]

Hey Mike,

Unfortunately, that did not fix it. I will keep you informed if I come up with anything.

Ali

[Jingqi]

 It happened sporadically. For me, I just cycle awskill i-******************** and awslaunch *** until it had 100% success.

[Ali Khan]

Hey Jingqi and Mike,

I was curious if any solutions were found for this problem. I unfortunately haven't figured out a solution. I tried to use a side route to attach the volume using aws_ebs_attach, but even in this scenario, the attached EBS drive in not in the folder /data, though aws says it is successfully attached.

Ali

[mcia...@umich.edu]

Hi Ali,

This is very weird - I still suspect this is a python / software version issue. I have yet to reproduce this on my machines at Michigan so it's hard for me to troubleshoot. Do you want to summarize any new attempts you made and what you saw? Maybe a fresh look could fix this.

Mike