Spinning up an instance using a new keypair

[am4]

Im looking to do something like this -

1. create a new keypair (eg - new_keypair)
2. write the key to some place on disk
3. use this new keypair to spin up an instance
4. do some computations, save off results, etc.
5. Save off the instance as an AMI

I'm able to generate a new keypair, and use the bootstrap() method to start a new instance (giving it the name to the new keypair), but it relies on the FOG_RC env variable (which currently Im exporting manually, pointing to the place where I write out the key).

I've also tried to move away from the env variable (by passing the private_key_path to bootstrap), but that doesnt seem to work, and fails with a backtrace like this:

    from /var/lib/gems/1.9.1/gems/net-ssh-2.3.0/lib/net/ssh/ruby_compat.rb:22:in `io_select'
    from /var/lib/gems/1.9.1/gems/net-ssh-2.3.0/lib/net/ssh/transport/packet_stream.rb:94:in `block (2 levels) in next_packet'
    from /var/lib/gems/1.9.1/gems/net-ssh-2.3.0/lib/net/ssh/transport/packet_stream.rb:93:in `loop'
    from /var/lib/gems/1.9.1/gems/net-ssh-2.3.0/lib/net/ssh/transport/packet_stream.rb:93:in `block in next_packet'
    from /var/lib/gems/1.9.1/gems/net-ssh-2.3.0/lib/net/ssh/transport/packet_stream.rb:89:in `loop'
    from /var/lib/gems/1.9.1/gems/net-ssh-2.3.0/lib/net/ssh/transport/packet_stream.rb:89:in `next_packet'
    from /var/lib/gems/1.9.1/gems/net-ssh-2.3.0/lib/net/ssh/transport/session.rb:171:in `block in poll_message'
    from /var/lib/gems/1.9.1/gems/net-ssh-2.3.0/lib/net/ssh/transport/session.rb:166:in `loop'
    from /var/lib/gems/1.9.1/gems/net-ssh-2.3.0/lib/net/ssh/transport/session.rb:166:in `poll_message'
    from /var/lib/gems/1.9.1/gems/net-ssh-2.3.0/lib/net/ssh/transport/session.rb:151:in `next_message'
    from /var/lib/gems/1.9.1/gems/net-ssh-2.3.0/lib/net/ssh/transport/kex/diffie_hellman_group_exchange_sha1.rb:45:in `get_parameters'
    from /var/lib/gems/1.9.1/gems/net-ssh-2.3.0/lib/net/ssh/transport/kex/diffie_hellman_group1_sha1.rb:110:in `generate_key'
    from /var/lib/gems/1.9.1/gems/net-ssh-2.3.0/lib/net/ssh/transport/kex/diffie_hellman_group1_sha1.rb:51:in `initialize'
    from /var/lib/gems/1.9.1/gems/net-ssh-2.3.0/lib/net/ssh/transport/algorithms.rb:327:in `new'
    from /var/lib/gems/1.9.1/gems/net-ssh-2.3.0/lib/net/ssh/transport/algorithms.rb:327:in `exchange_keys'
    from /var/lib/gems/1.9.1/gems/net-ssh-2.3.0/lib/net/ssh/transport/algorithms.rb:175:in `proceed!'
    from /var/lib/gems/1.9.1/gems/net-ssh-2.3.0/lib/net/ssh/transport/algorithms.rb:166:in `send_kexinit'
    from /var/lib/gems/1.9.1/gems/net-ssh-2.3.0/lib/net/ssh/transport/algorithms.rb:121:in `accept_kexinit'

[...snip]

Am I missing something? Is there a better way to do this?

Any help would be greatly appreciated.

[geemus (Wesley Beary)]

That error looks a bit like it is pointing at a path that doesn't
exist. Not sure but you may want to double check the paths to make
sure the file is getting saved where you are expecting. If that isn't
it, could you include the error message from the top of the stack
trace as well?
Thanks!
wes

[am4]

Thx geemus.

Here is the backtrace from the run where I explicitly pass in the private_key_file to bootstrap (no public key file, since I dont have it)

    from /var/lib/gems/1.9.1/gems/net-ssh-2.3.0/lib/net/ssh/ruby_compat.rb:22:in `io_select'
    from /var/lib/gems/1.9.1/gems/net-ssh-2.3.0/lib/net/ssh/transport/packet_stream.rb:94:in `block (2 levels) in next_packet'
    from /var/lib/gems/1.9.1/gems/net-ssh-2.3.0/lib/net/ssh/transport/packet_stream.rb:93:in `loop'
    from /var/lib/gems/1.9.1/gems/net-ssh-2.3.0/lib/net/ssh/transport/packet_stream.rb:93:in `block in next_packet'
    from /var/lib/gems/1.9.1/gems/net-ssh-2.3.0/lib/net/ssh/transport/packet_stream.rb:89:in `loop'
    from /var/lib/gems/1.9.1/gems/net-ssh-2.3.0/lib/net/ssh/transport/packet_stream.rb:89:in `next_packet'
    from /var/lib/gems/1.9.1/gems/net-ssh-2.3.0/lib/net/ssh/transport/session.rb:171:in `block in poll_message'
    from /var/lib/gems/1.9.1/gems/net-ssh-2.3.0/lib/net/ssh/transport/session.rb:166:in `loop'
    from /var/lib/gems/1.9.1/gems/net-ssh-2.3.0/lib/net/ssh/transport/session.rb:166:in `poll_message'
    from /var/lib/gems/1.9.1/gems/net-ssh-2.3.0/lib/net/ssh/transport/session.rb:151:in `next_message'
    from /var/lib/gems/1.9.1/gems/net-ssh-2.3.0/lib/net/ssh/transport/kex/diffie_hellman_group_exchange_sha1.rb:45:in `get_parameters'
    from /var/lib/gems/1.9.1/gems/net-ssh-2.3.0/lib/net/ssh/transport/kex/diffie_hellman_group1_sha1.rb:110:in `generate_key'
    from /var/lib/gems/1.9.1/gems/net-ssh-2.3.0/lib/net/ssh/transport/kex/diffie_hellman_group1_sha1.rb:51:in `initialize'
    from /var/lib/gems/1.9.1/gems/net-ssh-2.3.0/lib/net/ssh/transport/algorithms.rb:327:in `new'
    from /var/lib/gems/1.9.1/gems/net-ssh-2.3.0/lib/net/ssh/transport/algorithms.rb:327:in `exchange_keys'
    from /var/lib/gems/1.9.1/gems/net-ssh-2.3.0/lib/net/ssh/transport/algorithms.rb:175:in `proceed!'
    from /var/lib/gems/1.9.1/gems/net-ssh-2.3.0/lib/net/ssh/transport/algorithms.rb:166:in `send_kexinit'
    from /var/lib/gems/1.9.1/gems/net-ssh-2.3.0/lib/net/ssh/transport/algorithms.rb:121:in `accept_kexinit'

    from /var/lib/gems/1.9.1/gems/net-ssh-2.3.0/lib/net/ssh/transport/session.rb:188:in `block in poll_message'

    from /var/lib/gems/1.9.1/gems/net-ssh-2.3.0/lib/net/ssh/transport/session.rb:166:in `loop'
    from /var/lib/gems/1.9.1/gems/net-ssh-2.3.0/lib/net/ssh/transport/session.rb:166:in `poll_message'

    from /var/lib/gems/1.9.1/gems/net-ssh-2.3.0/lib/net/ssh/transport/session.rb:203:in `block in wait'
    from /var/lib/gems/1.9.1/gems/net-ssh-2.3.0/lib/net/ssh/transport/session.rb:201:in `loop'
    from /var/lib/gems/1.9.1/gems/net-ssh-2.3.0/lib/net/ssh/transport/session.rb:201:in `wait'
    from /var/lib/gems/1.9.1/gems/net-ssh-2.3.0/lib/net/ssh/transport/session.rb:80:in `initialize'
    from /var/lib/gems/1.9.1/gems/net-ssh-2.3.0/lib/net/ssh.rb:186:in `new'
    from /var/lib/gems/1.9.1/gems/net-ssh-2.3.0/lib/net/ssh.rb:186:in `start'
    from /var/lib/gems/1.9.1/gems/fog-1.3.0/lib/fog/core/ssh.rb:52:in `run'
    from /var/lib/gems/1.9.1/gems/fog-1.3.0/lib/fog/aws/models/compute/server.rb:209:in `block (2 levels) in setup'
    from /usr/lib/ruby/1.9.1/timeout.rb:58:in `timeout'
    from /var/lib/gems/1.9.1/gems/fog-1.3.0/lib/fog/aws/models/compute/server.rb:208:in `block in setup'
    from /var/lib/gems/1.9.1/gems/fog-1.3.0/lib/fog/aws/models/compute/server.rb:206:in `setup'
    from /var/lib/gems/1.9.1/gems/fog-1.3.0/lib/fog/aws/models/compute/servers.rb:102:in `bootstrap'

At this point, would you say this is the correct way of doing it, or would you recommend some other way of spinning up an instance using a new key?

On Monday, April 9, 2012 5:26:09 PM UTC-7, geemus wrote:

That error looks a bit like it is pointing at a path that doesn't
exist.  Not sure but you may want to double check the paths to make
sure the file is getting saved where you are expecting.  If that isn't
it, could you include the error message from the top of the stack
trace as well?
Thanks!
wes

[geemus (Wesley Beary)]

I would try lengthening the timeout and see if you have any better
luck as a starting point. What instance sizes are you using? (I
mostly ask because micros can be particularly finicky).
wes

[am4]

The same code would run to completion when provided with a path to the .fog file (via FOG_RC), so that makes me think it wasnt a timeout issue.

I finally resorted to something like this:

1. pkey is written out to a known location on disk (it corresponds to an existing aws keypair)
2. bootstrap a server, telling Fog.credentials about the name of the keypair (which has to be prefixed with fog_..)
3. bootstrap method also takes a param 'private_key_path' where i point it to the key (on disk).

An earlier experiment I tried (I guess I completely misunderstood what the lib was doing) was:
1. create a new keypair in aws
2. grab the key from the response, write it to disk
3. bootstrap a server, pointing it to the key i just wrote
4. Also provide the ;key_name to the bootstrap

IIRC, This would complain that the .fog file didnt contain a key with this name.

So Im still a little confused on how these entities interact (or precedence for each).
1. .fog file (pointed to by FOG_RC, maybe)
2. Fog.credentials
3. key_name (provided to bootstrap)
4. private_key_path, public_key_path (provided to the bootstrap method)

I tried looking at the source, but since Im still learning ruby, I didnt make much progress.
Any clarification would be appreciated.


Thanks for all the help so far.

On Tuesday, April 10, 2012 1:03:17 PM UTC-7, geemus wrote:

I would try lengthening the timeout and see if you have any better
luck as a starting point.  What instance sizes are you using?  (I
mostly ask because micros can be particularly finicky).
wes

[geemus (Wesley Beary)]

Got it, I'll try to clarify.

.fog is the fall back/defaults. If there is some value here it will
be read only if you don't specify something in particular later.

Fog.credentials just contains the parsed/in memory representation of .fog

key_name is needed to attach a public_key to a server at boot time on aws

private key is needed at ssh time for you to connect to the aws server
with the matching public key

Hope that helps clear it up. Sorry that it wasn't clear to begin
with, any suggestions (or especially patches) to documentation to make
it clearer for future users would be amazing.

Thanks!
wes

[am4]

Thanks Wes,

Just to clarify (so that I have the right stuff to put in my patch):

If you specify a key_name, and the path to the pvt key, then you dont need to deal with the .fog file or Fog.credentials in order to spin up a new instance and SSH into it. Correct?

Basically, the key_name is passed back transparently to AWS, so that it can spin up an instance with that key, and Fog will assume that the pvt key file you are specifying matches the public key that is associated with the keyname specified earlier.

LMK if Im on the wrong track.
Thx

On Sunday, April 15, 2012 12:46:09 PM UTC-7, geemus wrote:

Got it, I'll try to clarify.

.fog is the fall back/defaults.  If there is some value here it will
be read only if you don't specify something in particular later.

Fog.credentials just contains the parsed/in memory representation of .fog

key_name is needed to attach a public_key to a server at boot time on aws

private key is needed at ssh time for you to connect to the aws server
with the matching public key

Hope that helps clear it up.  Sorry that it wasn't clear to begin
with, any suggestions (or especially patches) to documentation to make
it clearer for future users would be amazing.

Thanks!
wes

[geemus (Wesley Beary)]

I think that sounds correct, yes. Any time you explicitly pass
something it should override whichever values might or might not exist
in .fog (and Fog.credentials is basically just .fog converted into
Ruby data structures). So yes, I think we are on the same page. Let
me know if you have other questions or concerns though.
Thanks!
wes

[geemus (Wesley Beary)]

I believe you are correct that public key shouldn't be needed for an existing key.

I do not believe there is a way to have a particular key name and key path associate with one another in ~/.fog at present.

Not sure about the timeouts, but we can continue that discussion in the other thread.

wes

On Thu, Mar 28, 2013 at 6:21 PM, Paul <catc...@paulwalker.tv> wrote:

So, in the common situation in which one already has a key_pair defined in AWS with a specific name, is the right thing to do to provide that name in the key_name parameter as well as the private_key_path?  Can you confirm that public_key_path is not needed.  Can one specify the private_key_path in the ~/.fog file under a section with the key_name?  e.g.

  :mykey:

     private_key_path: '~/.ssh/mykey'

Fog.credential = :mykey

connection.servers.bootstrap(key_name: 'mykey', ...)  # private_key_path nor public_key_path are arguments

Does it matter that they key_name value is a string in the bootstrap method while the key_name in the .fog file is a symbol?  Will Fog correctly associate the two?

I have yet to have the bootstrap method not timeout when it successfully creates an instance.

--
You received this message because you are subscribed to the Google Groups "ruby-fog" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ruby-fog+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.