I've been digging around in Net::SSH and I've found a trick to avoid
having the public key, assuming you're on a Linux box with ssh-agent
running.  Perhaps everyone already knows about this, but I'll post it
for posterity.

Also, in full disclosure, I'm no security expert, so this could be
absolutely terrible advice!  If so, someone please tell me!

I've only used it with Net:SSH directly, but since Capistrano relies
on Net::SSH, it should work as well.  I'd be curious if it works for
any of you...

Here's how it works: Net::SSH can make use of an ssh-agent to handle
the key management.  On my Linux box (Ubuntu, pretty standard) there
is already an ssh-agent running.  So, you simply need to hand off your
private key file to the agent.  This is done like so (from a command
line):

ssh-add /path/to/private/key/file

You can do it inside of Ruby by enclosing it in backticks:
`ssh-add #{my_key_path}`

If the key has a passphrase, you will be prompted.  This pretty much
kills it from an automation standpoint, so you'll want an unprotected
key.

At this point, the ssh-agent has the private key.  Net::SSH should
automatically detect the presence of the ssh-agent and allow it to
manage the keys.  You should now be able to connect and execute
commands on your remote server, all without ever getting a copy of the
public key!

Here is an example of how I do this in a script (using Net::SSH):

    # Assume t is our private key file

    # Pass the key file to ssh-agent
    `ssh-add #{t.path}`

    # Make the connection
    ssh = Net::SSH.start(ec2_instance_host, 'root', :auth_methods =>
["publickey"], :keys => [t.path])

    # Revoke the key from the ssh-agent
    `ssh-add -d #{t.path}`

    return ssh

Like I said, let me know if this works!