Strange error launching custom AMI

304 views
Skip to first unread message

Paolo Di Tommaso

unread,
May 19, 2012, 6:16:03 AM5/19/12
to jcl...@googlegroups.com
Hi guys, 

I'm getting a strange error launching an EC2 instance using a custom image that I'm able to start and access without any problem from the AWS  dashboard. 

2012-05-19 11:24:35,245 [user thread 2] ERROR jclouds.ssh - << (root:rsa[fingerprint(d4:2b:a2:52:39:37:14:f2:a7:37:26:80:2c:80:1e:57),sha1(6e:02:e7:ce:cd:36:86:38:45:4f:ef:f6:02:42:52:a5:94:2c:c4:76)]@23.20.71.204:22) error acquiring SFTPClient() (out of retries - max 7): Invalid packet: indicated length 1349281121 too large
java.lang.IllegalStateException: Invalid packet: indicated length 1349281121 too large
    at net.schmizz.sshj.sftp.PacketReader.readPacket(PacketReader.java:70) ~[sshj-0.8.0.jar:na]
    at net.schmizz.sshj.sftp.SFTPEngine.init(SFTPEngine.java:77) ~[sshj-0.8.0.jar:na]
    at net.schmizz.sshj.SSHClient.newSFTPClient(SSHClient.java:623) ~[sshj-0.8.0.jar:na]
    at org.jclouds.sshj.SshjSshClient$2.create(SshjSshClient.java:278) ~[jclouds-sshj-1.3.2.jar:1.3.2]
    at org.jclouds.sshj.SshjSshClient$2.create(SshjSshClient.java:261) ~[jclouds-sshj-1.3.2.jar:1.3.2]
    at org.jclouds.sshj.SshjSshClient.acquire(SshjSshClient.java:226) [jclouds-sshj-1.3.2.jar:1.3.2]
    at org.jclouds.sshj.SshjSshClient$PutConnection.create(SshjSshClient.java:341) [jclouds-sshj-1.3.2.jar:1.3.2]
    at org.jclouds.sshj.SshjSshClient$PutConnection.create(SshjSshClient.java:319) [jclouds-sshj-1.3.2.jar:1.3.2]
    at org.jclouds.sshj.SshjSshClient.acquire(SshjSshClient.java:226) [jclouds-sshj-1.3.2.jar:1.3.2]
    at org.jclouds.sshj.SshjSshClient.put(SshjSshClient.java:375) [jclouds-sshj-1.3.2.jar:1.3.2]
    at org.jclouds.sshj.SshjSshClient.put(SshjSshClient.java:164) [jclouds-sshj-1.3.2.jar:1.3.2]
    at org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSsh.doCall(RunScriptOnNodeAsInitScriptUsingSsh.java:119) [jclouds-compute-1.3.2.jar:1.3.2]
    at org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete.future(RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete.java:68) [jclouds-compute-1.3.2.jar:1.3.2]
    at org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete.doCall(RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete.java:60) [jclouds-compute-1.3.2.jar:1.3.2]
    at org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSsh.call(RunScriptOnNodeAsInitScriptUsingSsh.java:91) [jclouds-compute-1.3.2.jar:1.3.2]
    at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:150) [jclouds-compute-1.3.2.jar:1.3.2]
    at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:57) [jclouds-compute-1.3.2.jar:1.3.2]
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) [na:1.6.0_29]
    at java.util.concurrent.FutureTask.run(FutureTask.java:138) [na:1.6.0_29]
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [na:1.6.0_29]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [na:1.6.0_29]
    at java.lang.Thread.run(Thread.java:680) [na:1.6.0_29]


I was thinking it was related to this bug https://github.com/shikhar/sshj/pull/73

But the SSHJ guys says: "The packet length is garbage, it would imply a 1287 MB SFTP packet, so the exception is being correctly raised" 


Any idea ? 


Thanks, 
Paolo



Adrian Cole

unread,
May 19, 2012, 11:53:47 AM5/19/12
to jcl...@googlegroups.com

This isn't an issue launching, rather one sshing in.  When you launch manually, what user do you connect with?  What ami or base ami is this made from.

-A

--
You received this message because you are subscribed to the Google Groups "jclouds" group.
To view this discussion on the web visit https://groups.google.com/d/msg/jclouds/-/PZYhAqlXdGcJ.
To post to this group, send email to jcl...@googlegroups.com.
To unsubscribe from this group, send email to jclouds+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/jclouds?hl=en.

Aled Sage

unread,
May 19, 2012, 6:00:57 PM5/19/12
to jcl...@googlegroups.com
Hi Paolo,

The symptoms sound very similar to those reported in [1].

There, we fixed sshj to report the more helpful "
Invalid packet: indicated length %d too large", rather than going into an infinite loop.

Could the underlying cause for you be related to what Alex thought:

I think I am using the wrong credentials, based on a log message, but I 
think the credentials is confusing so maybe that could be cleaned up. 

The log message was:

Received SSH_MSG_DEBUG (display=false) 'Forced command: echo 'Please login as the ubuntu user rather than root user.';echo;sleep 10'

Aled

[1] https://groups.google.com/group/jclouds-dev/browse_thread/thread/c17d71019c7fc932/a3a990bb7c271efd?lnk=gst&q=sshj+hangs#a3a990bb7c271efd
[2] https://gist.github.com/2489273

Aled Sage

unread,
May 19, 2012, 7:06:03 PM5/19/12
to jcl...@googlegroups.com
I've just submitted issue https://github.com/shikhar/sshj/issues/75: Invalid packet exception ssh'ing as root when "Please login as the ubuntu user rather than root user

My guess is that's the underlying cause.

Aled

Paolo Di Tommaso

unread,
May 19, 2012, 7:35:34 PM5/19/12
to jcl...@googlegroups.com
Hi guys,

Thanks for your feedbac. I'm also thinking that something related to wrong credentials, but also there should be something broken in my workflow. 

I'm trying to create a custom AMI starting from a Amazon Linux AMI. What I do is the following: 

1) Launch the Amazon Linux AMI from AWS dashboard using the default settings 
2) Log in to the instance using the Amazon provided key and install sw packages  
3) Create a new AMI with the  ec2-create-image command 
4) When the AMI is ready I try to launch it with JClouds using my own key-pair and user-name. I use this code: 

AdminAccess admin = new AdminAccess.Builder()
                            .adminUsername( conf.userName ) 
                            .adminPublicKey( conf.publicKeyFile )   
                            .adminPrivateKey( conf.privateKeyFile )
                            .build()
templateOptions.runScript(admin)


After that I get the Invalid packet: indicated length 1349281121 too large exception. 



I've tried also the following: 

1) Launch the Amazon Linux AMI with JClouds using my own key-pair 
2) Customize the instance 
3) Create a new image 
4) Launching the new AMI with JClouds with the same credentials, but in this case I get another exception 

net.schmizz.sshj.userauth.UserAuthException: Exhausted available authentication methods
at net.schmizz.sshj.userauth.UserAuthImpl.authenticate(UserAuthImpl.java:114) ~[sshj-0.8.0.jar:na]
at net.schmizz.sshj.SSHClient.auth(SSHClient.java:205) ~[sshj-0.8.0.jar:na]
at net.schmizz.sshj.SSHClient.authPublickey(SSHClient.java:305) ~[sshj-0.8.0.jar:na]
at net.schmizz.sshj.SSHClient.authPublickey(SSHClient.java:324) ~[sshj-0.8.0.jar:na]
:


How do you create custom AMI to be used with JClouds ? 


Cheers,
Paolo

Adrian Cole

unread,
May 19, 2012, 8:27:34 PM5/19/12
to jcl...@googlegroups.com

Inline


On May 19, 2012 4:36 PM, "Paolo Di Tommaso" <paolo.d...@gmail.com> wrote:
>
> Hi guys,
>

> Thanks for your feedbac. I'm also thinking that something related to wrong credentials, but also there should be something broken in my workflow. 
>
> I'm trying to create a custom AMI starting from a Amazon Linux AMI. What I do is the following: 
>
> 1) Launch the Amazon Linux AMI from AWS dashboard using the default settings 
> 2) Log in to the instance using the Amazon provided key and install sw packages  
> 3) Create a new AMI with the  ec2-create-image command 
> 4) When the AMI is ready I try to launch it with JClouds using my own key-pair and user-name. I use this code: 
>
> AdminAccess admin = new AdminAccess.Builder()
>                             .adminUsername( conf.userName ) 
>                             .adminPublicKey( conf.publicKeyFile )   
>                             .adminPrivateKey( conf.privateKeyFile )
>                             .build()
> templateOptions.runScript(admin)
>
>
> After that I get the Invalid packet: indicated length 1349281121 too large exception. 

Ok. AdminAccess is about the user to install once the vm is booted.  Jclouds needs to login in order to create this user.  Jclouds has parsers to decide the name of the login user for some common amis.  This is by convention as EC2 exposes no API about the login name of a particular image.  For custom amis, jclouds will have no idea what the login user should be, and blindly attempts with root.  If you are using a custom ami which doesn't allow root login, you need to specify the login user and likely also the private key you installed.

This would be the TemplateOption overrideLoginCredentials

Again AdminAcess is to install a user overrideLoginCredentials is to specify one.

Paolo Di Tommaso

unread,
May 19, 2012, 8:59:33 PM5/19/12
to jcl...@googlegroups.com
I see, much more clear. Thanks Adrian. 

But I never understood one thing. How JCouds does to install a new user without having to provide the Amazon generated private key? As far I know is the only way to ssh access the instance. 


Best,
Paolo

Adrian Cole

unread,
May 19, 2012, 9:05:16 PM5/19/12
to jcl...@googlegroups.com

We generate a temporary key.. that's the secret ingredient ;)

We are missing an example of using our image extension to cover the use case you are doing.  Ill try to nudge that along this week.

Paolo Di Tommaso

unread,
May 20, 2012, 5:48:02 PM5/20/12
to jcl...@googlegroups.com
Hi Adrian, 

Finally I've solved the problem following your tips. Thanks.


Let me know about the example you are referring. I'm interested in that. 


Best,
Paolo
Reply all
Reply to author
Forward
0 new messages