EC2, VPC and security groups

[barkbarkuk]

I've been trying to use pallet and jclouds together to create some machines in the VPC, but have been struggling a little with security groups.

After a while I stumbled across the VPC section in http://www.jclouds.org/documentation/userguide/using-ec2/ which states that the VPC and security groups are mutually exclusive but I don't believe this is the case anymore.

I've had a bit of a dig through the code and lost myself a few times but it looks like the interface to make calls to create a security group with VPC specified already exists in the aws-ec2 provider.
When creating a new instance however it looks like when a subnet id is specified (i.e. the machine should be created in a specific subnet in the VPC) any named security groups are intentionally not added. This does make sense as the amazon api's don't allow a security group name to be passed when creating a VPC instance though they do allow a security group id to be passed.

I'm wondering if there are docs/discussions where this has been talked about?
If not, where's best to target my efforts as I'm happy to get my hands into the code with a little direction?

I think when a non-VPC instance is created, passing the name of a security group that doesn't exist, the group is automatically created by AWS as part of the same call that creates the instance. For a VPC security group it seems like the group would have to be created manually by jclouds code first, the id retreived and then passed as an option into the instance creation details. I was originally thinking about doing this in the code that chooses not to supply security group names when a subnet id is detected, using the subnet id to get the VPC id needed to create the new security group. This seems a bit wrong though as the code is more about building up options for the create instance call to AWS.

Maybe the right thing is to be explicitly creating VPC security groups in pallet and making sure the ids can be passed on through jclouds?

The other thing to consider is the security group name restrictions are different when making a VPC security group, notably the # symbol is not allowed.

Sorry if this is a bit rambly, trying to understand a lot of new code and how all the parts fit together.
Any direction gratefully received.
- Pete.

[Adrian Cole]

Hi, Pete.

Let me try to summarize a bit. Please correct below, if they aren't correct.

You are using pallet/clojure to create ec2 instances in clojure.

To use existing resources for these, you'd use :subnet-id and
:security-group-ids kwargs

If you don't supply :security-group-ids, you'd expect a default group
to be created in the correct subnet with the ingress rules from
:inbound-ports, but this doesn't work.

Is this right?
-A

> --
> You received this message because you are subscribed to the Google Groups "jclouds" group.
> To view this discussion on the web visit https://groups.google.com/d/msg/jclouds/-/rZYbYsCY7jAJ.
> To post to this group, send email to jcl...@googlegroups.com.
> To unsubscribe from this group, send email to jclouds+u...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/jclouds?hl=en.
>

[Peter Hibbert]

Hi Adrian,

You are using pallet/clojure to create ec2 instances in clojure.

Yes.

(long answer: I'm actually using storm-deploy (clojure) -> pallet -> pallet-jclouds -> jclouds)

 

To use existing resources for these, you'd use :subnet-id and
:security-group-ids kwargs

Ah, ok thanks, I'll have a bit of a dig and see if I can make that work out for me.  

IIRC pallet was creating a security group name automatically to group together multiple instances and then failing to pull those instances back into groups following their creation because the group name wasn't attached to the instance. The group name being created contained '#' symbols so wouldn't work as a VPC security group name. I need to go back into pallet land and check the plumbing to see if I can create the security group myself upfront and have it used to successfully pull those machines together after instance creation.

 

If you don't supply :security-group-ids, you'd expect a default group
to be created in the correct subnet with the ingress rules from
:inbound-ports, but this doesn't work.

Yes, I expected a default group to be created either by jclouds or pallet, not really sure where that responsibility lies. If the responsibility is in the calling code going into pallet, then that's ok as long as I know it has to be done.

I guess what I hoped would happen is when I supplied a subnet id in the node-spec for pallet that everything would work just the same as without subnet specified (Admittedly coming in a bit naive as to how all the layers plugged together).

 

Is this right?

Yes, I think you've understood my issue, probably somewhat better than I have! 

Thanks.

  - Pete. 

[Adrian Cole]

I'm inclined to drop use of security groups in aws-ec2 for tags, as
we've had way too many issues related to encody things. thoughts?

http://code.google.com/p/jclouds/issues/detail?id=835

-A

> https://groups.google.com/d/msg/jclouds/-/VN00MtrDHiAJ.

[Peter Hibbert]

Good idea is my initial thought. 

It seems like it would be cleaner to be able to group on something other than security groups, though I'm still trying to organise a few brain cells to see how the pieces fit together so that may be a naive comment.

Also, in AWS there is a limit on number of security groups (which is quite a bit lower when using VPC security groups), so just from that perspective it would be a good thing.