Ec2 Plugin: Preparing PR to add support for Launch Templates

[Ian Murray]

Hi,

I am in the process of putting together a PR for Ec2 plugin to support launch templates. The advantage for me is that I can use an external tool such as terraform to manage my slave configurations. I have made quite decent progress in as much as I have the plugin able to launch with a template.

It is a bit of a paradigm shift because it means Jenkins doesn't know as much as it used to about a launched instance, i.e. AMI, type, etc.

However, I have come across two problems that are causing me to rethink the approach.

Firstly, reading the code, it seems that it filters for stopped instances that can be restarted  based on known things like AMI, type etc. If I simply tag on the launch template at launch time, then Jenkins doesn't know these things. I thought I would be able to filter by launch template/version, but this doesn't seem to be possible. So I am thinking that I am better pulling what is needed from the launch template and populating the request with those values, i.e. the plugin handles things, rather than the AWS launch template subsystem. This feels wrong, but I can't see a simpler solution.

The second problem is somewhat similar. I was looking how to implement this for spot requests and it seems that the API doesn't allow you to specify a launch template (unless I missed it!). You can launch it in the gui, but I am guessing that the gui does the pulling of the data rather than the API. So, I think I am going to have to implement something like the above for spot, anyway,

This is my first attempt at anything like this, so I might have misunderstood some conventions, so please feel free to comment on anything that is seen as amiss. 

https://github.com/dlcc/ec2-plugin/pull/1/files

I also intend to throw in `unlimited` t2 option as well, while I am on.

Thanks for reading,

Ian

[Baptiste Mathus]

Hello Ian,

Seems like you're designing something pretty important and ambitious for the EC2 plugin, thanks a lot!

I was thinking: given you're doing this big change, I think it would make sense to write a Jenkins Enhancement Proposal (aka JEP) for it WDYT?

JEPs are now the official way in the Jenkins Project to get feedback that goes beyond a single & trivial PR, and gather feedback on design before diving into it. In that case for instance, I guess the people having shown their interest for the Cloud-native SiG in the other thread would be put in the list of reviewers for your proposal.

I fear/feel that going through this stage would help you get feedback, and then the PR would be an implementation /detail/ (not saying it's easy though).

In your case, the fact you have already started implementing it is *not* an issue, it's actually even a good thing as JEPs refer explicitly to impls.

Given your PR already has 17 files modified and hundreds of lines added, I think the JEP would actually help move it forward in a sane way.

I can definitely help with the process if you are OK to do it. Even jump into a hangout to give details, just tell me.

-- Baptiste

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/60e2ba77-f170-4772-a3ea-5fb771e9327a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ian Murray]

Hi,

Thank you for the interest. I wasn't aware of JEP, so it would make sense to go in that direction. It doesn't look too onerous. I don't regret what I have done so far, because only now do I understand the real challenges, which are much bigger than I originally anticipated. I am still not confident I understand the existing code, even.

Thank you for the offer of help. I think I can manage the JEP by myself, but would certainly appreciate your advice and commentary as I progress. As I already have a WIP, early comments would be appreciated, because I would probably abandon the idea right now, if I was completely off track.

Thank-you again,

Ian.

[Ian Murray]

Just so I can find this when I need it, I put a support call into AWS regarding the launching of spot with launch template, and they responded:

Hello, The documentation for request-spot-instances confirms there is no parameter for launch configurations[1], and consequently this option is not available via the API. The console actually doesn't specifically call request-spot-instances, it is always a spot fleet request(you can verify this when creating a "one time request", which returns sfr-xxxxxxx) no matter which you option you select. This is why you see launch configuration as an option in the console, and explains why it's also available via the API for spot fleet requests and not request-spot-instances. With this in mind, this leaves you with one of 2 options: - Use spot fleet and specify 1 as the target capacity for single requests. - As you mentioned, you can use RunInstances API call and specify spot[2] under "MarketType". This will launch a spot-request and use the specified launch template under "LaunchTemplateName". I hope the above answers your query, let me know if you have any further questions. [1]https://docs.aws.amazon.com/cli/latest/reference/ec2/request-spot-instances.html [2]lhttps://docs.aws.amazon.com/cli/latest/reference/ec2/run-instances.html Best regards, Amazon Web Services

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/788613ae-448c-4505-9216-94d241160f17%40googlegroups.com.