Linode libcloud support completed (differently)
Jed Smith
Hash: SHA1
Alex,
This commit in my fork of libcloud:
http://github.com/jedsmith/libcloud/commit/d9c9b513ffeaf44b550e662538ac012a3f7de0f5
...completes the entirety of Linode support in libcloud. It is tested
and works. This takes a different approach from Ryan Tucker -- which I
believe is what you had the issue with -- and implements Linode support
in a manner similar to your other providers (with no external dependencies).
We're extremely interested in getting on board with you, and changing
all those red Nos on your homepage to at least three Yeses. Does this
diff fit better with your vision for libcloud? Please let me know if it
does not, and I will work with you to get it in.
I'm also ready to commit create support when you decide upon an API. We
are more than willing to assist with that process.
Yours,
Jed Smith
Systems Developer
Linode, LLC
jsm...@linode.com
+1 (609) 593-7103 x1209
PGP: 0xA6611ED6
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Darwin)
iEYEARECAAYFAkp5FVMACgkQgNwvNaZhHtYubwCffoa1iT690jvstW5UuBADNK0k
q5UAni/v+nHROpCHq9aTx7qtLFGhhxRR
=7IQk
-----END PGP SIGNATURE-----
Alex Polvi
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Alex,
>
> This commit in my fork of libcloud:
>
> http://github.com/jedsmith/libcloud/commit/d9c9b513ffeaf44b550e662538ac012a3f7de0f5
Jed, looking better! I'm going to push some interfaces ASAP, and
you'll probably need to take another look at writing a driver.
Take a lot at this slicehost implementation ... the new stuff will
look a lot more like this:
> I'm also ready to commit create support when you decide upon an API. We
> are more than willing to assist with that process.
... in the mean time, this would be awesome. Right now we're trying to
define the API -- the hard part about this is we need to send provider
specific details. Here is one approach, what do you think?
Essentially we are assuming that each provider needs a size and an
image in order to boot. Of course there are more details to this, but
this is the bare minimum.
Any feedback here would be awesome.
-Alex
--
co-founder, cloudkick.com
twitter.com/cloudkick
541 231 0624
Jed Smith
This is really, really strange code. Why are the private methods of the
node driver (i.e., to_nodes) now mandated to be public in INodeDriver?
For that matter, why are functions like that -- which /work/ on response
objects -- members of INodeDriver and not IResponse? What the heck is
the purpose of IResponse::parse_error, and why does it fall through to
returning the body if there is no error...?
This design is probably the strangest object-oriented structure I have
ever seen in my programming career. I'm sorry, but it is. This is the
entirely wrong direction to pull libcloud in, and you'll suffer for it.
The assumption that every API works over HTTP is problematic. While all
the providers you work with today do, what's to stop a VPS company from
offering an entirely different protocol for transfer? You are then
looking at reimplementing your entire library to provide support for
them. Instead, you really need to let drivers abstract away their API
details from the rest of the library.
I would love to implement Linode support for this, but I am having
extreme difficulty grokking the new libcloud. In my opinion, in
LinodeNodeDriver I should be completely isolating all the details of my
API away from the rest of libcloud...Connection details, Response
details...why is this now no longer constrained to my driver? Why is
the rest of the library getting involved in API details -- something I
thought libcloud was designed to avoid?
This change has shot the complexity of libcloud through the roof, and
made it extremely difficult for outsiders to contribute code. Why have
we overcomplicated libcloud to fifteen separate interfaces?
I'd like to see something like this --
libcloud/
__init__.py
libcloud.py General package functionality
Node.py The Node object
Credential.py A general Credential object
IDriver.py The interface for drivers
drivers/
__init__.py Package-o-drivers
Linode.py Linode's IDriver
Slicehost.py Slicehost's IDriver
Amazon.py Amazon's EC2 IDriver and EU IDriver
import libcloud
lcreds = libcloud.Credential()
screds = libcloud.Credential()
lcreds["key"] = "abc123"
screds["key"] = "abc123"
screds["http"] = ("user, "pass")
linode = libcloud.driver("Linode", lcreds)
slhost = libcloud.driver("Slicehost", screds)
nodes = libcloud.enumerate([linode, slhost])
for node in nodes:
node.reboot()
lplan = {
"Plan": 360,
"OS": "Debian",
"PrivateIP": True,
"Linode:Datacenter": "Fremont",
"Linode:BillingCycle": 12
}
splan = {
"Plan": 256,
"OS": "Debian",
"PrivateIP": True,
"Slicehost:UniqueFeat": 5,
"Slicehost:ETC": True
}
lnode = linode.create(lplan)
snode = slhost.create(splan)
lnode.boot()
snode.boot()
I was hoping simplicity like this is the direction you were heading for,
and I'm really disappointed that it's not.
Obviously it'd be a conflict of interest for me to rewrite libcloud for
you, but ... Alex ... I need your help here. Are you strongly committed
to this direction? There are a million better ways to do what you're
trying to do, while minimizing overhead for both you and the providers
you're trying to support.
I'll try to implement Linode's support into your current schema, but I
really hope I'm not wasting effort again.
Paul Querna
>
> Alex,
>
> This is really, really strange code. Why are the private methods of the
> node driver (i.e., to_nodes) now mandated to be public in INodeDriver?
I agree, some of the things, specifically to_nodes likely should not
be in the INodeDriver, but almost all of the rest of them really do
work well, at least that is my opinion after seeing most of it for the
first time last night while implementing the Rackspace driver.
> For that matter, why are functions like that -- which /work/ on response
> objects -- members of INodeDriver and not IResponse? What the heck is
> the purpose of IResponse::parse_error, and why does it fall through to
> returning the body if there is no error...?
> This design is probably the strangest object-oriented structure I have
> ever seen in my programming career. I'm sorry, but it is. This is the
> entirely wrong direction to pull libcloud in, and you'll suffer for it.
>
> The assumption that every API works over HTTP is problematic.
I disagree, the API doesn't really mandate HTTP -- much more about
requests / response patterns, which even if your transport was
something else, is a pretty consistent programming pattern for
communication over the internet.
> While all
> the providers you work with today do, what's to stop a VPS company from
> offering an entirely different protocol for transfer? You are then
> looking at reimplementing your entire library to provide support for
> them. Instead, you really need to let drivers abstract away their API
> details from the rest of the library.
I disagree, that road takes you to just wrapping random APIs, and the
consistency between the different providers drops considerably, while
with this approach it is very easy to write full test cases for all
providers, and to drive them completely offline without credentials,
which if it was not done this way, would need to be implemented
completely from scratch for every provider.
> I would love to implement Linode support for this, but I am having
> extreme difficulty grokking the new libcloud. In my opinion, in
> LinodeNodeDriver I should be completely isolating all the details of my
> API away from the rest of libcloud...Connection details, Response
> details...why is this now no longer constrained to my driver? Why is
> the rest of the library getting involved in API details -- something I
> thought libcloud was designed to avoid?
>
> This change has shot the complexity of libcloud through the roof, and
> made it extremely difficult for outsiders to contribute code. Why have
> we overcomplicated libcloud to fifteen separate interfaces?
I don't agree with your conclusions, I hadn't done much in libcloud
before last night, and it only took me like 2 hours to knock out
Rackspace providers using the new interfaces, and they have full test
cases. Moving to a less centralized API, I doubt I could of gotten it
done nearly as quickly.
Thanks,
Paul
Jed Smith
> requests / response patterns, which even if your transport was
> something else, is a pretty consistent programming pattern for
> communication over the internet.
base.ConnectionKey disagrees with you. In fact, most of base.py does.
> I disagree, that road takes you to just wrapping random APIs, and the
> consistency between the different providers drops considerably,
How is that any different from what libcloud is doing today? A standard
is trying to be forged on how to communicate between providers --
wrapping each provider's API is a big part of that. That's essentially
what the design of libcloud mandates.
> I don't agree with your conclusions, I hadn't done much in libcloud
> before last night, and it only took me like 2 hours to knock out
> Rackspace providers using the new interfaces, and they have full test
> cases.
You had Slicehost to build upon. I banged out my first Linode plugin in
a shade under 3 hours, and that included familiarizing myself with the
code. That did not include test cases. With this new design,
everything is completely different, and I have been studying how to
tackle this all day.
There are larger strategic decisions at play in the design of our
plugin, as we've never forced a datacenter upon our customers before
(and we're weighing how best to do that). The new create_node system
does not give us any capability whatsoever to specify that.
Regards,
-Jed
Tom Davis
For that matter, why are functions like that -- which /work/ on response objects -- members of INodeDriver and not IResponse?
Think of an HTTP response. It knows its status code and whether or not it was a success, and it knows any errors it may have (Response.parse_error), but aside from that, it isn't the job of the response to interpret its contents. Sure, it knows the type of data its returning (text/html, etc.), but a response doesn't realize its carrying XML then parse that XML for you into some standardized object notation. That'd be silly -- the response doesn't know what you want to do with it! Like your browser, which is kind enough to display a response in an appropriate manner based on its mimetype and contents, NodeDrivers are aware of how to parse responses into various forms (sizes, images, nodes...)
What the heck is the purpose of IResponse::parse_error, and why does it fall through to returning the body if there is no error...?
The assumption that every API works over HTTP is problematic. While all the providers you work with today do, what's to stop a VPS company from offering an entirely different protocol for transfer? You are then looking at reimplementing your entire library to provide support for them.
I agree it is problematic, but after that I start to lose you. Whether it be via HTTP (highly likely for the foreseeable future), SOAP over SMTP, or something even crazier, it is a fair assumption that in order to use the API, one must make requests to which responses are received. This is the only thing our API really assumes.
Of course, at this moment base.py contains mostly examples for working with HTTP APIs and our interfaces assume that type as well -- it's all we've got! If a new type comes along, not a single thing has to be rewritten; we merely add an interface for the given transport. Here's what a Connection assumes to be true right now:
- Secure and insecure connections which (maybe) have different clients and (maybe) use different ports
- You will want to "connect" to something, make a "request" to some "action" over that connection
...why is this now no longer constrained to my driver?
Because that would mean we would be stuck finding a way to reliably test, vet, and maintain your driver... and Jim's driver, and Tim's driver, and Jane's driver, and...
Libcloud needs to maintain drivers for myriad providers, and we (the royal "we"; I don't control the master branch) simply cannot do that if we allow contributions which are untestable and unmaintainable to a base standard. If you know how one driver works, you effectively know how they all work. More importantly, you know how to test them. When Linode updates their API to support some new feature or changes the name of some parameter or whatever the case, I know almost exactly where to go to fix that issue and (first) where to go to write a new failing test for it... and I haven't even written an implementation!
Not only do we need to maintain many drivers, we also theoretically want to support many APIs as well. As Paul attested to earlier, once you look over the interfaces and check out an example driver, you can implement a new one extremely easily.
There are a million better ways to do what you're trying to do, while minimizing overhead for both you and the providers you're trying to support.
If strict standards of implementation and testing aren't in place, there's no way we could maintain viability as complexity, contributors, and providers increase. You say there are a million better ways, but you fail to offer even one of those million. The example you give basically offers up a wild-west of driver implementation which ignores both of these necessary standards. The code you originally supplied for inclusion is devoid of all documentation and tests; we've made it super easy for you test it now, though! Your destroy_node() blocks for up to 3 minutes then just gives up. If anything, this is a failing on our part to not be strict enough -- we don't even demand that methods respond timely with a suitable status (such as "pending" -- to check again later)!
base.ConnectionKey disagrees with you. In fact, most of base.py does.
Covered previously, but just in case: this assumption is made because it is currently the only protocol used and is thus a 100% safe assumption at this time. The file can be appended, however! It's turtles, errr, text all the way down!!
A standard is trying to be forged on how to communicate between providers -- wrapping each provider's API is a big part of that
This may be a semantic misunderstanding, but it seems like you want to wrap existing Python APIs with more Python to arrive at a unified API, which is synonymous with jamming whatever-sided peg you write into our square hole. This goes back to maintaining and reliably vetting your octagon against our square and John's pentagon.
With this new design, everything is completely different, and I have been studying how to tackle this all day.
But it's only more work for you. In aggregate, it greatly reduces the work load, which was the goal of this entire design. And while it may not be perfect yet (see below), it's a pretty damn good start, in my opinion. There are a few things I'd change in the newer code, but only because of inconsistency (INodeDriver deals with "images" and "sizes", but INode doesn't require an image or size associated with the node, etc), not "strangeness" (as an aside to that: potentially-private methods are public because interfaces can't define private methods and it makes a lot of sense to require distinct methods for operations which are prone to regression).
The new create_node system does not give us any capability whatsoever to specify that.
Fair enough, now it will: http://github.com/tdavis/libcloud/commit/771b477a529482fcf60fe0c33d2c84a16d5f4030
There are very few places where this need applies, however. create_node() was easily the most disparate of all the necessary interface attributes in terms of implementation, which is why I was too lazy to write it. I think Alex did an admirable job on his first pass, given the disparity. With this solution, you can even use dictionaries to pass arguments... without that superfluous and odd "Provider:arg" notation.
These interfaces and example implementations were released not because anybody thinks they're completely done (Alex has espoused an iterative style since the beginning), but because everyone involved feels they are a very good starting point. A couple providers have been implemented with much ease using the new style, but I'm pretty sure everybody realized it wasn't going to be a perfect fit for everybody. But we'll get there!
Thank you for the criticism, even if it was less-than constructive. And sorry for breaking your career-long streak of idyllic object-orientation.
Alex Polvi
> On Fri, Aug 7, 2009 at 2:37 PM, Jed Smith<jsm...@linode.com> wrote:
>>
>> Alex,
>>
>> This is really, really strange code. Why are the private methods of the
>> node driver (i.e., to_nodes) now mandated to be public in INodeDriver?
>
> I agree, some of the things, specifically to_nodes likely should not
> be in the INodeDriver, but almost all of the rest of them really do
> work well, at least that is my opinion after seeing most of it for the
> first time last night while implementing the Rackspace driver.
Based on this feedback, I've removed to_nodes/to_images/to_sizes from
the interface. You guys are right, it does not really make sense
there.
I've also merged in tdavis's create_node kwargs commit -- I was
actually thinking about doing the exact same thing, and he beat me to
it!
>> This design is probably the strangest object-oriented structure I have
>> ever seen in my programming career. I'm sorry, but it is.
... and I'm sorry you're having a hard time with it. It's really good
to know, because linode is one of the first providers that has stepped
up with development resources to help be part of libcloud. It's
important to the project that you guys can figure it out! Like I
mentioned in my other email, my next top priority is writing a drivers
writers guide. Maybe wait for that to be complete, then would to be
willing to give it another shot? I think it will help a bunch. Another
option, if it would be helpful, is I would be willing to write the
base driver (or maybe someone else?) and let you take over
maintainer-ship. Not ideal, but would help move things along!
Like others have mentioned, the architecture is meant to remove
complexity, add consistency, and make things maintainable. I really
think it does satisfy those goals.
Thanks again for your feedback!
-Alex
Jed Smith
> Because that would mean we would be stuck finding a way to reliably test,
> vet, and maintain your driver... and Jim's driver, and Tim's driver, and
> Jane's driver, and...
Not if the driver tests itself. Why does the global library need to
know about my connection details in order to test it? Why can't I
provide a test framework and cases, using 'unittest' as the case may be,
that you then call from whatever code you like?
The fact that you've reimplemented all sorts of HTTP craziness -- and
then excuse that by saying "we need it to unit test you," is pure
hogwash and you know it. What sort of Wild West are you referring to?
That a driver knows how it works and provides a consistent interface to
the client code calling it?
Heavens, the thought of that.
> Libcloud needs to maintain drivers for myriad providers,
No, it does not. Linode has made a commitment -- to Alex Polvi -- to
maintain our driver. The only drivers you need to maintain are the ones
you write; Slicehost is really getting a free ride here. We will own
it, and you don't have to get your panties in a bunch maintaining
something we throw at you. That's what having git and github and e-mail
and mailing lists is all about.
Us. Working with you.
> When Linode updates their API to
> support some new feature or changes the name of some parameter or whatever
> the case, I know almost exactly where to go to fix that issue and (first)
> where to go to write a new failing test for it...
We'd update the libcloud support as soon as we change something. That's
part of us officially saying "we support libcloud," linking to libcloud
in our API documentation...and that's part of my job description. Work
with you on this.
I must say, though, I'm getting less motivated as I communicate more
with the people supposedly in charge.
> Your destroy_node() blocks for up to 3 minutes then just gives up.
> If anything, this is a failing on our part to not be strict enough -- we
> don't even demand that methods respond timely with a suitable status (such
> as "pending" -- to check again later)!
You don't have any sort of standard whatsoever. Or documentation for
what things do. This is completely undocumented library that seems to
think rewriting itself every now and then is A Good Move and then yells
at providers who try to bow before its every whim.
> This may be a semantic misunderstanding, but it seems like you want to wrap
> existing Python APIs with more Python to arrive at a unified API, which is
> synonymous with jamming whatever-sided peg you write into our square hole.
I do? That patch was rejected, so I supplied a new one that didn't have
any external dependencies at all, save for simplejson in < Python 2.6.
I reimplemented a lot of the work in our Python bindings, which I
expected is what Alex wanted, and completely took our prewritten Linode
Python bindings out of the picture.
I felt bad enough snubbing the fine gentlemen who have put weeks of
testing and work and documentation into those bindings, as they are
customers and good friends of ours, but I felt it was important to
demonstrate to the libcloud steering folks that we are committed to
offering official Linode support.
It's pretty obvious you -- and I mean only you, Tim -- don't want that
from us, though.
> With this
> solution, you can even use dictionaries to pass arguments... without that
> superfluous and odd "Provider:arg" notation.
The idea there was that you wouldn't be changing ABI every time you want
to provide a new feature. I provided Provider:blah so that you could
define what is standard, what won't change...and then providers are free
to add things to that.
Every provider must implement "image", "size", etc...but for those
drivers who have differing implementations than the standard, a
guaranteed way to not collide with other drivers is set in stone. By
the people who are supposed to be setting things in stone.
It's a move toward standardization, which I got the impression was a
main interest of libcloud. Apparently not?
> Thank you for the criticism, even if it was less-than constructive. And
> sorry for breaking your career-long streak of idyllic object-orientation.
Can't separate criticism of the work from criticism of yourself?
Personal attacks against those attempting to improve your product -- who
are interested in helping you and providing what Linode support we can
-- is a good way for all of us providers to lose complete interest in
your product here and leave you to implement it.
I must say that all of us in the Linode community who read your message
were quite surprised by your response, and you've gotten our attention.
This is a dealbreaker for us, and I'm glad I waited for Alex to respond
to my message before pushing send, because I was ready to drop the
"remove our trademarks from everything" stanza.
Hesitantly yours,
-Jed
jcsalterego
Tom Davis
I came to this list (and Alex) following the initial libcloud release announcement. I took one look at the code and realized from experience that some things needed to change to ensure long-term ease of development. I made my case to the list and then followed up via IRC with some of the other initial contributors to the project. I implemented the core interfaces, base classes, and testing tools. I now spend most of my libcloud time fixing random bugs I find in the existing code and rewriting pieces of implementations where appropriate.
I didn't start this project and I certainly am not in charge of it. I wrote some code that people liked and it was merged in -- that's it. Do I think it is the only reasonable solution? Of course not. Do I feel it is far better than what was here? Yes. You are free to disagree with me as much as you like on that point.
Linode has made a commitment -- to Alex Polvi -- to maintain our driver.
Hey, it's *great* that Linode has stepped up to the plate and promised to maintain their own driver. But Slicehost isn't really getting a "free ride"; the assumption was never made that providers would maintain their own drivers. If that were the case, none of the structure would be necessary as responsibility would lie solely on each provider's shoulders to make sure things were maintained; Alex could write a few guidelines and go on vacation.
Unfortunately, that isn't the case. As it stands, libcloud only exists in its current form by the grace of non-provider contributions. The changes were put in place to make it *easier* to contribute -- not harder! We've already succeeded in doing that to some extent, but I'm sure it will only improve as more areas are found to generalize and abstract away.
The fact that you've reimplemented all sorts of HTTP craziness
Huh? Nothing has been re-implemented. We've simply provided convenient wrappers around request/response objects.
We'd update the libcloud support as soon as we change something. That's part of us officially saying "we support libcloud," linking to libcloud in our API documentation...
Right, fair enough. I shouldn't have used Linode as an example since you're committed to maintaining your own driver. The fact is though (as stated above), Linode is the exception and not the rule.
It's pretty obvious you -- and I mean only you, Tim -- don't want that from us, though.
My name isn't Tim, but that's neither here nor there. I was *in no way shape or form* a factor in getting your initial contribution (or any contribution, for that matter) rejected. In fact, I didn't even *look* at your code until last night when I pointed out the issues I took with it.
Every provider must implement "image", "size", etc...but for those
drivers who have differing implementations than the standard, a
guaranteed way to not collide with other drivers is set in stone.
There's an extremely simple guaranteed way: provide your own attributes and ivars to your NodeImage / NodeSize! **kwargs was added to create_node, so there's really no way for collisions to take place. We're not expecting the absence of additional attributes, we're simply setting a baseline for what you should have. If you don't have some of those attributes, then we made faulty presumptions.
Can't separate criticism of the work from criticism of yourself?
I was simply pointing out you did it in a way that was less than constructive. You are more than welcome to your opinion, of course, but starting off with an email titled "What the heck?" and expressing outrage over the choices that have been made while supplying rationale that only accounts for a single perspective (that of a provider who is committed to maintaining their own driver) and no concrete recommendations for improvement is not a great way to make friends. Following it up with a reply dripping with sarcastic and quasi-threatening remarks is an even worse way.
Maybe things were taken a little too far with the interfaces; I didn't write all of them, but I'll gladly accept your criticisms personally, if you like. Things aren't perfect and I have every confidence in Alex's ability to alter requirements where others agree they should be (as he's already done).
At this point it is obvious that the current contributors' opinion of this new design differs from yours and trying to convince you of its merits is a poor use of otherwise good coding time; this is the last time I'll be opining on it. If you feel we're wrong, I'm sure everybody (myself included) would gladly welcome code which improves the foundation of this project as a whole. Saying, "this makes life more difficult for me" does not such code make.
Cheers,
Tom