Linode branch
Jed Smith
Requesting a pull of my fork:
http://github.com/jedsmith/libcloud/tree/master
I added a CONTRIBUTORS file similar to how the Linux kernel does things;
my thought here is that a one-stop shop of "who's responsible for what?"
would be beneficial. I got tempted to fill out README and NOTICE and
such but I figured there's a plan for those.
The work done here:
- Linode has everything implemented, save for "Create" (--> 22c419b)
- CONTRIBUTORS created and a shell filled in (b1a060e, 05e96ed)
- Small typos fixed (72ef99, db4ada)
- One particularly annoying bug fixed, kinda (dceccb0)
The Linode API does not Keep Alive. httplib would exception
if the same connection were used twice. There might be a better
way to handle this, but this looked right at the time.
Looking for feedback.
Also of note: we're debating the best way to implement Create support.
The Create workflow in the Linode API goes like this:
linode.create Create the Linode
linode.disk.createfromdistribution Deploy Linux
linode.disk.create Make a swap
linode.config.create Make a config to tie it all
linode.boot And boot
Steps 1, 2, and 3 have to be executed sequentially, and blocking done to
wait for them to succeed before moving on. How best can we implement
this in the libcloud mode?
We modified linode.delete to rectify Tom's (quite valid) concern about
delete_node() blocking for 3 minutes waiting on disks to delete.
delete_node() now fires off the job and forgets about it, with an added
"please remove the safety catch so I can slice off my finger" parameter.
Regards,
Jed Smith
Systems Developer
Linode, LLC
jsm...@linode.com
+1 (609) 593-7103 x1209
Paul Querna
Can't linode put it under the same license as the rest of the code?
(The effective 'real world difference between MIT and Apache is
minimal, is there a problem with the Apache Licnese I'm not seeing?)
Thanks,
Paul
Jed Smith
> I am opposed to license diversification.
Then why, in commit b13303f, did you add this to every source file:
# Licensed to libcloud.org under one or more
# contributor license agreements.
That's an invitation for contributors to license code to libcloud at
will, with a license of their choice. I, personally, prefer MIT and I
read that comment as permission to use it.
If this is a hang-up issue, I'll be happy to revisit it.
The way I see it, Linode is licensing the code to libcloud.org, who is
then relicensing it in their distribution. Am I wrong?
-Jed
Alex Polvi
> If this is a hang-up issue, I'll be happy to revisit it.
>
> The way I see it, Linode is licensing the code to libcloud.org, who is
> then relicensing it in their distribution. Am I wrong?
Would you be willing to license it as same as the rest of the code,
just to make everything consistent and easy? The actual effect of
MIT/Apache is nearly identical.
If we wanted to make everything really "proper" we would need to sign
contributor license agreements ... but I'm trying to avoid that until
absolutely necessary. If you need that level of formality to make a
contribution -- happy to go down that path...
If this is about Linode and you getting credit for the contributions
-- we will definitely do that! (on the website, in the NOTICES file,
etc)
Still working on reviewing the rest of the patch....
-Alex
--
co-founder, cloudkick.com
twitter.com/cloudkick
541 231 0624
Jed Smith
anything. I'm not the self-horn-toot type :)
The multiple license agreements clause in the comment needs to go, then
-- can I propose this?
#
# libcloud
# A Standard Interface into the Cloud
#
# Copyright (C) 2009 libcloud.org and contributors.
# Released under license. See LICENSE for more information.
#
Throwing a patch in my tree now to relicense.
-Jed
Jed Smith
> Still working on reviewing the rest of the patch....
Moar for you to review!
http://github.com/jedsmith/libcloud/tree/linode_create
The tip of that branch is create_node() support. It's untested in a
live, credit-card-charging environment but works (with some coaxing)
against our beta stack. master remains at "all but create_node(),"
since you're already reviewing that.
That brings Linode support to full -- I'll be throwing it around among
our customers and having them try it out. You are certainly welcome to
do the same, and I'd love your feedback on it. It's amazing how big
create_node() is compared to the rest of the driver...wowsas.
Is MockHttp the preferred method for unit tests? Unit tests are next
and will likely come early next week. I tried running unit tests in the
current distribution but there seems to be symbol issues with EC2?
Should I wait for that to settle down?
Regards,
Jed
Jed Smith
> Feel free to merge this in too.
Done and pushed.
> Seriously, might consider "pick all the defaults for me" option for the API
I know you're kidding, but in seriousness we've weighed the benefits of
both. Linode has never really been a 'defaults' VPS provider, and it's
tough to draw the line between picking what we think is right with the
API, and still giving customers the configurability we've always
offered. I think I've struck a healthy balance, but the code weight
suffers for it.
> Yes please, as my only hesitation from merging it are that there are
> no test cases. Just need some tests that make sure each of the driver
> functions work.
In the queue.
> What errors did you see with EC2? Everything looks fine from my end.
---------------------
jsmith@lindev7:~/libcloud$ python2.5 setup.py test
running test
Traceback (most recent call last):
File "setup.py", line 56, in <module>
cmdclass = { 'test': TestCommand }
File "/usr/lib/python2.5/distutils/core.py", line 151, in setup
dist.run_commands()
File "/usr/lib/python2.5/distutils/dist.py", line 974, in run_commands
self.run_command(cmd)
File "/usr/lib/python2.5/distutils/dist.py", line 994, in run_command
cmd_obj.run()
File "setup.py", line 42, in run
tests = TestLoader().loadTestsFromNames(testfiles)
File "/usr/lib/python2.5/unittest.py", line 565, in loadTestsFromNames
suites = [self.loadTestsFromName(name, module) for name in names]
File "/usr/lib/python2.5/unittest.py", line 541, in loadTestsFromName
parent, obj = obj, getattr(obj, part)
AttributeError: 'module' object has no attribute 'test_ec2'
---------------------
Maybe the way I'm running it?
-Jed
Jed Smith
> Yes please, as my only hesitation from merging it are that there are
> no test cases.
I've added test cases for all calls that do not change API state, in my
lintests branch (and slurped your base into it, thanks):
http://github.com/jedsmith/libcloud/tree/lintests
For the calls that alter API state, I'm fighting with how to make that
work with MockHttp. It seems counterproductive to test these calls with
static data, given that three of them are supposed to change
state...reducing the API to "if create_was_called: return
ResponsePlusNewNode" just doesn't feel right. Thoughts?
What sucks is that I can't really think of an alternative to this
system. We provide a sandbox environment currently, but it will go
away. I don't know, I'm coming up dry.
-Jed
Alex Polvi
> Alex Polvi wrote:
>> Yes please, as my only hesitation from merging it are that there are
>> no test cases.
>
> I've added test cases for all calls that do not change API state, in my
> lintests branch (and slurped your base into it, thanks):
>
> http://github.com/jedsmith/libcloud/tree/lintests
The only extra test I see is list_images and list_sizes, is that correct?
>
> For the calls that alter API state, I'm fighting with how to make that
> work with MockHttp. It seems counterproductive to test these calls with
> static data, given that three of them are supposed to change
> state...reducing the API to "if create_was_called: return
> ResponsePlusNewNode" just doesn't feel right. Thoughts?
Definitely could be better -- but at least static data is a start.
This allows us to make sure interfaces and response parsing are not
broken.
Just curious, why did you base64 encode the responses? The python """
quoting should be able to do whatever you want, and maintain
readability.
> What sucks is that I can't really think of an alternative to this
> system. We provide a sandbox environment currently, but it will go
> away. I don't know, I'm coming up dry.
Yeah, lets just start with basic tests, so we at least have a sample
of what all responses should look like.
Jed Smith
> The only extra test I see is list_images and list_sizes, is that correct?
Correct. The rest of the calls change something in our API, and I'm not
sure of the best way to attack that.
> Just curious, why did you base64 encode the responses? The python """
> quoting should be able to do whatever you want, and maintain
> readability.
Inline JSON, to me, really pollutes the Python. I'm not a giant fan of
Python heredoc quoting nonetheless (due to Python's indenting
requirements), and code that jumps from indented two blocks in to flush
left really bugs me in a text editor.
Granted, I've replaced it with something 4.5k characters wide...
I don't know, that was a stylistic kneejerk.
J
Alex Polvi
Yeah, I would much prefer the weird indentation over not being able to
read the response w/out decoding.
Jed Smith
> Yeah, I would much prefer the weird indentation over not being able to
> read the response w/out decoding.
Fixed and pushed in fed868c.
Jed
Alex Polvi
These look better. Could you get them merged into your main branch?
Also, please just take a stab at the rest of the interface methods ...
just so we can have tests that fail if we change interfaces. I realize
they are state altering, but something is better than nothing, IMHO.
Once all that is settled and merged into your main branch, I'll do
another review of everything and get it merged in to trunk. Also,
pquerna, tdavis, and jcsalterego all have commit access to the main
branch, so any of them can review if needed. :)
Have a good weekend,
Jed Smith
> These look better. Could you get them merged into your main branch?
>
> Also, please just take a stab at the rest of the interface methods ...
> just so we can have tests that fail if we change interfaces. I realize
> they are state altering, but something is better than nothing, IMHO.
Finished and merged into master. I'd appreciate another look :)
Regards,
-Jed
Alex Polvi
Jed -- sorry for the slow review. I keep getting errors when running the tests ... is this expected?
local-cloudkick-com:libcloud polvi$ python setup.py test
running test
..........EE.E.E.............
======================================================================
ERROR: test_create_node (test.test_linode.LinodeTest)
----------------------------------------------------------------------
Traceback (most recent call last):
node = self.driver.create_node("Test", distro, size, root="test")
File "/Users/polvi/Code/libcloud/libcloud/drivers/linode.py", line 238, in create_node
raise LinodeException(0xFB, "Root password is too short")
LinodeException: (251) Root password is too short
======================================================================
ERROR: test_destroy_node (test.test_linode.LinodeTest)
----------------------------------------------------------------------
Traceback (most recent call last):
self.driver.destroy_node(node)
File "/Users/polvi/Code/libcloud/libcloud/drivers/linode.py", line 154, in destroy_node
self.connection.request(LINODE_ROOT, params=params)
File "/Users/polvi/Code/libcloud/libcloud/base.py", line 218, in request
headers=headers)
File "/Users/polvi/Code/libcloud/test/__init__.py", line 121, in request
meth = getattr(self, meth_name)
AttributeError: 'LinodeMockHttp' object has no attribute '_linode_delete'
======================================================================
ERROR: test_list_nodes (test.test_linode.LinodeTest)
----------------------------------------------------------------------
Traceback (most recent call last):
self.assertEqual(node.id, 8098)
NameError: global name 'node' is not defined
======================================================================
ERROR: test_reboot_node (test.test_linode.LinodeTest)
----------------------------------------------------------------------
Traceback (most recent call last):
self.driver.reboot_node(node)
File "/Users/polvi/Code/libcloud/libcloud/drivers/linode.py", line 147, in reboot_node
self.connection.request(LINODE_ROOT, params=params)
File "/Users/polvi/Code/libcloud/libcloud/base.py", line 218, in request
headers=headers)
File "/Users/polvi/Code/libcloud/test/__init__.py", line 121, in request
meth = getattr(self, meth_name)
AttributeError: 'LinodeMockHttp' object has no attribute '_linode_reboot'
----------------------------------------------------------------------
Ran 29 tests in 0.088s
FAILED (errors=4)
Regards,
-Jed
Jed Smith
> Jed -- sorry for the slow review. I keep getting errors when running the
> tests ... is this expected?
Negative.
[master 06fe6a9] Fix test failures reported by Alex Polvi
1 files changed, 10 insertions(+), 1 deletions(-)
Pushed. You can tell I'm working without a test framework; I still
haven't been able to actually /run/ the tests. Your feedback is
helpful, and that commit should fix all four failures.
-J
Alex Polvi
Alex Polvi wrote:Negative.
> Jed -- sorry for the slow review. I keep getting errors when running the
> tests ... is this expected?
[master 06fe6a9] Fix test failures reported by Alex Polvi
1 files changed, 10 insertions(+), 1 deletions(-)
Almost:
local-cloudkick-com:libcloud polvi$ python setup.py test
running test
======================================================================
ERROR: test_create_node (test.test_linode.LinodeTest)
----------------------------------------------------------------------
Traceback (most recent call last):
node = self.driver.create_node("Test", distro, size, root="test123")
File "/Users/polvi/Code/libcloud/libcloud/drivers/linode.py", line 258, in create_node
kernels = self.connection.request(LINODE_ROOT, params=params).object
File "/Users/polvi/Code/libcloud/libcloud/base.py", line 218, in request
headers=headers)
File "/Users/polvi/Code/libcloud/test/__init__.py", line 121, in request
meth = getattr(self, meth_name)
----------------------------------------------------------------------
Ran 29 tests in 0.087s
FAILED (errors=1)
Pushed. You can tell I'm working without a test framework; I still
haven't been able to actually /run/ the tests. Your feedback is
helpful, and that commit should fix all four failures.
What is keeping you from running the tests?
-Alex
Jed Smith
> Almost:
Knew I missed a few. When I read create_node() again, I missed dummy
HTTP calls for all methods above step 1. Forgot I did some sanity
checks in there.
[master 33c5871] And fix one more failure
1 files changed, 4 insertions(+), 0 deletions(-)
> What is keeping you from running the tests?
I honestly haven't had time to research the alternative frameworks you
proposed. I will rectify that one of these days (you know how our
career is).
-Jed
Alex Polvi
Alex Polvi wrote:
> Almost:
Knew I missed a few. When I read create_node() again, I missed dummy
HTTP calls for all methods above step 1. Forgot I did some sanity
checks in there.
[master 33c5871] And fix one more failure
1 files changed, 4 insertions(+), 0 deletions(-)
running test
..........E..................
======================================================================
ERROR: test_create_node (test.test_linode.LinodeTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/polvi/Code/libcloud/test/test_linode.py", line 51, in test_create_node
node = self.driver.create_node("Test", distro, size, root="test123")
File "/Users/polvi/Code/libcloud/libcloud/drivers/linode.py", line 258, in create_node
kernels = self.connection.request(LINODE_ROOT, params=params).object
response = self.responseCls(self.connection.getresponse())
File "/Users/polvi/Code/libcloud/libcloud/drivers/linode.py", line 65, in __init__
raise self.errors[0]
LinodeException: (255) Invalid JSON received from server
----------------------------------------------------------------------
Ran 29 tests in 0.060s
FAILED (errors=1)
> What is keeping you from running the tests?I honestly haven't had time to research the alternative frameworks you
proposed. I will rectify that one of these days (you know how our
career is).
All you should need to do is this:
python setup.py test
.. from the libcloud directory.
-Alex
-Jed
Jed Smith
> LinodeException: (255) Invalid JSON received from server
I thought that was fixed on our side -- that's not in this code, it's in
the API stack. I'll look at this.
> All you should need to do is this:
>
> python setup.py test
>
> .. from the libcloud directory.
We discussed this before:
jsmith@lindev7:~/libcloud$ python setup.py test
running test
Traceback (most recent call last):
File "setup.py", line 56, in <module>
cmdclass = { 'test': TestCommand }
File "/usr/lib/python2.6/distutils/core.py", line 152, in setup
dist.run_commands()
File "/usr/lib/python2.6/distutils/dist.py", line 975, in run_commands
self.run_command(cmd)
File "/usr/lib/python2.6/distutils/dist.py", line 995, in run_command
cmd_obj.run()
File "setup.py", line 42, in run
tests = TestLoader().loadTestsFromNames(testfiles)
File "/usr/lib/python2.6/unittest.py", line 613, in loadTestsFromNames
suites = [self.loadTestsFromName(name, module) for name in names]
File "/usr/lib/python2.6/unittest.py", line 584, in loadTestsFromName
parent, obj = obj, getattr(obj, part)
AttributeError: 'module' object has no attribute 'test_ec2'
-j
Tom Davis
Have you tried the latest cloudkick/master? I had this issue crop up again too in a previous tree, but it appears to be fixed... again.
Jed Smith
> Jed,
>
> Have you tried the latest cloudkick/master? I had this issue crop up
> again too in a previous tree, but it appears to be fixed... again.
Yes, I'm branched from HEAD of cloudkick/master (6b08f4).
I've never been able to run tests, for what it's worth.
Jed
Jed Smith
In my tree I disabled all tests except mine by renaming them to
DISABLED_{...} and was able to get `python2.5 setup.py test` to work.
I've now banged out the remainders:
[master 79a3aa7] Fix tests; all run now
1 files changed, 1 insertions(+), 4 deletions(-)
Please verify that these now run for you.
Regards,
Jed
Alex Polvi
Your changes are merged into trunk! Thank you so much for all your help on this. Would you (or one of the Linode founders) be at all interested in providing a comment for libcloud.org (see slicehost/rackspace/gogrid quotes on the top of the site)?
Thanks again for all your work getting Linode integrated. I look forward to continuing to improve this project with you as it grows!
-Alex
Jed Smith
> Jed,
>
> Your changes are merged into trunk!
Fantastic! We all celebrated a bit in the office. It looks great, all
that Yes Yes Yes on libcloud.org.
> Would you (or one of the Linode founders) be at all interested in
> providing a comment for libcloud.org (see slicehost/rackspace/gogrid quotes
> on the top of the site)?
Sure!
Here's mine -- I'm sure Chris will have another (the more the merrier,
I'm assuming):
"In the IaaS market it's important for all of us to demonstrate the
effectiveness of cloud computing. libcloud is a valuable
standardization effort that allows vendors to connect with customers in
new ways. We are very excited about the potential of this project."
Jed Smith
Systems Developer
Linode, LLC
Regards,
Jed
Christopher S. Aker
>> Your changes are merged into trunk!
Great news...
> I'm sure Chris will have another
libcloud represents a fundamental change in the way clouds are managed,
breaking the barriers of proprietary, closed clouds. We at Linode
believe this is of the utmost importance and fully support this effort.
Christopher S. Aker, Founder, Linode
-Chris