Request to improve RDS module to use boto rds2

[iain wright]

Hi folks,

The goal was to provision an rds instance using ansible, which was easy enough using this module: http://docs.ansible.com/rds_module.html

Unfortunately this module doesn't have the option to allow the RDS to pass in publicly_accessible and ec2 defaults making it publicly accessible to false. We'd like this seed our data on instance creation, proper security groups are used ofcourse.

I tried modifying the module to include publicly_accessible where needed but it looks the boto rds doesn't support it:

<localhost> REMOTE_MODULE rds size=10 password=VALUE_HIDDEN publicly_accessible=true command=create instance_type=db.m1.small instance_name=prod-rds2 engine_version=5.6 aws_access_key=AKIAJKMT2FVVHTPMEI6Q aws_secret_key=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 region=us-west-1 vpc_security_groups=sg-905fb6f5 db_engine=MySQL subnet=prod-rds username=admin
<localhost> EXEC ['/bin/sh', '-c', 'mkdir -p $HOME/.ansible/tmp/ansible-tmp-1402373676.78-265376028601009 && chmod a+rx $HOME/.ansible/tmp/ansible-tmp-1402373676.78-265376028601009 && echo $HOME/.ansible/tmp/ansible-tmp-1402373676.78-265376028601009']
<localhost> PUT /tmp/tmpDEylEv TO /home/iain/.ansible/tmp/ansible-tmp-1402373676.78-265376028601009/rds
<localhost> EXEC ['/bin/sh', '-c', u'LC_CTYPE=C LANG=C /usr/bin/python /home/iain/.ansible/tmp/ansible-tmp-1402373676.78-265376028601009/rds; rm -rf /home/iain/.ansible/tmp/ansible-tmp-1402373676.78-265376028601009/ >/dev/null 2>&1']
failed: [localhost] => (item={'username': 'admin', 'subnet': 'prod-public-rds', 'db_engine': 'MySQL', 'name': 'prod-rds2', 'instance_type': 'db.m1.small', 'password': 'omitted_from_mailing_list', 'engine_version': 5.6, 'size': 10}) => {"failed": true, "item": {"db_engine": "MySQL", "engine_version": 5.6, "instance_type": "db.m1.small", "name": "prod-rds2", "password": "omitted_from_mailing_list", "size": 10, "subnet": "prod-public-rds", "username": "admin"}, "parsed": false}
invalid output was: Traceback (most recent call last):
  File "/home/iain/.ansible/tmp/ansible-tmp-1402373676.78-265376028601009/rds", line 2048, in <module>
    main()
  File "/home/iain/.ansible/tmp/ansible-tmp-1402373676.78-265376028601009/rds", line 487, in main
    result = conn.create_dbinstance(instance_name, size, instance_type, username, password, publicly_accessible, **params)
TypeError: create_dbinstance() got an unexpected keyword argument 'publicly_accessible'

Boto does have rds2 which does seem to support this: https://github.com/boto/boto/blob/develop/boto/rds2/layer1.py

Is anyone interested in creating an rds2 module in ansible for this? Is this the proper place to request such a module or are they generally user provided?

Thank you,
iain

[Michael DeHaan]

It seems very non-Ansible-way to have another copy of the module that just uses a different protocol.

Couldn't this just be changed up to support the new parameter?

--Michael

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
To post to this group, send email to ansible...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/a4dd2940-0158-474a-9ffc-d2ec73e9b77a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[iain wright]

Hi Michael,

Thanks for the reply! That makes sense, I'm still learning the ansible/correct way :)

Do you think we should we pursue modifying the boto rds interface to support the param or should we modify the ansible rds module to use rds2 (which seems to support option groups and a couple other things like publicly_accessible). 

I don't really understand & couldn't find any history as to why boto added "rds2" instead of iterating on the existing rds interface:

https://github.com/boto/boto/tree/develop/boto/rds

https://github.com/boto/boto/tree/develop/boto/rds2

Cheers,

iain

--

Iain Wright

Cell: (562) 852-5916

This email message is confidential, intended only for the recipient(s) named above and may contain information that is privileged, exempt from disclosure under applicable law. If you are not the intended recipient, do not disclose or disseminate the message to anyone except the intended recipient. If you have received this message in error, or are not the named recipient(s), please immediately notify the sender by return email, and delete all copies of this message.

--
You received this message because you are subscribed to a topic in the Google Groups "Ansible Project" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ansible-project/jPcfjXH0cc0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to ansible-proje...@googlegroups.com.

To post to this group, send email to ansible...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/CA%2BnsWgwzaKnTxPaWEjD7N7ApQBmtfkpm1EeBSmyNFOpDt9Tdmw%40mail.gmail.com.

[Michael DeHaan]

This I don't know.

Maybe so other RDS using folks would have input.

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/CAMhkWdDJRMs78uS02TWUtRtuJKRXm%2BYr6fkTO6Ck7MKMPQ8V%3DQ%40mail.gmail.com.

[Will Thames]

Digging up an old thread (as I came across https://github.com/ansible/ansible/issues/7564 looking at tagging RDS instances).

It looks like the rds2 module is automatically generated so that it keeps up to date with upstream AWS capabilities, and this change broke backward compatibility.
http://boto.readthedocs.org/en/latest/migrations/rds_v1_to_v2.html

Do we have a minimum supported boto version? We could rewrite the RDS module to use rds2, but that might affect users on earlier boto versions. Supporting both seems like a headache. However, I suspect we could just use rds2 for the features that just aren't in rds (so make a boto.rds2.RDSConnection if we need to do instance tagging or publicly available).

I'm hoping I can use the ec2_tag module to work around my problem but it's worth discussing in more general terms - we can deprecate old Ansible conventions, but when do we deprecate old boto libraries?

Will

[Michael DeHaan]

"we can deprecate old Ansible conventions, but when do we deprecate old boto libraries?"

Generally each module is responsible for yelling if it tries to do something it cannot.

But boto is obviously not something we release, so we can't "deprecate it", we can only raise an error when the version is too old.

I am ok with forcing folks to update their boto from time to time, especially how AWS usually forces the issue on it's own occasionally by not working with older versions.

--Michael

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/b0fc5833-d03b-40db-b0fc-76fb78dc0976%40googlegroups.com.

[Dan Vaida]

Ah-ha!

This might very well be the cause for my issues: https://github.com/ansible/ansible/issues/7958 & https://github.com/ansible/ansible/issues/7952

[Will Thames]

I’m going to say rds vs rds2 is unrelated to your two issues, unfortunately.

I’ve updated 7958 - my refactoring* of the rds module won’t currently affect this behaviour (although it could if I understood the requirements better).

7952 is part of a different module (rds_params_group) that I currently have no plans to touch.

* I say refactoring, git diff thinks it’s a complete rewrite, even though I haven’t touched some lines at all (mostly the module declaration). I am reusing a lot of the existing logic.

Will

On 27 Aug 2014, at 19:33, Dan Vaida <vaid...@gmail.com> wrote:

Ah-ha!

This might very well be the cause for my issues: https://github.com/ansible/ansible/issues/7958& https://github.com/ansible/ansible/issues/7952

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/5b988e58-e04b-429d-ac03-3fff64b56812%40googlegroups.com.

[Dan Vaida]

Thanks for checking that.

You're totally write about the #7952; was thinking about something else there.

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/FC8944FC-3025-47E8-B0B6-4B1E1536739B%40thames.id.au.

[Will Thames]

I've rewritten rds to be able to use boto.rds and boto.rds2. The latter is only six months old, and insisting on it for all RDS users might be a little disruptive.

Removing boto.rds support would greatly simplify the results, but I'm happier with the new structure of the rds module than with the old structure. I could probably remove a lot of the additional classes that I'm using to unify the rds and rds2 code were we to insist on rds2 only.

My tests have been reasonably extensive (and I've greatly improved rds test coverage in general) but I'd welcome further tests.

https://github.com/ansible/ansible/pull/8834

Will