No longer able to apply commands to AWS instances after upgrade to 2.5

[Ilsa Loving]

Hi all,

I have a bunch of scripts I inherited that do a variety of things, primarily for setting up new environments in AWS.

Using v2.4, everything worked just fine.  As of v2.5, modules are applied to the currently running host instead of the target instance.  Needless to say this caused a really big confusing mess until we figured out what happened and forced everything to stay at v2.4.

For example, we have a playbook of:

- hosts: localhost

gather_facts: no

roles:

- { role: create_web_tier, tags: 'web' }

The role includes a 'create_ec2_instance' task which does the following:

- name: "create_ec2_instance | AWS EC2 | Create server"

local_action:

module: ec2

region: "{{ aws_region }}"

key_name: "{{ ec2_key_name }}"

image: "{{ ec2_image }}"

instance_type: "{{ ec2_instance_type }}"

vpc_subnet_id: "{{ ec2_vpc_subnet_id }}"

assign_public_ip: "{{ ec2_assign_public_ip }}"

group_id: "{{ ec2_group_id }}"

count: "{{ ec2_count }}"

instance_tags: "{{ ec2_instance_tags}}"

wait: yes

register: ec2

- debug: var=ec2 verbosity=4

- name: create_ec2_instance | Ansible | Add instance to host group

add_host:

hostname: "ec2_instance_host"

ansible_ssh_host: "{{ item.private_ip }}"

# ansible_ssh_user: "{{ global_ansible_ssh_user }}"

# ansible_ssh_private_key_file: "{{ global_ansible_ssh_private_key_file }}"

with_items: "{{ ec2.instances }}"

This should theoretically add the instance to inventory so that when we perform the following task later:

# Perform server default tasks

- include_tasks: set_server_defaults.yml

delegate_to: "ec2_instance_host"

become: true

The task applies a number of changes such as yum updates, etc.

But as soon as ansible is updated to 2.5+, this behaviour breaks and instead applies all those server settings to the local host running the playbook.

So far our only solution has been to block updates and to keep Ansible at 2.4, which is far from ideal.

Does anyone have any insight as to why Ansible's behaviour would change so fundamentally?  This is a catastrophic disruption that has seriously shaken our confidence in Ansible.

[Kai Stian Olstad]

On Thursday, 16 August 2018 22.36.17 CEST Ilsa Loving wrote:
> This should theoretically add the instance to inventory so that when we
> perform the following task later:
> # Perform server default tasks
> - include_tasks: set_server_defaults.yml
> delegate_to: "ec2_instance_host"
> become: true
>
>
> The task applies a number of changes such as yum updates, etc.
>
> But as soon as ansible is updated to 2.5+, this behaviour breaks and
> instead applies all those server settings to the local host running the
> playbook.
>
> So far our only solution has been to block updates and to keep Ansible at
> 2.4, which is far from ideal.
>
> Does anyone have any insight as to why Ansible's behaviour would change so
> fundamentally? This is a catastrophic disruption that has seriously shaken
> our confidence in Ansible.

Before upgrading it is crucial to read the porting guide
https://docs.ansible.com/ansible/2.5/porting_guides/porting_guide_2.5.html

for you, especial this
https://docs.ansible.com/ansible/2.5/porting_guides/porting_guide_2.5.html#dynamic-includes-and-attribute-inheritance

The short answer (for the long one read the links) the delegate_to only applies to the include_tasks, not the tasks inside the include.


--
Kai Stian Olstad

[Ilsa Loving]

I'll skip the story since it's long, convoluted, and frustrating, but it boils down to the fact that the 'upgrade' was unintentional and cost us three weeks of head scratching.

Thanks for the info.  Now that I know how capricious Ansible is, we will need to reconsider how it is used, and how heavily.  This kind of fundamental instability make Ansible a very high risk product to use.

Thanks!

[Karl Auer]

In three weeks of head-scratching you didn't realise the version of

A

nsible had changed, meaning you allow uncontrolled upgrades to your production systems?

Or in three weeks of head-scratching, knowing that Ansible had upgraded, it didn't occur to you to read the release notes?

It's a bit rough to blame Ansible for those...

Regards, K.

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-project+unsubscribe@googlegroups.com.
To post to this group, send email to ansible-project@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/804d6159-b9bb-4283-a848-a6a30a82eb72%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Karl Auer

Email  : ka...@2pisoftware.com
Website: http://2pisoftware.com

GPG/PGP : 958A 2647 6C44 D376 3D63 86A5 FFB2 20BC 0257 5816
Previous: F0AB 6C70 A49D 1927 6E05 81E7 AD95 268F 2AB6 40EA

[Jonathan Lozada De La Matta]

your issue was in  " but it boils down to the fact that the 'upgrade' was unintentional and cost us three weeks of head scratching.". Ansible docs, changelogs and release announcements talk about all the changes that happened. If its something that happened in your environment then its not really ansible's fault.

To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
To post to this group, send email to ansible...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/804d6159-b9bb-4283-a848-a6a30a82eb72%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Karl Auer

Email  : ka...@2pisoftware.com
Website: http://2pisoftware.com

GPG/PGP : 958A 2647 6C44 D376 3D63 86A5 FFB2 20BC 0257 5816
Previous: F0AB 6C70 A49D 1927 6E05 81E7 AD95 268F 2AB6 40EA

--

You received this message because you are subscribed to the Google Groups "Ansible Project" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
To post to this group, send email to ansible...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/CA%2B%2BT08R4vDm%3Dr5AGUUNox2yC-aPzO%3DAhfK_vP-orLHzaLVoGpg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Jonathan lozada de la matta

AUTOMATION CONSULTANT - AUTOMATION PRACTICE

Red Hat Consulting Services

jloz...@redhat.com   

 

[Ilsa Loving]

The issue was that the person that originally wrote the scripts quit, and we were parachuted in as emergency resources to try to get everything going.  And Ansible just happened to put out an update to 2.5 in the same time frame, so when we did 'yum install ansible', we had no way of knowing that the version we were installing was different from the one he was using.

But you're right, it was completely our fault.  We should have known (apparently by osmosis) that Ansible changes it's APIs capriciously with every single version, that backward compatibility is for losers, and that all Ansible packages are treated as minor in-place updates so if you haven't locked down the version ahead of time, a routine yum update will break your entire setup.

I mean, heaven forbid that someone might expect a critical operations tool like Ansible to have a modicum of stability between minor releases.

[Jonathan Lozada De La Matta]

I think you misunderstood our comment and taking it a little far. They are going to be changes to software, all of them makes changes. But, like mentioned before we have docs and notices for this exact reason. 

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/35b35e98-d3eb-4c61-94f4-4273557b91cb%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Ilsa Loving]

And yes, I know that.  Now.

My point is that that's not good enough.  It's not acceptable to make wide-ranging fundamental changes to the system, and then say "It's in the docs.  You need to read the docs."

If Ansible is going to make changes so significant that it is guaranteed to break existing playbooks, then it *cannot* follow the traditional upgrade path where you simply update the version number of the rpm/deb/whatever, and toss it into the pile for upgrading.  Each release needs it's own versioning as part of the name (ie: have ansible2.5, ansible2.6, etc.) so that people can be assured that when they do a yum update, then their stuff will still work afterwards.  Apache did this when they went from 2.2 to 2.4.  People had plenty of issues during the transition, but surprise upgrades that broke their infrastructure wasn't one of them.

I've been doing a lot of reading, and have come across a very large number of people who share the same frustrations. Ansible is supposed to make sysadmin lives easier.  Instead, people have had to resort to version-locking Ansible (as we have done), or implementing entire continuous delivery paths just for Ansible.  Or they've given up on it entirely and reverted back to bash scripts, cause bash scripts don't break every 3-6 months.

IMO that is absolutely absurd.  Infrastructure is not software.  It's infrastructure.  It has to be solid, because everything else depends on it.  If Ansible is to be a key element in managing infrastructure, it too has to be solid.  Why should I use something that adds to my workload instead of reduces it?

Ilsa

[Jonathan Lozada De La Matta]

 You are welcome to join the meetings, proposals and submit issues for changes.

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/54e706d0-fd39-438b-87f4-cc8ac0c943d0%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Stephen Granger]

It's almost fundamental that you treat all of your deployment and systems tools in the same way, they are flaky and will introduce drastic changes that will almost always cause system outages on upgrade (similar to public cloud infrastructure, it fails a LOT). Every tool, (ansible role) playbook and inventory, api reference, software package EVERYTHING needs to be versioned. Blind updates are going to wipe you out and if other people (business revenue too :) ) rely on the system it's even more critical to lock everything down to a version.

You will experience this time and time again with various tools, OS's, a lot of Ansible users already have and this is why they are using Ansible in the first place to enforce Ansible version. 

https://www.jeffgeerling.com/blog/2016/require-minimum-ansible-version-your-playbook

This is along the line of what you can implement to ensure Ansible is at a specific version. You can use this outage as an excuse to implement this check across all of your playbooks. This is a hard lesson to learn and a lot of other infrastructure tools take the same approach enforcing a version, especially those still considered beta i.e. terraform.

Good luck!

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/54e706d0-fd39-438b-87f4-cc8ac0c943d0%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Steve

[yaakov....@sparkpost.com]

Part of the problem is that important new functionality and modules come out, but aren't backported, so if you want to be able to do everything you need to do, you either need to do extra work to backport it yourself (when possible), or deal with the upgrades. It'd be nice if non-breaking updates were brought to stable long-term releases by default.

I don't think general flakiness is a good excuse for the instability - ansible could be the one that *doesn't* cause problems in that case.  It'd be a sign of the tool's maturity.

I don't mind submitting a proposal and advocating for it, where do I sign?