The new Python API is extremely complex.

[Mitchell Ludwig]

Ansible's tagline:
"Ansible is a radically simple IT orchestration engine that makes your applications and systems easier to deploy."

Radically. Simple.

Now, check out the new Python API in v2.0:
http://docs.ansible.com/ansible/developing_api.html

I have no idea how to just run a normal ad-hoc command. No clue. And I've been digging at it for hours now. The old API? With Runner? Radically simple. Love it. Flawless.

Why was Runner removed? Like, I'd re-develop it, and Pull Request it in, gladly, but why was it removed in the first place? It's so so so much simpler. It does what I need it to.

Further, could we have simple helper classes for all these random loading things? Like, something like:

inventory = Inventory()

Bam, now you've got your standard inventory, whatever you'd have if you didn't specify an inventory on the command line. Is that not viable? Or even:

import defaults from somewhere

inventory = defaults.inventory()

Something to make life easier for those of us who don't need to reinvent the whole wheel.

[Brian Coca]

The API is not the product, the Ansible command line tools are, the API is there for the tool usage and it not the main interface. This has always been the case and stated in many occasions and in the documentation.

Runner was not simple (no class with a thousand+ line __init__ method is) and was full of errors as many features were 'bolted in' and was sorely needing a redesign. It might have been 'simple to call' but it was not easy to maintain nor extend.

We removed runner and derived over a dozen classes that now handle the same functions in a much cleaner way internally. This also allowed us to correctly test the code and now validate plays, many previously silent errors are now caught. Adding new features is much simpler and clearly defined, as is the inheritance of such features, things are much more well defined and predictable. We also made many parts of the code more configurable and were able to push features to the plugins that were previously hardcoded.

As much as this has made it easier to maintain and extend Ansible, it has made the API interface more complex, but that, again, is something that we use internally for the command line tools, which have not really changed their interface and maintained their simplicity.

​Long story short, NO we do NOT want runner back. It is easy enough to create other helper classes to simplify the interface for 3rd party, but it is not something the current project focuses on, our focus is the set of CLI tools which our main user base depends on.​

----------

Brian Coca

[Stephen Granger]

Any chance there will be documentation for the api similar to Ansible modules? (I'm looking for an AWS Lambda example using the 2.0 api after seeing Jose article)

I like the way things have been broken apart with Ansible 2.0 into smaller contained components, especially with how some of the modules have been rewritten.. It was slightly painful to begin with but it certainly does allow for flexibility and makes it easier for more complicated use cases.

Admittedly I haven't figured out how to run a playbook via the new api yet either.

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
To post to this group, send email to ansible...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/CACVha7e3%2BLLcJpSq8%3DpzGMqKNfFtetVjcpiN6FcHQ9VTavUzdg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Steve

[Brian Coca]

It is 'on the list' to document it, but we also have many other things on that list that are much higher priority: documenting other plugins, migration path to 2.0, making plugins work under 1.9 AND 2.0, document testing, and in general bugfixes and features galore ....

----------

Brian Coca

[Mike Biancaniello]

I have found that if you want to wrap the running of a playbook inside of a python script (so you can make the interface a bit more user friendly and translate cmdline args to --extra-vars, etc), the 2.0 API is much simpler.

from ansible.cli.playbook import PlaybookCLI

cliargs = {
    'playbook':pb,
    'args': {
        '-i':host_list,
        '-e':json.dumps(xvars),
    }
}

def _run_playbook_v20(cliargs):
    ansargs=['ansible-playbook',cliargs['playbook']]
    [ansargs.extend([k,v]) for k,v in cliargs['args'].items()]
    print "executing: {}".format(' '.join(ansargs))
    cli = PlaybookCLI(ansargs)
    cli.parse()
    return cli.run()

[Greg DeKoenigsberg]

On Wed, Feb 24, 2016 at 10:34 AM, Stephen Granger <vipe...@gmail.com> wrote:
> Any chance there will be documentation for the api similar to Ansible
> modules? (I'm looking for an AWS Lambda example using the 2.0 api after
> seeing Jose article)
>
> I like the way things have been broken apart with Ansible 2.0 into smaller
> contained components, especially with how some of the modules have been
> rewritten.. It was slightly painful to begin with but it certainly does
> allow for flexibility and makes it easier for more complicated use cases.
>
> Admittedly I haven't figured out how to run a playbook via the new api yet
> either.

It is my hope that the 2.0 API will be stable for a good long time,
and documenting it would be a great and useful project for someone
that isn't the core team.

--g

> https://groups.google.com/d/msgid/ansible-project/CA%2BemtqsScdVQ6kZd8UaX74dk2uNzSFygoik%3DTLnSO_Z5UoZVFQ%40mail.gmail.com.

>
> For more options, visit https://groups.google.com/d/optout.



--

Greg DeKoenigsberg
Ansible Community Guy

[Brian Coca]

Keep in mind that the API is in service of the CLI tools and that we still might change it as needed, also parts of it are still scheduled for a revamp, like inventory, roles and some other subsystems that we did not have time to finish up for 2.0.

----------

Brian Coca

[Dag Wieers]

I noticed the "making plugins work under 1.9 AND 2.0" and I wrote my first
hybrid lookup plugin today. I consider this an odd thing to have and
didn't consider it useful to bring up.

Would it be useful to share what I did ?
It may be a starting point to improve the mechanism and document it in the
process.

--
Dag

[Brian Coca]

Dag, yes please! We've been meaning to write that but have never had the time.

--

----------

Brian Coca

[Dag Wieers]

On Wed, 24 Feb 2016, Brian Coca wrote:

> Dag, yes please! We've been meaning to write that but have never had the
> time.

Not sure how to proceed next, but let me quickly show what I did for the
filetree plugin. Since we have a need to run the existing playbooks with
v1.9.4, but also test new stuff with v2.0.0.2 we have a need for a hybrid
plugin.

The original v2.0 filetree plugin is available from:

https://github.com/ansible/ansible/pull/14332

And the v1.9 filetree plugin is available from:

https://github.com/ansible/ansible/pull/14628


The resulting hybrid lookup plugin required the following changes:

- We need an implementation of LookupBase class on v1, with a custom
v1 __init__()

- In v1 we get the basedir from runner through __init__

- Since we use the warning infrastructure, we need to have it imported
from the right location (different from v1 and v2)

- v1 also needs to import specialized functions, and we use the
existence of these fuctions in globals() as the condition to see
whether we are doing v1 or v2

- We modified v1 LookupBase __init__() so it behaves a bit as v2
LookupBase for our purpose, so the calls would be identical (we could
have done something similar for other v1 functions, but prefered to
stick as closely to v2 as possible, and consider v1 the exception as
much as possible.

Below is the hybrid filetree plugin. Feedback is much appreciated:

--------
# (c) 2016 Dag Wieers <d...@wieers.com>
#
# This file is part of Ansible
#
# Ansible is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# Ansible is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with Ansible. If not, see <http://www.gnu.org/licenses/>.
from __future__ import (absolute_import, division, print_function)
__metaclass__ = type

import os
import glob
import pwd
import grp

from ansible.errors import AnsibleError

HAVE_SELINUX=False
try:
import selinux
HAVE_SELINUX=True
except ImportError:
pass

try:
# Ansible v2
from ansible.plugins.lookup import LookupBase
except ImportError:
# Ansible v1
class LookupBase(object):
def __init__(self, basedir=None, runner=None, **kwargs):
if runner:
self.runner = runner
self.basedir = self.runner.basedir
elif basedir:
self.basedir = basedir

def get_basedir(self, variables):
return self.basedir

try:
# Ansible v1
from ansible.utils import (listify_lookup_plugin_terms, path_dwim, warning)
except ImportError:
# Ansible v2
from __main__ import display
warning = display.warning

def _to_filesystem_str(path):
'''Returns filesystem path as a str, if it wasn't already.


Used in selinux interactions because it cannot accept unicode
instances, and specifying complex args in a playbook leaves
you with unicode instances. This method currently assumes
that your filesystem encoding is UTF-8.

'''
if isinstance(path, unicode):
path = path.encode("utf-8")
return path

# If selinux fails to find a default, return an array of None
def selinux_context(path):
context = [None, None, None, None]
if HAVE_SELINUX and selinux.is_selinux_enabled():
try:
ret = selinux.lgetfilecon_raw(_to_filesystem_str(path))
except OSError:
return context
if ret[0] != -1:
# Limit split to 4 because the selevel, the last in the list,
# may contain ':' characters
context = ret[1].split(':', 3)
return context

def file_props(root, path):
''' Returns dictionary with file properties, or return None on failure '''
abspath = os.path.join(root, path)
ret = dict(root=root, path=path, exists=True)

try:
if os.path.islink(abspath):
ret['state'] = 'link'
ret['src'] = os.readlink(abspath)
elif os.path.isdir(abspath):
ret['state'] = 'directory'
elif os.path.isfile(abspath):
ret['state'] = 'file'
ret['src'] = abspath
except OSError, e:
warning('filetree: Error querying path %s (%s)' % (abspath, e))
return None

try:
stat = os.stat(abspath)
except OSError, e:
warning('filetree: Error using stat() on path %s (%s)' % (abspath, e))
return None

ret['uid'] = stat.st_uid
ret['gid'] = stat.st_gid
ret['owner'] = pwd.getpwuid(stat.st_uid).pw_name
ret['group'] = grp.getgrgid(stat.st_gid).gr_name
ret['mode'] = str(oct(stat.st_mode))
ret['size'] = stat.st_size
ret['mtime'] = stat.st_mtime
ret['ctime'] = stat.st_ctime

if HAVE_SELINUX and selinux.is_selinux_enabled() == 1:
context = selinux_context(abspath)
ret['seuser'] = context[0]
ret['serole'] = context[1]
ret['setype'] = context[2]
ret['selevel'] = context[3]

return ret

def run(self, terms, inject=None, variables=None, **kwargs):

basedir = self.get_basedir(variables)

# Ansible v1
if 'listify_lookup_plugin_terms' in globals():
terms = listify_lookup_plugin_terms(terms, basedir, inject)

ret = []
for term in terms:
term_file = os.path.basename(term)
if 'path_dwim' in globals():
# Ansible v1
dwimmed_path = path_dwim(basedir, os.path.dirname(term))
else:
# Ansible v2
dwimmed_path = self._loader.path_dwim_relative(basedir, 'files', os.path.dirname(term))
path = os.path.join(dwimmed_path, term_file)
for root, dirs, files in os.walk(path, topdown=True):
for entry in dirs + files:
relpath = os.path.relpath(os.path.join(root, entry), path)
props = file_props(path, relpath)
if props != None:
ret.append(props)

return ret
--------

--
Dag

[Mike Biancaniello]

I have done similar things to enable my plugins for v1 AND v2. Essentially:

from ansible import __version__ as ANSIBLE_VERSION
    if ANSIBLE_VERSION.startswith('2'):
    else:

More recently, I am encouraging everyone to upgrade to v2 and trying to remove the v1 support esp as some things that I'm doing now simply aren't possible in v1.

My concern, however, is that all of this is going to break with v3 or maybe even v2.2, resulting in some plugins requiring more code to deal with the version than actually do the thing it's actually trying to do.

[Dag Wieers]

I wrote a small section about hybrid plugins with some advice.
Feel free to add your thoughts and experiences to it.

Here is the PR:

https://github.com/ansible/ansible/pull/14660

--
Dag

[Mike Biancaniello]

interesting. My hybrid (custom) lookup_plugin:

    def run(self, terms='', **kwargs):
        '''Normalize terms and args and perform lookup.

        Args:
          terms (Optional[str,list]): Comma-delimited string (or python list) of terms.
          kwargs (**dict): See ``Store()`` object for details. All kwargs are sent
            when creating instance.

        Caveats:
           Ansible 2.0 sends all args as list::

             "{{ lookup('myplug', ['key1', 'key2']) }}" ==> [['key1', 'key2']]
             "{{ lookup('myplug', 'key1', 'key2') }}" ==> ['key1', 'key2']

           Ansible 1.9 sends first arg as list::

             "{{ lookup('myplug', ['key1', 'key2']) }}" ==> ['key1', 'key2']
             "{{ lookup('myplug', 'key1', 'key2') }}" ==> ERROR! Too many args.

        '''
        ## Normalize terms to be a list
        if not isinstance(terms, list):
            terms = terms.split()
        ## Ansible 2.0 sends all args as list
        ## Ansible 1.9 sends first arg as list
        if len(terms) == 1 and isinstance(terms[0], list):
            terms = terms[0]
        ...etc...

Now, I want to switch to use listify_lookup_plugin_terms!