Ansible for academic distributed experiment

[Matthieu Simonin]

Dear All,

I'd like to share what we are doing using Ansible in an academic context
and maybe bootstrap some discussions about our specific implementation and
needs.

---

TLDR;
In our context, distributed system experimenters aren't expert in system
provisionning and with Ansible we are able to give them nice abstractions
with a good degree of expressivity and safety (e.g idempotency). In this
direction, we built EnOSlib https://discovery.gitlabpages.inria.fr/enoslib/
to provide a smooth integration of Ansible with Python.

Some initial questions/discussions:
- We use "in-memory" inventory, but we deem our implementation fragile
  Is their a canonical way to populate an Inventory datastructure from scratch ?
  (see our implementation[1])

- In some specific use case we are pushing our deployments to the scale of
  thousands of machines and we are seeing the Ansible SSH transport suffer.
  What would be a good way to scale ? (We already identified few things [5])

---

Some details on the Ansible integration which intensively uses the Python
API so that experimenter can stick with their python code as much as
possible.

- We target different infrastructure but don't rely on dynamic inventory.
  We rather provide a way to generate an inventory from python. Once the
  infrastructure resources are ready the inventory is either:
  + dumped on the file system (and reloaded when needed)
  + or generated on the fly when needed (in memory)[1]. In my opinion the
    corresponding code is fragile as it subclasses the Inventory
    datastructure and populate it manually.

- We provide a `run_ansible`[2] function to run any Ansible playbook.

- We provide a Python context manager[3] to allow programmer to call Ansible Modules
  directly from Python. This is considered as a very convenient way to script
  remote actions on nodes from Python.

- Built on this two, we provide built-in services to deploy common tool used
  when experimenting with distributed systems (e.g [4])

[1] https://gitlab.inria.fr/discovery/enoslib/blob/master/enoslib/enos_inventory.py
[2] https://discovery.gitlabpages.inria.fr/enoslib/apidoc/api.html#enoslib.api.run_ansible
[3] https://discovery.gitlabpages.inria.fr/enoslib/apidoc/api.html#enoslib.api.play_on
[4] https://discovery.gitlabpages.inria.fr/enoslib/apidoc/service.html#monitoring
[5] https://discovery.gitlabpages.inria.fr/enoslib/performance_tuning.html