Good practices Ansible + GitHub + docker-compose

[Juan Luis Cano Rodríguez]

hello folks, I'm looking for good practices on how to deploy a docker-compose from GitHub to Ansible.

at the moment I have a repository https://github.com/Mercado-Social-de-Madrid/ansible-takahe that contains the playbook and some instructions (install Docker, generate SSL certificate). in another repository https://github.com/AlbertoMoreta/takahe-docker there's a docker-compose.yml I want to deploy to my target machines.

I am about to create a systemd .service file that will take care of running docker compose up in the appropriate directory. but before that, I need to upload that compose file to the target servers. there are naturally several ways of doing this but I'm looking for best practices. should I merge both repositories (or add one as a submodule of the other)? should I command a `git clone` inside the target machine? should I `scp` the contents? what do I do when the compose definition changes?

any references to public repos or guides that do similar things are more than welcome.

--
Sent with HEY

[Pierre TOURON]

Hi,

Ultimately it depends on what process you envision; you could, as you suggest, create a systemd service piloting your app and components deployment / teardown on target machine, or you could deploy them directly from ansible, running your playbook from a CI pipeline, cron job or anything really. As for fetching sources, it depends on the location you want to deploy from, or if you need to have some of your repo files available on target machine (though you should probably deploy those files from an ansible tasks instead IMO).
I won't list a bunch of options here, but I can give you my input on how I'd do things (don't know if it could be considered as best practices though).

Regarding your repos structure, I'd do it a bit differently; I'd suggest splitting files in separate repositories :
    - Each role in it's own, with the lint part as pre-commit / pre-receive hook
    - 'Deployment' files (requirements.yml, playbook calling your roles, inventory and variables files, and perhaps CI pipeline files if that's how you'd like to trigger your deployments) in a seperate one

The idea is to standardize your projects structures and allow from easy roles components reuse accross eventual multiple projects. Here is a structure exemple from a project I manage :
    - 'gbt_psg' role repo :
    /home/ptouron/Infra_GIT/ansible-roles/gbt_psg/
    ├── .ansible-lint
    ├── .gitignore
    ├── .pre-commit-config.yaml
    ├── .yamllint
    ├── README.md
    ├── defaults
    │   └── main.yml
    ├── files
    │   ├── cacerts.vault
    │   └── dockerFindNextAvailableSubnet.sh
    ├── meta
    │   └── main.yml
    ├── molecule
    │   └── docker
    │       ├── converge.yml
    │       ├── molecule.yml
    │       ├── prepare.yml
    │       └── requirements.yml
    ├── tasks
    │   ├── backend_conf_lint.yml
    │   ├── build_frontend-es_image.yml
    │   ├── copy_backend_cacerts.yml
    │   ├── copy_keycloak_realm_exports.yml
    │   ├── create_psg_docker_containers.yml
    │   ├── create_psg_docker_networks.yml
    │   ├── create_psg_docker_volumes.yml
    │   └── main.yml
    ├── templates
    │   ├── Dockerfile_es.j2
    │   ├── backend_config.json.j2
    │   └── realm-export.json.j2
    └── vars
        ├── main.yml
        └── secrets.vault

    - 'psg-deploy' repo :
    /home/ptouron/TEMP/gitlab/psg-deploy/
    ├── .gitignore
    ├── .gitlab-ci.yml
    ├── README.md
    ├── ansible.cfg
    ├── inventories
    │   ├── host_vars
    │   │   ├── <production-srv-hostname>.yml # Masked to avoid information leak
    │   │   └── <staging-srv-hostname>.yml # Masked to avoid information leak
    │   ├── hosts_psg_production
    │   └── hosts_psg_staging
    ├── main.yml
    └── requirements.yml

    $ less requirements.yml
    ---

    roles:
      - src: <repo-clone-url> # Masked to avoid information leak
        scm: git
        name: gbt_psg
        version: main

    collections:
      - name: community.docker
        version: 3.4.8

    $ less ~/Devops_GIT/devops-conf/roles/gbt_psg/meta/main.yml # gbt_psg meta role file excerpt
    ...
    dependencies:
      - role: gbt_docker # Role to provision machines as docker hosts. Here deployed as a dependency of gbt_psg role, but I also could have just added it to main requirements.yml and file, and called from playbook
        src: <repo-clone-url> # Masked to avoid information leak
        scm: git
        version: main
        when: "('psg' in run_in_docker|d()|lower) and (inventory_hostname|lower is not match 'test_*')"

    $ less main.yml
    ---

    - name: Apply PSG app configuration
      gather_facts: false
      hosts: psg
      become: "{{ 'no' if (root_access|d()|bool) else 'yes' }}"
      roles:
        - role: gbt_psg

This way I can reuse both roles across multiple similar projects, and roles development is not part of deploy configuration.

Just a quick note about the docker objects deployment; for this project, I used community.docker collection's docker_(network|volume|container) modules. There is also a docker_compose one you could use to manage your compose files if you so wish, though it currently doesn't support compose v2. Some awesome people are working on it :) : https://github.com/ansible-collections/community.docker/pull/586.

In your case, I'd probably deploy this way, from role tasks (which you call from a playbook). Alternatively, you could just template out the compose file from template module on target host, then maybe create systemd service (also from ansible task) you wanted to use to pilot deployments locally. Perhaps you'd also like to make a role for the environment (certbot, ansible, nginx) setup / configure part, or adding seperate task(s) to your existing webdb role.

One last thing, ansible install from pip might require you to use a python venv on some more recent distributions; you mentionned support for Ubuntu 22.04 which doesn't AFAIK, though I'm pretty sure Ubuntu 23.04 does. Just in case you'd like to mention it.

Anyways, I hope it somehow responds to what you asked, or at least might give you some insight.
Have a nice Sunday !