ansible task to cat a file with sql commands and pipe the commands into database using psql with synchronise module (remotely)

[jonin]

Hi,

I have created the following task to be run on a remote host:

- hosts: nodes

  remote_user: root

  tasks:

    - name: Step Seven - remove all current postcode data and then re-populate the database with most recent data

      become_user: postgres

      become: yes

      command: "cat /var/lib/pgsql/temp/truncate_sql  /var/lib/pgsql/temp/postcodesio-* | psql postcodesiodb"

but it is failing with the below memory issue. A google makes me suspect it is because I am trying to make ansible handle too large a file (a postcode database)  which it is loading into RAM. : https://stackoverflow.com/questions/41367278/how-to-fix-memory-error-in-ansible/51320108. The suggestion is to use the synchronise module or rsync directly as a command. For either I am struggling to visualise the syntax / structure to do this.  Any help much appreciated. 

TASK [Step Seven - remove all current postcode data and then re-populate the database with most recent data] *****************************************************************

fatal: [vm015018.bskyb.com]: FAILED! => {"changed": false, "module_stderr": "Shared connection to vm015018.bskyb.com closed.\r\n", "module_stdout": "Traceback (most recent call last):\r\n  File \"/tmp/ansible_ar0FKd/ansible_module_command.py\", line 213, in <module>\r\n    main()\r\n  File \"/tmp/ansible_ar0FKd/ansible_module_command.py\", line 207, in main\r\n    module.fail_json(msg='non-zero return code', **result)\r\n  File \"/tmp/ansible_ar0FKd/ansible_modlib.zip/ansible/module_utils/basic.py\", line 2294, in fail_json\r\n  File \"/tmp/ansible_ar0FKd/ansible_modlib.zip/ansible/module_utils/basic.py\", line 2273, in _return_formatted\r\n  File \"/tmp/ansible_ar0FKd/ansible_modlib.zip/ansible/module_utils/basic.py\", line 2227, in jsonify\r\n  File \"/tmp/ansible_ar0FKd/ansible_modlib.zip/ansible/module_utils/basic.py\", line 782, in jsonify\r\n  File \"/usr/lib64/python2.7/json/__init__.py\", line 250, in dumps\r\n    sort_keys=sort_keys, **kw).encode(obj)\r\n  File \"/usr/lib64/python2.7/json/encoder.py\", line 207, in encode\r\n    chunks = self.iterencode(o, _one_shot=True)\r\n  File \"/usr/lib64/python2.7/json/encoder.py\", line 270, in iterencode\r\n    return _iterencode(o, 0)\r\nMemoryError\r\n", "msg": "MODULE FAILURE", "rc": 1}

        to retry, use: --limit @/home/sco94/mwe-playbooks/playbooks/patching/postcodes.retry

[Toshio Kuratomi]

What happens if you use shell instead of command?

I'm not 100% certain but I think what's happening is that since you're
using the ansible command module instead of the shell module, it is
running the equivalent of:
cat '/var/lib/pgsql/temp/truncate_sql'
'/var/lib/pgsql/temp/postcodesio-*' '|' 'psql' 'postcodesiodb'

That's sending the entire output of the first couple files to stdout
which the command module is then buffering to send back to the
controller.

If you use the shell module then the pipe will actually be interpreted
by the shell and so it will do what you want. (If I'm wrong about why
the MemoryError is happening, there might be another problem in
addition but I am hopeful that it's all tied together).

-Toshio

> --
> You received this message because you are subscribed to the Google Groups "Ansible Development" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to ansible-deve...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-devel/6ff52565-cbdd-4dca-96ee-85382530d7b5%40googlegroups.com.

[jonin]

Thank you Toshio.

The use of shell has worked perfectly. Much appreciated.

> To unsubscribe from this group and stop receiving emails from it, send an email to ansibl...@googlegroups.com.