Memory error when downloading 3.4GB file

[Sindhuja Koneru]

Hello,

Has anyone come across this Memory error? It occurs when running backup.yml script which is to backup DB and SECRET KEY.

Script is completely successful, until it reaches to a point when it needs to download a common.tar.gz file which is 3.4GB --Fails with an unpredictable memory error.

Any suggestions or help please?

./setup.sh -b -i inventoryfile -e backup_dir=/tower/bkpsetup

TASK [backup : Download the common backup.] ************************************************************************************************************************

task path: /TowerSetup/tower_binaries/ansible-tower-setup-bundle-3.4.2-1.el7/roles/backup/tasks/download.yml:31

<abc.corp..com> ESTABLISH SSH CONNECTION FOR USER: root

<abc.corp..com> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o 'IdentityFile="/export/home/zsjkvh0/.ssh/id_rsaDev"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/e37aeee5bc abc.corp..com '/bin/sh -c '"'"'echo ~root && sleep 0'"'"''

<abc.corp..com> (0, '/root\n', '')

<abc.corp..com> ESTABLISH SSH CONNECTION FOR USER: root

<abc.corp..com> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o 'IdentityFile="/export/home/zsjkvh0/.ssh/id_rsaDev"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/e37aeee5bc abc.corp..com '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /root/.ansible/tmp/ansible-tmp-1553783711.94-90505061625143 `" && echo ansible-tmp-1553783711.94-90505061625143="` echo /root/.ansible/tmp/ansible-tmp-1553783711.94-90505061625143 `" ) && sleep 0'"'"''

skipping: [abc2.corp..com] => {

    "changed": false,

    "skip_reason": "Conditional result was False"

}

skipping: [abc3.corp..com] => {

    "changed": false,

    "skip_reason": "Conditional result was False"

}

<abc.corp..com> (0, 'ansible-tmp-1553783711.94-90505061625143=/root/.ansible/tmp/ansible-tmp-1553783711.94-90505061625143\n', '')

Using module file /usr/lib/python2.7/site-packages/ansible/modules/net_tools/basics/slurp.py

<abc.corp..com> PUT /root/.ansible/tmp/ansible-local-19291WC3_1C/tmprxDde7 TO /root/.ansible/tmp/ansible-tmp-1553783711.94-90505061625143/slurp.py

<abc.corp..com> SSH: EXEC sftp -b - -C -o ControlMaster=auto -o ControlPersist=60s -o 'IdentityFile="/export/home/zsjkvh0/.ssh/id_rsaDev"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/e37aeee5bc '[abc.corp..com]'

<abc.corp..com> (0, 'sftp> put /root/.ansible/tmp/ansible-local-19291WC3_1C/tmprxDde7 /root/.ansible/tmp/ansible-tmp-1553783711.94-90505061625143/slurp.py\n', '')

<abc.corp..com> ESTABLISH SSH CONNECTION FOR USER: root

<abc.corp..com> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o 'IdentityFile="/export/home/zsjkvh0/.ssh/id_rsaDev"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/e37aeee5bc abc.corp..com '/bin/sh -c '"'"'chmod u+x /root/.ansible/tmp/ansible-tmp-1553783711.94-90505061625143/ /root/.ansible/tmp/ansible-tmp-1553783711.94-90505061625143/slurp.py && sleep 0'"'"''

<abc.corp..com> (0, '', '')

<abc.corp..com> ESTABLISH SSH CONNECTION FOR USER: root

<abc.corp..com> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o 'IdentityFile="/export/home/zsjkvh0/.ssh/id_rsaDev"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/e37aeee5bc -tt abc.corp..com '/bin/sh -c '"'"'/usr/bin/python /root/.ansible/tmp/ansible-tmp-1553783711.94-90505061625143/slurp.py && sleep 0'"'"''

<abc.corp..com> ESTABLISH SSH CONNECTION FOR USER: root

<abc.corp..com> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o 'IdentityFile="/export/home/zsjkvh0/.ssh/id_rsaDev"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/e37aeee5bc abc.corp..com '/bin/sh -c '"'"'rm -f -r /root/.ansible/tmp/ansible-tmp-1553783711.94-90505061625143/ > /dev/null 2>&1 && sleep 0'"'"''

<abc.corp..com> (0, '', '')

The full traceback is:

Traceback (most recent call last):

  File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 138, in run

    res = self._execute()

  File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 576, in _execute

    result = self._handler.run(task_vars=variables)

  File "/usr/lib/python2.7/site-packages/ansible/plugins/action/fetch.py", line 95, in run

    slurpres = self._execute_module(module_name='slurp', module_args=dict(src=source), task_vars=task_vars)

  File "/usr/lib/python2.7/site-packages/ansible/plugins/action/__init__.py", line 831, in _execute_module

    res = self._low_level_execute_command(cmd, sudoable=sudoable, in_data=in_data)

  File "/usr/lib/python2.7/site-packages/ansible/plugins/action/__init__.py", line 938, in _low_level_execute_command

    rc, stdout, stderr = self._connection.exec_command(cmd, in_data=in_data, sudoable=sudoable)

  File "/usr/lib/python2.7/site-packages/ansible/plugins/connection/ssh.py", line 1007, in exec_command

    (returncode, stdout, stderr) = self._run(cmd, in_data, sudoable=sudoable)

  File "/usr/lib/python2.7/site-packages/ansible/plugins/connection/ssh.py", line 276, in wrapped

    display.vvv(return_tuple, host=self.host)

  File "/usr/lib/python2.7/site-packages/ansible/utils/display.py", line 183, in vvv

    return self.verbose(msg, host=host, caplevel=2)

  File "/usr/lib/python2.7/site-packages/ansible/utils/display.py", line 206, in verbose

    self.display("<%s> %s" % (host, msg), color=C.COLOR_VERBOSE, screen_only=True)

MemoryError

fatal: [abc.corp..com]: FAILED! => {

    "msg": "Unexpected failure during module execution.",

    "stdout": ""

}

[Stephen John Smoogen]

What architectures are you using? 3.4 gb is a common failure size for 32 bit processors

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
To post to this group, send email to ansible...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/64035943-9d99-4677-b49e-1703dd003b1e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Sindhuja Koneru]

It's Linux system with 64 bit (x86_64)

Running on 2 cpu x 8 memory, there are no other processes running except ansible. Im running the setup.sh as root user.

I can see enough disk space and available memory, no clue why it throws memory error?

Is it trying to load the entire file to some cache and fails?

To unsubscribe from this group and stop receiving emails from it, send an email to ansible...@googlegroups.com.

[Sebastian Meyer]

Hi,

On 29.03.19 00:55, Sindhuja Koneru wrote:
> '"'"'/usr/bin/python
> /root/.ansible/tmp/ansible-tmp-1553783711.94-90505061625143/slurp.py &&
> sleep 0'"'"''

Any reason you are using slurp and not fetch?

slurp reads the whole file into memory, afterwards it does base64
encoding still in memory, and then transfers the content.

This is basically what happens in slurp:

---
$ python
Python 2.7.14 (default, Oct 12 2017, 15:50:02) [GCC] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import base64
>>> import os
>>> with open('/isos/rhel-server-5.11-x86_64-dvd.iso', 'rb') as source:
... source_content = source.read()
...
>>> data = base64.b64encode(source_content)
>>>
---

My testfile is a 4.2GB and ram usage for that single python process goes
up to 12.5GB before settling at 9.7GB.

Sooooo you might not have enough RAM on the system you're slurping the
file from.

Sebastian

--
Sebastian Meyer
Linux Consultant & Trainer
Mail: me...@b1-systems.de

B1 Systems GmbH
Osterfeldstraße 7 / 85088 Vohburg / http://www.b1-systems.de
GF: Ralph Dehner / Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537

[Jonathan Lozada De La Matta]

I think you might not have enough space to do this. You should contact red hat support and get them to help you.

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.

To post to this group, send email to ansible...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/31cc4d14-b626-8eaf-18ef-5a73dfd9c832%40b1-systems.de.

For more options, visit https://groups.google.com/d/optout.

--

Jonathan lozada de la matta

AUTOMATION PRACTICE

 

[Sindhuja Koneru]

Thanks for the reply Sebastian.

It(backup.yml) is a inbuilt module provided in the ansible bundle. It is also running fetch.py during the download 

 File "/usr/lib/python2.7/site-packages/ansible/plugins/action/fetch.py", line 95, in run

    slurpres = self._execute_module(module_name='slurp', module_args=dict(src=source), task_vars=task_vars)

[Sindhuja Koneru]

To download, ansible is using fetch module.

fetch:

   dest: ' '

   fail_on_missing: yes

   flat: yes

   src: ' '

[Sindhuja Koneru]

I found that it’s a big in the Ansible module.
Module fetch cannot download larger files , instead we need to try synchronize module with mode set to pull and it is working.

[Brian Coca]

fetch will automatically use slurp if become is true, fetch itself
should not have issues with large files, but slurp uses RAM > x2 the
file size due to copying the full file into memory and then base64
encoding it.


--
----------
Brian Coca

[Dominic D'Apice]

https://access.redhat.com/articles/3344101

https://docs.ansible.com/ansible/2.5/modules/fetch_module.html