Memory consumption when working with large input args

19 views

Skip to first unread message

Paco Ramon Penella

unread,

Jan 30, 2023, 7:45:01 AM1/30/23

to Ansible Development

Hi all,

I realized that when working with large input arguments in a task, this causes problems of high memory consumption. Let me give you more details.

I developed a custom module, called get_artifacts, which receives as an input argument (among others), a list of artifacts to be downloaded in each host.

This list of artifacts is list of json objects containing a lot of properties for each artifact (like size, url, permissions, path, etc.) which means it can weigh around 60MB.

When I execute my playbook, I have noticed that memory is increased every time that a host finish the custom task.

Detailed examples:

My set up:

>>ansible --version

ansible [core 2.11.12]
config file = memory_consumption/ansible/ansible.cfg
configured module search path = [u'memory_consumption/ansible/role_deploy/library']
ansible python module location = /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/ansible
ansible collection location = /Users/user/.ansible/collections:/usr/share/ansible/collections
executable location = /Library/Frameworks/Python.framework/Versions/2.7/bin/ansible
python version = 2.7.16 (v2.7.16:413a49145e, Mar 2 2019, 14:32:10) [GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)]
jinja version = 2.10.1
libyaml = False

>>cat ansible.cfg
[defaults]
library = ./role_deploy/library
module_utils = ./role_deploy/library/module_utils
callback_whitelist = profile_memory
forks = 1

When the task starts:

TASK [role_deploy : Download the artifacts] ***********************************************************************************

---------->[started task TASK: role_deploy : Download the artifacts]

Filename: memory_consumption/ansible/role_deploy/callback_plugins/profile_memory.py

Line # Mem usage Increment Line Contents

================================================

18 343.113 MiB 343.113 MiB @profile(precision=3)

19 def v2_playbook_on_task_start(self, task, is_conditional):

20 343.113 MiB 0.000 MiB self._display.display("---------->[started task %s]" % (task), color=C.COLOR_WARN)

When the first host starts and finishes the task:

---------------------------------->[started task TASK: role_deploy : Download the artifacts on host 18.203.162.10]

Filename: memory_consumption/ansible/role_deploy/callback_plugins/profile_memory.py

Line # Mem usage Increment Line Contents

================================================

22 343.113 MiB 343.113 MiB @profile(precision=3)

23 def v2_runner_on_start(self, host, task):

24 343.113 MiB 0.000 MiB self._display.display("---------------------------------->[started task %s on host %s]" % (task, host), color=C.COLOR_WARN)

---------------------------------->[finished result {u'total_failed': 0, u'total_ok': 3238, '_ansible_no_log': False, u'deleted': False, u'changed': True, u'response_action': {u'deleted': [], u'failed': [], u'changed': 6, u'ok': 3238}, u'total_changed': 6, u'total_deleted': 0, u'invocation': {}, u'total': 3244}]

Filename: memory_consumption/ansible/role_deploy/callback_plugins/profile_memory.py

Line # Mem usage Increment Line Contents

================================================

26 420.051 MiB 420.051 MiB @profile(precision=3)

27 def v2_runner_on_ok(self, result):

28 420.051 MiB 0.000 MiB self._display.display("---------------------------------->[finished result %s]" % (result._result), color=C.COLOR_WARN)

When the second host starts and finishes the task:

---------------------------------->[started task TASK: role_deploy : Download the artifacts on host 54.216.154.16]

Filename: memory_consumption/ansible/role_deploy/callback_plugins/profile_memory.py

Line # Mem usage Increment Line Contents

================================================

22 420.051 MiB 420.051 MiB @profile(precision=3)

23 def v2_runner_on_start(self, host, task):

24 420.051 MiB 0.000 MiB self._display.display("---------------------------------->[started task %s on host %s]" % (task, host), color=C.COLOR_WARN)

Filename: memory_consumption/ansible/role_deploy/callback_plugins/profile_memory.py

Line # Mem usage Increment Line Contents

================================================

26 480.516 MiB 480.516 MiB @profile(precision=3)

27 def v2_runner_on_ok(self, result):

28 480.516 MiB 0.000 MiB self._display.display("---------------------------------->[finished result %s]" % (result._result), color=C.COLOR_WARN)

When the third host starts and finishes the task:

---------------------------------->[started task TASK: role_deploy : Download the artifacts on host 18.202.233.9]

Filename: memory_consumption/ansible/role_deploy/callback_plugins/profile_memory.py

Line # Mem usage Increment Line Contents

================================================

22 480.516 MiB 480.516 MiB @profile(precision=3)

23 def v2_runner_on_start(self, host, task):

24 480.516 MiB 0.000 MiB self._display.display("---------------------------------->[started task %s on host %s]" % (task, host), color=C.COLOR_WARN)

Filename: memory_consumption/ansible/role_deploy/callback_plugins/profile_memory.py

Line # Mem usage Increment Line Contents

================================================

26 537.219 MiB 537.219 MiB @profile(precision=3)

27 def v2_runner_on_ok(self, result):

28 537.219 MiB 0.000 MiB self._display.display("---------------------------------->[finished result %s]" % (result._result), color=C.COLOR_WARN)

As you can see, It can be a real problem when managing a lot of nodes because the master is going into OOM exception... just as it is my case.

Going into detail, I realized that ansible is keeping/storing the input args of the task in two places in the each host result: in the _result.invocation.module_args key and in the _task_fields.args.

I found that the invocation data can be forced to be empty in my custom module using something like:

...

res_args.update({"invocation": dict()})

module.exit_json(**res_args)

But I haven't found any solution to force ansible not to storing the input args under the _task_fields.args key.

Is there any way to do it? How can ansible work more efficiently when working with large input parameters?

Thank you in advance!!

PD: Is my first message in this community so please be benevolent. :)

Reply all

Reply to author

Forward

0 new messages