slow inventory build

[Michael DiDomenico]

i have a flat list of hosts, no groups, no variables, total is over
15k. when i run

$ head inv-expanded.yml
all:
hosts:
admin:
admin1:
admin2:
admin3:
admin4:
admin5:
admin6:
...snipped...

$ time ansible-inventory -i inv-expanded.yml --list > a
real 0m28.334s
user 0m23.725s
sys 0m4.452s

$ ansible-inventory --version
ansible-inventory 2.9.27
config file = /etc/ansible/ansible.cfg
configured module search path = ['/usr/share/ansible/plugins/modules']
ansible python module location = /usr/lib/python3.6/site-packages/ansible
executable location = /usr/bin/ansible-inventory
python version = 3.6.8 (default, Oct 20 2022, 09:31:56) [GCC 8.5.0
20210514 (Red Hat 8.5.0-15)]

$ head -n20 /proc/cpuinfo
processor : 0
vendor_id : AuthenticAMD
cpu family : 23
model : 49
model name : AMD EPYC 7302P 16-Core Processor
stepping : 0
microcode : 0x8301055
cpu MHz : 3000.000
cache size : 512 KB

when i include the groups and variables this slows down even more.

this ticket https://github.com/ansible/ansible/issues/30534 seems to
claim that there used to be an issue in the v2.3/v2.4 era, but it was
fixed and i should be seeing ~6s or less

is there a way to speed this up without splitting the inventory file
into multiple smaller files?

[Brian Coca]

Do you have libyaml installed? it is the C version of the YAML
tokenizer, if not the processing will be done in python, which is much
slower. The difference is not noticeable with small files, but it
really becomes visible with large ones.


--
----------
Brian Coca

[Michael DiDomenico]

it is installed. is there a way for me to confirm that ansible is
actually using it?

$ rpm -qa | grep -i yaml
perl-CPAN-Meta-YAML-0.018-397.el8.noarch
python3-pyyaml-3.12-12.el8.x86_64
libyaml-0.1.7-5.el8.x86_64

> --
> You received this message because you are subscribed to the Google Groups "Ansible Project" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/CACVha7fT33czos32m%3DTVxzMpWMdg5M7EAnvf44oUuOwHV0VLUQ%40mail.gmail.com.

[Brian Coca]

It could be related to this
https://github.com/ansible/ansible/issues/79652 (see linked PR for
fix), but you really don't provide enough information.

--
----------
Brian Coca

[Will McDonald]

You can strace the ansible-inventory execution to check what libraries and syscalls are used:

$ strace ansible-inventory -i inventory/hosts --list 2>&1 | grep libya

openat(AT_FDCWD, "/lib64/libyaml-0.so.2", O_RDONLY|O_CLOEXEC) = 3

$ rpm -qf /lib64/libyaml-0.so.2
libyaml-0.2.5-8.fc37.x86_64

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/CABOsP2MUpfN_KoVqfTLr-hmTPTba%2BvLvR%3DwBJ6hVxcNgTNyAKw%40mail.gmail.com.

[Michael DiDomenico]

thanks i did see that yesterday when i opened my ticket.
https://github.com/ansible/ansible/issues/79664

if there's more info i can provide please let me know

> --
> You received this message because you are subscribed to the Google Groups "Ansible Project" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/CACVha7cwa-5DCzg03GDj%3DKtKE3BYH2eZDLRmG3MRsE1LWX6atw%40mail.gmail.com.

[Michael DiDomenico]

ah, yes your right for some reason that didn't jump into my head. i
am in fact using libyaml

inventory-testing]$ strace ansible-inventory -i ./inv.py --list 1>a 2>&1
[inventory-testing]$ grep libyaml a

openat(AT_FDCWD, "/lib64/libyaml-0.so.2", O_RDONLY|O_CLOEXEC) = 3

i also found this interesting, seems excessive

[[inventory-testing]$ grep open a | sort | uniq -c | grep -v "^ [1-9]"
18591 openat(AT_FDCWD, "/home/user/.ansible/plugins/connection",
O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = -1 ENOENT (No such file
or directory)
111536 openat(AT_FDCWD, "/home/user/.ansible/plugins/vars",
O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = -1 ENOENT (No such file
or directory)
18590 openat(AT_FDCWD,
"/usr/lib/python3.6/site-packages/ansible/plugins/connection",
O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 3
18590 openat(AT_FDCWD,
"/usr/lib/python3.6/site-packages/ansible/plugins/connection/__pycache__",
O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 3
111535 openat(AT_FDCWD,
"/usr/lib/python3.6/site-packages/ansible/plugins/vars",
O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 3
111535 openat(AT_FDCWD,
"/usr/lib/python3.6/site-packages/ansible/plugins/vars/__pycache__",
O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 3
18591 openat(AT_FDCWD, "/usr/share/ansible/plugins/connection",
O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 3
111536 openat(AT_FDCWD, "/usr/share/ansible/plugins/vars",
O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 3
22 stat("/usr/lib64/python3.6/site-packages/cryptography/hazmat/backends/openssl",
{st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0

> To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/CAKtKohR%2BBLLeJ8OoQk9gvJn-O%3D_JHJ9u%3DvtDi8wkedfLLHTmGg%40mail.gmail.com.