Hi,
I'm trying to get some pillar data from external source so I wrote a module that calls https api of the source and returns the data but it so slow and I guess it's because how saltstack runs external pillar. I want to share my scenario and see what it's the best way to do it.
As an example I have 500 minions and 100 of password which are stored in an external source so I defined 100 external pillar that all these pillar use same module to get the password.
Example of ext pillar definition:
ext_pillar:
- module_name:
- data_key_in_external_source
- targeting_condition(like 'kernel:linux')
- targeting_type(like 'grain')
- pillar_key(the key that should be appeared in pillar data)
- pillar_subKey
20 passwords are shared between all minions and the rest should be used based on minion role, as an example mysql servers should get mysql password.
Problem is for each minion 100 ext_pillar definitions should be run that mean for 500 minions it should be run 50K times!!!!
I used salt.utils.minions.CkMinions() class to do targeting that has huge load on the salt-master and if minion has targeting condition then module tries to get password from external source and creates a data structure with pilla_key, pillar_subKeys and password as final value.
I'm going to change this structure to put data_key_in_external_source, targeting_condition, targeting_type, pillar_key and pillar_subKey to a separate yaml file and then I'll load file in the module and I'll start to match minion condition and if minion has proper condition, data_key will be assigned to the minion and finally based on all assigned data_key, I'll get all required password in one session for the minion. By this way, I just need to define 1 ext_pillar instead of 100.
But the problem is even with this way, module should be run for each minion which mean 500 times and it could be slow too so, is there any way that I can get all data for all minion in one run instead of running module for each minion?
reagrds