run external pillar once for all minions

[mohammad Samadzadegan]

Hi,

I'm trying to get some pillar data from external source so I wrote a module that calls https api of the source and returns the data but it so slow and I guess it's because how saltstack runs external pillar. I want to share my scenario and see what it's the best way to do it.

As an example I have 500 minions and 100 of password which are stored in an external source so I defined 100 external pillar that all these pillar use same module to get the password.

Example of ext pillar definition:

ext_pillar:

  - module_name:

    - data_key_in_external_source

    - targeting_condition(like 'kernel:linux')

    - targeting_type(like 'grain')

    - pillar_key(the key that should be appeared in pillar data)

    - pillar_subKey

20 passwords are shared between all minions and the rest should be used based on minion role, as an example mysql servers should get mysql password.

Problem is for each minion 100 ext_pillar definitions should be run that mean for 500 minions it should be run 50K times!!!!

I used salt.utils.minions.CkMinions() class to do targeting that has huge load on the salt-master and if minion has targeting condition then module tries to get password from external source and creates a data structure with pilla_key, pillar_subKeys and password as final value.

I'm going to change this structure to put data_key_in_external_source, targeting_condition, targeting_type, pillar_key and pillar_subKey to a separate yaml file and then I'll load file in the module and I'll start to match minion condition and if minion has proper condition, data_key will be assigned to the minion and finally based on all assigned data_key, I'll get all required password in one session for the minion. By this way,  I just need to define 1 ext_pillar instead of 100.

But the problem is even with this way, module should be run for each minion which mean 500 times and it could be slow too so, is there any way that I can get all data for all minion in one run instead of running module for each minion?

reagrds

[mohammad Samadzadegan]

one more finding,

I changed module to run once per minion and get all assigned password but it's still slow and I guess, problem is with  salt.utils.minions.CkMinions() class that I used for targeting. below is an example of yaml file that declares what the targeting is, which data_key should be used and what the pillar key/subKeys should be appeared in the pillar :

pwd1:
  tgt1:
    - 'kernel:Linux'
    - 'grain'
  tag:
    data_key1
  keys:
    - pillar_key

pwd2:
  tgt2:
    - 'G@kernel:Linux and G@custom:data1'
    - 'compound'
  tag:
    data_key2
  keys:
    - pillar_key
    - pillar_subKey

for each top key (pwd1/pwd2/...) per minion I'm calling  salt.utils.minions.CkMinions() class for targeting purpose, is there any lighter and faster method/class that I can use?

[mohammad Samadzadegan]

I found salt.matchers.compound_match.match function which is much faster that CkMinion class. any suggestion faster than this function or is there a way to run external_pillar module once for all minion instead of running module for each minion separately?

[Phipps, Thomas]

that isn't how pillar works. ext_pillar is rendered on a per minion bases because minions can request pillars at different times. and pillar is a pull based system. all pillar knows is which minion requested data and it will render for that minion. however it shouldn't be running 100 times for each minion. it should only run once and grab all definitions at that time.

if you want to elevate that, you might say make an engine that pre-pulls the data and then have your ext_pillar pull from the local cache that your engine builds instead of hitting your service directly.

--
You received this message because you are subscribed to the Google Groups "Salt-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to salt-users+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/salt-users/84f0ec7f-8451-4014-a23e-d284ffcf009an%40googlegroups.com.

[Abdellatif Housni]

Hello Mohammad,

Some approach to reduce the burden would be :

- try to use async calls with your minions

- use a Secret management System (Hashicorp Vault comes to mind)

- use a specific salt syndic to distribute the load, but it requires additional configuration compared to single master configuration

- use the salt mine but it comes with a security requierement for you to put in place : ACL and or encryption of your credential data

Hope it helps

Best regards

[mohammad Samadzadegan]

  Thanks. I changed module to get all data for each minion in one loop and get data from memory for duplicated data with different condition or different pillar keys b. I don't know how to use Salt cache, could you please send me a link about how I can use salt cache if there is any document?

[mohammad Samadzadegan]

Hello

Thank you for your suggestions, to make them clear to myself 

- use a Secret management System (Hashicorp Vault comes to mind), I guess, there should not be much different in performance by using different data source system(Vault or DB), The problem is with number of calls that should be done for refreshing pillar for all minions. in both cases for each minion, all passwords should have been retrieved from source. 
- use the salt mine: could you please explain your idea more?, I can't connect dots in this case. how sharing passwords for one minion to the other can help?

regards 

[Abdellatif Housni]

Hello Mohammad,

there is a way to specify ACL of mine data described here : 

https://salt.tips/whats-new-in-salt-neon/#salt-mine

Hope it 

Regards