Why does the salt-master uses the pure-Python, slow YAML SafeLoader?

[Markus Kramer]

Hi

I have in issue with runners being very CPU intensive.

Therefore I profiled a salt-master and was surprised to find the pure Python YAML scanner at 65% of a RunnerClient.

 

From the profiler call-graph:

```

RunnerClient.low  salt/client/mixins.py:246  // 88%

safe_load salt/utils/yamlloader.py:218   // 57%

…

r.check_event yaml/parser.py:94  // 56%

check_token yaml/scanner.py:113   // 56%

fetch_more_tokens yaml/scanner.py:153   // 55%

scan_to_next_token yaml/scanner.py:753   // 54%

```

The pure Python YAML Loader is said to be much slower than the YAML CLoader.

Therefore, file `salt/utils/yamlloader.py` contains this replacement:

 

```

import yaml

try:

    yaml.Loader = yaml.CLoader

    yaml.Dumper = yaml.CDumper

except Exception:

    pass

```

But variable `yaml.Loader` is never used, instead there is  `class SaltYamlSafeLoader(yaml.SafeLoader)`

 

I guess the yaml.SafeLoader calls the pure Python yaml.parser and yaml.scanner

 

Does the salt-master uses the slow, pure-Python SafeLoader?

Why not use CSafeLoader?

 

Thank you

[Markus Kramer]

Seems that CSafeLoader  misses these functions:

   'fetch_plain'  

   'peek'  

   'fetch_single'  

   'fetch_double' 

Can the CSafeLoader replace the SafeLoader none the less?

How can I prevent a Runner to spends 2/3 of its ticks on YAML loading/parsing scanning?

The runner seems to load grains. 

As I do not need any grains, can I skip the loading of the grains?

Which other YAML a Runners reads?

[Daniel Wallace]

Here is the place where salt replaced the default yaml loader with the
libyaml loaders.

https://github.com/saltstack/salt/pull/48309

If something uses salt.utils.yamlloader it will use the CSafeLoader,
and we overwrite it so all salt stuff uses the CSafeLoader.

Starting in 2019.2

> --
> You received this message because you are subscribed to the Google Groups "Salt-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to salt-users+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/salt-users/b899d038-8967-4953-9ae4-0b5ef4a9696a%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Markus Kramer]

Thank you for the good news, I would have missed that pull request. 

Before upgrading, I will test a Runner that does not load YAML.

You received this message because you are subscribed to a topic in the Google Groups "Salt-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/salt-users/LIxzVCrPc1E/unsubscribe.
To unsubscribe from this group and all its topics, send an email to salt-users+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/salt-users/CAKw0-%2BOeD6-%2B%3DPsDKasvc4%2BhKUU91wVn5cM%2B9j7QhVESgAwX5Q%40mail.gmail.com.

[David Boucha]

In general loading json is much faster than yaml.  If you can load your data in your runner with json you'll likely see even greater results.

To view this discussion on the web visit https://groups.google.com/d/msgid/salt-users/CACAsXuMcDa2jCixHG5j3_4ZBycfs8W%3DF8sjq46rP6FHMRdNyHA%40mail.gmail.com.

[Markus Kramer]

Hi David,

long time no see!

That is good to kow, but my runner does not load any YAML. Instead each RunnerClient creates a MasterMinion which loads grains with YAML.

Because my runner does need no grains, I experimentally assign grains to the empty set {}. This caused the load of the salt-master to beautifully drop to less then a tenth (from 0.5 to 0.03)! 

I'll write up a PR.

Best regards,

Markus

To view this discussion on the web visit https://groups.google.com/d/msgid/salt-users/CAJJqF90NFCcKjJHQ4Xi%2BMDCo0KsmZvZaihXAvrqABB673Wg%2BvA%40mail.gmail.com.

[David Boucha]

ah, cool!

To view this discussion on the web visit https://groups.google.com/d/msgid/salt-users/CACAsXuM5bNEj12ee%3DavsZLpnTZc2Ox7fefbuMUGmN2z%3DZf%3DQOQ%40mail.gmail.com.