how to use salt restful api efficiently

[ramfs init0]

Hi

I had deployed the salt-stack with salt-api and followed the tutorial, but i find the salt api is slow.

Such as this situation, i use api with Requests, the python http lib. As shown below pseudo-code, api has spent 40seconds on executed the simple linux order('ls /data').

class SaltHost:
   def init(get api token)
   def execute_ord(send command string with token):
        requests.post(url, headers={token auth dict}, data={tgt is ip, func is cmd.run, arg is linux order})

salt_host = new SaltHost()
x = salt_host.execute_ord(ip, 'ls /data')

i don`t know where is key to improve the execute time. Maybe my request class is i inefficient?  or something wrong in deploy process?

Thanks!

[Boris FELD]

Hi,

Which netapi do you use? Do you use rest_cherrypy or rest_tornado?

Do you launch a job and actively waiting for its result or do you poll the salt-api to check if the result is present?

--
You received this message because you are subscribed to the Google Groups "Salt-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to salt-users+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Szymon Juraszczyk]

W dniu 2016-08-03 o 10:12, Boris FELD pisze:

> Hi,
>
> Which netapi do you use? Do you use rest_cherrypy or rest_tornado?
> Do you launch a job and actively waiting for its result or do you poll
> the salt-api to check if the result is present?

Hi

I'm not sure how polling works for you but I found it very inefficient.
For some reason it takes almost one second to poll for one task. An
interesting approach is to start asynchronous tasks with API and then
listen on the event bus for the events with their results. It is an
overkill for one task but may hopefully be more justified when you
submit hundred/thousands of them from one central interface. Events just
come "for free" and you can passively listen for them.

--
Regards,
Szymon Juraszczyk, szy...@juraszczyk.com

[Seth House]

As Boris said, there's a lot of ways to use salt-api.

If you're running jobs synchronously and waiting for the return the
performance should be very close to Salt's CLI. (They use most of the
same code paths.) For example, if you have any downed minions Salt
will wait for them to timeout before returning just like it does at
the CLI.

A very common way to use salt-api is to fire a job asynchronously to
get the JID and then fetch the jobs results for that JID sometime
later. This matches how Salt itself works behind the scenes and pairs
well with the norm of short-lived HTTP connections. Of course that
means you have to add a way to get the result later, often by polling
or watching the event stream.

[Seth House]

On Wed, Aug 3, 2016 at 11:26 AM, Szymon Juraszczyk
<szy...@juraszczyk.com> wrote:
> I'm not sure how polling works for you but I found it very inefficient.

You're right about that. Polling for returns has all the same overhead
as invoking the jobs runner from the CLI -- the two are functionally
equivalent. There's overhead in instantiating the RunnerClient which
in turn starts a new listener to the event bus, and then the jobs
runner also stats quite a few files on the file system. It's pretty
heavy. For most installations I probably wouldn't poll more frequently
than once or twice a minute, and for places with a large job cache
even less frequently.

Listening to the event stream, as you suggested, is a great
lightweight alternative to polling and it's close to real-time. You
just need a good way to manage the backgrounded listener which is
pretty language & client dependent. :-P

[Daniel Jung]

I use reactor plus returner for remote execution. Returner i am using is redis. I chose reactor over api for auth reasons.

I have multiple syndics in the environment. Once you introduce syndic into the salt architect, its whole lot more difficult to do things like retrieving jids and returns async.

By using external returner to retrieve results, u offload the burden on the master and faster retrieval of results from the minions.

[Szymon Juraszczyk]

On 2016-08-04 at 09:21, Daniel Jung wrote:
> I use reactor plus returner for remote execution. Returner i am using is redis. I chose reactor over api for auth reasons.
>
> I have multiple syndics in the environment. Once you introduce syndic into the salt architect, its whole lot more difficult to do things like retrieving jids and returns async.

Why is that? JID is still JID and the returns are passed back to the
master on which the job was created (that is what the extra master_id
attribute is for).

> By using external returner to retrieve results, u offload the burden on the master and faster retrieval of results from the minions.

That is certainly true that masters can be heavily loaded with job
results. When I start a job on the master via API, ultimately the result
comes back to it and appears on the event bus into which you can also
tap via API.

Can you share more details how you avoided passing job results to the
master? And when you run 'salt -L minionname test.ping, where does it
get the results from?

[Daniel Jung]

On Aug 4, 2016 09:47, "Szymon Juraszczyk" <szy...@juraszczyk.com> wrote:
>
> On 2016-08-04 at 09:21, Daniel Jung wrote:
> > I use reactor plus returner for remote execution. Returner i am using is redis. I chose reactor over api for auth reasons.
> >
> > I have multiple syndics in the environment. Once you introduce syndic into the salt architect, its whole lot more difficult to do things like retrieving jids and returns async.
>
> Why is that? JID is still JID and the returns are passed back to the
> master on which the job was created (that is what the extra master_id
> attribute is for)

Are you running syndic in your enviornment? Do you get JID when you sent to salt-api (cherry) as return?

>
> > By using external returner to retrieve results, u offload the burden on the master and faster retrieval of results from the minions.
>
> That is certainly true that masters can be heavily loaded with job
> results. When I start a job on the master via API, ultimately the result
> comes back to it and appears on the event bus into which you can also
> tap via API.
>
> Can you share more details how you avoided passing job results to the
> master?

Srry if i gave the impression you can. I just dont use the returna from the master. By the way i have requested as a feature as an option to not send returns when returners are used.

And when you run 'salt -L minionname test.ping, where does it
> get the results from?
>
> --
> Regards,
> Szymon Juraszczyk, szy...@juraszczyk.com
>

> --
> You received this message because you are subscribed to a topic in the Google Groups "Salt-users" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/salt-users/BZDAhHXgj8c/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to salt-users+...@googlegroups.com.

[Daniel Jung]

On Aug 4, 2016 14:14, "Daniel Jung" <mimian...@gmail.com> wrote:
>
>
> On Aug 4, 2016 09:47, "Szymon Juraszczyk" <szy...@juraszczyk.com> wrote:
> >
> > On 2016-08-04 at 09:21, Daniel Jung wrote:
> > > I use reactor plus returner for remote execution. Returner i am using is redis. I chose reactor over api for auth reasons.
> > >
> > > I have multiple syndics in the environment. Once you introduce syndic into the salt architect, its whole lot more difficult to do things like retrieving jids and returns async.
> >
> > Why is that? JID is still JID and the returns are passed back to the
> > master on which the job was created (that is what the extra master_id
> > attribute is for)
> Are you running syndic in your enviornment? Do you get JID when you sent to salt-api (cherry) as return?

Actually ignore my last comment, i meant to say result returns and not JId.

[Szymon Juraszczyk]

On 2016-08-04 at 23:14, Daniel Jung arote:

>

> Are you running syndic in your enviornment? Do you get JID when you sent
> to salt-api (cherry) as return?

Yes. Don't you? :-) The only real issue with JIDs I found when I started
using syndics is that without syndics if you tried to start a job on a
non-existent minion using list target (not sure about the glob one), the
job wouldn't be created. With syndics the master apparently has no clue
about what minions there are so it always creates the job and forwards
it to the syndics. That makes my use case a bit tricky as I may need to
install minions on demand when they are not present.

[Szymon Juraszczyk]

On 2016-08-04 at 23:18, Daniel Jung wrote:
>> Are you running syndic in your enviornment? Do you get JID when you
> sent to salt-api (cherry) as return?
> Actually ignore my last comment, i meant to say result returns and not JId.

Too late ;-) The results of asynchronous jobs do appear on the top-level
master's event bus where they can be consumed from. There are shorn of
some attributes (such as retcode for cmd.run) as we spoke recently
(reported as https://github.com/saltstack/salt/issues/34992), which
makes them useless for me, but I did a crude hack/fix for that. The bug
is one of 1471 open High Severity bugs so I guess it will take a while
before we can see it fixed properly. I couldn't share my changes there
but I wrote in the comment of that issue where in my opinion the code
should be fixed, in case you are still interested in that after 1.5
years of finding it before I did.

[Daniel Jung]

On Aug 4, 2016 14:48, "Szymon Juraszczyk" <szy...@juraszczyk.com> wrote:
>
> On 2016-08-04 at 23:18, Daniel Jung wrote:
> >> Are you running syndic in your enviornment? Do you get JID when you
> > sent to salt-api (cherry) as return?
> > Actually ignore my last comment, i meant to say result returns and not JId.
>
> Too late ;-) The results of asynchronous jobs do appear on the top-level
> master's event bus where they can be consumed from.

Yeah i dont rely on the returns from the syndics as with scale grows it starts to drop returns and what not.

There are shorn of
> some attributes (such as retcode for cmd.run) as we spoke recently
> (reported as https://github.com/saltstack/salt/issues/34992), which
> makes them useless for me, but I did a crude hack/fix for that. The bug
> is one of 1471 open High Severity bugs so I guess it will take a while
> before we can see it fixed properly. I couldn't share my changes there
> but I wrote in the comment of that issue where in my opinion the code
> should be fixed, in case you are still interested in that after 1.5
> years of finding it before I did

Haha i thought ur name sounded familiar.
I have also found alternative sol using returner :)

> --
> Regards,
> Szymon Juraszczyk, szy...@juraszczyk.com
>

[Szymon Juraszczyk]

On 2016-08-05 at 01:11, Daniel Jung wrote:
> Yeah i dont rely on the returns from the syndics as with scale grows it
> starts to drop returns and what not.

Seems like the best fun is still ahead of me. I already start loving
SaltStack... It sounds scary that job results may never make it to the
master. I thought it was supposed to be scalable.

Thanks for advice, who knows, I may need to go the same path as you did
although it may ruin the integration I have already started.

[Daniel Jung]

Once you have so many minions spread among syndics and want to get all the returns to a single master all at once and saltmaster firing  saltutil.find_job..to see if the jobs still running after some minions dont return. It starts to cause issues.  Recently zmq highwater mark plus other zmq settings have been bumped to accommodate high number minions but i have yet to give it a try. With limited targets returning, it works really well, just not when you target too many.