Salt-mine not updating reliably

[Karl Schafer]

Hi all,

I recently got salt-mine working and am able to use it to generate some nifty configuration templates.

However, I've noticed that the data in salt-mine isn't being updated reliably. 

I have a group of nodes I am deploying with Terraform, with the following mine configuration in the minion:

mine_functions:

  grains.item:

    - fqdn

    - ipv4

Some nodes never update, and aren't available in salt-mine after being re-deployed. 

After several restarts and "salt-call mine.update" commands, most are showing up except for one node.

It don't show up in this output from a mine.get command, like this:

salt-call mine.get 'os:Ubuntu' 'grains.item' grain

What is the right way to update the salt-mine data? Is this behaviour expected?

Thank you,

Karl

[Karl Schafer]

So in some more testing, I am able to get the minions that are missing in the mine.get output to show up after restarting the minions several times.

Without doing that, it doesn't get updated.

Whats the right way to do this?

[saltycdr]

Hi Karl,

You can set mine_interval via pillar or config options. I run mine.update in states that should trigger a mine update to always have fresh mine data.

My 2 cents.

cDR

[Karl Schafer]

Hi there,

I have set mine_interval to 5 minutes, and I am regularly running mine.update. The mine output doesn't still get updated until I've restarted minions multiple times.

[Karl Schafer]

I also notice in the minion logs w/ DEBUG on that I get these messages, which seem to indicate mine updates are being sent. (This node still doesn't show up in the mine output):

2017-02-07 19:55:22,599 [salt.utils.event ][DEBUG   ][2920] Sending event: tag = _minion_mine; data = {'clear': False, 'cmd': '_mine', 'data': {'grains.item': {'fqdn': 'my-host.blah.net', 'ipv4': ['1.2.3.4', '127.0.0.1']}}, 'id': 'my-host.blah.net', '_stamp': '2017-02-07T19:55:22.598564'}

[C. R. Oldham]

Hey Karl,

This sounds serious enough that I'd recommend opening an issue on our Github repo.  The issue template will ask for some specific details that will help us figure out if it's really a bug or maybe just a misconfiguration.

--cro

C. R. Oldham

Senior Platform Engineer, Integrations Team Lead, SaltStack

c...@saltstack.com / 801-564-4673

On Tue, Feb 7, 2017 at 1:00 PM, Karl Schafer <kschaf...@gmail.com> wrote:

I also notice in the minion logs w/ DEBUG on that I get these messages, which seem to indicate mine updates are being sent. (This node still doesn't show up in the mine output):

2017-02-07 19:55:22,599 [salt.utils.event ][DEBUG   ][2920] Sending event: tag = _minion_mine; data = {'clear': False, 'cmd': '_mine', 'data': {'grains.item': {'fqdn': 'my-host.blah.net', 'ipv4': ['1.2.3.4', '127.0.0.1']}}, 'id': 'my-host.blah.net', '_stamp': '2017-02-07T19:55:22.598564'}

--
You received this message because you are subscribed to the Google Groups "Salt-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to salt-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/salt-users/885c42cd-2f60-4978-a428-8ce588c571af%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Karl Schafer]

Thanks, I just opened issue 39277: https://github.com/saltstack/salt/issues/39277

On Tuesday, February 7, 2017 at 3:42:18 PM UTC-5, C. R. Oldham wrote:

Hey Karl,

This sounds serious enough that I'd recommend opening an issue on our Github repo.  The issue template will ask for some specific details that will help us figure out if it's really a bug or maybe just a misconfiguration.

--cro

C. R. Oldham

Senior Platform Engineer, Integrations Team Lead, SaltStack

c...@saltstack.com / 801-564-4673

On Tue, Feb 7, 2017 at 1:00 PM, Karl Schafer <kschaf...@gmail.com> wrote:

I also notice in the minion logs w/ DEBUG on that I get these messages, which seem to indicate mine updates are being sent. (This node still doesn't show up in the mine output):

2017-02-07 19:55:22,599 [salt.utils.event ][DEBUG   ][2920] Sending event: tag = _minion_mine; data = {'clear': False, 'cmd': '_mine', 'data': {'grains.item': {'fqdn': 'my-host.blah.net', 'ipv4': ['1.2.3.4', '127.0.0.1']}}, 'id': 'my-host.blah.net', '_stamp': '2017-02-07T19:55:22.598564'}

--
You received this message because you are subscribed to the Google Groups "Salt-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to salt-users+...@googlegroups.com.

[Karl Schafer]

I have some updates.

The issue I'm having is related to having a multi-master configuration. The minion cache isn't maintained between masters, which is why I was getting sporadic results. 

Testing using a single master appears to have fixed the issue. 

I was pointed at using Consul to act as an external cache as that feature is available in version 2016.11

https://github.com/saltstack/salt/issues/39277