Duplicate exported resources

Galed Friedmann

unread,

Oct 26, 2011, 4:01:23 AM10/26/11

to puppet...@googlegroups.com

Hey all,

I'm exporting a Host resource across my puppet managed client for various synchronizations across my servers. Lately I'm seeing duplicate exported resources problems and have a hard time understanding what is causing this, I'll appreciate your help :)

This happens almost every day at various times, I see in the log file:

Oct 26 07:43:57 ubuntu puppet-master[6164]: Exported resource Host[proxy1back] cannot override local resource on node lb5back

Oct 26 07:44:00 ubuntu puppet-master[6164]: Exported resource Host[proxy1back] cannot override local resource on node lb3

Oct 26 07:44:01 ubuntu puppet-master[6164]: Exported resource Host[proxy1back] cannot override local resource on node lb3back

As you can see it happens on several servers and not on a specific one, the duplicate resource happens on random servers and the error affects random servers (some of my servers are operating fine and does not report duplicate errors).

Looking at the puppet DB resources table I see this:

mysql> select * from resources where title = 'proxy1back';

+--------+------------+---------+---------+----------------+----------+------+---------------------+---------------------+

+--------+------------+---------+---------+----------------+----------+------+---------------------+---------------------+

| 573048 | proxy1back | Host | 39640 | 26 | 1 | 63 | 2011-10-26 02:01:22 | 2011-10-26 02:01:22 |

| 576542 | proxy1back | Host | 40107 | 2 | 0 | 63 | 2011-10-26 04:01:05 | 2011-10-26 04:01:05 |

| 581182 | proxy1back | Host | 40844 | 26 | 1 | 63 | 2011-10-26 07:01:35 | 2011-10-26 07:01:35 |

+--------+------------+---------+---------+----------------+----------+------+---------------------+---------------------+

It seems like at some point the server is getting a different host_id and then exporting the resource again with the same title, which causes the duplicate. But I have no idea why..

My Host resource looks like this:

@@host { "$hostname":

comment => "$hostname",

ip => "$ec2_local_ipv4",

name => "stable_host$proxy_id",

tag => "production-proxy",

}

Thanks in advance,

Galed.

jcbollinger

unread,

Oct 26, 2011, 9:16:08 AM10/26/11

to Puppet Users

On Oct 26, 3:01 am, Galed Friedmann <galed.friedm...@onavo.com> wrote:
> Hey all,
> I'm exporting a Host resource across my puppet managed client for various
> synchronizations across my servers. Lately I'm seeing duplicate exported
> resources problems and have a hard time understanding what is causing this,
> I'll appreciate your help :)
>
> This happens almost every day at various times, I see in the log file:
> Oct 26 07:43:57 ubuntu puppet-master[6164]: Exported resource
> Host[proxy1back] cannot override local resource on node lb5back
> Oct 26 07:44:00 ubuntu puppet-master[6164]: Exported resource
> Host[proxy1back] cannot override local resource on node lb3
> Oct 26 07:44:00 ubuntu puppet-master[6164]: Exported resource
> Host[proxy1back] cannot override local resource on node lb3
> Oct 26 07:44:01 ubuntu puppet-master[6164]: Exported resource
> Host[proxy1back] cannot override local resource on node lb3back
> Oct 26 07:44:01 ubuntu puppet-master[6164]: Exported resource
> Host[proxy1back] cannot override local resource on node lb3back
>
> As you can see it happens on several servers and not on a specific one, the
> duplicate resource happens on random servers and the error affects random
> servers (some of my servers are operating fine and does not report
> duplicate errors).
>
> Looking at the puppet DB resources table I see this:
> mysql> select * from resources where title = 'proxy1back';

> +--------+------------+---------+---------+----------------+----------+------+---------------------+---------------------+

> +--------+------------+---------+---------+----------------+----------+------+---------------------+---------------------+

> | 573048 | proxy1back | Host | 39640 | 26 | 1 |
> 63 | 2011-10-26 02:01:22 | 2011-10-26 02:01:22 |
> | 576542 | proxy1back | Host | 40107 | 2 | 0 |
> 63 | 2011-10-26 04:01:05 | 2011-10-26 04:01:05 |
> | 581182 | proxy1back | Host | 40844 | 26 | 1 |
> 63 | 2011-10-26 07:01:35 | 2011-10-26 07:01:35 |

> +--------+------------+---------+---------+----------------+----------+------+---------------------+---------------------+

>
> It seems like at some point the server is getting a different host_id and
> then exporting the resource again with the same title, which causes the
> duplicate. But I have no idea why..
>
> My Host resource looks like this:
> @@host { "$hostname":
> comment => "$hostname",
> ip => "$ec2_local_ipv4",
> name => "stable_host$proxy_id",
> tag => "production-proxy",
>
> }

I think you're looking for the problem in the wrong place. The way I
read the error messages, the problem is that you are trying to collect
resources that conflict with ordinary resources already defined on the
affected nodes. Check for declarations of host resources other than
the one you show above in the manifests applicable to the affected
nodes.

John

Galed Friedmann

unread,

Oct 27, 2011, 3:23:07 AM10/27/11

to puppet...@googlegroups.com

Hmm..

I'm not setting up local Host resources any where in the manifests, my nodes only collect the exported resources.

Almost all of my nodes are exporting their Host resource, and I have several hosts that collect the resource according to their tag name:

Host <<| tag == "production-proxy" |>> {

ensure => present,

}

Host <<| tag =="backup-proxy" |>> {

ensure => present,

}

This also happens to me for an Opsviewmonitored resource I'm exporting, which is being collected only by one server that has no local resources like this set..

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To post to this group, send email to puppet...@googlegroups.com.
To unsubscribe from this group, send email to puppet-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.

jcbollinger

unread,

Oct 27, 2011, 9:10:08 AM10/27/11

to Puppet Users

On Oct 27, 2:23 am, Galed Friedmann <galed.friedm...@onavo.com> wrote:
> Hmm..
> I'm not setting up local Host resources any where in the manifests, my nodes
> only collect the exported resources.
>
> Almost all of my nodes are exporting their Host resource, and I have several
> hosts that collect the resource according to their tag name:
>
> Host <<| tag == "production-proxy" |>> {
> ensure => present,
> }
>
> Host <<| tag =="backup-proxy" |>> {
> ensure => present,
> }
>
> This also happens to me for an Opsviewmonitored resource I'm exporting,
> which is being collected only by one server that has no local resources like
> this set..

Then perhaps you're collecting the same resource twice. I wouldn't
normally expect that to be a problem, but when you add property
overrides to the mix then that could change things. Even if the
overrides match. That also ties in even closer to the error messages,
which mention overriding.

Why do you need to override 'ensure' anyway? Can't you just export it
with that 'ensure' value in the first place?

John

Galed Friedmann

unread,

Oct 30, 2011, 3:42:47 AM10/30/11

to puppet...@googlegroups.com

I didn't know there was such a difference between exporting the ensure and setting it when collecting, I thought it was supposed to be the same thing ...

Anyways, I fixed this but I still see the error happening from time to time, got a few of those this morning:

Oct 30 07:34:16 ubuntu puppet-master[10501]: Exported resource Host[proxy4] cannot override local resource on node lb2

Oct 30 07:34:17 ubuntu puppet-master[10337]: Exported resource Host[proxy4] cannot override local resource on node lb5

Oct 30 07:34:19 ubuntu puppet-master[10337]: Exported resource Host[proxy4] cannot override local resource on node lb3back

This is the new manifest:

proxy* servers:

@@host { "$hostname":

comment => "$hostname",

ip => "$ec2_local_ipv4",

name => "stable_host$proxy_id",

tag => "production-proxy",

ensure => present,

}

lb* servers:

Host <<| tag == "production-proxy" |>> {

}

Host <<| tag =="backup-proxy" |>> {

}

Still no idea why this is happening. I have some hosts that sometime change their IP address and I have to sync this with the other servers. Is it possible that the cause is that there is already a host record in the /etc/hosts file that has another IP address in it? (Although I'd assume puppet should just change it and not take it as a local resource...)

I'd appreciate the help, quite desperate about this...

Thanks!

John

jcbollinger

unread,

Oct 31, 2011, 11:47:35 AM10/31/11

to Puppet Users

On Oct 30, 2:42 am, Galed Friedmann <galed.friedm...@onavo.com> wrote:
> I didn't know there was such a difference between exporting the ensure and
> setting it when collecting, I thought it was supposed to be the same thing

As I seem to be saying frequently of late, the effect on the client
should be the same if the catalog compiles, but the effect on catalog
compilation is not necessarily the same. Also, it's cleaner design-
wise to collect resources without overriding their properties.
Overriding has legitimate uses, but you should prefer to export the
full target state of a resource when you can do so.

> ...
>
> Anyways, I fixed this but I still see the error happening from time to
> time, got a few of those this morning:
> Oct 30 07:34:16 ubuntu puppet-master[10501]: Exported resource Host[proxy4]
> cannot override local resource on node lb2
> Oct 30 07:34:16 ubuntu puppet-master[10501]: Exported resource Host[proxy4]
> cannot override local resource on node lb2
> Oct 30 07:34:17 ubuntu puppet-master[10337]: Exported resource Host[proxy4]
> cannot override local resource on node lb5
> Oct 30 07:34:17 ubuntu puppet-master[10337]: Exported resource Host[proxy4]
> cannot override local resource on node lb5
> Oct 30 07:34:19 ubuntu puppet-master[10337]: Exported resource Host[proxy4]
> cannot override local resource on node lb3back
>
> This is the new manifest:
> proxy* servers:
> @@host { "$hostname":
> comment => "$hostname",
> ip => "$ec2_local_ipv4",
> name => "stable_host$proxy_id",
> tag => "production-proxy",
> ensure => present,
> }
>
> lb* servers:
> Host <<| tag == "production-proxy" |>> {
> }
>
> Host <<| tag =="backup-proxy" |>> {
> }

If you're not performing any actual overrides then you should remove
the empty braces, too. It seems unlikely that that would solve the
problem, but not completely out of the question. I'd remove for
clarity, though, if for no other reason.

> Still no idea why this is happening. I have some hosts that sometime change
> their IP address and I have to sync this with the other servers. Is it
> possible that the cause is that there is already a host record in the
> /etc/hosts file that has another IP address in it? (Although I'd assume
> puppet should just change it and not take it as a local resource...)

It is conceivable that Puppet gets confused if in /etc/hosts it finds
separate records, one having the same host name as the collected
resource but a different address, and the other having the same IP
address but a different name. I agree, however, that that *shouldn't*
be a problem, and I would be surprised if it were. You should be able
to test that case.

Do you have hosts with duplicate hostnames (perhaps in different
domains)? Are any of the affected hosts flipping between being
primary and backup proxies, or have they done since you turned on
storeconfigs? If you clear out your storeconfig database do the
problems continue and/or come back later?

John

Galed Friedmann

unread,

Nov 2, 2011, 8:51:36 AM11/2/11

to puppet...@googlegroups.com

Hi again,

I'm always deleting exported resources for hosts that are no longer in use, also changed what you suggested.

It seems like this problem is still happening once in a while but much less than it used to so maybe things are starting to work out here.