Fleetd Engine Leadership change every 2 seconds

Jon Borgonia

unread,

Aug 11, 2016, 12:39:55 AM8/11/16

to CoreOS User

is this normal behavior?

journalctl -f

Aug 11 04:36:55 10.0.0.2 fleetd[831]: INFO engine.go:81: Engine leadership changed from 0aa381ddd8bc492ebf5448071f757f4b to bdede29cd5654fcaa9022331455a05fd
Aug 11 04:37:01 10.0.0.2 fleetd[831]: INFO engine.go:81: Engine leadership changed from bdede29cd5654fcaa9022331455a05fd to 5531cdd1721641af9ab96c43504609e7
Aug 11 04:37:25 10.0.0.2 fleetd[831]: INFO engine.go:81: Engine leadership changed from 5531cdd1721641af9ab96c43504609e7 to e4f0d7acc41e414a9b2ac2fe03990b61
Aug 11 04:37:31 10.0.0.2 fleetd[831]: INFO engine.go:81: Engine leadership changed from e4f0d7acc41e414a9b2ac2fe03990b61 to 5531cdd1721641af9ab96c43504609e7
Aug 11 04:37:33 10.0.0.2 fleetd[831]: INFO engine.go:81: Engine leadership changed from 5531cdd1721641af9ab96c43504609e7 to e4f0d7acc41e414a9b2ac2fe03990b61
Aug 11 04:37:35 10.0.0.2 fleetd[831]: INFO engine.go:81: Engine leadership changed from e4f0d7acc41e414a9b2ac2fe03990b61 to 5531cdd1721641af9ab96c43504609e7
Aug 11 04:37:37 10.0.0.2 fleetd[831]: INFO engine.go:81: Engine leadership changed from 5531cdd1721641af9ab96c43504609e7 to e4f0d7acc41e414a9b2ac2fe03990b61
Aug 11 04:37:39 10.0.0.2 fleetd[831]: INFO engine.go:81: Engine leadership changed from e4f0d7acc41e414a9b2ac2fe03990b61 to 5531cdd1721641af9ab96c43504609e7
Aug 11 04:37:41 10.0.0.2 fleetd[831]: INFO engine.go:81: Engine leadership changed from 5531cdd1721641af9ab96c43504609e7 to e4f0d7acc41e414a9b2ac2fe03990b61
Aug 11 04:37:43 10.0.0.2 fleetd[831]: INFO engine.go:81: Engine leadership changed from e4f0d7acc41e414a9b2ac2fe03990b61 to 6d03b00fe69b4af5be39cf67ad0f16e2

Brandon Philips

unread,

Aug 11, 2016, 2:45:21 PM8/11/16

to Jon Borgonia, CoreOS User

Not normal. There are a variety of reasons this could happen. Can you provide some information on the environment you are running in and the hardware configuration?

--
You received this message because you are subscribed to the Google Groups "CoreOS User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to coreos-user...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Jon Borgonia

unread,

Aug 12, 2016, 8:59:07 PM8/12/16

to CoreOS User, jon.bo...@gmail.com

$ cat /etc/os-release

NAME=CoreOS
ID=coreos
VERSION=1068.8.0
VERSION_ID=1068.8.0
BUILD_ID=2016-07-18-0616
PRETTY_NAME="CoreOS 1068.8.0 (MoreOS)"
ANSI_COLOR="1;32"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://github.com/coreos/bugs/issues"

using stable HVM AMI ami-ee57148e on AWS ec2 t2.medium

Kyle Brown

unread,

Aug 15, 2016, 2:51:28 PM8/15/16

to Jon Borgonia, CoreOS User

Jon,

Please time a write to your etcd cluster:

$ time etcdctl set /test/foo bar

If this is higher than 1 fleet will time out. You can tune this timeout via FLEET_REQUEST_TIMEOUT. Additional fleet tuning options can be found here. If you find that your etcd cluster is being hammered by fleet it may help to reduce the load on the cluster by tripling the default fleet parameters.

FLEET_ENGINE_RECONCILE_INTERVAL=8
FLEET_AGENT_TTL=60s
FLEET_ETCD_REQUEST_TIMEOUT=3.0

Cheers,

Kyle Brown

To unsubscribe from this group and stop receiving emails from it, send an email to coreos-user+unsubscribe@googlegroups.com.

Reply all

Reply to author

Forward