[ANN] Nomad 0.11.4 Released
Michael Schurter
We are releasing Nomad 0.11.4 containing a backported bug fix that is critical for some users to successfully upgrade existing Nomad servers which rely on Vault. If you are already running Nomad 0.11.3 successfully, there is no need to upgrade.
Nomad client agents are unaffected and do not need to be upgraded.
Revoking Vault Tokens
Prior to Nomad 0.11.2, a bug could prevent Nomad from properly revoking and purging Vault tokens. While these tokens would be safely revoked by Vault, Nomad would still have a record of them and continually try to remove them.
While Nomad 0.11.2 fixed this bug, users with a large number of old Vault tokens to revoke (hundreds of thousands to millions), could see their Nomad servers run out of memory or timeout after being elected leader due to a bug in batching revocation. Users affected by this bug will likely see the following in their leader server’s logs:
background token revocation errored
The following metric will likely stay at or near 0:
nomad.vault.distributed_tokens_revoked
Upon upgrading servers to v0.11.4 you can confirm you were affected by this bug and that it was fixed by looking for the new log line:
batching tokens to be revoked
That log line is only emitted when there are multiple batches of tokens to be revoked which should only occur when upgrading from pre-0.11.2 versions of Nomad. Monitoring the following metric should show the tokens being periodically revoked until the backlog of tokens to be revoked is processed:
nomad.vault.distributed_tokens_revoked
Go 1.14.6
Nomad 0.11.4 also upgrades to Go 1.14.6 from 1.14.3 to address two Go CVEs. We believe these CVEs are low severity for Nomad users and have only upgraded Go out of an abundance of caution. See issue 8441 for details.
The Nomad Team
Links:
Changelog - https://github.com/hashicorp/nomad/blob/v0.11.4/CHANGELOG.md
Binaries - https://releases.hashicorp.com/nomad/0.11.4/