Service registration is working but I still see "Coordinate update blocked by ACLs"

[Василий Александрович]

Hello.

Sometimes I feel that ACL is beta functionality. After sometime of playing with ACLs I've managed to configure it when it works more or less. Despite services can register, I still see following errors in logs.

2018/07/23 13:09:48 [ERR] consul: "Coordinate.Update" RPC failed to server 192.168.44.12:58300: rpc error making call: rpc error making call: Permission denied

2018/07/23 13:09:48 [WARN] agent: Coordinate update blocked by ACLs

If add

service ""
{
  policy = "write"
}

to anonymous ACL, errors stop appearing.

-log-level=debug gives no clue.

Consul version 1.2.1

[Justin DynamicD]

Best way to help would be if you are willing to share your token policies for both the servers and nodes.  My guess is you've not configured client/server tokens correctly for consul to use in order to allow updates.  It can be kind of tricky out the gate to get things going but I've found it stable once deployed.  I generally have three policies: and anonymous read-only policy that lets you read anything, a client token/policy tat allow service management, then a 3rd token/policy for the server cluster.

[Smit Jain]

Василий Александрович <mage...@gmail.com>	Mon 23 Jul, 18:53 (7 hours ago)
to Consul Unsubscribe

You need to first sort out which is accessing which policy,  just setting the service policy to have WRITE access is not enough.  There are a lot of touch points in the ACL in Consul. ACL works great in the Consul but requires a lot of testing.

Read this first very carefully and this link : https://www.consul.io/docs/guides/acl.html#complete-acl-coverage-in-consul-0-8 

Policy	Scope
agent	Utility operations in the Agent API, other than service and check registration
event	Listing and firing events in the Event API
key	Key/value store operations in the KV Store API
keyring	Keyring operations in the Keyring API
node	Node-level catalog operations in the Catalog API, Health API, Prepared Query API, Network Coordinate API, and Agent API
operator	Cluster-level operations in the Operator API, other than the Keyring API
query	Prepared query operations in the Prepared Query API
service	Service-level catalog operations in the Catalog API, Health API, Prepared Query API, and Agent API
session	Session operations in the Session API

Thanks & Regards,

------------------------------------------------

Smit Jain
Contact no.-9582921008

Email: smitj...@gmail.com

 

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/consul/issues
Community chat: https://gitter.im/hashicorp-consul/Lobby
---
You received this message because you are subscribed to the Google Groups "Consul" group.
To unsubscribe from this group and stop receiving emails from it, send an email to consul-tool...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/consul-tool/bf10f286-d34b-4bda-9b8f-9b4a890f9137%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Василий Александрович]

Server config.

{
  "datacenter": "local_dc",
  "acl_datacenter": "local_dc",
  "acl_default_policy": "deny",
  "acl_down_policy": "extend-cache",
  "ca_file": "/opt/consul/SSL/ca.cert",
  "cert_file": "/opt/consul/SSL/consul.cert",
  "key_file": "/opt/consul/SSL/consul.key",
  "verify_outgoing": true,
  "verify_server_hostname": true,
  "verify_incoming": true,
  "data_dir": "/opt/consul/data",
  "log_level": "INFO",
  "server": true,
  "client_addr": "0.0.0.0",
  "addresses": {
    "http": "0.0.0.0",
    "https": "0.0.0.0"
  },
  "ports": {
    "dns": 58600,
    "http": 58500,
    "serf_lan": 58301,
    "serf_wan": 58302,
    "server": 58300
},
  "bind_addr": "192.168.44.12",
  "advertise_addr": "192.168.44.12",
  "enable_script_checks": true,
  "domain": "consul.local",
  "bootstrap_expect": 3,
  "ui": true,
  "start_join": ["192.168.44.10", "192.168.44.11", "192.168.44.12"],
  "encrypt": "cg8StVXbQJ0gPvMd9o7yrg=="
}

client config

{
  "datacenter": "local_dc",
  "acl_datacenter": "local_dc",
  "acl_default_policy": "deny",
  "acl_down_policy": "extend-cache",
  "ca_file": "/opt/consul/SSL/ca.cert",
  "cert_file": "/opt/consul/SSL/consul.cert",
  "key_file": "/opt/consul/SSL/consul.key",
  "verify_outgoing": true,
  "verify_server_hostname": true,
  "verify_incoming": true,
  "data_dir": "/opt/consul/data",
  "log_level": "INFO",
  "server": false,
  "client_addr": "0.0.0.0",
  "addresses": {
    "http": "0.0.0.0",
    "https": "0.0.0.0"
  },
  "ports": {
    "dns": 58600,
    "http": 58500,
    "serf_lan": 58301,
    "serf_wan": 58302,
    "server": 58300
},
  "bind_addr": "192.168.44.14",
  "advertise_addr": "192.168.44.14",
  "enable_script_checks": true,
  "domain": "consul.local",
  "start_join": ["192.168.44.10", "192.168.44.11", "192.168.44.12"],
  "encrypt": "cg8StVXbQJ0gPvMd9o7yrg=="
}

Agent ACL:

node ""
{
  policy = "write"
}
service ""
{
  policy = "write"
}
key "_rexec"
{
  policy = "write"
}

Anonymous ACL

node ""
{
  policy = "read"
}
service ""
{
  policy = "read"
}

I use API to apply token. That's why therea no config options in config files.

acl_agent_token and acl_token use the same ACL - agent ACL. One more time. "Everything wroks" with this configuration.(At least service can register) But log file is full of error messages.

Cluster consists of 3 master nodes and 2 client nodes. Cluster build in virtualbox and provisioned with ansible.

понедельник, 23 июля 2018 г., 21:32:14 UTC+2 пользователь Justin DynamicD написал:

[Василий Александрович]

This is test cluster, built in virtualbox. I just want consul client registers service. I have only two ACLs, agent and anonymous.

Agent ACL:

node ""
{
  policy = "write"
}
service ""
{
  policy = "write"
}
key "_rexec"
{
  policy = "write"
}

Anonymous ACL

node ""
{
  policy = "read"
}
service ""
{
  policy = "read"
}

The simplest configuration. Where is a pitfall?

понедельник, 23 июля 2018 г., 23:02:35 UTC+2 пользователь Smit Jain написал:

[Василий Александрович]

понедельник, 23 июля 2018 г., 23:02:35 UTC+2 пользователь Smit Jain написал:

 but requires a lot of testing.

Don't you think that this is no OK?

Why should I test hard?  I just want to enter debug mode and see, what ACL blocks what request, on what host.

[Василий Александрович]

I've managed to make narrower the search of my problem.
Coordinate.Update means https://www.consul.io/api/coordinate.html#update-lan-coordinates-for-a-node
I just want consul agent/client (not server) to be able to update "Update LAN Coordinates for a node"
But the agent ACL already has
node ""
{
  policy = "write"
}
No other apps try to update Coordinate because it is fresh install in virtualbox.
So I have no choise, other than give anonymous ACL ability to write
node ""
{
  policy = "write"
}
But I do not want to.

Moreover. During consul restart I see following messages.
2018/07/23 15:39:40 [ERR] consul: "Catalog.Register" RPC failed to server 192.168.44.12:58300: rpc error making call: rpc error making call: Permission denied
2018/07/23 15:39:40 [WARN] agent: Node info update blocked by ACLs
2018/07/23 15:39:41 [ERR] consul: "Catalog.Register" RPC failed to server 192.168.44.10:58300: rpc error making call: rpc error making call: Permission denied
2018/07/23 15:39:41 [WARN] agent: Node info update blocked by ACLs
Thanks god it appears only once.

And again at first glance, everything works fine. But log files are flooded with error messages.

First I noticed flood of errors when I tried to configure connect. Connect did not work and I decided to step few steps back to better understand ACLs.

[Paul Banks]

Hey, I commented on your more specific post before I saw this thread.

As I mentioned there having ACLs only fail during background sync is an unfortunate side-effect of the way Consul works right now. It's not obvious how that can be made a lot clearer.

Also we're well aware that ACLs are very complicated - that's mostly due to all the flexibility demanded of them by different use-cases but we are aware and designing both changes to ACL and better tooling/documentation to make it easier to get them setup in a sensible configuration easily!

Thanks for your feedback.