Service registration blocked by ACL

[Rajinder Singh]

I have a 3 node consul 8.0 cluster running in AWS.

I have gone through the new ACL guide.

Cluster is up but I am seeing these  errors in the consul log on all three servers

2017/05/02 03:34:06 [WARN] agent: Service 'consul' registration blocked by ACLs

Here is my consul config file.

{

  "log_level": "INFO",

  "server": true,

  "bootstrap_expect": 3,

  "datacenter": "dc-useast",

  "acl_datacenter": "dc-useast",

  "acl_default_policy": "deny",

  "acl_master_token": "xxxxx",

  "acl_agent_token": "yyyyy",

  "acl_down_policy": "extend-cache",

  "data_dir": "C:\\opt\\consul\\data",

  "ui_dir": "C:\\opt\\consul\\ui",

  "encrypt": "zzzzz==",

  "client_addr": "0.0.0.0",

  "bind_addr": "0.0.0.0",

  "node_name": "consul-IP-0AE30519",

  "skip_leave_on_interrupt": true,

  "leave_on_terminate": true,

  "service": {

    "name": "consul",

    "tags": ["consul-IP-0AE30519"]

  }

}

retry join config is in a separate file

I used the Consul UI to define the ACL policy for  acl_agent_token

It is a client token(not management)

agent "" {

  policy = "write"

}

key "" {

  policy = "write"

}

node "" {

  policy = "write"

}

service "" {

  policy = "write"

}

as you can see acl_agent_token does have write access to agent, node and service.

Why am I still  seeing this error in my logs:

2017/05/02 03:34:06 [WARN] agent: Service 'consul' registration blocked by ACLs

[James Phillips]

Hi Rajinder - your example policy looks right I'm assuming that got set to yyyyy in your example config. Did you restart each server to pick up the token from it's config?

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/consul/issues
IRC: #consul on Freenode
---
You received this message because you are subscribed to the Google Groups "Consul" group.
To unsubscribe from this group and stop receiving emails from it, send an email to consul-tool+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/consul-tool/f9f1f880-8bd9-4e64-a3d9-8251db316f21%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Rajinder Singh]

Yes I was hiding the token values.

Yes I diid restart my consul server and I am still seeing error messages.

Do you work out of your San Fran office by any chance?

I am attending hashi training delivered by Seth tomorrow.

If you are available I can show you the configuration.

I will post consul logs after restarting consul service on one of my servers.

To unsubscribe from this group and stop receiving emails from it, send an email to consul-tool...@googlegroups.com.

[James Phillips]

I'm in Los Angeles unfortunately. Logs would be good if you have them and I'll try to reproduce this as well. 

To unsubscribe from this group and stop receiving emails from it, send an email to consul-tool+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/consul-tool/249ca7e9-30f0-466e-ba82-8dc4e693f6fd%40googlegroups.com.

[Rajinder Singh]

Here are the logs after restarting one of the consul servers.

 2017/05/02 04:52:24 [INFO] raft: Restored from snapshot 3-155654-1493682744283

    2017/05/02 04:52:25 [INFO] raft: Initial configuration (index=1): [{Suffrage:Voter ID:10.227.134.127:8300 Address:10.227.134.127:8300} {Suffrage:Voter ID:10.227.7.109:8300 Address:10.227.7.109:8300} {Suffrage:Voter ID:10.227.5.25:8300 Address:10.227.5.25:8300}]

    2017/05/02 04:52:25 [INFO] raft: Node at 10.227.7.109:8300 [Follower] entering Follower state (Leader: "")

    2017/05/02 04:52:25 [INFO] serf: EventMemberJoin: consul-IP-0AE3076D 10.227.7.109

    2017/05/02 04:52:25 [INFO] serf: Attempting re-join to previously known node: consul-IP-0AE30519: 10.227.5.25:8301

    2017/05/02 04:52:25 [INFO] consul: Adding LAN server consul-IP-0AE3076D (Addr: tcp/10.227.7.109:8300) (DC: dc-useast)

    2017/05/02 04:52:25 [INFO] consul: Raft data found, disabling bootstrap mode

    2017/05/02 04:52:25 [INFO] serf: EventMemberJoin: consul-IP-0AE30519 10.227.5.25

    2017/05/02 04:52:25 [WARN] memberlist: Refuting a suspect message (from: consul-IP-0AE3076D)

    2017/05/02 04:52:25 [INFO] serf: EventMemberJoin: jenkins01 10.227.4.228

    2017/05/02 04:52:25 [INFO] serf: EventMemberJoin: consul-IP-0AE3867F 10.227.134.127

    2017/05/02 04:52:25 [INFO] serf: Re-joined to previously known node: consul-IP-0AE30519: 10.227.5.25:8301

    2017/05/02 04:52:25 [INFO] consul: Adding LAN server consul-IP-0AE30519 (Addr: tcp/10.227.5.25:8300) (DC: dc-useast)

    2017/05/02 04:52:25 [INFO] consul: Adding LAN server consul-IP-0AE3867F (Addr: tcp/10.227.134.127:8300) (DC: dc-useast)

    2017/05/02 04:52:25 [INFO] consul: New leader elected: consul-IP-0AE3867F

    2017/05/02 04:52:25 [INFO] serf: EventMemberJoin: consul-IP-0AE3076D.dc-useast 10.227.7.109

    2017/05/02 04:52:25 [WARN] serf: Failed to re-join any previously known node

    2017/05/02 04:52:25 [INFO] consul: Handled member-join event for server "consul-IP-0AE3076D.dc-useast" in area "wan"

    2017/05/02 04:52:25 [INFO] agent: Joining cluster...

    2017/05/02 04:52:25 [INFO] agent: No EC2 region provided, querying instance metadata endpoint...

    2017/05/02 04:52:25 [INFO] agent: Discovered 4 servers from EC2

    2017/05/02 04:52:25 [INFO] agent: (LAN) joining: [10.227.134.127 10.227.5.25 10.227.7.109 10.227.4.228]

    2017/05/02 04:52:25 [INFO] agent: (LAN) joined: 4 Err: <nil>

    2017/05/02 04:52:25 [INFO] agent: Join completed. Synced with 4 initial agents

    2017/05/02 04:52:25 [WARN] agent: Service 'consul' registration blocked by ACLs

    2017/05/02 04:52:25 [INFO] agent: Synced node info

    2017/05/02 04:52:26 [WARN] raft: Rejecting vote request from 10.227.5.25:8300 since we have a leader: 10.227.134.127:8300

==> Newer Consul version available: 0.8.1 (currently running: 0.8.0)

[Eugen Mayer]

Is it probably just the new 0.8 ACL system? http://stackoverflow.com/questions/41968048/consul-0-8-acl-migration-how-to-migrate

[Rajinder Singh]

I want to share how I got past the registration blocked by ACL error.

On all of Consul Servers I already had a policy with appropriate permissions.

agent "" {

  policy = "write"

}

key "" {

  policy = "write"

}

node "" {

  policy = "write"

}

service "" {

  policy = "write"

}

Here is how my original configuration looked like:

{

  "log_level": "INFO",

  "server": true,

  "bootstrap_expect": 3,

  "datacenter": "dc-useast",

  "acl_datacenter": "dc-useast",

  "acl_default_policy": "deny",

  "acl_master_token": "xxxxx",

  "acl_agent_token": "yyyyy",

  "acl_down_policy": "extend-cache",

  "data_dir": "C:\\opt\\consul\\data",

  "ui_dir": "C:\\opt\\consul\\ui",

  "encrypt": "zzzzz==",

  "client_addr": "0.0.0.0",

  "bind_addr": "0.0.0.0",

  "node_name": "consul-IP-0AE30519",

  "skip_leave_on_interrupt": true,

  "leave_on_terminate": true,

  "service": {

    "name": "consul",

    "tags": ["consul-IP-0AE30519"]

  }

}

I had to add acl_token and set its value to the same token I was using for acl_agent_token

 "acl_token": "yyyyy",

Here is the working configuration

{

  "log_level": "INFO",

  "server": true,

  "bootstrap_expect": 3,

  "datacenter": "dc-useast",

  "acl_datacenter": "dc-useast",

  "acl_default_policy": "deny",

  "acl_master_token": "xxxxx",

  "acl_agent_token": "yyyyy",

 "acl_token": "yyyyy",

  "acl_down_policy": "extend-cache",

  "data_dir": "C:\\opt\\consul\\data",

  "ui_dir": "C:\\opt\\consul\\ui",

  "encrypt": "zzzzz==",

  "client_addr": "0.0.0.0",

  "bind_addr": "0.0.0.0",

  "node_name": "consul-IP-0AE30519",

  "skip_leave_on_interrupt": true,

  "leave_on_terminate": true,

  "service": {

    "name": "consul",

    "tags": ["consul-IP-0AE30519"]

  }

}

This token did have agent, key, service and node write permissions.

Once I made the same change in all 3 of my consul server I stopped seeing this error:

        Service 'consul' registration blocked by ACLs

We have a jenkins server where I started consul agent as a client.

Even on that server I had to add acl_token . Once acl_token was added jenkins service was able to register with Consul servers.

{

  "log_level": "INFO",

  "datacenter": "dc-useast",

  "acl_datacenter": "dc-useast",

  "acl_default_policy": "deny",

  "acl_agent_token": "yyyyy",

  "acl_token": "yyyyy",    

  "acl_down_policy": "extend-cache",

  "data_dir": "C:\\opt\\consul\\data",

  "encrypt": "zzzzzzzzzzzzz",

  "client_addr": "0.0.0.0",

  "bind_addr": "0.0.0.0",

  "node_name": "jenkins01",

  "skip_leave_on_interrupt": true,

  "leave_on_terminate": true

}

Here is the ACL guide:

https://www.consul.io/docs/guides/acl.html

It has this table:

Special Token	Servers	Clients	Purpose
acl_agent_master_token	OPTIONAL	OPTIONAL	Special token that can be used to access Agent API when the ACL datacenter isn't available, or servers are offline (for clients); used for setting up the cluster such as doing initial join operations
acl_agent_token	OPTIONAL	OPTIONAL	Special token that is used for an agent's internal operations with the Catalog API; this needs to have at least node policy access so the agent can self update its registration information
acl_master_token	REQUIRED	N/A	Special token used to bootstrap the ACL system, see details below.
acl_token	OPTIONAL	OPTIONAL	Default token to use for client requests where no token is supplied; this is often configured with read-only access to services to enable DNS service discovery on agents

From this table it was not obvious that I need to set acl_token on my Consul servers as well as consul client VM's.

This table above says that acl_token is optional and is only used when client requests with no token.

So it seems to me that consul service running on server does need acl_token set.

Why do we have to set acl_token if acl_agent_token has been configured with correct permissions?

[James Phillips]

> Why do we have to set acl_token if acl_agent_token has been configured with correct permissions?

That sounds like a bug - I'll investigate.

--

This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/consul/issues
IRC: #consul on Freenode
---
You received this message because you are subscribed to the Google Groups "Consul" group.

To unsubscribe from this group and stop receiving emails from it, send an email to consul-tool+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/consul-tool/54b7f5ff-f3ce-4316-8920-98fdac261ac7%40googlegroups.com.

[James Phillips]

Hi Rajinder,

I was unable to reproduce this - by configuring the "acl_agent_token" the server was able to sync itself up. Here's a fully worked example:

https://gist.github.com/slackpad/d89ce0e1cc0802c3c4f2d84932fa3234

Going through this it was clear this is a painful process (especially having to restart the servers to pick up the token), so I think the right thing is to not require the trusted-already servers from needing a token. That's captured here https://github.com/hashicorp/consul/issues/2971.

Please let me know if you can't figure out what happened with your setup, it should be possible to bootstrap with v0.8.1, even though it takes a few steps to do.

-- James

[Rajinder Singh]

Thanks for looking into this and providing an example. I had started with a 0.8.0 cluster. I ran into unique node id issues as well because I was creating AMI's using packer.

I had to write code to generate unique node ids during node creation. 

Another difference is that I am on setting up consul on windows.

I am going to setup a new 0.8.1 cluster on windows. I will use the new setting that creates unique node ids. I will use steps you provided in the gist and report back.

To unsubscribe from this group and stop receiving emails from it, send an email to consul-tool...@googlegroups.com.