Zookeeper authentication

[Etienne Champetier]

Hello Clickhouse team,

I'm unable to connect to my zookeeper cluster using auth

Before I dig too deep if someone see an obvious error

######################################################################

# cat /etc/clickhouse-server/conf.d/zookeeper.xml
<yandex>
    <zookeeper>
        <node>
            <host>172.27.209.171</host>
        </node>
        <node>
            <host>172.27.209.172</host>
        </node>
        <node>
            <host>172.27.209.173</host>
        </node>
        <session_timeout_ms>30000</session_timeout_ms>
        <identity>admin:deruai4czzj5BgoxljklkjtyeR</identity>
    </zookeeper>
</yandex>

######################################################################

# cat /etc/zookeeper/zookeeper_jaas.conf
Server {
    org.apache.zookeeper.server.auth.DigestLoginModule required
    user_admin="deruai4czzj5BgoxljklkjtyeR";
};

QuorumServer {
    org.apache.zookeeper.server.auth.DigestLoginModule required
    user="admin"
    password="der>ai4czzj5BgoxtyljklkleS"
    user_admin="der>ai4czzj5BgoxtyljklkleS";
};

######################################################################

# cat /etc/zookeeper/zoo.cfg

tickTime=2000
initLimit=10
syncLimit=5
dataDir=/opt/zookeeper-data
clientPort=2181
server.1=172.27.209.171:2888:3888
server.2=172.27.209.172:2888:3888
server.3=172.27.209.173:2888:3888

authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
requireClientAuthScheme=sasl
jaasLoginRenew=3600000
zookeeper.allowSaslFailedClients=false

######################################################################

# tail -n100 /var/log/clickhouse-server/clickhouse-server.log

...

2018.08.22 21:12:04.073935 [ 19 ] <Trace> ZooKeeper: initialized, hosts: 172.27.209.172:2181,172.27.209.173:2181,172.27.209.171:2181
2018.08.22 21:12:04.110762 [ 19 ] <Error> DDLWorker: Terminating. Cannot initialize DDL queue.: Code: 999, e.displayText() = ZooKeeperImpl::Exception: Not authenticated, path: /clickhouse/task_queue, e.what() = ZooKeeperImpl::Exception, Stack trace:

0. /usr/bin/clickhouse-server(StackTrace::StackTrace()+0x16) [0x5041ba6]
1. /usr/bin/clickhouse-server(ZooKeeperImpl::Exception::Exception(std::string const&, int, int)+0x28) [0x4f8fc88]
2. /usr/bin/clickhouse-server(ZooKeeperImpl::Exception::Exception(int, std::string const&)+0x68) [0x4f90ad8]
3. /usr/bin/clickhouse-server(zkutil::ZooKeeper::createAncestors(std::string const&)+0x206) [0x4f823f6]
4. /usr/bin/clickhouse-server(DB::DDLWorker::run()+0xa5) [0x49450a5]
5. /usr/bin/clickhouse-server() [0x597b32f]
6. /lib64/libpthread.so.0(+0x7e25) [0x7f6428367e25]
7. /lib64/libc.so.6(clone+0x6d) [0x7f6427b87bad]

2018.08.22 21:12:06.069946 [ 23 ] <Debug> ConfigReloader: Loading config `/etc/clickhouse-server/config.xml'

######################################################################

# tail /var/log/zookeeper/zookeeper.log

...

2018-08-22 21:12:04,077 - INFO  [ProcessThread(sid:3 cport:-1)::PrepRequestProcessor@653] - Got user-level KeeperException when processing sessionid:0x200010a86f00004 type:create cxid:0x1 zxid:0x6000000b6 txntype:-1 reqpath:n/a Error Path:/clickhouse Error:KeeperErrorCode = NodeExists for /clickhouse
2018-08-22 21:12:04,084 - INFO  [ProcessThread(sid:3 cport:-1)::PrepRequestProcessor@653] - Got user-level KeeperException when processing sessionid:0x200010a86f00004 type:create cxid:0x2 zxid:0x6000000b7 txntype:-1 reqpath:n/a Error Path:null Error:KeeperErrorCode = NoAuth

The zookeeper cluster was always configured with auth, so if /clickhouse exists, it means clickhouse was able to talk to zookeeper at one point

Thanks in advance

[Etienne Champetier]

Found the issue, I started the server once with "user:password" as identity, so /clickhouse acl was set to that