Running sgadmin - no permissions for indices:admin/exists

[Michael]

OS: Red Hat Enterprise Linux Server release 7.3

JVM: openjdk version "1.8.0_111"

Search Guard: 5.5.0-14

Elasticsearch: 5.5.0-1

Number of nodes in cluster: 3

I upgraded from Search Guard 5.3.0 & Elasticsearch 5.3.0 and found I cannot update my Search Guard configuration using sgadmin.sh anymore. Whenever I run the the sgadmin.sh command, I get the following below error:

> plugins/search-guard-5/tools/sgadmin.sh -h myhost -cd plugins/search-guard-5/sgconfig/ -ks mykeystore.jks -kspass kspass -ts mytruststore.jks -tspass tspass -nhnv -icl


Contacting elasticsearch cluster 'elasticsearch' and wait for YELLOW clusterstate ...
Clustername: mycluster
Clusterstate: GREEN
Number of nodes: 3
Number of data nodes: 2
ERR: An unexpected ElasticsearchSecurityException occured: no permissions for indices:admin/exists
Trace:
ElasticsearchSecurityException[no permissions for indices:admin/exists]
 at com.floragunn.searchguard.filter.SearchGuardFilter.apply(SearchGuardFilter.java:147)
 at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:168)
 at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:142)
 at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:64)
 at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:54)
 at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceivedDecorate(SearchGuardSSLRequestHandler.java:177)
 at com.floragunn.searchguard.transport.SearchGuardRequestHandler.messageReceivedDecorate(SearchGuardRequestHandler.java:191)
 at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceived(SearchGuardSSLRequestHandler.java:139)
 at com.floragunn.searchguard.SearchGuardPlugin$2$1.messageReceived(SearchGuardPlugin.java:336)
 at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69)
 at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1544)
 at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
 at org.elasticsearch.common.util.concurrent.EsExecutors$1.execute(EsExecutors.java:110)
 at org.elasticsearch.transport.TcpTransport.handleRequest(TcpTransport.java:1501)
 at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:1385)
 at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:74)
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
 at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
 at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:297)
 at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:413)
 at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
 at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
 at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1267)
 at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1078)
 at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
 at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)
 at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
 at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334)
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
 at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926)
 at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134)
 at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644)
 at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544)
 at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498)
 at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
 at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
 at java.lang.Thread.run(Thread.java:745)

Using the Search Guard REST Management API, I gotten the relevant current configuration items.

actiongroups

{
  "CLUSTER_ALL" : [
    "cluster:*"
  ],
  "ALL" : [
    "indices:*"
  ],
  "CRUD" : [
    "READ",
    "WRITE"
  ],
  "SEARCH" : [
    "indices:data/read/search*",
    "indices:data/read/msearch*",
    "SUGGEST"
  ],
  "MONITOR" : [
    "indices:monitor/*"
  ],
  "DATA_ACCESS" : [
    "indices:data/*",
    "indices:admin/mapping/put"
  ],
  "CREATE_INDEX" : [
    "indices:admin/create",
    "indices:admin/mapping/put"
  ],
  "WRITE" : [
    "indices:data/write*",
    "indices:admin/mapping/put"
  ],
  "MANAGE_ALIASES" : [
    "indices:admin/aliases*"
  ],
  "READ" : [
    "indices:data/read*"
  ],
  "DELETE" : [
    "indices:data/write/delete*"
  ],
  "CLUSTER_COMPOSITE_OPS" : [
    "indices:data/write/bulk",
    "indices:admin/aliases*",
    "CLUSTER_COMPOSITE_OPS_RO"
  ],
  "CLUSTER_COMPOSITE_OPS_RO" : [
    "indices:data/read/mget",
    "indices:data/read/msearch",
    "indices:data/read/mtv",
    "indices:data/read/coordinate-msearch*",
    "indices:admin/aliases/exists*",
    "indices:admin/aliases/get*"
  ],
  "GET" : [
    "indices:data/read/get*",
    "indices:data/read/mget*"
  ],
  "MANAGE" : [
    "indices:monitor/*",
    "indices:admin/*"
  ],
  "CLUSTER_MONITOR" : [
    "cluster:monitor/*"
  ],
  "INDEX" : [
    "indices:data/write/index*",
    "indices:data/write/update*",
    "indices:admin/mapping/put"
  ],
  "SUGGEST" : [
    "indices:data/read/suggest*"
  ]
}

roles

{

  "sg_all_access" : {
    "cluster" : [
      "*"
    ],
    "indices" : {
      "*" : {
        "*" : [
          "*"
        ]
      }
    }
  },
  "sg_kibana" : {
    "cluster" : [
      "CLUSTER_MONITOR",
      "CLUSTER_COMPOSITE_OPS_RO"
    ],
    "indices" : {
      "*" : {
        "*" : [
          "READ",
          "indices:admin/mappings/fields/get*"
        ]
      },
      "?kibana" : {
        "*" : [
          "ALL"
        ]
      }
    }
  },
  "sg_public" : {
    "cluster" : [
      "cluster:monitor/main",
      "CLUSTER_COMPOSITE_OPS_RO"
    ]
  },
  "sg_own_index" : {
    "cluster" : [
      "CLUSTER_COMPOSITE_OPS"
    ],
    "indices" : {
      "${user_name}" : {
        "*" : [
          "ALL"
        ]
      }
    }
  },
  "sg_logstash" : {
    "cluster" : [
      "indices:admin/template/get",
      "indices:admin/template/put",
      "CLUSTER_MONITOR",
      "CLUSTER_COMPOSITE_OPS"
    ],
    "indices" : {
      "*beat*" : {
        "*" : [
          "CRUD",
          "CREATE_INDEX"
        ]
      },
      "logstash-*" : {
        "*" : [
          "CRUD",
          "CREATE_INDEX"
        ]
      }
    }
  },
  "sg_readall" : {
    "cluster" : [
      "CLUSTER_COMPOSITE_OPS_RO"
    ],
    "indices" : {
      "*" : {
        "*" : [
          "READ"
        ]
      }
    }
  }
}

rolesmapping

{
  "sg_all_access" : {
    "users" : [
      "admin"
    ]
  },
  "sg_kibana" : {
    "users" : [
      "kibana"
    ]
  },
  "sg_logstash" : {
    "users" : [
      "logstash"
    ]
  },
  "sg_readall" : {
    "users" : [
      "ronly1"
    ]
  }
}

Any suggestions for overcoming this issue?

[SG]

Thats most likely a problem with your ssl certificates.

Pls. share elasticsearch.yml and make sure mykeystore.jks contains a client admin certificate (the one which is configured in elasticsearch.yml) see https://github.com/floragunncom/search-guard-docs/blob/master/sgadmin.md#configuring-the-admin-certificate
Is this a production issue or just in devel/uat/staging ...?

(maybe your certificates have worked prior to SG 14 - if this is the case then we need to document this as a breaking change)

> --
> You received this message because you are subscribed to the Google Groups "Search Guard" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.
> To post to this group, send email to search...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/e643a359-df02-4989-b163-eb3c13b6d460%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Michael]

This is a proof of concept, but one of the acceptance criteria is to ensure TLS is implemented. Therefore, the configuration feels like it's for production. The root, signing CA, client and node certificates were generated using https://floragunn.com/tls-certificate-generator/ and are valid at least until 2019.

Here's the relevant snippet in elasticsearch.yml

# --------------------------------- Search Guard SSL Configuration -------------
#
searchguard.ssl.http.enabled: true
searchguard.ssl.http.keystore_filepath: node1-keystore.jks
searchguard.ssl.http.keystore_password: node1pass
searchguard.ssl.http.truststore_filepath: mytruststore.jks
searchguard.ssl.http.truststore_password: tspass
searchguard.ssl.transport.keystore_filepath: node1-keystore.jks
searchguard.ssl.transport.keystore_password: node1pass
searchguard.ssl.transport.truststore_filepath: mytruststore.jks
searchguard.ssl.transport.truststore_password: tspass
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.authcz.admin_dn:
  - CN=sgadmin
  - CN=node1-keystore

Here's the relevant keystores

> keytool -list -keystore plugins/search-guard-5/sgconfig/sgadmin-keystore.jks -storepass sgpass


Keystore type: JKS
Keystore provider: SUN


Your keystore contains 1 entry


cn=sgadmin, Apr 5, 2017, PrivateKeyEntry,
Certificate fingerprint (SHA1): **:**:**:**:**:**:**:**:**:**:**:**:**:**:**:**:**:**:**:**


> keytool -list -keystore plugins/search-guard-5/sgconfig/node1-keystore.jks -storepass node1pass


Keystore type: JKS
Keystore provider: SUN


Your keystore contains 1 entry


cn=node1-keystore, Apr 5, 2017, PrivateKeyEntry,
Certificate fingerprint (SHA1): **:**:**:**:**:**:**:**:**:**:**:**:**:**:**:**:**:**:**:**


> keytool -list -keystore plugins/search-guard-5/sgconfig/mytruststore.jks -storepass tspass


Keystore type: JKS
Keystore provider: SUN


Your keystore contains 1 entry


root-ca-chain, Apr 5, 2017, trustedCertEntry,
Certificate fingerprint (SHA1): **:**:**:**:**:**:**:**:**:**:**:**:**:**:**:**:**:**:**:**

The keystores were not changed as part of the upgrade and the same sgadmin command provided was able to run sgadmin.sh in SG 5.3.0 and ES 5.3.0. 

[Michael]

So a bit of a breakthrough / workaround - when I include the keystore and truststore passwords as part of the sgadmin command on the command line, it fails with the reported error.

If I exported the same passwords as environment variables SG_KS_PASS and SG_TS_PASS, then sgadmin works as expected.

On Thursday, 13 July 2017 00:14:25 UTC+10, Michael wrote:

[SG]

thats pretty strange, i will investigate this

> --
> You received this message because you are subscribed to the Google Groups "Search Guard" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.
> To post to this group, send email to search...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/68e68964-b57c-42a0-8998-1e3981875c73%40googlegroups.com.

[Search Guard]

see also https://github.com/floragunncom/search-guard/issues/366

[Search Guard]

i am glad that it is working for you but i cannot reproduce this (for me it always working with command line passwords as well as environment passwords)

Maybe you passwords contains special characters which interfere with shell expansion or globbing (especially & and ! signs are good candidates here)

For your original issue there is also now one on github: https://github.com/floragunncom/search-guard/issues/366

Long story short: Use admin client certificates instead of node certificates for sgadmin