Apply renewed searchguard license to elasticsearch cluster

[Rudra]

When asking questions, please provide the following information:

* Search Guard and Elasticsearch version 6.0

* Installed and used enterprise modules, if any yes-ES 6.1 version

* JVM version and operating system version centos 6.8

* Search Guard configuration files 

* Elasticsearch log messages on debug level 

* Other installed Elasticsearch or Kibana plugins, if any yes-kibana

[Rudra]

What is the procedure to apply renewed licience.

[Jochen Kressin]

A renewed license can be applied just like the initial license. You can install it by using:

• sg_config.yml and uploading it with sgadmin
• using the REST API
• using the Kibana Config GUI

Please see this chapter in the docs:

https://docs.search-guard.com/latest/search-guard-enterprise-edition#applying-an-enterprise-or-compliance-license

[Rudra]

Thank you for the response.

I could not find the existing license in my sgconf.yml file.

[Jochen Kressin]

I do not fully understand. Your initial question was how to apply a Search Guard license which is described in the documentation link I posted. If you do not have a license yet you probably run Search Guard with the trial license. Or, someone has uploaded the license directly via the REST API or the Search Guard configuration GUI. In this case, the license will not show up in the sg_config.yml file of course.

What is the output of:

https://localhost:9200/_searchguard/license

[Rudra]

Yeah, Thanks.

 I am trying to understand the current configuration and it will be renewed.

[Jochen Kressin]

So if you don't see the license in the config files I assume someone used Kibana or the REST API to upload it. The license you attached is definitely valid and active.

You can also use sgadmin with the -r/--retrieve switch to download the currently active configuration from the cluster. This will include the license string in sg_config.yml

[rud]

 I am trying to retrieve current configuration with proper credentials. Am i missing here.

./sgadmin.sh -cacert /config/bdm/bdm-es-server/config/ca-bundle.cer -cn BDM-ES-DEV-HQ -p 10150 -cd /upapps/bdm/bdm-es-server/plugins/search-guard-6/sgconfig -cert /privdir/dbdm100/dbdm-admin.cer -key /privdir/dbdm100/dbdm-admin.key.pk8 -keypass /privdir/dbdm100/dbdmadmin.pass -r

Search Guard Admin v6

Will connect to localhost:10150 ... done

01:20:20.597 [main] ERROR com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore - Your keystore or PEM does not contain a key. If you sepcified a key password try removing it. If you not sepcified a key password maybe you one because the key is password protected. Maybe you just confused keys and certificates.

ERR: An unexpected IllegalStateException occured: failed to load plugin class [com.floragunn.searchguard.SearchGuardPlugin]

Trace:

java.lang.IllegalStateException: failed to load plugin class [com.floragunn.searchguard.SearchGuardPlugin]

        at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:452)

        at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:105)

        at org.elasticsearch.client.transport.TransportClient.newPluginService(TransportClient.java:103)

        at org.elasticsearch.client.transport.TransportClient.buildTemplate(TransportClient.java:128)

        at org.elasticsearch.client.transport.TransportClient.<init>(TransportClient.java:251)

        at com.floragunn.searchguard.tools.SearchGuardAdmin$TransportClientImpl.<init>(SearchGuardAdmin.java:823)

        at com.floragunn.searchguard.tools.SearchGuardAdmin.main0(SearchGuardAdmin.java:403)

        at com.floragunn.searchguard.tools.SearchGuardAdmin.main(SearchGuardAdmin.java:120)

Caused by: java.lang.reflect.InvocationTargetException

        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)

        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)

        at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:443)

        ... 7 more

Caused by: ElasticsearchSecurityException[Error while initializing transport SSL layer from PEM: java.lang.IllegalArgumentException: File does not contain valid private key: /privdir/dbdm100/dbdm-admin.key.pk8]; nested: IllegalArgumentException[File does not contain valid private key: /privdir/dbdm100/dbdm-admin.key.pk8]; nested: InvalidKeySpecException[Cannot retrieve the PKCS8EncodedKeySpec]; nested: BadPaddingException[Given final block not properly padded];

        at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.initSSLConfig(DefaultSearchGuardKeyStore.java:292)

        at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.<init>(DefaultSearchGuardKeyStore.java:145)

        at com.floragunn.searchguard.ssl.SearchGuardSSLPlugin.<init>(SearchGuardSSLPlugin.java:192)

        at com.floragunn.searchguard.SearchGuardPlugin.<init>(SearchGuardPlugin.java:182)

        ... 12 more

Caused by: java.lang.IllegalArgumentException: File does not contain valid private key: /privdir/dbdm100/dbdm-admin.key.pk8

        at io.netty.handler.ssl.SslContextBuilder.keyManager(SslContextBuilder.java:267)

        at io.netty.handler.ssl.SslContextBuilder.forServer(SslContextBuilder.java:90)

        at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.buildSSLServerContext(DefaultSearchGuardKeyStore.java:613)

        at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.initSSLConfig(DefaultSearchGuardKeyStore.java:287)

        ... 15 more

Caused by: java.security.spec.InvalidKeySpecException: Cannot retrieve the PKCS8EncodedKeySpec

        at javax.crypto.EncryptedPrivateKeyInfo.getKeySpec(EncryptedPrivateKeyInfo.java:255)

        at io.netty.handler.ssl.SslContext.generateKeySpec(SslContext.java:965)

        at io.netty.handler.ssl.SslContext.getPrivateKeyFromByteBuffer(SslContext.java:1013)

        at io.netty.handler.ssl.SslContext.toPrivateKey(SslContext.java:993)

        at io.netty.handler.ssl.SslContextBuilder.keyManager(SslContextBuilder.java:265)

        ... 18 more

Caused by: javax.crypto.BadPaddingException: Given final block not properly padded

        at com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:991)

        at com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:847)

        at com.sun.crypto.provider.PBES1Core.doFinal(PBES1Core.java:416)

        at com.sun.crypto.provider.PBEWithMD5AndDESCipher.engineDoFinal(PBEWithMD5AndDESCipher.java:316)

        at javax.crypto.Cipher.doFinal(Cipher.java:2165)

        at javax.crypto.EncryptedPrivateKeyInfo.getKeySpec(EncryptedPrivateKeyInfo.java:250)

[Jochen Kressin]

-keypass /privdir/dbdm100/dbdmadmin.pass

Here you have to provide the actual password, not a file. If you are worried about having the password in the bash history you can also use the  -prompt/--prompt-for-password switch which will turn on interactive mode.

[rud]

Thank You , I can retrive license by connecting any node in the cluster with admin certificate but we do not have Search Guard configuration GUI for adding license.

Applying the new license with REST API below leaves same existing access permissions?

********

PUT /_searchguard/api/license/

{ 

  "sg_license": <licensestring>

}

*********

we have  REST api is only exposed to client nodes with "Cname", 

*********

kibana:

      multitenancy_enabled: true

      server_username: "db156"

  

*********

[Jochen Kressin]

Yes, it just updates the license. The Kibana config GUI uses the same endpoint under the hood.

[rud]

Hi Jochen,

Thank You for the support. Here

1)**

Using REST endpoint , I am trying to PUT  the renewed licience - As license string is not visible sg_config.yml file 

PUT /_searchguard/api/license/

{ 

  "sg_license": <licensestring>

}

Here i am using nss to SSL for curl| certificate database and imported certificates

SSL_DIR=~/nss curl -vk --cert bdsys --pass xxxx -sS -XGET 'https://xxxx:xxx/_searchguard/api/license/'  (same for all the /_searchguard/api/<configuration type>/)

* About to connect() to xxx.com port xxx (#0)

*   Trying 45.54.150.170... connected

* Connected to dxxx.com (45.54.150.170) port xxx (#0)

* Initializing NSS with certpath: sql:/xx/xx/nss

* warning: ignoring value of ssl.verifyhost

* skipping SSL peer certificate verification

* NSS: using client certificate: bdsys

*       subject: CN=bdsys,O=xxx,L=xxx,ST=xxx,C=US

*       start date: Jan 03 20:28:37 2018 GMT

*       expire date: Jan 02 20:28:37 2023 GMT

*       common name: bdsys

*       issuer: CN=BD-sd,O=xxx,ST=xxx,C=US

* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256

* Server certificate:

*       subject: CN=actualserver,OU=xxx,O=UP,L=xxx,ST=xxx,C=US

*       start date: Oct 19 21:41:00 2017 GMT

*       expire date: Oct 18 21:41:00 2020 GMT

*       common name: actualserver

*       issuer: CN=web CA,DC=xx,DC=xx,DC=xx,DC=com

> GET /_searchguard/api/actiongroups/ HTTP/1.1

> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.21 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2

> Host: dxx.com:xxx

> Accept: */*

>

< HTTP/1.1 403 Forbidden

< access-control-allow-credentials: true

< content-type: application/json; charset=UTF-8

< content-length: 191

<

* Connection #0 to host xxx.com left intact

* Closing connection #0

{"status":"FORBIDDEN","message":"No permission to access REST API: Role based access not enabled.. SG admin permissions required but CN=bdsys,O=xxx,L=xxx,ST=xxx,C=US is not an admin"}  

How should i PUT the updated license with REST endpoint with admin certificate?

---sg_roles_mappimg.yml--

sg_all_access:

  users:

  - "CN=bdsys,O=X,L=xxx,ST=xx,C=x"

  

--elasticsearch.yml---

  searchguard.authcz.admin_dn:

  - "CN=bd-admin,O=X,L=xxx,ST=xx,C=x"  --admin certificate

curl -k --cert bdsys:**** -sS -XGET 'https://devxxxxx.com:xxx/_searchguard/authinfo'?pretty

{

  "user" : "User [name=bdsys, roles=[], requestedTenant=null]",

  "user_name" : "bdsys",

  "user_requested_tenant" : null,

  "remote_address" : "xxxxx:56276",

  "backend_roles" : [ ],

  "custom_attribute_names" : [ ],

  "sg_roles" : [

    "sg_all_access",

    "sg_own_index"

  ],

  "sg_tenants" : {

    "test_tenant_ro" : true,

    "adm_tenant" : true,

    "bdmsys" : true

  },

  "principal" : "CN=bdsys,O=xx,L=xxx,ST=xxx,C=xxx",

  "peer_certificates" : "2"

}

--------------------------------------------------------

2)**

--sg_config.yml----

  kibana:

      multitenancy_enabled: true

      server_username: "db345"

      index: ".kibana"

As i kw the "Kibana server user"used for maintenance, managing the .kibana index 

We are not using any visualizations, When do i actually use the kibana server user name for any operations in the process of SG license renewel?

-------------------------------

[rud]

How should i upload the license with REST API put with admin certificate.

[Jochen Kressin]

You are using the wrong certificate for your curl call. You need to use an admin certificate, in your case this one here:

 searchguard.authcz.admin_dn:
  - "CN=bd-admin,O=X,L=xxx,ST=xx,C=x"  --admin certificate

Since you used a node or client certificate you see this error message:

{"status":"FORBIDDEN","message":"No permission to access REST API: Role based access not enabled.. SG admin permissions required but CN=bdsys,O=xxx,L=xxx,ST=xxx,C=US is not an admin"}  

The internal Kibana server user is never used for anything other than the Kibana index maintenance. So it needs to be configured, but it's not an account for direct use.

If you want to 

[rud]

Thank You, I can upload the license to individual hosts with sgadmin.sh.

Is there a way to apply license at cluster level via sgadmin.sh.

[Jochen Kressin]

You do not need to upload the license to all nodes. The license is stored in the Search Guard configuration index and is this propagated to all nodes automatically.

[rud]

I got it. Thanks a Bunch.

Here, What is the difference and where should we use both of them.

curl -vk --cert ./xxx.cer  --key ./xxx.key -XGET 'https://xxxx1.bxxxm:xxx/_searchguard/api/configuration/config'  (Here i don't see LDAP passwords in the configuration file)

/sgadmin.sh --diagnose -cacert xxxxx -cn xxxx -h xxx-p xxxx-cd ./sgconfig -cert xxxxx -keyxxx -keypass xxx -r -nhnv   (Here see LDAP passwords in the configuration file)

[rud]

Another Question :(I am trying to understand NSS vs OPENSSL)

Got confused by reading the information on different websites, I really appreciate if you could share the information.

When i use curl with nssdb with certificates imported it uses  (SSL_DIR=~/nss curl -vk --cert bdmsys:xxxx -XGET 'https:/xxxxxxx/_cat/health')

 ###SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256  (more secure than TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384)

 

 ###User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.21 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2

 When i use curl with --cert ./bdmsys.cer  --key ./bdmsysd.key (curl -vk --cert ./bdmsys.cer  --key ./bdmsysd.key -XGET 'https://xxxxxx/_cat/health')

 

 ###SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384

 

  ###User-Agent: curl/7.29.0

[rud]

Can some one check this.

[rud]

* Search Guard and Elasticsearch version 6.0

* Installed and used enterprise modules, if any yes-ES 6.1 version

* JVM version and operating system version centos 6.8

* Search Guard configuration files 

* Elasticsearch log messages on debug level 

* Other installed Elasticsearch or Kibana plugins, if any yes-kibana

Hi Jochen,

My license is about to expire in couple of days on Production and we are working on it to get the renewed license. What would be the solution and procedure In case if i  receive my license keys after expiration.

Is it the same procedure to apply renewed license as before and after expiration ?   https://search-guard.com/faq/ says my cluster will still work.

Quick response would be appreciated.

Thanks

[rud]

Can some one please respond.

[Jochen Kressin]

The procedure is exactly the same, yes. Your production cluster will continue to work even if the current license has expired. You will see warning messages about the expired license in many places, but Search Guard will continue to function. We will never put a production system at risk just because a renewal process is taking slightly longer than expected!

[rud]

Thank You, For how long my cluster will work with expired license?