Automating creation of Node Certificate

[apt221]

Elasticsearch Version: 5.6.14
Search Guard Version: 5.6.14-19.2

I am in the process of figuring out how to transfer from a single node elasticsearch cluster with SG installed to a multinode ES cluster (in kubernetes) and I would like some help understanding the process. I would like to automate the process of generating the certs required for the ES cluster. So far I have used the offline TLS tool to generate a root-ca and intermediate cert. These certs are shared to every node in the ES cluster via kubernetes secrets.

1. How do I generate the required node certificates for the cluster?

If I had 50 nodes in my elasticsearch cluster, would that mean that in my tlsconfig.yml I would need 50 entries under the nodes config section or do I just need a minimum of 1 node certificate?

2. Is it fine to simply run the sgadmin script from one node and expect the rest of the cluster to have the updated configurations? 

I can't seem to find the post right now but I remember reading in one of the SG posts a reply that stated that you would only need to update the SG indexes from one node and the changes would be updated in all nodes (I assume after you restart all these nodes).

Please let me know if any more information is required. Thanks!

[SG]

See https://github.com/floragunncom/search-guard-helm

> Am 19.03.2019 um 00:07 schrieb apt221 <akash...@sysdig.com>:
>
> Elasticsearch Version: 5.6.14
> Search Guard Version: 5.6.14-19.2
>
>
> I am in the process of figuring out how to transfer from a single node elasticsearch cluster with SG installed to a multinode ES cluster (in kubernetes) and I would like some help understanding the process. I would like to automate the process of generating the certs required for the ES cluster. So far I have used the offline TLS tool to generate a root-ca and intermediate cert. These certs are shared to every node in the ES cluster via kubernetes secrets.
>
> 1. How do I generate the required node certificates for the cluster?
> If I had 50 nodes in my elasticsearch cluster, would that mean that in my tlsconfig.yml I would need 50 entries under the nodes config section or do I just need a minimum of 1 node certificate?

In the above mentioned helm chart we run the tls tool in an init container for every pod. In kubernetes you typically cannot generate them upfront because everything is dynamic.

>
> 2. Is it fine to simply run the sgadmin script from one node and expect the rest of the cluster to have the updated configurations?

Yes. Run sgadmin from one node (or from outside the cluster) and all nodes will be updated with the new configs without restart.
Thats called hot reload. You only need to restart nodes if youmake changes in elasticsearch.yml or if you update Search Guard or Elasticsearch version.
The above mentioned helm will perform a rolling restart in that cases so (given thereare enough nodes in the cluster) that no downtime occurs.

> I can't seem to find the post right now but I remember reading in one of the SG posts a reply that stated that you would only need to update the SG indexes from one node and the changes would be updated in all nodes (I assume after you restart all these nodes).

No restart required

See https://docs.search-guard.com/latest/sgadmin#using-sgadmin-to-configure-search-guard and https://docs.search-guard.com/latest/main-concepts#the-search-guard-index

>
> Please let me know if any more information is required. Thanks!
>

> --
> You received this message because you are subscribed to the Google Groups "Search Guard Community Forum" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.
> To post to this group, send email to search...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/1a0f026b-df99-47d7-8d6d-1deff654afdb%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[apt221]

I was thinking of using the docker entrypoint script for the elasticsearch image to create the node certificates. Would it be fine to use the same config every time? For example:

./sgtlstool.sh -c ../config/tlsconfig.yml -crt

ca:

  root:

     dn: CN=root.ca.example.com,OU=CA,O=Example Com\, Inc.,DC=example,DC=com

     keysize: 2048

     pkPassword: password

     validityDays: 3650

     file: root-ca.pem

  intermediate:

     dn: CN=signing.ca.example.com,OU=CA,O=Example Com\, Inc.,DC=example,DC=com

     keysize: 2048

     validityDays: 3650

     pkPassword: password

     file: intermediate-ca.pem

defaults:

     validityDays: 3650

     pkPassword: password

     nodeOid: "1.2.3.4.5.5"

     httpsEnabled: true

nodes:

 - name: esnode

   dn: CN=esnode.example.com,OU=Ops,O=Example Com\, Inc.,DC=example,DC=com

   dns: esnode.example.com

This way the same snippet would be appended to each node's elasticsearch config

# This is a configuration snippet for the node esnode

# This snippet needs to be inserted into the file config/elasticsearch.yml of the respective node.

# If the config file already contains SearchGuard configuration, this needs to be replaced.

# Furthermore, you need to copy the files referenced below into the same directory.

# Please refer to http://docs.search-guard.com/latest/configuring-tls for further configuration of your installation.

searchguard.ssl.transport.pemcert_filepath: esnode.pem

searchguard.ssl.transport.pemkey_filepath: esnode.key

searchguard.ssl.transport.pemkey_password: password

searchguard.ssl.transport.pemtrustedcas_filepath: root-ca.pem

searchguard.ssl.transport.enforce_hostname_verification: false

searchguard.ssl.transport.resolve_hostname: false

searchguard.ssl.http.enabled: true

searchguard.ssl.http.pemcert_filepath: esnode_http.pem

searchguard.ssl.http.pemkey_filepath: esnode_http.key

searchguard.ssl.http.pemkey_password: password

searchguard.ssl.http.pemtrustedcas_filepath: root-ca.pem

searchguard.authcz.admin_dn:

- CN=kirk.example.com,OU=Ops,O=Example Com\, Inc.,DC=example,DC=com

searchguard.cert.oid: 1.2.3.4.5.5

Also since sgadmin only needs to update the configuration index. If I pass in my custom sg_config files to the config before starting elasticsearch in my docker entrypoint will I still need to run sgadmin or will ES correctly configure both the ES config and SG config on startup?

[SG]

You can use the same node certificate for every node but in that case you cannot use hostname verification which makes TLS less secure.
And of course you must use the same root ca/signing ca for all nodes so its not possible to create a new ca on every node. So that approach will not work.

> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/91bc2644-c6d5-4d2c-a3bc-3609b56a265a%40googlegroups.com.

[apt221]

So the current plan is to generate the root/intermediate certs externally and pass them to each node when they are starting up. Then I can use these certs to generate a node cert and append the snippet to elasticsearch.yml (this is all happening in the docker entrypoint before elasticsearch is run). Is there a way to also initialize the cluster before elasticsearch has started? I'm not sure if I can run sgadmin once elasticsearch has started

[SG]

You can use the auto init feature https://docs.search-guard.com/latest/demo-installer-generated-artefacts#allow-demo-certificates-and-auto-initialization
Have also look here https://github.com/floragunncom/search-guard-helm how we deal with it.

> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/3b57d45d-5f79-438a-bbde-0c0830136f78%40googlegroups.com.

[apt221]

I was able to figure out a way to run ES and sgadmin after it has started up. The process is as follows in the docker entrypoint:

1. Copy over admin/root certs to the es config folder and the tools folder (so we can use the tools script to generate a node cert)

2. Generate node cert and append the snippet to elasticsearch.yml

3. Start running elasticsearch as a background process  (adding & at the end of the command) and record the PID

4. Run sgadmin to initialize SG and then kill the old es process and start ES again.

I had this almost working except that sgadmin would throw me an error when I tried to run it (step 4)

[2019-03-22T23:56:56,016][WARN ][com.floragunn.searchguard.auth.BackendRegistry] Transport authentication finally failed for CN=kirk.example.com,OU=Ops,O=Example Com\, Inc.,DC=example,DC=com

[2019-03-22T23:56:56,017][ERROR][com.floragunn.searchguard.transport.SearchGuardRequestHandler] Cannot authenticate null

Unable to check whether cluster is sane: None of the configured nodes are available: [{#transport#-1}{zwAs1hI5RJOvq2aeJHhBgQ}{100.97.216.234}{100.97.216.234:9300}]

Contacting elasticsearch cluster 'elasticsearch' and wait for YELLOW clusterstate ...

Cannot retrieve cluster state due to: None of the configured nodes are available: [{#transport#-1}{zwAs1hI5RJOvq2aeJHhBgQ}{100.97.216.234}{100.97.216.234:9300}]. This is not an error, will keep on trying ...

  Root cause: NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{zwAs1hI5RJOvq2aeJHhBgQ}{100.97.216.234}{100.97.216.234:9300}]] (org.elasticsearch.client.transport.NoNodeAvailableException/org.elasticsearch.client.transport.NoNodeAvailableException)

   * Try running sgadmin.sh with -icl (but no -cl) and -nhnv (If thats works you need to check your clustername as well as hostnames in your SSL certificates)

   * Make also sure that your keystore or cert is a client certificate (not a node certificate) and configured properly in elasticsearch.yml

   * If this is not working, try running sgadmin.sh with --diagnose and see diagnose trace log file)

   * Add --accept-red-cluster to allow sgadmin to operate on a red cluster.

My tlsconfig that is run to generate the node cert is the following (I don't generate admin/root since those were created externally and copied into every node)

ca:

  root:

     dn: CN=root.ca.example.com,OU=CA,O=Example Com\, Inc.,DC=example,DC=com

     keysize: 2048

     pkPassword: password

     validityDays: 3650

     file: root-ca.pem

defaults:

     validityDays: 3650

     pkPassword: password

     nodeOid: "1.2.3.4.5.5"

     httpsEnabled: true

nodes:

 - name: esnode

   dn: CN=esnode.example.com,OU=Ops,O=Example Com\, Inc.,DC=example,DC=com

   dns: esnode.example.com

When I checked the elasticsearch.yml file of the generated node i noticed that the DN was empty in the generated snippet:

searchguard.authcz.admin_dn: []

So what I did was also generate the node certificate externally and copy the same node cert into every node. I also add the same code snippet to every node and this time there is a value for the DN:

searchguard.authcz.admin_dn:
- CN=kirk.example.com,OU=Ops,O=Example Com\, Inc.,DC=example,DC=com

With this configuration sgadmin initalizes correctly and works. But I would like to know if there's a way for me to generate a node cert for each node (instead of copying the same one) and have a DN as part of snippet which appends to the config like above instead of it being '[]'

[SG]

Search Guard Community Forum - We are moving!

We are moving the Search Guard Community Forum to a new home:
https://forum.search-guard.com/

All content hosted has been migrated to the new forum.

Starting from 2019/03/30 please ask your questions on https://forum.search-guard.com/ only.

The Google Group forum will not be maintained anymore.
Thanks!

> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/69edf698-3d1a-4651-8349-dba36f75216a%40googlegroups.com.