Generating all required TLS certificates for SG with OpenSSL alone

[apt221]

* Search Guard Version: com.floragunn:search-guard-5:5.6.14-19.2
* Elasticsearch version: 5.6.14

* JVM version:  

openjdk version "1.8.0_191"

OpenJDK Runtime Environment (IcedTea 3.10.0) (Alpine 8.191.12-r0)

OpenJDK 64-Bit Server VM (build 25.191-b12, mixed mode)

* operating system version: Alpine Linux v3.8

I am not very familiar with OpenSSL and I wanted some clarification on the process of generating the required certs for SG. I am running a test with a single node elasticsearch cluster locally on my machine via docker. My process so far has been the following:

1. Created CA
 - located under /root/ca in the container
 - created root key/root certificate

   - openssl genrsa -aes256 -out private/ca.key.pem 4096

   - openssl req -config openssl.cnf \

      -key private/ca.key.pem \
      -new -x509 -days 7300 -sha256 -extensions v3_ca \
      -out certs/ca.cert.pem

2. Create intermediate pair

 - intermediate key/CSR

  - openssl genrsa -aes256 \

      -out intermediate/private/intermediate.key.pem 4096 

  - openssl req -config intermediate/openssl.cnf -new -sha256 \

      -key intermediate/private/intermediate.key.pem \
      -out intermediate/csr/intermediate.csr.pem

- sign certificate
 - openssl ca -config openssl.cnf -extensions v3_intermediate_ca \

      -days 3650 -notext -md sha256 \
      -in intermediate/csr/intermediate.csr.pem \
      -out intermediate/certs/intermediate.cert.pem

- create certificate chain file
 - cat intermediate/certs/intermediate.cert.pem \

      certs/ca.cert.pem > intermediate/certs/ca-chain.cert.pem

I am trying to figure out what the next step would be and what the correct approach would be to create the client/node certificates.

Once I scale from single node to multi-node I believe that I will need to pass the Root CA/Intermediate CA/Certificate Chain File to every new node so that each node certificate can be signed. 
What I am not sure about is 
1. What is the different between a client node cert and a node cert as seen in the example.sh file?

- Do I need both? In the demo installer it looks like they only use the node cert/key and not a client node cert
2. WIthout using those scripts in the example-pki-scripts directory is it possible for me to create the client node cert/node cert/admin cert with just simple OpenSSL commands? 

- I see at the bottom of the scripts that they use openssl commands but it looks like it builds off the previous keytool commands.
3. On the installation page it says the minimum requirements to install are only the TLS certificates for transport/REST, the TLS config in the ES config and initalizing the SG index. Does this include the keystore/trustore? If so, how do I create those without using the scripts.

Essentially I am trying to get SG installed and running with just using OpenSSL for all the cert generation and i'm not sure if that is possible. If any could help clear up some of my confusion help point me in the right direction through resources that would be great, thank you!

[SG]

It is possible to do this with openssl only. You need node certificates (one for each node or one for all nodes without hostname resolution which is not recommended) and at least one client certificates which serveres as admin certificate so that you can run sgadmin to initialize and configure Search Guard. They can be plain pem files or PKCS12 keystores.
The demo certificates are the minimal required set of certificates.

Please have a look here https://docs.search-guard.com/latest/tls-in-production#tls-for-production-environments and here https://github.com/floragunncom/search-guard-ssl/blob/master/example-pki-scripts/gen_node_cert_openssl.sh

The recommended way to generate the certificates is with our TLS tool: https://docs.search-guard.com/latest/offline-tls-tool

> Am 06.03.2019 um 14:49 schrieb apt221 <akash...@sysdig.com>:
>
>
> * Search Guard Version: com.floragunn:search-guard-5:5.6.14-19.2
> * Elasticsearch version: 5.6.14
> * JVM version:
>
> openjdk version "1.8.0_191"
>
>
> OpenJDK Runtime Environment (IcedTea 3.10.0) (Alpine 8.191.12-r0)
>
>
> OpenJDK 64-Bit Server VM (build 25.191-b12, mixed mode)
>
>
> * operating system version:

> NAME="Alpine Linux"
>
>
> ID=alpine
>
>
> VERSION_ID=3.8.2
>
>
> PRETTY_NAME="Alpine Linux v3.8"

> Once I scale from single node to multi-node I believe that I will need to pass the Root Certificate/CA and Intermediate cert to every new node so that each node certificate can be signed.

> What I am not sure about is
> 1. What is the different between a client node cert and a node cert as seen in the example.sh file?
> - Do I need both? In the demo installer it looks like they only use the node cert/key and not a client node cert
> 2. WIthout using those scripts in the example-pki-scripts directory is it possible for me to create the client node cert/node cert/admin cert with just simple OpenSSL commands?
> - I see at the bottom of the scripts that they use openssl commands but it looks like it builds off the previous keytool commands.
> 3. On the installation page it says the minimum requirements to install are only the TLS certificates for transport/REST, the TLS config in the ES config and initalizing the SG index. Does this include the keystore/trustore? If so, how do I create those without using the scripts.
>
> Essentially I am trying to get SG installed and running with just using OpenSSL for all the cert generation and i'm not sure if that is possible. If any could help clear up some of my confusion help point me in the right direction through resources that would be great, thank you!
>

> --
> You received this message because you are subscribed to the Google Groups "Search Guard Community Forum" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.
> To post to this group, send email to search...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/daac6347-8a55-4095-9830-dd40c15e453b%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[apt221]

I used the offline TLS tool to generate all the required certificates. 
Here is my config:

ca:                                                                                      

  root:                                                  

     dn: CN=root.ca.example.com,OU=CA,O=Example Com\, Inc.,DC=example,DC=com                                                                      

     keysize: 2048                                                          

     validityDays: 3650                                                    

     pkPassword: password

     file: root-ca.pem  

                     

defaults:                                                                                

     validityDays: 3650                                  

     pkPassword: password                                                                                          

     nodeOid: "1.2.3.4.5.5"                                                                                        

     httpsEnabled: true                                                                                                                                                                                                        

nodes:                                                                                                              

 - name: node1                                                                                                    

   dn: CN=node1.example.com,OU=Ops,O=Example Com\, Inc.,DC=example,DC=com                                          

   ip: 172.17.0.2                                                                                                  

                                                                                                                   

clients:                                                                                                            

 - name: spock                                                                                                    

   dn: CN=spock.example.com,OU=Ops,O=Example Com\, Inc.,DC=example,DC=com                                          

 - name: kirk                                                                              

   dn: CN=kirk.example.com,OU=Ops,O=Example Com\, Inc.,DC=example,DC=com                  

   admin: true                                                                            

Using the sgtlstool.sh -c ../config/tlsconfig.yml -ca -crt command I created all the required files 

bash-4.4# ls out/

client-certificates.readme              node1.key                               node1_http.key                          root-ca.pem

kirk.key                                node1.pem                               node1_http.pem                          spock.key

kirk.pem                                node1_elasticsearch_config_snippet.yml  root-ca.key                             spock.pem

I then added the config snippet to my elasticsearch.yml and also copied the node1, node1_http, root-ca and kirk files to /elasticsearch/config

Once I tried to enable shard allocation I got an error message after running the following command and I'm not sure why 

./plugins/search-guard-5/tools/sgadmin.sh --enable-shard-allocation -cert config/kirk.pem -key config/kirk.key -cacert config/root-ca.pem -keypass password
...

Unable to check whether cluster is sane: None of the configured nodes are available: [{#transport#-1}{X5vEOTP6QkiyQUilzhm_4Q}{localhost}{127.0.0.1:9300}]

ERR: Cannot connect to elasticsearch. Please refer to elasticsearch logfile for more information

Trace:

NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{X5vEOTP6QkiyQUilzhm_4Q}{localhost}{127.0.0.1:9300}]]

      at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:347)

at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:245)

at org.elasticsearch.client.transport.TransportProxyClient.execute(TransportProxyClient.java:59)

       at org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:366)

      at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:408)

    at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:397)

    at org.elasticsearch.client.support.AbstractClient$ClusterAdmin.execute(AbstractClient.java:724)

       at org.elasticsearch.client.support.AbstractClient$ClusterAdmin.updateSettings(AbstractClient.java:791)

at com.floragunn.searchguard.tools.SearchGuardAdmin.main0(SearchGuardAdmin.java:466)

       at com.floragunn.searchguard.tools.SearchGuardAdmin.main(SearchGuardAdmin.java:115)

Could this be because i am running sgadming from inside the elasticsearch docker container itself or is this a networking issue? I am able to properly access the elasticsearch api if I go to localhost:9200 on my browser but I don't know why it won't run with sgadmin.


UPDATE:

After trying with the certificates generated by the offline tool yesterday, I decided to try with the demo script/certificates just to confirm that I could get search guard running on the single node elasticsearch docker container. I disabled the node allocation shards and then installed the plugin on a brand new container and then ran the install_demo_configuration.sh file. This added the following files to my /elasticsearch/config directory: esnode-key.pem, esnode.pem, kirk-key.pem, kirk.pem and root-ca.pem. The script added all the required configurations to my elasticsearch config exept for searchguard.allow_unsafe_democertificates: true which i manually added.

http.host: 0.0.0.0

# Uncomment the following lines for a production cluster deployment

#transport.host: 0.0.0.0

#discovery.zen.minimum_master_nodes: 1

######## Start Search Guard Demo Configuration ########

# WARNING: revise all the lines below before you go into production

searchguard.allow_unsafe_democertificates: true

searchguard.ssl.transport.pemcert_filepath: esnode.pem

searchguard.ssl.transport.pemkey_filepath: esnode-key.pem

searchguard.ssl.transport.pemtrustedcas_filepath: root-ca.pem

searchguard.ssl.transport.enforce_hostname_verification: false

searchguard.ssl.http.enabled: true

searchguard.ssl.http.pemcert_filepath: esnode.pem

searchguard.ssl.http.pemkey_filepath: esnode-key.pem

searchguard.ssl.http.pemtrustedcas_filepath: root-ca.pem

searchguard.authcz.admin_dn:

 - CN=kirk,OU=client,O=client,L=test, C=de

cluster.routing.allocation.disk.threshold_enabled: false

cluster.name: searchguard_demo

network.host: 0.0.0.0

discovery.zen.minimum_master_nodes: 1

node.max_local_storage_nodes: 3

######## End Search Guard Demo Configuration ########

After this I ran the sgadmin_demo.sh script and got the following error

Unable to check whether cluster is sane: None of the configured nodes are available: [{#transport#-1}{JfHlZxdjRiCTHaoP_tU8Tw}{localhost}{127.0.0.1:9300}]

Diagnostic trace written to: /usr/share/elasticsearch/plugins/search-guard-5/tools/sgadmin_diag_trace_2019-Mar-08_22-15-21.txt

Contacting elasticsearch cluster 'elasticsearch' and wait for YELLOW clusterstate ...

Cannot retrieve cluster state due to: None of the configured nodes are available: [{#transport#-1}{JfHlZxdjRiCTHaoP_tU8Tw}{localhost}{127.0.0.1:9300}]. This is not an error, will keep on trying ...

 Root cause: NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{JfHlZxdjRiCTHaoP_tU8Tw}{localhost}{127.0.0.1:9300}]] (org.elasticsearch.client.transport.NoNodeAvailableException/org.elasticsearch.client.transport.NoNodeAvailableException)

  * Try running sgadmin.sh with -icl (but no -cl) and -nhnv (If thats works you need to check your clustername as well as hostnames in your SSL certificates)

  * Make also sure that your keystore or cert is a client certificate (not a node certificate) and configured properly in elasticsearch.yml

  * If this is not working, try running sgadmin.sh with --diagnose and see diagnose trace log file)

  * Add --accept-red-cluster to allow sgadmin to operate on a red cluster.

I am not sure why this fails now as I have not tampered with any of the configs or made any custom certificates. I simply ran the two demoscripts on a single node cluster. Is there a step that I could be missing here? I have attached the diagnose trace file to this post for more information. Thanks again!

[Search Guard]

Anything in the elasticsearch logfile?

[apt221]

My issue ended up being that I was following the docs for the latest SG while i should have been following the v5 documents (which did not require shard deallocation and only asked that I restart elasticsearch after appending the es snippet to elasticsearch.yml).