How to configure the index in the cloud

[Xander Steinmann]

Hi all,

We have a dotCMS 5.3.8.4 with its ES index in AWS. To make it work we set the following settings in dotcms-config-cluster-ext.properties:

ES_ENDPOINTS=https://xxx.eu-central-1.es.amazonaws.com

ES_HOSTNAME=xxx.eu-central-1.es.amazonaws.com

ES_PORT=443

ES_AUTH_TYPE=NONE

ES_TLS_ENABLED=false

DIST_INDEXATION_ENABLED=false

AUTOWIRE_CLUSTER_TRANSPORT=false

Since the AWS server is in our own managed environment we don't need authentication or TLS. We can access the https URL on our server.

This all works fine until we restart dotCMS and/or redeploy the plugins. Then dotCMS can't find the index any more, or any index really, and we need to reindex to make everything work again. When looking in the database I do see the current index which exists in AWS. For some reason dotCMS can't find the existing indexes in AWS... It works fine locally with Docker or a local ES. Maybe we misconfigured the ES settings or it requires security? Any ideas?

Kind regards,

Xander

[Will Ezell]

All dotCMS elasticsearch indexes are prefixed with the DOTCMS_CLUSTER_ID for the dotCMS cluster to which they belong - that way multiple dotCMS environments can share a single elasticsearch cluster without stepping on each other.  The DOTCMS_CLUSTER_ID is generated at startup and is stored in the dot_cluster table or it can be passed in as an system or environmental variable.  You should make sure that you are not deleting the DOTCMS_CLUSTER_ID and that DOTCMS_CLUSTER_ID persists across restarts.

If that's not the case, then yeah, maybe permissions on ES?  Fire up Kibana and make sure the indexes are there.

[Xander Steinmann]

Hi Will,

Yes, you were correct. I also needed to set the DOTCMS_CLUSTER_ID and DOTCMS_CLUSTER_SALT to make it work.

One remark: if there is already something in the dot_cluster table other than the DOTCMS_CLUSTER_ID then dotCMS will keep on trying to add the DOTCMS_CLUSTER_ID and fail because of a duplicate key. After clearing the dot_cluster table I was able to stop/start dotCMS without issues. Maybe you would want to fix this in the ClusterFactory .writeClusterData() method?

Thanks!

Xander

Op dinsdag 4 mei 2021 om 17:53:40 UTC+2 schreef Will Ezell: