ELK+SG migration from 5.6.6 to 6.6.1 problem on Rolling upgrade

[Przemysław Danysz]

Hi :)

* Search Guard and Elasticsearch version

- ELK 5.6.6 & search-guard-5-5.6.6-18

* Installed and used enterprise modules, if any

 dlic-search-guard-authbackend-ldap-5.6-11-jar-with-dependencies.jar

 dlic-search-guard-module-auditlog-5.3-5-jar-with-dependencies.jar

 dlic-search-guard-module-dlsfls-5.3-7-jar-with-dependencies.jar

 dlic-search-guard-module-kibana-multitenancy-5.4-5-jar-with-dependencies.jar

 dlic-search-guard-rest-api-5.3-6-jar-with-dependencies.jar

The case:

On our TEST ENV we have 4 data nodes, 2 masters and 1 coordinator.

All of them are in 5.6.6 with SG of course. We are doing a rolling migration.

 

Today we made some test, and one of data node was upgraded to 6.6.1 with correspond SG plugin.

Configuration was also adjusted as docs says, and so on.

 

So, after starting the Node, its connected to the rest of the cluster, and I can see him on the node list and Shards are available again on him. But , the problem is with authentication to this specific node. If our Logstash or scripts or anybody want to connect to “6x” version of node, the node throw errors with those logs:

 

ELK NODE version 6:

[2019-03-21T15:50:38,401][ERROR][c.f.s.a.BackendRegistry  ] [<<nodename>>] Not yet initialized (you may need to run sgadmin)

 

Logstash tried to push data to it:

2019-03-21T15:57:03,451][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://logstash:xxx...@xx.xx.xx.83:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :error=>"Got response code '503' contacting Elasticsearch at URL 'https://xx.xx.xx.83:9200/'"}

 

 

And question is What happened? Why? How can we debug it further ? 

How we can fix it and enable authentication again on this host when rest of the cluster are still in version of 5 ?

Had someone this problem also ?

Any ideas ?

thanks in advance

Przemek