Fabric8 elasticsearch-cloud-kubernetes exception

[Alex T]

Hi

I am having a look at the fabric8 ElasticSearch :

https://github.com/fabric8io/elasticsearch-cloud-kubernetes

I have created a service account 'elasticsearch' and also create the template kubernetes.json in my Vagrant Fabric8 latest version.  

I can see the 3 Elasticsearch pods creates successfully, however, I am getting the following error :

[2016-02-05 14:03:31,588][INFO ][discovery                ] [Vampire by Night] elasticsearch/oZGPpiw_TlWKsnnaK-J3YA

9 [2016-02-05 14:03:32,397][WARN ][io.fabric8.elasticsearch.discovery.kubernetes.KubernetesUnicastHostsProvider] [Vampire by Night] Exception caught during discovery: Failure executing: GET at: https://kubernetes.default.svc/api/v1/namespaces/fxdev/endpoints/elasticsearch-masters. Received status: Status(apiVersion=v1, code=403, details=null, kind=Status, message=Forbidden!Configured service account doesn't have access. Service account may have been revoked., metadata=null, reason=null, status=null, additionalProperties={}). Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked..

10 io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://kubernetes.default.svc/api/v1/namespaces/fxdev/endpoints/elasticsearch-masters. Received status: Status(apiVersion=v1, code=403, details=null, kind=Status, message=Forbidden!Configured service account doesn't have access. Service account may have been revoked., metadata=null, reason=null, status=null, additionalProperties={}). Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked..

11 at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:236)

12 at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:207)

13 at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:184)

14 at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:173)

15 at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleGet(BaseOperation.java:472)

16 at io.fabric8.kubernetes.client.dsl.base.BaseOperation.get(BaseOperation.java:105)

17 at io.fabric8.elasticsearch.cloud.kubernetes.KubernetesAPIServiceImpl.endpoints(KubernetesAPIServiceImpl.java:35)

18 at 

I can see servicesaccount elasticsearch has been created :

Name:           elasticsearch

Namespace:      fxdev

Labels:         <none>

Image pull secrets:     elasticsearch-dockercfg-l55vs

Mountable secrets:      elasticsearch-token-9a5rm

                        elasticsearch-dockercfg-l55vs

Tokens:                 elasticsearch-token-1yzsb

                        elasticsearch-token-9a5rm

So, what am I missing ?  

Thanks

Alex

[Jimmi Dyson]

Hi Alex,

Are you running Kubernetes or OpenShift?

Either way, the service account needs to be able to query the API server. For openshift you need to add the appropriate role to the service account. Something like `oadm policy add-role-to-user view system:serviceaccount:<NAMESPACE>:elasticsearch` should do it.

Thanks,

Jimmi

--
You received this message because you are subscribed to the Google Groups "fabric8" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fabric8+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Alex T]

Hi Jimmi

I am using Openshift. This works ! Thanks.

As this is working now, I want my client to connect to the port 9300 using Java client as :

Client client = new TransportClient()
        .addTransportAddress(new InetSocketTransportAddress(System.getenv("ELASTICSEARCH_DISCOVERY_SERVICE_HOST"), 9300))

Which I am expecting where the  elasticsearch service  is listening. 

Deployed in Openshift, I can see the Elasticsearch IP :  

ElasticSearchClient            - ---->>>ElasticSearch Server : 172.30.4.119:9300

However, my client is getting the Exception :

Caused by: org.elasticsearch.client.transport.NoNodeAvailableException: None of the configured nodes are available: []

So, it isn't quite correct ? 

This code works nicely if I am using an Elasticsearch server outside Openshift.

Thanks

Alex

[Jimmi Dyson]

Is this running in the same namespace (project) as the Elasticsearch cluster? Is the environment variable `ELASTICSEARCH_DISCOVERY_SERVICE_HOST` set properly in the containers? You can check with `oc rsh` into your app pod & see what's going on.

[Alex T]

[Alex T]

Hi Jimmi

thanks for the reply. I did have a look at the app pod. 

These are the services :

elasticsearch             172.30.168.226                 9200/TCP

elasticsearch-masters     172.30.202.183   <none>        9300/TCP

These are the endpoints for the services :

elasticsearch             10.1.3.77:9200    9m
elasticsearch-masters     10.1.2.216:9300   9m

These are the ENV variable for the services no port 9300:

ELASTICSEARCH_MASTERS_SERVICE_HOST=172.30.202.183
ELASTICSEARCH_MASTERS_SERVICE_PORT=9300

In the app  pod,  I do see the client IP correctly targeting the ElasticSearch Master Service, but the client failed to connect to ElasticSearch :

2016-02-08 13:50:44,087 [I/O worker #10}] INFO transport - [Toad-In-Waiting] failed to get local cluster state for [#transport#-1][workmatch-8-cx2wq][inet[/172.30.202.183:9300]], disconnecting...

Any ideas ?

Thanks

Alex

[Alex T]

Got it working. My bad. I forgot to upgrade the elasticsearch client after I upgraded the ElasticSearch pods. :( 

Many Thanks

Alex