seldon-apiserver external IP is shown in pending status

[rishikesh bansod]

Hi 

      I am trying to run example given at : https://github.com/SeldonIO/seldon-core/blob/master/notebooks/kubectl_demo_gcp.ipynb  . But after running kubectl get svc -n seldon seldon-apiserver . seldon-apiserver external IP is shown in pending status. API  server is not getting allocated external IP address. 

      I have  set up kubernetes on bare metal cluster with weave net pod network. What might be the issue here?

Thanks,

Rishikesh

[Gurminder Sunner]

Hi,

Just wanted to check something.

After you "set up kubernetes on bare metal cluster", did try to create a simple deployment or pod to check that the cluster is working?

- Gurminder

[rishikesh bansod]

Hi,

    I have tried example at link https://kubernetes.io/docs/tasks/run-application/run-stateless-application-deployment/ and it is working.

[Gurminder Sunner]

Hi,

The seldon-core example you are following "https://github.com/SeldonIO/seldon-core/blob/master/notebooks/kubectl_demo_gcp.ipynb" is for GCP (Google Cloud Platform).

On bare metal its not going to automatically provision a load balancer.

Are you using the option "--set apife_service_type=LoadBalancer" for stating seldon-core as per the instructions? This is not likely to work.

Instead start seldon-core as follows

helm install helm-charts/seldon-core --name seldon-core --namespace seldon

This will give you a NodePort by deault that you can then access.

- Gurminder

On Monday, 19 March 2018 08:22:54 UTC, rishikesh bansod wrote:

[rishikesh bansod]

Hi,

      I removed load balancer option for bare-metal cluster. But how do I get API server IP address? What should I replace in place of statement : 

SELDON_API_IP=getoutput("kubectl get svc -n "+NAMESPACE+" seldon-apiserver -o jsonpath='{.status.loadBalancer.ingress[0].ip}'") .

As now there is no loadbalancer ?

Thanks,

Rishikesh

[Gurminder Sunner]

To just test that it is working, you can forward the ports locally:

Use the following in two separate terminals

kubectl port-forward -n seldon $(kubectl get pod --selector=app=seldon-apiserver-container-app -o jsonpath='{.items..metadata.name}' -n seldon) 8080:8080

kubectl port-forward -n seldon $(kubectl get pod --selector=app=seldon-apiserver-container-app -o jsonpath='{.items..metadata.name}' -n seldon) 5000:5000

Then change, SELDON_API_IP in the notebook as follows:

SELDON_API_IP="127.0.0.1"

Then try rest of notebook.

For a more permanent solution, you can use ingress.

https://kubernetes.io/docs/concepts/services-networking/ingress/

To connect the "seldon-apiserver" service to static ips.

On Monday, 19 March 2018 08:22:54 UTC, rishikesh bansod wrote:

[rishikesh bansod]

Hi,

    I have forwarded ports as you have mentioned but getting following error now when I do either rest_request() or grpc_request(). Where temp_defs1.py contains python code for requests as mentioned in the example

Traceback (most recent call last): File "temp_defs1.py", line 50, in <module> rest_request() File "temp_defs1.py", line 24, in rest_request token = get_token() File "temp_defs1.py", line 20, in get_token token = response.json()["access_token"] File "/usr/local/lib/python3.4/site-packages/requests/models.py", line 892, in json return complexjson.loads(self.text, **kwargs) File "/usr/local/lib/python3.4/json/__init__.py", line 318, in loads return _default_decoder.decode(s) File "/usr/local/lib/python3.4/json/decoder.py", line 343, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/local/lib/python3.4/json/decoder.py", line 361, in raw_decode raise ValueError(errmsg("Expecting value", s, err.value)) from None ValueError: Expecting value: line 1 column 1 (char 0)

Thanks,

Rishikesh

[Gurminder Sunner]

Did you deploy the example model using "model.json" before making the call?

[rishikesh bansod]

Hi,

      Yes I have deployed model.json  . But after running "kubectl get seldondeployments seldon-deployment-example -o jsonpath='{.status}' -n seldon" I did not get any output

Thanks,

Rishikesh

[rishikesh bansod]

I have deployed model.json. But after running "kubectl get seldondeployments seldon-deployment-example -o jsonpath='{.status}' -n seldon" I did not get any output. 

After running "kubectl describe seldondeployments seldon-deployment-example -n seldon"  I get following output :

Name:         seldon-deployment-example

Namespace:    seldon

Labels:       app=seldon

Annotations:  <none>

API Version:  machinelearning.seldon.io/v1alpha1

Kind:         SeldonDeployment

Metadata:

  Cluster Name:

  Creation Timestamp:  2018-03-19T10:54:33Z

  Resource Version:    12813

  Self Link:           /apis/machinelearning.seldon.io/v1alpha1/namespaces/seldon/seldondeployments/seldon-deployment-example

  UID:                 e8655271-2b63-11e8-be36-9418820a1de4

Spec:

  Annotations:

    Deployment _ Version:  v1

    Project _ Name:        FX Market Prediction

  Name:                    test-deployment

  Oauth _ Key:             oauth-key

  Oauth _ Secret:          oauth-secret

  Predictors:

    Annotations:

      Predictor _ Version:  v1

    Component Spec:

      Spec:

        Containers:

          Image:              seldonio/mock_classifier:1.0

          Image Pull Policy:  IfNotPresent

          Name:               classifier

          Resources:

            Requests:

              Memory:                      1Mi

        Termination Grace Period Seconds:  20

    Graph:

      Children:

      Endpoint:

        Type:  REST

      Name:    classifier

      Type:    MODEL

    Name:      fx-market-predictor

    Replicas:  1

Events:        <none>

[rishikesh bansod]

Hi, 

    above response was whn I query on master node. But when I tried to query on worker node I got following error

{'path': '/oauth/token', 'message': 'Bad credentials', 'timestamp': 1521527066875, 'status': 401, 'error': 'Unauthorized'}

Traceback (most recent call last):

  File "temp_defs1.py", line 51, in <module>

    rest_request()

  File "temp_defs1.py", line 25, in rest_request

    token = get_token()

  File "temp_defs1.py", line 21, in get_token

    token =  response.json()["access_token"]

KeyError: 'access_token'

[Gurminder Sunner]

If you could modify "temp_defs1.py" to print the value of response, that would be useful.

Currently it just says that "access_token" is not part of the response, but would be useful to know the full response.

[rishikesh bansod]

Hi

  {'path': '/oauth/token', 'message': 'Bad credentials', 'timestamp': 1521527066875, 'status': 401, 'error': 'Unauthorized'} . This is full response

[rishikesh bansod]

Hi Gurminder,

       Same error is seen for https://github.com/SeldonIO/seldon-core/blob/master/examples/models/keras_mnist/README.md example.

{"timestamp":1521552610576,"status":401,"error":"Unauthorized","message":"Bad credentials","path":"/oauth/token"}

Traceback (most recent call last):

  File "../../../util/api_tester/api-tester.py", line 202, in <module>

    run(args)

  File "../../../util/api_tester/api-tester.py", line 159, in run

    token = get_token(args)

  File "../../../util/api_tester/api-tester.py", line 136, in get_token

    token =  response.json()["access_token"]

KeyError: 'access_token'

[Clive Cox]

This looks like a oauth error. Are you sure your oauth key and secret are correct in your seldondeployment and match what you send when getting a token?

Maybe you can create an issue at https://github.com/SeldonIO/seldon-core/issues to continue the conversation?

--
You received this message because you are subscribed to the Google Groups "Seldon Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to seldon-users+unsubscribe@googlegroups.com.
To post to this group, send email to seldon...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/seldon-users/85d31121-4585-4ae8-9420-39fbfc16fa12%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.