CDAP in K8s Setup

[Arun Sundarlal]

Hi,

I'm trying to setup CDAP in K8s by following the blog: https://medium.com/cdapio/cdap-in-kubernetes-deployment-guide-7c0777dbc3a6

I'm facing issue in the CDAP Operator steps:

i don’t see “crds” dir under “cdap-operator/config/”. There is “crd” dir.

I tried the command with “crd” and got the below error

$ kubectl apply -f ./cdap-operator/config/crd

error validating “cdap-operator\\config\\crd\\kustomization.yaml”: error validating data: [apiVersion not set, kind not set]; if you choose to ignore these errors, turn validation off with — validate=false

error validating “cdap-operator\\config\\crd\\kustomizeconfig.yaml”: error validating data: [apiVersion not set, kind not set]; if you choose to ignore these errors, turn validation off with — validate=false

I tried with validate=false flag. Now i get another error.

$ kubectl apply -f ./cdap-operator/config/crd — validate=false

unable to decode “cdap-operator\\config\\crd\\kustomization.yaml”: Object ‘Kind’ is missing in ‘{“configurations”:[“kustomizeconfig.yaml”],”patchesStrategicMerge”:null,”resources”:[“bases/cdap.cdap.io_cdapmasters.yaml”]}’

unable to decode “cdap-operator\\config\\crd\\kustomizeconfig.yaml”: Object ‘Kind’ is missing in ‘{“nameReference”:[{“fieldSpecs”:[{“group”:”apiextensions.k8s.io”,”kind”:”CustomResourceDefinition”,”path”:”spec/conversion/webhookClientConfig/service/name”}],”kind”:”Service”,”version”:”v1"}],”namespace”:[{“create”:false,”group”:”apiextensions.k8s.io”,”kind”:”CustomResourceDefinition”,”path”:”spec/conversion/webhookClientConfig/service/namespace”}],”varReference”:[{“path”:”metadata/annotations”}]}’

Similarly, i don’t see “rbac” and “manager” dirs under “cdap-operator/config/default”. I can see them under “cdap-operator/config/”

Could you please help me to resolve these errors. Thanks.

[Yaroslav Kornachevskyi]

Seems that you have to go with the -k key, just like the following:

kubectl apply -k config/crd

-k stands for kustomize, see more at https://github.com/kubernetes-sigs/kustomize/blob/master/docs/INSTALL.md

[Arun Sundarlal]

Thanks a lot Yaroslav...That worked... :)

Now i tried the next 2 steps with -k flag. As i mentioned earlier, i don’t see “rbac” and “manager” dirs under “cdap-operator/config/default”. I can see them under “cdap-operator/config/”.

The cdap-operator/config/default/kustomization.yaml seems to be referring to the crd, rbac and manager dirs internally. So instead of running 3 separate commands, i tried to execute:

$ kubectl apply -k ./cdap-operator/config/default/
namespace/cdap-operator-system created
customresourcedefinition.apiextensions.k8s.io/cdapmasters.cdap.cdap.io created
role.rbac.authorization.k8s.io/cdap-operator-leader-election-role created
clusterrole.rbac.authorization.k8s.io/cdap-operator-manager-role created
clusterrole.rbac.authorization.k8s.io/cdap-operator-proxy-role created
rolebinding.rbac.authorization.k8s.io/cdap-operator-leader-election-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/cdap-operator-manager-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/cdap-operator-proxy-rolebinding created
service/cdap-operator-controller-manager-metrics-service created
deployment.apps/cdap-operator-controller-manager created

The command was successful.  Now checked the pod status and it's failing with "ImagePullBackOff" status.

$ kubectl get pod --namespace=cdap-operator-system
NAME                                              READY   STATUS             RESTARTS   AGE
cdap-operator-controller-manager-58964fdf-br28l   1/2     ImagePullBackOff   0          4m18s

On describing the pod, i found that, it's failing due to:

Failed to pull image "controller:latest": rpc error: code = Unknown desc = Error response from daemon: pull access denied for controller, repository does not exist or may require 'docker login': denied: requested access to the resource is denied

$ kubectl describe pod cdap-operator-controller-manager-58964fdf-br28l --namespace=cdap-operator-system
Name:         cdap-operator-controller-manager-58964fdf-br28l
Namespace:    cdap-operator-system
Priority:     0
Node:         minikube/192.168.99.104
Start Time:   Tue, 19 May 2020 23:22:33 +0100
Labels:       control-plane=controller-manager
              pod-template-hash=58964fdf
Annotations:  <none>
Status:       Pending
IP:           172.17.0.4
IPs:
  IP:           172.17.0.4
Controlled By:  ReplicaSet/cdap-operator-controller-manager-58964fdf
Containers:
  kube-rbac-proxy:
    Container ID:  docker://6799e52d1873b73d976bf78b8aa97b9f2d0ef57d718808cac5c0d540a0d92653
    Image:         gcr.io/kubebuilder/kube-rbac-proxy:v0.4.1
    Image ID:      docker-pullable://gcr.io/kubebuilder/kube-rbac-proxy@sha256:6c915d948d4781d366300d6e75d67a7830a941f078319f0fecc21c7744053eff
    Port:          8443/TCP
    Host Port:     0/TCP
    Args:
      --secure-listen-address=0.0.0.0:8443
      --upstream=http://127.0.0.1:8080/
      --logtostderr=true
      --v=10
    State:          Running
      Started:      Tue, 19 May 2020 23:22:41 +0100
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-6cjhb (ro)
  manager:
    Container ID:
    Image:         controller:latest
    Image ID:
    Port:          <none>
    Host Port:     <none>
    Command:
      /manager
    Args:
      --metrics-addr=127.0.0.1:8080
      --enable-leader-election
    State:          Waiting
      Reason:       ImagePullBackOff
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     100m
      memory:  30Mi
    Requests:
      cpu:        100m
      memory:     20Mi
    Environment:  <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-6cjhb (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  default-token-6cjhb:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-6cjhb
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  12m                   default-scheduler  Successfully assigned cdap-operator-system/cdap-operator-controller-manager-58964fdf-br28l to minikube
  Normal   Pulling    12m                   kubelet, minikube  Pulling image "gcr.io/kubebuilder/kube-rbac-proxy:v0.4.1"
  Normal   Pulled     12m                   kubelet, minikube  Successfully pulled image "gcr.io/kubebuilder/kube-rbac-proxy:v0.4.1"
  Normal   Created    12m                   kubelet, minikube  Created container kube-rbac-proxy
  Normal   Started    12m                   kubelet, minikube  Started container kube-rbac-proxy
  Normal   Pulling    11m (x3 over 12m)     kubelet, minikube  Pulling image "controller:latest"
  Warning  Failed     11m (x3 over 12m)     kubelet, minikube  Failed to pull image "controller:latest": rpc error: code = Unknown desc = Error response from daemon: pull access denied for controller, repository does not exist or may require 'docker login': denied: requested access to the resource is denied
  Warning  Failed     11m (x3 over 12m)     kubelet, minikube  Error: ErrImagePull
  Normal   BackOff    10m (x6 over 12m)     kubelet, minikube  Back-off pulling image "controller:latest"
  Warning  Failed     2m13s (x44 over 12m)  kubelet, minikube  Error: ImagePullBackOff

Will you be help to resolve this issue? Thanks in advance.

[b...@ros.io]

When installing on Kubernetes I had to build and push the image to an image repository myself:

git clone https://github.com/cdapio/cdap-operator
cd cdap-operator
(export IMG=<REGISTRY>/cdap-controller:latest; make docker-build && make docker-push)

In the kustomization.yaml file, add the following to update the image reference:

images:
  - name: controller
    newName: <REGISTRY>/cdap-controller
    newTag: latest # TODO

[Terence Yim]

Hi,

I've updated the blog to reflect all the latest updates to run CDAP in k8s.

Terence

--
You received this message because you are subscribed to the Google Groups "CDAP User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cdap-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cdap-user/e0696008-496b-4256-88fa-77b1e1f7c6d3%40googlegroups.com.

--

Terence Yim | Staff Software Engineer | tere...@google.com | 

[Arun Sundarlal]

Thanks a lot Terence...The latest updates in the blog works and i'm able to launch CDAP UI in minikube successfully... :)

Few observations while trying your latest updates:

I faced few issues while trying to setup CDAP using latest K8s & kubectl versions. Though the installation step  use --kubernetes-version="v1.14.10" it's not explicit. Also the old kubectl version has to be manually setup. Do we need to make a note for it?

I had the below setup initially in my laptop:

minikube v1.10.1 on Microsoft Windows 10 Home 10.0.18363 Build 18363

Kubernetes v1.18.2 on Docker 19.03.8

kubectl v1.18.0

I faced few issues while trying to install elasticsearch

$ helm install elasticsearch elastic/elasticsearch --version 6.5.3-alpha1 --set replicas=1 --set minimumMasterNodes=1 --set resources.requests.memory=500Mi

Error: unable to build kubernetes objects from release manifest: unable to recognize "": no matches for kind "StatefulSet" in version "apps/v1beta1"

Then i tried without --version tag and it worked and installed latest version (think 7.7.0). But after completing the CDAP deployment, the CDAP services failed to start with this latest version of ES.

Caused by: org.elasticsearch.client.ResponseException: method [PUT], host [http://elasticsearch-master:9200], URI [/cdap.metadata?master_timeout=30s&timeout=30s], status line [HTTP/1.1 400 Bad Request]

....

Caused by: ElasticsearchException[Elasticsearch exception [type=mapper_parsing_exception, reason=Root mapping definition has unsupported parameters:  [meta : {_meta={cdap_version=6.3.0-SNAPSHOT-1590615123435, mapping_checksum=-1901211791440276194, metadata_version=1}, properties={metadata={enabled=false}, system={copy_to=text, analyzer=text_analyzer, type=text}, hidden=

...

So i stopped this approach and then decided to try with the k8s version mentioned in your post.

$ minikube start --cpus 4 --memory 8192 --kubernetes-version="v1.14.10"

minikube v1.10.1 on Microsoft Windows 10 Home 10.0.18363 Build 18363

Kubernetes v1.14.10 on Docker 19.03.8

kubectl v1.14.10

Now the ES helm install worked with --version 6.5.3-alpha1

I faced another minor issue while creating the hdfs dir within the hadoop container.

$ kubectl exec -it hadoop -- /usr/local/hadoop/bin/hdfs dfs -mkdir /cdap

Unable to use a TTY - input is not a terminal or the right kind of file
OCI runtime exec failed: exec failed: container_linux.go:349: starting container process caused "exec: \"D:/Arun/MyLab/Git/usr/local/hadoop/bin/hdfs\": stat D:/Arun/MyLab/Git/usr/local/hadoop/bin/hdfs: no such file or directory": unknown
command terminated with exit code 126


I worked around this by launching the minikube dashboard and then using the UI exec option to launch terminal window for the hadoop container and then created the /cdap hdfs dir.

Another cosmetic one, the controller pod name was shown as "cdap-controller-0" in the blog wheres as the name was like "controller-manager-xxxx".
$ kubectl get pod --namespace=system

NAME READY STATUS RESTARTS AGE

controller-manager-588486cd7b-gqs4h 1/1 Running 0 11s

My mistake:

When i deleted the minikube instance and started all the steps from start, i skipped the cdap-controller git clone step, as i already had the clone.
But i followed the sed replace step also in a hurry which replaced the string again and created an invalid url.

$ sed -i 's/controller:latest/gcr.io\/cdapio\/cdap-controller:latest/g' cdap-operator/config/manager/manager.yaml

Finally, all the pods were running and the CDAP UI launched successfully...

$ kubectl get pods

NAME                                       READY   STATUS    RESTARTS   AGE

cdap-test-appfabric-0                      1/1     Running   0          4m17s

cdap-test-logs-0                           1/1     Running   0          4m17s

cdap-test-messaging-0                      1/1     Running   0          4m17s

cdap-test-metadata-7565474f75-p2zkg        1/1     Running   0          4m17s

cdap-test-metrics-0                        1/1     Running   0          4m17s

cdap-test-preview-0                        1/1     Running   0          4m17s

cdap-test-router-f969bcdbc-8fjhw           1/1     Running   0          4m17s

cdap-test-runtime-0                        1/1     Running   0          4m17s

cdap-test-userinterface-54bf6bcb78-rwcxg   1/1     Running   0          4m17s

elasticsearch-master-0                     1/1     Running   0          14m

hadoop                                     1/1     Running   0          11m

postgres-postgresql-0                      1/1     Running   0          15m

Thanks again for your help in resolving this issue.

To unsubscribe from this group and stop receiving emails from it, send an email to cdap...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cdap-user/e0696008-496b-4256-88fa-77b1e1f7c6d3%40googlegroups.com.