On-prem k8s cluster and k8s plugin

[Quang Truong]

Hi,

I start working with k8s and my working environment is the on-prem so I try to build the on-prem k8s via kubeadm and calico cni:

http://www.centinosystems.com/blog/sql/getting-started-with-kubernetes-on-prem/

https://docs.projectcalico.org/v3.7/getting-started/kubernetes/

I try to configure the service account for k8s plugin and run a simple test but not sure which part I have done wrong the 1st run (when first download the image to the cluster) the jnlp pod will use jenkins user

+ id
uid=1000(jenkins) gid=1000(jenkins) groups=1000(jenkins)
+ ls -lh /home/jenkins
total 8
drwxr-sr-x    2 jenkins  jenkins     4.0K Apr 29 11:54 agent
drwxr-xr-x    3 jenkins  jenkins     4.0K May 23 22:45 workspace

But from the second, when the image is on the machine then the pod will run with root user

+ id uid=0(root) gid=0(root) groups=0(root),1(bin),2(daemon),3(sys),4(adm),6(disk),10(wheel),11(floppy),20(dialout),26(tape),27(video) 

+ ls -lh /home/jenkins 

total 8 

drwxr-sr-x 2 jenkins jenkins 4.0K Apr 29 11:54 agent 

drwxr-xr-x 3 root root 4.0K May 23 22:45 workspace

Here is the pod describe

Name:               builder-0322cb0d-37be-438d-949c-fce0149039e5-5c743-2830b
Namespace:          default
Priority:           0
PriorityClassName:  <none>
Node:               <my_node_name>/<my_node_IP>
Start Time:         Thu, 23 May 2019 16:18:00 -0700
Labels:             jenkins=slave
                    jenkins/builder-0322cb0d-37be-438d-949c-fce0149039e5=true
Annotations:        buildUrl: <my_jenkins_build_url>
                    cni.projectcalico.org/podIP: 192.168.243.226/32
Status:             Running
IP:                 192.168.243.226
Containers:
  jnlp:
    Container ID:   docker://325602117ce4b0df6ef1d73e44ed7426251a0ea801990c065bce993e9af25cb4
    Image:          jenkins/jnlp-slave:alpine
    Image ID:       docker-pullable://jenkins/jnlp-slave@sha256:3c4227433a1bbd070b250d491bdee1696e6c163cff8a470df9c848da94306693
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Thu, 23 May 2019 16:18:02 -0700
    Ready:          True
    Restart Count:  0
    Environment:
      JENKINS_SECRET:      <my_jenkins_secret>
      JENKINS_AGENT_NAME:  builder-0322cb0d-37be-438d-949c-fce0149039e5-5c743-2830b
      JENKINS_NAME:        builder-0322cb0d-37be-438d-949c-fce0149039e5-5c743-2830b
      JENKINS_URL:         <my_jenkins_url>
      HOME:                /home/jenkins
    Mounts:
      /home/jenkins from workspace-volume (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-tmzcc (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  workspace-volume:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:    
    SizeLimit:  <unset>
  default-token-tmzcc:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-tmzcc
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  30s   default-scheduler  Successfully assigned default/builder-0322cb0d-37be-438d-949c-fce0149039e5-5c743-2830b to <>
  Normal  Pulled     28s   kubelet, ubuntu    Container image "jenkins/jnlp-slave:alpine" already present on machine
  Normal  Created    28s   kubelet, ubuntu    Created container jnlp
  Normal  Started    28s   kubelet, ubuntu    Started container jnlp

I'm not sure where the problem is, from my k8s cluster or configuration for jenkins connection, what should be the troubleshooting steps

Any comments will be helped, much appreciate on reading my issue.

Thanks,

Quang

[Ben Rich]

Have a look at https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-pod

You can assign a securityContext for the pod as such:

securityContext:
  fsGroup: 1000