Pipeline examples

446 views
Skip to first unread message

boris.l...@lightbend.com

unread,
Dec 12, 2018, 2:43:03 PM12/12/18
to kubeflow-discuss
The majority of pipeline's examples are very GCP centric. As I am trying to run on Openshift, I can't really try them.
that should work anywhere.
Unfortunately when I run it fails (see attached screen).

The error that I am getting

This step is in Error state with this message: pods "calculation-pipeline-nt8fv-1275480913" is forbidden: unable to validate against any security context constraint: [spec.volumes[1]: Invalid value: "hostPath": hostPath volumes are not allowed to be used spec.volumes[2]: Invalid value: "hostPath": hostPath volumes are not allowed to be used]

Is an Openshift specific and can be fixed by modifying yaml. Unfortunately, I have no idea where to find  corresponding yaml file. Is there any place, where I can fix it to validate my assumption?

Additional questions:
Is there a way delete experiments/runs from the console?
I do see run and experiment, but not the pipeline itself
Finally, it is my understanding that pipelines are build on top of Argo, but when I bring up Argo UI, its completely blank
Screen Shot 2018-12-12 at 1.33.03 PM.png

Pavel Dournov

unread,
Dec 12, 2018, 6:13:11 PM12/12/18
to boris.l...@lightbend.com, Alexey Volkov, Qi-ming Jiang, Yang Pan, kubeflow...@googlegroups.com
Boris -

Thanks for trying out kubeflow pipelines.

This cell in the notebook
compiler.Compiler().compile(pipeline_func, pipeline_filename)

Creates a *gz file with the pipeline yaml. You can look into this to try to debug why it breaks. 

-Pavel



--
You received this message because you are subscribed to the Google Groups "kubeflow-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubeflow-discu...@googlegroups.com.
To post to this group, send email to kubeflow...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubeflow-discuss/650d9932-a25e-4ea9-b812-dcd3cae8058a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


--
Thanks!
-Pavel

Boris Lublinsky

unread,
Dec 12, 2018, 6:29:15 PM12/12/18
to Pavel Dournov, Alexey Volkov, Qi-ming Jiang, Yang Pan, kubeflow...@googlegroups.com
Thanks Pavel,
This is argo definition yaml.
I was looking for yaml, that creates pods, but I am guessing this is part of Argo.
What about other questions


Boris Lublinsky

unread,
Dec 12, 2018, 6:45:25 PM12/12/18
to Pavel Dournov, Alexey Volkov, Qi-ming Jiang, Yang Pan, kubeflow...@googlegroups.com
Also digging further into problems with Argo
Argo rolebinding has to be updated
On Dec 12, 2018, at 5:12 PM, Pavel Dournov <dou...@google.com> wrote:

Boris Lublinsky

unread,
Dec 12, 2018, 6:52:43 PM12/12/18
to Alexey Volkov, Pavel Dournov, Qi-ming Jiang, Yang Pan, kubeflow...@googlegroups.com
Thanks Alexey,
The attachment just had a screenshot with the same error

As for error, I am not doing any manual mounting.
At the moment I am experimenting with Argo itself, this is how I discovered Clusterrolebinding issue.
Again, are there any other not GKS specific samples, that I can try
 
On Dec 12, 2018, at 5:47 PM, Alexey Volkov <avo...@google.com> wrote:

Hello Boris.

The error you're getting is a standard Kubernetes error when you use hostPath volumes on a multi-node cluster (as hostPath only mounts the host node file system that is not shared with other nodes). hostPath works on single-node cluster installations like Minikube
I do not see any mentions of hostPath in our code. Are you mounting the volume manually?

P.S. I do not think your attachment has gotten through. Is there any other way you can share it?

Jeremy Lewi

unread,
Dec 15, 2018, 7:15:04 PM12/15/18
to Boris Lublinsky, Alexey Volkov, Pavel Dournov, Qi-ming Jiang, Yang Pan, kubeflow-discuss
Boris,

You could probably take this pipeline

And modify it to not use GCP by removing the first step that downloads a GCS file.

J

To view this discussion on the web visit https://groups.google.com/d/msgid/kubeflow-discuss/514A94A0-12EF-4E89-9334-E6C2F5628D6D%40lightbend.com.

Abhishek Gupta

unread,
Dec 15, 2018, 8:25:20 PM12/15/18
to Jeremy Lewi, Alexey Volkov, Boris Lublinsky, Pavel Dournov, Qi-ming Jiang, Yang Pan, kubeflow-discuss
Boris,

Are you trying to mount a local volume? As mentioned earlier the hostPath method won't work. You might be able to leverage the tank from 
https://github.com/kubeflow/kubeflow/blob/master/scripts/util-minikube.sh Where it creates a Persistent Volume and a corresponding claim.

If it's a distributed setup, local-storage class is not going to be helpful.

Abhishek

Boris Lublinsky

unread,
Dec 16, 2018, 5:20:41 PM12/16/18
to Abhishek Gupta, Jeremy Lewi, Alexey Volkov, Pavel Dournov, Qi-ming Jiang, Yang Pan, kubeflow-discuss
Do you guys ever look at the pod’s yaml generated by either Argo or ML Pipelines?
Here is an example:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    openshift.io/scc: privileged
    workflows.argoproj.io/node-name: pipeline-flip-coin-xlkfl.flip
      {"parameters":[{"name":"flip-output","value":"tails","valueFrom":{"path":"/tmp/output"}}],"artifacts":[{"name":"mlpipeline-ui-metadata","path":"/mlpipeline-ui-metadata.json","s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"runs/30850dfb-0180-11e9-bd47-063a66a580a8/pipeline-flip-coin-xlkfl-3596557372/mlpipeline-ui-metadata.tgz"}},{"name":"mlpipeline-metrics","path":"/mlpipeline-metrics.json","s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"runs/30850dfb-0180-11e9-bd47-063a66a580a8/pipeline-flip-coin-xlkfl-3596557372/mlpipeline-metrics.tgz"}}]}
      {"name":"flip","inputs":{},"outputs":{"parameters":[{"name":"flip-output","valueFrom":{"path":"/tmp/output"}}],"artifacts":[{"name":"mlpipeline-ui-metadata","path":"/mlpipeline-ui-metadata.json","s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"runs/30850dfb-0180-11e9-bd47-063a66a580a8/pipeline-flip-coin-xlkfl-3596557372/mlpipeline-ui-metadata.tgz"}},{"name":"mlpipeline-metrics","path":"/mlpipeline-metrics.json","s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"runs/30850dfb-0180-11e9-bd47-063a66a580a8/pipeline-flip-coin-xlkfl-3596557372/mlpipeline-metrics.tgz"}}]},"metadata":{},"container":{"name":"","image":"python:alpine3.6","command":["sh","-c"],"args":["python
      -c \"import random; result = 'heads' if random.randint(0,1) == 0 else
      'tails'; print(result)\" | tee
      /tmp/output"],"resources":{}},"archiveLocation":{}}
  creationTimestamp: '2018-12-16T22:16:09Z'
  labels:
    workflows.argoproj.io/workflow: pipeline-flip-coin-xlkfl
  name: pipeline-flip-coin-xlkfl-3596557372
  namespace: kubeflow
  ownerReferences:
    - apiVersion: argoproj.io/v1alpha1
      blockOwnerDeletion: true
      controller: true
      kind: Workflow
      name: pipeline-flip-coin-xlkfl
      uid: 30850dfb-0180-11e9-bd47-063a66a580a8
  resourceVersion: '14833825'
  selfLink: /api/v1/namespaces/kubeflow/pods/pipeline-flip-coin-xlkfl-3596557372
  uid: 309010c0-0180-11e9-ac4e-0abcca1e707a
spec:
  containers:
    - args:
        - >-
          python -c "import random; result = 'heads' if random.randint(0,1) == 0
          else 'tails'; print(result)" | tee /tmp/output
      command:
        - sh
        - '-c'
      image: 'python:alpine3.6'
      imagePullPolicy: IfNotPresent
      name: main
      resources: {}
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      volumeMounts:
        - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
          name: pipeline-runner-token-wffsv
          readOnly: true
    - args:
        - wait
      command:
        - argoexec
      env:
        - name: ARGO_POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
      image: 'argoproj/argoexec:v2.2.1'
      imagePullPolicy: IfNotPresent
      name: wait
      resources: {}
      securityContext:
        privileged: false
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      volumeMounts:
        - mountPath: /argo/podmetadata
          name: podmetadata
        - mountPath: /var/lib/docker
          name: docker-lib
          readOnly: true
        - mountPath: /var/run/docker.sock
          name: docker-sock
          readOnly: true
        - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
          name: pipeline-runner-token-wffsv
          readOnly: true
  dnsPolicy: ClusterFirst
  imagePullSecrets:
    - name: pipeline-runner-dockercfg-xpbn2
  nodeName: ip-10-0-48-147.us-east-2.compute.internal
  nodeSelector:
  restartPolicy: Never
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: pipeline-runner
  serviceAccountName: pipeline-runner
  terminationGracePeriodSeconds: 30
  volumes:
    - downwardAPI:
        defaultMode: 420
        items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.annotations
            path: annotations
      name: podmetadata
    - hostPath:
        path: /var/lib/docker
        type: Directory
      name: docker-lib
    - hostPath:
        path: /var/run/docker.sock
        type: Socket
      name: docker-sock
    - name: pipeline-runner-token-wffsv
      secret:
        defaultMode: 420
        secretName: pipeline-runner-token-wffsv
status:
  conditions:
    - lastProbeTime: null
      lastTransitionTime: '2018-12-16T22:16:09Z'
      reason: PodCompleted
      status: 'True'
      type: Initialized
    - lastProbeTime: null
      lastTransitionTime: '2018-12-16T22:16:09Z'
      reason: PodCompleted
      status: 'False'
      type: Ready
    - lastProbeTime: null
      lastTransitionTime: '2018-12-16T22:16:09Z'
      status: 'True'
      type: PodScheduled
  containerStatuses:
    - containerID: >-
      image: 'docker.io/python:alpine3.6'
      imageID: >-
      lastState: {}
      name: main
      ready: false
      restartCount: 0
      state:
        terminated:
          containerID: >-
          exitCode: 0
          finishedAt: '2018-12-16T22:16:15Z'
          reason: Completed
          startedAt: '2018-12-16T22:16:15Z'
    - containerID: >-
      imageID: >-
      lastState: {}
      name: wait
      ready: false
      restartCount: 0
      state:
        terminated:
          containerID: >-
          exitCode: 0
          finishedAt: '2018-12-16T22:16:16Z'
          reason: Completed
          startedAt: '2018-12-16T22:16:16Z'
  hostIP: 10.0.48.147
  phase: Succeeded
  podIP: 10.129.2.12
  qosClass: BestEffort
  startTime: '2018-12-16T22:16:09Z'

If you do , you will notice the usage of hostPath there.
So its not me, its Argo that is doing this.

Jeremy Lewi

unread,
Dec 17, 2018, 9:28:18 PM12/17/18
to Boris Lublinsky, Abhishek Gupta, Alexey Volkov, Pavel Dournov, Qi-ming Jiang, Yang Pan, kubeflow-discuss
Hi Boris,

thank you very much. It looks like the pod is trying to mount the docker socket. I don't know where its coming from. I suspect kubeflow/pipelines.
I filed an issue against the pipelines folks.

Sorry for the trouble.

J

Jeremy Lewi

unread,
Dec 18, 2018, 12:54:23 AM12/18/18
to Alexey Volkov, Boris Lublinsky, Abhishek Gupta, Pavel Dournov, Qi-ming Jiang, Yang Pan, kubeflow-discuss
Thanks Alexey.

I didn't realize Argo was mounting the docker socket. Does it always do that or only if you configure Argo to collect artifacts?

On Mon, Dec 17, 2018 at 8:23 PM Alexey Volkov <avo...@google.com> wrote:
Hello, Jeremy

>I don't know where its coming from. I suspect kubeflow/pipelines.
Is there a particular reason to suspect that? As I mentioned in this thread last week, "I do not see any mentions of hostPath in our code." "the error might be related to the way Argo is using to get the artifacts from the main container to the sidecar".
A cursory GitHub search reveals that Argo uses hostPath volumes in the WorkflowController https://github.com/argoproj/argo/blob/master/workflow/controller/workflowpod.go#L46 since v2.1.0 https://github.com/argoproj/argo/search?q=hostPath&unscoped_q=hostPath
// volumeDockerSock provides the wait container direct access to the minion's host docker daemon.
// The primary purpose of this is to make available `docker cp` to collect an output artifact
// from a container. Alternatively, we could use `kubectl cp`, but `docker cp` avoids the extra
// hop to the kube api server.

Fortunately, Boris has already debugged the issue and found the root cause - the way the KFP Argo installation interacts with a secure OpenShift cluster. Secure on-prem K8s clusters is one area where we might need more tests and improved support.

We might need to make the Kubeflow deployer handle the access and credentials setup for OpenShift.

Boris Lublinsky

unread,
Dec 18, 2018, 9:18:35 AM12/18/18
to Jeremy Lewi, Alexey Volkov, Abhishek Gupta, Pavel Dournov, Qi-ming Jiang, Yang Pan, kubeflow-discuss
Always

Hongye Sun

unread,
Dec 18, 2018, 1:33:59 PM12/18/18
to kubeflow-discuss
Hello Boris,

Could you relax the hostPath privilege in your cluster by following the link to workaround this issue for now?

Thanks,
Hongye
To unsubscribe from this group and stop receiving emails from it, send an email to kubeflow-discuss+unsub...@googlegroups.com.


-- 
Thanks!
-Pavel



--
You received this message because you are subscribed to the Google Groups "kubeflow-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubeflow-discu...@googlegroups.com.
To post to this group, send email to kubeflow...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubeflow-discuss/514A94A0-12EF-4E89-9334-E6C2F5628D6D%40lightbend.com.
For more options, visit https://groups.google.com/d/optout.

Boris Lublinsky

unread,
Dec 18, 2018, 1:37:34 PM12/18/18
to Hongye Sun, kubeflow-discuss
Sure, but end result is the same. You just create a semi privileged account.
The general problem is that Openshift sec is not a k8 resource, so you can’t put it into helm.
So no matter you do, you have to add additional commands to install.

Hongye Sun

unread,
Dec 18, 2018, 1:55:13 PM12/18/18
to kubeflow-discuss
Understood. This is a temporary workaround. In the meantime, we are looking into ways to get rid of this dependency (which seems to be unnecessary to have). Jeremy's bug will be used to track it.

I currently don't have a openshift cluster to test. Let us know if you encounter other problems.

Thanks,
Hongye
Reply all
Reply to author
Forward
0 new messages