Failed to push image to Harbor registry

[Qian Zhang]

I am running Kubernetes v1.15.5 cluster and I deployed Harbor 1.9.1 in the cluster via https://github.com/goharbor/harbor-helm, Harbor was exposed as a Kubernetes service (named `harbor`) of clusterIP type in the default namespace. And then I created a Kaniko pod to build and push an image to Harbor, here is the pod.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: kaniko
spec:
  containers:
  - name: kaniko
    image: gcr.io/kaniko-project/executor:latest
    args: ["--dockerfile=/workspace/Dockerfile",
           "--context=dir://workspace",
           "--destination=harbor.default/library/test-kaniko"]
    volumeMounts:
      - name: kaniko-secret
        mountPath: /root
      - name: dockerfile-storage
        mountPath: /workspace
  restartPolicy: Never
  volumes:
    - name: kaniko-secret
      secret:
        secretName: regcred
        items:
          - key: .dockerconfigjson
            path: .docker/config.json
    - name: dockerfile-storage
      persistentVolumeClaim:
        claimName: dockerfile-claim

But I found the pod failed, here is its logs:

error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: checking push permission for "harbor.default/library/test-kaniko:latest": creating push check transport for harbor.default failed: Get https://harbor.default/v2/: x509: certificate signed by unknown authority

So it seems Kaniko needs Harbor's certificate, but I do not know how to provide the certificate to Kaniko pod, is there a Kaniko command line arg for that?

To workaround the above issue, I modified Kaniko pod yaml by adding the `--skip-tls-verify` arg, this time the pod failed with a different error:

error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: checking push permission for "harbor.default/library/test-kaniko:latest": UNAUTHORIZED: authentication required; [map[Action:pull Class: Name:library/test-kaniko Type:repository] map[Action:push Class: Name:library/test-kaniko Type:repository]]

So the issue seems the authentication between Kaniko and Harbor failed, but I think I have created the secret needed by Kaniko correctly:

$ kubectl get secret regcred --output="jsonpath={.data.\.dockerconfigjson}" | base64 -d

{"auths":{"harbor.default":{"username":"admin","password":"Harbor12345","email":"ad...@harbor.com","auth":"YWRtaW46SGFyYm9yMTIzNDU="}}}

 

It seems this secret was not used by Kaniko at all?