storage migration check error: error="open /vault/file/core/_migration: permission denied"

548 views
Skip to first unread message

Arun Prasath

unread,
Dec 27, 2019, 3:31:37 PM12/27/19
to Vault
I facing permission denied error when deploying the vault in openshift(minishift).

what is causing or would have been the issue (any clue) for this error
how to debug and where to start to get more info on this error
what am i missing or any workaround 
any suggestion to move forward to dig more

I see below error messages in the vault pod logs.
2019-12-27T19:26:07.178Z [INFO] proxy environment: http_proxy= https_proxy= no_proxy=
2019-12-27T19:26:07.178Z [WARN] storage migration check error: error="open /vault/file/core/_migration: permission denied"
WARNING! Unable to read storage migration status.
2019-12-27T19:26:09.181Z [WARN] storage migration check error: error="open /vault/file/core/_migration: permission denied"
(truncated)

I have the below role mapping to service account "vault-auth".
oc adm policy add-cluster-role-to-user system:auth-delegator system:serviceaccount:vault-demo:vault-auth
cluster role "system:auth-delegator" added: "system:serviceaccount:vault-demo:vault-auth"

output from pvc:
~/github/hashitvault$ oc get pvc
NAME                 STATUS    VOLUME    CAPACITY   ACCESS MODES   STORAGECLASS   AGE
vault-file-backend   Bound     pv0063    100Gi      RWO,ROX,RWX                   6m
vault-logs-backend   Bound     pv0009    100Gi      RWO,ROX,RWX                   6m


My deployment.yaml file is below. 
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: vault-auth
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: vault-config
  labels:
    app: vault
data:
  vault-config: |-
    {
      "backend": {
        "file": {
          "path": "/vault/file"
        }
      }, 
      "default_lease_ttl": "168h", 
      "max_lease_ttl": "720h" , 
      "disable_mlock": true,
      "ui": true, 
      "listener": { 
        "tcp" : { 
          "address" : "0.0.0.0:8200" , 
        }
      } 
    }
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: vault-file-backend
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: vault-logs-backend
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
kind: Deployment
apiVersion: apps/v1
metadata:
  labels:
    app: vault
  name: vault
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vault
  template:
    metadata:
      labels:
        app: vault
    spec:
      serviceAccountName: vault-auth
      containers:
        name: vault
        ports:
        - containerPort: 8200
          name: vaultport
          protocol: TCP
        args:
        - server
        - -log-level=debug    
        env:
        - name: SKIP_SETCAP
          value: 'true' 
        - name: SKIP_CHOWN
          value: 'true'
        - name: VAULT_LOCAL_CONFIG
          valueFrom:
            configMapKeyRef:
              name: vault-config
              key: vault-config
        volumeMounts:      
        - name: vault-file-backend
          mountPath: /vault/file
          readOnly: false
        - name: vault-logs-backend
          mountPath: /vault/logs
          readOnly: false
          #- name: vault-cert
          #mountPath: /var/run/secrets/kubernetes.io/certs
        livenessProbe:
          httpGet:
            path: 'v1/sys/health?standbyok=true&standbycode=200&sealedcode=200&uninitcode=200'
            port: 8200
            scheme: HTTPS
        readinessProbe:
          httpGet:
            path: 'v1/sys/health?standbyok=true&standbycode=200&sealedcode=200&uninitcode=200'
            port: 8200
            scheme: HTTPS                                        
      volumes:
      - name: vault-file-backend
        persistentVolumeClaim:
          claimName: vault-file-backend
      - name: vault-logs-backend
        persistentVolumeClaim:
          claimName: vault-logs-backend
          #- name: vault-cert
          #secret:
          #secretName: vault-cert          
---
apiVersion: v1
kind: Service
metadata:
  name: vault
  annotations:
  labels:
    app: vault
spec:
  ports:
  - name: vault
    port: 8200
  selector:
    app: vault

Arun Prasath

unread,
Dec 27, 2019, 4:49:15 PM12/27/19
to Vault
I found the issue from https://github.com/minishift/minishift/issues/599
The issue is with the pv,pvc. minishift provided pvs are causing this error. 

to solve this issue, created the pv manually which solves the error. 

Raghavendra A

unread,
Jan 17, 2020, 4:54:23 AM1/17/20
to Vault
was this happened as part of SW reboot or upgrade and occasssional?

Arun Prasath

unread,
Jan 20, 2020, 12:33:36 PM1/20/20
to Vault
Not upgrade or reboot. The minishift is not having any storageclass by default to provision the storage when pvc is requested. In the vault, I need the PVC for vault storage so had PVC configuration which has created PVC but in pending state due to error with respect to storage class. Then I created storage class pointing to a path in minishift vm and then recreated the PVC which solved the issue.
Reply all
Reply to author
Forward
0 new messages