coreos k8s centos nfs

[tech qi3]

I am trying to mount a nfs shared to kubernetes pod on coreos but can't get it work.

I am using:

coreos alpha 1097.0

kubernetes 1.2.4

centos 7 nfs

virtualbox 5.0.24 

my environment:

etcd cluster with 3 nodes

1 k8s master

3 k8s worker

and 1 centos nfs 

 

I can ping nfs centos server and showmount displayed all the shared directories from all nodes.  

When i describe the pod I got:

31s   31s     1       {kubelet 172.17.7.102}          Warning FailedSync      Error syncing pod, skipping: Mount failed: exit status 32

Mounting arguments: 192.168.20.101:/var/kube/mysql /var/lib/kubelet/pods/6958f240-47fa-11e6-8ca6-0800273afbf4/volumes/kubernetes.io~nfs/mysql-pv nfs []

Output: mount.nfs: access denied by server while mounting 192.168.20.101:/var/kube/mysql

Logs from kubelet: 

Jul 12 16:59:58 node-02 kubelet-wrapper[711]: E0712 08:59:58.116618     711 pod_workers.go:138] Error syncing pod 6958f240-47fa-11e6-8ca6-0800273afbf4, skipping: Mount failed: exit status 32

Jul 12 16:59:58 node-02 kubelet-wrapper[711]: Mounting arguments: 192.168.20.101:/var/kube/mysql /var/lib/kubelet/pods/6958f240-47fa-11e6-8ca6-0800273afbf4/volumes/kubernetes.io~nfs/mysql-pv nfs []

Jul 12 16:59:58 node-02 kubelet-wrapper[711]: Output: mount.nfs: access denied by server while mounting 192.168.20.101:/var/kube/mysql

Jul 12 17:04:13 node-02 kubelet-wrapper[711]: E0712 09:04:13.040337     711 kubelet.go:1780] Unable to mount volumes for pod "mysql_default(6958f240-47fa-11e6-8ca6-0800273afbf4)": Mount failed: exit status 32

Jul 12 17:04:13 node-02 kubelet-wrapper[711]: Mounting arguments: 192.168.20.101:/var/kube/mysql /var/lib/kubelet/pods/6958f240-47fa-11e6-8ca6-0800273afbf4/volumes/kubernetes.io~nfs/mysql-pv nfs []

Jul 12 17:04:13 node-02 kubelet-wrapper[711]: Output: mount.nfs: access denied by server while mounting 192.168.20.101:/var/kube/mysql

Jul 12 17:04:13 node-02 kubelet-wrapper[711]: ; skipping pod

Although the logs showed this was due to the nfs server, but I have tested my nfs server with another centos nfs-client and mount successfully. 

I tried disabling selinux and restart the nfs server and service but fail. I have double checked all the persistent volume, persistent volume claim and pod file. 

Before i proceed to mount the nfs to coreos host instead, I just wondering did i missed something? 

[Brandon Philips]

Hello-

Can you gather logs on the NFS server. Generally there is something wrong with the /etc/exports when you get this sort of error.

Cheers,

Brandon

--
You received this message because you are subscribed to the Google Groups "CoreOS User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to coreos-user...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[tech qi3]

Hi Brandon, thanks for the hints. I managed to get it work at the end, at least for the nfs part. I've set the wrong ip range in the /etc/exports, I am not sure whether it was due to virtualbox or other reason, the client ip that mount to nfs server is not worker node ip. And now everything works except when the container is mysql, it kept failing and restart new container until the pod completely delete itself. The logs from container i got: 

Initializing database
2016-07-23T18:50:51.657082Z 0 [Warning] InnoDB: New log files created, LSN=45790
2016-07-23T18:51:17.210722Z 0 [Warning] InnoDB: Retry attempts for reading partial data failed.
2016-07-23T18:51:17.210779Z 0 [ERROR] InnoDB: Tried to read 16384 bytes at offset 81920, but was only able to read 0
2016-07-23T18:51:17.210797Z 0 [ERROR] InnoDB: Operating system error number 5 in a file operation.
2016-07-23T18:51:17.213322Z 0 [ERROR] InnoDB: Error number 5 means 'Input/output error'
2016-07-23T18:51:17.213353Z 0 [ERROR] InnoDB: File (unknown): 'read' returned OS error 105. Cannot continue operation
2016-07-23T18:51:17.213359Z 0 [ERROR] InnoDB: Cannot continue operation.

The files it generated in nfs shared directory are "ibdata1  ib_logfile0  ib_logfile1  ibtmp1". I am using mysql:latest image from docker hub. Does it relate to the docker image? 

[tech qi3]

I switch the nfs host os to coreos and everything work fine now.