spark 2.3 and kubernetes 1.8.3 throws UnknownHostException: kubernetes.default.svc

[vignesh kennady]

Hi Team, 
I am using spark 2.3 and kubernetes 1.8.3 and intermittently spark-submit throws below error. This is causing the pod to fail but kube-dns seems to be properly configured under kube-system namespace. 

kubectl get ep kube-dns gives the endpoints IP's as well 

        at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:62) 
        at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:71) 
        at io.fabric8.kubernetes.client.dsl.base.BaseOperation.getMandatory(BaseOperation.java:228) 
        at io.fabric8.kubernetes.client.dsl.base.BaseOperation.get(BaseOperation.java:184) 
        at org.apache.spark.scheduler.cluster.k8s.KubernetesClusterSchedulerBackend.<init>(KubernetesClusterSchedulerBackend.scala:70) 
        at org.apache.spark.scheduler.cluster.k8s.KubernetesClusterManager.createSchedulerBackend(KubernetesClusterManager.scala:120) 
        at org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2741) 
        ... 12 more 
Caused by: java.net.UnknownHostException: kubernetes.default.svc: Try again 
        at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method) 
        at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928) 
        at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323) 
        at java.net.InetAddress.getAllByName0(InetAddress.java:1276) 
        at java.net.InetAddress.getAllByName(InetAddress.java:1192) 
        at java.net.InetAddress.getAllByName(InetAddress.java:1126)