Alluxio not working on k8s

27 views
Skip to first unread message

Sushil Kumar Sah

unread,
Jun 8, 2018, 7:15:43 AM6/8/18
to Alluxio Developers
I am trying alluxio 1.7.1 with docker 1.13.1, kubernetes 1.9.6, 1.10.1

I created the alluxio docker image as per the instructions on https://www.alluxio.org/docs/1.7/en/Running-Alluxio-On-Docker.html

Then I followed the https://www.alluxio.org/docs/1.7/en/Running-Alluxio-On-Kubernetes.html guide to run alluxio on kubernetes. I was able to bring up the alluxio master pod properly, but when I try to bring up alluxio worker I get  the error that Address in use. I have not modified anything in the yamls which I downloaded from alluxio git. Only change I did was for alluxio docker image name and api version in yamls for k8s to match properly.

I checked ports being used in my k8s cluster setup, and even on the nodes also. There are no ports that alluxio wants being used by any other process, but I still get address in use error. I am unable to understand what I can do to debug further or what I should change to make this work. I don't have any other application running on my k8s cluster setup. I tried with single node k8s cluster setup and multi node k8s cluster setup also. I tried k8s version 1.9 and 1.10 also.

There is definitely some issue from alluxio worker side which I am unable to debug.

This is the log that I get from worker pod:

[root@vm-sushil-scrum1-08062018-alluxio-1 kubernetes]# kubectl logs po/alluxio-worker-knqt4
Formatting Alluxio Worker @ vm-sushil-scrum1-08062018-alluxio-1
2018-06-08 10:09:55,723 INFO  Configuration - Configuration file /opt/alluxio/conf/alluxio-site.properties loaded.
2018-06-08 10:09:55,845 INFO  Format - Formatting worker data folder: /alluxioworker/
2018-06-08 10:09:55,845 INFO  Format - Formatting Data path for tier 0:/dev/shm/alluxioworker
2018-06-08 10:09:55,856 INFO  Format - Formatting complete
2018-06-08 10:09:56,357 INFO  Configuration - Configuration file /opt/alluxio/conf/alluxio-site.properties loaded.
2018-06-08 10:09:56,549 INFO  TieredIdentityFactory - Initialized tiered identity TieredIdentity(node=10.194.11.7, rack=null)
2018-06-08 10:09:56,866 INFO  BlockWorkerFactory - Creating alluxio.worker.block.BlockWorker
2018-06-08 10:09:56,866 INFO  FileSystemWorkerFactory - Creating alluxio.worker.file.FileSystemWorker
2018-06-08 10:09:56,942 WARN  StorageTier - Failed to verify memory capacity
2018-06-08 10:09:57,082 INFO  log - Logging initialized @1160ms
2018-06-08 10:09:57,509 INFO  AlluxioWorkerProcess - Domain socket data server is enabled at /opt/domain.
Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: io.netty.channel.unix.Errors$NativeIoException: bind(..) failed: Address in use
        at alluxio.worker.AlluxioWorkerProcess.<init>(AlluxioWorkerProcess.java:164)
        at alluxio.worker.WorkerProcess$Factory.create(WorkerProcess.java:45)
        at alluxio.worker.WorkerProcess$Factory.create(WorkerProcess.java:37)
        at alluxio.worker.AlluxioWorker.main(AlluxioWorker.java:56)
Caused by: java.lang.RuntimeException: io.netty.channel.unix.Errors$NativeIoException: bind(..) failed: Address in use
        at alluxio.util.CommonUtils.createNewClassInstance(CommonUtils.java:224)
        at alluxio.worker.DataServer$Factory.create(DataServer.java:45)
        at alluxio.worker.AlluxioWorkerProcess.<init>(AlluxioWorkerProcess.java:159)
        ... 3 more
Caused by: io.netty.channel.unix.Errors$NativeIoException: bind(..) failed: Address in use
        at io.netty.channel.unix.Errors.newIOException(Errors.java:117)
        at io.netty.channel.unix.Socket.bind(Socket.java:259)
        at io.netty.channel.epoll.EpollServerDomainSocketChannel.doBind(EpollServerDomainSocketChannel.java:75)
        at io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:504)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1226)
        at io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:495)
        at io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:480)
        at io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:973)
        at io.netty.channel.AbstractChannel.bind(AbstractChannel.java:213)
        at io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:354)
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:399)
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:305)
        at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
        at java.lang.Thread.run(Thread.java:748)

-----------------------
[root@vm-sushil-scrum1-08062018-alluxio-1 kubernetes]# kubectl get all
NAME                DESIRED   CURRENT   READY     UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
ds/alluxio-worker   1         1         0         1            0           <none>          42m
ds/alluxio-worker   1         1         0         1            0           <none>          42m

NAME                          DESIRED   CURRENT   AGE
statefulsets/alluxio-master   1         1         44m

NAME                      READY     STATUS    RESTARTS   AGE
po/alluxio-master-0       1/1       Running   0          44m
po/alluxio-worker-knqt4   0/1       Error     12         42m

NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)               AGE
svc/alluxio-master   ClusterIP   None         <none>        19998/TCP,19999/TCP   44m
svc/kubernetes       ClusterIP   10.254.0.1   <none>        443/TCP               1h

---------------------

[root@vm-sushil-scrum1-08062018-alluxio-1 kubernetes]# kubectl describe po/alluxio-worker-knqt4
Name:           alluxio-worker-knqt4
Namespace:      default
Node:           vm-sushil-scrum1-08062018-alluxio-1/10.194.11.7
Start Time:     Fri, 08 Jun 2018 10:09:05 +0000
Labels:         app=alluxio
                controller-revision-hash=3081903053
                name=alluxio-worker
                pod-template-generation=1
Annotations:    <none>
Status:         Running
IP:             10.194.11.7
Controlled By:  DaemonSet/alluxio-worker
Containers:
  alluxio-worker:
    Container ID:  docker://40a1eff2cd4dff79d9189d7cb0c4826a6b6e4871fbac65221e7cdd341240e358
    Image:         alluxio:1.7.1
    Image ID:      docker://sha256:b080715bd53efc783ee5f54e7f1c451556f93e7608e60e05b4615d32702801af
    Ports:         29998/TCP, 29999/TCP, 29996/TCP
    Command:
      /entrypoint.sh
    Args:
      worker
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Fri, 08 Jun 2018 11:01:37 +0000
      Finished:     Fri, 08 Jun 2018 11:02:02 +0000
    Ready:          False
    Restart Count:  14
    Limits:
      cpu:     1
      memory:  2G
    Requests:
      cpu:     500m
      memory:  2G
    Environment Variables from:
      alluxio-config  ConfigMap  Optional: false
    Environment:
      ALLUXIO_WORKER_HOSTNAME:   (v1:status.hostIP)
    Mounts:
      /dev/shm from alluxio-ramdisk (rw)
      /opt/domain from alluxio-domain (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-7xlz7 (ro)
Conditions:
  Type           Status
  Initialized    True
  Ready          False
  PodScheduled   True
Volumes:
  alluxio-ramdisk:
    Type:    EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:  Memory
  alluxio-domain:
    Type:          HostPath (bare host directory volume)
    Path:          /tmp/domain
    HostPathType:  Directory
  default-token-7xlz7:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-7xlz7
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/disk-pressure:NoSchedule
                 node.kubernetes.io/memory-pressure:NoSchedule
                 node.kubernetes.io/not-ready:NoExecute
                 node.kubernetes.io/unreachable:NoExecute
Events:
  Type     Reason                 Age                 From                                          Message
  ----     ------                 ----                ----                                          -------
  Normal   SuccessfulMountVolume  56m                 kubelet, vm-sushil-scrum1-08062018-alluxio-1  MountVolume.SetUp succeeded for volume "alluxio-domain"
  Normal   SuccessfulMountVolume  56m                 kubelet, vm-sushil-scrum1-08062018-alluxio-1  MountVolume.SetUp succeeded for volume "alluxio-ramdisk"
  Normal   SuccessfulMountVolume  56m                 kubelet, vm-sushil-scrum1-08062018-alluxio-1  MountVolume.SetUp succeeded for volume "default-token-7xlz7"
  Normal   Pulled                 53m (x5 over 56m)   kubelet, vm-sushil-scrum1-08062018-alluxio-1  Container image "alluxio:1.7.1" already present on machine
  Normal   Created                53m (x5 over 56m)   kubelet, vm-sushil-scrum1-08062018-alluxio-1  Created container
  Normal   Started                53m (x5 over 56m)   kubelet, vm-sushil-scrum1-08062018-alluxio-1  Started container
  Warning  BackOff                1m (x222 over 55m)  kubelet, vm-sushil-scrum1-08062018-alluxio-1  Back-off restarting failed container

---------------------

[root@vm-sushil-scrum1-08062018-alluxio-1 kubernetes]# netstat -tuplena
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       User       Inode      PID/Program name
tcp        0      0 10.194.11.7:5002        0.0.0.0:*               LISTEN      0          149989     26448/docker-proxy-
tcp        0      0 10.194.11.7:10250       0.0.0.0:*               LISTEN      0          104935     19283/kubelet
tcp        0      0 127.0.0.1:10251         0.0.0.0:*               LISTEN      0          100472     17920/kube-schedule
tcp        0      0 10.194.11.7:6380        0.0.0.0:*               LISTEN      99         155827     26659/redis-server
tcp        0      0 127.0.0.1:10252         0.0.0.0:*               LISTEN      0          97624      17764/kube-controll
tcp        0      0 10.194.11.7:8879        0.0.0.0:*               LISTEN      0          143624     23758/nginx: master
tcp        0      0 10.194.11.7:10255       0.0.0.0:*               LISTEN      0          107608     19283/kubelet
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      0          12481      1/systemd
tcp        0      0 127.0.0.1:10256         0.0.0.0:*               LISTEN      0          100461     17793/kube-proxy
tcp        0      0 0.0.0.0:8081            0.0.0.0:*               LISTEN      0          143622     23758/nginx: master
tcp        0      0 0.0.0.0:53              0.0.0.0:*               LISTEN      0          156660     27123/dnsmasq
tcp        0      0 127.0.0.1:8085          0.0.0.0:*               LISTEN      0          97730      17857/kube-apiserve
tcp        0      0 127.0.0.1:60053         0.0.0.0:*               LISTEN      0          61130      11533/skydns
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      0          42466      7310/sshd
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      0          11047      1017/master
tcp        0      0 10.194.11.7:8443        0.0.0.0:*               LISTEN      0          99373      17857/kube-apiserve
tcp        0      0 10.194.11.7:16380       0.0.0.0:*               LISTEN      99         154397     26839/redis-sentine
tcp        0      0 0.0.0.0:19998           0.0.0.0:*               LISTEN      0          452707     22265/java
tcp        0      0 0.0.0.0:19999           0.0.0.0:*               LISTEN      0          451738     22265/java
tcp        0      0 10.194.11.7:49152       0.0.0.0:*               LISTEN      0          121927     22081/glusterfsd
tcp        0      0 0.0.0.0:9090            0.0.0.0:*               LISTEN      0          143623     23758/nginx: master
tcp        0      0 10.194.11.7:24007       0.0.0.0:*               LISTEN      0          114455     21997/glusterd
tcp        0      0 10.194.11.7:5000        0.0.0.0:*               LISTEN      0          143625     23758/nginx: master
tcp        0      0 127.0.0.1:10248         0.0.0.0:*               LISTEN      0          102303     19283/kubelet
tcp        0      0 10.194.11.7:5001        0.0.0.0:*               LISTEN      0          157094     27208/docker-proxy-
tcp        0      0 127.0.0.1:10249         0.0.0.0:*               LISTEN      0          100463     17793/kube-proxy
tcp        0      0 127.0.0.1:8085          127.0.0.1:51610         ESTABLISHED 0          174185     17857/kube-apiserve
tcp        0      0 127.0.0.1:49512         127.0.0.1:8085          ESTABLISHED 0          97736      17793/kube-proxy
tcp        0      0 10.194.11.7:51556       10.194.11.7:2379        ESTABLISHED 0          73319      13004/confd
tcp        0      0 127.0.0.1:49514         127.0.0.1:8085          ESTABLISHED 0          92137      17793/kube-proxy
tcp        0      0 10.194.11.7:52036       10.194.11.7:2379        ESTABLISHED 0          95747      17857/kube-apiserve
tcp        0      0 10.194.11.7:43206       10.194.11.7:2379        TIME_WAIT   0          0          -    
tcp        0      0 10.194.11.7:43088       10.194.11.7:2379        TIME_WAIT   0          0          -    
tcp        0      0 10.194.11.7:52046       10.194.11.7:2379        ESTABLISHED 0          92074      17857/kube-apiserve
tcp        0      0 127.0.0.1:57840         127.0.0.1:8085          ESTABLISHED 0          283917     17764/kube-controll
tcp        0      0 10.194.11.7:43254       10.194.11.7:2379        TIME_WAIT   0          0          -    
tcp        0      0 10.194.11.7:43242       10.194.11.7:2379        TIME_WAIT   0          0          -    
tcp        0      0 10.194.11.7:52012       10.194.11.7:2379        ESTABLISHED 0          98944      17857/kube-apiserve
tcp        0      0 10.194.11.7:8443        192.168.1.0:43213       ESTABLISHED 0          104055     17857/kube-apiserve
tcp        0      0 10.194.11.7:6380        10.194.11.7:35385       ESTABLISHED 99         154404     26659/redis-server
tcp        0      0 127.0.0.1:49658         127.0.0.1:8085          ESTABLISHED 0          100657     17764/kube-controll
tcp        0      0 127.0.0.1:41282         127.0.0.1:8085          ESTABLISHED 0          384379     17764/kube-controll
tcp        0      0 10.194.11.7:43120       10.194.11.7:2379        TIME_WAIT   0          0          -    
tcp        0      0 127.0.0.1:8085          127.0.0.1:49610         ESTABLISHED 0          99469      17857/kube-apiserve
tcp        0      0 127.0.0.1:59714         127.0.0.1:8085          ESTABLISHED 0          599303     17764/kube-controll
tcp        0      0 127.0.0.1:46484         127.0.0.1:8085          ESTABLISHED 0          449507     17764/kube-controll
tcp        0      0 127.0.0.1:8085          127.0.0.1:57998         ESTABLISHED 0          583873     17857/kube-apiserve
tcp        0      0 127.0.0.1:8085          127.0.0.1:49530         ESTABLISHED 0          92150      17857/kube-apiserve
tcp        0      0 10.194.11.7:51986       10.194.11.7:2379        ESTABLISHED 0          98941      17857/kube-apiserve
tcp        0      0 127.0.0.1:8085          127.0.0.1:41282         ESTABLISHED 0          384381     17857/kube-apiserve
tcp        0      0 127.0.0.1:49580         127.0.0.1:8085          ESTABLISHED 0          95894      17764/kube-controll
tcp        0      0 10.194.11.7:34222       10.194.11.7:8443        ESTABLISHED 0          96516      17857/kube-apiserve
tcp        0      0 10.194.11.7:51546       10.194.11.7:2379        ESTABLISHED 0          72583      13004/confd
tcp        0      0 10.194.11.7:43202       10.194.11.7:2379        TIME_WAIT   0          0          -    
tcp        0      0 127.0.0.1:44862         127.0.0.1:8085          ESTABLISHED 0          435921     17764/kube-controll
tcp        0      0 10.194.11.7:43364       10.194.11.7:2379        TIME_WAIT   0          0          -    
tcp        0      0 127.0.0.1:8085          127.0.0.1:37618         ESTABLISHED 0          350305     17857/kube-apiserve
tcp        0      0 10.194.11.7:51566       10.194.11.7:2379        ESTABLISHED 0          75882      13004/confd
tcp        0      0 127.0.0.1:8085          127.0.0.1:37900         ESTABLISHED 0          346954     17857/kube-apiserve
tcp        0      0 10.194.11.7:43208       10.194.11.7:2379        TIME_WAIT   0          0          -    
tcp        0      0 127.0.0.1:8085          127.0.0.1:45628         ESTABLISHED 0          439238     17857/kube-apiserve
tcp        0      0 10.194.11.7:52040       10.194.11.7:2379        ESTABLISHED 0          95749      17857/kube-apiserve
tcp        0      0 127.0.0.1:51610         127.0.0.1:8085          ESTABLISHED 0          168623     17764/kube-controll
tcp        0      0 10.194.11.7:52042       10.194.11.7:2379        ESTABLISHED 0          96495      17857/kube-apiserve
tcp        0      0 127.0.0.1:10252         127.0.0.1:52924         TIME_WAIT   0          0          -    
tcp        0      0 127.0.0.1:37224         127.0.0.1:8085          ESTABLISHED 0          651407     17764/kube-controll
tcp        0      0 127.0.0.1:8085          127.0.0.1:35422         ESTABLISHED 0          634065     17857/kube-apiserve
tcp        0      0 127.0.0.1:8085          127.0.0.1:49514         ESTABLISHED 0          92138      17857/kube-apiserve
tcp        0      0 10.194.11.7:43164       10.194.11.7:2379        TIME_WAIT   0          0          -    
tcp        0      0 10.194.11.7:52038       10.194.11.7:2379        ESTABLISHED 0          95748      17857/kube-apiserve
tcp        0      0 127.0.0.1:8085          127.0.0.1:37698         ESTABLISHED 0          347661     17857/kube-apiserve
tcp        0      0 127.0.0.1:49596         127.0.0.1:8085          ESTABLISHED 0          96571      17764/kube-controll
tcp        0      0 10.194.11.7:43390       10.194.11.7:2379        TIME_WAIT   0          0          -    
tcp        0      0 10.194.11.7:43114       10.194.11.7:2379        TIME_WAIT   0          0          -    
tcp        0      0 10.194.11.7:51998       10.194.11.7:2379        ESTABLISHED 0          99378      17857/kube-apiserve
tcp        0      0 127.0.0.1:8085          127.0.0.1:59714         ESTABLISHED 0          599304     17857/kube-apiserve
tcp        0      0 10.194.11.7:43112       10.194.11.7:2379        TIME_WAIT   0          0          -    
tcp        0      0 127.0.0.1:8085          127.0.0.1:43490         ESTABLISHED 0          423791     17857/kube-apiserve
tcp        0      0 10.194.11.7:52064       10.194.11.7:2379        ESTABLISHED 0          98958      17857/kube-apiserve
tcp        0      0 127.0.0.1:49524         127.0.0.1:8085          ESTABLISHED 0          100506     17920/kube-schedule
tcp        0      0 10.194.11.7:52022       10.194.11.7:2379        ESTABLISHED 0          93987      17857/kube-apiserve
tcp        0      0 127.0.0.1:8085          127.0.0.1:49618         ESTABLISHED 0          100646     17857/kube-apiserve
tcp        0      0 127.0.0.1:53774         127.0.0.1:8085          ESTABLISHED 0          245656     17764/kube-controll
tcp        0      0 10.194.11.7:52024       10.194.11.7:2379        ESTABLISHED 0          93988      17857/kube-apiserve
tcp        0      0 127.0.0.1:8085          127.0.0.1:46484         ESTABLISHED 0          454863     17857/kube-apiserve
tcp        0      0 127.0.0.1:8085          127.0.0.1:49512         ESTABLISHED 0          97737      17857/kube-apiserve
tcp        0      0 127.0.0.1:8085          127.0.0.1:49606         ESTABLISHED 0          99467      17857/kube-apiserve
tcp        0      0 10.194.11.7:24007       10.194.11.7:49151       ESTABLISHED 0          119971     21997/glusterd
tcp        0      0 127.0.0.1:58966         127.0.0.1:8085          ESTABLISHED 0          297833     17764/kube-controll
tcp        0      0 10.194.11.7:43234       10.194.11.7:2379        TIME_WAIT   0          0          -    
tcp        0      0 10.194.11.7:52074       10.194.11.7:2379        ESTABLISHED 0          92080      17857/kube-apiserve
tcp        0      0 10.194.11.7:51564       10.194.11.7:2379        ESTABLISHED 0          71013      13004/confd
tcp        0      0 127.0.0.1:33658         127.0.0.1:8085          TIME_WAIT   0          0          -    
tcp        0      0 127.0.0.1:49602         127.0.0.1:8085          ESTABLISHED 0          101494     17764/kube-controll
tcp        0      0 127.0.0.1:40456         127.0.0.1:8085          TIME_WAIT   0          0          -    
tcp        0      0 10.194.11.7:43278       10.194.11.7:2379        TIME_WAIT   0          0          -    
tcp        0      0 127.0.0.1:8085          127.0.0.1:49658         ESTABLISHED 0          100658     17857/kube-apiserve
tcp        0      0 127.0.0.1:40638         127.0.0.1:8085          TIME_WAIT   0          0          -    
tcp        0      0 10.194.11.7:51577       10.194.11.7:2379        ESTABLISHED 0          72597      13005/calico-felix
tcp        0      0 10.194.11.7:43224       10.194.11.7:2379        TIME_WAIT   0          0          -    
tcp        0      0 10.194.11.7:35814       10.194.11.7:2379        ESTABLISHED 0          316826     23849/uwsgi
tcp        0      0 127.0.0.1:8085          127.0.0.1:36752         ESTABLISHED 0          339687     17857/kube-apiserve
tcp        0      0 10.194.11.7:6380        172.17.0.3:57050        ESTABLISHED 99         158883     26659/redis-server
tcp        0      0 10.194.11.7:52086       10.194.11.7:2379        ESTABLISHED 0          92083      17857/kube-apiserve
tcp        0      0 10.194.11.7:43302       10.194.11.7:2379        TIME_WAIT   0          0          -    
tcp        0      0 127.0.0.1:10252         127.0.0.1:52984         TIME_WAIT   0          0          -    
tcp        0      0 127.0.0.1:49528         127.0.0.1:8085          ESTABLISHED 0          92146      17920/kube-schedule
tcp        0      0 10.194.11.7:52122       10.194.11.7:2379        ESTABLISHED 0          98974      17857/kube-apiserve
tcp        0      0 10.194.11.7:52026       10.194.11.7:2379        ESTABLISHED 0          95742      17857/kube-apiserve
tcp        0      0 127.0.0.1:8085          127.0.0.1:37224         ESTABLISHED 0          651408     17857/kube-apiserve
tcp        0      0 10.194.11.7:43266       10.194.11.7:2379        TIME_WAIT   0          0          -    
tcp        0      0 10.194.11.7:51992       10.194.11.7:2379        ESTABLISHED 0          100487     17857/kube-apiserve
tcp        0      0 127.0.0.1:49526         127.0.0.1:8085          ESTABLISHED 0          100507     17920/kube-schedule
tcp        0      0 10.194.11.7:49151       10.194.11.7:24007       ESTABLISHED 0          121253     22081/glusterfsd
tcp        0      0 10.194.11.7:52054       10.194.11.7:2379        ESTABLISHED 0          92078      17857/kube-apiserve
tcp        0      0 10.194.11.7:52060       10.194.11.7:2379        ESTABLISHED 0          95751      17857/kube-apiserve
tcp        0      0 127.0.0.1:8085          127.0.0.1:41404         ESTABLISHED 0          387180     17857/kube-apiserve
tcp        0      0 10.194.11.7:52078       10.194.11.7:2379        ESTABLISHED 0          92082      17857/kube-apiserve
tcp        0      0 127.0.0.1:40394         127.0.0.1:8085          TIME_WAIT   0          0          -    
tcp        0      0 10.194.11.7:51996       10.194.11.7:2379        ESTABLISHED 0          92068      17857/kube-apiserve
tcp        0      0 10.194.11.7:51538       10.194.11.7:2379        ESTABLISHED 0          75881      13004/confd
tcp        0      0 127.0.0.1:37618         127.0.0.1:8085          ESTABLISHED 0          350304     17764/kube-controll
tcp        0      0 127.0.0.1:8085          127.0.0.1:49590         ESTABLISHED 0          99459      17857/kube-apiserve
tcp        0      0 10.194.11.7:51480       10.194.11.7:2379        ESTABLISHED 0          61129      11533/skydns
tcp        0      0 10.194.11.7:39841       10.194.11.7:6380        ESTABLISHED 99         154403     26839/redis-sentine
tcp        0      0 10.194.11.7:51970       10.194.11.7:2379        ESTABLISHED 0          100479     17857/kube-apiserve
tcp        0      0 127.0.0.1:10251         127.0.0.1:47324         TIME_WAIT   0          0          -    
tcp        0      0 127.0.0.1:38706         127.0.0.1:8085          ESTABLISHED 0          665790     19283/kubelet
tcp        0      0 10.194.11.7:10250       10.194.11.7:51198       ESTABLISHED 0          389728     19283/kubelet
tcp        0      0 10.194.11.7:43174       10.194.11.7:2379        TIME_WAIT   0          0          -    
tcp        0      0 10.194.11.7:51554       10.194.11.7:2379        ESTABLISHED 0          73318      13004/confd
tcp        0      0 127.0.0.1:8085          127.0.0.1:43352         ESTABLISHED 0          426445     17857/kube-apiserve
tcp        0      0 10.194.11.7:52020       10.194.11.7:2379        ESTABLISHED 0          93986      17857/kube-apiserve
tcp        0      0 10.194.11.7:6380        10.194.11.7:39841       ESTABLISHED 99         154405     26659/redis-server
tcp        0      0 10.194.11.7:52014       10.194.11.7:2379        ESTABLISHED 0          96491      17857/kube-apiserve
tcp        0      0 10.194.11.7:43276       10.194.11.7:2379        TIME_WAIT   0          0          -    
tcp        0      0 10.194.11.7:8443        192.168.1.2:48620       ESTABLISHED 0          113268     17857/kube-apiserve
tcp        0      0 10.194.11.7:43380       10.194.11.7:2379        TIME_WAIT   0          0          -    
tcp        0      0 10.194.11.7:43374       10.194.11.7:2379        TIME_WAIT   0          0          -    
tcp        0      0 127.0.0.1:52048         127.0.0.1:8085          ESTABLISHED 0          212102     17764/kube-controll
tcp        0      0 127.0.0.1:36752         127.0.0.1:8085          ESTABLISHED 0          339686     17764/kube-controll
tcp        0      0 127.0.0.1:8085          127.0.0.1:41764         ESTABLISHED 0          391077     17857/kube-apiserve
tcp        0      0 127.0.0.1:10251         127.0.0.1:47384         TIME_WAIT   0          0          -    
tcp        0      0 10.194.11.7:43096       10.194.11.7:2379        TIME_WAIT   0          0          -    
tcp        0      0 10.194.11.7:43146       10.194.11.7:2379        TIME_WAIT   0          0          -    
tcp        0      0 10.194.11.7:43180       10.194.11.7:2379        TIME_WAIT   0          0          -    
tcp        0      0 127.0.0.1:53482         127.0.0.1:8085          ESTABLISHED 0          241659     17764/kube-controll
tcp        0      0 10.194.11.7:52082       10.194.11.7:2379        ESTABLISHED 0          95754      17857/kube-apiserve
tcp        0      0 10.194.11.7:52084       10.194.11.7:2379        ESTABLISHED 0          95755      17857/kube-apiserve
tcp        0      0 10.194.11.7:43270       10.194.11.7:2379        TIME_WAIT   0          0          -    
tcp        0      0 10.194.11.7:6380        172.17.0.4:59172        ESTABLISHED 99         159935     26659/redis-server
tcp        0      0 10.194.11.7:52016       10.194.11.7:2379        ESTABLISHED 0          93985      17857/kube-apiserve
tcp        0      0 127.0.0.1:45628         127.0.0.1:8085          ESTABLISHED 0          439237     17764/kube-controll
tcp        0      0 10.194.11.7:52030       10.194.11.7:2379        ESTABLISHED 0          95744      17857/kube-apiserve
tcp        0      0 10.194.11.7:49152       10.194.11.7:49148       ESTABLISHED 0          121982     22081/glusterfsd
tcp        0      0 127.0.0.1:49618         127.0.0.1:8085          ESTABLISHED 0          94094      17764/kube-controll
tcp        0      0 127.0.0.1:37900         127.0.0.1:8085          ESTABLISHED 0          350562     17764/kube-controll
tcp        0      0 10.194.11.7:51984       10.194.11.7:2379        ESTABLISHED 0          100486     17857/kube-apiserve
tcp        0      0 127.0.0.1:49648         127.0.0.1:8085          ESTABLISHED 0          99473      17764/kube-controll
tcp        0      0 127.0.0.1:8085          127.0.0.1:53774         ESTABLISHED 0          245657     17857/kube-apiserve
tcp        0      0 10.194.11.7:52066       10.194.11.7:2379        ESTABLISHED 0          100491     17857/kube-apiserve
tcp        0      0 10.194.11.7:43148       10.194.11.7:2379        TIME_WAIT   0          0          -    
tcp        0      0 10.194.11.7:43306       10.194.11.7:2379        TIME_WAIT   0          0          -    
tcp        0      0 127.0.0.1:57998         127.0.0.1:8085          ESTABLISHED 0          583872     17764/kube-controll
tcp        0      0 127.0.0.1:43490         127.0.0.1:8085          ESTABLISHED 0          428316     17764/kube-controll
tcp        0      0 127.0.0.1:8085          127.0.0.1:49592         ESTABLISHED 0          99460      17857/kube-apiserve
tcp        0      0 10.194.11.7:43182       10.194.11.7:2379        TIME_WAIT   0          0          -    
tcp        0      0 127.0.0.1:8085          127.0.0.1:49578         ESTABLISHED 0          100637     17857/kube-apiserve
tcp        0      0 127.0.0.1:49516         127.0.0.1:8085          ESTABLISHED 0          99028      17920/kube-schedule
tcp        0      0 10.194.11.7:43338       10.194.11.7:2379        TIME_WAIT   0          0          -    
tcp        0      0 10.194.11.7:51198       10.194.11.7:10250       ESTABLISHED 0          392377     17857/kube-apiserve
tcp        0      0 10.194.11.7:51580       10.194.11.7:2379        ESTABLISHED 0          73323      13005/calico-felix
tcp        0      0 127.0.0.1:8085          127.0.0.1:40224         ESTABLISHED 0          373511     17857/kube-apiserve
tcp        0      0 127.0.0.1:37698         127.0.0.1:8085          ESTABLISHED 0          352270     17764/kube-controll
tcp        0      0 127.0.0.1:8085          127.0.0.1:49626         ESTABLISHED 0          96575      17857/kube-apiserve
tcp        0      0 127.0.0.1:49522         127.0.0.1:8085          ESTABLISHED 0          92141      17920/kube-schedule
tcp        0      0 10.194.11.7:10255       192.168.1.1:38392       ESTABLISHED 0          114969     19283/kubelet
tcp        0      0 10.194.11.7:43320       10.194.11.7:2379        TIME_WAIT   0          0          -    
tcp        0      0 127.0.0.1:10251         127.0.0.1:47498         TIME_WAIT   0          0          -    
tcp        0      0 10.194.11.7:52080       10.194.11.7:2379        ESTABLISHED 0          95753      17857/kube-apiserve
tcp        0      0 10.194.11.7:22          10.194.11.6:43312       ESTABLISHED 0          345299     3986/sshd: cloud-us
tcp        0      0 10.194.11.7:43130       10.194.11.7:2379        TIME_WAIT   0          0          -    
tcp        0      0 127.0.0.1:53002         127.0.0.1:8085          ESTABLISHED 0          532077     17764/kube-controll
tcp        0      0 127.0.0.1:49532         127.0.0.1:8085          ESTABLISHED 0          92152      17920/kube-schedule
tcp        0      0 127.0.0.1:38196         127.0.0.1:8085          ESTABLISHED 0          656149     19283/kubelet
tcp        0      0 127.0.0.1:40546         127.0.0.1:8085          TIME_WAIT   0          0          -    
tcp        0      0 10.194.11.7:8443        10.194.11.7:34222       ESTABLISHED 0          94044      17857/kube-apiserve
tcp        0      0 127.0.0.1:36838         127.0.0.1:8085          ESTABLISHED 0          339759     17764/kube-controll
tcp        0      0 10.194.11.7:51536       10.194.11.7:2379        ESTABLISHED 0          72581      13004/confd
tcp        0      0 127.0.0.1:40224         127.0.0.1:8085          ESTABLISHED 0          377277     17764/kube-controll
tcp        0      0 127.0.0.1:49590         127.0.0.1:8085          ESTABLISHED 0          96569      17764/kube-controll
tcp        0      0 127.0.0.1:40516         127.0.0.1:8085          TIME_WAIT   0          0          -    
tcp        0      0 127.0.0.1:8085          127.0.0.1:49528         ESTABLISHED 0          92147      17857/kube-apiserve
tcp        0      0 127.0.0.1:8085          127.0.0.1:49624         ESTABLISHED 0          100649     17857/kube-apiserve
tcp        0      0 10.194.11.7:52058       10.194.11.7:2379        ESTABLISHED 0          95750      17857/kube-apiserve
tcp        0      0 10.194.11.7:51978       10.194.11.7:2379        ESTABLISHED 0          100480     17857/kube-apiserve
tcp        0      0 10.194.11.7:43140       10.194.11.7:2379        TIME_WAIT   0          0          -    
tcp        0      0 10.194.11.7:43098       10.194.11.7:2379        TIME_WAIT   0          0          -    
tcp        0      0 10.194.11.7:43262       10.194.11.7:2379        TIME_WAIT   0          0          -    
tcp        0      0 10.194.11.7:43156       10.194.11.7:2379        TIME_WAIT   0          0          -    
tcp        0      0 10.194.11.7:51994       10.194.11.7:2379        ESTABLISHED 0          92067      17857/kube-apiserve
tcp        0      0 10.194.11.7:52056       10.194.11.7:2379        ESTABLISHED 0          92079      17857/kube-apiserve
tcp        0      0 10.194.11.7:6380        172.17.0.4:59178        ESTABLISHED 99         159938     26659/redis-server
tcp        0      0 10.194.11.7:52034       10.194.11.7:2379        ESTABLISHED 0          95746      17857/kube-apiserve
tcp        0      0 10.194.11.7:52044       10.194.11.7:2379        ESTABLISHED 0          96496      17857/kube-apiserve
tcp        0      0 10.194.11.7:51982       10.194.11.7:2379        ESTABLISHED 0          100485     17857/kube-apiserve
tcp        0      0 127.0.0.1:8085          127.0.0.1:41280         ESTABLISHED 0          384380     17857/kube-apiserve
tcp        0      0 10.194.11.7:43218       10.194.11.7:2379        TIME_WAIT   0          0          -    
tcp        0      0 10.194.11.7:43314       10.194.11.7:2379        TIME_WAIT   0          0          -    
tcp        0      0 127.0.0.1:40486         127.0.0.1:8085          TIME_WAIT   0          0          -    
tcp        0      0 127.0.0.1:49606         127.0.0.1:8085          ESTABLISHED 0          101496     17764/kube-controll
tcp        0      0 10.194.11.7:43118       10.194.11.7:2379        TIME_WAIT   0          0          -    
tcp        0      0 10.194.11.7:49150       10.194.11.7:24007       ESTABLISHED 0          121981     22230/glusterfs
tcp        0      0 10.194.11.7:43200       10.194.11.7:2379        TIME_WAIT   0          0          -    
tcp        0      0 10.194.11.7:52048       10.194.11.7:2379        ESTABLISHED 0          92075      17857/kube-apiserve
tcp        0      0 10.194.11.7:43240       10.194.11.7:2379        TIME_WAIT   0          0          -    
tcp        0      0 10.194.11.7:43216       10.194.11.7:2379        TIME_WAIT   0          0          -    
tcp        0      0 127.0.0.1:10252         127.0.0.1:52804         TIME_WAIT   0          0          -    
tcp        0      0 127.0.0.1:8085          127.0.0.1:36362         ESTABLISHED 0          641716     17857/kube-apiserve
tcp        0      0 127.0.0.1:8085          127.0.0.1:54290         ESTABLISHED 0          261185     17857/kube-apiserve
tcp        0      0 10.194.11.7:52008       10.194.11.7:2379        ESTABLISHED 0          97680      17857/kube-apiserve
tcp        0      0 10.194.11.7:52018       10.194.11.7:2379        ESTABLISHED 0          96492      17857/kube-apiserve
tcp        0      0 10.194.11.7:6380        172.17.0.4:59176        ESTABLISHED 99         159937     26659/redis-server
tcp        0      0 127.0.0.1:8085          127.0.0.1:38706         ESTABLISHED 0          664785     17857/kube-apiserve
tcp        0      0 10.194.11.7:43260       10.194.11.7:2379        TIME_WAIT   0          0          -    
tcp        0      0 10.194.11.7:43388       10.194.11.7:2379        TIME_WAIT   0          0          -    
tcp        0      0 127.0.0.1:8085          127.0.0.1:49580         ESTABLISHED 0          100638     17857/kube-apiserve
tcp        0      0 127.0.0.1:10252         127.0.0.1:52754         TIME_WAIT   0          0          -    
tcp        0      0 127.0.0.1:41764         127.0.0.1:8085          ESTABLISHED 0          391076     17764/kube-controll
tcp        0      0 127.0.0.1:40422         127.0.0.1:8085          TIME_WAIT   0          0          -    
tcp        0      0 127.0.0.1:49518         127.0.0.1:8085          ESTABLISHED 0          92140      17920/kube-schedule
tcp        0      0 127.0.0.1:49592         127.0.0.1:8085          ESTABLISHED 0          101492     17764/kube-controll
tcp        0      0 10.194.11.7:43284       10.194.11.7:2379        TIME_WAIT   0          0          -    
tcp        0      0 127.0.0.1:36362         127.0.0.1:8085          ESTABLISHED 0          634794     19283/kubelet
tcp        0      0 127.0.0.1:54290         127.0.0.1:8085          ESTABLISHED 0          261184     17764/kube-controll
tcp        0      0 127.0.0.1:8085          127.0.0.1:49596         ESTABLISHED 0          99462      17857/kube-apiserve
tcp        0      0 10.194.11.7:52004       10.194.11.7:2379        ESTABLISHED 0          95741      17857/kube-apiserve
tcp        0      0 10.194.11.7:52068       10.194.11.7:2379        ESTABLISHED 0          98959      17857/kube-apiserve
tcp        0      0 127.0.0.1:49796         127.0.0.1:8085          ESTABLISHED 0          109679     17920/kube-schedule
tcp        0      0 10.194.11.7:24007       10.194.11.7:49150       ESTABLISHED 0          139084     21997/glusterd
tcp        0      0 127.0.0.1:49520         127.0.0.1:8085          ESTABLISHED 0          99031      17920/kube-schedule
tcp        0      0 127.0.0.1:49530         127.0.0.1:8085          ESTABLISHED 0          92149      17920/kube-schedule
tcp        0      0 127.0.0.1:49664         127.0.0.1:8085          ESTABLISHED 0          100663     17764/kube-controll
tcp        0      0 10.194.11.7:52002       10.194.11.7:2379        ESTABLISHED 0          99379      17857/kube-apiserve
tcp        0      0 10.194.11.7:43136       10.194.11.7:2379        TIME_WAIT   0          0          -    
tcp        0      0 127.0.0.1:43352         127.0.0.1:8085          ESTABLISHED 0          428141     17764/kube-controll
tcp        0      0 10.194.11.7:51980       10.194.11.7:2379        ESTABLISHED 0          100481     17857/kube-apiserve
tcp        0      0 10.194.11.7:52006       10.194.11.7:2379        ESTABLISHED 0          97679      17857/kube-apiserve
tcp        0      0 10.194.11.7:43294       10.194.11.7:2379        TIME_WAIT   0          0          -    
tcp        0      0 127.0.0.1:35422         127.0.0.1:8085          ESTABLISHED 0          635152     19283/kubelet
tcp        0      0 10.194.11.7:52010       10.194.11.7:2379        ESTABLISHED 0          93981      17857/kube-apiserve
tcp        0      0 10.194.11.7:43086       10.194.11.7:2379        TIME_WAIT   0          0          -    
tcp        0      0 10.194.11.7:51968       10.194.11.7:2379        ESTABLISHED 0          100478     17857/kube-apiserve
tcp        0      0 127.0.0.1:8085          127.0.0.1:49664         ESTABLISHED 0          96581      17857/kube-apiserve
tcp        0      0 127.0.0.1:8085          127.0.0.1:58966         ESTABLISHED 0          297834     17857/kube-apiserve
tcp        0      0 10.194.11.7:8443        192.168.1.1:41190       ESTABLISHED 0          111471     17857/kube-apiserve
tcp        0      0 10.194.11.7:51990       10.194.11.7:2379        ESTABLISHED 0          98943      17857/kube-apiserve
tcp        0      0 10.194.11.7:51540       10.194.11.7:2379        ESTABLISHED 0          72582      13004/confd
tcp        0      0 10.194.11.7:51988       10.194.11.7:2379        ESTABLISHED 0          98942      17857/kube-apiserve
tcp        0      0 10.194.11.7:16380       172.17.0.3:38514        ESTABLISHED 99         156638     26839/redis-sentine
tcp        0      0 127.0.0.1:10251         127.0.0.1:47264         TIME_WAIT   0          0          -    
tcp        0      0 127.0.0.1:8085          127.0.0.1:49796         ESTABLISHED 0          105122     17857/kube-apiserve
tcp        0      0 10.194.11.7:43322       10.194.11.7:2379        TIME_WAIT   0          0          -    
tcp        0      0 127.0.0.1:40666         127.0.0.1:8085          TIME_WAIT   0          0          -    
tcp        0      0 10.194.11.7:51544       10.194.11.7:2379        ESTABLISHED 0          69600      13004/confd
tcp        0      0 127.0.0.1:8085          127.0.0.1:52048         ESTABLISHED 0          212103     17857/kube-apiserve
tcp        0      0 127.0.0.1:8085          127.0.0.1:49522         ESTABLISHED 0          92142      17857/kube-apiserve
tcp        0      0 127.0.0.1:49626         127.0.0.1:8085          ESTABLISHED 0          94098      17764/kube-controll
tcp        0      0 10.194.11.7:52028       10.194.11.7:2379        ESTABLISHED 0          95743      17857/kube-apiserve
tcp        0      0 10.194.11.7:43366       10.194.11.7:2379        TIME_WAIT   0          0          -    
tcp        0      0 10.194.11.7:43194       10.194.11.7:2379        TIME_WAIT   0          0          -    
tcp        0      0 127.0.0.1:8085          127.0.0.1:49648         ESTABLISHED 0          97783      17857/kube-apiserve
tcp        0      0 10.194.11.7:43336       10.194.11.7:2379        TIME_WAIT   0          0          -    
tcp        0      0 10.194.11.7:43328       10.194.11.7:2379        TIME_WAIT   0          0          -    
tcp        0      0 127.0.0.1:41404         127.0.0.1:8085          ESTABLISHED 0          385467     17764/kube-controll
tcp        0      0 10.194.11.7:52000       10.194.11.7:2379        ESTABLISHED 0          92069      17857/kube-apiserve
tcp        0      0 10.194.11.7:51548       10.194.11.7:2379        ESTABLISHED 0          73316      13004/confd
tcp        0      0 10.194.11.7:51526       10.194.11.7:2379        ESTABLISHED 0          73313      13005/calico-felix
tcp        0      0 10.194.11.7:51974       10.194.11.7:2379        ESTABLISHED 0          95738      17857/kube-apiserve
tcp        0      0 10.194.11.7:52062       10.194.11.7:2379        ESTABLISHED 0          95752      17857/kube-apiserve
tcp        0      0 10.194.11.7:43232       10.194.11.7:2379        TIME_WAIT   0          0          -    
tcp        0      0 10.194.11.7:43382       10.194.11.7:2379        TIME_WAIT   0          0          -    
tcp        0      0 127.0.0.1:8085          127.0.0.1:49524         ESTABLISHED 0          92144      17857/kube-apiserve
tcp        0      0 10.194.11.7:43352       10.194.11.7:2379        TIME_WAIT   0          0          -    
tcp        0      0 127.0.0.1:8085          127.0.0.1:57840         ESTABLISHED 0          283918     17857/kube-apiserve
tcp        0      0 10.194.11.7:52052       10.194.11.7:2379        ESTABLISHED 0          92077      17857/kube-apiserve
tcp        0      0 127.0.0.1:8085          127.0.0.1:49532         ESTABLISHED 0          92153      17857/kube-apiserve
tcp        0      0 127.0.0.1:8085          127.0.0.1:49518         ESTABLISHED 0          95804      17857/kube-apiserve
tcp        0      0 10.194.11.7:51972       10.194.11.7:2379        ESTABLISHED 0          96479      17857/kube-apiserve
tcp        0      0 10.194.11.7:43354       10.194.11.7:2379        TIME_WAIT   0          0          -    
tcp        0      0 127.0.0.1:8085          127.0.0.1:36838         ESTABLISHED 0          339760     17857/kube-apiserve
tcp        0      0 10.194.11.7:35385       10.194.11.7:6380        ESTABLISHED 99         154402     26839/redis-sentine
tcp        0      0 127.0.0.1:8085          127.0.0.1:49516         ESTABLISHED 0          99029      17857/kube-apiserve
tcp        0      0 10.194.11.7:43330       10.194.11.7:2379        TIME_WAIT   0          0          -    
tcp        0      0 127.0.0.1:10251         127.0.0.1:47444         TIME_WAIT   0          0          -    
tcp        0      0 127.0.0.1:8085          127.0.0.1:49602         ESTABLISHED 0          99465      17857/kube-apiserve
tcp        0      0 127.0.0.1:49598         127.0.0.1:8085          ESTABLISHED 0          100643     17764/kube-controll
tcp        0      0 127.0.0.1:10252         127.0.0.1:52864         TIME_WAIT   0          0          -    
tcp        0      0 10.194.11.7:43292       10.194.11.7:2379        TIME_WAIT   0          0          -    
tcp        0      0 10.194.11.7:52050       10.194.11.7:2379        ESTABLISHED 0          92076      17857/kube-apiserve
tcp        0      0 127.0.0.1:8085          127.0.0.1:49598         ESTABLISHED 0          99463      17857/kube-apiserve
tcp        0      0 10.194.11.7:35702       10.194.11.7:2379        ESTABLISHED 0          314132     23850/uwsgi
tcp        0      0 127.0.0.1:49578         127.0.0.1:8085          ESTABLISHED 0          101491     17764/kube-controll
tcp        0      0 10.194.11.7:52072       10.194.11.7:2379        ESTABLISHED 0          98961      17857/kube-apiserve
tcp        0      0 10.194.11.7:51976       10.194.11.7:2379        ESTABLISHED 0          97678      17857/kube-apiserve
tcp        0      0 127.0.0.1:41280         127.0.0.1:8085          ESTABLISHED 0          382722     17764/kube-controll
tcp        0      0 127.0.0.1:45640         127.0.0.1:8085          ESTABLISHED 0          441092     17764/kube-controll
tcp        0      0 127.0.0.1:8085          127.0.0.1:38196         ESTABLISHED 0          657211     17857/kube-apiserve
tcp        0      0 127.0.0.1:49610         127.0.0.1:8085          ESTABLISHED 0          101498     17764/kube-controll
tcp        0      0 127.0.0.1:8085          127.0.0.1:53002         ESTABLISHED 0          533051     17857/kube-apiserve
tcp        0      0 127.0.0.1:8085          127.0.0.1:45640         ESTABLISHED 0          445073     17857/kube-apiserve
tcp        0      0 10.194.11.7:49148       10.194.11.7:49152       ESTABLISHED 0          113410     22230/glusterfs
tcp        0      0 10.194.11.7:43346       10.194.11.7:2379        TIME_WAIT   0          0          -    
tcp        0      0 127.0.0.1:8085          127.0.0.1:53482         ESTABLISHED 0          241660     17857/kube-apiserve
tcp        0      0 10.194.11.7:52032       10.194.11.7:2379        ESTABLISHED 0          95745      17857/kube-apiserve
tcp        0      0 10.194.11.7:6380        172.17.0.4:36110        ESTABLISHED 99         264809     26659/redis-server
tcp        0      0 10.194.11.7:43158       10.194.11.7:2379        TIME_WAIT   0          0          -    
tcp        0      0 127.0.0.1:8085          127.0.0.1:49526         ESTABLISHED 0          100508     17857/kube-apiserve
tcp        0      0 127.0.0.1:40576         127.0.0.1:8085          TIME_WAIT   0          0          -    
tcp        0      0 10.194.11.7:51558       10.194.11.7:2379        ESTABLISHED 0          72585      13004/confd
tcp        0      0 10.194.11.7:43102       10.194.11.7:2379        TIME_WAIT   0          0          -    
tcp        0      0 10.194.11.7:43172       10.194.11.7:2379        TIME_WAIT   0          0          -    
tcp        0      0 10.194.11.7:52076       10.194.11.7:2379        ESTABLISHED 0          92081      17857/kube-apiserve
tcp        0      0 127.0.0.1:8085          127.0.0.1:49520         ESTABLISHED 0          95806      17857/kube-apiserve
tcp        0      0 127.0.0.1:8085          127.0.0.1:44862         ESTABLISHED 0          434789     17857/kube-apiserve
tcp        0      0 10.194.11.7:52070       10.194.11.7:2379        ESTABLISHED 0          98960      17857/kube-apiserve
tcp        0      0 172.17.0.1:48140        172.17.0.2:2379         ESTABLISHED 0          157044     10751/docker-proxy-
tcp        0      0 127.0.0.1:40606         127.0.0.1:8085          TIME_WAIT   0          0          -    
tcp        0      0 127.0.0.1:49624         127.0.0.1:8085          ESTABLISHED 0          94097      17764/kube-controll
tcp6       0      0 :::2379                 :::*                    LISTEN      0          51143      10751/docker-proxy-
tcp6       0      0 :::2380                 :::*                    LISTEN      0          51131      10739/docker-proxy-
tcp6       0      0 :::111                  :::*                    LISTEN      0          12480      1/systemd
tcp6       0      0 :::53                   :::*                    LISTEN      0          156662     27123/dnsmasq
tcp6       0      0 :::22                   :::*                    LISTEN      0          42464      7310/sshd
tcp6       0      0 ::1:25                  :::*                    LISTEN      0          11048      1017/master
tcp6       0      0 10.194.11.7:2379        172.17.0.3:53392        ESTABLISHED 0          157042     10751/docker-proxy-
udp        0      0 127.0.0.1:60053         0.0.0.0:*                           0          70013      11533/skydns
udp        0      0 0.0.0.0:49033           0.0.0.0:*                           0          10858      701/dhclient
udp        0      0 0.0.0.0:53              0.0.0.0:*                           0          156659     27123/dnsmasq
udp        0      0 0.0.0.0:68              0.0.0.0:*                           0          10868      701/dhclient
udp        0      0 0.0.0.0:111             0.0.0.0:*                           0          116733     21995/rpcbind
udp        0      0 0.0.0.0:123             0.0.0.0:*                           0          44810      8231/chronyd
udp        0      0 127.0.0.1:323           0.0.0.0:*                           0          44807      8231/chronyd
udp        0      0 0.0.0.0:970             0.0.0.0:*                           0          116734     21995/rpcbind
udp6       0      0 :::53                   :::*                                0          156661     27123/dnsmasq
udp6       0      0 :::111                  :::*                                0          116735     21995/rpcbind
udp6       0      0 :::123                  :::*                                0          44811      8231/chronyd
udp6       0      0 :::61685                :::*                                0          10859      701/dhclient
udp6       0      0 ::1:323                 :::*                                0          44808      8231/chronyd
udp6       0      0 :::970                  :::*                                0          116736     21995/rpcbind

--------------------

The alluxio-worker container is always restarting and failing again and again for the same error.

Please guide me what I can do to solve this.

Thanks


Sushil Kumar Sah

unread,
Jun 12, 2018, 4:57:36 AM6/12/18
to Alluxio Developers
The problem was short circuit domain socket path. I was using whatever was present by default in alluxio git. In the default integration/kubernetes/conf/alluxio.properties.template the address for ALLUXIO_WORKER_DATA_SERVER_DOMAIN_SOCKET_ADDRESS was not complete. This is properly explained in https://www.alluxio.org/docs/1.7/en/Running-Alluxio-On-Docker.html for enabling short circuit reads in alluxio worker containers using unix domain sockets.
Just because of a missing complete path for unix domain socket the alluxio worker was not able to come up in kubernetes when short circuit read was enabled for alluxio worker.

When I corrected the path in integration/kubernetes/conf/alluxio.properties for ALLUXIO_WORKER_DATA_SERVER_DOMAIN_SOCKET_ADDRESS=/opt/domain/d
Then things started wokring properly. Now also some tests are failing but alteast the alluxio setup is properly up. Now I will debug why some tests are failing.

I have submitted this fix in alluxio git for them to merge it in master branch.
https://github.com/Alluxio/alluxio/pull/7376
...

Haoyuan Li

unread,
Jun 12, 2018, 4:49:28 PM6/12/18
to sushilk...@gmail.com, Alluxio Users, Alluxio Developers
Please try our user mailing list.

Best regards,

Haoyuan (HY)



Reply all
Reply to author
Forward
0 new messages