Unable to run scylladb/scylla docker image in Docker for Mac's Kubernetes

[Douglas Mendizábal]

Hello Scylla friends,

I've been unable to run the official scylladb/scylla docker image on the Kubernetes install that comes bundled with Docker for Mac.  Image seems to be working correctly if I run it on Docker directly.  Any tips to get it running under k8s would be greatly appreciated.  Here's the logs for the failed scylla container:

Scylla version 2.1.0-0.20180216.58b90ceee starting ...
WARN  2018-03-05 22:04:40,698 [shard 0] init - Only 256 MiB per shard; this is below the recommended minimum of 1 GiB/shard; continuing since running in developer mode
Connecting to http://localhost:10000
Starting the JMX server
JMX is enabled to receive remote connections on port: 7199
2018-03-05 22:04:41,533 INFO success: scylla-jmx entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-03-05 22:04:41,533 INFO success: scylla entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
ERROR 2018-03-05 22:05:10,741 [shard 0] seastar - Exiting on unhandled exception: std::system_error (error C-Ares:4, Not found)
scylla: ./seastar/core/sharded.hh:374: seastar::sharded<T>::~sharded() [with Service = seastar::httpd::http_server]: Assertion `_instances.empty()' failed.
Aborting on shard 0.
Backtrace:
  0x00000000004a38ab
  0x00000000004a397c
  0x00000000004a3a33
  /lib64/libpthread.so.0+0x000000000000f5df
  /lib64/libc.so.6+0x00000000000351f6
  /lib64/libc.so.6+0x00000000000368e7
  /lib64/libc.so.6+0x000000000002e265
  /lib64/libc.so.6+0x000000000002e311
  0x00000000008e8e72
  0x00007fff6cd71faf
  0x05ffffffffffffff
2018-03-05 22:05:10,959 INFO exited: scylla (terminated by SIGABRT; not expected)
2018-03-05 22:05:11,964 INFO spawned: 'scylla' with pid 42
tuning /sys/dev/block/0:75
tuning /sys/dev/block/0:75
Unexpected end of /proc/mounts line `overlay / overlay rw,relatime,lowerdir=/var/lib/docker/overlay2/l/YQEXED672TKOXKDL3E4UN44SO7:/var/lib/docker/overlay2/l/3MZQ7M4EHG6H6FLZ2PISBIUKHN:/var/lib/docker/overlay2/l/DXXKUCXEWF4YLXFUHYFILJVJDY:/var/lib/docker/overlay2/l/OTMXIKFPSZFVVQ5KNRMV647MIG:/var/lib/docker/overlay2/l/SYH3PB5P2Y7DS33NNO6PCQWD4V:/var/lib/docker/overlay2/l/TG2545SZUBCJDBXUGJIJYUOJMI:/var/lib/docker/overlay2/l/4FUZBXRB6NO6S46EWC6UDESUVP:/var/lib/docker/overlay2/l/UAU7NTVQKLJDWN7J2VJRGMDJVK:/var/lib/docker/overlay2/l/PO4CCRGGOVCM2'
Unexpected end of /proc/mounts line `overlay / overlay rw,relatime,lowerdir=/var/lib/docker/overlay2/l/YQEXED672TKOXKDL3E4UN44SO7:/var/lib/docker/overlay2/l/3MZQ7M4EHG6H6FLZ2PISBIUKHN:/var/lib/docker/overlay2/l/DXXKUCXEWF4YLXFUHYFILJVJDY:/var/lib/docker/overlay2/l/OTMXIKFPSZFVVQ5KNRMV647MIG:/var/lib/docker/overlay2/l/SYH3PB5P2Y7DS33NNO6PCQWD4V:/var/lib/docker/overlay2/l/TG2545SZUBCJDBXUGJIJYUOJMI:/var/lib/docker/overlay2/l/4FUZBXRB6NO6S46EWC6UDESUVP:/var/lib/docker/overlay2/l/UAU7NTVQKLJDWN7J2VJRGMDJVK:/var/lib/docker/overlay2/l/PO4CCRGGOVCM2'
WARN  2018-03-05 22:05:12,135 [shard 0] seastar - Unable to set SCHED_FIFO scheduling policy for timer thread; latency impact possible. Try adding CAP_SYS_NICE
Scylla version 2.1.0-0.20180216.58b90ceee starting ...
WARN  2018-03-05 22:05:12,142 [shard 0] init - Only 256 MiB per shard; this is below the recommended minimum of 1 GiB/shard; continuing since running in developer mode
2018-03-05 22:05:13,144 INFO success: scylla entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)

And here's the k8s spec I'm using:

apiVersion: v1
kind: Service
metadata:
  name: scylla
spec:
  selector:
    app: scylla
  ports:
  - port: 9044
    targetPort: 9042
    protocol: TCP
  type: LoadBalancer
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: scylla
spec:
  serviceName: scylla
  selector:
    matchLabels:
      app: scylla
  replicas: 1
  template:
    metadata:
      labels:
        app: scylla
    spec:
      containers:
      - name: scylla
        image: scylladb/scylla:2.1.0
        args:
        - "--broadcast-rpc-address"
        - "localhost:9044"
        - "--memory"
        - "1G"
        - "--overprovisioned"
        - "1"
        ports:
        - containerPort: 7000
        - containerPort: 7001
        - containerPort: 9042
        - containerPort: 9160
        - containerPort: 10000
        volumeMounts:
        - name: scylladb
          mountPath: /var/lib/scylla
  volumeClaimTemplates:
    - metadata:
        name: scylladb
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 1Gi


Regards,
- Douglas Mendizábal

[Avi Kivity]

On 03/06/2018 12:07 AM, Douglas Mendizábal wrote:

Hello Scylla friends,

Hi,

I've been unable to run the official scylladb/scylla docker image on the Kubernetes install that comes bundled with Docker for Mac.  Image seems to be working correctly if I run it on Docker directly.  Any tips to get it running under k8s would be greatly appreciated.  Here's the logs for the failed scylla container:

Scylla version 2.1.0-0.20180216.58b90ceee starting ...
WARN  2018-03-05 22:04:40,698 [shard 0] init - Only 256 MiB per shard; this is below the recommended minimum of 1 GiB/shard; continuing since running in developer mode
Connecting to http://localhost:10000
Starting the JMX server
JMX is enabled to receive remote connections on port: 7199
2018-03-05 22:04:41,533 INFO success: scylla-jmx entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-03-05 22:04:41,533 INFO success: scylla entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
ERROR 2018-03-05 22:05:10,741 [shard 0] seastar - Exiting on unhandled exception: std::system_error (error C-Ares:4, Not found)

The problem here is that DNS was not configured correctly, and in addition Scylla did not handle the error gracefully. Please file an issue so we can make the failure more informative (i.e. report which DNS name did not resolve, and not crash afterwards).

Can you check the DNS configuration? What's /etc/resolv.conf in the container? Are the DNS servers reachable?

I'm no kubernetes expert, but IIUC LoadBalancer destroys the token-aware driver capability that most drivers offer. It's better to offer clients direct access to Scylla IP addresses.

If you specify --memory 1GB, also specify --smp 1, to prevent the small amount of memory from being further divided among many cores.

        ports:
        - containerPort: 7000
        - containerPort: 7001
        - containerPort: 9042
        - containerPort: 9160
        - containerPort: 10000
        volumeMounts:
        - name: scylladb
          mountPath: /var/lib/scylla
  volumeClaimTemplates:
    - metadata:
        name: scylladb
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 1Gi


Regards,
- Douglas Mendizábal

--
You received this message because you are subscribed to the Google Groups "ScyllaDB users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scylladb-user...@googlegroups.com.
To post to this group, send email to scyllad...@googlegroups.com.
Visit this group at https://groups.google.com/group/scylladb-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/scylladb-users/4fb76067-5e38-4e02-86fd-89e8edbe9668%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jesse Haber-Kucharsky]

That seems correct, accordinga to the documentation [1] ("Type LoadBalancer"), `type: LoadBalancer` provisions an external load-balancer in a provider-specific way.

> > <mailto:scylladb-user...@googlegroups.com>.

> > To post to this group, send email to scyllad...@googlegroups.com

> > <mailto:scyllad...@googlegroups.com>.

> > Visit this group at https://groups.google.com/group/scylladb-users.

> > To view this discussion on the web visit https://groups.google.com/d/msgid/scylladb-users/4fb76067-5e38-4e02-86fd-89e8edbe9668%40googlegroups.com <https://groups.google.com/d/msgid/scylladb-users/4fb76067-5e38-4e02-86fd-89e8edbe9668%40googlegroups.com?utm_medium=email&utm_source=footer>.

> > For more options, visit https://groups.google.com/d/optout.
>

> --
> You received this message because you are subscribed to the Google Groups "ScyllaDB users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to scylladb-user...@googlegroups.com.
> To post to this group, send email to scyllad...@googlegroups.com.
> Visit this group at https://groups.google.com/group/scylladb-users.

> To view this discussion on the web visit https://groups.google.com/d/msgid/scylladb-users/60fc1962-dbaa-e86e-c3b8-78b4786f4b20%40scylladb.com.

> For more options, visit https://groups.google.com/d/optout.

Douglas,

I'm also by no means a Kubernetes expert, but perhaps the example on GitHub [2] will help?

[1] https://kubernetes.io/docs/concepts/services-networking/service/

[2] https://github.com/scylladb/scylla-code-samples/tree/master/kubernetes-scylla