New install on RHEL7 fails to start

747 views
Skip to first unread message

Mohamed Lrhazi

<lrhazi@gmail.com>
unread,
Mar 17, 2016, 4:05:06 AM3/17/16
to ScyllaDB users
Hello,

This is my first attempt at installing Scylla. When I try to start I get this error:


[root@avesterra-prod-1 ~]# systemctl status scylla-server                                                                                                                                     
● scylla-server.service - Scylla Server
   Loaded: loaded (/usr/lib/systemd/system/scylla-server.service; disabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Thu 2016-03-17 03:55:47 EDT; 13s ago
  Process: 11905 ExecStopPost=/usr/bin/sudo -E /usr/lib/scylla/scylla_stop (code=exited, status=0/SUCCESS)
  Process: 11870 ExecStart=/usr/bin/scylla $SCYLLA_ARGS $SCYLLA_IO (code=exited, status=1/FAILURE)
  Process: 11867 ExecStartPre=/usr/bin/sudo -E /usr/lib/scylla/scylla_prepare (code=exited, status=0/SUCCESS)
 Main PID: 11870 (code=exited, status=1/FAILURE)

Mar 17 03:55:47 avesterra-prod-1.uis.georgetown.edu scylla[11870]: EAL: lcore 12 is ready (tid=c507c700;cpuset=[12])
Mar 17 03:55:47 avesterra-prod-1.uis.georgetown.edu scylla[11870]: EAL: PCI device 0000:0b:00.0 on NUMA socket -1
Mar 17 03:55:47 avesterra-prod-1.uis.georgetown.edu scylla[11870]: EAL:   probe driver: 15ad:7b0 rte_vmxnet3_pmd
Mar 17 03:55:47 avesterra-prod-1.uis.georgetown.edu scylla[11870]: EAL:   Not managed by a supported kernel driver, skipped
Mar 17 03:55:47 avesterra-prod-1.uis.georgetown.edu scylla[11870]: Scylla version 0.19-20160310.7c12687 starting ...
Mar 17 03:55:47 avesterra-prod-1.uis.georgetown.edu scylla[11870]: Exiting on unhandled exception of type 'boost::exception_detail::clone_impl<boost::exception_detail::error_inf...d argument
Mar 17 03:55:47 avesterra-prod-1.uis.georgetown.edu systemd[1]: scylla-server.service: main process exited, code=exited, status=1/FAILURE
Mar 17 03:55:47 avesterra-prod-1.uis.georgetown.edu systemd[1]: Failed to start Scylla Server.
Mar 17 03:55:47 avesterra-prod-1.uis.georgetown.edu systemd[1]: Unit scylla-server.service entered failed state.
Mar 17 03:55:47 avesterra-prod-1.uis.georgetown.edu systemd[1]: scylla-server.service failed.



Version :

Package scylla-server-0.19-20160310.7c12687.el7.centos.x86_64 already installed and latest version
Package scylla-jmx-0.19-20160310.ac51e7f.el7.centos.noarch already installed and latest version
Package scylla-tools-0.19-20160310.0bd0b18.el7.centos.noarch already installed and latest version


Thanks,
Mohamed.

Pekka Enberg

<penberg@scylladb.com>
unread,
Mar 17, 2016, 4:10:25 AM3/17/16
to ScyllaDB users, Mohamed Lrhazi, Avi Kivity
Looks like an early startup issue. Can you post the full error message
from "journalctl"?

- Pekka

Mohamed Lrhazi

<lrhazi@gmail.com>
unread,
Mar 17, 2016, 4:18:08 AM3/17/16
to ScyllaDB users, lrhazi@gmail.com, avi@scylladb.com
Error msg looks truncated.. how do I find more:


-- Unit scylla-server.service has begun starting up.
Mar 17 04:17:25 avesterra-prod-1.uis.georgetown.edu scylla[12640]: EAL: Detected lcore 0 as core 0 on socket 0
Mar 17 04:17:25 avesterra-prod-1.uis.georgetown.edu scylla[12640]: EAL: Detected lcore 1 as core 1 on socket 0
Mar 17 04:17:25 avesterra-prod-1.uis.georgetown.edu scylla[12640]: EAL: Detected lcore 2 as core 0 on socket 1
Mar 17 04:17:25 avesterra-prod-1.uis.georgetown.edu scylla[12640]: EAL: Detected lcore 3 as core 1 on socket 1
Mar 17 04:17:25 avesterra-prod-1.uis.georgetown.edu scylla[12640]: EAL: Detected lcore 4 as core 0 on socket 2
Mar 17 04:17:25 avesterra-prod-1.uis.georgetown.edu scylla[12640]: EAL: Detected lcore 5 as core 1 on socket 2
Mar 17 04:17:25 avesterra-prod-1.uis.georgetown.edu scylla[12640]: EAL: Detected lcore 6 as core 0 on socket 3
Mar 17 04:17:25 avesterra-prod-1.uis.georgetown.edu scylla[12640]: EAL: Detected lcore 7 as core 1 on socket 3
Mar 17 04:17:25 avesterra-prod-1.uis.georgetown.edu scylla[12640]: EAL: Detected lcore 8 as core 0 on socket 4
Mar 17 04:17:25 avesterra-prod-1.uis.georgetown.edu scylla[12640]: EAL: Detected lcore 9 as core 1 on socket 4
Mar 17 04:17:25 avesterra-prod-1.uis.georgetown.edu scylla[12640]: EAL: Detected lcore 10 as core 0 on socket 5
Mar 17 04:17:25 avesterra-prod-1.uis.georgetown.edu scylla[12640]: EAL: Detected lcore 11 as core 1 on socket 5
Mar 17 04:17:25 avesterra-prod-1.uis.georgetown.edu scylla[12640]: EAL: Detected lcore 12 as core 0 on socket 6
Mar 17 04:17:25 avesterra-prod-1.uis.georgetown.edu scylla[12640]: EAL: Detected lcore 13 as core 1 on socket 6
Mar 17 04:17:25 avesterra-prod-1.uis.georgetown.edu scylla[12640]: EAL: Detected lcore 14 as core 0 on socket 7
Mar 17 04:17:25 avesterra-prod-1.uis.georgetown.edu scylla[12640]: EAL: Detected lcore 15 as core 1 on socket 7
Mar 17 04:17:25 avesterra-prod-1.uis.georgetown.edu scylla[12640]: EAL: Support maximum 128 logical core(s) by configuration.
Mar 17 04:17:25 avesterra-prod-1.uis.georgetown.edu scylla[12640]: EAL: Detected 16 lcore(s)
Mar 17 04:17:25 avesterra-prod-1.uis.georgetown.edu scylla[12640]: EAL: VFIO modules not all loaded, skip VFIO support...
Mar 17 04:17:25 avesterra-prod-1.uis.georgetown.edu scylla[12640]: EAL: Setting up physically contiguous memory...
Mar 17 04:17:26 avesterra-prod-1.uis.georgetown.edu scylla[12640]: EAL: TSC frequency is ~3000004 KHz
Mar 17 04:17:26 avesterra-prod-1.uis.georgetown.edu scylla[12640]: EAL: Master lcore 0 is ready (tid=ae719600;cpuset=[0])
Mar 17 04:17:26 avesterra-prod-1.uis.georgetown.edu scylla[12640]: EAL: lcore 2 is ready (tid=a174e700;cpuset=[2])
Mar 17 04:17:26 avesterra-prod-1.uis.georgetown.edu scylla[12640]: EAL: lcore 7 is ready (tid=9ef49700;cpuset=[7])
Mar 17 04:17:26 avesterra-prod-1.uis.georgetown.edu scylla[12640]: EAL: lcore 4 is ready (tid=a074c700;cpuset=[4])
Mar 17 04:17:26 avesterra-prod-1.uis.georgetown.edu scylla[12640]: EAL: lcore 1 is ready (tid=a1f4f700;cpuset=[1])
Mar 17 04:17:26 avesterra-prod-1.uis.georgetown.edu scylla[12640]: EAL: lcore 8 is ready (tid=9e748700;cpuset=[8])
Mar 17 04:17:26 avesterra-prod-1.uis.georgetown.edu scylla[12640]: EAL: lcore 10 is ready (tid=9d746700;cpuset=[10])
Mar 17 04:17:26 avesterra-prod-1.uis.georgetown.edu scylla[12640]: EAL: lcore 11 is ready (tid=9cf45700;cpuset=[11])
Mar 17 04:17:26 avesterra-prod-1.uis.georgetown.edu scylla[12640]: EAL: lcore 14 is ready (tid=9b742700;cpuset=[14])
Mar 17 04:17:26 avesterra-prod-1.uis.georgetown.edu scylla[12640]: EAL: lcore 15 is ready (tid=9af41700;cpuset=[15])
Mar 17 04:17:26 avesterra-prod-1.uis.georgetown.edu scylla[12640]: EAL: lcore 3 is ready (tid=a0f4d700;cpuset=[3])
Mar 17 04:17:26 avesterra-prod-1.uis.georgetown.edu scylla[12640]: EAL: lcore 9 is ready (tid=9df47700;cpuset=[9])
Mar 17 04:17:26 avesterra-prod-1.uis.georgetown.edu scylla[12640]: EAL: lcore 13 is ready (tid=9bf43700;cpuset=[13])
Mar 17 04:17:26 avesterra-prod-1.uis.georgetown.edu scylla[12640]: EAL: lcore 6 is ready (tid=9f74a700;cpuset=[6])
Mar 17 04:17:26 avesterra-prod-1.uis.georgetown.edu scylla[12640]: EAL: lcore 5 is ready (tid=9ff4b700;cpuset=[5])
Mar 17 04:17:26 avesterra-prod-1.uis.georgetown.edu scylla[12640]: EAL: lcore 12 is ready (tid=9c744700;cpuset=[12])
Mar 17 04:17:26 avesterra-prod-1.uis.georgetown.edu scylla[12640]: EAL: PCI device 0000:0b:00.0 on NUMA socket -1
Mar 17 04:17:26 avesterra-prod-1.uis.georgetown.edu scylla[12640]: EAL:   probe driver: 15ad:7b0 rte_vmxnet3_pmd
Mar 17 04:17:26 avesterra-prod-1.uis.georgetown.edu scylla[12640]: EAL:   Not managed by a supported kernel driver, skipped
Mar 17 04:17:26 avesterra-prod-1.uis.georgetown.edu scylla[12640]: Scylla version 0.19-20160310.7c12687 starting ...
Mar 17 04:17:26 avesterra-prod-1.uis.georgetown.edu scylla[12640]: Exiting on unhandled exception of type 'boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<bo
Mar 17 04:17:26 avesterra-prod-1.uis.georgetown.edu systemd[1]: scylla-server.service: main process exited, code=exited, status=1/FAILURE
Mar 17 04:17:26 avesterra-prod-1.uis.georgetown.edu systemd[1]: Failed to start Scylla Server.
-- Subject: Unit scylla-server.service has failed
-- Defined-By: systemd
-- 
-- Unit scylla-server.service has failed.
-- 
-- The result is failed.
Mar 17 04:17:26 avesterra-prod-1.uis.georgetown.edu systemd[1]: Unit scylla-server.service entered failed state.
Mar 17 04:17:26 avesterra-prod-1.uis.georgetown.edu systemd[1]: scylla-server.service failed.
lines 1524-1578/1578 (END)

Pekka Enberg

<penberg@scylladb.com>
unread,
Mar 17, 2016, 4:34:12 AM3/17/16
to ScyllaDB users, Mohamed Lrhazi, Avi Kivity
On Thu, Mar 17, 2016 at 10:18 AM, Mohamed Lrhazi <lrh...@gmail.com> wrote:
> Error msg looks truncated.. how do I find more:

I am on Fedora 23 where you can see the full message by using
right-arrow to move (like with less).

If it really *is* truncated on RHEL, you should be able to see the
full thing with "journalctl --full".

- Pekka

Mohamed Lrhazi

<lrhazi@gmail.com>
unread,
Mar 17, 2016, 4:39:29 AM3/17/16
to ScyllaDB users, lrhazi@gmail.com, avi@scylladb.com
Thanks! I can see the full line indeed, by moving with the right-arrow:

Mar 17 04:18:36 avesterra-prod-1.uis.georgetown.edu scylla[12690]: Exiting on unhandled exception of type 'boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<bo

on of type 'boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::system::system_error> >': Invalid argument

Not all that helpful, is it?

Avi Kivity

<avi@scylladb.com>
unread,
Mar 17, 2016, 4:44:13 AM3/17/16
to Mohamed Lrhazi, ScyllaDB users
Indeed the error is somewhat less than self-explanatory.

The error is probably in the area of the configuration files.  Can you make sure that scylla.yaml is valid, and that it is readable by the scylla user?

I'll try to see what could possible have thrown this exception (and make it more readable).

Pekka Enberg

<penberg@scylladb.com>
unread,
Mar 17, 2016, 4:46:55 AM3/17/16
to ScyllaDB users, Mohamed Lrhazi, Avi Kivity
On Thu, Mar 17, 2016 at 10:44 AM, Avi Kivity <a...@scylladb.com> wrote:
> Indeed the error is somewhat less than self-explanatory.
>
> The error is probably in the area of the configuration files. Can you make
> sure that scylla.yaml is valid, and that it is readable by the scylla user?
>
> I'll try to see what could possible have thrown this exception (and make it
> more readable).

It's a system error which suggests that a system call failed with EINVAL, no?

- Pekka

Avi Kivity

<avi@scylladb.com>
unread,
Mar 17, 2016, 4:48:47 AM3/17/16
to Pekka Enberg, ScyllaDB users, Mohamed Lrhazi
It looks like that, yes.

Mohamed Lrhazi

<lrhazi@gmail.com>
unread,
Mar 17, 2016, 4:51:29 AM3/17/16
to ScyllaDB users, lrhazi@gmail.com, avi@scylladb.com
I did:

chown -R scylla /etc/scylla
senenforce 0

same error.

and:

[root@avesterra-prod-1 ~]# strace systemctl restart scylla-server 2> /tmp/foo  
                                                                                                          
[root@avesterra-prod-1 ~]# cat   /tmp/foo | grep /etc/                                                                                                                                        
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
stat("/etc/sysconfig/64bit_strstr_via_64bit_strstr_sse2_unaligned", 0x7ffdb401d000) = -1 ENOENT (No such file or directory)
access("/etc/system-fips", F_OK)        = -1 ENOENT (No such file or directory)
[root@avesterra-prod-1 ~]# 
[root@avesterra-prod-1 ~]# 


You guys want the that full trace? or some other trace?

Avi Kivity

<avi@scylladb.com>
unread,
Mar 17, 2016, 4:57:35 AM3/17/16
to Mohamed Lrhazi, ScyllaDB users


On 03/17/2016 10:51 AM, Mohamed Lrhazi wrote:
I did:

chown -R scylla /etc/scylla
senenforce 0

same error.

and:

[root@avesterra-prod-1 ~]# strace systemctl restart scylla-server 2> /tmp/foo  
                                                                                                          
[root@avesterra-prod-1 ~]# cat   /tmp/foo | grep /etc/                                                                                                                                        
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
stat("/etc/sysconfig/64bit_strstr_via_64bit_strstr_sse2_unaligned", 0x7ffdb401d000) = -1 ENOENT (No such file or directory)
access("/etc/system-fips", F_OK)        = -1 ENOENT (No such file or directory)
[root@avesterra-prod-1 ~]# 
[root@avesterra-prod-1 ~]# 


You guys want the that full trace? or some other trace?

Try:

   . /etc/sysconfig/scylla-server
   . /etc/scylla.d/io.conf
   strace -fF -o /tmp/trace /usr/bin/scylla $SCYLLA_ARGS $SEASTAR_IO

and look for EINVAL in /tmp/trace.

Mohamed Lrhazi

<lrhazi@gmail.com>
unread,
Mar 17, 2016, 5:01:28 AM3/17/16
to Avi Kivity, ScyllaDB users
Much better! thanks! I had cassandra installed then removed.. must be some left over config files?


[root@avesterra-prod-1 ~]# strace -fF -o /tmp/trace /usr/bin/scylla $SCYLLA_ARGS $SEASTAR_IO
EAL: Detected lcore 0 as core 0 on socket 0
EAL: Detected lcore 1 as core 1 on socket 0
EAL: Detected lcore 2 as core 0 on socket 1
EAL: Detected lcore 3 as core 1 on socket 1
EAL: Detected lcore 4 as core 0 on socket 2
EAL: Detected lcore 5 as core 1 on socket 2
EAL: Detected lcore 6 as core 0 on socket 3
EAL: Detected lcore 7 as core 1 on socket 3
EAL: Detected lcore 8 as core 0 on socket 4
EAL: Detected lcore 9 as core 1 on socket 4
EAL: Detected lcore 10 as core 0 on socket 5
EAL: Detected lcore 11 as core 1 on socket 5
EAL: Detected lcore 12 as core 0 on socket 6
EAL: Detected lcore 13 as core 1 on socket 6
EAL: Detected lcore 14 as core 0 on socket 7
EAL: Detected lcore 15 as core 1 on socket 7
EAL: Support maximum 128 logical core(s) by configuration.
EAL: Detected 16 lcore(s)
EAL: VFIO modules not all loaded, skip VFIO support...
EAL: Setting up physically contiguous memory...
EAL: TSC frequency is ~3000062 KHz
EAL: Master lcore 0 is ready (tid=9c1d8600;cpuset=[0])
EAL: lcore 1 is ready (tid=8fa0e700;cpuset=[1])
EAL: lcore 2 is ready (tid=8f20d700;cpuset=[2])
EAL: lcore 4 is ready (tid=8e20b700;cpuset=[4])
EAL: lcore 5 is ready (tid=8da0a700;cpuset=[5])
EAL: lcore 3 is ready (tid=8ea0c700;cpuset=[3])
EAL: lcore 7 is ready (tid=8ca08700;cpuset=[7])
EAL: lcore 6 is ready (tid=8d209700;cpuset=[6])
EAL: lcore 9 is ready (tid=8ba06700;cpuset=[9])
EAL: lcore 10 is ready (tid=8b205700;cpuset=[10])
EAL: lcore 8 is ready (tid=8c207700;cpuset=[8])
EAL: lcore 11 is ready (tid=8aa04700;cpuset=[11])
EAL: lcore 12 is ready (tid=8a203700;cpuset=[12])
EAL: lcore 13 is ready (tid=89a02700;cpuset=[13])
EAL: lcore 14 is ready (tid=89201700;cpuset=[14])
EAL: lcore 15 is ready (tid=88a00700;cpuset=[15])
EAL: PCI device 0000:0b:00.0 on NUMA socket -1
EAL:   probe driver: 15ad:7b0 rte_vmxnet3_pmd
EAL:   Not managed by a supported kernel driver, skipped
Scylla version 0.19-20160310.7c12687 starting ...
Exiting on unhandled exception of type 'std::invalid_argument': Could not open file at conf/scylla.yaml. Make sure it exists


Mohamed Lrhazi

<lrhazi@gmail.com>
unread,
Mar 17, 2016, 5:03:44 AM3/17/16
to ScyllaDB users, lrhazi@gmail.com
What it seems to not find is apparently really:  conf/scylla.yaml

[root@avesterra-prod-1 ~]# cat /tmp/trace | grep .yaml
14150 open("/lib64/libyaml-cpp.so.0.5", O_RDONLY|O_CLOEXEC) = 3
14167 stat("conf/scylla.yaml",  <unfinished ...>

Pekka Enberg

<penberg@scylladb.com>
unread,
Mar 17, 2016, 5:07:39 AM3/17/16
to ScyllaDB users, Mohamed Lrhazi, Avi Kivity
On Thu, Mar 17, 2016 at 11:03 AM, Mohamed Lrhazi <lrh...@gmail.com> wrote:
> What it seems to not find is apparently really: conf/scylla.yaml
>
> [root@avesterra-prod-1 ~]# cat /tmp/trace | grep .yaml
> 14150 open("/lib64/libyaml-cpp.so.0.5", O_RDONLY|O_CLOEXEC) = 3
> 14167 stat("conf/scylla.yaml", <unfinished ...>
> [root@avesterra-prod-1 ~]#
> [root@avesterra-prod-1 ~]#

You need to start scylla in the "/var/lib/scylla" directory:

. /etc/sysconfig/scylla-server
. /etc/scylla.d/io.conf
cd /var/lib/scylla
strace -fF -o /tmp/trace /usr/bin/scylla $SCYLLA_ARGS $SEASTAR_IO

- Pekka

Mohamed Lrhazi

<lrhazi@gmail.com>
unread,
Mar 17, 2016, 5:11:10 AM3/17/16
to ScyllaDB users, lrhazi@gmail.com, avi@scylladb.com
Cool. now a better error:

EAL: PCI device 0000:0b:00.0 on NUMA socket -1
EAL:   probe driver: 15ad:7b0 rte_vmxnet3_pmd
EAL:   Not managed by a supported kernel driver, skipped
Scylla version 0.19-20160310.7c12687 starting ...
Exiting on unhandled exception of type 'std::runtime_error': NOFILE rlimit too low
[root@avesterra-prod-1 scylla]# pwd
/var/lib/scylla
[root@avesterra-prod-1 scylla]# 

Pekka Enberg

<penberg@scylladb.com>
unread,
Mar 17, 2016, 5:15:37 AM3/17/16
to ScyllaDB users, Mohamed Lrhazi, Avi Kivity
On Thu, Mar 17, 2016 at 11:11 AM, Mohamed Lrhazi <lrh...@gmail.com> wrote:
> Cool. now a better error:
>
> EAL: PCI device 0000:0b:00.0 on NUMA socket -1
> EAL: probe driver: 15ad:7b0 rte_vmxnet3_pmd
> EAL: Not managed by a supported kernel driver, skipped
> Scylla version 0.19-20160310.7c12687 starting ...
> Exiting on unhandled exception of type 'std::runtime_error': NOFILE rlimit
> too low
> [root@avesterra-prod-1 scylla]# pwd
> /var/lib/scylla
> [root@avesterra-prod-1 scylla]#

It's still unfortunately not the error we're after because NOFILE is
set by the systemd.

Try with:

ulimit -n 200000

before you launch scylla.

- Pekka

Mohamed Lrhazi

<lrhazi@gmail.com>
unread,
Mar 17, 2016, 5:25:49 AM3/17/16
to ScyllaDB users, lrhazi@gmail.com, avi@scylladb.com
OK. back to the previous error.... but no EINVAL in the trace.


EAL:   probe driver: 15ad:7b0 rte_vmxnet3_pmd
EAL:   Not managed by a supported kernel driver, skipped
Scylla version 0.19-20160310.7c12687 starting ...
Exiting on unhandled exception of type 'boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::system::system_error> >': Invalid argument

[root@avesterra-prod-1 scylla]# cat /tmp/trace | grep .yaml                                                                                                                                   
15308 open("/lib64/libyaml-cpp.so.0.5", O_RDONLY|O_CLOEXEC) = 3
15325 stat("conf/scylla.yaml",  <unfinished ...>
15325 open("conf/scylla.yaml", O_RDONLY|O_DIRECT|O_CLOEXEC <unfinished ...>
15325 open("conf/scylla.yaml", O_RDONLY|O_DIRECT|O_CLOEXEC <unfinished ...>
[root@avesterra-prod-1 scylla]# 

[root@avesterra-prod-1 scylla]# 
[root@avesterra-prod-1 scylla]# cat   /tmp/trace | grep EINVAL                                                                                                                                
[root@avesterra-prod-1 scylla]# 

Pekka Enberg

<penberg@scylladb.com>
unread,
Mar 17, 2016, 5:27:43 AM3/17/16
to ScyllaDB users, Mohamed Lrhazi, Avi Kivity
On Thu, Mar 17, 2016 at 11:25 AM, Mohamed Lrhazi <lrh...@gmail.com> wrote:
> OK. back to the previous error.... but no EINVAL in the trace.
>
> EAL: probe driver: 15ad:7b0 rte_vmxnet3_pmd
> EAL: Not managed by a supported kernel driver, skipped
> Scylla version 0.19-20160310.7c12687 starting ...
> Exiting on unhandled exception of type
> 'boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::system::system_error>
>>': Invalid argument
>
> [root@avesterra-prod-1 scylla]# cat /tmp/trace | grep .yaml
> 15308 open("/lib64/libyaml-cpp.so.0.5", O_RDONLY|O_CLOEXEC) = 3
> 15325 stat("conf/scylla.yaml", <unfinished ...>
> 15325 open("conf/scylla.yaml", O_RDONLY|O_DIRECT|O_CLOEXEC <unfinished ...>
> 15325 open("conf/scylla.yaml", O_RDONLY|O_DIRECT|O_CLOEXEC <unfinished ...>
> [root@avesterra-prod-1 scylla]#
>
> [root@avesterra-prod-1 scylla]#
> [root@avesterra-prod-1 scylla]# cat /tmp/trace | grep EINVAL
> [root@avesterra-prod-1 scylla]#

Can you please post the full strace log?

Mohamed Lrhazi

<lrhazi@gmail.com>
unread,
Mar 17, 2016, 5:30:22 AM3/17/16
to ScyllaDB users, lrhazi@gmail.com, avi@scylladb.com
This is all my chnages to the config file:

[root@avesterra-prod-1 scylla]# diff /etc/scylla/scylla.yaml /etc/scylla/scylla.yaml.new
29c29
<     - /var/lib/scylla/data
---
>     - /data2/scylla/data
34c34
< commitlog_directory: /var/lib/scylla/commitlog
---
> commitlog_directory: /data2/scylla/commitlog
76c76
<           - seeds: "127.0.0.1"
---
>           - seeds: "10.212.18.168,10.212.18.169"
82c82
< listen_address: localhost
---
> listen_address: eth0
[root@avesterra-prod-1 scylla]# 


I am hoping you ll show me my error there... else I ll try to post the trace.... might too big to post here?

Mohamed Lrhazi

<lrhazi@gmail.com>
unread,
Mar 17, 2016, 5:33:10 AM3/17/16
to ScyllaDB users, lrhazi@gmail.com, avi@scylladb.com
Looks like the directories being missing is the issue!! apparently it is happy now:

 [root@avesterra-prod-1 scylla]# mkdir  /data2/scylla/commitlog
[root@avesterra-prod-1 scylla]# mkdir  /data2/scylla/data
[root@avesterra-prod-1 scylla]# chown -R scylla /etc/scylla  /data2/scylla  
[root@avesterra-prod-1 scylla]# systemctl restart scylla-server                                                                                                                               
[root@avesterra-prod-1 scylla]# 

Mohamed Lrhazi

<lrhazi@gmail.com>
unread,
Mar 17, 2016, 5:35:34 AM3/17/16
to ScyllaDB users, lrhazi@gmail.com, avi@scylladb.com
Actually not quite....

Still not really starting:

Exiting on unhandled exception of type 'std::runtime_error': invalid configuration: path "/var/lib/scylla/commitlog" on unsupported filesystem

why is it not using what I specified? or how do I make it use /data2/scylla  for eveything?

Avi Kivity

<avi@scylladb.com>
unread,
Mar 17, 2016, 5:36:39 AM3/17/16
to Mohamed Lrhazi, ScyllaDB users
Great.  We will fix the error messages to be less horrible.

Asias He

<asias@scylladb.com>
unread,
Mar 17, 2016, 5:37:18 AM3/17/16
to ScyllaDB users, lrhazi@gmail.com, Avi Kivity
On Thu, Mar 17, 2016 at 5:35 PM, Mohamed Lrhazi <lrh...@gmail.com> wrote:
Actually not quite....

Still not really starting:

Exiting on unhandled exception of type 'std::runtime_error': invalid configuration: path "/var/lib/scylla/commitlog" on unsupported filesystem

why is it not using what I specified? or how do I make it use /data2/scylla  for eveything?

Are you using xfs for the dir?
 

--
You received this message because you are subscribed to the Google Groups "ScyllaDB users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scylladb-user...@googlegroups.com.
To post to this group, send email to scyllad...@googlegroups.com.
Visit this group at https://groups.google.com/group/scylladb-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/scylladb-users/f7c1e637-d39f-49d2-84b5-43125086828e%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--
Asias

Avi Kivity

<avi@scylladb.com>
unread,
Mar 17, 2016, 5:39:22 AM3/17/16
to Mohamed Lrhazi, ScyllaDB users
From the diff, looks like you need to copy scylla.yaml.new to scylla.yaml?

Asias He

<asias@scylladb.com>
unread,
Mar 17, 2016, 5:39:23 AM3/17/16
to ScyllaDB users, Mohamed Lrhazi
On Thu, Mar 17, 2016 at 5:36 PM, Avi Kivity <a...@scylladb.com> wrote:
Great.  We will fix the error messages to be less horrible.

I think we will create the dir.

$ cat scylla1.yaml
seed_provider:
    - class_name: org.apache.cassandra.locator.SimpleSeedProvider
      parameters:
          - seeds: "127.0.0.1,127.0.0.2"
listen_address: 127.0.0.1
start_native_transport: true
start_rpc: true
rpc_address: 127.0.0.1
developer_mode: true
data_file_directories:
    - tmp/1/d1
    - tmp/1/d2
    - tmp/1/d2
commitlog_directory: tmp/1/commitlog

$ rm -rf tmp

$ ./scylla --options-file scylla1.yaml

Scylla starts happily. 

 

--
You received this message because you are subscribed to the Google Groups "ScyllaDB users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scylladb-user...@googlegroups.com.
To post to this group, send email to scyllad...@googlegroups.com.
Visit this group at https://groups.google.com/group/scylladb-users.

For more options, visit https://groups.google.com/d/optout.



--
Asias

Mohamed Lrhazi

<lrhazi@gmail.com>
unread,
Mar 17, 2016, 5:44:31 AM3/17/16
to ScyllaDB users, lrhazi@gmail.com
right.... I got confused... reinstalled the rpms to get the original config file so I can do the diff.. then forgot to restore my config... now back to square one I think:

I am attahcing the trace.. hopefully correct one.


[root@avesterra-prod-1 scylla]# strace -fF -o /tmp/trace /usr/bin/scylla $SCYLLA_ARGS $SEASTAR_IO;echo
EAL: Detected lcore 0 as core 0 on socket 0
EAL: Detected lcore 1 as core 1 on socket 0
EAL: Detected lcore 2 as core 0 on socket 1
EAL: Detected lcore 3 as core 1 on socket 1
EAL: Detected lcore 4 as core 0 on socket 2
EAL: Detected lcore 5 as core 1 on socket 2
EAL: Detected lcore 6 as core 0 on socket 3
EAL: Detected lcore 7 as core 1 on socket 3
EAL: Detected lcore 8 as core 0 on socket 4
EAL: Detected lcore 9 as core 1 on socket 4
EAL: Detected lcore 10 as core 0 on socket 5
EAL: Detected lcore 11 as core 1 on socket 5
EAL: Detected lcore 12 as core 0 on socket 6
EAL: Detected lcore 13 as core 1 on socket 6
EAL: Detected lcore 14 as core 0 on socket 7
EAL: Detected lcore 15 as core 1 on socket 7
EAL: Support maximum 128 logical core(s) by configuration.
EAL: Detected 16 lcore(s)
EAL: VFIO modules not all loaded, skip VFIO support...
EAL: Setting up physically contiguous memory...
EAL: TSC frequency is ~2999963 KHz
EAL: Master lcore 0 is ready (tid=a125f600;cpuset=[0])
EAL: lcore 1 is ready (tid=94a95700;cpuset=[1])
EAL: lcore 2 is ready (tid=94294700;cpuset=[2])
EAL: lcore 3 is ready (tid=93a93700;cpuset=[3])
EAL: lcore 5 is ready (tid=92a91700;cpuset=[5])
EAL: lcore 4 is ready (tid=93292700;cpuset=[4])
EAL: lcore 7 is ready (tid=91a8f700;cpuset=[7])
EAL: lcore 6 is ready (tid=92290700;cpuset=[6])
EAL: lcore 9 is ready (tid=90a8d700;cpuset=[9])
EAL: lcore 8 is ready (tid=9128e700;cpuset=[8])
EAL: lcore 10 is ready (tid=9028c700;cpuset=[10])
EAL: lcore 12 is ready (tid=8f28a700;cpuset=[12])
EAL: lcore 11 is ready (tid=8fa8b700;cpuset=[11])
EAL: lcore 13 is ready (tid=8ea89700;cpuset=[13])
EAL: lcore 14 is ready (tid=8e288700;cpuset=[14])
EAL: lcore 15 is ready (tid=8da87700;cpuset=[15])
EAL: PCI device 0000:0b:00.0 on NUMA socket -1
EAL:   probe driver: 15ad:7b0 rte_vmxnet3_pmd
EAL:   Not managed by a supported kernel driver, skipped
Scylla version 0.19-20160310.7c12687 starting ...
Exiting on unhandled exception of type 'boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::system::system_error> >': Invalid argument

[root@avesterra-prod-1 scylla]# 




trace.txt

Asias He

<asias@scylladb.com>
unread,
Mar 17, 2016, 5:47:40 AM3/17/16
to ScyllaDB users, Mohamed Lrhazi
Do not use eth0 for listen_address.

 

For more options, visit https://groups.google.com/d/optout.



--
Asias

Asias He

<asias@scylladb.com>
unread,
Mar 17, 2016, 5:48:54 AM3/17/16
to ScyllaDB users, Mohamed Lrhazi
$ cat scylla0.yaml
seed_provider:
    - class_name: org.apache.cassandra.locator.SimpleSeedProvider
      parameters:
          - seeds: "127.0.0.1"
listen_address: eth0
start_native_transport: true
start_rpc: true
rpc_address: 127.0.0.1
developer_mode: true
data_file_directories:
    - tmp/0/d1
    - tmp/0/d2
    - tmp/0/d2
commitlog_directory: tmp/0/commitlog

 $ ./scylla --options-file scylla0.yaml
Scylla version 666.development-20160317.6a3872b starting ...
WARN  2016-03-17 17:47:40,597 [shard 0] config - Option start_native_transport is not (yet) used.
WARN  2016-03-17 17:47:40,597 [shard 0] init - NOFILE rlimit too low (recommended setting 200000, minimum setting 10000; you may run out of file descriptors.



--
Asias

Pekka Enberg

<penberg@scylladb.com>
unread,
Mar 17, 2016, 5:50:31 AM3/17/16
to ScyllaDB users, Mohamed Lrhazi, Asias He, Avi Kivity
On Thu, Mar 17, 2016 at 11:47 AM, Asias He <as...@scylladb.com> wrote:
> Do not use eth0 for listen_address.

Aah, that's the problem here:

[penberg@nero scylla]$ ./build/release/scylla --developer-mode=1
Scylla version 666.development-20160316.49af399 starting ...
WARN 2016-03-17 11:49:14,221 [shard 0] config - Option partitioner is
not (yet) used.
WARN 2016-03-17 11:49:14,222 [shard 0] init - NOFILE rlimit too low
(recommended setting 200000, minimum setting 10000; you may run out of
file descriptors.
Exiting on unhandled exception of type
'boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::system::system_error>
>': Invalid argument

We fail to parse "eth0" (because expect an IP address) but the error
message is just so confusing it's impossible to tell that.

- Pekka

Pekka Enberg

<penberg@scylladb.com>
unread,
Mar 17, 2016, 5:55:30 AM3/17/16
to ScyllaDB users, Mohamed Lrhazi, Asias He, Avi Kivity
I opened an issue about this:

https://github.com/scylladb/scylla/issues/1051

- Pekka

Mohamed Lrhazi

<lrhazi@gmail.com>
unread,
Mar 17, 2016, 6:02:27 AM3/17/16
to Pekka Enberg, ScyllaDB users, Asias He, Avi Kivity
Thanks guys. Now it starts when I set the ip of the interface, instead of the interface... note that the comment says that an interface is OK though.

# Address or interface to bind to and tell other Scylla nodes to connect to.
# You _must_ change this if you want multiple nodes to be able to communicate!
#
# Setting listen_address to 0.0.0.0 is always wrong.
listen_address: 10.212.18.168


Quick question, how do I get nodetool status to work?

[root@avesterra-prod-1 scylla]# nodetool -h $(uname -n) status
nodetool: Failed to connect to 'avesterra-prod-1.uis.georgetown.edu:7199' - ConnectException: 'Connection refused'.
[root@avesterra-prod-1 scylla]# 

Pekka Enberg

<penberg@scylladb.com>
unread,
Mar 17, 2016, 6:04:47 AM3/17/16
to Mohamed Lrhazi, ScyllaDB users, Asias He, Avi Kivity
On Thu, Mar 17, 2016 at 12:02 PM, Mohamed Lrhazi <lrh...@gmail.com> wrote:
> Thanks guys. Now it starts when I set the ip of the interface, instead of
> the interface... note that the comment says that an interface is OK though.
>
> # Address or interface to bind to and tell other Scylla nodes to connect to.
> # You _must_ change this if you want multiple nodes to be able to
> communicate!
> #
> # Setting listen_address to 0.0.0.0 is always wrong.
> listen_address: 10.212.18.168

Oh, sorry about that, we'll need to fix it up.

> Quick question, how do I get nodetool status to work?
>
> [root@avesterra-prod-1 scylla]# nodetool -h $(uname -n) status
> nodetool: Failed to connect to 'avesterra-prod-1.uis.georgetown.edu:7199' -
> ConnectException: 'Connection refused'.
> [root@avesterra-prod-1 scylla]#

There's a separate "scylla-jmx" proxy that should start up
automatically when you use systemd to start scylla.

- Pekka

Avi Kivity

<avi@scylladb.com>
unread,
Mar 17, 2016, 6:05:31 AM3/17/16
to Mohamed Lrhazi, Pekka Enberg, ScyllaDB users, Asias He
systemctl start scylla-jmx
nodetool status

Mohamed Lrhazi

<lrhazi@gmail.com>
unread,
Mar 17, 2016, 6:08:13 AM3/17/16
to Avi Kivity, Pekka Enberg, ScyllaDB users, Asias He
Thanks  a lot everyone!

Pekka Enberg

<penberg@scylladb.com>
unread,
Mar 18, 2016, 3:33:05 AM3/18/16
to Mohamed Lrhazi, ScyllaDB users, Asias He, Avi Kivity
On Thu, Mar 17, 2016 at 12:04 PM, Pekka Enberg <pen...@scylladb.com> wrote:
> On Thu, Mar 17, 2016 at 12:02 PM, Mohamed Lrhazi <lrh...@gmail.com> wrote:
>> Thanks guys. Now it starts when I set the ip of the interface, instead of
>> the interface... note that the comment says that an interface is OK though.
>>
>> # Address or interface to bind to and tell other Scylla nodes to connect to.
>> # You _must_ change this if you want multiple nodes to be able to
>> communicate!
>> #
>> # Setting listen_address to 0.0.0.0 is always wrong.
>> listen_address: 10.212.18.168
>
> Oh, sorry about that, we'll need to fix it up.

I opened an issue about this:

https://github.com/scylladb/scylla/issues/1058

- Pekka
Reply all
Reply to author
Forward
0 new messages