How to find out the reason the service cannot be started?

566 views
Skip to first unread message

Kiswono Prayogo

<kiswono@gmail.com>
unread,
Jun 23, 2017, 6:12:33 AM6/23/17
to ScyllaDB users
I'm using ubuntu 17.04 64-bit 
Kernel Linux ter 4.10.0-19-generic #21-Ubuntu SMP Thu Apr 6 17:04:57 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
Version: 1.7.1-0.20170607.dbbf99d-0ubuntu1

Journalctl -r: 
Jun 23 17:05:24 ter systemd[1]: scylla-server.service: Failed with result 'exit-code'.
Jun 23 17:05:24 ter systemd[1]: scylla-server.service: Unit entered failed state.
Jun 23 17:05:24 ter systemd[1]: Failed to start Scylla Server.
Jun 23 17:05:24 ter systemd[1]: scylla-server.service: Main process exited, code=exited, status=71/n/a
Jun 23 17:05:24 ter scylla_prepare[1733]: tuning /sys/devices/pci0000:00/0000:00:01.1/ata1/host0/target0:0:0/0:0:0:0/block/sda/sda1
Jun 23 17:05:24 ter scylla_prepare[1733]: already tuned: /sys/devices/pci0000:00/0000:00:01.1/ata1/host0/target0:0:0/0:0:0:0/block/sda/
Jun 23 17:05:24 ter scylla_prepare[1733]: already tuned: /sys/devices/pci0000:00/0000:00:01.1/ata1/host0/target0:0:0/0:0:0:0/block/sda/
Jun 23 17:05:24 ter scylla_prepare[1733]: tuning /sys/devices/pci0000:00/0000:00:01.1/ata1/host0/target0:0:0/0:0:0:0/block/sda
Jun 23 17:05:24 ter scylla_prepare[1733]: tuning /sys/devices/pci0000:00/0000:00:01.1/ata1/host0/target0:0:0/0:0:0:0/block/sda/sda1
Jun 23 17:05:24 ter scylla_prepare[1733]: hugeadm:WARNING: Directory /var/lib/hugetlbfs/pagesize-2MB is already mounted.
Jun 23 17:05:24 ter scylla_prepare[1733]: Setting mask 00000003 in /sys/class/net/ens3/queues/tx-0/xps_cpus
Jun 23 17:05:24 ter scylla_prepare[1733]: Setting mask 00000003 in /proc/irq/11/smp_affinity
Jun 23 17:05:24 ter scylla_prepare[1733]: Restarting irqbalance (via systemctl): irqbalance.service.
Jun 23 17:05:24 ter scylla_prepare[1733]: Original irqbalance configuration is in /etc/default/irqbalance.scylla.orig
Jun 23 17:05:24 ter scylla_prepare[1733]: Restarting irqbalance: going to ban the following IRQ numbers: 11 ...
Jun 23 17:05:24 ter scylla_prepare[1733]: Setting a physical interface ens3...
Jun 23 17:05:24 ter systemd[1]: Starting Scylla Server...


Configuration:
sudo scylla_setup
# NO -> It is recommended to use RAID0 and XFS for Scylla data. If you select yes, you will be prompt to choose which unmounted disks to use for Scylla data. Selected disks will be formatted in the process.
# NO -> Do you want to setup coredump? Answer yes to enable core dumps; this allows to do post-mortem analysis of Scylla state after a crash. Answer no to do nothing.
# Generating evaluation file sized 10GB... timed out before we could write the entire file. Will continue but accuracy may suffer.
# 2.92871GB written in 144 seconds
# Refining search for maximum. So far, 3968 IOPS 
# Maximum throughput: 3968 IOPS

sudo sed -i 's|/usr/bin/scylla $SCYLLA_ARGS|/usr/bin/scylla -m 1G -c 1 --developer-mode 1 $SCYLLA_ARGS|g' /lib/systemd/system/scylla-server.service

echo '
experimental: true
' | sudo tee -a /etc/scylla/scylla.yaml
sudo mv /var/lib/scylla /home/
sudo ln -s /home/scylla /var/lib/scylla

the similar configuration works fine on larger computer (with 32GB RAM), but this one installed on a computer with 4GB RAM.

Computer spec:
# inxi -F
System:    Host: ter Kernel: 4.10.0-19-generic x86_64 (64 bit) Console: tty 0 Distro: Ubuntu 17.04
Machine:   Device: kvm System: Red Hat product: KVM v: RHEL 6.6.0 PC                                                                    
          Mobo: N/A model: N/A BIOS: Seabios v: 0.5.1 date: 01/01/2007                                                                 
CPU(s):    2 Single core QEMU Virtual version (cpu64-rhel6)s (-HT-SMP-) cache: 8192 KB                                                  
          clock speeds: max: 2599 MHz 1: 2599 MHz 2: 2599 MHz                                                                          
Graphics:  Card: Cirrus Logic GD 5446                                                                                                   
          Display Server: N/A driver: N/A tty size: 135x72 Advanced Data: N/A for root out of X                                        
Audio:     Card Intel 82801FB/FBM/FR/FW/FRW (ICH6 Family) High Definition Audio Controller driver: snd_hda_intel                        
          Sound: Advanced Linux Sound Architecture v: k4.10.0-19-generic                                                               
Network:   Card: Intel 82540EM Gigabit Ethernet Controller driver: e1000                                                                
          IF: ens3 state: up speed: 1000 Mbps duplex: full mac: 00:16:3e:00:4e:c3                                                      
Drives:    HDD Total Size: 42.9GB (12.0% used)                                                                                          
          ID-1: /dev/sda model: QEMU_HARDDISK size: 42.9GB                                                                             
Partition: ID-1: / size: 40G used: 4.9G (13%) fs: ext4 dev: /dev/sda1                                                                   
RAID:      No RAID devices: /proc/mdstat, md_mod kernel module present                                                                  
Sensors:   None detected - is lm-sensors installed and configured?                                                                      
Info:      Processes: 120 Uptime: 7 min Memory: 249.1/3949.5MB Init: systemd runlevel: 5                                                
          Client: Shell (bash) inxi: 2.3.8 

How to find out which part that are wrong?

Glauber Costa

<glauber@scylladb.com>
unread,
Jun 23, 2017, 8:07:33 AM6/23/17
to ScyllaDB users
Recommended to look at the full logs with journalctl.

The command is sudo journalctl -u scylla-server.
> --
> You received this message because you are subscribed to the Google Groups
> "ScyllaDB users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to scylladb-user...@googlegroups.com.
> To post to this group, send email to scyllad...@googlegroups.com.
> Visit this group at https://groups.google.com/group/scylladb-users.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/scylladb-users/3e7a7f8b-53c1-4284-85b3-eaef877e862b%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

Kiswono Prayogo

<kiswono@gmail.com>
unread,
Jun 23, 2017, 8:14:07 AM6/23/17
to ScyllaDB users
That's already the full log from journalctl.

> email to scylladb-users+unsubscribe@googlegroups.com.
> To post to this group, send email to scylladb-users@googlegroups.com.
--
You received this message because you are subscribed to the Google Groups "ScyllaDB users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scylladb-users+unsubscribe@googlegroups.com.
To post to this group, send email to scylladb-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--
Best Regards,
Kiswono P
GB

Glauber Costa

<glauber@scylladb.com>
unread,
Jun 23, 2017, 9:25:13 AM6/23/17
to ScyllaDB users
That is strange.

The log doesn't really say anything. Maybe it wasn't scylla that
failed, but one of its dependencies?

systemctl status scylla-server right after a failed run should tell you that
>> > email to scylladb-user...@googlegroups.com.
>> > To post to this group, send email to scyllad...@googlegroups.com.
>> > Visit this group at https://groups.google.com/group/scylladb-users.
>> > To view this discussion on the web visit
>> >
>> > https://groups.google.com/d/msgid/scylladb-users/3e7a7f8b-53c1-4284-85b3-eaef877e862b%40googlegroups.com.
>> > For more options, visit https://groups.google.com/d/optout.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "ScyllaDB users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to scylladb-user...@googlegroups.com.
>> To post to this group, send email to scyllad...@googlegroups.com.
>> Visit this group at https://groups.google.com/group/scylladb-users.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/scylladb-users/CAD-J%3DzYQ-zjch%3DJ0%3DSX%3D9edLViS0A3NweKtZ%2BOzfYjnFK6OLzw%40mail.gmail.com.
>> For more options, visit https://groups.google.com/d/optout.
>
>
>
>
> --
> Best Regards,
> Kiswono P
> GB
>
> --
> You received this message because you are subscribed to the Google Groups
> "ScyllaDB users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to scylladb-user...@googlegroups.com.
> To post to this group, send email to scyllad...@googlegroups.com.
> Visit this group at https://groups.google.com/group/scylladb-users.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/scylladb-users/CANiF8TQP6PwdRPY6B%2B4gs7wmNP%3DOWKVAEWZor1qVeg11bywKhw%40mail.gmail.com.

Kiswono Prayogo

<kiswono@gmail.com>
unread,
Jun 23, 2017, 9:26:31 AM6/23/17
to ScyllaDB users
It shows this:

# systemctl status scylla-server
scylla-server.service - Scylla Server
  Loaded: loaded (/lib/systemd/system/scylla-server.service; enabled; vendor preset: enabled)
  Active: failed (Result: exit-code) since Fri 2017-06-23 19:53:49 WIB; 32min ago
Main PID: 1643 (code=exited, status=71)
     CPU: 379ms

Jun 23 19:53:47 ter scylla_prepare[1131]: Setting mask 00000003 in /sys/class/net/ens3/queues/tx-0/xps_cpus
Jun 23 19:53:48 ter scylla_prepare[1131]: tuning /sys/devices/pci0000:00/0000:00:01.1/ata1/host0/target0:0:0/0:0:0:0/block/sda/sda1
Jun 23 19:53:48 ter scylla_prepare[1131]: tuning /sys/devices/pci0000:00/0000:00:01.1/ata1/host0/target0:0:0/0:0:0:0/block/sda
Jun 23 19:53:48 ter scylla_prepare[1131]: tuning: /sys/devices/pci0000:00/0000:00:01.1/ata1/host0/target0:0:0/0:0:0:0/block/sda/queue/scheduler noop
Jun 23 19:53:48 ter scylla_prepare[1131]: tuning: /sys/devices/pci0000:00/0000:00:01.1/ata1/host0/target0:0:0/0:0:0:0/block/sda/queue/nomerges 2
Jun 23 19:53:48 ter scylla_prepare[1131]: tuning /sys/devices/pci0000:00/0000:00:01.1/ata1/host0/target0:0:0/0:0:0:0/block/sda/sda1
Jun 23 19:53:49 ter systemd[1]: scylla-server.service: Main process exited, code=exited, status=71/n/a
Jun 23 19:53:49 ter systemd[1]: Failed to start Scylla Server.
Jun 23 19:53:49 ter systemd[1]: scylla-server.service: Unit entered failed state.
Jun 23 19:53:49 ter systemd[1]: scylla-server.service: Failed with result 'exit-code'.


>> > email to scylladb-users+unsubscribe@googlegroups.com.
>> > To post to this group, send email to scylladb-users@googlegroups.com.

>> > Visit this group at https://groups.google.com/group/scylladb-users.
>> > To view this discussion on the web visit
>> >
>> > https://groups.google.com/d/msgid/scylladb-users/3e7a7f8b-53c1-4284-85b3-eaef877e862b%40googlegroups.com.
>> > For more options, visit https://groups.google.com/d/optout.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "ScyllaDB users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to scylladb-users+unsubscribe@googlegroups.com.
>> To post to this group, send email to scylladb-users@googlegroups.com.

>> Visit this group at https://groups.google.com/group/scylladb-users.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/scylladb-users/CAD-J%3DzYQ-zjch%3DJ0%3DSX%3D9edLViS0A3NweKtZ%2BOzfYjnFK6OLzw%40mail.gmail.com.
>> For more options, visit https://groups.google.com/d/optout.
>
>
>
>
> --
> Best Regards,
> Kiswono P
> GB
>
> --
> You received this message because you are subscribed to the Google Groups
> "ScyllaDB users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to scylladb-users+unsubscribe@googlegroups.com.
> To post to this group, send email to scylladb-users@googlegroups.com.

> Visit this group at https://groups.google.com/group/scylladb-users.
> To view this discussion on the web visit
>
> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "ScyllaDB users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scylladb-users+unsubscribe@googlegroups.com.
To post to this group, send email to scylladb-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Tomer Sandler

<tomer@scylladb.com>
unread,
Jun 23, 2017, 9:37:20 AM6/23/17
to scylladb-users@googlegroups.com
Ubuntu 17.04 is not a supported version. Only 14.04 + 16.04 are supported

--
Tomer Sandler
ScyllaDB

(Sent from my android)

--

Glauber Costa

<glauber@scylladb.com>
unread,
Jun 23, 2017, 9:52:34 AM6/23/17
to ScyllaDB users
reviewing your original instructions:


>
> sudo sed -i 's|/usr/bin/scylla $SCYLLA_ARGS|/usr/bin/scylla -m 1G -c 1
> --developer-mode 1 $SCYLLA_ARGS|g' /lib/systemd/system/scylla-server.service

Why are you doing the above?

Changing the systemd file directly is not recommended.
To begin with, when that is done you need to reload it with systemctl
daemon-reload (did you do that?)

But also that just invite errors - systemd is quite sensitive...
> --
> You received this message because you are subscribed to the Google Groups
> "ScyllaDB users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to scylladb-user...@googlegroups.com.
> To post to this group, send email to scyllad...@googlegroups.com.

Kiswono Prayogo

<kiswono@gmail.com>
unread,
Jun 23, 2017, 10:38:48 AM6/23/17
to ScyllaDB users
well, this works on my pc and another one that that use the same OS..

So where should i change those config?

And yes, daemon-reload is a must, because if i dont do that, it would complain that i must do that before start/restart service..


> email to scylladb-users+unsubscribe@googlegroups.com.
> To post to this group, send email to scylladb-users@googlegroups.com.
--
You received this message because you are subscribed to the Google Groups "ScyllaDB users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scylladb-users+unsubscribe@googlegroups.com.
To post to this group, send email to scylladb-users@googlegroups.com.

Kiswono Prayogo

<kiswono@gmail.com>
unread,
Jun 23, 2017, 11:10:09 AM6/23/17
to ScyllaDB users
From the error code 71, i looked at the source code
    if (!cpu_sanity()) {
       _exit(71);
    }

so my cpu on that computer not supported?

CPU(s):    2 Single core QEMU Virtual version (cpu64-rhel6)s (-HT-SMP-) cache: 8192 KB 
           clock speeds: max: 2399 MHz 1: 2399 MHz 2: 2399 MHz


> To post to this group, send email to scyllad...@googlegroups.com.

> Visit this group at https://groups.google.com/group/scylladb-users.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/scylladb-users/3e7a7f8b-53c1-4284-85b3-eaef877e862b%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "ScyllaDB users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scylladb-users+unsubscribe@googlegroups.com.
To post to this group, send email to scyllad...@googlegroups.com.

Glauber Costa

<glauber@scylladb.com>
unread,
Jun 23, 2017, 11:31:15 AM6/23/17
to ScyllaDB users
On Fri, Jun 23, 2017 at 11:10 AM, Kiswono Prayogo <kis...@gmail.com> wrote:
> From the error code 71, i looked at the source code
> if (!cpu_sanity()) {
> _exit(71);
> }
>
> so my cpu on that computer not supported?
>
> CPU(s): 2 Single core QEMU Virtual version (cpu64-rhel6)s (-HT-SMP-)
> cache: 8192 KB
> clock speeds: max: 2399 MHz 1: 2399 MHz 2: 2399 MHz
>

Interesting. This should have printed a message. I guess the message
goes to stderr, and systemd is not picking up those.

We have a patch in our master branch that will teach systemd to
properly redirect stderr messages to the logger - then analysing
things like this will be easier.
>> > email to scylladb-user...@googlegroups.com.
>> > To post to this group, send email to scyllad...@googlegroups.com.
>> > Visit this group at https://groups.google.com/group/scylladb-users.
>> > To view this discussion on the web visit
>> >
>> > https://groups.google.com/d/msgid/scylladb-users/3e7a7f8b-53c1-4284-85b3-eaef877e862b%40googlegroups.com.
>> > For more options, visit https://groups.google.com/d/optout.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "ScyllaDB users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to scylladb-user...@googlegroups.com.
>> To post to this group, send email to scyllad...@googlegroups.com.
>> Visit this group at https://groups.google.com/group/scylladb-users.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/scylladb-users/CAD-J%3DzawyS-4otyoeVwEEa9z9xVidwJyGuDM4LeyFm3ZtnE8pA%40mail.gmail.com.
>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>
>
>
> --
> Best Regards,
> Kiswono P
> GB
>
> --
> You received this message because you are subscribed to the Google Groups
> "ScyllaDB users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to scylladb-user...@googlegroups.com.
> To post to this group, send email to scyllad...@googlegroups.com.
> Visit this group at https://groups.google.com/group/scylladb-users.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/scylladb-users/CANiF8TT%2BCRebd%2BmzzMSi7LdULPiS1Gj11%2BXvG7ZMU28Osz5-bA%40mail.gmail.com.

Avi Kivity

<avi@scylladb.com>
unread,
Jun 29, 2017, 4:46:41 AM6/29/17
to scylladb-users@googlegroups.com, Kiswono Prayogo

Scylla needs sse4.2 (the sse4_2 flag in /proc/cpuinfo).


Adjust your virtual machine configuration to enable that feature in the virtual CPU. If you are using Virtual Machine Manager, you can do that in the CPUs tab, under Configuration, select "Copy host CPU configuration".

To unsubscribe from this group and stop receiving emails from it, send an email to scylladb-user...@googlegroups.com.

To post to this group, send email to scyllad...@googlegroups.com.
Visit this group at https://groups.google.com/group/scylladb-users.
Reply all
Reply to author
Forward
0 new messages