Quadstor incredibly slow/not working after unclean shutdown

[b.n...@goldspark.solutions]

I am running Quadstor Storage Virtualization 3.2.12.1 on Debian 7 as iSCSI target for my XenServer 7 servers. After an unclean shutdown, Quadstor suddenly slows down the entire machine to a grinding halt, up to a point where there are multiple messages that 'gdevq' blocked for more than 120 seconds. The XenServers also cannot access the files on the Quadstor machine.

After uninstalling Quadstor, the machine is as fast as it has always been. Reinstalling Quadstor didn't change that, but after rescanning the database, all the symptoms came back and I still can't access my data.

Does anyone have any idea what to do? I can't access any of my data at the moment.

[QUADStor Support]

Please send the diagnostics to sup...@quadstor.com

What was the reason for the shutdown ? gdevq is the thread which
processes incoming requests, and if that is blocked it could indicate
a problem with the underlying storage.

> --
> You received this message because you are subscribed to the Google Groups
> "QUADStor Storage Virtualization" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to quadstor-vir...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[b.n...@goldspark.solutions]

Due to the load, I unfortunately can't access the GUI. Is there another way to create the diagnostics?

The shutdown was due to a power outage. The underlying storage is a hardware RAID 1 on PERC 6/i; the RAID BIOS tells me everything is fine with both disks. After booting the server after the power outage, the RAID controller was busy writing it's BBU cache to disk for approx. a minute.

Op dinsdag 5 september 2017 13:45:54 UTC+2 schreef quadstor:

Please send the diagnostics to sup...@quadstor.com

What was the reason for the shutdown ? gdevq is the thread which
processes incoming requests, and if that is blocked it could indicate
a problem with the underlying storage.

[QUADStor Support]

Please send across the following files

/var/log/dmesg*
/var/log/kern.log*
/var/log/syslog*

And the output of
dmesg
top -b -d 1 -n 60

[b.n...@goldspark.solutions]

What's the status on the fix? The server just crashed and rebooting is going to take a couple of hours again due to this bug. This is unacceptable in a production environment.

Op dinsdag 5 september 2017 14:07:58 UTC+2 schreef quadstor:

Please send across the following files

/var/log/dmesg*
/var/log/kern.log*
/var/log/syslog*

And the output of
dmesg
top -b -d 1 -n 60

[QUADStor Support]

Do you have a stack trace for the crash. We are interested in the
reason for the crash.

The problem was attributed to the slowness in loading the
deduplication tables. When a master disk is added to a pool the
initial set of deduplication tables are sequential on disk and these
are quickly loaded back into memory on a reboot. The size of these
tables are around 1/8 the size of the memory. The second set of
deduplication tables are added on demand when more table entries are
needed and the first set of tables are full. The problem is with the
second set, the location of these tables can be spread across the
disk. For your setup as per our calculations around 2GB are the
sequential tables and 14GB are the second set. The disk is a RAID 1
disk and it seems that with just the max speed of two disks reading
back 14GB of data which are mainly 4K blocks can take many minutes to
hours.

There are two things which we are doing.
1. Create all the deduplication tables sequentially. This will mean
that only one storage pool can maintain the deduplication tables
(mostly the Default pool). Most configurations are this way => This
change is complete
2. For existing implementation we realign the random blocks to
sequential blocks using a command line tool => This is still being
implemented and ETA is the end of this month. You would need this
feature

[b.n...@goldspark.solutions]

I unfortunately do not have a stack trace. It seems however that the crash itself was not necessarily caused by the Quadstor software. The real problem lies within the fact that rebooting the server (for any reason) is impossible because it takes hours to come back up.

I can confirm the issue is caused by the loading of the deduplication tables; ddloadt is taking up 99,99% of IO according to iotop.

As there is nothing I can do about it, I will just have to wait for the deduplication tables to be loaded. I look forward to the release of the new command line tool.

Op maandag 6 november 2017 12:08:37 UTC+1 schreef quadstor:

Do you have a stack trace for the crash. We are interested in the
reason for the crash.

The problem was attributed to the slowness in loading the
deduplication tables. When a master disk is added to a pool the
initial set of deduplication tables are sequential on disk and these
are quickly loaded back into memory on a reboot. The size of these
tables are around 1/8 the size of the memory. The second set of
deduplication tables are added on demand when more table entries are
needed and the first set of tables are full. The problem is with the
second set, the location of these tables can be spread across the
disk. For your setup as per our calculations around 2GB are the
sequential tables and 14GB are the second set. The disk is a RAID 1
disk and it seems that with just the max speed of two disks reading
back 14GB of data which are mainly 4K blocks can take many minutes to
hours.

There are two things which we are doing.
1. Create all the deduplication tables sequentially. This will mean
that only one storage pool can maintain the deduplication tables
(mostly the Default pool). Most configurations are this way => This
change is complete
2. For existing implementation we realign the random blocks to
sequential blocks using a command line tool => This is still being
implemented and ETA is the end of this month. You would need this
feature

[QUADStor Support]

Please try 3.2.13 from
http://www.quadstor.com/storage-virtualization-enterprise-edition-downloads.html

The procedure will be

1. Uninstall the old package (dpkg -r quadstor-virt)
2. Install the new package (dpkg -i
quadstor-virt-3.2.13-debian7-x86_64.deb). Do not restart or attempt to
start the quadstor service as that will take the usual amount of time.
3. /quadstor/pgsql/etc/pgsql start
4. echo "update sysinfo set dalign=1" | /quadstor/pgsbin/psql -U scdbuser qsdb
5. service quadstor start

A few points to remember
a. Please ensure that there are backups done
b. The downtime will be atleast twice the time taken to start the
quadstor service
c. Send us the diagnostic logs after step 5
d. In case you run into issues leave the system as is (no
restart/reboot etc) and send us an email to sup...@quadstor.com

[QUADStor Support]

On Thu, Nov 30, 2017 at 3:53 PM, QUADStor Support <sup...@quadstor.com> wrote:

> 4. echo "update sysinfo set dalign=1" | /quadstor/pgsbin/psql -U scdbuser qsdb

This should be

echo "update sysinfo set dalign=1" | /quadstor/pgsql/bin/psql -U scdbuser qsdb