Out of space
98 views
Skip to first unread message
Sergey Kamyanov
Jun 5, 2019, 4:13:35 AM6/5/19
to QUADStor VTL
Hello, could you please explain what Quadstor VTL actually does in case of out of space condition on the storage pools and how to bring it back online after that?
The only solution I found is to restart the service which takes a lot of time (~30 min) due to dd tables dump and load.
The only solution I found is to restart the service which takes a lot of time (~30 min) due to dd tables dump and load.
QUADStor VTL Support
Jun 5, 2019, 4:32:48 AM6/5/19
to quadstor-vtl
Hi,
Service shutdown/load time has been addressed in 3.0.35
https://www.quadstor.com/vtl-extended-edition-downloads.html
Please generate diagnostics logs before and after the upgrade
The shutdown or load can be slow based on the speed of the master disk
and in older installations where all the deduplication tables are not
created upfront.
When an out-of-space condition occurs, any pending writes are
terminated with error. From the backup application point of view the
running backups will fail. But the devices should still be accessible
and things can continue either by deleting/erasing older/expired
tapes, or by adding more storage to the pool. What exactly is the
behavior seen ?
On Wed, Jun 5, 2019 at 1:43 PM Sergey Kamyanov <s.kam...@gmail.com> wrote:
>
> Hello, could you please explain what Quadstor VTL actually does in case of out of space condition on the storage pools and how to bring it back online after that?
>
> The only solution I found is to restart the service which takes a lot of time (~30 min) due to dd tables dump and load.
>
> --
> Virtual Tape Library (VTL) https://www.quadstor.com/virtual-tape-library.html
> VTL software download https://www.quadstor.com/vtl-extended-edition-downloads.html
> VTL documentation https://www.quadstor.com/vtl-documentation.html
> ---
> You received this message because you are subscribed to the Google Groups "QUADStor VTL" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to quadstor-vtl...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/quadstor-vtl/656e8aa3-0a74-4a8e-8c04-8c2e23cd9547%40googlegroups.com.
Service shutdown/load time has been addressed in 3.0.35
https://www.quadstor.com/vtl-extended-edition-downloads.html
Please generate diagnostics logs before and after the upgrade
The shutdown or load can be slow based on the speed of the master disk
and in older installations where all the deduplication tables are not
created upfront.
When an out-of-space condition occurs, any pending writes are
terminated with error. From the backup application point of view the
running backups will fail. But the devices should still be accessible
and things can continue either by deleting/erasing older/expired
tapes, or by adding more storage to the pool. What exactly is the
behavior seen ?
On Wed, Jun 5, 2019 at 1:43 PM Sergey Kamyanov <s.kam...@gmail.com> wrote:
>
> Hello, could you please explain what Quadstor VTL actually does in case of out of space condition on the storage pools and how to bring it back online after that?
>
> The only solution I found is to restart the service which takes a lot of time (~30 min) due to dd tables dump and load.
>
> Virtual Tape Library (VTL) https://www.quadstor.com/virtual-tape-library.html
> VTL software download https://www.quadstor.com/vtl-extended-edition-downloads.html
> VTL documentation https://www.quadstor.com/vtl-documentation.html
> ---
> You received this message because you are subscribed to the Google Groups "QUADStor VTL" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to quadstor-vtl...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/quadstor-vtl/656e8aa3-0a74-4a8e-8c04-8c2e23cd9547%40googlegroups.com.
Sergey Kamyanov
Jun 5, 2019, 4:45:26 AM6/5/19
to QUADStor VTL
I'm running 3.0.33 now, will schedule the update window soon.
The master disk is not so fast for random io (12x 4T NLSAS on Netapp).
I see about 2k iops and 100% disk load on dd tables load.
As for the behavior - all drives just go offline in TSM and the only way to bring them back online is quadstor service restart.
On Wednesday, June 5, 2019 at 11:32:48 AM UTC+3, quadstor wrote:
Hi,
Service shutdown/load time has been addressed in 3.0.35
https://www.quadstor.com/vtl-extended-edition-downloads.html
Please generate diagnostics logs before and after the upgrade
The shutdown or load can be slow based on the speed of the master disk
and in older installations where all the deduplication tables are not
created upfront.
When an out-of-space condition occurs, any pending writes are
terminated with error. From the backup application point of view the
running backups will fail. But the devices should still be accessible
and things can continue either by deleting/erasing older/expired
tapes, or by adding more storage to the pool. What exactly is the
behavior seen ?
On Wed, Jun 5, 2019 at 1:43 PM Sergey Kamyanov <s.ka...@gmail.com> wrote:
>
> Hello, could you please explain what Quadstor VTL actually does in case of out of space condition on the storage pools and how to bring it back online after that?
>
> The only solution I found is to restart the service which takes a lot of time (~30 min) due to dd tables dump and load.
>
> --
> Virtual Tape Library (VTL) https://www.quadstor.com/virtual-tape-library.html
> VTL software download https://www.quadstor.com/vtl-extended-edition-downloads.html
> VTL documentation https://www.quadstor.com/vtl-documentation.html
> ---
> You received this message because you are subscribed to the Google Groups "QUADStor VTL" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to quadst...@googlegroups.com.
QUADStor VTL Support
Jun 5, 2019, 5:38:54 AM6/5/19
to quadstor-vtl
On Wed, Jun 5, 2019 at 2:15 PM Sergey Kamyanov <s.kam...@gmail.com> wrote:
>
> I'm running 3.0.33 now, will schedule the update window soon.
> The master disk is not so fast for random io (12x 4T NLSAS on Netapp).
> I see about 2k iops and 100% disk load on dd tables load.
This is addressed in 3.0.35. Please generate the diagnostic logs
>
> I'm running 3.0.33 now, will schedule the update window soon.
> The master disk is not so fast for random io (12x 4T NLSAS on Netapp).
> I see about 2k iops and 100% disk load on dd tables load.
(HTML UI -> System Page -> Run Diagnostics) before and after the
upgrade and send them across to vtlsu...@quadstor.com
>
> As for the behavior - all drives just go offline in TSM and the only way to bring them back online is quadstor service restart.
i.e the /dev/IBMtape[n] or similar devices should be accessible
The next time this occurs does a manual "UPDATE PATH" work ?
For example
update path QUADSTOR1 drive0 srctype=server desttype=drive
library=lib0 device=/dev/IBMtape0 online=yes
The command is similar to the define path command but with the
trailing "online=yes"
Follow up with a "q path" command to see if all paths are online
> To view this discussion on the web visit https://groups.google.com/d/msgid/quadstor-vtl/dcf6c57f-7c7e-4ba6-92cd-119be0b5a01c%40googlegroups.com.
Sergey Kamyanov
Jun 16, 2019, 4:41:31 PM6/16/19
to QUADStor VTL
Happened again :)
update drive/path works but drives are immediately going offline/polling again.
Total Size: 45056.00 GB
Used Size: 36170.66 GB
VCartridge Usage: 104433.43 GB
Deduped Size: 12290.02 GB
Uncompressed Size: 1254.45 GB
Compressed Size: 34155.99 GB
Compression Hits: 90888.96 GB
Dedupe Ratio: 2.949
messages on the vtl server:
Jun 16 05:58:05 srv-vtl-00 vtmdaemon: WARN: node_usr_notify:700 bdev_alloc_for_pgdata:5317 Cannot find a free bint for tape VT0161L4 size 512
Jun 16 05:58:05 srv-vtl-00 vtmdaemon: WARN: node_usr_notify:700 tape_alloc_blocks:2796 pgdata alloc for entry size 512 failed
Jun 16 05:58:05 srv-vtl-00 kernel: WARN: bdev_alloc_for_pgdata:5317 Cannot find a free bint for tape VT0161L4 size 512
Jun 16 05:58:05 srv-vtl-00 kernel: WARN: tape_alloc_blocks:2796 pgdata alloc for entry size 512 failed
Jun 16 05:58:05 srv-vtl-00 kernel: WARN: bdev_alloc_for_pgdata:5317 Cannot find a free bint for tape VT0172L4 size 512
Jun 16 05:58:05 srv-vtl-00 kernel: WARN: tape_alloc_blocks:2796 pgdata alloc for entry size 512 failed
Jun 16 05:58:05 srv-vtl-00 vtmdaemon: WARN: node_usr_notify:700 bdev_alloc_for_pgdata:5317 Cannot find a free bint for tape VT0172L4 size 512
Jun 16 05:58:05 srv-vtl-00 vtmdaemon: WARN: node_usr_notify:700 tape_alloc_blocks:2796 pgdata alloc for entry size 512 failed
Jun 16 05:58:05 srv-vtl-00 vtmdaemon: WARN: node_usr_notify:700 bdev_alloc_for_pgdata:5317 Cannot find a free bint for tape VT0173L4 size 512
Jun 16 05:58:05 srv-vtl-00 kernel: WARN: bdev_alloc_for_pgdata:5317 Cannot find a free bint for tape VT0173L4 size 512
Jun 16 05:58:05 srv-vtl-00 kernel: WARN: tape_alloc_blocks:2796 pgdata alloc for entry size 512 failed
Updating 3.0.33 to 3.0.35 now. Will send before and after logs to the support address.
Forgot to mention that the server began to crash reboot during service stop since 3.0.33 update.
On Wednesday, June 5, 2019 at 12:38:54 PM UTC+3, quadstor wrote:
QUADStor VTL Support
Jun 16, 2019, 4:46:13 PM6/16/19
to quadstor-vtl
Before update path is the problem with the out-of-space condition
fixed ? i.e either additional storage needs to be added or some of the
older tapes need to be expired+erased.
For the crash on shutdown please check if there are any
vmcore-dmesg.txt files under /var/crash and send them across.
fixed ? i.e either additional storage needs to be added or some of the
older tapes need to be expired+erased.
For the crash on shutdown please check if there are any
vmcore-dmesg.txt files under /var/crash and send them across.
> --
> You received this message because you are subscribed to the Google Groups "QUADStor VTL" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to quadstor-vtl...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/quadstor-vtl/3844308b-2902-4fe4-b77a-fb33847c5a34%40googlegroups.com.
> You received this message because you are subscribed to the Google Groups "QUADStor VTL" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to quadstor-vtl...@googlegroups.com.
Sergey Kamyanov
Jun 17, 2019, 6:25:39 AM6/17/19
to QUADStor VTL
No, i have done nothing except VTL service restart.
This normally happens during 10TB DB2 or 14TB Oracle backup in the condition when I have about backup size or slightly more (Total minus Used) free space on VTL, I believe TSM somehow removes failed objects itself.
Numbers from my previous message (Total Size: 45056.00 GB / Used Size: 36170.66 GB) was during the 'offline drives', about 9TB free, TSM was trying to write 14TB Oracle database L0 backup.
Now it is (Total Size: 45056.00 GB / Used Size: 30487.29 GB)
Can't wait my TS3310 to be fixed so i can return to 'move everything older than one week to the physical tapes' scheme :)
I have older crash dumps from May and nothing from June, can upload somewhere if you want.
1.5G 127.0.0.1-2019-05-12-04:05:54
1.2G 127.0.0.1-2019-05-22-23:01:33
8.9G 127.0.0.1-2019-05-31-20:06:04
QUADStor VTL Support
Jun 17, 2019, 6:43:44 AM6/17/19
to quadstor-vtl
Hi,
Under /var/crash there would be smaller vmcore-dmesg.txt files. These
should be small and contain the stack trace for the crash. Please send
them to vtlsu...@quadstor.com.
For update path to work we would need that an out-of-space condition
is first cleared at the VTL side
When we bring the drive paths online, TSM will most likely retry the
job and then encounter the same out-of-space condition and then mark
the paths offline again. If this is not the case we will try this out
at our end.
The restart of the VTL service is not required be here in our opinion.
If TSM is deleting objects and the space is therefore reclaimed at the
VTL side, then update path should be run after space has been
reclaimed by the VTL and then things should work.
Please also check if RELABELSCRATCH is set to YES for the library. If
not then please run the following command
update library <library> RELABELSCRATCH=YES
With RELABELSCRATCH set to YES when a tape is moved from private to
scratch, an erase command is also sent for the tape.
Under /var/crash there would be smaller vmcore-dmesg.txt files. These
should be small and contain the stack trace for the crash. Please send
them to vtlsu...@quadstor.com.
For update path to work we would need that an out-of-space condition
is first cleared at the VTL side
When we bring the drive paths online, TSM will most likely retry the
job and then encounter the same out-of-space condition and then mark
the paths offline again. If this is not the case we will try this out
at our end.
The restart of the VTL service is not required be here in our opinion.
If TSM is deleting objects and the space is therefore reclaimed at the
VTL side, then update path should be run after space has been
reclaimed by the VTL and then things should work.
Please also check if RELABELSCRATCH is set to YES for the library. If
not then please run the following command
update library <library> RELABELSCRATCH=YES
With RELABELSCRATCH set to YES when a tape is moved from private to
scratch, an erase command is also sent for the tape.
> --
> You received this message because you are subscribed to the Google Groups "QUADStor VTL" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to quadstor-vtl...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/quadstor-vtl/9fa3a5f2-98ef-4442-9870-abb23bcfaf37%40googlegroups.com.
> You received this message because you are subscribed to the Google Groups "QUADStor VTL" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to quadstor-vtl...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages