Backup issues due to assetstore size
55 views
Skip to first unread message
Michael White
Aug 11, 2020, 9:34:52 AM8/11/20
to DSpace Tech
Hi,
Our DSpace instance has been steadily growing over the years and now has over 18,000 records, all with one or more full text files attached (taking up around 106GB of disk space). We are on DSpace 6.2 and currently only have one assetstore configured.
Whilst we have no issues with dspace operation/performance, my System and Network colleagues are reporting problems when their scheduled backup jobs of the DSpace server are trying to run as they are timing out, which they suspect is due to the large number of files on the server.
To resolve this, they have asked me about the possibility of splitting the current dspace filesystem in to a number of smaller filesystems (which they can then back up in parallel, reducing the overall time to back up the dspace server).
To that end they have asked about the possibility of splitting the assetstore in to a number of filesystems and creating softlinks to these from the assetstore - e.g. adding 9 new filesystems for clusters of 10 assetstore subdirectories - i.e.:
.../dspace/assetstore/10 -> .../assetstores10-19/10
.../dspace/assetstore/11 -> .../assetstores10-19/11
.../dspace/assetstore/12 -> .../assetstores10-19/12
.... ...
.../dspace/assetstore/20 -> ../assetstores20-29/20
Has anyone ever done anything like this? Any reasons why that wouldn't work?
I can, of course, add another assetstore alongside the existing one, but my understanding is that this would only be used going forwards, and so wouldn't address the large number of files in the current assetstore - unless there is a way to then distribute the existing assetstore across a number of smaller assetstores?
Has anyone else in the community had issues like this? And, if so, any (other) suggestions on how best to address the problem?
Cheers,
Mike
|
Michael White Business Applications and Integrations
University of Stirling |
|
|
|
|
The University achieved an overall 5 stars in the QS World University Rankings 2020
UK Sports University of the Year 2020 (Times Higher Good University Guide)
The University of Stirling is a charity registered in Scotland, number SC 011159.
Mark H. Wood
Aug 11, 2020, 11:07:38 AM8/11/20
to DSpace Tech
On Tue, Aug 11, 2020 at 01:34:48PM +0000, Michael White wrote:
> Our DSpace instance has been steadily growing over the years and now has over 18,000 records, all with one or more full text files attached (taking up around 106GB of disk space). We are on DSpace 6.2 and currently only have one assetstore configured.
>
> Whilst we have no issues with dspace operation/performance, my System and Network colleagues are reporting problems when their scheduled backup jobs of the DSpace server are trying to run as they are timing out, which they suspect is due to the large number of files on the server.
>
> To resolve this, they have asked me about the possibility of splitting the current dspace filesystem in to a number of smaller filesystems (which they can then back up in parallel, reducing the overall time to back up the dspace server).
>
> To that end they have asked about the possibility of splitting the assetstore in to a number of filesystems and creating softlinks to these from the assetstore - e.g. adding 9 new filesystems for clusters of 10 assetstore subdirectories - i.e.:
>
> .../dspace/assetstore/10 -> .../assetstores10-19/10
> .../dspace/assetstore/11 -> .../assetstores10-19/11
> .../dspace/assetstore/12 -> .../assetstores10-19/12
> .... ...
> .../dspace/assetstore/20 -> ../assetstores20-29/20
>
> Has anyone ever done anything like this? Any reasons why that wouldn't work?
I see no obvious reason why it wouldn't work.
> Our DSpace instance has been steadily growing over the years and now has over 18,000 records, all with one or more full text files attached (taking up around 106GB of disk space). We are on DSpace 6.2 and currently only have one assetstore configured.
>
> Whilst we have no issues with dspace operation/performance, my System and Network colleagues are reporting problems when their scheduled backup jobs of the DSpace server are trying to run as they are timing out, which they suspect is due to the large number of files on the server.
>
> To resolve this, they have asked me about the possibility of splitting the current dspace filesystem in to a number of smaller filesystems (which they can then back up in parallel, reducing the overall time to back up the dspace server).
>
> To that end they have asked about the possibility of splitting the assetstore in to a number of filesystems and creating softlinks to these from the assetstore - e.g. adding 9 new filesystems for clusters of 10 assetstore subdirectories - i.e.:
>
> .../dspace/assetstore/10 -> .../assetstores10-19/10
> .../dspace/assetstore/11 -> .../assetstores10-19/11
> .../dspace/assetstore/12 -> .../assetstores10-19/12
> .... ...
> .../dspace/assetstore/20 -> ../assetstores20-29/20
>
> Has anyone ever done anything like this? Any reasons why that wouldn't work?
"They" must think that the backup storage device is the bottleneck, if
they believe that parallel backups will improve throughput. I would
do some exploring to see if this is true, before doing a lot of work
to redistribute all those files. Is there another lurking bottleneck,
nearly as narrow, currently covered by the one they are working on?
Each of those ten volumes will continue to grow, but at 1/10 the
current rate, so this puts the problem off for some time but doesn't
solve it in the long run. That will give you some time to ponder
whether there is a smarter way to back up your repository.
> I can, of course, add another assetstore alongside the existing one, but my understanding is that this would only be used going forwards, and so wouldn't address the large number of files in the current assetstore - unless there is a way to then distribute the existing assetstore across a number of smaller assetstores?
"current assetstore".
There is a "BitStore migration tool" which moves *all* content from one
assetstore to another:
https://wiki.lyrasis.org/display/DSDOC6x/Storage+Layer#StorageLayer-MigrateBitStores
That isn't quite what you want, but it might serve as a model for a
tool to split up an assetstore.
If you do go this route, you might look at changing the way you do
backup. The materials in the "old" assetstores are probably subject
to very infrequent deletions but are otherwise static storage -- all
new and replacement content goes into the "current" assetstore. You
might devise a very different backup schedule for the old assetstores
vs. current.
--
Mark H. Wood
Lead Technology Analyst
University Library
Indiana University - Purdue University Indianapolis
755 W. Michigan Street
Indianapolis, IN 46202
317-274-0749
www.ulib.iupui.edu
Tony Brian Albers
Aug 12, 2020, 1:59:09 AM8/12/20
to dspac...@googlegroups.com, mwood...@gmail.com
assetstore itself instead. That would make it easier to administrate.
>
> I see no obvious reason why it wouldn't work.
>
> "They" must think that the backup storage device is the bottleneck,
> if
> they believe that parallel backups will improve throughput.
What I think they want to achieve is a higher client parallelism. This
means that instead of one save stream running on one file system they
can have several savestreams running, one on each filesystem. If they
had backup storage performance issues that would not make sense.
So, going for higher parallelism by having several smaller file systems
makes sense.
> I would
> do some exploring to see if this is true, before doing a lot of work
> to redistribute all those files. Is there another lurking
> bottleneck,
> nearly as narrow, currently covered by the one they are working on?
traffic. Can the network?
>
> Each of those ten volumes will continue to grow, but at 1/10 the
> current rate, so this puts the problem off for some time but doesn't
> solve it in the long run. That will give you some time to ponder
> whether there is a smarter way to back up your repository.
>
> > I can, of course, add another assetstore alongside the existing
> > one, but my understanding is that this would only be used going
> > forwards, and so wouldn't address the large number of files in the
> > current assetstore - unless there is a way to then distribute the
> > existing assetstore across a number of smaller assetstores?
>
> Your understanding is correct: all new submissions go into the
> "current assetstore".
>
> There is a "BitStore migration tool" which moves *all* content from
> one
> assetstore to another:
>
> https://wiki.lyrasis.org/display/DSDOC6x/Storage+Layer#StorageLayer-MigrateBitStores
>
> That isn't quite what you want, but it might serve as a model for a
> tool to split up an assetstore.
>
> If you do go this route, you might look at changing the way you do
> backup. The materials in the "old" assetstores are probably subject
> to very infrequent deletions but are otherwise static storage -- all
> new and replacement content goes into the "current" assetstore. You
> might devise a very different backup schedule for the old assetstores
> vs. current.
once or twice a year combined with incremental and cumulative
incremental backups.
I'd suggest something like this:
.../dspace/assetstore/10 --mountpoint, filesystem10
.../dspace/assetstore/11 --mountpoint, filesystem11
.../dspace/assetstore/12 --mountpoint, filesystem12
.... ...
.../dspace/assetstore/20 --mountpoint, filesystem20
That would do for the few houndreds of gigabytes you have. If you had >
10x that, I'd probably go for something like ZFS with replication and
snapshot backup
Good luck,
/tony
>
> --
> Mark H. Wood
> Lead Technology Analyst
>
> University Library
> Indiana University - Purdue University Indianapolis
> 755 W. Michigan Street
> Indianapolis, IN 46202
> 317-274-0749
> www.ulib.iupui.edu
>
Tony Albers - Systems Architect - IT Development
Royal Danish Library, Victor Albecks Vej 1, 8000 Aarhus C, Denmark
Tel: +45 2566 2383 - CVR/SE: 2898 8842 - EAN: 5798000792142
Mark H. Wood
Aug 12, 2020, 9:14:56 AM8/12/20
to dspac...@googlegroups.com
first-level assetstore directories are 00 through 99. They are named
by the first two decimal digits of a checksum of the Bitstream's content.
> >
> > I see no obvious reason why it wouldn't work.
> >
> > "They" must think that the backup storage device is the bottleneck,
> > if
> > they believe that parallel backups will improve throughput.
>
> Ehm, not following you here, Mark.
> What I think they want to achieve is a higher client parallelism. This
> means that instead of one save stream running on one file system they
> can have several savestreams running, one on each filesystem. If they
> had backup storage performance issues that would not make sense.
> So, going for higher parallelism by having several smaller file systems
> makes sense.
the bottleneck is CPU or memory, higher parallelism doesn't help. If
the bottleneck is disk, higher parallelism only helps if those ten
volumes are on separate physical disks. (Where do you buy 10GB disks
these days?) Same for the storage controller(s). That left me to
consider the backup system itself, where the slowest thing in the
entire process, by far, is the tape drive(s).
The problem *could* be the local disks. I'd want to prove that first.
Tony Brian Albers
Aug 13, 2020, 1:44:20 AM8/13/20
to dspac...@googlegroups.com, mwood...@gmail.com
I hope I'm not being annoying, please bear with me.
I'd like to explain how parallelism actually works in some(not all)
backup systems. See my answers below if you're interested.
On Wed, 2020-08-12 at 09:14 -0400, Mark H. Wood wrote:
>
>
> If the bottleneck is network, higher parallelism doesn't help.
Right, it won't.
> If
> the bottleneck is CPU or memory, higher parallelism doesn't help.
Yes, but that really depends on whether the CPU/MEM is actually used
properly.
> If
> the bottleneck is disk, higher parallelism only helps if those ten
> volumes are on separate physical disks.
Not necessarily. In general backup software scans the file system
looking for changes etc. before actually starting to stream data to the
backup storage. Often the scan and the save stream are handled by
single-threaded processes who traverses the file and folder structure,
and in this case where we have a large number of files, this will take
a large amount of time even though the disk might not be struggling at
all. But it will still be seen as a disk bottleneck.
By employing several scanning and streaming processes on the same file
system/disk, we can actually speed things up a lot.
For instance, EMC NetWorker version > 8.1 actually can do parallel save
streams in one backup job on the same file system. I've used this a
number of times in situations like this and it helps a lot.
> (Where do you buy 10GB disks
> these days?) Same for the storage controller(s). That left me to
> consider the backup system itself, where the slowest thing in the
> entire process, by far, is the tape drive(s).
If they use tape drives and write directly to them, that might be true.
It's hard to utilize a tape drive's performance by saving directly to
it from a client. However, for sequential reads/writes tape is often
extremely fast and can easily outperform a RAID-5 storage system of the
same capacity. LTO-8 for example has a compressed write speed of
900MB/sec.
So if you have a fast temporary storage area on the backup system, you
can stream to that from the client and let the backup system dump it to
tape and save huge amounts of time.
>
> The problem *could* be the local disks. I'd want to prove that
> first.
It actually could. And that's probably a good place to start since it's
quite easy to check.
/tony
(who is actually a certified EMC NetWorker Specialist - Implementation
Engineer)
>
> --
> Mark H. Wood
> Lead Technology Analyst
>
> University Library
> Indiana University - Purdue University Indianapolis
> 755 W. Michigan Street
> Indianapolis, IN 46202
> 317-274-0749
> www.ulib.iupui.edu
>
I'd like to explain how parallelism actually works in some(not all)
backup systems. See my answers below if you're interested.
On Wed, 2020-08-12 at 09:14 -0400, Mark H. Wood wrote:
>
>
> If the bottleneck is network, higher parallelism doesn't help.
> If
> the bottleneck is CPU or memory, higher parallelism doesn't help.
properly.
> If
> the bottleneck is disk, higher parallelism only helps if those ten
> volumes are on separate physical disks.
looking for changes etc. before actually starting to stream data to the
backup storage. Often the scan and the save stream are handled by
single-threaded processes who traverses the file and folder structure,
and in this case where we have a large number of files, this will take
a large amount of time even though the disk might not be struggling at
all. But it will still be seen as a disk bottleneck.
By employing several scanning and streaming processes on the same file
system/disk, we can actually speed things up a lot.
For instance, EMC NetWorker version > 8.1 actually can do parallel save
streams in one backup job on the same file system. I've used this a
number of times in situations like this and it helps a lot.
> (Where do you buy 10GB disks
> these days?) Same for the storage controller(s). That left me to
> consider the backup system itself, where the slowest thing in the
> entire process, by far, is the tape drive(s).
It's hard to utilize a tape drive's performance by saving directly to
it from a client. However, for sequential reads/writes tape is often
extremely fast and can easily outperform a RAID-5 storage system of the
same capacity. LTO-8 for example has a compressed write speed of
900MB/sec.
So if you have a fast temporary storage area on the backup system, you
can stream to that from the client and let the backup system dump it to
tape and save huge amounts of time.
>
> The problem *could* be the local disks. I'd want to prove that
> first.
quite easy to check.
/tony
(who is actually a certified EMC NetWorker Specialist - Implementation
Engineer)
>
> --
> Mark H. Wood
> Lead Technology Analyst
>
> University Library
> Indiana University - Purdue University Indianapolis
> 755 W. Michigan Street
> Indianapolis, IN 46202
> 317-274-0749
> www.ulib.iupui.edu
>
dc...@prosentient.com.au
Aug 13, 2020, 2:07:52 AM8/13/20
to Tony Brian Albers, dspac...@googlegroups.com, mwood...@gmail.com
Hey Tony,
I haven't been following this thread very closely, but I enjoyed your email.
I have some personal experience using backup software that scans and saves in parallel and it's wonderful. Last time I looked, it took about 30 seconds on a 10 year old system with 2 CPUs and 1 HDD to scan over 30,000 files for changes on an incremental backup.
I wonder what software the OP's IT staff are using that it's timing out...
David Cook
Software Engineer
Prosentient Systems
72/330 Wattle St
Ultimo, NSW 2007
Australia
Office: 02 9212 0899
Online: 02 8005 0595
--
All messages to this mailing list should adhere to the DuraSpace Code of Conduct: https://duraspace.org/about/policies/code-of-conduct/
---
You received this message because you are subscribed to the Google Groups "DSpace Technical Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dspace-tech...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dspace-tech/ada0a42220c81cfbd244fe6741e93225d5f76e35.camel%40kb.dk.
I haven't been following this thread very closely, but I enjoyed your email.
I have some personal experience using backup software that scans and saves in parallel and it's wonderful. Last time I looked, it took about 30 seconds on a 10 year old system with 2 CPUs and 1 HDD to scan over 30,000 files for changes on an incremental backup.
I wonder what software the OP's IT staff are using that it's timing out...
David Cook
Software Engineer
Prosentient Systems
72/330 Wattle St
Ultimo, NSW 2007
Australia
Office: 02 9212 0899
Online: 02 8005 0595
All messages to this mailing list should adhere to the DuraSpace Code of Conduct: https://duraspace.org/about/policies/code-of-conduct/
---
You received this message because you are subscribed to the Google Groups "DSpace Technical Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dspace-tech...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dspace-tech/ada0a42220c81cfbd244fe6741e93225d5f76e35.camel%40kb.dk.
Michael White
Aug 13, 2020, 3:46:15 AM8/13/20
to DSpace Technical Support
Hi,
I just want to thank folk for the replies and discussion around this issue - I will pass all this on to my network colleague and hopefully it will give him some food for thought and a way forwards.
Cheers,
Mike
Alan Orth
Aug 18, 2020, 3:30:54 AM8/18/20
to Michael White, DSpace Technical Support
Dear Michael,
Your assetstore doesn't seem unpractically large to me. We have ~90,000 items in our DSpace repository and our assetstore looks like this:
# du -sh /home/dspace/assetstore/
76G /home/dspace/assetstore/
# find /home/dspace/assetstore/ -type f | wc -l
143062
76G /home/dspace/assetstore/
# find /home/dspace/assetstore/ -type f | wc -l
143062
We do a simple `s3cmd sync` to Amazon S3 every night and have never, ever had an issue in ten years. I regularly rsync our production assetstore to dev and it's super fast... The overwhelming majority of your assetstore won't change every day so backups should only copy new files and those that have changed. If your colleagues are doing file-level backups it should be trivial to only sync what has changed...? BTW even rsync can be parallelized¹ with a little bit of shell scripting.
Regards,
--
All messages to this mailing list should adhere to the DuraSpace Code of Conduct: https://duraspace.org/about/policies/code-of-conduct/
---
You received this message because you are subscribed to the Google Groups "DSpace Technical Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dspace-tech...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dspace-tech/99a08863-e58e-4954-9cfe-79b5fabe5f3bo%40googlegroups.com.
--
Reply all
Reply to author
Forward
0 new messages