Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss
Re: Very slow Backup (Many little file in many subdirectories)
9 views
Skip to first unread message
DrumDude
Feb 12, 2010, 9:36:24 AM2/12/10
to
Matt Karwowski wrote:
> - I have a Netware 6.5 SP8 server with the post SP8 NSS patch installed
> (NSS version 3.27.01 April 7, 2009)
> - iSCSI Initiator Version 1.06.05 October 30, 2008 connecting a 410 GB
> NSS Volume (Lefthand Networks iSCSI SAN)
> - the volume contains user data, and contains, from the Root, a USERDATA
> directory, which then contains 32,749 individual user directories.
> - The majority (90%) of the user directories are empty
> - each user directory has it's own trustee assignment, and space
> restrictions
> - compression is enabled
> - 376,269 Files using 45,556,460,661 bytes in 221,439 directories so the
> average would be around 121K per file and as you can see it's very
> Directory intensive
>
> TSATEST, indicates approximately 850 MB/min result, but when I attempt
> to back up the data using my backup software (NetVault, which
> uses NDMP), the performance is hideous. A Full Backup runs around
> 135k/sec or 8 MB/min (try getting that done in any size backup
> window) . I have also tried other backup solutions, with the same basic
> result.
>
> I assume the issue is with indexing, but I'm not sure what to check at
> this point. I've been trying every suggestion I could find. I've
> gotten the throughput up to about 1.5M/s, but obviously need
> better. Just wanted to know if anyone here had any suggestions that
> might help me to make this function more efficiently, or any TID's that
> have proven helpful.
>
> Thank You!
> - I have a Netware 6.5 SP8 server with the post SP8 NSS patch installed
> (NSS version 3.27.01 April 7, 2009)
> - iSCSI Initiator Version 1.06.05 October 30, 2008 connecting a 410 GB
> NSS Volume (Lefthand Networks iSCSI SAN)
> - the volume contains user data, and contains, from the Root, a USERDATA
> directory, which then contains 32,749 individual user directories.
> - The majority (90%) of the user directories are empty
> - each user directory has it's own trustee assignment, and space
> restrictions
> - compression is enabled
> - 376,269 Files using 45,556,460,661 bytes in 221,439 directories so the
> average would be around 121K per file and as you can see it's very
> Directory intensive
>
> TSATEST, indicates approximately 850 MB/min result, but when I attempt
> to back up the data using my backup software (NetVault, which
> uses NDMP), the performance is hideous. A Full Backup runs around
> 135k/sec or 8 MB/min (try getting that done in any size backup
> window) . I have also tried other backup solutions, with the same basic
> result.
>
> I assume the issue is with indexing, but I'm not sure what to check at
> this point. I've been trying every suggestion I could find. I've
> gotten the throughput up to about 1.5M/s, but obviously need
> better. Just wanted to know if anyone here had any suggestions that
> might help me to make this function more efficiently, or any TID's that
> have proven helpful.
>
> Thank You!
Known issue....
http://www.novell.com/support/viewContent.do?externalId=3341011&sliceId=1
I have seen this in Java development shops with millions of 1kb files
and folders. The nature of Java.
- Use tar or zip on the local server,not across the wire to compress
those types of data folders.
- Exclude those folders from backup
- Then backup only the zip or tar files.
or maybe look into rsync as I think is copies by hardware blocks not
files?????
- maybe need toreview your environment if it is a development shop like
Java.Maybe switch to GPFS or Reiser for many small files if you can, EXT
could handle it for your small amount of data though.
0 new messages