Big Directory file and its performance

[Joe]

Hello All,

In our environment, we have a scratch directory in which on an average 3000 files are getting created and moved or deleted on a daily basis.Currently, this directory is hosting around 80,000 files which are of past 60 days.

Today I created a blank file and did a test backup and please find the output below for the time taken.

NODE09 > Show time
1-AUG-2013 11:43:02
NODE09 > BACKUP/LOG/NEW_VERSION/NOVERIFY LOGICAL_ROOT:[SCRATCH]TEST_ORIG.RPT; LOGICAL_RETAIN:*.*
%BACKUP-S-CREATED, created PRISMWORK_ROOT:[RETAIN]TEST_ORIG.RPT;4
NODE09 > show time
1-AUG-2013 11:52:45

Directory output:
dir LOGICAL_ROOT:[000000]scratch.dir/full

Directory LOGICAL_ROOT:[000000]

SCRATCH.DIR;1 File ID: (32,1,0)
Size: 11725/21845 Owner: [XXX_XXX,XXX_XXX]
Created: 28-APR-1996 16:35:00.52
Revised: 1-AUG-2013 13:14:30.28 (29217)
Expires: 1-AUG-2013 13:14:30.29
Backup: 27-JUL-2013 18:28:42.69
Effective: <None specified>
Recording: <None specified>
Accessed: <None specified>
Attributes: <None specified>
Modified: <None specified>
Linkcount: 1
File organization: Sequential
Shelved state: Online
Caching attribute: Writethrough
File attributes: Allocation: 21845, Extend: 0, Global buffer count: 0, No default version limit, Contiguous, Directory file
Record format: Variable length, maximum 512 bytes, longest 512 bytes
Record attributes: No carriage control, Non-spanned
RMS attributes: None
Journaling enabled: None
File protection: System:RWED, Owner:RWED, Group:RE, World:RE
Access Cntrl List: (IDENTIFIER=XXXXXX_READ_ONLY,ACCESS=READ+EXECUTE)
(IDENTIFIER=XXXXXX_READ_ONLY,OPTIONS=DEFAULT,ACCESS=READ+EXECUTE)
(IDENTIFIER=XXXXXX_ADMIN,OPTIONS=DEFAULT,ACCESS=READ+WRITE+DELETE+CONTROL)
(IDENTIFIER=XXXXXX_ADMIN,ACCESS=READ+WRITE)
Client attributes: None

Total of 1 file, 11725/21845 blocks.

Now, Im in process of moving more files out of this directory and putting in sub Directories.

Please help me with the below questions:

If I move the files and free up the directory, will the response time decrease automatically? Or Anyfurther action needs to be taken to improve this?

Is there a tool available to measure the response time of a directory file?

How do I identify that a directory is taking a long time to respond? (say if I'm clearing the files today and after a week if the same issue occurs, I want to put some script to notify me)


Some Background:

Disk DSA1240:, device type Generic SCSI disk ------ Its a two member shadow
Members:
Disk $1$DGA1395:, device type EMC SYMMETRIX,
Disk $1$DGA1983:, device type EMC SYMMETRIX,

Total blocks 94396800
Free blocks 45623665
Mount Count 8
Its a mixed cluster with 5 Alpha and 4 Itanium - One Alpha is just a quorum node
OpenVMS V8.4

All other directories in the same disk is responding fine.

I ran Analyze/disk/norepair and got the below warnings.
%ANALDISK-W-BAD_NAMEORDER, filename ordering incorrect in VBN 1489 -- There are multiple entries similar to this
%ANALDISK-W-BADDIRENT, invalid file identification in directory entry
%ANALDISK-W-BADHEADER, file (139726,281,0)
invalid file header
%ANALDISK-W-BADHEADER, file (140192,4,0)
invalid file header
%ANALDISK-W-BADHEADER, file (140200,26,0)
invalid file header
%ANALDISK-W-BADHEADER, file (140201,17,0)
invalid file header


If I just give $ dire/Grand , I'm getting response in around 2 seconds. But If I give any additional operation like /Since or /before, then it takes a lot of time to respond around 30 seconds.

Please provide your inputs

Thanks,
Joseph

[Simon Clubley]

On 2013-08-01, Joe <joslo...@gmail.com> wrote:
>
> Some Background:
>
> Disk DSA1240:, device type Generic SCSI disk ------ Its a two member shadow
> Members:
> Disk $1$DGA1395:, device type EMC SYMMETRIX,
> Disk $1$DGA1983:, device type EMC SYMMETRIX,
>
> Total blocks 94396800
> Free blocks 45623665
> Mount Count 8
> Its a mixed cluster with 5 Alpha and 4 Itanium - One Alpha is just a quorum node
> OpenVMS V8.4
>

Are all the machines running V8.4 ?

> All other directories in the same disk is responding fine.
>
> I ran Analyze/disk/norepair and got the below warnings.
> %ANALDISK-W-BAD_NAMEORDER, filename ordering incorrect in VBN 1489 -- There are multiple entries similar to this
> %ANALDISK-W-BADDIRENT, invalid file identification in directory entry
> %ANALDISK-W-BADHEADER, file (139726,281,0)
> invalid file header
> %ANALDISK-W-BADHEADER, file (140192,4,0)
> invalid file header
> %ANALDISK-W-BADHEADER, file (140200,26,0)
> invalid file header
> %ANALDISK-W-BADHEADER, file (140201,17,0)
> invalid file header
>

There is a known directory corruption problem which it looks like you may
be experiencing. You _really_ need to read through the previous threads
on this subject here in comp.os.vms as a matter of urgency.

Simon.

--
Simon Clubley, clubley@remove_me.eisner.decus.org-Earth.UFP
Microsoft: Bringing you 1980s technology to a 21st century world

[Joe]

> Are all the machines running V8.4 ?

Yes, all are running 8.4

> There is a known directory corruption problem which it looks like you may
> be experiencing. You _really_ need to read through the previous threads
> on this subject here in comp.os.vms as a matter of urgency.

oh ok, thanks for you suggestion. I will start checking the same now.

[abrsvc]

A few questions about your initial backup test and directory commands:

1) The target directory ( PRISMWORK_ROOT:[RETAIN]) is highwater marking turned on?
If so, then the creation of the file may be taking time rather than either
the directory acceses or the data transfer.
2) With the addition of "other" parameters to the /grand, additional processing
of entry information is required other than just the file size. Thus, the
more processing, the longer the time to report.

In general, the most obvious directory functions are usually seen during mass file delete operations. This is due to the way that empty directory blocks are handled. Large delete operations are more efficient if the files are deleted in reverse diretory list order. This too has been discussed many times here in comp.os.vms

Dan

[Joe]

On Thursday, August 1, 2013 6:31:02 PM UTC+5:30, abrsvc wrote:
> A few questions about your initial backup test and directory commands:
>
>
>
> 1) The target directory ( PRISMWORK_ROOT:[RETAIN]) is highwater marking turned on?
>
> If so, then the creation of the file may be taking time rather than either
>
> the directory acceses or the data transfer.
>

NO, High water marking is not turned on

Volume Status: ODS-2, subject to mount verification, write-through XFC
caching enabled, write-back XQP caching enabled.

> 2) With the addition of "other" parameters to the /grand, additional processing
>
> of entry information is required other than just the file size. Thus, the
>
> more processing, the longer the time to report.
>

Ok, so to scan through the directory file and get the output with specific condition it is taking that much time. And I guess 30 seconds is because of some corruption in the directory file, correct me if I'm wrong.

>
>
> In general, the most obvious directory functions are usually seen during mass file delete operations. This is due to the way that empty directory blocks are handled. Large delete operations are more efficient if the files are deleted in reverse diretory list order. This too has been discussed many times here in comp.os.vms

Ok, thanks, I will continue to go through them.

>
>
>
> Dan

[Simon Clubley]

On 2013-08-01, Joe <joslo...@gmail.com> wrote:
>

> Ok, so to scan through the directory file and get the output with specific
> condition it is taking that much time. And I guess 30 seconds is because of
> some corruption in the directory file, correct me if I'm wrong.
>

The directory corruption (if that's what it does turn out to be) may or may
not be related to your performance issues. However, it's the first thing
you need to verify and fix (before looking at the performance issues).

One thing I do when running analyze/disk on a active volume is to run it
about 3 times and only take account of the error messages which show up in
all 3 reports.

That tells me which errors are genuine and which are false alarms caused
by the disk volume changing while analyze/disk is being run.

If you have not done so already, you should run analyze/disk multiple
times and make sure the directory errors show up in each report.

[Simon Clubley]

On 2013-08-01, Joe <joslo...@gmail.com> wrote:

[snip]

>
> Total of 1 file, 11725/21845 blocks.
>
> Now, Im in process of moving more files out of this directory and putting in sub Directories.
>
> Please help me with the below questions:
>
> If I move the files and free up the directory, will the response time
> decrease automatically? Or Anyfurther action needs to be taken to improve
> this?
>

As the files are moved out of the directory, you should find it continues to
shrink in size. However, 11725 blocks is still a large directory, which needs
to be scanned every time you walk the directory.

If you moving files out of the directory in strict alphabetical order,
or in otherwise contiguous alphabetical chunks, then the directory blocks
left should be pretty tightly filled already.

However, if you are moving by, say, filetype, then the remaining directory
blocks may be sparsely filled with directory entries and you could try
recreating the directory from scratch to make the directory file itself
more compact.

The downside of that is that you may see some initial performance issues
as the directory file expands once again to handle newly added files.

[Stephen Hoffman]

On 2013-08-01 11:55:29 +0000, Joe said:

> In our environment, we have a scratch directory in which on an average
> 3000 files are getting created and moved or deleted on a daily
> basis.Currently, this directory is hosting around 80,000 files which
> are of past 60 days.

> ....

> dir LOGICAL_ROOT:[000000]scratch.dir/full
>
> Directory LOGICAL_ROOT:[000000]
>
> SCRATCH.DIR;1 File ID: (32,1,0)
> Size: 11725/21845 Owner: [XXX_XXX,XXX_XXX]

> ...
> Please provide your inputs

Make yourself the proud parent of a flock of little scratch directories.

Either all of them on one disk if there's not a whole lot of I/O load
involved, or scattered around around various disk spindles if the disks
are busier.

Spreading the I/O load and the storage does depend on what processes
are using the scratch directories, and how; details not in evidence.

Make yourself a reaper procedure, too. Some DCL procedure that
automatically deletes any scratch storage files older than some value,
rotates log files, looks for files with versions approaching the limit,
and the rest of the usual nightly server maintenance tasks.

I'd also consider move non-critical scratch storage off of a volume set
(barring full-on HBVS licenses and a surfeit of disk storage and disk
slots), but that's another discussion.

Patch to current, too, as there are some file system bugs around.


--
Pure Personal Opinion | HoffmanLabs LLC

[Keith Parris]

On 8/1/2013 5:55 AM, Joe wrote:
> If I move the files and free up the directory,
> will the response time decrease automatically?

> Or any further action needs to be taken to improve this?

It is the number of in-use blocks (not allocated blocks)
which affects directory performance. So the improvement
in performance will be automatic once you clear the
directory out.

> Is there a tool available to measure the response
> time of a directory file?
>
> How do I identify that a directory is taking a long
> time to respond? (say if I'm clearing the files today
> and after a week if the same issue occurs, I want to
> put some script to notify me)

I like to use lock contention (lock queues) as a proactive
indicator of potential future performance problems.
Availability Manager can detect lock contention (if you
enable lock contention data collection), and will report
lock contention events, which you can find in the Event
window or after-the-fact in the AMDS$LOCK_LOG.LOG file.

Or you can use the LCKQUE.COM tool from the V6 Freeware
CD directory [KP_LOCKTOOLS] to sample for lock queues.
A directory that is growing and starting to be a
performance bottleneck will start to generate lock
queues on its file serialization lock (F11B$s prefix).

[glen herrmannsfeldt]

Joe <joslo...@gmail.com> wrote:

> In our environment, we have a scratch directory in which on
> an average 3000 files are getting created and moved or deleted
> on a daily basis.Currently, this directory is hosting
> around 80,000 files which are of past 60 days.

I am not sure how VMS directories work internally, but at 80,000
it will run slow on many unix systems.

Unix traditionally does linear search through directories.

-- glen

[JF Mezei]

Not sure if applicable to your case:

Spread files into multiple directories with basically a searchlist to
find them.

say you have temp01.dir temp02.dir.... temp99.dir

You create your logical "temp" as a searchlist that points to all
tempxx.dir files.

To populate the directories, you can have logical such as tempnew which
points to a specific directory and is changed to point to another
directory every hour or whenever time period you want. (another way to
do this is to "increment" tempnew whenever a new file is created, and
whyen it reaches the limit, you reset tempnew to pointo to temp01.dir).
This way, you have round robbin file distribution in all directories.

This allows you to function with minimal changes to your code, while
spreading the temporary file load between multiple directories, reducing
directory handling overhead. But in exchange, you do have searchlist
overhead to lookup a file.

[David Froble]

Joe wrote:
> Hello All,
>
> In our environment, we have a scratch directory in which on an
> average 3000 files are getting created and moved or deleted on a
> daily basis.Currently, this directory is hosting around 80,000 files
> which are of past 60 days.

While not being very productive at this time, it needs to be understood
that such a bad design should never have happened.

Devise a scheme for distributing the scratch files a bit more. Perhaps
subdirectories in the main scratch directory. Others have given some
hints also. Perhaps something such as "A", "B", "C", etc
subdirectories, and storing scratch files based upon the first character
in the filename. Or whatever works.

Some preparation for the following would be good. This assumes that you
can find a time when the scratch directory is not in use.

1) Rename the scratch directory to something else.
2) Create a new scratch directory.
3) Rename all the files into the newly designed multitude of scratch
directories / subdiectories.
4) Delete the old renamed scratch directory

or

1) Make a backup save set of the directory
2) Delete the directory and all files
3) Restore

But the latter doesn't address breaking up your scratch space into
something reasonable.

[Paul Sture]

In article <kte53v$b73$1...@dont-email.me>,

Simon Clubley <clubley@remove_me.eisner.decus.org-Earth.UFP> wrote:

> On 2013-08-01, Joe <joslo...@gmail.com> wrote:
>
> [snip]
>
> >
> > Total of 1 file, 11725/21845 blocks.
> >
> > Now, Im in process of moving more files out of this directory and putting
> > in sub Directories.
> >
> > Please help me with the below questions:
> >
> > If I move the files and free up the directory, will the response time
> > decrease automatically? Or Anyfurther action needs to be taken to improve
> > this?
> >
>
> As the files are moved out of the directory, you should find it continues to
> shrink in size. However, 11725 blocks is still a large directory, which needs
> to be scanned every time you walk the directory.
>
> If you moving files out of the directory in strict alphabetical order,
> or in otherwise contiguous alphabetical chunks, then the directory blocks
> left should be pretty tightly filled already.
>
> However, if you are moving by, say, filetype, then the remaining directory
> blocks may be sparsely filled with directory entries and you could try
> recreating the directory from scratch to make the directory file itself
> more compact.

I haven't looked at this in detail since V4 days, but back then deleting
a single file near the beginning of the directory would cause the
remaining entries to be shuffled down. IIRC this would happen in kernel
mode, which meant that other processes would be affected.

> The downside of that is that you may see some initial performance issues
> as the directory file expands once again to handle newly added files.

--
Paul Sture

[Jan-Erik Soderholm]

Joe wrote 2013-08-01 13:55:
> Hello All,
>
> In our environment, we have a scratch directory in which on an average
> 3000 files are getting created and moved or deleted on a daily
> basis.Currently, this directory is hosting around 80,000 files which are
> of past 60 days.

How are these files "used"? Just saved "just in case" ?

That is aprox 1.300 files a day. You could have a batch
job that run after each midnight that ZIP's all files
from the day before (or the day 7 days before, if you
want a full week to be available). That will reduce
the number of files VMS has to take care of.

Then, when needed, you can unzip the files somewhere in
some temp-dir to run SEARCH (or whatever you need them for).
Then just empty this temp-dir until next time...

It all boils down to what the actual use of these files is!


Now, below you write that you are alrady moving these files
into subdirs, so maybe this is non-issue anyway (?).


Jan-Erik.


>
> Today I created a blank file and did a test backup...

Why? Did you see signs of some problems?

[Hein RMS van den Heuvel]

>> Currently, this directory is hosting around 80,000 files which are of past 60 days.

>> 11725/21845

That is well beyond the point the point of noticeably diminishing performance.
The math is really simple....

If a file with a name which sort early is added requiring the directory to make room, or if it is deleted creating an empty (512 byte) block, then the rest of the directory is shuffled up resp moved down in ACP_MAXREAD chunck.

ACP_MAXREAD is typically 32 blocks, so that 1 file will cause 11725/32=366 read and write IOs during which the directory is locked out and invalidating the cache on all other cluster nodes. The reads are likely from cache. Verify the size with MCR SYSGEN SHOW /ACP

> > Today I created a blank file and did a test backup and please find the output below for the time taken.

I do not understand that test. Is is unlikely to depend a lot on the directory size since is did a simple direct lookup. Is that backup command similar to how the directory is typically use? What is a blank file? Zero block, or many zero filled blocks ? How big?
So you are indicating it took almost 10 minutes to copy one file from the problem directory with backup?

Now backup takes a retarded approach... it reads the directory in chunks doing an access and de-access as it goes. It can easily require 400+ accesses for a 10,000+ block directory and if that directory is write accessed on other nodes I could see it take 'forever'. Still... 10 minutes?
How does 'COPY' perform for you?
It does a single lookup and lets the XQP deal with it.


Anyway... JFM has the right idea for now... turn in into a search list.
How do applications find/use the directory?
Can you create a fresh directory
$ CREATE/ALLO=2000 LOGICAL_ROOT:[SCRATCH1]
and next re-define the logical as
LOGICAL_ROOT:[SCRATCH1], LOGICAL_ROOT:[SCRATCH]

Existing files can still be found, but new ones will be create in the (pre-allocated) new directory.
Now start cleaning [SCRATCH] ... from HI to LO, Z to A
Repeat monthly, weekly or daily as needed.

I create a test directory with 55,000 files in 11,000 blocks with the scripts below. So 5 files/block. You'll see just a few IO's to delete files 1..4 in a block, but delete file # and the now empty the block will need 690 IOs to get expunged when early in the sort, bu just a few IOs late in the sort order.

More importantly... how does the application use the scratch directory?
Are files handed over from one process to the next with a strict design,
or is it perhaps just a 'free-for-all' zone out of the quota / backup realm?
Can you not just give each process its own scratch directory based on username or UIC or some modest grouping based on that?

> Now, Im in process of moving more files out of this directory and putting in sub Directories.

ok...

> If I move the files and free up the directory, will the response time decrease automatically? Or Anyfurther action needs to be taken to improve this?

Yes. No need to shrink the directory, only EOF size is considered.
The more files are 'spread' within those sub directories in the first few bytes of the name, the better the system works. The worst you can do is to have files like MAIL$xxx ... a sin OpenVMS itself made. You may want to consider per-node subdirectories or searchlist where the 'home node' comes first, yet others are there for incidental access as needed

> Is there a tool available to measure the response time of a directory file?

Just do it and measure?

It would vary too much due to unpredictable shuffles and cross cluster-node directory datablock cache invalidations
But be sure to do something similar to what the application uses.
Don't use BACKUP if the application uses COPY or DELETE.
Use Wildcards or not as the application would.
Keep context, as the application would (F$SEARCH ?)

> How do I identify that a directory is taking a long time to respond?
(say if I'm clearing the files today and after a week if the same issue occurs, I want to put some script to notify me)

If it is big then it is bad. Simple.

> Disk DSA1240:, device type Generic SCSI disk ------ Its a two member shadow

ok... so worst case write. Is one of the member geograpically distant?
Then 300 Io's could start to take 300*50 ms = 15 seconds... or worse.

>> Its a mixed cluster with 5 Alpha and 4 Itanium - One Alpha is just a quorum

> %ANALDISK-W-BAD_NAMEORDER, filename ordering incorrect in VBN 1489 -- There are multiple entries similar to this

Please urgently report to OpenVMS Engineering via your support channels.
There are currently a few customers with up to date patches which are experiencing directory corruptions and we have little to go expect for hi-activity and mix-cluster usage. Serious stuff. Really bad if true.

> If I just give $ dire/Grand , I'm getting response in around 2 seconds. But If I give any additional operation like /Since or /before, then it takes a lot of time to respond around 30 seconds.

That is absolutely normal and predictable. Learn to use DFU!

If you do a directory requiring anything other than the name then the system will have to access the file header. The header is a block in INDEXF.SYS where dates and size and actually mapping is recorded.
So a DIR/SINCE looking at 50,000 files will require 50,000 IOs. No ifs or buts.
If that is done in 30 seconds, then your IO system is performing about 1500 read IO's per second. Not bad.

Cheers,
Hein

ps... when testing this stuff, consider a RAM disk:

$ mcr sysman io connect mda1:/driver=sys$mddriver/noadap
$ init mda1:/size=50000 ram
$ moun /sys mda1: ram ram
%MOUNT-I-MOUNTED, RAM mounted on _EISNER$MDA1:
$ cre/dir ram:[test]
$ cre ram:[000000]tmp.tmp
$
$ create fill.com
$ i = 1
$ max = p1
$ if f$type(max).nes."INTEGER" then exit 16
$ loop:
$ SET FILE/ENT=RAM:[TEST]'f$fao("!6ZL!33*x.!39*y",i)' RAM:[000000]tmp.tmp
$ i = i + 1
$ if i .lt. max then goto loop
$
$ @fill 55000 ! 10,000,000+ IO's
$ dir/size=all ram:[000000]test.dir

$ @time delete ram:[test]000011*.*;* ! yes, SET FILE/REMOVE is quicker.
Dirio= 9 Bufio= 18 Kernel= 5 RMS= 11 DCL=0 User= 0 Elapsed= 17
$ @time delete ram:[test]000012*.*;*
Dirio= 3 Bufio= 18 Kernel= 5 RMS= 11 DCL=0 User= 0 Elapsed= 16
$ @time delete ram:[test]000013*.*;*
Dirio= 3 Bufio= 18 Kernel= 5 RMS= 11 DCL=0 User= 0 Elapsed= 16
$ @time delete ram:[test]000014*.*;*
Dirio= 3 Bufio= 18 Kernel= 4 RMS= 10 DCL=0 User= 0 Elapsed= 16
$ @time delete ram:[test]000015*.*;*/lo
Dirio= 691 Bufio= 18 Kernel= 10 RMS= 10 DCL=0 User= 0 Elapsed= 22
:
$ @time delete ram:[test]054011*.*;*
Dirio= 22 Bufio= 18 Kernel= 7 RMS= 18 DCL=0 User= 0 Elapsed= 25
$ @time delete ram:[test]054012*.*;*
Dirio= 16 Bufio= 18 Kernel= 7 RMS= 19 DCL=0 User= 0 Elapsed= 26
$ @time delete ram:[test]054013*.*;*
Dirio= 15 Bufio= 18 Kernel= 4 RMS= 18 DCL=0 User= 1 Elapsed= 26
$ @time delete ram:[test]054014*.*;*
Dirio= 14 Bufio= 18 Kernel= 6 RMS= 20 DCL=0 User= 0 Elapsed= 26
$ @time delete ram:[test]054015*.*;*
Dirio= 27 Bufio= 18 Kernel= 6 RMS= 18 DCL=0 User= 0 Elapsed= 26

[Joe]

On Thursday, August 1, 2013 10:14:42 PM UTC+5:30, Simon Clubley wrote:
> On 2013-08-01, Joe <joslo...@gmail.com> wrote:
>
> >
>
> > Ok, so to scan through the directory file and get the output with specific
>
> > condition it is taking that much time. And I guess 30 seconds is because of
>
> > some corruption in the directory file, correct me if I'm wrong.
>
> >
>
>
>
> The directory corruption (if that's what it does turn out to be) may or may
>
> not be related to your performance issues. However, it's the first thing
>
> you need to verify and fix (before looking at the performance issues).
>
>
>
> One thing I do when running analyze/disk on a active volume is to run it
>
> about 3 times and only take account of the error messages which show up in
>
> all 3 reports.
>
>
>
> That tells me which errors are genuine and which are false alarms caused
>
> by the disk volume changing while analyze/disk is being run.
>
>
>
> If you have not done so already, you should run analyze/disk multiple
>
> times and make sure the directory errors show up in each report.
>

We ran it only once, We will run it multiple times to get rid of any false alarms as you suggested. Thanks

[Joe]

>
> As the files are moved out of the directory, you should find it continues to
>
> shrink in size. However, 11725 blocks is still a large directory, which needs
>
> to be scanned every time you walk the directory.
>
>
>
> If you moving files out of the directory in strict alphabetical order,
>
> or in otherwise contiguous alphabetical chunks, then the directory blocks
>
> left should be pretty tightly filled already.
>
>
>
> However, if you are moving by, say, filetype, then the remaining directory
>
> blocks may be sparsely filled with directory entries and you could try
>
> recreating the directory from scratch to make the directory file itself
>
> more compact.
>
>
> The downside of that is that you may see some initial performance issues
>
> as the directory file expands once again to handle newly added files.
>
>
>

OK, I will try that.

[Joe]

>
> Make yourself the proud parent of a flock of little scratch directories.

Yes, I'm planning for the same. I have taken up this environment only a weeks ago.

>
>
>
> Either all of them on one disk if there's not a whole lot of I/O load
>
> involved, or scattered around around various disk spindles if the disks
>
> are busier.
>
>
>
> Spreading the I/O load and the storage does depend on what processes
>
> are using the scratch directories, and how; details not in evidence.
>
>

yes, sure.

>
> Patch to current, too, as there are some file system bugs around.
>

Yes, servers are patched to current.

[Joe]

>
> It is the number of in-use blocks (not allocated blocks)
>
> which affects directory performance. So the improvement
>
> in performance will be automatic once you clear the
>
> directory out.

oh ok, Thanks.

> I like to use lock contention (lock queues) as a proactive
>
> indicator of potential future performance problems.
>
> Availability Manager can detect lock contention (if you
>
> enable lock contention data collection), and will report
>
> lock contention events, which you can find in the Event
>
> window or after-the-fact in the AMDS$LOCK_LOG.LOG file.
>
>
>
> Or you can use the LCKQUE.COM tool from the V6 Freeware
>
> CD directory [KP_LOCKTOOLS] to sample for lock queues.
>
> A directory that is growing and starting to be a
>
> performance bottleneck will start to generate lock
>
> queues on its file serialization lock (F11B$s prefix).

We are using cockpitmgr to monitor this cluster, I will check, if its possible to enable to detect lock contention. Thanks

[Joe]

>
>
> Some preparation for the following would be good. This assumes that you
>
> can find a time when the scratch directory is not in use.
>
>
>
> 1) Rename the scratch directory to something else.
>
> 2) Create a new scratch directory.
>
> 3) Rename all the files into the newly designed multitude of scratch
>
> directories / subdiectories.
>
> 4) Delete the old renamed scratch directory
>

OK, Since the scratch directory is used by our applications many files are opened from scratch directory at any point of time.Will plan for this, when application is getting stopped or in between bounce.

[Joe]

> How are these files "used"? Just saved "just in case" ?

Our Applications write their ouput files, errorlogs and some temporary files here.
We use some DCL scripts and Exe's to scan through these files to get the required details and analyze the failures. We get requests to provide reports of these details up to 60 days old, that the reason the files are in here for 60 days.

>
>
>
> That is aprox 1.300 files a day. You could have a batch
>
> job that run after each midnight that ZIP's all files
>
> from the day before (or the day 7 days before, if you
>
> want a full week to be available). That will reduce
>
> the number of files VMS has to take care of.
>
>
>
> Then, when needed, you can unzip the files somewhere in
>
> some temp-dir to run SEARCH (or whatever you need them for).
>
> Then just empty this temp-dir until next time...
>
>
>
> It all boils down to what the actual use of these files is!
>
>
>
>
>
> Now, below you write that you are alrady moving these files
>
> into subdirs, so maybe this is non-issue anyway (?).

Yes, after I moved the files to sub directories, the response is better now. I want to implement a solution to this problem so that this issue doesn't come up again.

>
>
>
>
>
> Jan-Erik.
>
>
>
>
>
> >
>
> > Today I created a blank file and did a test backup...
>
>
>
> Why? Did you see signs of some problems?
>

Yes, one of the batch job was taking long time than usual and when I checked, it was at moving the old files to retain directory.

[Simon Clubley]

I don't know how it worked in the 4.x days, but from 5.4-3 (which is when
I started with VMS) it works differently on all the systems I have looked
at.

Imagine a 3 block directory with the following layout:

Block 1: Directory entries for A.TXT and A.PDF
Block 2: Directory entries for B.TXT and B.PDF
Block 3: Directory entries for C.TXT and C.PDF

Assume there are no other directory entries.

If you just delete A.TXT, no directory shuffling takes place and block 1
is left with just 1 directory (A.PDF).

If you delete A.*, then block 1 will contain zero directory entries and
hence blocks 2 and 3 will be shuffled down one block.

However, if you instead delete *.PDF, then all the directory blocks will
contain 1 directory entry each and no directory shuffling will take place.

It's this latter scenario which is of interest here. If the OP deletes
contiguous directory entries, leaving zero directory entries in a block,
then the directory will be shuffled down.

However, if the OP deletes files which are not packed together in the
directory, and hence on average leaves some directory entries in the
directory blocks, then you will see far less shuffling down of the
directory and the directory will become sparsely populated.

However, this sparsely populated directory still needs to be walked by
code performing directory operations and for large directories, the OP
might see a benefit from recreating the directory from scratch and hence
making it more densely populated and hence smaller in size.

The downside will be when you start expanding the directory again if
there are still a lot of files in the directory.

[Joe]

> ACP_MAXREAD is typically 32 blocks, so that 1 file will cause 11725/32=366 read and write IOs during which the directory is locked out and invalidating the cache on all other cluster nodes. The reads are likely from cache. Verify the size with MCR SYSGEN SHOW /ACP
>
>

yes, Its 32

MCR SYSGEN SHOW ACP_MAX
Parameter Name Current Default Min. Max. Unit Dynamic
-------------- ------- ------- ------- ------- ---- -------
ACP_MAXREAD 32 32 1 64 Blocks D

>
>
>
> > > Today I created a blank file and did a test backup and please find the output below for the time taken.
>
>
>
> I do not understand that test. Is is unlikely to depend a lot on the directory size since is did a simple direct lookup. Is that backup command similar to how the directory is typically use? What is a blank file? Zero block, or many zero filled blocks ? How big?

Yes, some of our batches does self housekeeping by taking backup of old files to retain directory before creating files.

Its a blank file with no content in it. 0 blocks in size.

>
> How does 'COPY' perform for you?
>
> It does a single lookup and lets the XQP deal with it.
>

Copy was also taking time, but I did not capture the exact time taken :(

>
>
>
> Anyway... JFM has the right idea for now... turn in into a search list.
>

Yes, planning for the same.

> How do applications find/use the directory?

Application just translates the logical name which is sitting in application table which is defined system wide and use the directory.

Actual application only writes to the scratch directory. There are other tools which actually scans throught the directory and reads and moves the files accordingly.

>
> More importantly... how does the application use the scratch directory?
>
> Are files handed over from one process to the next with a strict design,
>
> or is it perhaps just a 'free-for-all' zone out of the quota / backup realm?
>
> Can you not just give each process its own scratch directory based on username or UIC or some modest grouping based on that?

I will check this later.

> > %ANALDISK-W-BAD_NAMEORDER, filename ordering incorrect in VBN 1489 -- There are multiple entries similar to this
>
>
>
> Please urgently report to OpenVMS Engineering via your support channels.
>
> There are currently a few customers with up to date patches which are experiencing directory corruptions and we have little to go expect for hi-activity and mix-cluster usage. Serious stuff. Really bad if true.

Yes, ok, will raise a case with HP on the same.

>
>
>
> > If I just give $ dire/Grand , I'm getting response in around 2 seconds. But If I give any additional operation like /Since or /before, then it takes a lot of time to respond around 30 seconds.
>
>
>
> That is absolutely normal and predictable. Learn to use DFU!
>

OK

[VAXman-]

In article <bc96c0ea-163b-4405...@googlegroups.com>, Hein RMS van den Heuvel <heinvand...@gmail.com> writes:
>{...snip...}
>
>>> Its a mixed cluster with 5 Alpha and 4 Itanium - One Alpha is just a quo=
>rum=20
>> %ANALDISK-W-BAD_NAMEORDER, filename ordering incorrect in VBN 1489 -- Th=

>ere are multiple entries similar to this
>
>Please urgently report to OpenVMS Engineering via your support channels.

>There are currently a few customers with up to date patches which are exper=
>iencing directory corruptions and we have little to go expect for hi-activi=

>ty and mix-cluster usage. Serious stuff. Really bad if true.

Yeah! Yet another data point! Well, at least we know what it isn't! ;)

I'll have to spend some time perusing the sources. I have my suspicions,
but only a good read and head-check will prove or disprove their veracity.

--
VAXman- A Bored Certified VMS Kernel Mode Hacker VAXman(at)TMESIS(dot)ORG

Well I speak to machines with the voice of humanity.

[David Froble]

Joe wrote:
>> How are these files "used"? Just saved "just in case" ?
> Our Applications write their ouput files, errorlogs and some temporary files here.
> We use some DCL scripts and Exe's to scan through these files to get the required details and analyze the failures. We get requests to provide reports of these details up to 60 days old, that the reason the files are in here for 60 days.

If the date of the file creation is of any significance, you could have
subdirectories with names such as yyyymmdd which contain only files from
that particular day. Then when it comes time to delete that day's
files, you also delete the subdirectory. Doing so would avoid having
old directories with lots of churn.

1300 (someone else's number) isn't too many files for a directory.

Procedures for deleting files from high value to low value could be
helpful if you experience any issues with performance, but, if done in a
batch job in the middle of the night, who really cares if the deletions
take some time.

Another batch job which re-queues itself and runs at 0:01 could create
the new day's subdirectory. Having the applications check for the new
day's directory and cause it to be created if it doesn't exist would be
another design. Bit more overhead, bit less calls because the batch job
didn't run for some reason. Frankly, if something happens to your
mandatory batch jobs, then the entire system probably needs to be looked at.

[Stephen Hoffman]

On 2013-08-02 15:05:21 +0000, David Froble said:

> Procedures for deleting files from high value to low value could be
> helpful if you experience any issues with performance, but, if done in
> a batch job in the middle of the night, who really cares if the
> deletions take some time.

Or use the existing delete-on-close mechanism as the file disposition
setting for the temporary files. No subsequent regular clean-up is
then required.

http://h71000.www7.hp.com/doc/73final/4523/4523pro_005.html

If you're storing entirely scratch data and don't need to share the
stuff across processes and applications, then the scratch files can be
created entirely ex-directory; hidden.

Related: FAB$V_DLT, FAB$V_TMD, FAB$V_TMP, etc.

Once in a while, you'll need an ANALYZE /DISK /REPAIR pass to deal with
accumulated file system cruft, as is normal and typical with ODS-2 and
ODS-5 disks. In the case of these temporary and hidden files, the pass
will find and correct the effects of system crashes; for dealing with
the delete-on-close files that were open when VMS crashed.

[VAXman-]

In article <ktgme4$ve1$1...@dont-email.me>, Stephen Hoffman <seao...@hoffmanlabs.invalid> writes:
>On 2013-08-02 15:05:21 +0000, David Froble said:
>
>> Procedures for deleting files from high value to low value could be
>> helpful if you experience any issues with performance, but, if done in
>> a batch job in the middle of the night, who really cares if the
>> deletions take some time.
>
>Or use the existing delete-on-close mechanism as the file disposition
>setting for the temporary files. No subsequent regular clean-up is
>then required.
>
>http://h71000.www7.hp.com/doc/73final/4523/4523pro_005.html

Marking the file FAB$V_TMP as well, for no directory muddling, might help
ease the directory corruption pains too.

[Phillip Helbig---undress to reply]

In article <bc871fc8-5ec8-4025...@googlegroups.com>, Joe
<joslo...@gmail.com> writes:

> Hello All,
>
> In our environment, we have a scratch directory in which on an average 3000 files are getting created and moved or deleted on a daily basis.Currently, this directory is hosting around 80,000 files which are of past 60 days.

>
> Today I created a blank file and did a test backup and please find the output below for the time taken.

I don't know what creates your "scratch files", but for example Fortran
on VMS, by default, does not place scratch files in a directory at all.
They are automatically deleted by default as well, of course. If they
are true scratch files, then they are not needed after the application
exits anyway.

If you really need that many, then define a search-list logical for read
access, pointing to many directories, then have something periodically
redefine a logical for write access to point to one of these.

[Paul Sture]

In article <a1348902-a6a4-4763...@googlegroups.com>,

What I would do here is as others have suggested and use a search list.

The first item in a search list is the default for newly created files
so we can use this to our advantage. Here's a quick example:

-------------------------------------
$ sh def
SYS$SYSDEVICE:[PAUL]
$ cre /dir [.a]
$ cre /dir [.b]
$ def scratch SYS$SYSDEVICE:[PAUL.a],sys$sysdevice:[paul.b]
$ $ cre/log scratch:x.x
Exit
%CREATE-I-CREATED, SYS$SYSDEVICE:[PAUL.A]X.X;1 created
$ $ cre/log scratch:y.y
Exit
%CREATE-I-CREATED, SYS$SYSDEVICE:[PAUL.A]Y.Y;1 created

-------------------------------------
Note how the files were created in the first directory
of the search list.

Now we can change the search list on the fly and see
the new files get created in what is now the first member
of the search list
-------------------------------------
$ def scratch SYS$SYSDEVICE:[PAUL.b],sys$sysdevice:[paul.a]
$ cre/log scratch:z.z
Exit
%CREATE-I-CREATED, SYS$SYSDEVICE:[PAUL.B]Z.Z;1 created

-------------------------------------
and the directory entries look like this:
-------------------------------------

$ dir /da scratch

Directory SYS$SYSDEVICE:[PAUL.B]

Z.Z;1 2-AUG-2013 20:55:19.72

Total of 1 file.

Directory SYS$SYSDEVICE:[PAUL.A]

X.X;1 2-AUG-2013 20:53:56.61
Y.Y;1 2-AUG-2013 20:54:49.97

Total of 2 files.

Grand total of 2 directories, 3 files.
-------------------------------------

so we have changed the target directory for new files simply
by redefining the search list.

--
Paul Sture

[Paul Sture]

In article <51fabf37$0$5827$c3e8da3$5077...@news.astraweb.com>,

Agreed. By keeping the existing (huge) directory at the end of the
searchlist the older files will still be available for retrieval by the
applications. Once the system is implemented newer file lookups will be
satisfied by searching the newer and smaller directories, and by the end
of 60+ days the large directory should be no longer needed.

--
Paul Sture

[Paul Sture]

In article <ktg5r9$11r$1...@dont-email.me>,

Thanks for the clear explanation.

--
Paul Sture

[Paul Sture]

In article <ktghb2$15o$1...@dont-email.me>,

David Froble <da...@tsoft-inc.com> wrote:

> Another batch job which re-queues itself and runs at 0:01 could create
> the new day's subdirectory. Having the applications check for the new
> day's directory and cause it to be created if it doesn't exist would be
> another design. Bit more overhead, bit less calls because the batch job
> didn't run for some reason. Frankly, if something happens to your
> mandatory batch jobs, then the entire system probably needs to be looked at.

Since we are talking about clusters here, you need to be careful about
time drift between cluster members.

$ SUBMIT /AFTER="TOMORROW+00:01:00"

can go horribly wrong if the batch job gets executed on a different
cluster member from the submitting node and those two systems have times
which differ by a minute or more. I have variously used 5 or more
minutes here to be on the safe side.

Yes, I know there's NTP or DTSS in the case of DECnet Phase V, but both
those have been known to break somewhere.

--
Paul Sture

[Paul Sture]

In article <kth3g1$ed0$1...@online.de>,
hel...@astro.multiCLOTHESvax.de (Phillip Helbig---undress to reply)
wrote:

> In article <bc871fc8-5ec8-4025...@googlegroups.com>, Joe
> <joslo...@gmail.com> writes:
>
> > Hello All,
> >
> > In our environment, we have a scratch directory in which on an average
> > 3000 files are getting created and moved or deleted on a daily
> > basis.Currently, this directory is hosting around 80,000 files which are
> > of past 60 days.
> >
> > Today I created a blank file and did a test backup and please find the
> > output below for the time taken.
>
> I don't know what creates your "scratch files", but for example Fortran
> on VMS, by default, does not place scratch files in a directory at all.
> They are automatically deleted by default as well, of course. If they
> are true scratch files, then they are not needed after the application
> exits anyway.

As I understand it, the idea here is to be able to retrieve any of these
particular files produced in the last 60 days, so it isn't 'scratch' in
the Fortran sense of the word.

> If you really need that many, then define a search-list logical for read
> access, pointing to many directories, then have something periodically
> redefine a logical for write access to point to one of these.

I think we are all agreed on that, though it's hard to say without
knowing the application.

--
Paul Sture

[David Froble]

Paul Sture wrote:

> What I would do here is as others have suggested and use a search list.

VMS usually allows many different solutions to a problem.

When I read JF's post, I liked the idea.

I still don't think that we know enough details to get very specific.
For example, I suggested a scratch subdirectory for each day. Under
some circumstances this would work well, but not under other circumstances.

Been a number of good general approaches and suggestions in this thread.

[Paul Sture]

In article <nospam-BFD0F6....@news.chingola.ch>,

To nitpick my own post, I am assuming that the applications aren't using
DIRECTORY to do the lookups. DIRECTORY of course will plough through
the entire searchlist...

--
Paul Sture

[Paul Sture]

In article <kthgdj$g5v$1...@dont-email.me>,

David Froble <da...@tsoft-inc.com> wrote:

> Paul Sture wrote:
>
> > What I would do here is as others have suggested and use a search list.
>
> VMS usually allows many different solutions to a problem.
>
> When I read JF's post, I liked the idea.
>
> I still don't think that we know enough details to get very specific.
> For example, I suggested a scratch subdirectory for each day. Under
> some circumstances this would work well, but not under other circumstances.

Indeed. There could be a pile of documents representing month end
invoices and/or statements in there falling on the same day of the
month, or there could simply be certain days which see magnitudes more
activity than others.

> Been a number of good general approaches and suggestions in this thread.

--
Paul Sture

[Paul Sture]

In article <214840f4-50e5-4e91...@googlegroups.com>,

Joe <joslo...@gmail.com> wrote:

>
> > ACP_MAXREAD is typically 32 blocks, so that 1 file will cause 11725/32=366
> > read and write IOs during which the directory is locked out and
> > invalidating the cache on all other cluster nodes. The reads are likely
> > from cache. Verify the size with MCR SYSGEN SHOW /ACP
> >
> >
>
> yes, Its 32
>
> MCR SYSGEN SHOW ACP_MAX
> Parameter Name Current Default Min. Max. Unit Dynamic
> -------------- ------- ------- ------- ------- ---- -------
> ACP_MAXREAD 32 32 1 64 Blocks D

There's not a lot of scope for tuning here, so time and effort would
probably be better spent elsewhere.

FWIW VAX/VMS V7.3 has the same values.

--
Paul Sture

[Phillip Helbig---undress to reply]

In article <nospam-8D6613....@news.chingola.ch>, Paul Sture

<nos...@sture.ch> writes:

> Yes, I know there's NTP or DTSS in the case of DECnet Phase V, but both
> those have been known to break somewhere.

SYSMAN> CONFIG SET TIME once per day should do it.