Resizing files based on jstat output

[[Xze]]

Dear all,

I'm using the jstat utility to check the attributes of hashed files and for some of them i can see data in primary space but for others in both, primary and secondary

My question is how to find whether a file needs resize or not based on jstat output (for both cases) and if it needs how to calculate the new modulo

As an example you can consider the following files:
 
jsh ~ -->jstat -v F.LOCKING
File ../bnk.data/eb/F.LOCKING
Type=J4 , Hash method = 5
Created at Wed Jan  6 19:45:35 2010
Groups = 1629 , Frame size = 8192 bytes , Secondary Record Size = 16384 bytes
Restore re-size parameters : (none)
File size = 16457728 bytes , Inode = 1248028 , Device = Id 9223372212948434946
Last Accessed Tue Mar  2 08:40:18 2010 , Last Modified Tue Mar  2 08:40:18 2010
Backup = YES , Log = YES , Rollback = YES , Network = AUTO
 

Record Count = 24207 , Record Bytes = 10265930
Bytes/Record = 424 , Bytes/Group = 6301
Primary   file space:   Total Frames = 1950 , Total Bytes = 9840241
Secondary file space:   Total Frames = 56 , Total Bytes = 425689
 

jsh ~ -->jstat -v FBNK.EB.CONTRACT.BALANCES
File ../bnk.data/re/FBNK_EB_CONTRACT_BALANCES
Type=J4 , Hash method = 5
Created at Wed Feb 24 10:56:56 2010
Groups = 250001 , Frame size = 4096 bytes , Secondary Record Size = 8192 bytes
Restore re-size parameters : (none)
File size = 1061769216 bytes , Inode = 1230014 , Device = Id 9223372212948434946
Last Accessed Wed Feb 24 14:34:34 2010 , Last Modified Wed Feb 24 14:34:34 2010
Backup = YES , Log = YES , Rollback = YES , Network = AUTO
 

Record Count = 1346947 , Record Bytes = 387310863
Bytes/Record = 287 , Bytes/Group = 1549
Primary   file space:   Total Frames = 259220 , Total Bytes = 387310863
Secondary file space:   Total Frames = 0 , Total Bytes = 0

System Information
==================

 

OS  AIX 5.3.9.0
jbase: Major 5.0 , Minor 20 , Patch 0364 (Change 85159)

Regards,
Xze

[pat]

Hi Xze,

Secondary space is not a bad thing

Secondary space is for storing ( individual ) records that exceed the
Primary frame size, eg items that are larger than 4096 bytes

The actual item data ( ie the individual item exceeding 4096 bytes, or
8192 bytes in the case of 'F.LOCKING' above( I assume a Seperation of
2 was used when creating the 'F.LOCKING' file ) is held in the
Secondary space
The Primary space used for such ( large ) items simply holds a
'pointer' to the 'out of group' data, so the 'Primary space'
requirement for large items is minimal, and the actual data for the
item is efficiently held 'out of group'

The 'thing to compare', when deciding whether the file needs resizing,
is the 'Groups' ( '250001' in the case of the
'FBNK.EB.CONTRACT.BALANCES' file )
with the current 'Primary file space: Total Frames' ( '259220' in the
case of the 'FBNK.EB.CONTRACT.BALANCES' file )

If the 'Primary file space: Total Frames is in excess of the 'Groups',
then the 'Primary file space' contains 'linked' frames

Meaning for each update / access of an item within any of the
'Groups' , with a 'linked' frame, instead of having a maximum of
'4096' bytes to scan in order to find the item within the Group,
at least 2 frames of data may be scanned in order to find the required
item

The higher the ratio of 'Total Frames ( in Primary space)' to the
'Groups' the higher the likelihood that there are more than just one
linked frame in each 'Group', eg for the 'FBNK.EB.CONTRACT.BALANCES'
file above

Groups : 250001
Frames in Primary Space : 259220

So there are at least 9219 linked Frames in Primary space, ie
approximately 4% of the Groups in the 'FBNK.EB.CONTRACT.BALANCES' file
have a 'linked' frame
[ which on the whole is not too bad ], but could be improved by
resizing this file with a modulo of 26501 ( say )

Pat.

[praveen kumar]

Hi,

 

Use this command jrf -R F.LOCKING.

 

Regards,

 

--
Please read the posting guidelines at: http://groups.google.com/group/jBASE/web/Posting%20Guidelines
 
IMPORTANT: Type T24: at the start of the subject line for questions specific to Globus/T24
 
To post, send email to jB...@googlegroups.com
To unsubscribe, send email to jBASE-un...@googlegroups.com
For more options, visit this group at http://groups.google.com/group/jBASE?hl=en

[Srinivas]

Dear ,

 

In the above output for F.LOCKING, it is confrimed that the file F.LOCKING requires to be resized . When the count of Total Frames (i.e,1950) becomes more than the Groups Allocated (i.e. 1629) , then the File is said to be Badly sized.

 

To Resize , the No of Groups or Frames count must be increased  so that the total size of the frames will be  within the limit of the allocated Frame Size.

 

New Modulo can be calculated based on the total no of bytes divided by the frame size currently hold by the file.

 

total no of bytes = 16457728

current frame size = DIV(9840241,1950)

                           = 5046 bytes

New Modulo = total no of bytes / current frame size

                  = 3261

Using jrf utility , it has to be re-sized.

 

Regards,

 

Srinivas Yadav.

 

--
Please read the posting guidelines at: http://groups.google.com/group/jBASE/web/Posting%20Guidelines
 
IMPORTANT: Type T24: at the start of the subject line for questions specific to Globus/T24
 
To post, send email to jB...@googlegroups.com
To unsubscribe, send email to jBASE-un...@googlegroups.com
For more options, visit this group at http://groups.google.com/group/jBASE?hl=en

--

[fft...@aol.com]

I think an important metric here is a comparison between your frame size, in this case 8192 and your bytes/group which was smaller than this.  The number of frames used in primary space will tell you how many groups have another frame piggy-backed onto the default single frame they get when they are created.

I can't see that your example is badly sized, at least not so much that you would notice much if any speed improvement in resizing.  In my experience, only the worst files actually cause a noticeable issue.  The biggest issues are with.... bad programming.  If you're having speed issues, I'd examine the code to look for bottle-necks.

I have a funny story about a whole-file-process which, as it ran, output a status message on how many records it had processed, etc.  Of course the file ran pretty quickly on new hardware, but the program still output a status line for *each* record processed. Record 1, 2, 3, 4, 5, etc up to 500,000 or whatever it was.  The messages went by now, so quickly you couldn't really read them, so it was pointless.

The whole process took about 2 hours.  I changed that status to only output every 1000th record.  The process from that point on, now took 10 minutes.  The vast consumption of time, was just in printing status messages.  Funny isn't it?

Some things just don't scale well, when the hardware speeds up.

Will J.

[Tony Gravagno]

I noticed the situation you describe sometime in the early 90's
.... or was it in the 80's? Serial lines and the Telnet protocol
both throttle a server, forcing it to complete the write before
it moves to the next instruction. Individual instructions may
take a few milliseconds to process, but it may take full a second
to transport text down a wire and render it to a UI, thus
crippling the performance of an app.

One solution (as you said) is to do something like this:
IF MOD(COUNTER,1000) THEN CRT COUNTER

Another solution (depending on the technology involved, is simply
to minimize the window on a telnet client which is receiving
streamed output. You still incur transmission time but you don't
incur time in the UI to render text, which can also be
substantial.

HTH

Tony Gravagno
Nebula Research and Development
TG@ remove.pleaseNebula-RnD.com
Nebula R&D sells mv.NET and other Pick/MultiValue products
worldwide, and provides related development services
remove.pleaseNebula-RnD.com/blog
Visit PickWiki.com! Contribute!
http://Twitter.com/TonyGravagno

[Jim Idle]

Actually, you are misunderstanding a little bit here.

While the render time on a fairly weak terminal emulator, such as some of those popular with MV users (bit which I will restrain myself from knocking) can be an issue as it causes flow control delays, once the data has left the process and entered the socket layer on the transmitting process, it does not affect the speed of the running process. On earlier RS232 system then the effect depended on the hardware implementation. Later Reality systems used plan, which acted in a similar manner to sockets. However many systems had hardwired ports and the transmission time did affect things because the RS232 buffer was very small and would quickly fill up and block the process.

However, by far the more significant issue is the fact that even before the days of pre-emptive kernels, queuing yourself up for some IO meant that you lost your time slice and had to wait for the monitor to come back around and pick you up again. Pre-emptive scheduling helped this a little but even so, you have given up your time slice for no reason and this is the real issue. Minimizing the local telnet client will only [possibly] save you some local CPU cycles.

A crude benchmarking system could be created on dedicated hardware, which would give you an indication of the load on the CPU at any point in time by seeing how many single '*' characters the process could send to the screen in a given period of time (say a second or 5 seconds). Because the process would RQM as soon as the IO was dished up, then in order to queue the next '*' you had to wait for another time slice and that meant that the higher the load, the smaller the number of '*' sent to the screen.

Of course I know of nobody at all that has fallen for this, most especially not any ex-Microdata programmers who might have written a transaction journaling system and printed '*' the screen to indicate transactions being processed/restored/etc. Oh no, yes indeed, definitely didn't happen to any mate of mine and definitely not on the journal restore program.

Jim

HTH

--

[Daniel Klein]

It would be better to use ELSE, i.e.,

IF MOD(COUNTER,1000) ELSE CRT COUNTER

otherwise you will be displaying every number except every thousandth ;-)

Dan

[Tony Gravagno]

> From: Daniel Klein

> It would be better to use ELSE, i.e.,
> IF MOD(COUNTER,1000) ELSE CRT COUNTER
>
> otherwise you will be displaying every number except every
> thousandth ;-)

My responses can be fast, free, or debugged - pick two. :)

Thanks Dan.
T

[[Xze]]

Thank you for your ideas, but i believe something is definitely wrong here.

You said that the citeriea for a badly sized file is the following condition: Total Frames > Groups Allocated.
Ok, i calculated the new modulo and re-sized the file, check it out:

jsh ~ -->jstat -v F.LOCKING
File ../bnk.data/eb/F.LOCKING
Type=J4 , Hash method = 5
Created at Wed Jan  6 19:45:35 2010
Groups = 1629 , Frame size = 8192 bytes , Secondary Record Size = 16384 bytes
Restore re-size parameters : (none)

File size = 16457728 bytes , Inode = 2272742 , Device = Id 9223372217243402242
Last Accessed Wed Mar  3 08:30:39 2010 , Last Modified Wed Mar  3 08:30:39 2010

Backup = YES , Log = YES , Rollback = YES , Network = AUTO

Record Count = 24206 , Record Bytes = 10265821

Bytes/Record = 424 , Bytes/Group = 6301

Primary   file space:   Total Frames = 1950 , Total Bytes = 9840132

Secondary file space:   Total Frames = 56 , Total Bytes = 425689

jsh ~ -->jrf -ES3261,2 F.LOCKING

jsh ~ -->jstat -v F.LOCKING
File ../bnk.data/eb/F.LOCKING
Type=J4 , Hash method = 5

Created at Wed Mar  3 08:32:26 2010
Groups = 3261 , Frame size = 8192 bytes , Secondary Record Size = 16384 bytes
Restore re-size parameters : (none)
File size = 27828224 bytes , Inode = 2295685 , Device = Id 9223372217243402242
Last Accessed Wed Mar  3 08:32:27 2010 , Last Modified Wed Mar  3 08:32:27 2010

Backup = YES , Log = YES , Rollback = YES , Network = AUTO

Record Count = 24206 , Record Bytes = 10265821
Bytes/Record = 424 , Bytes/Group = 3148
Primary   file space:   Total Frames = 3340 , Total Bytes = 9840132

Secondary file space:   Total Frames = 56 , Total Bytes = 425689

With that kind of approach i'll be resizing in an infinite loop

Can anybody clarify the issue?

[Mike Preece]

On Mar 3, 1:22 am, Daniel Klein <danielklei...@gmail.com> wrote:
> It would be better to use ELSE, i.e.,
>
> IF MOD(COUNTER,1000) ELSE CRT COUNTER
>
> otherwise you will be displaying every number except every thousandth ;-)
>
> Dan

It would be better to use THEN, i.e.,

IF MOD(COUNTER,1000) = 0 THEN CRT COUNTER

Mike.

[Daniel Klein]

My responses are fast, free AND accurate ;-)

Dan

[Daniel Klein]

Not. Using the ELSE form results in less object (executable) code.

Dan

[fft...@aol.com]

Xze like you discovered you can't really win that game.
You want to focus on bytes/group.  As long as that number is reasonably under your frame size of 8192, you're about as good as you can get in the "speed" department here.  As you can see, your resizing drastically lowered this number from 6301 to 3148

That's what you gained by the resize.
I really don't think you're going to notice any speed improvement however :)
Just my two cents!
I've resized my share of files in my lifetime.
If you're having speed issues, your programming is more likely to be at fault, then this file.
Or perhaps you have some oddly very badly sized file... or you're running an Anti-Virus on your server.

Will

[FAISAL FAYYAZ]

Hi,

 

Have you tried system enquiry EB.BAD.SIZE.FILES?

 

Regards,

 

Faisal

[jaro]

usually I tried to follow the rules:
1. Keeping the primary group size (frame size) in balance with the
operating system IO operations. aix is normally performs the IO in 4KB
blocks, so trying to use the frames in the same size to get the group
on one go. in your example the F.LOCKING file has frame size 8KB so
for every READ/WRITE the system must do 2 IO operations because I
think jbase must read the whole logical block and then searching for
the record within the block.
2. Avoiding too many records per frame especially for the heavy used
tables even the records size is too small and can fit in the frame.
Before system reads/writes the record in the frame the whole frame is
locked for the integrity purposes that is released after the IO
operation is completed. During this moment all the records within the
particular frame are not available for other requests. And jbase call
must spend some time to locate the record position within too many
records in the frame. Then it's better to increase number of frames to
reduce the number of records per frame even you will have more free
space in the frames.
3. this is out of topic but the F.LOCKING is one of the most
frequently used file. In this file you have too many records (more
than usually) and I assume that majority of the records are out of
date or obsolete. sometimes the proper cleaning (also in other
important files) and then reducing the file size will help a lot. for
the second file EB.CONTRACT.BALANCES I don't see a reason to resize it
to increase the number of frames because the physical size of the file
is more than 1GB, but the data are about 400MB only, so a lot of free
space.
4. and there are many other possibilities.

[FFT...@aol.com]

In a message dated 3/3/2010 7:14:49 AM Pacific Standard Time, daniel...@gmail.com writes:

My responses are fast, free AND accurate ;-)

What I like about you Dan is that you're soooooo humble!

[Daniel Klein]

Hey, if your good and you know it, why hide it...right, Jimi?

Dan

[VK]

Hi,
a bit offtopic but... there shouldn't be so many as 24,000 records in
F.LOCKING. Count records with IDs starting with "UNIQUE.KEY.". You can
delete them without harm (when no one is at the system though).

VK

On Mar 4, 2:08 am, Daniel Klein <danielklei...@gmail.com> wrote:
> Hey, if your good and you know it, why hide it...right, Jimi?
>
> Dan
>

> On Wed, Mar 3, 2010 at 3:26 PM, <FFT2...@aol.com> wrote:
> > In a message dated 3/3/2010 7:14:49 AM Pacific Standard Time,

[[Xze]]

Thank you for your reply.

Actually, my task is as follows:
I'm checking with jstat every file on a regular basis (once per two weeks or once per month). My next step is to apply a formula, something like
IF (Total.Frames.In.Primary* Frame.Size + Total.Frames.In.Secondary*Frame.Size)/Record.Bytes > File.size
or whatever and to know whether this particular file requires resize or not.

So, the question was what is that 'magic' formula and what is the formula for new modulo, assuming that my file requires resize

Regards,
Xze

[Jim Idle]

Just use jrf.

 

However I feel that the algorithm has been tuned detrimentally in recent times, but it is the one that TEMENOS want you to use. But, start with jbase.markmail.org to search for older posts on this.

 

Jim

[[Xze]]

Unfortunatelly jrf is not suitable for the files that have separation 2, 4, 8

That's why i'm trying to rely on jstat

Thanks for the link, will give a try

Xze

[fft...@aol.com]

I'll give you a simple formula that has served me for 20 years.  Not in Jbase  in particular, but hopefully it's not too much different from other Pick systems in this regard.

If bytes/group > 2*framesize then *maybe* it's time to resize.
Of course you'd want to address the files that exceed this in the grossest way first.

Will

[jaro]

send me your email, I can pass you a program I did in past. you can
try it and you will see if helpful for you or you adjust it by
yourself.

j

On Mar 4, 3:36 pm, "[Xze]" <bhdistort...@gmail.com> wrote:
> Unfortunatelly jrf is not suitable for the files that have separation 2, 4,
> 8
>
> That's why i'm trying to rely on jstat
>
> Thanks for the link, will give a try
>
> Xze
>
>
>
> On Thu, Mar 4, 2010 at 16:25, Jim Idle <j...@temporal-wave.com> wrote:
> >  Just use jrf.
>
> > However I feel that the algorithm has been tuned detrimentally in recent
> > times, but it is the one that TEMENOS want you to use. But, start with
> > jbase.markmail.org to search for older posts on this.
>
> > Jim
>

> > *From:* jb...@googlegroups.com [mailto:jb...@googlegroups.com] *On Behalf
> > Of *[Xze]
> > *Sent:* Thursday, March 04, 2010 3:57 AM
> > *To:* jb...@googlegroups.com
>
> > *Subject:* Re: Resizing files based on jstat output

>
> > Thank you for your reply.
>
> > Actually, my task is as follows:
> > I'm checking with jstat every file on a regular basis (once per two weeks
> > or once per month). My next step is to apply a formula, something like
> > IF (Total.Frames.In.Primary* Frame.Size +
> > Total.Frames.In.Secondary*Frame.Size)/Record.Bytes > File.size
> > or whatever and to know whether this particular file requires resize or
> > not.
>
> > So, the question was what is that 'magic' formula and what is the formula
> > for new modulo, assuming that my file requires resize
>
> > Regards,
> > Xze
>

> > Groups = *1629 *, Frame size = 8192 bytes , Secondary Record Size = 16384

> > bytes
> > Restore re-size parameters : (none)
> > File size = 16457728 bytes , Inode = 2272742 , Device = Id
> > 9223372217243402242
> > Last Accessed Wed Mar  3 08:30:39 2010 , Last Modified Wed Mar  3 08:30:39
> > 2010
> > Backup = YES , Log = YES , Rollback = YES , Network = AUTO
>
> > Record Count = 24206 , Record Bytes = 10265821
> > Bytes/Record = 424 , Bytes/Group = 6301

> > Primary   file space:   Total Frames = *1950 *, Total Bytes = 9840132

> > Secondary file space:   Total Frames = 56 , Total Bytes = 425689
>

> > jsh ~ -->jrf -ES*3261*,2 F.LOCKING

>
> > jsh ~ -->jstat -v F.LOCKING
> > File ../bnk.data/eb/F.LOCKING
> > Type=J4 , Hash method = 5
> > Created at Wed Mar  3 08:32:26 2010

> > Groups = *3261 *, Frame size = 8192 bytes , Secondary Record Size = 16384

> > bytes
> > Restore re-size parameters : (none)
> > File size = 27828224 bytes , Inode = 2295685 , Device = Id
> > 9223372217243402242
> > Last Accessed Wed Mar  3 08:32:27 2010 , Last Modified Wed Mar  3 08:32:27
> > 2010
> > Backup = YES , Log = YES , Rollback = YES , Network = AUTO
>
> > Record Count = 24206 , Record Bytes = 10265821
> > Bytes/Record = 424 , Bytes/Group = 3148

> > Primary   file space:   Total Frames = *3340 *, Total Bytes = 9840132