What might cause a D3 system to suddenly drop to zero Free Space.

[Neil Pratt]

Recently from the morning to the evening we lost 11 % of our free space on our D3 system. We didn't find anything that might have gobbled up the space.  Then we had a worse disaster where over a period of minutes we lost ALL our free space.  Fortunately Rocket managed to recover 38% . Apparently the pointers to free space were corrupted (I think it's kind of like truncating a file with a GFE, I'm sure many will recall those?) .  I was just wondering if anyone else has experienced this and if they knew what might cause it?  We do have an API that was affected when AWS (Amazon Web Services) was having issues. Some of our D3 ports where shelled out to linux and got hung up. Could that perhaps have caused our sudden drop to zero space?  Rocket haven't come up with a cause/reason yet.  We have 209 Gig for D3 and usually hover around 50% usage. Currently it is at 34% and apparently we will need to do a full restore of the system to reclaim the rest.

[geneb]

The only time I've ever seen that was with a runaway print job that was
going to a hold file.

g.

--
Proud owner of F-15C 80-0007
http://www.f15sim.com - The only one of its kind.
http://www.diy-cockpits.org/coll - Go Collimated or Go Home.
Some people collect things for a hobby. Geeks collect hobbies.

ScarletDME - The red hot Data Management Environment
A Multi-Value database for the masses, not the classes.
http://scarlet.deltasoft.com - Get it _today_!

[Neil Pratt]

Thanks G ,  the first thing we did was SORT PEQS BY SIZE .... :)   We found nothing out of the ordinary there.

[Mauricio Caneda]

Did you check the POINTER-FILE?  Maybe someone or some process created a very large list?

On Wed, Dec 15, 2021 at 3:54 PM 'Neil Pratt' via Pick and MultiValue Databases <mvd...@googlegroups.com> wrote:

Recently from the morning to the evening we lost 11 % of our free space on our D3 system. We didn't find anything that might have gobbled up the space.  Then we had a worse disaster where over a period of minutes we lost ALL our free space.  Fortunately Rocket managed to recover 38% . Apparently the pointers to free space were corrupted (I think it's kind of like truncating a file with a GFE, I'm sure many will recall those?) .  I was just wondering if anyone else has experienced this and if they knew what might cause it?  We do have an API that was affected when AWS (Amazon Web Services) was having issues. Some of our D3 ports where shelled out to linux and got hung up. Could that perhaps have caused our sudden drop to zero space?  Rocket haven't come up with a cause/reason yet.  We have 209 Gig for D3 and usually hover around 50% usage. Currently it is at 34% and apparently we will need to do a full restore of the system to reclaim the rest.

--
You received this message because you are subscribed to
the "Pick and MultiValue Databases" group.
To post, email to: mvd...@googlegroups.com
To unsubscribe, email to: mvdbms+un...@googlegroups.com
For more options, visit http://groups.google.com/group/mvdbms
---
You received this message because you are subscribed to the Google Groups "Pick and MultiValue Databases" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mvdbms+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mvdbms/4cbd0c68-3909-4ef7-826e-ac357cc1f1f3n%40googlegroups.com.

[Roland Ruiz]

If you are actively losing overflow, take a snapshot of the overflow table segments using "ovf (a"

Then after some loss, do another "ovf (a" and identify any blocks of overflow that were consumed.

Then examine the content of those frames to hopefully identify the consumer.

Regards,

Roland 

To view this discussion on the web visit https://groups.google.com/d/msgid/mvdbms/CAGL1E6SAGXw0ry8un_%3D2_p_%3D%2Bk5EJu5Q636o80LxfDLEyWNmcQ%40mail.gmail.com.

[Mike Wooding]

I've seen this twice on D3 and both instances were a considerable time ago.

The first instance was a runaway phantom process which was eating disk space whilst creating a report.  From memory, terminating the process released the space and the code was modified to prevent it happening again.

D3 have since introduced the "set-runaway-limit" command to prevent just this situation.  What are your runaway process value traps set at?   Do you have any code that disables these by setting it to zero system wide, on a specific PIB or on the current port?

Providing "set-runaway-autoquit" isn't in effect, when a process consumes more frames than the maximum values it should halt the process and issue a prompt along the lines  of "Runaway process detected (continue/quit)?"  As suggested, "Continue" allows the process to continue (until the same number of frames have been eaten again when the prompt returns) whereas "Quit" aborts the process, returning the frames to the system.

The second instance was a glitch in the free space / overflow table.  (I'm sorry, it was a long time ago and I can't remember precisely which.)

What I do remember is that there was a discrepancy between output of the "free" verb and the "povf" verb.

As Rocket have suggested, the lost space was recovered when doing a full save and restore using ":files".

Hope you get it sorted.

With Kindest Regards

Mike Wooding

[SLove]

we have found that improper shutdowns, ie: crashes, cause loss of space - when the system shutdowns properly the workspace is restored to the overflow table.  Crashes do NOT return the space, so slowly, or quickly when lots of users are signed in,  the space disappears.  And on older versions using LOGOFF also did not return the space.   The only solution i know is a full VME restore.

lastly, though Rocket tells me not to do this,  i increase the VME size to 4096 MB since most systems i install have 8 or more gigs of ram and this helps reduce the number of vme restores.  Also, if you do this change the VME and FSI flush periods from 10 to 3,  so far i have not seen system performance affected by these changes

[Neil Pratt]

Thanks Mike,

set-runaway-limit 40000

I'll go with ..  we had a " glitch in the free space / overflow table" and somehow Rocket managed to repair some of it.  "Available disk space" reported by the free command has been sitting at 34% since Rocket made the fix, so we aren't rapidly losing free space anymore. Thank goodness. :) 

On Thursday, December 16, 2021 at 11:23:37 AM UTC-5 Mike Wooding wrote:

[Neil Pratt]

Thanks Slove,

I think we "lost" free space due to a   " glitch in the free space / overflow table"  which occurred when some of our logged on ports were hung up whilst being shelled out to Linux.

Isn't VME something the D3-NT requires?  It's been a long time since I've seen a D3-NT system.

[Neil Pratt]

Thanks Rolando,

Good to know about the "ovf" command but I don't believe we actually lost free space just that,  Pick/D3 lost access to it, via the 
Overflow Table" as mentioned by Mike Wooding.

[Neil Pratt]

Thanks  Moe Caneda but we don't usually create large save-lists . I just checked to make sure. :)