How do I explain REXX Storage Usage

[Lizette Koehler]

I have the need to provide a simple explanation as to why sometimes when running
a REXX with REGION=0M or REGION=128M in batch it fails. And other times it is
okay.
I am unable to post the code, but am willing to answer any questions about it.


The batch job runs IKJEFT01

The process reads an input file that could have from between 5 input lines and
100's or 1000's of input lines.

An EXECIO is used to read the entire Input file. Then the process does a DO i=1
to src.0 looking for a header record.

Once the header record is found they spin the data to an output file until the
next header record is found.

Occasionally they will get an error message stating: Machine storage is
exhausted.

There are no new stack/del stack process in this code. A lot of Signal
statements are used.


This is trying to take on large input file and split out all of the reports in
it into separate files. So long as there are not many input lines, it works.

I would like to provide a simple explanation as to why it does not work some of
the time.

I will also be working with my users and using a different process that will be
more successful. But for now, I need to explain why this is a coding technique
challenge.

Thanks.

Lizette Koehler
statistics: A precise and logical method for stating a half-truth inaccurately

----------------------------------------------------------------------
For TSO-REXX subscribe / signoff / archive access instructions,
send email to LIST...@VM.MARIST.EDU with the message: INFO TSO-REXX

[Dyck, Lionel B. , TRA]

This is basic but hopefully will help

1. Each line that is read in via execio into a stem takes storage. A few records takes less storage than a lot of records.
2. Assuming you are copying from the src.x stem into another stem for output then that consumes additional storage

Hope this helps


--------------------------------------------------------------------------
Lionel B. Dyck (Contractor)
Mainframe Systems Programmer
Enterprise Infrastructure Support (Station 200) (005OP6.3.10)
VA OI&T Service Delivery & Engineering

[Itschak Mugzach]

Which command caused this msg? I usually see it at exec io. You may Use drop scr.i after processing the record to free up storage.

Best.
ITschak

נשלח מה-iPad שלי

‫ב-6 ביוני 2016, בשעה 16:07, ‏‏Dyck, Lionel B. (TRA) ‏<Lione...@VA.GOV> כתב/ה:‬

[richard rozentals]

Hi Lizette,
I suspect that the EXECIO is not the cause. You can prove this by using the STACK instead of a stem variable. My testing found that the STACK is stored in the spool. So you can read 100s of thousands of records . I suspect this may be the signal. I've written a lot of REXX code though the years and only had this problem when it was the execio to a stem variable. That problem was when I was reading a file with over 100,000 records. I do not use the signal as a GOTO. I assume they do. It may be that the signal is not internally written as GOTO. It may be that signal uses a lot of storage based on the number of times it's used.This is just a guess.

Can you reproduce this at will? Does just a simple rerun with the same data work? Does using different data clear the problem? Does the storage used on the batch job stats change based on the data read, not just number of records? More interested in the failure with REGION=128M.

Richard Rozentals

[Hobart Spitz]

If you have the PIPE command, use it.

--
OREXXMan

[Sevetson, Phil]

Lizette,

I was told by my systems programmer that requests for memory are fulfilled on an as-available basis. The implication is that you might not get all the region size which you request. I'm unable to verify this independently -- I don't know how to read dumps and have no systems programming background.

That said, one of your options is to EXECIO 1 DISKW to write out your output as it occurs. It will definitely be quite a bit slower than putting all your output to stack-variables and writing it all when done... but it won't take a lot of memory, and won't blow up if your input suddenly decides to grow huge.

Or, if your developers are willing to write the rather-more-complex code, you could do a tradeoff and write a batch every hundredth record (EXECIO 100 DISKW), or even every "n-th" record (EXECIO chunksize DISKW), setting a variable. That would eliminate 99% of the duplicate EXECIO processing and still not take more than a few megabytes of storage.

--Phil Sevetson
psev...@fisa.nyc.gov
City of New York / Financial Info Svcs Agcy
New York, NY USA 10001

-----Original Message-----
From: TSO REXX Discussion List [mailto:TSO-...@VM.MARIST.EDU] On Behalf Of Lizette Koehler
Sent: Monday, June 06, 2016 8:59 AM
To: TSO-...@VM.MARIST.EDU

[Lizette Koehler]

So with regard to SIGNAL statements.

There are 7 SIGNAL ON coded in the REXX. Each is SIGNAL ON NAME routine, then some code, EXIT.

The use it to determine
O Did the dataset get allocated?
O Did the file get written to okay?
O Did the file close okay?
O Did the file delete okay?

So might this process be part of the issue?

I would have used the RETURN CODES from EXECIO, ALLOC, DELETE, etc...

Lizette

> -----Original Message-----
> From: TSO REXX Discussion List [mailto:TSO-...@VM.MARIST.EDU] On Behalf Of
> richard rozentals
> Sent: Monday, June 06, 2016 6:37 AM
> To: TSO-...@VM.MARIST.EDU
> Subject: Re: [TSO-REXX] [EXTERNAL] [TSO-REXX] How do I explain REXX Storage
> Usage
>

> Hi Lizette,
> I suspect that the EXECIO is not the cause. You can prove this by using the
> STACK instead of a stem variable. My testing found that the STACK is stored in
> the spool. So you can read 100s of thousands of records . I suspect this may
> be the signal. I've written a lot of REXX code though the years and only had
> this problem when it was the execio to a stem variable. That problem was when
> I was reading a file with over 100,000 records. I do not use the signal as a
> GOTO. I assume they do. It may be that the signal is not internally written as
> GOTO. It may be that signal uses a lot of storage based on the number of times
> it's used.This is just a guess.
>
> Can you reproduce this at will? Does just a simple rerun with the same data
> work? Does using different data clear the problem? Does the storage used on
> the batch job stats change based on the data read, not just number of records?
> More interested in the failure with REGION=128M.
>
> Richard Rozentals
>
>
> From: Itschak Mugzach <imug...@GMAIL.COM>
> To: TSO-...@VM.MARIST.EDU
> Sent: Monday, June 6, 2016 9:11 AM

> Subject: Re: [TSO-REXX] [EXTERNAL] [TSO-REXX] How do I explain REXX Storage
> Usage
>

> Which command caused this msg? I usually see it at exec io. You may Use drop
> scr.i after processing the record to free up storage.
>
> Best.
> ITschak
>
> נשלח מה-iPad שלי
>
> ‫ב-6 ביוני 2016, בשעה 16:07, ‏‏Dyck, Lionel B. (TRA) ‏<Lione...@VA.GOV> כתב/ה:‬
>
> > This is basic but hopefully will help
> >
> > 1. Each line that is read in via execio into a stem takes storage. A few
> records takes less storage than a lot of records.
> > 2. Assuming you are copying from the src.x stem into another stem for
> > output then that consumes additional storage
> >
> > Hope this helps
> >
> >
> > ----------------------------------------------------------------------
> > ----
> > Lionel B. Dyck (Contractor)
> > Mainframe Systems Programmer
> > Enterprise Infrastructure Support (Station 200) (005OP6.3.10) VA OI&T
> > Service Delivery & Engineering
> >

[Roach, Dennis]

Here is what I use for looping through large files
Buffer_amt = # records to read at a time
"execio" buffer_amt "diskr input (stem in."
do while in.0 > 0
do i = 1 to in.0
<process record>
end
drop in.
"execio" buffer_amt "diskr input (stem in."
end

Dennis Roach, CISSP, PMP
AIG
IAM Access Administration – Consumer | Identy & Access Management

2929 Allen Parkway, America Building, 3rd Floor | Houston, TX 77019
Phone: 713-831-8799

Dennis...@AIG.com | www.aig.com

All opinions expressed by me are mine and may not agree with my employer or any person, company, or thing, living or dead, on or near this or any other planet, moon, asteroid, or other spatial object, natural or manufactured, since the beginning of time.

[Lizette Koehler]

Dennis,

I have a similar snippet of REXX but it lacks the DROP Statement.

I will see if they will provide the file on the next failure so I can work with this concept.

Thanks everyone so far with these awesome suggestions.

[christel.u.w.pachl christel.u.w.pachl]

Buffer the Input with something like this

Do While readrc=
'EXECIO DISKR 1000 (STEM X.
readrc=rc
Do i=1 To x.0
Call process x.i
End
End
process:
Parse Arg l
If ... Then Call o l
Return

o:
z=o.0+1
o.z=arg(1)
o.0=z
If o.0>=1000 Then Do
'EXECIO' DISKW o.0
o.0=0
End

To be completed and tested

Regards

Walter

[Paul Gilmartin]

On 2016-06-06, at 07:11, Itschak Mugzach wrote:

> Which command caused this msg? I usually see it at exec io. You may Use drop scr.i after processing the record to free up storage.
>

No. When last I researched it, DROPping individual members of
a compound symbol did not free the storage; only DROPping the
entire compound (DROP scr.) did that. I have a contrived test
in which DROPping individual members of a compound actually
increases the storage requirement. I don't know whether exiting
a PROCEDURE with a non-EXPOSEd compound is equivalent to DROP.
I should hope so.

Mark Zelden has a tool (thanks!) that nicely reports storage
usage.

I had a colleague who once naively misused SIGNAL repeatedly
to exit a nest of CALLs. Worked fine on simple test cases
With a larger data set it failed with call/return stack
exhausted. Rexx desperately needs longjump, or at least
longITERATE/longLEAVE.

-- gil

[Anania, Philip D]

If you write to the queue (stack) instead of a stem variable, then you will
free up storage for each pull. I've had to switch from one to the other on
occasion to get by this. If you are reading from an array and adding to an
array or the stack for your writes you'll end up with double the amount of
records read. My general rule of thumb - lots O' data - use the stack, not
lots O' data - use an array.

On Mon, Jun 6, 2016 at 10:23 AM, Lizette Koehler <star...@mindspring.com>
wrote:

--

"Go about your Music citizen" - PdotDdot

[richard rozentals]

Hi Lizette,
Yes your approach is simpler and more efficient.  I was thinking if the signal was used for each record it would be a problem. I can't see 7 executions of signal being a issue.I can't see the execio being the cause. 1000 records is not that much.
I would ask your user for the listings of the failed and working runs of the rexx(I assume this is run in batch). Look for the pattern. If the file size or type of data does not increase the region used then it can't be that.

I also assume this is not a real big REXX EXEC. I found that if it get over 5 thousand lines it gets to be a problem.
You can also try compiling the REXX. Not to run it that way but to see how big it is compiled. 

I expect this is a issue of odd coding or environmental. Environmental issues may be harder to pinpoint. Make sure your user's CPU is not running close to 100% and has all the resources available. Could it be a limitation on the USERID?
RichardRozentals

[Kopischke, David G.]

I think the DROP is what's going to free memory.

If your EXECIO goes to a different stem for every report, it's going to use a lot of memory. Memory isn't freed. I believe the DROP addresses this to a certain extent, but I'm not sure it's fool proof.

Is there some reason the logic can't be streamed ??? You mentioned a header of some kind is a decision point, so can't you EXECIO 1 READ to a buffer and EXECIO 1 WRITE to a different file between headers ???

This e-mail transmission may contain information that is proprietary, privileged and/or confidential and is intended exclusively for the person(s) to whom it is addressed. Any use, copying, retention or disclosure by any person other than the intended recipient or the intended recipient's designees is strictly prohibited. If you are not the intended recipient or their designee, please notify the sender immediately by return e-mail and delete all copies. OppenheimerFunds may, at its sole discretion, monitor, review, retain and/or disclose the content of all email communications.

[christel.u.w.pachl christel.u.w.pachl]

BEWARE of EXECIO 1

It's slooooooooooooooooooooooooooooooooow

Walter

[Farley, Peter x23353]

I use this technique all the time, with an argument to the EXEC to control how many to read at one time rather than a fixed amount I the EXEC, but with some reasonable default for the expected volume of records. For known small-volume files, I can then pass "*" as the argument to read all of them at once.

Also with appropriate validation of the argument value of course.

Peter

-----Original Message-----
From: TSO REXX Discussion List [mailto:TSO-...@VM.MARIST.EDU] On Behalf Of Roach, Dennis
Sent: Monday, June 06, 2016 10:41 AM
To: TSO-...@VM.MARIST.EDU
Subject: Re: How do I explain REXX Storage Usage

Here is what I use for looping through large files Buffer_amt = # records to read at a time
"execio" buffer_amt "diskr input (stem in."
do while in.0 > 0
do i = 1 to in.0
<process record>
end
drop in.
"execio" buffer_amt "diskr input (stem in."
end

--


This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail and delete the message and any attachments from your system.

[richard rozentals]

Hi,
This is true. Each execio does a open,read and close. Vary slow reading a file.
That is another reason to use STACK for large files. 1 open and 1 close, full use of buffering. Then using the "get" for each record.
Richard Rozentals

From: christel.u.w.pachl christel.u.w.pachl <christel....@CHELLO.AT>
To: TSO-...@VM.MARIST.EDU
Sent: Monday, June 6, 2016 12:25 PM
Subject: Re: [TSO-REXX] How do I explain REXX Storage Usage

[Robin]

There is a trick to using EXECIO 1 that doesn't slow down processing too
severely.

Presuming the organization (RECFM, LRECL, etc) of input file is stable for
each execution of the REXX, you have the REXX program allocate to the file
as RECFM=U. Herein the EXECIO will read a physical block (with x number of
logical record) with 1 EXECIO. It is then the responsibility to 'deblock'
the physical file block of records. Handling 'short blocks' (e.g. last block
of the file)
Is also the responsibility of the REXX program. 'Deblocking' full physical
blocks is relatively simple for files where the true RECFM=FB.

-----Original Message-----
From: TSO REXX Discussion List [mailto:TSO-...@VM.MARIST.EDU] On Behalf Of

Lizette Koehler
Sent: June 6, 2016 8:59 AM
To: TSO-...@VM.MARIST.EDU

[Hobart Spitz]

Instead of EXECIO 1, you can use LINEIN().

--
OREXXMan

[Paul Gilmartin]

On 2016-06-06, at 11:15, Hobart Spitz wrote:

> Instead of EXECIO 1, you can use LINEIN().
>

I'd be surprised if that wasn't even slower.

But which LINEIN()? There's one with compiled Rexx that handles
only Classic data sets, and one with OMVS Rexx that handles only
UNIX files. They gotta fix that.

> On Mon, Jun 6, 2016 at 1:11 PM, Robin wrote:
>>
>> Presuming the organization (RECFM, LRECL, etc) of input file is stable for
>> each execution of the REXX, you have the REXX program allocate to the file
>> as RECFM=U. Herein the EXECIO will read a physical block (with x number of
>> logical record) with 1 EXECIO. It is then the responsibility to 'deblock'
>> the physical file block of records. Handling 'short blocks' (e.g. last
>> block
>> of the file)
>> Is also the responsibility of the REXX program. 'Deblocking' full physical
>> blocks is relatively simple for files where the true RECFM=FB.
>>

Deblocking RECFM=VB is only a little harder, and VBS is only a little
more harder.

Or, if you're determined to RYO, you can just write an Assembler program.

-- gil

[Mickey Bee]

You can also do this with any file type.

For example, I create a file 80 bytes, optimum half track blocking of 27920.

I do an execio 349 DISKR, and one EXCP gets me 349 records. Then it's just a
matter of interrogating the return code.

Mickey

---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus

[Lizette Koehler]

So I wanted a little clearer understanding at this point as to storage
allocation during REXX execution (I do not have the compiler so always
interpreted).

So if I have a simple REXX

A = myvar
B = myvar

Both of those take an individual storage location. Or does REXX point A and B
to the same storage location?



But if I now do

Do I = 10
Parse Var stem.i myvar .
A = myvar
B.i = myvar
End

I think on each iteration it is a new address for the storage location of the
vars A & B.i .

Do I have that correct.

If so, then Dropping the stem would free up that storage and reduce what storage
is allocated for my variables.

So instead of having 10 storage locations for A and B.i - I would have one for
A and 10 for B.i ????? So I would need to drop B.i before finishing the loop?



Lizette


>
> If the same stem is being used for each output report (e.g. outrec.) then
> doing a "DROP" of the stem between reports might take care of the problem.
>
> I'm not sure why this works but there seems to be some cultch that only gets
> cleaned up w/ a DROP.
>
> Jamie

>
> -----Original Message-----
> From: TSO REXX Discussion List [mailto:TSO-...@VM.MARIST.EDU] On Behalf Of
> Lizette Koehler
> Sent: Monday, June 06, 2016 8:59 AM
> To: TSO-...@VM.MARIST.EDU
> Subject: [TSO-REXX] How do I explain REXX Storage Usage
>

[christel.u.w.pachl christel.u.w.pachl]

One Location for A which gets reused

ten Locations for all the B.i's

[retired mainframer]

> -----Original Message-----
> From: TSO REXX Discussion List [mailto:TSO-...@VM.MARIST.EDU] On Behalf
> Of Lizette Koehler

> Sent: Thursday, June 23, 2016 12:15 PM
> To: TSO-...@VM.MARIST.EDU

> Subject: Re: How do I explain REXX Storage Usage
>
> So I wanted a little clearer understanding at this point as to storage
> allocation during REXX execution (I do not have the compiler so always
> interpreted).
>
> So if I have a simple REXX
>
> A = myvar
> B = myvar
>
> Both of those take an individual storage location. Or does REXX point A
and B
> to the same storage location?

Can A and B point to the same location? Do interpreters do optimizations of
this sort? Would
A = B
cause A and B to point to the same location?

If you run this under TSO TEST or XDC, you should be able to determine where
A and B point.

How about
A = myvar
myvar = myvar+1
B = myvar

After the first assignment, does A point to the same place myvar points or
to a new area holding a copy of myvar's data? After the second, does myvar
point to a new area? If so and A pointed to the old area, REXX has to know
not to release that memory because A is still using it. Did the value in A
change? Is A equal to B?

> But if I now do
>
> Do I = 10
> Parse Var stem.i myvar .
> A = myvar
> B.i = myvar
> End
>
> I think on each iteration it is a new address for the storage location of
the
> vars A & B.i .

All REXX variables are stored as strings. Even the simple sequence
A = myvar
A = myvar
requires either new memory OR a determination that the new value can fit in
the space already allocated. Does the routine that performs assignments try
to determine if space can be reused? If new space is allocated, is the old
space released immediately or does some threshold trigger a garbage
collection algorithm?

If you test, does the address pointed to be A change after the second
assignment?

>
> Do I have that correct.
>
> If so, then Dropping the stem would free up that storage and reduce what
storage
> is allocated for my variables.

Is the memory for dropped variables freed immediately or does garbage
collection occur later?

> So instead of having 10 storage locations for A and B.i - I would have
one for
> A and 10 for B.i ????? So I would need to drop B.i before finishing the
loop?

If there is a real reason for using B.i instead of B, then you probably
don't want to drop any B.i until well after the loop is finished. If you
drop B.i before the loop finishes, you cannot refer to B.(i-1) in the next
iteration so having a stem element seems a waste.

[Pedro Vera]

My recollection from a similar discussion many years ago was that a DROP
marks the memory as unused, but does not necessarily result in a FREEMAIN
of the memory. It is still allocated and could be used for something
else. The FREEMAIN could be done sometime later, as determined by the rexx
memory management routines.

I think you should drop large stem variables when you are done using them,
but not necessarily drop each simple variable that you use.



Pedro Vera
DB2 Admin Tool
http://www-01.ibm.com/software/data/db2imstools/db2tools/db2admin/