Read Dataset the last Record
Peter Uhrig
i have the following question: i want to read a dataset ( sequential or
a po-member) , but I need only the last record . how I can resolve this
problem with rexx ? have anyone an example ?
best regards
Peter
----------------------------------------------------------------------
For TSO-REXX subscribe / signoff / archive access instructions,
send email to LIST...@VM.MARIST.EDU with the message: INFO TSO-REXX
kpin...@tiaa-cref.org
can code EXECIO to read a specific record:
"EXECIO 1 DISKR ddname linenum"
Lizette Koehler
You would not be able to get all the records into the STEM before it fails
on storage.
Perhaps a loop reading one record at a time until all records are read? A
little slower but does not require as much region size.
Lizette
>
> Peter,
>
> You have to read the whole dataset, but you can put the records on a
> data stack and process just the one you need.
>
> "execio * disr DDNAME (finis stem YourStem" /* put the data on a
> stack */
>
> N=YourStem.0 /* N = # of records on
> the stack */
> YourRecord=YourStem.N /* get the record you
> need */
>
>
>
> Subject: [TSO-REXX] Read Dataset the last Record
George, William , DHCS-ITSD
Mark Zelden
08:17:38 AM:
> Only issue I see is what if it is a HUGE data set with Millions or
Records?
> You would not be able to get all the records into the STEM before it
fails
> on storage.
>
> Perhaps a loop reading one record at a time until all records are read?
A
> little slower but does not require as much region size.
>
> Lizette
>
That would be a lot slower when talking about a "large" number of
records. I usually do something like this:
do forever
"EXECIO 10000 DISKR SCANIN (STEM INREC."
if RC > 2 | inrec.0 == 0 then leave /* rc = 2 for partial read */
do I = 1 to inrec.0
...
...
end
end
Mark
--
Mark Zelden
Sr. Software and Systems Architect
Zurich North America / Farmers Insurance Group - ZFUS G-ITO
mailto:mark....@zurichna.com
z/OS and OS390 expert at http://searchDataCenter.com/ateExperts/com/
Systems Programming expert at http://expertanswercenter.techtarget.com/
Mark's MVS Utilities: http://home.flash.net/~mzelden/mvsutil.html
******************* PLEASE NOTE *******************
This E-Mail/telefax message and any documents accompanying this
transmission may contain privileged and/or confidential information and is
intended solely for the addressee(s) named above. If you are not the
intended addressee/recipient, you are hereby notified that any use of,
disclosure, copying, distribution, or reliance on the contents of this
E-Mail/telefax information is strictly prohibited and may result in legal
action against you. Please reply to the sender advising of the error in
transmission and immediately delete/destroy the message and any
accompanying documents. Thank you.
George, William , DHCS-ITSD
is sorted and the primary sort key is the first characters, do an
initial sort in the opposite direction and then read just the first
record.
-----Original Message-----
From: TSO REXX Discussion List [mailto:TSO-...@VM.MARIST.EDU] On Behalf
Paul Gilmartin
> Date: Fri, 20 Jul 2007 08:40:33 -0500
>
> TSO REXX Discussion List <TSO-...@VM.MARIST.EDU> wrote on 07/20/2007
> 08:17:38 AM:
>
> > Only issue I see is what if it is a HUGE data set with Millions or
> Records?
> > You would not be able to get all the records into the STEM before it
> fails
> > on storage.
> >
This should always be a design consideration; else there's
a really unpleasant astonishment factor when the input data
finally exceed REGION capacity.
> That would be a lot slower when talking about a "large" number of
> records. I usually do something like this:
>
> do forever
> "EXECIO 10000 DISKR SCANIN (STEM INREC."
> if RC > 2 | inrec.0 == 0 then leave /* rc = 2 for partial read */
> do I = 1 to inrec.0
> ...
> ...
> end
> end
>
Isn't the "inrec.0 == 0" test superfluous?
-- gil
--
StorageTek
INFORMATION made POWERFUL
Hawley, Scott R
fast.
First, get the record count with sort.
//COUNT EXEC PGM=SYNCTOOL,REGION=2M
//TOOLMSG DD SYSOUT=*
//DFSMSG DD SYSOUT=*
//IN1 DD DISP=SHR,DSN=<big.file>
//TOOLIN DD *
* COUNT NUMBER OF RECORDS IN INPUT FILE
COUNT FROM(IN1)
By parsing the TOOLMSG output for the number of records you could do an
execio to skip to the last record.
"EXECIO 1 DISKR INDD" SKIP_COUNT "(STEM T. FINIS"
Paul Gilmartin
> Date: Fri, 20 Jul 2007 06:30:45 -0700
>
> If the file is large like mentioned below and on the off chance the file
> is sorted and the primary sort key is the first characters, do an
> initial sort in the opposite direction and then read just the first
> record.
>
Gulp. Sounds like the worst performance suggestion yet. But
if DFSORT's I/O is twice as efficient as Rexx's, it may actually
come out ahead.
-- gil
--
StorageTek
INFORMATION made POWERFUL
----------------------------------------------------------------------
Paul Gilmartin
> Date: Fri, 20 Jul 2007 06:23:06 -0700
>
> You might try using an Edit Macro and process just the last record.
>
Same as Lizette's objection: this is guaranteed to fail on
a sufficiently large data set.
How about the following in a z/OS UNIX Services address SYSCALL:
cp "//'DATA.SET.NAME(MEMBER)'" /dev/fd/1 | tail -1
.. z/OS UNIX rocks!
-- gil
--
StorageTek
INFORMATION made POWERFUL
----------------------------------------------------------------------
Dave Salt
> > a po-member) , but I need only the last record . how I can resolve this
> > problem with rexx ? have anyone an example ?
You could do it with a VIEW macro. Here's an example of the calling program:
/* REXX */
address ispexec
"VIEW DATASET('your.dsn(memb)') MACRO(YOURMAC)"
"VGET (LASTLINE) SHARED"
say "The last line in 'your.dsn(memb)' is: "lastline
Here's an example of YOURMAC:
/* REXX */
address isredit
"MACRO"
"(LASTLINE) = LINE .ZLAST"
address ispexec "VPUT (LASTLINE) SHARED"
"END"
The above will work both with members and sequential data sets. Hope that
helps,
Dave Salt
See the new SimpList(tm) rollover image at:
http://www.mackinney.com/products/SIM/simplist.htm
_________________________________________________________________
See Fireworks On Live Image Search
http://search.live.com/images/results.aspx?q=Fireworks&mkt=en-ca&FORM=SERNEP
Farley, Peter x23353
> From: Lizette Koehler [mailto:star...@MINDSPRING.COM]
> Sent: Friday, July 20, 2007 9:18 AM
> To: TSO-...@VM.MARIST.EDU
> Subject: Re: [TSO-REXX] Read Dataset the last Record
>
> Records?
> You would not be able to get all the records into the STEM before it fails
> on storage.
>
> little slower but does not require as much region size.
Better would be to EXECIO 5000 or 10000 or some fixed "chunk size" at a time
in a loop, saving the last record from each chunk, stopping when you get EOF
on the EXECIO. Then the last saved record from your loop is the last record
in the file.
I have found that "chunked" EXECIO's are far more efficient than
single-record calls (which makes sense when you think about it, less EXECIO
overhead per record to start with).
I frequently make the "chunk" size a parameter to the outermost REXX and
pass it down to the "I/O" subroutine. Then I can experiment with the
"chunk" value for different REGION sizes in a batch job to find the most
efficient value for that file and process.
HTH
Peter
This message and any attachments are intended only for the use of the addressee and
may contain information that is privileged and confidential. If the reader of the
message is not the intended recipient or an authorized representative of the
intended recipient, you are hereby notified that any dissemination of this
communication is strictly prohibited. If you have received this communication in
error, please notify us immediately by e-mail and delete the message and any
attachments from your system.
Mark Zelden
> In a recent note, Mark Zelden said:
>
> > Date: Fri, 20 Jul 2007 08:40:33 -0500
> >
> > TSO REXX Discussion List <TSO-...@VM.MARIST.EDU> wrote on 07/20/2007
> > 08:17:38 AM:
> >
> > > Only issue I see is what if it is a HUGE data set with Millions or
> > Records?
> > > You would not be able to get all the records into the STEM before it
> > fails
> > > on storage.
> > >
> This should always be a design consideration; else there's
> a really unpleasant astonishment factor when the input data
> finally exceed REGION capacity.
>
> > That would be a lot slower when talking about a "large" number of
> > records. I usually do something like this:
> >
> > do forever
> > "EXECIO 10000 DISKR SCANIN (STEM INREC."
> > if RC > 2 | inrec.0 == 0 then leave /* rc = 2 for partial read */
> > do I = 1 to inrec.0
> > ...
> > ...
> > end
> > end
> >
> Isn't the "inrec.0 == 0" test superfluous?
>
I don't think so. It would loop forever without it once there
were no more records to read. The RC would still be 2 at that
point (not greater than 2), but inrec.0 would be 0, so that is
what will cause the code to leave the do forever read loop.
Mark
--
Mark Zelden
Sr. Software and Systems Architect - z/OS Team Lead
Zurich North America / Farmers Insurance Group - ZFUS G-ITO
mailto:mark....@zurichna.com
z/OS and OS390 expert at http://searchDataCenter.com/ateExperts/
Systems Programming expert at http://expertanswercenter.techtarget.com/
Mark's MVS Utilities: http://home.flash.net/~mzelden/mvsutil.html
******************* PLEASE NOTE *******************
This E-Mail/telefax message and any documents accompanying this
transmission may contain privileged and/or confidential information and is
intended solely for the addressee(s) named above. If you are not the
intended addressee/recipient, you are hereby notified that any use of,
disclosure, copying, distribution, or reliance on the contents of this
E-Mail/telefax information is strictly prohibited and may result in legal
action against you. Please reply to the sender advising of the error in
transmission and immediately delete/destroy the message and any
accompanying documents. Thank you.
----------------------------------------------------------------------
Ken MacKenzie
this:
/* REXX */
"ALLOC F(SYSIN) SPACE(1 1) TRACKS REUS"
Push '$$DD01 COPYBACK OUT=1'
"EXECIO 1 DISKW SYSIN (FINIS"
"ALLOC F(DD01) DA(MY.INPUT.FILE) SHR REUS"
"ALLOC F(DD01O) DA(MY.OUTPUT.FILE) SHR REUS"
Address LINKMVS "FILEAID"
A few months ago, I did tests of this against sort and other solutions and
this was by far the most efficient - I don't know how they do it.
Ken MacKenzie
Phone: +1 973 716 6630
+353 74 918 8612
ken.ma...@pramerica.ie
www.pramerica.ie
--------------------------------------------------------------------------------------------------------------------------
Pramerica Systems Ireland Limited, Registered in Ireland No. 319900, 6th
Floor, South Bank House, Barrow Street, Dublin 4.
Peter Uhrig
<peter.uhrig@BHF-
BANK.COM> To
TSO-...@VM.MARIST.EDU
Sent by: TSO REXX cc
Discussion List
<TSO-...@VM.MARI Subject
ST.EDU> [TSO-REXX] Read Dataset the last
Record
Fri 07/20/2007
08:49 AM
Please respond to
TSO REXX
Discussion List
<TSO-...@VM.MARI
ST.EDU>
Hamilton, Robert L
Then at the end of the loop
IF REOORDS.0 < LIMIT THEN LEAVE
BOBH
-----Original Message-----
From: TSO REXX Discussion List [mailto:TSO-...@VM.MARIST.EDU] On Behalf
Of Mark Zelden
Sent: Friday, July 20, 2007 9:15 AM
To: TSO-...@VM.MARIST.EDU
richard rozentals
1st way use the IDCAMS REPRO command. With this command you can read from a specific record to the end. The problem you may have is that you don't know how many records you have. One way to do this is to again use IDCAMS repro to copy to a dummy file and trap the # of records copied.
The simplest way is to use EXECIO with the with LIFO and write to QUEUE. Yes this does take lots of CPU and I/O. But it's simple. You write the entire file to the QUEUE read the records you want and delete the queue. The amount of records that you can write is dependant on your system. I've written over 500cyls of data one time.
Mark Zelden <mark....@ZURICHNA.COM> wrote: TSO REXX Discussion List wrote on 07/20/2007
08:17:38 AM:
---------------------------------
Yahoo! oneSearch: Finally, mobile search that gives answers, not web links.
Dunkel, Martin
If you want to only use Rexx, you will have to read the whole file and
keep the last line you read before end-of-file. An "execio 1" into a
stem will go directly into stem.1 each time, wiping out the stem.1 from
the previous record. This way you do no need to be concerned with
storage, and you do not need to do an additional parse pull from the
data stack for each execio; this cuts your IO's in half, but keeping the
last line doubles your IO's again.
Here's my code:
/* rexx */
dsn="'your.big.dataset.goes.here'"
"alloc ddname(ddin) dataset("||dsn||") shr reuse"
"execio 0 diskr ddin (open)"
rc=0; recs=0; drop line.; keepline=""
do until rc=2
"execio 1 diskr ddin (stem line.)"
if rc=0 then do
recs=recs+1
keepline=line.1
end
end
say "Records read:"
say recs
say "Last line read:"
say keepline
"execio 0 diskr ddin (finis)"
"free ddname(ddin)"
Thanks! MPD
Martin Dunkel
National City Corporation
Turnover Support / ChangeMan ZMF
(w) 216-257-5354
(p) 80053...@archwireless.net
-----Original Message-----
From: TSO REXX Discussion List [mailto:TSO-...@VM.MARIST.EDU] On Behalf
Of Peter Uhrig
Sent: Friday, July 20, 2007 3:49 AM
To: TSO-...@VM.MARIST.EDU
best regards
Peter
-------------------------------------------------------------------------------------------
***National City made the following annotations
-------------------------------------------------------------------------------------------
This communication is a confidential and proprietary business communication.
It is intended solely for the use of the designated recipient(s). If this
communication is received in error, please contact the sender and delete
this communication.
===========================================================================================
George, William , DHCS-ITSD
TSO REXX Discussion List <TSO-...@VM.MARIST.EDU> wrote on 07/20/2007
08:52:13 AM:
> > do forever
> > "EXECIO 10000 DISKR SCANIN (STEM INREC."
> > if RC > 2 | inrec.0 == 0 then leave /* rc = 2 for partial read */
> > do I = 1 to inrec.0
> > ...
> > ...
> > end
> > end
> >
> Isn't the "inrec.0 == 0" test superfluous?
>
> I don't think so. It would loop forever without it once there
> were no more records to read. The RC would still be 2 at that
> point (not greater than 2), but inrec.0 would be 0, so that is
> what will cause the code to leave the do forever read loop.
>
> Mark
What about just coding
If RC > 1 then?
Mark Zelden
> -----Original Message-----
> From: TSO REXX Discussion List [mailto:TSO-...@VM.MARIST.EDU] On Behalf
> Of Mark Zelden
> Sent: Friday, July 20, 2007 7:15 AM
> To: TSO-...@VM.MARIST.EDU
> Subject: Re: Read Dataset the last Record
>
> TSO REXX Discussion List <TSO-...@VM.MARIST.EDU> wrote on 07/20/2007
> 08:52:13 AM:
>
> > > do forever
> > > "EXECIO 10000 DISKR SCANIN (STEM INREC."
> > > if RC > 2 | inrec.0 == 0 then leave /* rc = 2 for partial read */
> > > do I = 1 to inrec.0
> > > ...
> > > ...
> > > end
> > > end
> > >
> > Isn't the "inrec.0 == 0" test superfluous?
> >
>
> > I don't think so. It would loop forever without it once there
> > were no more records to read. The RC would still be 2 at that
> > point (not greater than 2), but inrec.0 would be 0, so that is
> > what will cause the code to leave the do forever read loop.
> >
> > Mark
>
>
> What about just coding
> If RC > 1 then?
>
That would leave the loop without processing any records!
BTW, I'm not saying that is the only way (or the best way)
to accomplish "reading a chunk" as someone else put it.
It was just an example. But the point was... not to do
EXECIO one record at a time.
Mark
--
Mark Zelden
Sr. Software and Systems Architect - z/OS Team Lead
Zurich North America / Farmers Insurance Group - ZFUS G-ITO
mailto:mark....@zurichna.com
z/OS and OS390 expert at http://searchDataCenter.com/ateExperts/
Systems Programming expert at http://expertanswercenter.techtarget.com/
Mark's MVS Utilities: http://home.flash.net/~mzelden/mvsutil.html
******************* PLEASE NOTE *******************
This E-Mail/telefax message and any documents accompanying this
transmission may contain privileged and/or confidential information and is
intended solely for the addressee(s) named above. If you are not the
intended addressee/recipient, you are hereby notified that any use of,
disclosure, copying, distribution, or reliance on the contents of this
E-Mail/telefax information is strictly prohibited and may result in legal
action against you. Please reply to the sender advising of the error in
transmission and immediately delete/destroy the message and any
accompanying documents. Thank you.
----------------------------------------------------------------------
Ryerse, Robin
only supports DSORG=PS and RECFM=F or RECFM=V. For RECFM=V datasets, you
can specify an 'average record length' to optimize the skipping
/* REXX get last record of a dataset without handling them all */
address tso
call listdsi 'SORTIN FILE'
if sysreason <> 0 then
do
say 'require SORTIN to be allocated to the input dataset.'
exit 8
end
select
when sysdsorg <> 'PS' then
do
say 'DSORG='sysdsorg ' can not be processed.'
exit 8
end
when left(sysrecfm,1) = 'F' then
scx.RECORD = 'TYPE=F,LENGTH=('syslrecl')'
when left(sysrecfm,1) = 'V' then
do
scx.RECORD = 'TYPE=V,LENGTH=('syslrecl','syslrecl','syslrecl')'
if arg(1,'E') &,
datatype(arg(1),'W') &,
arg(1) < syslrecl then syslrecl = arg(1)
end
otherwise
do
say 'recfm='sysrecfm ' can not be processed.'
exit 8
end
end
sysused = sysused - 1
rec_per_block = sysblksize % syslrecl
select
when sysunits = 'BLOCK' then
do
sysblkstrk = 1
systrkscyl = 1
end
when sysunits = 'TRACK' then
systrkscyl = 1
otherwise nop
end
skipped = sysused * rec_per_block * sysblkstrk * systrkscyl
scx.SORT = 'FIELDS=COPY,SKIPREC='skipped
call scxsort , 'R.'
last# = r.0
say 'skipped' skipped 'records then read' last# 'records in',
sysdsname 'to find:'
say r.last#
exit
Robin Ryerse
-----Original Message-----
From: TSO REXX Discussion List [mailto:TSO-...@VM.MARIST.EDU] On Behalf
Of Peter Uhrig
Sent: July 20, 2007 3:49 AM
To: TSO-...@VM.MARIST.EDU
Cruz, Robert
first,
the fundamental problem is that sorting, at best, is O(n*log(n)), while
simply reading the file is O(n). For large files log(n) becomes
significant
and would probably overwhelm the superior efficiency of SORT's I/O (this
applies
to both DFSORT and SYNCSORT). So I do not expect this to be a viable
solution
large files, even when the file has key field(s).
-----Original Message-----
From: TSO REXX Discussion List [mailto:TSO-...@VM.MARIST.EDU] On Behalf
Of Paul Gilmartin
Sent: Fri 20 Jul 2007 06:56
To: TSO-...@VM.MARIST.EDU
Subject: Re: [TSO-REXX] Read Dataset the last Record
In a recent note, "George, William (DHCS-ITSD)" said:
> Date: Fri, 20 Jul 2007 06:30:45 -0700
>
> If the file is large like mentioned below and on the off chance the
> file is sorted and the primary sort key is the first characters, do an
> initial sort in the opposite direction and then read just the first
> record.
>
Gulp. Sounds like the worst performance suggestion yet. But if
DFSORT's I/O is twice as efficient as Rexx's, it may actually come out
ahead.
-- gil
--
StorageTek
INFORMATION made POWERFUL
----------------------------------------------------------------------
Dunkel, Martin
From my understanding, it really does not matter if you read a large
file one record at a time, or 10000 records at a time. I don't think
you will see any more disk reads either way. With the file being
blocked (hopefully optimally), even if you just read in one record at a
time, the system is still going to pull in a block of records at a time.
Please correct me if I am wrong here.
Thanks! MPD
Martin Dunkel
National City Corporation
Turnover Support / ChangeMan ZMF
(w) 216-257-5354
(p) 80053...@archwireless.net
-----Original Message-----
From: TSO REXX Discussion List [mailto:TSO-...@VM.MARIST.EDU] On Behalf
HTH
Peter
-------------------------------------------------------------------------------------------
***National City made the following annotations
-------------------------------------------------------------------------------------------
This communication is a confidential and proprietary business communication.
It is intended solely for the use of the designated recipient(s). If this
communication is received in error, please contact the sender and delete
this communication.
===========================================================================================
----------------------------------------------------------------------
Alan May
//LASTN EXEC PGM=ICETOOL
//DFSMSG DD SYSOUT=*
//TOOLMSG DD SYSOUT=*
//IN DD DSN=... input file
//OUT DD DSN=... output file
//TOOLIN DD *
COPY FROM(IN) USING(CTL1)
COPY FROM(IN) TO(OUT) USING(CTL2)
/*
//CTL1CNTL DD *
* Set n in SKIPREC=n to the number of records needed.
* For example, if you want the last 10 records,
* use SKIPREC=10.
OPTION SKIPREC=n <-- set n to the number of records needed
OUTFIL FNAMES=CTL2CNTL,REMOVECC,NODETAIL,BUILD=(80X),VTOF,
TRAILER1=(3:C'OPTION SKIPREC=',COUNT=(M11,LENGTH=12))
/*
//CTL2CNTL DD DISP=(,PASS),DSN=&&C2,SPACE=(TRK,1),UNIT=SYSDA
So where does the REXX code come into the equation? It doesn't really
but you could always write a bit of REXX code to set the value of
SKIPREC in the JCL stream.
Devised by Graham Harris of WDR Training & Consultancy
Enjoy!
-----Original Message-----
From: TSO REXX Discussion List [mailto:TSO-...@VM.MARIST.EDU] On Behalf
best regards
Peter
**************************************************************
**CONFIDENTIALITY NOTICE AND LEGAL LIABILITY WAIVER**
The content of this e-mail and any attachments are CONFIDENTIAL
and may contain privileged information. If you are not the named
addressee you should not disclose, copy, distribute or otherwise
use the information contained herein.
Please notify the sender immediately by e-mail if you have received
this e-mail by mistake and delete it from your system. E-mail
transmission cannot be guaranteed to be secure or error-free as
information could be intercepted, corrupted, lost, destroyed,
arrive late or incomplete, or contain viruses. The sender
therefore does not accept liability for any errors or omissions in
the contents.
The content of the message and or attachments may not reflect the
view and opinions of the originating company or any party it is
representing.
**************************************************************
WDR Training and Consultancy,
Park Lodge,
60, London Road,
Horsham,
West Sussex.
RH12 1AY
Cruz, Robert
read a 10,000-record file one record at a time, that's 10,000
invocations of EXECIO. If you read a "chunk" of 10,000 records all at
once, it's a single invocation of EXECIO. This turns out to be
significantly better.
-----Original Message-----
From: TSO REXX Discussion List [mailto:TSO-...@VM.MARIST.EDU] On Behalf
Of Dunkel, Martin
Sent: Fri 20 Jul 2007 09:18
To: TSO-...@VM.MARIST.EDU
Subject: Re: [TSO-REXX] Read Dataset the last Record
Just a side note...
From my understanding, it really does not matter if you read a large
file one record at a time, or 10000 records at a time. I don't think
you will see any more disk reads either way. With the file being
blocked (hopefully optimally), even if you just read in one record at a
time, the system is still going to pull in a block of records at a time.
Please correct me if I am wrong here.
Thanks! MPD
Martin Dunkel
National City Corporation
Turnover Support / ChangeMan ZMF
(w) 216-257-5354
(p) 80053...@archwireless.net
-----Original Message-----
From: TSO REXX Discussion List [mailto:TSO-...@VM.MARIST.EDU] On Behalf
HTH
Peter
----------------------------------------------------------------------
For TSO-REXX subscribe / signoff / archive access instructions, send
email to LIST...@VM.MARIST.EDU with the message: INFO TSO-REXX
------------------------------------------------------------------------
-------------------
***National City made the following annotations
------------------------------------------------------------------------
-------------------
This communication is a confidential and proprietary business
communication.
It is intended solely for the use of the designated recipient(s). If
this communication is received in error, please contact the sender and
delete
this communication.
========================================================================
===================
----------------------------------------------------------------------
Paul Gilmartin
> Date: Fri, 20 Jul 2007 09:17:38 -0400
>
> Only issue I see is what if it is a HUGE data set with Millions or Records?
> You would not be able to get all the records into the STEM before it fails
> on storage.
>
> Perhaps a loop reading one record at a time until all records are read? A
> little slower but does not require as much region size.
>
I took this over to IBM-MAIN and got some good suggestions, particularly from
Frank Yaeger:
-- gil
--
StorageTek
INFORMATION made POWERFUL
----------------------------------------------------------------------
Jeremy Nicoll - lstsrvs
> Hi,
> i have the following question: i want to read a dataset ( sequential or
> a po-member) , but I need only the last record . how I can resolve this
> problem with rexx ? have anyone an example ?
Something is going to have to read the whole file, whether it is syncsort,
or an editor (for a macro to be used). I'd hesitate to have that reading
done by any interpreted code in your rexx exec.
If this is a one-off problem, or something that will only happen a few
times, I'd consider calling idcams repro to copy the whole file into a KSDS
(ie a file which essentially numbers the records).
Then use IDCAMS PRINT (or maybe it's another verb, I can't remember) to
extract the last record. Last time I did this I extracted the data in a
hex-dump format then used rexx to read the IDCAMS SYSPRINT to get that and
convert it back to textual data. Of course if you have an rexx/vsam
interface then it's easier.
The advantage of this method is that the bulk of the IO is done by a much
more efficient piece of code than your exec will ever be.
Lastly I'd look at some method of not having to do this in future - eg
making whatever program updates the sequential file in question also write a
copy of its last record somewhere else.
--
Jeremy C B Nicoll - my opinions are my own.
Bob Stark
EXECIO is a separate module which must be potentially loaded from disk for
each EXECIO call. I Think you are correct about the physical disk I/O's
being the same.
I usually try to read large datasets at several thousand records per call,
so that I get efficiency without hitting a storage limit.
Regards,
Bob Stark
ProTech - When you're serious about Systems Management
Consulting, Software, and Training for z/OS, UNIX and Internet
www.protechtraining.com 800-373-9188 x150 412-445-8072
Just a side note...
Thanks! MPD
HTH
Peter
--
If this email is spam, report it here:
http://www.OnlyMyEmail.com/reportSpam?Id=Mzg3MzE6MzU4MzY0OTI5OmJzdGFya0Bwcm9
0ZWNocHRzLmNvbQ%3D%3D
Robert Zenuk
to prove it to yourself.
/* rexx */
arg dsn
if dsn = '' then dsn = 'your.dsn.ps.or pds.member'
say dsn
"ALLOC F(INPUT) DA('"dsn"') SHR"
stime = time('r')
"EXECIO * DISKR INPUT (STEM INPUT. FINIS"
if RC <> 0 then sat 'EXECIO RC='RC
ttime = time('e') - stime
say input.0 'records, elapsed time is' ttime
say
count = 0
stime = time('r')
"EXECIO 1 DISKR INPUT (OPEN"
do forever
"EXECIO 1 DISKR INPUT (STEM INPUT."
IORC = RC
count = count + 1
if IORC <> 0 then leave
end
"EXECIO 1 DISKR INPUT (FINIS"
say count 'records, elapsed time is' time('e')
Rob
In a message dated 7/20/2007 9:45:00 AM US Mountain Standard Time,
Martin...@NATIONALCITY.COM writes:
From my understanding, it really does not matter if you read a large
file one record at a time, or 10000 records at a time. I don't think
you will see any more disk reads either way. With the file being
blocked (hopefully optimally), even if you just read in one record at a
time, the system is still going to pull in a block of records at a time.
************************************** Get a sneak peek of the all-new AOL at
http://discover.aol.com/memed/aolcom30tour
George, William , DHCS-ITSD
receives a rc = 0, the read that gets the remaining records, if partial,
gets the rc = 2 then leave the loop.
Hamilton, Robert L
Does it work if allotment = 0 or total nbr of records = 0
bobh
Mark Zelden
if RC > 2 | inrec.0 == 0 then leave
RC = 2 will not leave that loop!
Mark
--
Mark Zelden
Sr. Software and Systems Architect
Zurich North America / Farmers Insurance Group - ZFUS G-ITO
mailto:mark....@zurichna.com Office:847-605-6570 Cell:630-306-8166
z/OS and OS390 expert at http://searchDataCenter.com/ateExperts/com/
Systems Programming expert at http://expertanswercenter.techtarget.com/
Mark's MVS Utilities: http://home.flash.net/~mzelden/mvsutil.html
"George, William
(DHCS-ITSD)"
<William.George@D To
HCS.CA.GOV> TSO-...@VM.MARIST.EDU
Sent by: TSO REXX cc
Discussion List
<TSO-...@VM.MARI Subject
ST.EDU> Re: Read Dataset the last Record
07/20/2007 11:01
AM
Please respond to
TSO REXX
Discussion List
<TSO-...@VM.MARI
ST.EDU>
Walter u. Christel Pachl
My experience contradicts your assumtion dramatically.
Reading one record at a time is dead slow (on MVS)
Walter
Jeremy Nicoll - lstsrvs
> You are very wrong, I think.
> My experience contradicts your assumtion dramatically.
> Reading one record at a time is dead slow (on MVS)
Probably, because on each I/O operation (ie each EXECIO) your address space
hands over control to MVS (or the IO subsystem) and then after the IO is
complete has to comepete with everything else to get despatched again. If
the system is heavily loaded that could mean quite some delay over & above
the IO itself.
--
Jeremy C B Nicoll - my opinions are my own.
----------------------------------------------------------------------
David S Speake
For instance, out of Z/OS REXX SYNTAX V1R4 dicussion of EXECIO
If you specify a value of zero (0), no I/O operations are performed
unless you also specify either OPEN, FINIS, or both OPEN and FINIS.
o If you specify OPEN and the data set is closed, EXECIO opens the
data set but does not read any lines. If you specify OPEN and the
data set is open, EXECIO does not read any lines.
Is this WRONG? NO! Is it misread? !!YES!!
The records are NOT placed wherever you have indicated REXX should place them
but the ARE acquired by QSAM into memory at OPEN and BEFORE the very FIRST
read. At least one entire block of them. Hopefully MANY more than one.
If you can afford the memory put BUFNO= on the ALLOCATE to override the
millennia
old default of 5. How many? Depends :-)
QSAM schedules input when the first of the following occurs:
1) 15 buffers are available for the operation,
(It will NOT use more than 15, even if they are 80 byte blocks and
you specify BUFNO=255)
2) 240K bytes of buffer space is available for the operation
(Full track blocks (56K)? It wiil NOT use more than 5 buffers even
if......)
3) ALL the buffers allocated to the data set are available for the
operation
(5 buffers? It ALWAYS waits till ALL of them are available. You never,
ever get ANY IO overlap)
(O.K. if you have 5 buffers and 240K blocksize on 3590 tape.....)
Above is to initially SCHEDULE an IO. I do not know how it decides when to
dynamically modify a channel program
that is already in play - chaining new CCWs to the end of the program.
The heavies over at MAIN tell me that for half track blocking (about 27K)
about BUFNO=18
(a bit over 480K) gives just above knee of the curve performance in actual
testing. EXECIO just asks QSAM for
the next record. Actual IO occurs ONLY as above. This is in ANY language
whatsoever.
O.K. not if you are in Authorized ALC and don't use QSAM.
Compared to how good/bad the above is done, reasonable source REXX repetitive
interpretation HAS to be trivial.
Put out 10,000 80 byte blocks and try BUFNO=255 versus default, then block
half track and try again.
Carefull, your DASD manager will call you dirty names.
David
------------------( Forwarded letter 1 follows )---------------------
Date: Sun, 22 Jul 2007 21:14:41 +0200
Reply-To: TSO.REXX.Discussion.List[TSO-REXX]@VM.MARIST.EDU.INET
Sender: TSO.REXX.Discussion.List[TSO-REXX]@VM.MARIST.EDU.INET
From: Walter.u.Christel.Pachl[pachl]@CHELLO.AT.INET
Subject: Re: Read Dataset the last Record
To: TSO-...@VM.MARIST.EDU.INET
X-HDT-HopCount: 1
You are very wrong, I think.
My experience contradicts your assumtion dramatically.
Reading one record at a time is dead slow (on MVS)
Just a side note...
Thanks! MPD
HTH
Peter
----------------------------------------------------------------------
For TSO-REXX subscribe / signoff / archive access instructions,
send email to LIST...@VM.MARIST.EDU with the message: INFO TSO-REXX
-------------------------------------------------------------------------------------------
***National City made the following annotations
-------------------------------------------------------------------------------------------
This communication is a confidential and proprietary business communication.
It is intended solely for the use of the designated recipient(s). If this
communication is received in error, please contact the sender and delete
this communication.
===========================================================================================
----------------------------------------------------------------------
Jeremy Nicoll - lstsrvs
> ... EXECIO just asks QSAM for the next record.
But that means there's an SVC, which means your address space loses control
to the OS and doesn't get it back until it is redespatched after the IO
completes. You're right that that IO may be satisfied from a buffer that's
already got data in it, but you still have the SVC.
Doing umpteen iterations with EXECIO asking for 1 record at a time is still
going to waste elapsed time & CPU, compared with asking for chunks of many
records at a time.
--
Jeremy C B Nicoll - my opinions are my own.
----------------------------------------------------------------------
Bob Bridges
But I read some of the messages late and thought I'd contest this one
assertion, or at least question it.
It's true that REXX seems very inconveniently to reload all my external
REXX subroutines dynamically. (New REXXers see footnote below.) But I
don't believe that applies to the "TSO/E external functions" described in
section 4.4, nor the "TSO/E REXX commands" in chapter 10. Granted I've
never tested this... <pause> ...hmm...this assumption. <Pause again, more
thoughtful this time.> I now recall a program that ran longer than I
thought it should, but I was unable to figure out why. Maybe a
NEWSTACK/DELSTACK pair in the middle of a loop?
Ok, now I'm not asserting, I'm asking. I know it happens to my own REXX
routines, but what about those supplied with TSO/E? Are they reloaded each
time, or do they reside permanently with my TSO session?
---
Footnote for new guys: The reason it's inconvenient that REXX loads your
external subroutines and functions dynamically is that it does it not once
for each main-program run but once for every invocation of the subroutine,
no matter how many times you call it. Thus:
'EXECIO * DISKR DDI (FINIS'
do queued()
parse pull rec1 'DATE=' olddate rec2
queue rec1 'DATE='s2a(olddate) rec2; end
'EXECIO * DISKW DDO (FINIS'
The above program reads in a file and writes it out again with the date
reformatted by some external function I wrote myself and named s2a. Say
there are 25 000 records in the file: After gulping down the whole file
and putting it on the data stack, the loop proceeds to
1) Get the next record, separating the date
2) Look for s2a in the SYSEXEC and SYSPROC concatenations
3) Load s2a
4) Use s2a to convert the date format
5) Discard s2a
6) Write the reformatted data back on the bottom of the stack
...and then back around again for each record; thus it loads and discards
s2a 25 000 times! This slows your program down enormously; in my
experience I may get 10 or 20 records per second processed like this, but
not 40.
This doesn't matter if the subroutine is to be used once in a program, or
five or even twenty times. But in the above case, the easy solution -- the
only solution, as far as I know -- is to make an internal copy of s2a, ie
copy it into your program and add a label and maybe a PROC statement.
Without the dynamic reloading it can run 25 000 records in less than a
second. YMMV.
---
Bob Bridges, rhb...@attglobal.net, cell 336 382-7313
Robert...@westfieldgrp.com, 330 791-6576
/* The horror of the Same Old Thing is one of the most valuable passions we
have produced in the human heart -- an endless source of heresies in
religion, folly in counsel, infidelity in marriage, and inconstancy in
friendship. -advice to a tempter, from The Screwtape Letters by C S Lewis
*/
-----Original Message-----
From: Bob Stark
Sent: Friday, July 20, 2007 12:52
It matters greatly if you read a large file one record at a time, because
EXECIO is a separate module which must be potentially loaded from disk for
each EXECIO call. I Think you are correct about the physical disk I/O's
being the same.
I usually try to read large datasets at several thousand records per call,
so that I get efficiency without hitting a storage limit.
----------------------------------------------------------------------
Paul Gilmartin
> Date: Sat, 1 Sep 2007 16:00:42 -0400
>
> It's true that REXX seems very inconveniently to reload all my external
> REXX subroutines dynamically. (New REXXers see footnote below.) But I
> don't believe that applies to the "TSO/E external functions" described in
> section 4.4, nor the "TSO/E REXX commands" in chapter 10. Granted I've
> never tested this... <pause> ...hmm...this assumption. <Pause again, more
> thoughtful this time.> I now recall a program that ran longer than I
> thought it should, but I was unable to figure out why. Maybe a
> NEWSTACK/DELSTACK pair in the middle of a loop?
>
I don't know if this is a relevant or redundant data point. But several
years ago I was running an EXEC in a job step with a large STEPLIB.
I timestamped each line the program write with "date() time()".
I found the call to date() was taking several hundred milliseconds.
I opened a PMR and got a big WAD -- IBM diagnosed the overhead as
directory searches of STEPLIB for tne national language components
to format the date. They said there is no practical alternative
because STEPLIB must be searched at each function call lest the
content of the member be changed between calls, and suggested that
I obtain the date once at the beginning of the program.
"But what if the date changes during execution of the program?"
"That's not likely enough to be a concern, is it?"
"Isn't it similarly unlikely that the content of the national
language routines change during execution of the program?
"Regardless how unlikely, we must account for the possibility.
.. etc.
I don't know if the situation has yet been ameliorated; I circumvented
by using a non-default variant of date() that was not locale-sensitive,
and so did not search STEPLIB for those routines.
-- gil
--
StorageTek
INFORMATION made POWERFUL
----------------------------------------------------------------------
Bob Bridges
messages that are timestamped; I reasonably sure it generates fifty or a
hundred of them in just a second or two. Oh, but wait; it establishes the
date of the top and bottom of the log and generates only the time in
between.
Ok, it looks as though benchmarking is called for on a few built-in
routines, which I didn't expect.
---
Bob Bridges, rhb...@attglobal.net, cell 336 382-7313
Robert...@westfieldgrp.com, 330 791-6576
/* I have never been hurt by anything I didn't say. -Calvin Coolidge */
-----Original Message-----
From: Paul Gilmartin
Sent: Monday, September 3, 2007 01:16
I don't know if this is a relevant or redundant data point. But several
years ago I was running an EXEC in a job step with a large STEPLIB.
I timestamped each line the program write with "date() time()".
I found the call to date() was taking several hundred milliseconds.
I opened a PMR and got a big WAD -- IBM diagnosed the overhead as
directory searches of STEPLIB for tne national language components
to format the date. They said there is no practical alternative
because STEPLIB must be searched at each function call lest the
content of the member be changed between calls, and suggested that
I obtain the date once at the beginning of the program.
"But what if the date changes during execution of the program?"
"That's not likely enough to be a concern, is it?"
"Isn't it similarly unlikely that the content of the national
language routines change during execution of the program?
"Regardless how unlikely, we must account for the possibility.
.. etc.
I don't know if the situation has yet been ameliorated; I circumvented
by using a non-default variant of date() that was not locale-sensitive,
and so did not search STEPLIB for those routines.
--- In a recent note, Bob Bridges said:
> Date: Sat, 1 Sep 2007 16:00:42 -0400
>
> It's true that REXX seems very inconveniently to reload all my external
> REXX subroutines dynamically. (New REXXers see footnote below.) But I
> don't believe that applies to the "TSO/E external functions" described in
> section 4.4, nor the "TSO/E REXX commands" in chapter 10. Granted I've
> never tested this... <pause> ...hmm...this assumption. <Pause again,
> more thoughtful this time.> I now recall a program that ran longer than
> I thought it should, but I was unable to figure out why. Maybe a
> NEWSTACK/DELSTACK pair in the middle of a loop?
----------------------------------------------------------------------
Mickey Bee
From: Paul Gilmartin <gil...@UNIX.STORTEK.COM>
> In a recent note, Bob Bridges said:
>
> > Date: Sat, 1 Sep 2007 16:00:42 -0400
> >
> > It's true that REXX seems very inconveniently to reload all my external
> > REXX subroutines dynamically. (New REXXers see footnote below.) But I
> > don't believe that applies to the "TSO/E external functions" described in
> > section 4.4, nor the "TSO/E REXX commands" in chapter 10. Granted I've
> > never tested this... <pause> ...hmm...this assumption. <Pause again, more
> > thoughtful this time.> I now recall a program that ran longer than I
> > thought it should, but I was unable to figure out why. Maybe a
> > NEWSTACK/DELSTACK pair in the middle of a loop?
> >
> years ago I was running an EXEC in a job step with a large STEPLIB.
> I timestamped each line the program write with "date() time()".
> I found the call to date() was taking several hundred milliseconds.
> I opened a PMR and got a big WAD -- IBM diagnosed the overhead as
> directory searches of STEPLIB for tne national language components
> to format the date. They said there is no practical alternative
> because STEPLIB must be searched at each function call lest the
> content of the member be changed between calls, and suggested that
> I obtain the date once at the beginning of the program.
>
> "But what if the date changes during execution of the program?"
>
> "That's not likely enough to be a concern, is it?"
>
> "Isn't it similarly unlikely that the content of the national
> language routines change during execution of the program?
>
> "Regardless how unlikely, we must account for the possibility.
>
This may be, at last and finally, the DUMBEST thing I have ever heard comeo ut of IBM, and given my 37 years in the field, that's saying a lot. Has it occured to them that even *IF* said routine has changed in mid-flight, your program would still be expecting the old one, and that to now load a new version of it would cause issues? Furthermore, do they not know the difference between system utilities and Rexx externals?
Sheesh!!!!
Mickey
Frank Clarke
<090420071448.3297.46DD70230000594...@comcast.net>
>This may be, at last and finally, the DUMBEST thing I have ever heard comeo ut of IBM,
>and given my 37 years in the field, that's saying a lot. Has it occured to them that
>even *IF* said routine has changed in mid-flight, your program would still be
>expecting the old one, and that to now load a new version of it would cause
>issues? Furthermore, do they not know the difference between system utilities
>and Rexx externals?
>
>Sheesh!!!!
Alas, IBM software these days is produced largely by PFCSKs (Pimply-Faced
Computer Science Kids) in C++. I suspect what you're seeing here is merely
another manifestation.
(change Arabic number to Roman numeral to email)
Mickey
> On 4 Sep 2007 07:48:27 -0700, mick...@COMCAST.NET (Mickey Bee) wrote:
> <090420071448.3297.46DD70230000594800000CE122007507840D970A050C0...@comcast.net>
>
> >This may be, at last and finally, the DUMBEST thing I have ever heard comeo ut of IBM,
> >and given my 37 years in the field, that's saying a lot. Has it occured to them that
> >even *IF* said routine has changed in mid-flight, your program would still be
> >expecting the old one, and that to now load a new version of it would cause
> >issues? Furthermore, do they not know the difference between system utilities
> >and Rexx externals?
>
> >Sheesh!!!!
>
> Alas, IBM software these days is produced largely by PFCSKs (Pimply-Faced
> Computer Science Kids) in C++. I suspect what you're seeing here is merely
> another manifestation.
>
> (change Arabic number to Roman numeral to email)
*snort* Computer **Science** indeed. If it was a science, you could
teach it to anyone. It was, is and always will be an art form, which
is why so few do it really well.
Mickey