How to select all by exculded info

[Lizette Koehler]

I have a request to take two lists and to select only those entries that do not appear in both lists.

So list one contains 3 fields, the third field has a userid.
The second list contains one field, the userid, which needs to be used against the first list.

The result should only be those records that do not match the userid in the second list.

I could use a double do loop but list one could be very large; so that might make it too much of a hog. I would like to run this in either BATCH or TSO Foreground.

Is there a simpler way of doing a match-exclude for this process?

A sample do loop might be:

Do i = 1 to list1.0
Do j = 1 to list2.0
If userid.i = userid.j Then exit
Else Say "USERID " userid.i " Has no match"
End j
End i

Or I was contemplating using SAS SQL and do it that way.

Any thoughts?

Lizette

----------------------------------------------------------------------
For TSO-REXX subscribe / signoff / archive access instructions,
send email to LIST...@VM.MARIST.EDU with the message: INFO TSO-REXX

[George.William]

Sounds like something you could run through DFSORT-ICETOOL. This is right up its alley I believe.

Any thoughts?

Lizette

______________________________________________________________________
CONFIDENTIALITY NOTICE: This email from the State of California is for the sole use of the intended recipient and may contain confidential and privileged information. Any unauthorized review or use, including disclosure or distribution, is prohibited. If you are not the intended recipient, please contact the sender and destroy all copies of this email.

[Hardee, Charles H]

Lizette,

If list 2 is truly small (I know, that's relative), consider making a blank delimited list out of it and then checking the userid in list 1 for being present.

For example:

List2 = ""
Do i = 1 to list2.0
List2 = list2 list2.i
End

Do i = 1 to list1.0

If wordpos(userid.1,list2) = 0 then say" User id" userid.i "not in list 2"
End

This way list 2 is loop across once where as in your example it will be looped across for each entry in list 1.

Chuck

-----Original Message-----
From: TSO REXX Discussion List [mailto:TSO-...@VM.MARIST.EDU] On Behalf Of Lizette Koehler
Sent: Tuesday, December 08, 2009 9:58 AM
To: TSO-...@VM.MARIST.EDU
Subject: [TSO-REXX] How to select all by exculded info

[George.William]

Another thought would be via a REXX Edit Macro
Very high level:
- Edit the main list
- read in the USERiD list
- Loop thru each USERID record
Do a FIND on the userid
If not found then Do something
End Loop

Only one loop process this way.

-----Original Message-----
From: TSO REXX Discussion List [mailto:TSO-...@VM.MARIST.EDU] On Behalf Of Lizette Koehler
Sent: Tuesday, December 08, 2009 7:58 AM
To: TSO-...@VM.MARIST.EDU
Subject: [TSO-REXX] How to select all by exculded info

Any thoughts?

Lizette

______________________________________________________________________

[Jeff Byrum]

This is the kind of problem that makes compound variables (stem variables) in REXX so useful. Basically, you loop only once through the second (userid only) list to set up another list of valid userids indexed by userid. Then you go through the first loop (the data), handling those records that don't have a valid userid:

/*-------------------------------------------------------------*/
/* Loop through list2 once to create new list of valid userids */
/*-------------------------------------------------------------*/
userid. = '' /* Set all occurrences to null */
Do i = 1 to list2.0
uid = WORD(list2.i,1)
userid.uid = 'Y' /* Yes, it's a valid userid */
End i

/*-------------------------------------------------------------*/
/* Report all invalid userids */
/*-------------------------------------------------------------*/

Do i = 1 to list1.0

uid = WORD(list1.i,3)
If userid.uid ^= 'Y' then
Say uid 'is not valid'
End i

(tested)

Jeff

-----Original Message-----
From: TSO REXX Discussion List [mailto:TSO-...@VM.MARIST.EDU] On Behalf Of Lizette Koehler
Sent: Tuesday, December 08, 2009 10:58 AM
To: TSO-...@VM.MARIST.EDU

[Paul Gilmartin]

You appear to be trying to put both files into a single
array. That causes problems anyway. So:

Indicator. = 0
do J = 1 to list2.0
UserID = list2.J
Indicator.UserID = 1; end J

do I = 1 to list1.0
parse value list1.I with . . UserID .
if \ Indicator.UserID then
say UserID "not in Indicator array."
end I

-- gil

[Paul Gilmartin]

On Dec 8, 2009, at 09:33, Jeff Byrum wrote:

> This is the kind of problem that makes compound variables (stem
> variables) in REXX so useful. Basically, you loop only once
> through the second (userid only) list to set up another list of
> valid userids indexed by userid. Then you go through the first
> loop (the data), handling those records that don't have a valid
> userid:
>

(My similar example was untested and less complete than
yours.)

This is the kind of problem that makes the widely-held
superstition that compound variables must be indexed by
positive integers with the count in the 0th entry so
harmful. I have sometimes suggested your technique only
to receive the response, "But that's illegal; all of the
examples in TFM use positive integers, and the count
must appear in the 0th entry!"

-- gil

[Robert Zenuk]

Here is another variation on Jeff's solution.

uidlist = ''
do l2=1 to list2.0
uidlist = uidlist word(list2.l2,1)
end
do l1=1 to list1.0
if wordpos(word(list1.l1,3),uidlist) = 0 then
say word(list1.l1,3) 'not found'
end

Rob

In a message dated 12/8/2009 9:33:53 A.M. US Mountain Standard Time,
Jeff....@ASG.COM writes:

This is the kind of problem that makes compound variables (stem variables)
in REXX so useful. Basically, you loop only once through the second
(userid only) list to set up another list of valid userids indexed by userid.
Then you go through the first loop (the data), handling those records that
don't have a valid userid:

/*-------------------------------------------------------------*/

/* Loop through list2 once to create new list of valid userids */
/*-------------------------------------------------------------*/
userid. = '' /* Set all occurrences to null */
Do i = 1 to list2.0
uid = WORD(list2.i,1)
userid.uid = 'Y' /* Yes, it's a valid userid */
End i

/*-------------------------------------------------------------*/
/* Report all invalid userids */
/*-------------------------------------------------------------*/

Do i = 1 to list1.0

uid = WORD(list1.i,3)
If userid.uid ^= 'Y' then
Say uid 'is not valid'
End i

(tested)

Jeff

-----Original Message-----
From: TSO REXX Discussion List [mailto:TSO-...@VM.MARIST.EDU] On Behalf
Of Lizette Koehler
Sent: Tuesday, December 08, 2009 10:58 AM
To: TSO-...@VM.MARIST.EDU
Subject: How to select all by exculded info

I have a request to take two lists and to select only those entries that
do not appear in both lists.

So list one contains 3 fields, the third field has a userid.
The second list contains one field, the userid, which needs to be used
against the first list.

The result should only be those records that do not match the userid in
the second list.

I could use a double do loop but list one could be very large; so that
might make it too much of a hog. I would like to run this in either BATCH or
TSO Foreground.

Is there a simpler way of doing a match-exclude for this process?

A sample do loop might be:

Do i = 1 to list1.0
Do j = 1 to list2.0
If userid.i = userid.j Then exit
Else Say "USERID " userid.i " Has no match"
End j
End i

Or I was contemplating using SAS SQL and do it that way.

Any thoughts?

Lizette

----------------------------------------------------------------------

[Thomas Berg]

I recommend this variant (or gil's) as it's
the most efficient (according to my experience).

Regards,
Thomas Berg
__________________________________________
Thomas Berg Specialist IT-U SWEDBANK

> -----Ursprungligt meddelande-----
> Fr�n: TSO REXX Discussion List
> [mailto:TSO-...@VM.MARIST.EDU] F�r Jeff Byrum
> Skickat: den 8 december 2009 17:34
> Till: TSO-...@VM.MARIST.EDU
> �mne: Re: [TSO-REXX] How to select all by exculded info

[Robert Zenuk]

I got curious about the efficiency...

Here are 3 variations of the EXEC

UMATCH1

do l1=1 to list1.0
do l2=1 to list2.0
if word(list2.l2,1) = word(list1.l1,3) then
iterate l1
end

say word(list1.l1,3) 'not found'
end

UMATCH2

uidlist = ''
do l2=1 to list2.0
uidlist = uidlist word(list2.l2,1)
end
do l1=1 to list1.0
if wordpos(word(list1.l1,3),uidlist) = 0 then
say word(list1.l1,3) 'not found'
end

UMATCH3

do l2=1 to list2.0
parse var list2.l2 uid .
userid.uid = 'Y'

end
do l1=1 to list1.0

parse var list1.l1 . . uid .
if userid.uid <> 'Y' then
say uid 'not found'
end

I ran with a LIST1 (file to cross reference - lots of duplicates) file with
33,813 records
I ran with a LIST2 (valid userids - also lots of duplicates) file with 900
ID's
I started with 9 and 13 records respectively and just replicated the
lines...

Initial data

LIST1

X Y USERA
X Y USER2
X Y USER3
X Y USER4
X Y USERB
X Y USER2
X Y USER3
X Y USER4
X Y USER2
X Y USER3
X Y USER4
X Y USERC
X Y USER9

LIST2

USER1
USER2
USER3
USER4
USER5
USER6
USER7
USER8
USER9

Here are the results:

STEPNAME PROCSTEP PROGRAM RC EXCP CPU SRB CLOCK
SERV
UMATCH1 UMATCH IKJEFT01 00 817 16.88 .00
23.00 612K
UMATCH2 UMATCH IKJEFT01 00 815 1.54 .00
2.09 56422
UMATCH3 UMATCH IKJEFT01 00 816 .26 .00
.50 9927

Jeff wins!

Slow morning... actually just lacking motivation for real work this
morning...

Rob



In a message dated 12/8/2009 10:38:40 A.M. US Mountain Standard Time,

[Pinnacle]

Gil's solution is better in that it uses a boolean value and tests for it.
It saves two lookups for literals and executes much faster. So use
userid.uid = 1 and if \userid.uid. I once did this for all my 'Y' flags in
a program and got a 20% performance boost just from that. YMMV, but it will
run faster with booleans.

Regards,
Tom Conley

[Pinnacle]

Rob,

Change UMATCH3 to use a boolean instead of 'Y' and post the results, please.

Thanks,
Tom Conley

[Lizette Koehler]

Wow - How Awesome!

I guess it is true what they say - Christmas Time is time for play (not work)

Thanks for the stats.

I will definitely work with Jeff's process. I have not worked enough with PARSE to have it as a function I usually use. I knew it was very powerful, just not this much.

Lizette

>In a message dated 12/8/2009 10:38:40 A.M. US Mountain Standard Time,
>thoma...@SWEDBANK.SE writes:
>
>I recommend this variant (or gil's) as it's
>the most efficient (according to my experience).
>
>
>
>Regards,
>Thomas Berg
>__________________________________________
>Thomas Berg Specialist IT-U SWEDBANK
>
>
>
>
>
>> -----Ursprungligt meddelande-----

>> Från: TSO REXX Discussion List
>> [mailto:TSO-...@VM.MARIST.EDU] För Jeff Byrum

>> Skickat: den 8 december 2009 17:34
>> Till: TSO-...@VM.MARIST.EDU

>> Ämne: Re: [TSO-REXX] How to select all by exculded info

>> -----Original Message-----
>> From: TSO REXX Discussion List
>> [mailto:TSO-...@VM.MARIST.EDU] On Behalf Of Lizette Koehler
>> Sent: Tuesday, December 08, 2009 10:58 AM
>> To: TSO-...@VM.MARIST.EDU
>> Subject: How to select all by exculded info
>>
>> I have a request to take two lists and to select only those
>> entries that do not appear in both lists.
>>
>> So list one contains 3 fields, the third field has a userid.
>> The second list contains one field, the userid, which needs
>> to be used against the first list.
>>
>> The result should only be those records that do not match the
>> userid in the second list.
>>
>> I could use a double do loop but list one could be very
>> large; so that might make it too much of a hog. I would like
>> to run this in either BATCH or TSO Foreground.
>>
>> Is there a simpler way of doing a match-exclude for this process?
>>
>> A sample do loop might be:
>>

>> Do i = 1 to list1.0

>> Do j = 1 to list2.0
>> If userid.i = userid.j Then exit
>> Else Say "USERID " userid.i " Has no match"
>> End j
>> End i
>>
>> Or I was contemplating using SAS SQL and do it that way.
>>
>> Any thoughts?
>>
>> Lizette
>>

[Robert Zenuk]

I used Tom's suggestion and here is the modified code and stats.

UMATCH4

userid. = 0

do l2=1 to list2.0
parse var list2.l2 uid .

userid.uid = 1

end
do l1=1 to list1.0
parse var list1.l1 . . uid .

if \userid.uid then

say uid 'not found'
end

I did not run UMATCH1 again. CPU was too high and was probably showing up
on someone's radar screen...

STEPNAME PROCSTEP PROGRAM RC EXCP CPU SRB CLOCK
SERV

UMATCH2 QMATCH IKJEFT01 00 819 1.53 .00
2.52 56088
UMATCH3 QMATCH IKJEFT01 00 818 .26 .00
.50 9738
UMATCH4 QMATCH IKJEFT01 00 818 .25 .00 .43
9357

Some minor improvement... Just under 4 percent. This is probably one of
those readability and maintainability versus performance decisions...

Rob

In a message dated 12/8/2009 11:40:06 A.M. US Mountain Standard Time,
pinn...@ROCHESTER.RR.COM writes:

----- Original Message -----
From: "Robert Zenuk" <Robz...@AOL.COM>
Newsgroups: bit.listserv.tsorexx
Sent: Tuesday, December 08, 2009 1:17 PM

Rob,

Change UMATCH3 to use a boolean instead of 'Y' and post the results,
please.

Thanks,
Tom Conley

----------------------------------------------------------------------

[George.William]

You'll grow to LOVE IT!

Wow - How Awesome!

Thanks for the stats.

Lizette

______________________________________________________________________

[Pinnacle]

----- Original Message -----
From: "Robert Zenuk" <Robz...@AOL.COM>
Newsgroups: bit.listserv.tsorexx
Sent: Tuesday, December 08, 2009 1:56 PM
Subject: Re: SV: How to select all by exculded info

Thanks to Rob for running it and posting the results. If I remember Rob's
numbers, the sample run took about 35000 iterations and booleans gave a 4%
CPU and 14% SRB savings over characters. Extrapolate that to millions of
iterations, like some of my Rexx code does, and you will get some serious
performance gains. I learned to use booleans instead of characters from no
less than Chip Davis, past SHARE Rexx project manager and currently maven
for RexxLA, the guardians of OoRexx (still owe him a steak dinner for that
one). I added this little goody to my Rexx Tips and Tricks session at
SHARE, and it always delivers. I use this feature that I call named stems,
and that Phil Smith calls associative arrays, to heavily index my stem
variables, so that I can instantly detect the existence of a value. It's
one of the most powerful uses for stem variables.

Regards,

[Mickey]

Whereas I, having seen the underlying code, avoid it like the plague. That
being said, if you use it ONCE in a program, you might as well use it 20
times, as once the macro is included, it's there.

Mickey

--------------------------------------------------
From: "George.William" <William...@FTB.CA.GOV>
Sent: Tuesday, December 08, 2009 10:56 AM

[Mickey]

--------------------------------------------------
From: "Pinnacle" <pinn...@ROCHESTER.RR.COM>
Sent: Tuesday, December 08, 2009 11:40 AM
To: <TSO-...@VM.MARIST.EDU>
Subject: Re: [TSO-REXX] SV: How to select all by exculded info

I do also, although in a slightly different manner. Mine would look more
like this.

Lookup. = 0
"EXECIO * DISKR somefile (STEM INPUT1. FINI"
"EXEICO ( DISKR smotherfile (STEM INPUT2. FINI"

Do i = 1 to INPUT1.0
trash = strip(INPUT1.I)
LookUp.Trash = 1
End

Do j = 1 to INPUT2.0
trash = Strip(INPUT2.J)
If ^LookUp.trash Then
Say Trash 'Not Found'
End

Might not be quite as fast, but I think it is FAR more readable.

Mickey

[Jeff Byrum]

Glad it's working out!

Just to clarify, my solution didn't use PARSE -- I used WORD to pick out the third field in the record. Or you could just use SUBSTR, if you're sure about the userid always being in a fixed column. The important part is setting up the third compound (stem) variable with the userids as indexes, instead of an array indexed by numbers that you have to loop through. And Gil is right -- for that kind of stem variable, you don't have to give a value to the "zero-th" occurrence.

Thanks to Robert for running the actual performance tests. I always had a sense it was pretty fast (I use the technique a lot), but it's good to have data to back that up!

Thanks also to Tom for the suggestion about Booleans. It's something I have tended to avoid for no good reason; just one of those things where there's a hole I haven't filled in yet-- mostly, I can never remember whether 1 is "yes" or "no"! But based on the results (and depending on the application) something I'll certainly consider more.

But I think the biggest thanks go to Mike Colishaw. I think this "named stem" feature (the ability to use strings as indexes) is one of the most powerful language features around. If anyone ever made me go back to writing assembler, the first thing I'd do is create a assembler subroutine that supported it.

The folks on this list are such a great resource. Thanks to all!

[Jeff Byrum]

I don't know if Mike Cowlishaw follows this list, but, Mike, if you do, sorry for misspelling your name!

[Robert Zenuk]

I took some literary license with Jeff's example (adding PARSE). I had
always heard it was faster... Well, I benchmarked my version with PARSE and
Mickey and Jeff's version with WORD and there is no difference... The
boolean approach was the only improvement, then only at 4%. I would tend to
agree with Mickey that the readability and maintainability factor has to be
considered (unless you are running on an old box with only 5 MIPS)...

New results:

UMATCH2 mine using WORDPOS
UMATCH3 Jeff's substituting PARSE
UMATCH4 UMATCH3 with boolean
UMATCH5 Mickey's
UMATCH6 Jeff's original

STEPNAME PROCSTEP PROGRAM RC EXCP CPU SRB CLOCK
SERV

UMATCH2 QMATCH IKJEFT01 00 801 1.48 .00
1.64 47996
UMATCH3 QMATCH IKJEFT01 00 800 .24 .00
.32 8074
UMATCH4 QMATCH IKJEFT01 00 801 .23 .00 .31
7900
UMATCH5 QMATCH IKJEFT01 00 805 .24 .00 .36
8282
UMATCH6 QMATCH IKJEFT01 00 811 .24 .00 .33
8230

BTW, there was no SRB values for comparison (all were zero), chalk that up
to column skewing with font changes... The second to last values is the
wall clock time... Service Units were slightly lower with PARSE, but it
did not translate to any savings in CPU time...

Rob




In a message dated 12/8/2009 1:34:29 P.M. US Mountain Standard Time,
Jeff....@ASG.COM writes:

Jeff

Wow - How Awesome!

Thanks for the stats.

Lizette

>> Fr�n: TSO REXX Discussion List
>> [mailto:TSO-...@VM.MARIST.EDU] F�r Jeff Byrum

>> Skickat: den 8 december 2009 17:34
>> Till: TSO-...@VM.MARIST.EDU

>> �mne: Re: [TSO-REXX] How to select all by exculded info

[Paul Gilmartin]

On 12/08/09 11:55, Robert Zenuk wrote:
>
> Some minor improvement... Just under 4 percent. This is probably one of
> those readability and maintainability versus performance decisions...
>
>

FORTRAN and BASIC have polluted programmers' brains with the
misbelief that the only allowable boolean primaries are
relational expressions. I have seen, for example in C:

#define FALSE 0
#define TRUE 1
...
if ( userid[ uid ] == TRUE ) { ... }

instead of:

if ( userid[ uid ] ) { ... }

I do not view this as contributing to readability and maintainability.

Ugh!

(I might consider naming the array "exists" instead of "userid"
for readability.)

> In a message dated 12/8/2009 11:40:06 A.M. US Mountain Standard Time,
> pinn...@ROCHESTER.RR.COM writes:
>
> Change UMATCH3 to use a boolean instead of 'Y' and post the results,
> please.

-- gil

[Paul Gilmartin]

On 12/08/09 11:56, George.William wrote:
> You'll grow to LOVE IT!
>
> -----Original Message-----
> From: TSO REXX Discussion List [mailto:TSO-...@VM.MARIST.EDU] On Behalf Of Lizette Koehler
> Sent: Tuesday, December 08, 2009 10:52 AM
>

> I will definitely work with Jeff's process. I have not worked enough with PARSE to have it as a function I usually use. I knew it was very powerful, just not this much.
>

Style. I overuse PARSE, so I perceive others as overusing WORD(), etc.

-- gil

[Paul Gilmartin]

On 12/08/09 12:40, Pinnacle wrote:
>
> one). I added this little goody to my Rexx Tips and Tricks session at
> SHARE, and it always delivers. I use this feature that I call named stems,

> and that Phil Smith calls associative arrays, ...
>
That's UNIXy jargon. Rexx and awk are a lot similar.

-- gil

[Paul Gilmartin]

On 12/08/09 13:33, Jeff Byrum wrote:
> ... for that kind of stem variable, you don't have to give a value to the "zero-th" occurrence.
>
If the tails are floating point numbers, what _would_ one put in
the 0th occurrence?

> Thanks to Robert for running the actual performance tests. I always had a sense it was pretty fast (I use the technique a lot), but it's good to have data to back that up!
>

> Booleans. It's something I have tended to avoid for no good reason;
> just one of those things where there's a hole I haven't filled in yet
> -- mostly, I can never remember whether 1 is "yes" or "no"!
>

With little loss in performance, one can set up:

YES = 1
NO = 0

... and copy it into all one's programs. Just remember to
use "if exists.tail" rather than "if exists.tail==YES". It's
less harm to use:

if word( user, LIST )>0 then exists.user = YES
else exists.user = NO

... but better:

exists.user = ( word( user, LIST )>0 )

(At this point, I'll steadfastly ignore all arguments about
"readability". Programmers should learn the languages they use.)

-- gil

[Paul Gilmartin]

On 12/08/09 13:33, Jeff Byrum wrote:
>

> If anyone ever made me go back to writing assembler,
> the first thing I'd do is create a assembler subroutine that supported it.
>

In other languages, likely assembler, this is called a "symbol table";
there must be reusable subroutine packages for this. Associative arrays
are supported by awk; C supports the facilities in /usr/include/search.h
(Alas, the POSIX specification precludes optimization by balancing the
tree.) And I can hardly doubt that Java has a class for this.

-- gil

[Arthur T.]

On 8 Dec 2009 16:48:21 -0800, in bit.listserv.tsorexx
(Message-ID:<4B1EF37A...@AIM.com>)
PaulGB...@AIM.COM (Paul Gilmartin) wrote:

>On 12/08/09 13:33, Jeff Byrum wrote:

>>Booleans. It's something I have tended to avoid for no
>>good reason;
>>just one of those things where there's a hole I haven't
>>filled in yet
> > -- mostly, I can never remember whether 1 is "yes" or
> "no"!
>With little loss in performance, one can set up:
>
> YES = 1
> NO = 0

I also have problems remembering which is which, and
even more problems remembering if it's the same across
languages. So in REXX I usually code (somewhere in my
initialization):

true = (0=0)
false = \true

--
I cannot receive mail at the address this was sent from.
To reply directly, send to ar23hur "at" intergate "dot" com

[Robert Zenuk]

One of my mentors had a great saying. This guy was the most prolific
programmer I ever met. As a former IBM'er he was personally responsible for
many amazing things in several IBM products on several platforms (Mainframe,
Unix, AS/400, PC's, Toaster's, etc) . He is an amazing Assembler, COBOL,
PL/I, APL, RPG, C, C++ and most recently (for the last 12 years) Java
programmer (he did not count interpretive languages as real languages).

He always said there is GOOD code and there is CLEVER code. He said he
never wrote CLEVER code because it was harder to maintain. His code was
always as efficient as anyone else's (usually more efficient), but was always
very readable. He did not believe in tricks, smoke or mirrors. He knew
them all and could explain why each worked and he could decipher anyone's
code, but he refused to use them himself. He would produce some amazing "1
liners" as examples and show us performance benchmarks proving his GOOD code
was just as efficient as the CLEVER example. His GOOD code was always the
shining example of how to do something. The net results was everyone loved
his code and following him on a project.

and that's all I have to say about that... and life is like a box of
chocolates... and run Forest run...

Rob

In a message dated 12/8/2009 5:48:17 P.M. US Mountain Standard Time,
PaulGB...@AIM.COM writes:

(At this point, I'll steadfastly ignore all arguments about
"readability". Programmers should learn the languages they use.)

[Thomas Giordano]

Lizette mentioned in the original post the option of using SAS, so how does SAS compare. I do realise SAS is designed for this sort of thing and REXX shouldn't really be expected to provide the same performance.

I don't agree with using PROC SQL as Lizette suggested, I think a data step such as the one below is a better approach and probably faster.

DATA OUTPUT;
MERGE LIST1(IN=IN_L1)
LIST2(IN=IN_L2);
BY USERID;
IF IN_L1 AND NOT IN_L2 THEN OUTPUT;

This expects that the SAS datasets LIST1 and LIST2 already exist and contain the input data sorted by userid.

I ran the same test as Robert (with the same amount of input data) but the CPU usage was too low to get meaningful numbers.
Below is from the 4 rexx solutions Robert has already provided results for.
PROCSTEP RC EXCP CPU SRB CLOCK
UMATCH1 00 2298 .28 .00 1.36
UMATCH2 00 2299 .02 .00 .20
UMATCH3 00 2299 .00 .00 .02
UMATCH4 00 2301 .00 .00 .05

I increased the size of both LIST1 and LIST2 by 5 (LIST1 = 202878 records, LIST2 = 5454 records) and compared the REXX performance to SAS. I left out UMATCH1 this time.

PROCSTEP RC EXCP CPU SRB CLOCK
UMATCH2 00 13597 .76 .00 2.46
UMATCH3 00 13597 .02 .00 .06
UMATCH4 00 13598 .02 .00 .12
SAS 00 15765 .01 .00 .18

The SAS solution uses two data steps to read the input lists, two proc sorts to sort the lists then the merge data step above. I'm not the most knowledgeable with SAS so it can probably be done better.

Who wants to try DFSORT.


Tom.

[Jeremy Nicoll - LStsrx]

Paul Gilmartin <PaulGB...@AIM.COM> wrote:

> I have sometimes suggested your technique only to receive the response,
> "But that's illegal; all of the examples in TFM use positive integers, and
> the count must appear in the 0th entry!"

You don't need to go past p.5 in the original TRL book by MC to see an
example of a stem indexed by random word data, and incidentally also the
values in the stem are boolean.

--
Jeremy C B Nicoll - my opinions are my own.

[Pinnacle]

----- Original Message -----
From: "Robert Zenuk" <Robz...@AOL.COM>
Newsgroups: bit.listserv.tsorexx
Sent: Tuesday, December 08, 2009 11:45 PM
Subject: Re: SV: How to select all by exculded info

> One of my mentors had a great saying. This guy was the most prolific
> programmer I ever met. As a former IBM'er he was personally responsible
> for
> many amazing things in several IBM products on several platforms
> (Mainframe,
> Unix, AS/400, PC's, Toaster's, etc) . He is an amazing Assembler, COBOL,
> PL/I, APL, RPG, C, C++ and most recently (for the last 12 years) Java
> programmer (he did not count interpretive languages as real languages).
>
> He always said there is GOOD code and there is CLEVER code. He said he
> never wrote CLEVER code because it was harder to maintain. His code was
> always as efficient as anyone else's (usually more efficient), but was
> always
> very readable. He did not believe in tricks, smoke or mirrors. He knew
> them all and could explain why each worked and he could decipher anyone's
> code, but he refused to use them himself. He would produce some amazing
> "1
> liners" as examples and show us performance benchmarks proving his GOOD
> code
> was just as efficient as the CLEVER example. His GOOD code was always
> the
> shining example of how to do something. The net results was everyone
> loved
> his code and following him on a project.
>

Since I grew up coding PL/I and Pascal, the use of boolean values (1 is true
or on, 0 is false or off) is second nature to me, and not something I
consider as "clever". I don't understand why booleans look so strange to
you guys, but that's probably just my background. I write a lot of Rexx
code that executes for a long time, so I look for every performance
advantage I can get. To me, the statements "if exist = 'N'" and "if \exist"
are functionally equivalent, and I have no trouble reading either of them.
I just know that "if \exist" is more efficient because it eliminates a
variable lookup. I don't understand why so many people here consider
booleans "clever", "unreadable", or "unmaintainable".

My $.02,
Tom

[Mickey]

--------------------------------------------------
From: "Pinnacle" <pinn...@ROCHESTER.RR.COM>
Sent: Wednesday, December 09, 2009 7:41 AM
To: <TSO-...@VM.MARIST.EDU>
Subject: Re: [TSO-REXX] SV: How to select all by exculded info

It comes as second nature to me also.... has to do with physically setting
UPSI switches (during an age when carnosaurs ruled the planet), and 1 was on
and 0 was off.

Mickey

[David S Speake]

DFSORT?

Whatsamatter - don't you have SYNCSORT JOIN?

:-)

Quickly donning asbestos suit.

David.Speake
David....@bcbssc.com
(803)-264-8003

[Paul Gilmartin]

On Dec 9, 2009, at 05:41, Pinnacle wrote:
>
> Since I grew up coding PL/I and Pascal, the use of boolean values
> (1 is true
> or on, 0 is false or off) is second nature to me, and not something I
> consider as "clever". I don't understand why booleans look so
> strange to
> you guys, but that's probably just my background. I write a lot of
> Rexx
> code that executes for a long time, so I look for every performance
> advantage I can get. To me, the statements "if exist = 'N'" and
> "if \exist"

ITYM "if exist = '0'" if "exist" is a Rexx boolean value.

This slip in some ways affirms and in others refutes Rob's point.

> are functionally equivalent, and I have no trouble reading either
> of them.
> I just know that "if \exist" is more efficient because it eliminates a
> variable lookup. I don't understand why so many people here consider
> booleans "clever", "unreadable", or "unmaintainable".
>

Overall I agree strongly.

-- gil

[Rick Woods]

ITYM?
- Rick

>>> Paul Gilmartin <PaulGB...@AIM.COM> 12/9/2009 8:35 AM >>>

ITYM "if exist = '0'" if "exist" is a Rexx boolean value.

----------------------------------------------------------------------

[Walter Pachl]

Nowadays one googles before asking :-)

ITYM - What does ITYM stand for? Acronyms and abbreviations by the ... -
[ Diese Seite �bersetzen ]
Acronym, Definition

I Think You Mean "I Think You Mean"

[Rick Woods]

LOL, I guess I haven't caught up yet, it didn't occur to me to google
(even though I use it all day long).
*sigh*
Thanks.

>>> Walter Pachl <pa...@CHELLO.AT> 12/9/2009 2:44 PM >>>

Nowadays one googles before asking :-)

ITYM - What does ITYM stand for? Acronyms and abbreviations by the ...
-

[ Diese Seite übersetzen ]

[John McKown]

I know what Syncsort is. What is Syncsort Join? We are not really very
open to changing from DFSORT to Syncsort. They always seem to be very
competitive and not likely to save us any money.

[max herman]

> send email to LISTS...@VM.MARIST.EDU with the message: INFO TSO-REXX

Have you considered using ICETOOL? I'd have thought it perfect for
this kind of job.

[Donald Johnson]

I ran two random files of this size through options 2, 3, and 4 of the REXX,
and through SAS and CA Easytrieve Plus (it has matched file processing
capabilities). In both the SAS and Easytrieve cases, it was easy to drop the
duplicates in the files before doing the match, thus cutting down on the
processing. I would think that in the "real world" the records would be
sorted and de-duped before trying to match anyway.

Here are my numbers (SAS and EZT include the initial file read, the sort and
the match. the REXX includes the EXECIO read of unsorted recs to stem vars
and the processing as defined before):

STEPNAME STEP PGM= CCODE EXCPS ELAPSED TOT-CPU
USRMTCH2 4 IKJEFT01 0000 104 00:00:07.86 00:00:03.38
USRMTCH3 5 IKJEFT01 0000 104 00:00:01.07 00:00:00.49
USRMTCH4 6 IKJEFT01 0000 104 00:00:00.98 00:00:00.48
SASMTCH 7 SAS 0000 1835 00:00:03.26 00:00:00.21
EZTMTCH 8 EZT 0000 7113 00:00:09.88 00:00:00.32

Increasing to filesizes of 500,000 and 15,000, here is what happened:
STEPNAME STEP PGM= CCODE EXCPS ELAPSED TOT-CPU
USRMTCH2 4 IKJEFT01 0000 1480 01:00:09.78 00:11:20.76
USRMTCH3 5 IKJEFT01 0000 1480 00:00:34.88 00:00:06.82
USRMTCH4 6 IKJEFT01 0000 1480 00:01:09.35 00:00:07.07
SAS9 7 SAS 0000 3512 00:00:12.44 00:00:01.04
EZTMTCH 8 EZT 0000 99651 00:02:19.41 00:00:03.23

It looks like SAS could be the big winner here...
Don

On Wed, Dec 9, 2009 at 1:24 AM, Thomas Giordano <
Thomas....@dcb.defence.gov.au> wrote:

> Lizette mentioned in the original post the option of using SAS, so how does
> SAS compare. I do realise SAS is designed for this sort of thing and REXX
> shouldn't really be expected to provide the same performance.
>
> I don't agree with using PROC SQL as Lizette suggested, I think a data step
> such as the one below is a better approach and probably faster.
>
> DATA OUTPUT;
> MERGE LIST1(IN=IN_L1)
> LIST2(IN=IN_L2);
> BY USERID;
> IF IN_L1 AND NOT IN_L2 THEN OUTPUT;
>
>

----------------------------------------------------------------------