Variables in FOR... NEXT...

[Bob Dubery]

I have been debugging a program that has multiple nested loops, with this statement for the outermost loop

FOR ElementCnt = 1 TO NumElements

Both of those variables may be modified within the loop.

NumElements is the number of attributes in a dynamic array - it runs into the 10000s. The array is modified in various ways

1) New attributes added to the end

2) New attributes inserted mid array (somewhere other than the end)

3) An attribute may be deleted

In each case NumElements is incremented or decremented (rather than a DCOUNT). ElementCnt may be decremented if the attribute deleted is at the position indicated by ElementCnt.

So this raises in my mind some questions about FOR. Z = 1 TO Y

Is Y evaluated for every iteration? Or is there some flag set to say that the variable has been updated and so next time around re-evaluate?

[Bob Dubery]

I should add that this is UV 10.1 (or 10.something else, but definitely 10.something) on NT (or whatever it's called these days)

[Wols Lists]

It certainly always used to be the case that Y was evaluated every
iteration. It would break a LOT of code if it isn't.

But what do you mean by "evaluated"? It is perfectly normal to compare
the current value of Z with the current value of Y every time the FOR
statement is evaluated. That makes a lot of sense.

It's only certain languages (FORTRAN springs to mind) where it is
explicitly legal to cache the value of Y such that any changes aren't
taken into account.

And given that code along the lines of

FOR element = 1 TO DCOUNT( array, @FM)

has long been known to be a major source of inefficiency (due to the
repeated evaluation of DCOUNT) I think any attempt to future-optimise
this in the compiler will be a losing strategy ...

(If you want to optimise the processing of these thousands of elements,
ask on the list. Normally I'd say "use a dimensioned array", but adding
and deleting elements in the middle of the array make this a little
tricky ...)

Cheers,
Wol

[Peter McMurray]

Hi Bob

The process is disastrously inefficient, but not for the position evaluation with a simple variable. Using a recalculated variable on every iteration is, as has been pointed out, extremely stupid.. 

To use a dynamic array consider this; every time a change is made the dynamic array has to be rebuilt. That is a chunk of memory has to be found, the first half moved in, the new element inserted/altered/deleted and then the next chunk moved in. If they had used an option that I believe exists in UV that maintains the current position of the pointer in a dynamic array and built a new dimensioned array not only would the job be more efficient and easier to maintain it would be orders of magnitude faster.

 In D3 one would always be better off reading in as a dimensioned array and building a new one allowing for the possible increase in size at the point of originally dimensioning it.

D3 Pick retains the original design of dimensioned arrays being blocks of 10 byte variables each being instantly addressable with a link to any variable that requires more space. I doubt that U2 changed this brilliant piece of design although I have not tested it.

By the way NT is now Windows server 2016 or Windows 10 - the latter being the preferred environment for smaller businesses when used in a peer to peer configuration. The file system on Windows 10 is NTFS whereas Server 2012 onwards is ReFS. Both are capable of very significant file storage size; backup, security and resilience are the major changes.

[Wols Lists]

On 29/07/17 22:46, Peter McMurray wrote:
> Hi Bob
> The process is disastrously inefficient, but not for the position
> evaluation with a simple variable. Using a recalculated variable on
> every iteration is, as has been pointed out, extremely stupid..
> To use a dynamic array consider this; every time a change is made the
> dynamic array has to be rebuilt. That is a chunk of memory has to be
> found, the first half moved in, the new element inserted/altered/deleted
> and then the next chunk moved in. If they had used an option that I
> believe exists in UV that maintains the current position of the pointer
> in a dynamic array and built a new dimensioned array not only would the
> job be more efficient and easier to maintain it would be orders of
> magnitude faster.

Except you've missed a crucial point ...

His requirements are to add and delete elements IN THE MIDDLE OF THE
ARRAY. How do you do that with a dimensioned array (other than an
extremely inefficient copy from one element to another ...) ?

Yes there are ways around it, but the simplistic "use a dimensioned
array" solution is a non-starter - this particular problem needs far
more information to solve.

Cheers,
Wol

[Wols Lists]

On 29/07/17 22:46, Peter McMurray wrote:

> In D3 one would always be better off reading in as a dimensioned array
> and building a new one allowing for the possible increase in size at the
> point of originally dimensioning it.
> D3 Pick retains the original design of dimensioned arrays being blocks
> of 10 byte variables each being instantly addressable with a link to any
> variable that requires more space. I doubt that U2 changed this
> brilliant piece of design although I have not tested it.

U2 arrays are SIGNIFICANTLY different from D3 arrays ...

DIM ARRAY(4)
DIM ARRAY(6)
ARRAY(0) = ""

Okay, D3 now implements U2- (actually PI-) style arrays and vice versa,
but the above code compiles - and works - fine in PI-mode. It's
completely illegal in classic D3-mode.

Cheers,
Wol

[Peter McMurray]

Hi Wol

It appears that you have completely missed my point. I did my first loads using this technique when picking up tens of thousands of debtors off BP's mainframe produced files - the fact that I had to convert the tapes from EBCDIC 32Kb to ASCII 1600 byte was another intriguing issue :-).

One simply keeps two counters one for the incoming record and one for the new array.

In the case described the FOR loop control variable can be the counter for the source and one simply increments another counter for the NEW array. every time one stores a record in it. Thus it will oscillate above and below the source count depending on whether a new record is going in or not.

[Peter McMurray]

Hi Wol

Your post reminded of the disastrous REDIM in R83. I have never used it but I thought I would give it a burn in D3. It is so long since I use PI I cannot remember what that did.

As you can see there is no problem adding dimensions so one does not even have to take an educated guess as to final size.

trydim

..

[820] Creating FlashBASIC Object ( Level 0 )

[241] Successful compile!   2 frame(s) used.

:run mmbp trydim

1

Hi There

:ct mmbp trydim

    trydim

001 gosub 1000

002 crt fred(a)

003 dim fred(2000)

004 fred(1500) = "Hi There"

005 crt fred(1500)

006 stop

007 1000 *

008 a = 99

009 dim fred(a)

010 mat fred = 1

011 return

[Bob Dubery]

On Saturday, 29 July 2017 23:46:53 UTC+2, Peter McMurray wrote:

Hi Bob

The process is disastrously inefficient, but not for the position evaluation with a simple variable. Using a recalculated variable on every iteration is, as has been pointed out, extremely stupid.. 

My question is whether or not that variable is recalculated.

If the loop is controlled by FOR Z = 1 TO Y, then clearly every time around the loop there is a test to see if Z <= Y.   

To use a dynamic array consider this; every time a change is made the dynamic array has to be rebuilt. That is a chunk of memory has to be found, the first half moved in, the new element inserted/altered/deleted and then the next chunk moved in. If they had used an option that I believe exists in UV that maintains the current position of the pointer in a dynamic array and built a new dimensioned array not only would the job be more efficient and easier to maintain it would be orders of magnitude faster.

I thought about that. But IIRC, when the array changes, the pointer is reset. 

I should reveal more about the program, though I'm not seeking to rewrite or refactor it. It's not going to be in use a lot longer, so I have to debug it now,

The program is navigating an XSD and trying to build up a tree structure, starting with the root element, of what actual elements are used and the hierarchy of those elements. It looks for bottom level elements, not types that may reference other elements. It has to resolve the types.

I would have done this differently, and whilst a discussion about the best way to do it would be interesting, see above! My brief is not to reinvent or improve the wheel as the program, in UV, will not be with us for much longer.

It has to work with a dynamic array, because it has no way of predicting how many elements will be in that array (last test it was in the order of 23000). It can't DIM a matrix, without first traversing the entire XSD and building the tree structure in the first place.

What is known is that the array will start with ONE element. So we can set a variable NumElements TO 1 and start a loop

FOR ElementCnt = 1 To NumElements.

But the array grows, and as it grows NumElements is incremented, not  by a DCOUNT, but by NumElements += 1.

It may also shrink, in which case NumElements -= 1.

I assume it is done this way rather than simply having NumElements =- DCOUNT(ElementList,@AM) at the bottom of the loop in order to save run time.

In some circumstances, ElementCnt is also reduced by ElementCnt -= 1.

So I started wondering about NumElements and if it is calculated every time around the loop.

ISTR a paper from my Prime Information days that said that FOR X = 1 TO DCOUNT(ARRAY) was wasteful, but also that there was a way to have Q in FOR X = 1 TO Q not be evaluated every time, but since a test X <= Q must take place every time, what is the use of that?

[Wols Lists]

On 30/07/17 07:03, Bob Dubery wrote:
> ISTR a paper from my Prime Information days that said that FOR X = 1 TO
> DCOUNT(ARRAY) was wasteful, but also that there was a way to have Q in
> FOR X = 1 TO Q not be evaluated every time, but since a test X <= Q must
> take place every time, what is the use of that?

There's no problem evaluating Q if Q is a variable. That's cheap (to the
price of non-existent :-)

The problem is where Q is a function - that needs to be called and
evaluated every single time. Basically, the compiler will fix that - it
will contain code that says "if Q is a function I need to calculate Q
before I compare". If Q isn't a function there will be none of that code
in the runtime. If Q is a function then the program will be slow because
it will call it every time round the loop.

I always used to code it as

LIMIT = DCOUNT( ARRAY, @FM); FOR I = 1 TO LIMIT
...

Cheers,
Wol

[Chuck Stevenson]

Learn to use VLIST.  It shows you a readable decompiled version of the compiled object.  You can what happens during execution.

Likewise, RAID's "I" command will let you step through each object instruction.  "X" displays the current object code & instruction.

Short answer: yes, Y is reevaluated every time.

This is why:

   MAXV = DCOUNT( X, @VM )

   FOR V = 1 TO MAXV

is preferable to

   FOR V = 1 TO DCOUNT( X, @VM )

[Chuck Stevenson]

VLIST example for CDS.BP BOB

0001       X = 1:@VM:2

0002 *

0003       FOR V = 1 TO DCOUNT( X, @VM )

0004          CRT X<1,V>

0005       NEXT V

0006 *

0007       MAXV = DCOUNT( X, @VM )

0008       FOR V = 1 TO MAXV

0009          CRT X<1,V>

0010       NEXT V

It reads a lot like any assembler I've ever read.

The general format of each line is:

The 1st number is source line number

The 2nd number is an object hex address  (This is the address PORT.STATUS command shows.)

Then comes the instruction, followed by operands.

>VLIST CDS.BP BOB

00001:       X = 1:@VM:2

00001 00000 : 104 multi_cat      1 "ý" 2  => X

00002: *

00003:       FOR V = 1 TO DCOUNT( X, @VM )

00003 0000C : 0F8 move           0  => V

00003 00012 : 051 dcount         X "ý"  => $R0      <--- NEXT V creates a jump back to here.  DCOUNT repeats.

00003 0001A : 098 forincr        V $R0 1 00040:

00004:          CRT X<1,V>

00004 00028 : 060 dyn_extract    X 1 V 0  => $R0

00004 00034 : 046 crtcrlf        $R0

00005:       NEXT V

00005 0003A : 0C2 jump           00012:

00006: *

00007:       MAXV = DCOUNT( X, @VM )

00007 00040 : 051 dcount         X "ý"  => MAXV

00008:       FOR V = 1 TO MAXV

00008 00048 : 0F8 move           0  => V

00008 0004E : 098 forincr        V MAXV 1 00074:        <--- NEXT V jumps to here.  DCOUNT not repeated.

00009:          CRT X<1,V>

00009 0005C : 060 dyn_extract    X 1 V 0  => $R0

00009 00068 : 046 crtcrlf        $R0

00010:       NEXT V

00010 0006E : 0C2 jump           0004E:

00011 00074 : 190 stop

>

[Peter McMurray]

Now we have a sensible explanation. It is NOT A PICK ITEM that is being handled it is an XML type file.

https://weblogs.asp.net/stevewellens/create-an-xsd-schema-without-knowing-a-damn-thing-about-xsd 

My immediate question is why are you messing around in Pick? It looks like the old saying "when all you have is a hammer then everything looks like a nail"

For heavens sake use the proper tools. Microsoft and 3rd party vendors supply heaps of them.

On Saturday, July 29, 2017 at 10:31:59 PM UTC+10, Bob Dubery wrote:

[Bob Dubery]

Hi Peter,

I might agree with you, and I might have done this very differently if I had written it from scratch. I might do it again differently if there were time for a rewrite. But those conditions don't apply. I have to fix what is already there. Which got me thinking about the variables for controlling a FOR ... NEXT...  loop, something I had always taken for granted and not given much thought to, other than avoiding the FOR X = 1 TO DCOUNT...

[Bob Dubery]

Thanks, Wol.

This is one of those cases where I needed to speak (so to speak (ouch)) to find out what I really thought. You have crystallised the matter for me.

In a case .like 

FOR X = 1 TO Y 

I have often manipulated X, but never Y. It had just never occurred to me to even try to manipulate Y.

[Will Johnson]

It seems clear this code was written by someone who did not have good training in Pick BASIC coding,  or in coding in the first place.

[Peter McMurray]

Hi I am intrigued as to why a method of data transmission is being used as a method of data storage. Glad to hear that you would not write it that way

My response was based on past experience with badly written code. I remember one where a file called PRICES contained one item called PRICE and every attribute referred to a stock item with each value/subvalue being an element of the pricing calculation. There were several thousand stock prices in the ITEM and your initial description seemed to describe that.

The only explanation I have ever been able to come up with is someone trying to get around case sensitivity. A stupid idea, dreamed up at the start of the Unix era by somebody who didn't understand ASCII. In fact those of us longer in the tooth are well aware that ASCII is designed to avoid the issue  - just ignore the lead bit.

[Bob Dubery]

On Monday, 31 July 2017 23:35:36 UTC+2, Peter McMurray wrote:

Hi I am intrigued as to why a method of data transmission is being used as a method of data storage.

It isn't. The program parses an XSD, to build up a tree structure which is then used to generate a set of rules that map the contents of certain files in a UV database to that XSD. Then XML can be generated that is valid. Given that the XSD runs to 1000s of lines, this is worth trying to do programatically.