Append stem contents to a file in Regina

[Arthur T.]

I use Regina REXX. For almost all file I/O, I use REGSTEMREAD &
REGSTEMWRITE. REGSTEMWRITE appears to always start writing at the
beginning of the file. I'd like to be able to append the data, i.e.
add it after the last line of the existing file.

I could read the file, play games with the stems, and then
rewrite the whole thing. This is inelegant, especially if the file
is large.

I could loop with LINEOUT. This is my probable fall-back
technique.

I tried opening the file with the STREAM command and APPEND
before the REGSTEMWRITE. No luck.

I couldn't find anything in the last few years of this
newsgroup, and my web search didn't turn up anything. The
information isn't there or my search-fu isn't strong today.

Can I use REGSTEMWRITE to append? If not, is looping through
LINEOUT the most reasonable alternative?

--
Arthur T. - ar23hur "at" pobox "dot" com

[Thom]

I searched the web for any examples of REGSTEMWRITE or any other APPEND and came up empty. Documentation for REGSTEMSWRITE is negligable. Using Regina Stream IO I wrote this to just play around. It works for me. Not sure what it would do with large files or a bunch of large files. There is extra stuff in here as it is a hack I used to strip numbers from a mainframe download.

/* TEST5 */
Trace e
say time('r')
/* load Rexx Utilities by Patrick TJ McPhee */
call rxfuncadd 'sysloadfuncs', 'rexxutil', 'sysloadfuncs'
call 'sysloadfuncs'

tmpstr.=''
mystem.=''
i=0
r=0
ptr=0
thisvar='NULL'
tmpvar='NULL'
myrc='MYRC'
suffix=''

path='C:\Users\USERNAME\My Rexxes\tmpfldr\'
call rxfuncadd 'SysFileTree', 'RexxUtil', 'SysFileTree'
myrc=SysFileTree(path, 'mystem.')

do i=1 to mystem.0
r=lastpos('\', mystem.i)
suffix=strip(substr(mystem.i,r+1))
infile=path||suffix
outfile=path||suffix
ptr=0
tmpstr.=''
Call ReadWrite
end
say time('e')
Exit

ReadWrite:
IF STREAM(infile, 'C', 'OPEN READ') == 'READY:' THEN
Do WHILE LINES(infile) > 0
thisvar=linein(infile)
ptr=ptr+1
tmpstr.ptr=strip(left(thisvar,72))
end
tmpstr.0=ptr
CALL STREAM infile, 'C', 'CLOSE'

IF STREAM(outfile, 'C', 'OPEN WRITE APPEND') == 'READY:' THEN
Do ptr =1 to tmpstr.0
tmpvar=tmpstr.ptr
myrc = lineout(outfile,tmpvar)
end
CALL STREAM outfile, 'C', 'CLOSE'
Return

[Arthur T.]

In
Message-ID:<fa5615b0-a19b-4c4c...@googlegroups.com>,

Thom <ts2...@gmail.com> wrote:

>On Sunday, September 10, 2017 at 6:44:48 PM UTC-5, Arthur T. wrote:
>> I use Regina REXX. For almost all file I/O, I use REGSTEMREAD &
>> REGSTEMWRITE. REGSTEMWRITE appears to always start writing at the
>> beginning of the file. I'd like to be able to append the data, i.e.
>> add it after the last line of the existing file.

<snip>

>> I couldn't find anything in the last few years of this
>> newsgroup, and my web search didn't turn up anything. The
>> information isn't there or my search-fu isn't strong today.

>I searched the web for any examples of REGSTEMWRITE or any other APPEND and came up empty. Documentation for REGSTEMSWRITE is negligable.

So, if there's anything out there, it's not just me who can't
find it. Thanks for trying. This is one of those times I was hoping
to be proved wrong.

>Using Regina Stream IO I wrote this to just play around.

And, thanks for the example. I've found that LINEOUT
automatically does append unless told otherwise, so I've added this
code to my program, in place of the REGSTEMWRITE:

if reply = "A" then /* user asked for APPEND */
do
do i = 1 for outline.0
x = lineout(filename, outline.i)
end
exit 0
end

I could do LINEOUT in stead of REGSTEMWRITE in both cases, but I
didn't want to change working code more than necessary. Plus,
several releases back, I found that LINEOUT was buggy. It's probably
fixed by now, but I still avoid it by habit. And, since I also came
from a mainframe background, I find REGSTEM... more familiarly
comfortable as a replacement for EXECIO.

[LesK]

No matter how it's done, on a pc the whole file must be read (by
something) to find the end of the last record in the bit stream.

--

Les (Change Arabic to Roman to email me)

[Arthur T.]

In Message-ID:<opa2l4$fmo$1...@gioia.aioe.org>,

LesK <5mr...@tampabay.rr.com> wrote:

>On 9/12/2017 7:31 PM, Arthur T. wrote:
>> In
>> Message-ID:<fa5615b0-a19b-4c4c...@googlegroups.com>,
>> Thom <ts2...@gmail.com> wrote:
>>
>>> On Sunday, September 10, 2017 at 6:44:48 PM UTC-5, Arthur T. wrote:
>>>> I use Regina REXX. For almost all file I/O, I use REGSTEMREAD &
>>>> REGSTEMWRITE. REGSTEMWRITE appears to always start writing at the
>>>> beginning of the file. I'd like to be able to append the data, i.e.
>>>> add it after the last line of the existing file.
>> <snip>

<snip>

>>
>No matter how it's done, on a pc the whole file must be read (by
>something) to find the end of the last record in the bit stream.

That appears not to be the case. I used LINEOUT to append a
line to a file that was almost 700 MiB. It took maybe one second. I
verified that the file had been updated via a hexedit program.

I made sure to use a file that I had not read since my last
boot, so there wouldn't be cached information about it. After doing
that append (and hexedit), I ran a REXX program to read the file; it
took over 20 seconds.

[LesK]

All the experts that I know assure me that my statement is correct. You
should research 'pc file system' to get a deeper understanding.

Btw: Visit http://rexx.org
to get a pointer to Regina support.

[Arthur T.]

In Message-ID:<opcqc8$1c3e$1...@gioia.aioe.org>,

"When the book disagrees with the bird, believe the bird." I'm
not saying you're wrong, but if you're right, I don't understand how
I can append in much less time than it takes to read the file.

>Btw: Visit http://rexx.org
>to get a pointer to Regina support.

I have opened bug reports. I think it's just one person
supporting Regina, and I don't want to bother him with questions.

[LesK]

The performance difference comes from comparing a c++ program to the
interpreter program. That's like comparing a Ferrari to a Volkswagen at
Le Mans!

It's very kind of you to worry about being a burden on Mark's time. He
certainly has a lot of responsibilities and some especially hot irons in
the fire right now. But you've exhausted this avenue that doesn't seem
to draw a lot of attention anymore. You might want to visit

http://rexxla.org

and click on 'Rexx Forum' at the top of the page and find an appropriate
place to pose your question. You can even post to the RexxLA List, which
is a shadow of the Mailing List on our server. However, only those who
follow the Nabble shadow will see your post if you're not a member of
RexxLA.

[Steven Levine]

On Thu, 14 Sep 2017 03:45:12 UTC, Arthur T. <art...@munged.invalid>
wrote:

Hi,

> >> That appears not to be the case. I used LINEOUT to append a
> >> line to a file that was almost 700 MiB. It took maybe one second. I
> >> verified that the file had been updated via a hexedit program.

The likely reason is that the entire file was not read. The vast
majority of file systems know the file size and the support seek
operations. To append to a binary file, one opens the file, seeks to
the end of file and writes out the new data. No file data is read.
To append to a text file, it's a bit more complicated depending how
robust you want the code to be. A robust solution will seek and read
in the last one or two bytes of the file, depending on the platform
EOL marker rules. If the EOL marker does not exist, it is written
followed by the new data. A less robust solution will just seek to
the end of the file and write out the new data, assuming that the EOL
marker exists for what used to be the last line in the file.

Steven

--
---------------------------------------------------------------------
Steven Levine <ste...@earthlink.bogus.net>
DIY/Warp/BlueLion etc. www.scoug.com www.arcanoae.com www.warpcave.com
---------------------------------------------------------------------

[LesK]

On 9/14/2017 12:18 PM, Steven Levine wrote:
> On Thu, 14 Sep 2017 03:45:12 UTC, Arthur T. <art...@munged.invalid>
> wrote:
>
> Hi,
>
>>>> That appears not to be the case. I used LINEOUT to append a
>>>> line to a file that was almost 700 MiB. It took maybe one second. I
>>>> verified that the file had been updated via a hexedit program.
>
> The likely reason is that the entire file was not read. The vast
> majority of file systems know the file size and the support seek
> operations. To append to a binary file, one opens the file, seeks to
> the end of file and writes out the new data. No file data is read.
> To append to a text file, it's a bit more complicated depending how
> robust you want the code to be. A robust solution will seek and read
> in the last one or two bytes of the file, depending on the platform
> EOL marker rules. If the EOL marker does not exist, it is written
> followed by the new data. A less robust solution will just seek to
> the end of the file and write out the new data, assuming that the EOL
> marker exists for what used to be the last line in the file.
>
> Steven
>

That *might* be true for a c++ program but on a pc all user files are
just binary bit streams, so I don't understand the difference you're
talking about. Certainly not true for a Rexx interpreter, specifically
OORexx and Regina. Maybe BRexx is different, but I'd bet your favorite
cold beverage that it's not.

[Rick McGuire]

You would lose the bet. They are binary streams, but because they are persistent streams, it is possible to move the read or write pointers of the open stream without needing to read the file. Additionally, when a file is opened for writing, the write pointer is automatically positioned at the end position for you. Since I wrote the ooRexx code, I can certainly attest that it works that way.

Rick

[roy]

On 9/14/2017 9:18 AM, Steven Levine wrote:
> On Thu, 14 Sep 2017 03:45:12 UTC, Arthur T. <art...@munged.invalid>
> wrote:
>
> Hi,
>
>>>> That appears not to be the case. I used LINEOUT to append a
>>>> line to a file that was almost 700 MiB. It took maybe one second. I
>>>> verified that the file had been updated via a hexedit program.
>
> The likely reason is that the entire file was not read. The vast
> majority of file systems know the file size and the support seek
> operations. To append to a binary file, one opens the file, seeks to
> the end of file and writes out the new data. No file data is read.
> To append to a text file, it's a bit more complicated depending how
> robust you want the code to be. A robust solution will seek and read
> in the last one or two bytes of the file, depending on the platform
> EOL marker rules. If the EOL marker does not exist, it is written
> followed by the new data. A less robust solution will just seek to
> the end of the file and write out the new data, assuming that the EOL
> marker exists for what used to be the last line in the file.
>
> Steven
>

Last time I had to read forward to find the end so I could append was a
10½-inch reel on a tape drive :-)

[LesK]

I can't find anything about "persistent streams", Rick. Can you provide
a reference that will clarify my understanding of pc file systems, since
all I've learned so far comes from posts by experts like you. Thanks!

[Arthur T.]

In Message-ID:<opd49b$1odb$1...@gioia.aioe.org>,

LesK <5mr...@tampabay.rr.com> wrote:

>>>>> No matter how it's done, on a pc the whole file must be read (by
>>>>> something) to find the end of the last record in the bit stream.
>>>>
>>>> That appears not to be the case. I used LINEOUT to append a
>>>> line to a file that was almost 700 MiB. It took maybe one second. I
>>>> verified that the file had been updated via a hexedit program.
>>>>
>>>> I made sure to use a file that I had not read since my last
>>>> boot, so there wouldn't be cached information about it. After doing
>>>> that append (and hexedit), I ran a REXX program to read the file; it
>>>> took over 20 seconds.
>>>>
>>> All the experts that I know assure me that my statement is correct. You
>>> should research 'pc file system' to get a deeper understanding.
>>
>> "When the book disagrees with the bird, believe the bird." I'm
>> not saying you're wrong, but if you're right, I don't understand how
>> I can append in much less time than it takes to read the file.

<snip>

>The performance difference comes from comparing a c++ program to the
>interpreter program. That's like comparing a Ferrari to a Volkswagen at
>Le Mans!

I found a C program I wrote long ago which copies a file. I
took out all of the write code so it just reads a file and compiled
it. It read that file in about 20 seconds. As I said, the append
took about 1 second.

You're right that usually a REXX program will be much slower
than a compiled or assembled program. But in this case, most of the
time is taken in the actual disk I/O which can't be sped up
programmatically.

[Rick McGuire]

A persistent stream is just a file as opposed to a non-persistent one such as an internet socket or a serial port. The file system apis allow easy movement along the stream (both forward and backward) without needing to read the file.

[LesK]

How can that be possible if end-of-record is encoded in the bit stream?

[Gil Barmwater]

The OS file system keeps track of where on the "disk" the file starts
AND its length so is not dependent on any characters in the stream to
denote EOF.

--
Gil Barmwater

[Rick McGuire]

In normal practice, it is only necessary to look at the line separators when reading an individual line. The reading starts from the current read location and only needs to read as far as the line separator.

[Steven Levine]

On Fri, 15 Sep 2017 13:37:53 UTC, LesK <5mr...@tampabay.rr.com> wrote:

Hi Les,

> How can that be possible if end-of-record is encoded in the bit stream?

As the others have noted, this is easily handled. IAC, this thread
started asking about appending to a file.

There was a time when file systems only knew the number of sectors in
a file and we needed to detect end of file markers, such as Ctrl-Z.
Those days are long gone. Modern file systems know how many bytes
there are in a file.

[LesK]

On 9/15/2017 9:43 PM, Steven Levine wrote:
> On Fri, 15 Sep 2017 13:37:53 UTC, LesK <5mr...@tampabay.rr.com> wrote:
>
> Hi Les,
>
>> How can that be possible if end-of-record is encoded in the bit stream?
>
> As the others have noted, this is easily handled. IAC, this thread
> started asking about appending to a file.
>
> There was a time when file systems only knew the number of sectors in
> a file and we needed to detect end of file markers, such as Ctrl-Z.
> Those days are long gone. Modern file systems know how many bytes
> there are in a file.
>
> Steven
>
>
>
>
>

Gil and you have both pointed that out and I *should* have realized that
on my own.

My apologies for diverting this thread.