Removing FORM FEED characters from file.

[John]

Hi,

We process some reports from open system into the Tandem using Cobol85
program. There is some recent chnage in report from open system.
Becuase of this there is on FORM FEED character (Hex 0C, Dec 20) gets
intorduced. Our existing cobol program is not able to process the
report correctly becuase of this FF character.

We don't want to modify the cobol program. Is there any way on Tandem
where we can remove these Form Feed characters from the file. Using
any Tacl macro or something else?

Please advice.

[wbreidbach]

John,
there is a program called CTOEDIT available, probably this would do
the trick.
It it doesn't, it would be important how the data is transferred to
the Tandem. Do you use FTP? Or has the data been created by OSS?

[Keith Dick]

I'm not sure that CTOEDIT will remove form-feed characters. Maybe it will, but I cannot find any documentation about exactly what CTOEDIT does.

Another possibility might be to use the TRIM option of FUP's COPY command. Depending on the details of your files, you might not be able to use FUP COPY.

As Wolfgang said, to make the best suggestion, we would have to know more facts about how the file is transferred and how the COBOL85 program accesses it.

It might be easiest to copy the report file into a work file on the open system, use some open system command to drop the form feed characters, then transfer the modified file to the NonStop system. Or do that in OSS on the NonStop system. I don't know the Unix utilities well enough to be sure, but I have a feeling that a pretty simple sed command could remove the form feeds from the file.

If nothing else works, it would take only a VERY short C program to copy a file, removing the form feed characters.

[John]

> If nothing else works, it would take only a VERY short C program to copy a file, removing the form feed characters.  - Hide quoted text -
>
> - Show quoted text -

Thank you for your prompt reply.

We are using FTP to load the file on Tandem as a structured file (Type
E, reclen 132). Cobol uses normal file-io to read the every 132 byte.
This report file has some headers on every page (page contains some 80
lines). FIle has FF before starting the new page. Program reads first
line of header to recognise the page start. Problem is that this FF
makes header shifts 1 byte towards right. Because of this program is
not able to read the header properly.

[Doug Miller]

On 12/20/2011 5:43 AM, Keith Dick wrote:
> wbreidbach wrote:
>> On 20 Dez., 08:36, John <johny.pont...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> We process some reports from open system into the Tandem using Cobol85
>>> program. There is some recent chnage in report from open system.
>>> Becuase of this there is on FORM FEED character (Hex 0C, Dec 20) gets
>>> intorduced. Our existing cobol program is not able to process the
>>> report correctly becuase of this FF character.
>>>
>>> We don't want to modify the cobol program. Is there any way on Tandem
>>> where we can remove these Form Feed characters from the file. Using
>>> any Tacl macro or something else?
>>>
>>> Please advice.
>>
>>
>> John,
>> there is a program called CTOEDIT available, probably this would do
>> the trick.
>> It it doesn't, it would be important how the data is transferred to
>> the Tandem. Do you use FTP? Or has the data been created by OSS?
>
> I'm not sure that CTOEDIT will remove form-feed characters. Maybe it
> will, but I cannot find any documentation about exactly what CTOEDIT does.

I'm almost 100% certain that it *won't*.

>
> Another possibility might be to use the TRIM option of FUP's COPY
> command. Depending on the details of your files, you might not be able
> to use FUP COPY.

Concur.

>
> As Wolfgang said, to make the best suggestion, we would have to know
> more facts about how the file is transferred and how the COBOL85 program
> accesses it.
> It might be easiest to copy the report file into a work file on the open
> system, use some open system command to drop the form feed characters,
> then transfer the modified file to the NonStop system. Or do that in OSS
> on the NonStop system. I don't know the Unix utilities well enough to be
> sure, but I have a feeling that a pretty simple sed command could remove
> the form feeds from the file.

That would work, but tr is easier to use than sed, and probably faster
as well. Instead of

$ my_process > desired_destination

do this

$ my_process | tr -d '\014' > desired_destination

[Keith Dick]

Given this description, if I were you, I think the way I would want to fix the problem is change the COBOL85 program to make it recognize the top of page by looking for the form feed character. I imagine making such a change would be rather simple, but you said that modifying the COBOL85 program is not desirable.

Since the form feed character is at the beginning of the record, FUP COPY with TRIM is not applicable.

In another post, Doug gave a tr command that will delete form feed characters. I'd say the simplest solution would be to copy the report file to a temporary file on the source system using such a tr command, then transfer the copied file to the NonStop system in the manner you currently do. No other changes should be needed.

If you cannot add that extra step on the source system, I believe the next best thing to do would be to do a binary ftp to an OSS file on the NonStop system, run the tr command on OSS, then get the data into the entry sequenced file by using ftp to transfer from the OSS file produced by the tr command to the entry sequenced file. One does not usually use ftp to move files around within the same system, but I imagine it is possible to do so (assuming the NonStop system has its FTP service configured), and that probably is the simplest way to convert the OSS file into an entry sequenced file (unless there is some other method I'm not thinking of right now).

If you cannot use ftp to move the file between OSS and Guardian, you could just make the output of the tr command be a Guardian file (/G/vol/subvol/name), use CTOEDIT to convert that Guardian file to an Edit file (assuming the report is small enough to fit into an Edit file), and either have the COBOL85 program read the Edit file (I believe no modifications to the program would be needed to do so) or use FUP COPY or FUP LOAD to copy the Edit file to an entry sequenced file.

In the last paragraph, I did not suggest having the COBOL85 program read the Guardian file that was the output of the tr command because I imagine that COBOL85 would not recognize the record boundaries in that file. If the COBOL85 runtime library does recognize the newline characters as record boundaries, then you could have the COBOL85 program read that file directly. I have not looked into the COBOL manual to see whether it says anything about reading that kind of file.

If you do not have OSS on your NonStop system, I cannot think of any standard utilities that could remove the form feed characters from the entry sequenced file (or even from an unstructured file). Writing a program to copy from one entry sequenced file to another, removing the form feeds is the simplest solution I can think of in that case (or modifying the original COBOL program). Make sure the output file has BUFFERED mode turned on, to make it run much faster.

One other possibility is not to change anything about how you transfer the file. Then write a program that reads its $RECEIVE, reads the entry sequenced file, removes the form feed when the record contains one, sends the record as the REPLY to its $RECEIVE file, and repeats until end of file. Run that program as a named process before starting your current COBOL85 program, and have the COBOL85 program read from that named process instead of from the entry sequenced file. I am pretty sure that no change would be necessary to the current COBOL85 program to let it read from the named process. (I assume you use TACL ASSIGN or DEFINE commands to specify the input file to the COBOL85 program.) The program to run as a named process is not very difficult to write for anyone who has written a server process, but might be a bit hard for someone without that experience. This approach has an advantage over the other methods of not making extra copies of the data file.

[dimandja]

It's so much easier to simply write a simple program (in any language)
to read the file, and replace FF characters to the ones your Cobol
program expects to see.

[wbreidbach]

It would be just one line in the COBOL program:

INSPECT <line> REPLACING HIGH-VALUE BY SPACE. (Or something else)

Meanwhile I know that CTOEDIT will not remove FF, so it is not an
option.

[Keith Dick]

I think form feed is not HIGH-VALUE. Besides, if I understood his explanation correctly, the change to the report generating program added the form feed to the beginning of the line, so everything his program checks for to recognize the top of page is shifted by one character. That means he does not want to replace the form feed with a space (or with anything else), he wants to remove the form feed altogether.

I would check the first character of the line to see whether it is a form feed and if so, move the record to a temp area, then move the second through the end back to the normal record. Or maybe just accept the line containing the form feed as the top of the page, without checking anything else (unless they need to extract some data from the heading). But they don't want to change the COBOL program. Perhaps there is some extensive testing that they must do if they change the program, so he is looking for something else to change (something that they don't have rules about testing, perhaps).

The tr command Doug suggested will remove the form feed, making the report file look like what they have been getting before the change was done. Yesterday I suggested a couple of ways he might be able to arrange things so he could run the file through a tr command, then move it to where his COBOL program could read it. I don't know whether he can use any of those suggestions. I hope he tells us what he finally did.

[wbreidbach]

I just referred to the given FF and that indeed is high-value. Form-
feed is decimal 12, hex 0C. But that would not make a big difference
for for INSPECT statement, you just would have to define from-feed as
a datafield.
The idea of moving the data could be done without a temp field, just
MOVE DATA(1:len) TO DATA(0).

[Keith Dick]

Oh, confusion between FF used as an abbreviation for form feed, and FF as a shorthand for 0xFF. Still, that particular INSPECT statement might have misled someone who thought it referred to transforming a form feed character into something else, so I'm glad I pointed it out.

As for utilizing overlapping moves, no thanks. I've been burned occasionally by stuff like that which is *supposed* to work, but contains subtle bugs. An extra move every 50 lines is not going to hurt the performance of the program. (I know he said the pages were 80 lines long, but I wonder whether that was a typo for 50.)

[wbreidbach]

> As for utilizing overlapping moves, no thanks.  I've been burned occasionally by stuff like that which is *supposed* to work, but contains subtle bugs.  An extra move every 50 lines is not going to hurt the performance of the program.  (I know he said the pages were 80 lines long, but I wonder whether that was a typo for 50.)- Zitierten Text ausblenden -
>
> - Zitierten Text anzeigen -

I agree, this is no performance issue and doing 2 moves instead of one
is not a problem.
But I did not check the CTOEDIT output in detail, it seems that
CTOEDIT replaces the foem-feed (x"0C") by a blank (x"20").
Maybe that would do the trick already.

[Keith Dick]

I don't believe replacing form feed with blank would help John, the original poster.

In his second post, he described the problem further. The altered report program puts a form feed at the beginning of the first line on each page. This moves all of the characters in that line one position to the right. His COBOL program was coded to recognize the start of each page by comparing some literals to specific columns of the line. Those comparisons no longer work because the content of the header lines has been shifted one character. Replacing the form feed with a space would not move the characters back into their original positions in the line. You must remove the form feed character altogether if you insist on not changing the COBOL program to recognize the new positions. (Actually, I would probably remove all the existing tests and just take the presence of the form feed character as the sign that this is the first line of the page rather than continuing to look at the text in the line, unless there is other information to be extracted from the header l
ine.)

[mustlearntandem]

On Dec 21, 6:15 am, wbreidbach <wolfgang.breidb..

>
> I agree, this is no performance issue and doing 2 moves instead of one
> is not a problem.
> But I did not check the CTOEDIT output in detail, it seems that
> CTOEDIT replaces the foem-feed (x"0C") by a blank (x"20").
> Maybe that would do the trick already.

I used CTOEDIT all the time for mixed MX and Enscribe development and
as I recall the output of CTOEDIT is a CODE 101 file. Is thre another
way to use it?

[Keith Dick]

I believe Wolfgang is referring to that same way to use CTOEDIT, to convert a C text file to an Edit file. I'm not sure what leads you to think he meant anything else. He had suggested using CTOEDIT in an earlier post. If you did not remember that earlier post, perhaps his short comment here is not as clear as it might be.

[mustlearntandem]

Well yes, that is exactly my point. My experience is that CTOEDIT
converts a C text file to a Guardian EDIT (code 101) I'm confused how
this will help the original poster. I don't see where John indicated
that his Cobol program uses the utilities to read EDIT files. I
wondered if perhaps there was some CTOEDIT option of which I was not
aware.

[Nik]

There is no problem handling form feed characters from a cobol program
if you wish to program in Cobol your own solution to the problem.

Declare a variable as comp-2, assign it the value of the form feed
character (12 in decimal I assume) and redefine it as a two bytes
character variable.

Something like this.

01 formfeed pic S9(4) comp value 12.

01 Twochars redefines formfeed.
05 filler pic x.
05 FormFeedChar pic x.

[mustlearntandem]

On Dec 21, 2:43 pm, Nik <nik...@gmail.com> wrote:

> There is no problem handling form feed characters from a cobol program
> if you wish to program in Cobol your own solution to the problem.
>

I believe you missed the two key points made by the OP. The problem
was introduced by a change in another application and he has no desire
to change his Cobol program to handle it. He asked for solutions
other than changing the Cobol.

[Keith Dick]

Wolfgang first mentioned CTOEDIT before John gave many details. So it isn't very surprising that one of the early suggestions might not work.

However, COBOL can read an Edit file without using any special utilities. What you wrote seems to indicate that you believe that is not true. If so, I believe you are mistaken on that point.

The only restriction I can think of at the moment is that the FD cannot be described to have alternate keys. There may be other restrictions I am not thinking of, but for most simple COBOL sequential files used for input, I believe you can assign the FD to an entry-sequenced file, a regular unstructured file, or an Edit file without changing the program, and the program will run correctly. (In the case of a regular unstructured file, the program will work correctly only if the logical records in the file are fixed-length.) So if there were a simple way to convert John's file into an Edit file that contained lines that look like the lines his program expects, that would solve John's problem -- he would not have to change the COBOL program. I imagine that is why Wolfgang thought CTOEDIT might help.

I cannot remember whether COBOL85 requires that the length of records read from a file exactly match the length of a file's description when it explicitly or implicitly is described as fixed-length. If that is true, then an additional restriction to be able to freely use an FD with an Edit file would be that the FD describe the file's records as variable-length. I have a vague memory of some situation that requires describing the FD has having variable-length records, but I cannot remember whether it is this case or some other one.

[mustlearntandem]

On Dec 21, 3:28 pm, Keith Dick <kd...@acm.org> wrote:

> I cannot remember whether COBOL85 requires that the length of records read from a file exactly match the length of a file's description when it explicitly or implicitly is described as fixed-length.  If that is true, then an additional restriction to be able to freely use an FD with an Edit file would be that the FD describe the file's records as variable-length.  I have a vague memory of some situation that requires describing the FD has having variable-length records, but I cannot remember whether it is this case or some other one.

==========

Yes I was confused, thank you for pointing out that Cobol can read
EDIT files just as it can other file types.

[dimandja]

On Dec 21, 3:34 pm, mustlearntandem <mustlearntan...@netscape.net>
wrote:

Cobol can read variable length records. The actual length is returned
in a special variable.

[Keith Dick]

That wasn't the issue. I know that can be done.

The issue is whether it is mandatory to tell COBOL that you will read variable length records when the records are not fixed-length and match the length indicated by the FD. The manual seems to contradict itself on that point. I have a vague memory that there is some situation in which you must tell COBOL85 that the records are variable-length to prevent an error, but I cannot remember whether it is when reading variable length records from a disk file or some other case.

[wbreidbach]

> The issue is whether it is mandatory to tell COBOL that you will read variable length records when the records are not fixed-length and match the length indicated by the FD.  The manual seems to contradict itself on that point.  I have a vague memory that there is some situation in which you must tell COBOL85 that the records are variable-length to prevent an error, but I cannot remember whether it is when reading variable length records from a disk file or some other case.- Zitierten Text ausblenden -

>
> - Zitierten Text anzeigen -

COBOL ist able to read variable length records but the filestatus
might be "04" in case of a record shorter than the defined maximum. If
the program only checks filestatus "00" there could be a problem.
If the program really would require a file with alternate keys it
would be a very simple thing to load the sequential file into the
requested one using FUP.
I am not an experienced TACL programmer but I cannot imagine that it
would be a big problem to write a TACL macro to eliminate the FF from
the file.

[Doug Miller]

On 12/22/2011 3:44 AM, wbreidbach wrote:

> I am not an experienced TACL programmer but I cannot imagine that it
> would be a big problem to write a TACL macro to eliminate the FF from
> the file.

No, it's not (although probably bigger than you think), but why reinvent
the wheel?

We already know that the report is created on a system running some
flavor of Unix, which means that it is almost assuredly created using a
command of the general form

report_prog > report_file

Pipe the output of report_prog through the Unix tr utility, and be done
with it in fifteen characters:

report_prog | tr -d "\014" > report_file

[dimandja]

> The issue is whether it is mandatory to tell COBOL that you will read variable length records when the records are not fixed-length and match the length indicated by the FD.

I have ran into that question so many times. Here's my answer.

Cobol can be coded to behave exactly as TAL does. One should really
code Cobol to expect variable length records at any time, on each and
every FD. There's no noticeable performance issue, except you get
efficient coding techniques. Why wrestle with File Status nuances
when you can firmly control the issue in your own code?

Many Cobol "purists" might be horrified by that suggestion, but when
you think about it, why should your Cobol program abend because you
read, say, a shorter record, when you really don't care about the
length of the record?

I instruct my trainees to always think "system resources" when coding
Cobol (or any other language), and to not keep reinventing the wheel.
For example, if TAL can do it, why should a Cobol programmer be
restricted from doing it?

[Keith Dick]

That position doesn't make much sense to me. "... why should your Cobol program abend because you read, say, a shorter record, when you really don't care about the length of the record?" How about because if the input data is supposed to be fixed-length records, getting a shorter record indicates that the input file has been corrupted in some way?

But that's a different topic, not relevant to this discussion. We aren't talking about how programs ought to be written.

The problem at hand is that we have a COBOL85 program already written and in production that is reading data from an entry-sequenced file. We aren't allowed to change that program.

An entry-sequenced file will preserve trailing blanks if they are present in the records written to the file. We don't know whether the existing process of writing the file writes fixed-length or variable-length records.

If we try to substitute an Edit file for the entry-sequenced file, and if the records have trailing blanks, those trailing blanks will be removed from the records, possibly turning a file of fixed-length records into a file of variable-length records.

So if a COBOL85 program must do something to accept a file of variable-length records that it does not have to do to accept a file of fixed-length records, we might not be able to solve the problem using a technique that results in putting the data into an Edit file. It does not matter whether it is possible to write a COBOL85 program that can accept variable length records -- it is more a question of whether every possible COBOL85 program will continue to work properly if its input file changes from fixed-length records to variable-length records (since we know *nothing* about the current program).

I am pretty sure that some COBOL85 programs that were written expecting fixed-length records would not work properly if given variable-length records, since I think that the record area is not cleared before reading the next record. So when reading a record shorter than the previous record, the data at the end of the previous record probably are left in the record area following the data from the shorter record. Only if the program can tell how much good data is present in each record by looking at something like a record code near the front of the record, or if the program never uses data from the variable part of the record, would that not affect the processing.

If the last column printed in a report always ends at the same point in the line and always has a nonblank character at that final character position, it might happen that, except for header, footer, and spacer lines, the report will be mostly fixed-length records when put into an Edit file. So, if the COBOL85 program does not get an error when reading the shorter records, there is some chance that it would function properly. Or if the report's detail lines are not fixed-length but the program does not use any fields in the variable-length part of the lines, it also could function properly if it does not get an error on the READ. So I'd say that there is a fair chance that the program would work properly, even if it was not explicitly written to recognize variable-length records, as long as it does not get an error on the READ due to records shorter than originally expected.

But a fair chance is not certainty, so we would have to be careful suggesting that the current problem be solved in a way that puts the data into an Edit file, even if READs of short records do not cause COBOL errors.

[Keith Dick]

I imagine it is more likely that the solution is not to pipe the output of the program through tr but rather something like this:

report_prog >report_file
tr -d "\014" <report_file >export_file

and transfer export_file to the NonStop system.

I say this because I imagine the report program was changed to add the form feed because they needed the form feed to get the report to print properly, so they would not look kindly on a solution that eliminates the form feed from the original report file. Also, there is some chance that report_prog does not produce the report on its standard output, so it might not even be possible to put tr in a pipeline to the report file.

This is not a big difference from your suggestion. Using tr on the source system probably is the simplest solution, unless John does not have any control over what gets done on the source system.

If the fix has to be done on the NonStop system, using tr is possible if OSS is installed, but I think that would involve making at least one extra copy of the whole file. I suggested how to do that in one of my other posts.

Writing a C, COBOL, or TACL program to run in the Guardian environment to remove the form feed and write a new entry-sequenced file probably is the most direct solution on the NonStop system. However, writing *any* new code (not just changing the existing COBOL85 program) might be undesirable for them, since they might have requirements for extensive testing of any code they write, but get a free pass on using system utilities. I can't think of any combination of Guardian utilities that can be used to remove the form feed from the beginning of some records of the file.

If they can't filter out the form feed on the source system before transferring the file, and can't or won't use tr in OSS and tolerate the extra copying of the file, I think they have to write a program to do it (or modify the existing program)..

[dimandja]

The OP specifically said they don't want to change the existing
programs.

We know Cobol can read variable length records, and that's hardly an
issue here.

We know that the existing Cobol programs can process the input file
(in whatever format it's in) just fine. The only problem is the extra
new characters (form feeds) will trip up those programs. How to deal
with (remove or replace) those characters is the only issue, I
believe, the OP wants solved as efficiently as possible.

Various solutions have been offered. These solutions are applied quite
frequently as a matter of routine. It's no big deal. I think it's up
to the OP to try them, and seek further help as needed.

[Keith Dick]

dimandja wrote:
> The OP specifically said they don't want to change the existing
> programs.
>
> We know Cobol can read variable length records, and that's hardly an
> issue here.

The COBOL manual says that there are some cases in which reading a record that is shorter than the fixed length stated or implied by the FD results in a read failure. Nearby in the manual, it also says that there are some cases in which reading a short record does not fail, but returns 04 in the file status variable. The conditions that distinguish between those two cases are not made clear, at least not in that portion of the manual.

So I believe there certainly is an issue. COBOL85 programs can read variable-length records, but, apparently, there are some ways of describing the file in the FD that require fixed-length records. We do not know whether the existing program can READ variable-length records without getting the COBOL error. It is possible that it cannot.

The rest of what you wrote here is mostly correct. (One of the approaches suggested doing the ftp into the OSS environment, running tr in OSS, then using ftp to transfer the corrected file from OSS to Guardian. I think ftp from OSS to Guardian is something that is not applied frequently.)

I hope John eventually tells us what method he used to solve the problem.

[dimandja]

Keith wrote:
> "The COBOL manual says that there are some cases in which reading a record that is shorter than the fixed length stated or implied by the FD results in a read failure. Nearby in the manual, it also says that there are some cases in which reading a short record does not fail, but returns 04 in the file status variable. The conditions that distinguish between those two cases are not made clear, at least not in that portion of the manual."

Interesting observation, Keith. Those manuals can be stuffy at
times. But this is what I remember from memory:

When you read a record described as fixed length in the FD, and then
you happen to read a shorter record, your program will fail ONLY IF
there is no defined record buffer that can accommodate the shorter
read length. Here is why: you can describe a buffer that is shorter
than that of a "fixed length" record description. In other words, you
can define multiple buffers for the same file, and they don't need to
be of the same length. When you read that shorter record, your program
will be happy because a matching record definition exists and no
characters are lost.

On the other hand, reading the same shorter record, when there is no
buffer defined to receive it, will clobber your program.

[Keith Dick]

Maybe your memory is correct; maybe not. But that doesn't matter at the moment.

The important point is that it seems that there are some ways to write an FD that will only work with fixed-length records. We don't know, but the report program might be writing fixed-length records, or the steps taken to move the file into the entry-sequenced file might pad the records to a fixed length. For that reason, we must be cautious about a solution that results in the report file being an Edit file. It might not cause a problem, but there is a chance it would.

[dimandja]

Keith wrote:
> Maybe your memory is correct; maybe not. But that doesn't matter at the moment.
The important point is that it seems that there are some ways to write
an FD that will only work with fixed-length records. We don't know,
but the report program might be writing fixed-length records, or the
steps taken to move the file into the entry-sequenced file might pad
the records to a fixed length. For that reason, we must be cautious
about a solution that results in the report file being an Edit file.
It might not cause a problem, but there is a chance it would.

You lost me, Keith.

What I posted about Cobol is exactly how Cobol behaves. Who is this
"we don't know" that you keep referring to%?; We cobol programmers
have been dealing with this issue for nearly 4 decades. Before you
junk my answers to your questions with "we don't know"'s, try and
check them out.

Also, I have no idea why you are so worried about this "report file"
format. All we suggested is to change/remove form feed characters. I
don't see any need to worry about the format. The OP doesn't.

[Keith Dick]

I'm referring to the original problem that John posted. He has an existing COBOL program that he does not want to change. This program reads a report file generated on another system and transferred to the NonStop system. That report file is the file that now contains a form feed at the beginning of the first line on each page.

We (any of us looking at this thread and trying to suggest a solution) don't know anything about how John's COBOL program is written. It doesn't matter that there is a way to write a COBOL program that can read variable-length records, or the details of how to do that, because we cannot change the program. If that program already works properly with an input file that contains variable-length records, then an attempted solution that ends up with variable-length records in the file to be read by that COBOL program could be okay. But we don't know whether that existing COBOL program will work properly with variable-length input records, so we should not depend on an approach that requires it to do so.