Re: EOF parameter of a file in Tandem.

[Keith Dick]

krish wrote:
> fileinfo command of a particular file in Tandem yields many info. What is the value below "EOF" indicates?
> Is it a total no of bytes of that particular file?
> Can anyone pls explain.
> If not, whr to get the total no of bytes of that file and wot is that value represent?

The answer to your question depends partly on what you mean by the "total no of bytes of that file".

The EOF value does report how many bytes the operating system has allocated for that file. However, for many types of files, not all of those bytes hold user data, so if you are asking whether EOF denotes the total amount of user data in the file, the answer would be "no".

As far as I know, there is no command or function call that will return the total amount of bytes of user data in a file (except one case). For Enscribe structured files, a FUP INFO filename,STAT can give a pretty close value, but you must look at the number of data records it reports and multiply that by the record size to get the total number of bytes of user data (and this assumes that all the records are the same length). For SQL/MP tables, the same method will give an approximation of the number of user data bytes in the table (there is some overhead for varchar lengths and null indicators included in the record length).

For non-Edit unstructured files, the EOF does report the total amount of user data in the file, since the operating system adds no overhead and the application is completely responsible for managing the data in the unstructured file. An unstructured file is very much like a Unix file in that regard.

An Edit file has some overhead for keeping track of the line numbers, and deleted lines take up space until you compress the Edit file, so the EOF reported by fileinfo is not very close to the amount of user data in the Edit file.

The EOF value does not even tell you how much disk space the file occupies, since disk space is allocated by whole extents, not a byte or block at a time. If you want to know the actual amount of disk space the file takes, you can use FUP INFO filename,DETAIL and look at the extent sizes and number of extents to calculate the total amount of disk space occupied by the file.

I'll bet this is more complicated than you thought it would be.

Now, there probably is some problem you wanted to solve that motivated your question. Maybe if you tell us what the problem is, we can help you solve it.

[Randall]

What Keith says is right on. In Guardian-land, for odd-unstructured files, the EOF represents the number of user-visible bytes in a file. The same can be approximated for C-files (code 180) and for OSS files. EDIT files (code 101), use run-length blank compression, so the EOF is likely shorter than what you will see in an editor or via NFS and SFTP.

However, like other non-PC-class operating systems, there are structured files that are managed by the operating system. This has been the case since the 1960's and vendors include IBM, NCR, Tandem(HP), DEC(HP). The EOF for structured files has always been problematic for programmers who want everything to be simple - a design goal of UNIX/OSS - not to be minimized. EOF is essentially meaningless for any SQL system, so ignore it. Select count(*) from table is useful but has nothing to do with the EOF particularly when varchar(x) is involved. Not even MySQL developers care about EOF, which is probably more meaningless that even in SQL/MP.

Cheers,
Randall

[Keith Dick]

Randall wrote:
> On Friday, October 18, 2013 7:23:26 PM UTC-4, Keith wrote:
>
>>krish wrote:
>>
>>
>>>fileinfo command of a particular file in Tandem yields many info. What is the value below "EOF" indicates?
>>
>>>Is it a total no of bytes of that particular file?
>>
>>>Can anyone pls explain.
>>
>>>If not, whr to get the total no of bytes of that file and wot is that value represent?
>>
>>

Randall wrote:

<snip>

> EDIT files (code 101), use run-length blank compression, so the EOF is likely shorter than what you will see in an editor or via NFS and SFTP.

That statement can be true of an Edit file that was created by writing lines sequentially and has not been touched after that. However, even in that state, since there are at least four bytes of overhead per line (for each line's line number) plus some other overhead, if the data does not contain many sequences of multiple blanks, the EOF easily can show a larger number than the actual number of bytes of user data. After an Edit file has been modified a bit since first being written, and if it has not been compacted again, it is quite possible that the observed EOF will be larger than the total number of bytes of user data. This is because of the way modified blocks are written to the end of the file (so as to enable recovery from a crash), making large chunks of the file be no longer in use. Compacting the file will squeeze out the unused space, returning it to a state where its EOF might, though not always, show fewer bytes than the total number of bytes of user data.

But those details aren't very important. The main point is that a Guardian Edit file is a form of structured file very different from a Unix text file, and as such, the EOF does not tell the number of bytes of user data stored in the file. That description applies only to Guardian Edit files. OSS text files are the same as Unix text files, which are just a stream of bytes with newline characters (\n) marking the ends of the lines. If you consider the newline characters to be part of the user data, then the EOF of an OSS text file does give the number of bytes of user data in the file.

[Randall]

Agreed, Keith. That's why I said "likely". A probability measure, not a statement of absolutes :). I think we're getting a little off of krish's original request - which is essentially that EOF doesn't always mean what it seams.

[krish]

@Keith : I just meant the size of the file when taken out of Tandem.

[Keith Dick]

krish wrote:
> @Keith : I just meant the size of the file when taken out of Tandem.
>

That is why I pointed out that, in many cases, the EOF shown for a file includes more than the actual user records. The EOF reported by fileinfo rarely gives the size that will be taken when you move the data outside the NonStop system. The size of the output depends a lot on how you go about moving the file to another system. If you have a specific file in mind and know the details of how you are going to move it to another system, we might be able to tell you how to get a good estimate of the amount of space the file will take on the other system. Give us more details about the case or cases you are interested in, and we'll see what we can do for you.

[Randall]

I think the key point to take away from this is that computing the off-host file size is unlikely to be practical in the general sense (I have a suspicion that it may actually be strictly impossible, as in theoretically uncomputable), but coming up with an approximation +/- delta should be achievable. Whatever process you design, functionally depending on an exact EOF may be a mistake.

[wbreidbach]

In fact the off-host EOF is depending on the type of file and the transfer itself. If you do a binary transfer, the size will be the about the same as on the NonStop. If you transfer a 101 file in ASCII, the EOF is not really predictable, it depends on the file contents and how much "old" data the file contains. For a structured Enscribe file transferred in ASCII the off-host EOF will usually be smaller than the one on the NonStop.
Do not transfer structured Enscribe files in binary mode to a non-NonStop target, the file on the target will be unusable.

[Randall]

Yes, I agree. I should have added "usable"