ZIP "-V" v. UNIX, et al.: Problem, possible solutions.
s...@antinode.org
produced on a VMS system using ZIP "-V" (to preserve the VMS file
attributes). The unpacking was done on a Tru64 UNIX system, but I'd bet
the results would be similar on any UNIX system.
All the files of interest to a non-VMS user were Record format:
Stream_LF (that is, maximally UNIX-friendly), and all were corrupted in
a particular way when unpacked on the UNIX system, specifically, each
file was padded with NULs until its size was a multiple of 16384.
(Which extra bytes sure startled the Compaq C compiler, by the way.)
After a bit of research, I learned that the Zip archive included
defective values for the uncompressed sizes of the files therein, and
that this problem arose only when "-V" was used. (Thus, UnZip on the
UNIX system was doing its best to try to follow bad instructions when it
was unpacking.)
The problem appears to be in the [.VMS]VMS_PK.C code, which uses
(fairly raw) block I/O to read the input file(s), specifically, that it
generally has only a vague idea of how many bytes it has read. I have
some modifications which appear to work correctly, within limits
discussed below. I also stole some of my code from Wget to enable the
DECC$ARGV_PARSE_STYLE feature (where possible), obviating the quotation
marks around "-V" (among other things) if the command-line parsing style
is extended (on modern, non-VAX systems).
The only problem I've seen so far (after limited testing) is some
complaints from BACKUP /COMPARE when a byte or two past the end of the
last record fails to match. A minor tune-up to UnZip helps, but does
not fix every case. (UnZip on VMS also tries to do (fairly raw) fast
I/O, so it always writes an even number of bytes to a file. Up to now,
when it incremented an odd byte count to make it even, it took the
existing garbage in the buffer and wrote that as the extra byte.
Clearing that extra byte fixes a lot of the mis-compares, but I have
some Alpha and VAX (but not IA64?) object files with an extra "FFFF"
after the end of the last record, and I don't see much to gain from
trying to reproduce any data from past the EOF.) DIFFERENCES is always
happy, but BACKUP /COMPARE is fussier.
So, my questions are:
Has anyone else noticed this problem?
Is anyone interested in pushing some changes back into the main
[Un]Zip code stream?
Does anyone see any potential problems?
Is anyone working on ODS5 support for the Info-Zip programs?
Potentially pertinent details (I quote):
This is Zip 2.3 (November 29th 1999), by Info-ZIP.
[^^^^--- And I'd bet that no one's touched
the I/O code in VMS_PK.C since about 1993.]
UnZip 5.51 of 22 May 2004, by Info-ZIP.
------------------------------------------------------------------------
Steven M. Schweda (+1) 651-699-9818
382 South Warwick Street sms@antinode-org
Saint Paul MN 55105-2547
Craig A. Berry
> Is anyone working on ODS5 support for the Info-Zip programs?
The GNV version of zip had some modifications for case preservation
that didn't work. I submitted a patch here:
http://sourceforge.net/tracker/?group_id=2506&atid=302506
I've seen no evidence that the GNV maintainers pay any attention to
patches submitted there, nor that there is any two-way communication
between the GNV project and Info-ZIP.
Tim Shoppa
> I recently had occasion to try to unpack a ZIP archive which was
> produced on a VMS system using ZIP "-V" (to preserve the VMS file
> attributes). The unpacking was done on a Tru64 UNIX system, but I'd bet
> the results would be similar on any UNIX system.
>
> All the files of interest to a non-VMS user were Record format:
> Stream_LF (that is, maximally UNIX-friendly), and all were corrupted in
> a particular way when unpacked on the UNIX system, specifically, each
> file was padded with NULs until its size was a multiple of 16384.
> (Which extra bytes sure startled the Compaq C compiler, by the way.)
>
> After a bit of research, I learned that the Zip archive included
> defective values for the uncompressed sizes of the files therein, and
> that this problem arose only when "-V" was used. (Thus, UnZip on the
> UNIX system was doing its best to try to follow bad instructions when it
> was unpacking.)
Well, info-zip works very differently in "-V" than otherwise, and this
is by design:
1. When "-V" is used, it doesn't try hard at all to determine Unix-style
line length. It's only goal is to get the blocks all there and get
the attributes right. The intention of "-V" is that you'll unpack
the file on a VMS machine and have it just as it originally was. But
if you unpack it on a non-VMS machine you'll get raw blocks (which
you seem to describe as padding, actually I'm not so sure that they'll
necessarily be NUL's, you may have random bytes instead in general.)
2. When "-V" isn't used it tries harder to determine file length in
a Unix-type way and will (if necessary, and sometimes if not necessary!)
convert the file in such a way that VMS-ism's are removed (e.g. record
lengths etc. if they're there.) This way when you unzip the file on
a non-VMS machine you usually get what you want and don't get what you
don't want.
You seem to be trying to use it in a hybrid way, and I see what
you're doing, but I suspect that if you used it without the "-V" you'd
get what you wanted right away.
Tim.
s...@antinode.org
> Well, info-zip works very differently in "-V" than otherwise, and this
> is by design:
>
> 1. When "-V" is used, it doesn't try hard at all to determine Unix-style
> line length. It's only goal is to get the blocks all there and get
> the attributes right. The intention of "-V" is that you'll unpack
> the file on a VMS machine and have it just as it originally was. But
> if you unpack it on a non-VMS machine you'll get raw blocks (which
> you seem to describe as padding, actually I'm not so sure that they'll
> necessarily be NUL's, you may have random bytes instead in general.)
Apparently, you haven't actually looked at the code, or run the
experiment. Please read carefully. Currently, ZIP "-V" will take a
four-byte file, with 1 VMS disk block used, and save info like this in
the Zip archive (as displayed by 'UNZIP "-Z" -v'):
compressed size: 38 bytes
uncompressed size: 16384 bytes
The suggestion that this error is "by design" is funny, but not
enlightening.
> 2. When "-V" isn't used it tries harder to determine file length in
> a Unix-type way and will (if necessary, and sometimes if not necessary!)
> convert the file in such a way that VMS-ism's are removed (e.g. record
> lengths etc. if they're there.) This way when you unzip the file on
> a non-VMS machine you usually get what you want and don't get what you
> don't want.
Please read carefully. The files in question are Record format:
Stream_LF, so what you see in raw form on VMS is exactly what you want
to see on the UNIX system. My choice of Stream_LF was not an accident.
Getting "the blocks all there" does not require getting 31 blocks
past the end of the file, attributes or no attributes. In a four-byte
file (one line, "ABC\n"), there is one raw block, not 32. Even DUMP
(not /RECORD) admits this. The extra 31 blocks on the UNIX system are
padding, not "raw blocks". Whether the padding comprises only NULs
depends on the implementation of things on the UNIX system, but it's the
result of the defective "uncompressed length" stored in the archive.
I assumed that the _intention_ of "-V" was to get the file attributes
right if the archive was unpacked on a VMS system, not necessarily to
produce corrupt files on non-VMS systems. Clearly, the current
_implementation_ offers file corruption as a bonus for a non-VMS user.
> You seem to be trying to use it in a hybrid way, and I see what
> you're doing, but I suspect that if you used it without the "-V" you'd
> get what you wanted right away.
Yes, because not using "-V" (or with SET PROCESS /PARSE = EXTENDED
and my changes, just plain -V), BYPASSES THE _DEFECTIVE_ CODE.
But thanks for playing.
Tim Shoppa
> right if the archive was unpacked on a VMS system, not necessarily to
> produce corrupt files on non-VMS systems. Clearly, the current
> _implementation_ offers file corruption as a bonus for a non-VMS user.
"-V" zip archives unpack exactly on real VMS systems. (Or, well, they
did, until you made your modification.) But you bypass
all conversions to/from "standard" Unix formats when you unpack on non-VMS
systems. You get extra stuff (not necessarily at the end of the file!)
when you unpack on a non-VMS machine. I hear your observation that all
that stuff comes out, and agree with you that it's there, but again,
that's "by design": there's no guarantee that a "-V" zipped file will
make sense on a foreign system.
Good to see that you're testing your modifications, BTW.
Tim.
s...@antinode.org
> "-V" zip archives unpack exactly on real VMS systems. (Or, well, they
> did, until you made your modification.) But you bypass
> all conversions to/from "standard" Unix formats when you unpack on non-VMS
> systems. You get extra stuff (not necessarily at the end of the file!)
> when you unpack on a non-VMS machine. I hear your observation that all
> that stuff comes out, and agree with you that it's there, but again,
> that's "by design": there's no guarantee that a "-V" zipped file will
> make sense on a foreign system.
If by "extra stuff" you mean the "extra fields" in the Zip archive
which hold OS-specific data such as VMS file attributes, then no, you
_don't_ get _that_ extra stuff when you unpack on a non-VMS machine.
UnZip discards those "extra fields". What you _do_ get is useless,
erroneous NUL padding. (Or do you actually believe that the almost 16K
of NULs is really a clever encoding of the VMS file attributes? Or
perhaps a file exactly 16K bytes long has exactly zero "extra stuff"?
Let's get serious.)
It may be that many people believe that this file corruption is the
result of the VMS attribute storage. This explanation of this problem
may even have become a computer legend in the lore of the misinformed
(probably some time around 1993), but it is NOT true.
While it may be true in general that a VMS file stored as a pile of
bytes (as with -V) does not appear in a useful form when unpacked on a
non-VMS system, it is simply not true that this is a problem for a
Stream_LF text file. For example:
ALP $ dump /byte TEST.SLF
Dump of file SYS$SYSDEVICE:[UTILITY.SOURCE.WGET]TEST.SLF;1 on 20-SEP-2004 16:43:30.93
File ID (144604,37,0) End of file block 1 / Allocated 35
Virtual block number 1 (00000001), 512 (0200) bytes
00 00 00 00 00 00 00 00 00 00 00 00 0A 43 42 41 ABC............. 000000
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 000010
[...]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 0001F0
urt# od -atx1 test.slf
0000000 A B C nl
41 42 43 0a
0000004
Same pile of bytes. Moving this pile of bytes from VMS to UNIX
without Zip-induced corruption gives the desired result. Moving this
pile of bytes and then padding them out to 16K with NULs gives an
incorrect result.
Please, before telling me again that ZIP -V _should_ work this way
because it _does_ work this way, do us all a favor and look at the code.
IT'S BROKEN. The fact that UnZip on VMS can find an accurate file size
in the VMS-specific attribute storage, and hence ignore the DEFECTIVE
file size in the general attribute storage does not mean that ZIP -V is
producing a proper archive.
> Good to see that you're testing your modifications, BTW.
Let me know if you'd like to help.
David J Dachtera
Steve - feel free to move on to the next message...
For those still reading:
I was able to duplicate Steve's observations. V2.3 of ZIP does indeed
appear to be "broken". Saw the results he observed even when using UNZIP
V5.51 on VMS(Alpha).
I still have a V2.2 ZIP distro. on the machine at work. I'll look into
that a bit deeper later on.
That said, in my DECUS presentation from Fall-99 (San Diego), I put
forth the concept that the target platform should be borne in mind when
creating the archive. Preserving RMS attributes is really only of value
when it's needed for OpenVMS. VMS .ZIP archives (using "-V" or /VMS)
frequently produce marginally usable results when UNZIPPed on non-VMS
platforms unless the content is either exclusively binary (Fixed-512) or
limited to binary and Stream-LF files.
My $0.02...
D.J.D.
s...@antinode.org
> I still have a V2.2 ZIP distro. on the machine at work. I'll look into
> that a bit deeper later on.
Considering the apparent age of the offending code, I'd be surprised
if you could find a version (not on punched paper tape) which was
significantly different.
> [...] Preserving RMS attributes is really only of value
> when it's needed for OpenVMS. VMS .ZIP archives (using "-V" or /VMS)
> frequently produce marginally usable results when UNZIPPed on non-VMS
> platforms unless the content is either exclusively binary (Fixed-512) or
> limited to binary and Stream-LF files.
The files of interest here were precisely ones for which this should
have worked. (This horse _is_ bloated-rotting-dead now, right?) The
idea was not to preserve the VMS/RMS file attributes for non-VMS
systems. I was just trying to save some disk space by offering one
(compatible) archive instead of two: one for VMS systems, one for other
systems. (Which I may yet do.)
For those with copious free time (and no songs to compose),
replacement source files with my proposed changes may be available at:
http://www.antinode.org/ftp/info-zip/
ftp://ftp.antinode.org/info-zip/
John Laird
> I assumed that the _intention_ of "-V" was to get the file attributes
>right if the archive was unpacked on a VMS system, not necessarily to
>produce corrupt files on non-VMS systems. Clearly, the current
>_implementation_ offers file corruption as a bonus for a non-VMS user.
Who shouldn't be using -V then. I suggest you also test your changes on
indexed files, especially ones not closed properly. It is vital to include
all allocated blocks and not pay attention to EOF.
--
Give me patience! RIGHT NOW!
Mail john rather than nospam...
Soterro
> That said, in my DECUS presentation from Fall-99 (San Diego), I put
> forth the concept that the target platform should be borne in mind when
> creating the archive. Preserving RMS attributes is really only of value
> when it's needed for OpenVMS. VMS .ZIP archives (using "-V" or /VMS)
True, but this avoids the real issue. As an user and not a developer, I
rather am of the original poster's opinion. I hate when developers come
to me saying 'do it because we know what is good for you' and then stop
listening to any feedback (this reminds me of the whole Mozilla
multi-user profiles crap).
In this case, when I create an archive with "-V" I'd certainly expect to
have a 'normal' archive (normal like in multiplatform, that's the ZIP
goal isn't it), and when and only when I unpack it on VMS to have the
extra goodies packed by "-V" (like attributes). 'Platform independent'
that is. I'm really looking forward for that, as at the moment I had to
write Perl utilities to shape files between VMS Unix and Windows... awful.
I agree it's more work to develop, but that's what are developers for :D
S
Tim Shoppa
> What you _do_ get is useless,
> erroneous NUL padding. (Or do you actually believe that the almost 16K
> of NULs is really a clever encoding of the VMS file attributes? Or
> perhaps a file exactly 16K bytes long has exactly zero "extra stuff"?
> Let's get serious.)
Whoa! Calm down there. I'm not saying that the extra stuff isn't there.
I agree fully that it is there. And if you do any VMS file format that
has RMS record lengths all those are there too.
But, please, stop, take a step back, and look at the info-zip documentation:
-V [VMS] Save VMS file attributes. zip archives cre-
ated with this option will generally not be usable
on other systems.
They (and I) make no guarantee that Info-zip's "-V" option will
produce what any particular person (including you) wants when unzipped on a
non-VMS system. In general there's a bunch of RMS junk between records
and potentially random junk at the end. The case you're all hyped up
about (stream LF) is indeed the best case but still
not perfect.
You're perfectly free to hack up the innards of Info-zip however you want
to do whatever you think it should do. But we still aren't into the
realm of "problem" much less "solution" because it's working exactly as
documented.
> Please, before telling me again that ZIP -V _should_ work this way
> because it _does_ work this way, do us all a favor and look at the code.
> IT'S BROKEN.
Look at the documentation. IT WORKS.
Tim.
s...@antinode.org
From: John Laird <nos...@laird-towers.org.uk>
> On Mon, 20 Sep 2004 09:48:39 -0500 (CDT), s...@antinode.org wrote:
>
> > I assumed that the _intention_ of "-V" was to get the file attributes
> >right if the archive was unpacked on a VMS system, not necessarily to
> >produce corrupt files on non-VMS systems. Clearly, the current
> >_implementation_ offers file corruption as a bonus for a non-VMS user.
> Who shouldn't be using -V then.
I'm curious. Is that your response to _all_ programs with features
that misbehave?
> I suggest you also test your changes on
> indexed files, especially ones not closed properly. It is vital to include
> all allocated blocks and not pay attention to EOF.
First, if I had to pick a defect, I'd choose to have properly
structured files come out right and badly structured files come out
wrong, rather than the other way around.
Second, what makes you think it does this now? (Not that I would
rule it out.) How would I test the results? For example, does BACKUP
/COMPARE check "all allocated blocks"?
> Subj: Re: ZIP "-V" v. UNIX, et al.: Problem, possible solutions.
Third, just as a reminder, I did say ---------^^^^^^^^.
John Laird
>Subj: Re: ZIP "-V" v. UNIX, et al.: Problem, possible solutions.
>
>From: John Laird <nos...@laird-towers.org.uk>
>
>> On Mon, 20 Sep 2004 09:48:39 -0500 (CDT), s...@antinode.org wrote:
>>
>> > I assumed that the _intention_ of "-V" was to get the file attributes
>> >right if the archive was unpacked on a VMS system, not necessarily to
>> >produce corrupt files on non-VMS systems. Clearly, the current
>> >_implementation_ offers file corruption as a bonus for a non-VMS user.
>
>> Who shouldn't be using -V then.
>
> I'm curious. Is that your response to _all_ programs with features
>that misbehave?
I refer you to the description of -V. There is no misbehaviour. It is not
intended to provide a zip file that may or may not work elsewhere. It is
intented to provide a zip file that will work on VMS systems. For this to
happen, it must include all file metadata as well as what one might term
"real" data. The simplest solution is to read all blocks up to and include
the EOF block for sequential files and ALQ block for others. However, once
this is stored in the archive, how is a non-VMS UNZIP supposed to know the
difference - all it will see is a stream of bytes.
>> I suggest you also test your changes on
>> indexed files, especially ones not closed properly. It is vital to include
>> all allocated blocks and not pay attention to EOF.
>
> First, if I had to pick a defect, I'd choose to have properly
>structured files come out right and badly structured files come out
>wrong, rather than the other way around.
Where does the notion of structure "goodness" come from ? All VMS file
structures are perfectly good in VMS terms.
> Second, what makes you think it does this now? (Not that I would
>rule it out.) How would I test the results? For example, does BACKUP
>/COMPARE check "all allocated blocks"?
I don't need to think, I *know* it currently works. You can see the code in
one of the source files you referred to. I, no doubt with hundreds or
thousands of others, have used ZIP/UNZIP to successfully store and restore
VMS files of all structures. You asked about BACKUP - yes this does ring a
bit of a bell in that I have seen /COMPARE complaining about data in the
last block but suspect this was more likely a result of having ftp'd simple
sequential files around. Since FTP is normally stream-oriented, it does
only transfer actual data and so the "random" bytes in the last block will
generally be different between source and destination files. VMS' own
utilities like COPY and BACKUP will usually work in block mode on
block-structured devices and thus copy the random data faithfully.
BACKUP/COMPARE can report differences that DIFFERENCE will not because the
two utilities are (normally) block and record-oriented by design.
To be honest, I would say that if you are not sure how to test the veracity
of your changes on all file types, then you should not be making those
changes. By all means do this for your own use but please don't take steps
to meld these into the publicly available versions until you are totally
satisfied there are no unwanted side-effects.
>> Subj: Re: ZIP "-V" v. UNIX, et al.: Problem, possible solutions.
> Third, just as a reminder, I did say ---------^^^^^^^^.
Fair enough. I did look at your source files but could not identify exact
changes very well without access to the unmodified versions, so will not
comment further.
--
Pardon our dust... nonsense under construction.
s...@antinode.org
> s...@antinode.org wrote in message news:<0409201...@antinode.org>...
> > What you _do_ get is useless,
> > erroneous NUL padding. (Or do you actually believe that the almost 16K
> > of NULs is really a clever encoding of the VMS file attributes? Or
> > perhaps a file exactly 16K bytes long has exactly zero "extra stuff"?
> > Let's get serious.)
>
> Whoa! Calm down there. I'm not saying that the extra stuff isn't there.
> I agree fully that it is there. And if you do any VMS file format that
> has RMS record lengths all those are there too.
I'm still not clear on which "extra stuff" you're talking about.
And, as I've tried repeatedly to make clear, I don't expect useful
results in this situation for a file with embedded record lengths. And,
of course, my files don't have them.
> But, please, stop, take a step back, and look at the info-zip documentation:
>
> -V [VMS] Save VMS file attributes. zip archives cre-
> ated with this option will generally not be usable
> on other systems.
>
> They (and I) make no guarantee that Info-zip's "-V" option will
> produce what any particular person (including you) wants when unzipped on a
> non-VMS system. In general there's a bunch of RMS junk between records
> and potentially random junk at the end. The case you're all hyped up
> about (stream LF) is indeed the best case but still
> not perfect.
I interpreted "generally" here as refering to files like those with
embedded record lengths. You may interpret it as refering to files
whose sizes are not N* 16K. We may have to agree to differ on this.
> You're perfectly free to hack up the innards of Info-zip however you want
> to do whatever you think it should do. But we still aren't into the
> realm of "problem" much less "solution" because it's working exactly as
> documented.
As you read and interpret it.
> > Please, before telling me again that ZIP -V _should_ work this way
> > because it _does_ work this way, do us all a favor and look at the code.
> > IT'S BROKEN.
>
> Look at the documentation. IT WORKS.
If you call describing a four-byte file (with sizes "1/35") as having
an "uncompressed size" of 16384 bytes "working", then I doubt that
there's much I can say to convince you otherwise.
David Froble
Developers have another purpose. To get paid. :-)
The problem with what you're asking is that if a version of ZIP were set up to
have a place in the archive for RMS attributes so that they could be ignored
when looking at/unpacking an archive on a non VMS system, every single variety
of ZIP in use would have to know about this 'place' so that it could be ignored.
You're basically asking for the ZIP format to change. Not feasable. How many
people out there will give a damn about a change VMS people want?
The non-standard option has been put into VMS versions of ZIP to allow
attributes to be preserved between VMS systems. It's not valid on other
systems. If you want it to be, as mentioned, you'll first have to change ZIP on
all other systems.
Ok, one of possibly many work arounds. Place the custom stuff in another file,
and need both files to extract attributes. The primary file would conform to
the format every other system expects.
If you want this, feel free to implement it, or, did you want to pay for it?
Dave
--
David Froble Tel: 724-529-0450
Dave Froble Enterprises, Inc. Fax: 724-529-0596
DFE Ultralights, Inc. E-Mail: da...@tsoft-inc.com
170 Grimplin Road
Vanderbilt, PA 15486
David J Dachtera
Um, well, no - it doesn't - at least, not in V2.3. Steve's correct there
- I duplicated his results on VMS using ZIP V2.3 with /VMS and "-V",
then used UNZIP v5.51 to /LIST (-l). The sizes displayed as he posted:
small files show as 16384.
$ dump/rec abc.seq
Dump of file USER$ROOT:[WORK]ABC.SEQ;1 on 21-SEP-2004 20:25:11.69
File ID (3875,2467,0) End of file block 1 / Allocated 35
Record number 1 (00000001), 3 (0003) bytes, RFA(0001,0000,0000)
434241 ABC............. 000000
$ dump/rec abc.slf
Dump of file USER$ROOT:[WORK]ABC.SLF;1 on 21-SEP-2004 20:25:17.69
File ID (3667,462,0) End of file block 1 / Allocated 35
Record number 1 (00000001), 3 (0003) bytes, RFA(0001,0000,0000)
434241 ABC............. 000000
$ dump/blo=cou=1 abc.seq
Dump of file USER$ROOT:[WORK]ABC.SEQ;1 on 21-SEP-2004 20:25:31.52
File ID (3875,2467,0) End of file block 1 / Allocated 35
Virtual block number 1 (00000001), 512 (0200) bytes
00000000 00000000 FFFF0043 42410003 ..ABC........... 000000
00000000 00000000 00000000 00000000 ................ 000010
00000000 00000000 00000000 00000000 ................ 000020
00000000 00000000 00000000 00000000 ................ 000030
00000000 00000000 00000000 00000000 ................ 000040
00000000 00000000 00000000 00000000 ................ 000050
00000000 00000000 00000000 00000000 ................ 000060
00000000 00000000 00000000 00000000 ................ 000070
00000000 00000000 00000000 00000000 ................ 000080
00000000 00000000 00000000 00000000 ................ 000090
00000000 00000000 00000000 00000000 ................ 0000A0
00000000 00000000 00000000 00000000 ................ 0000B0
00000000 00000000 00000000 00000000 ................ 0000C0
00000000 00000000 00000000 00000000 ................ 0000D0
00000000 00000000 00000000 00000000 ................ 0000E0
00000000 00000000 00000000 00000000 ................ 0000F0
00000000 00000000 00000000 00000000 ................ 000100
00000000 00000000 00000000 00000000 ................ 000110
00000000 00000000 00000000 00000000 ................ 000120
00000000 00000000 00000000 00000000 ................ 000130
00000000 00000000 00000000 00000000 ................ 000140
00000000 00000000 00000000 00000000 ................ 000150
00000000 00000000 00000000 00000000 ................ 000160
00000000 00000000 00000000 00000000 ................ 000170
00000000 00000000 00000000 00000000 ................ 000180
00000000 00000000 00000000 00000000 ................ 000190
00000000 00000000 00000000 00000000 ................ 0001A0
00000000 00000000 00000000 00000000 ................ 0001B0
00000000 00000000 00000000 00000000 ................ 0001C0
00000000 00000000 00000000 00000000 ................ 0001D0
00000000 00000000 00000000 00000000 ................ 0001E0
00000000 00000000 00000000 00000000 ................ 0001F0
$ dump/blo=cou=1 abc.slf
Dump of file USER$ROOT:[WORK]ABC.SLF;1 on 21-SEP-2004 20:25:35.13
File ID (3667,462,0) End of file block 1 / Allocated 35
Virtual block number 1 (00000001), 512 (0200) bytes
00000000 00000000 00000000 0A434241 ABC............. 000000
00000000 00000000 00000000 00000000 ................ 000010
00000000 00000000 00000000 00000000 ................ 000020
00000000 00000000 00000000 00000000 ................ 000030
00000000 00000000 00000000 00000000 ................ 000040
00000000 00000000 00000000 00000000 ................ 000050
00000000 00000000 00000000 00000000 ................ 000060
00000000 00000000 00000000 00000000 ................ 000070
00000000 00000000 00000000 00000000 ................ 000080
00000000 00000000 00000000 00000000 ................ 000090
00000000 00000000 00000000 00000000 ................ 0000A0
00000000 00000000 00000000 00000000 ................ 0000B0
00000000 00000000 00000000 00000000 ................ 0000C0
00000000 00000000 00000000 00000000 ................ 0000D0
00000000 00000000 00000000 00000000 ................ 0000E0
00000000 00000000 00000000 00000000 ................ 0000F0
00000000 00000000 00000000 00000000 ................ 000100
00000000 00000000 00000000 00000000 ................ 000110
00000000 00000000 00000000 00000000 ................ 000120
00000000 00000000 00000000 00000000 ................ 000130
00000000 00000000 00000000 00000000 ................ 000140
00000000 00000000 00000000 00000000 ................ 000150
00000000 00000000 00000000 00000000 ................ 000160
00000000 00000000 00000000 00000000 ................ 000170
00000000 00000000 00000000 00000000 ................ 000180
00000000 00000000 00000000 00000000 ................ 000190
00000000 00000000 00000000 00000000 ................ 0001A0
00000000 00000000 00000000 00000000 ................ 0001B0
00000000 00000000 00000000 00000000 ................ 0001C0
00000000 00000000 00000000 00000000 ................ 0001D0
00000000 00000000 00000000 00000000 ................ 0001E0
00000000 00000000 00000000 00000000 ................ 0001F0
$ zip/level=8/vms abcslf abc.*
adding: ABC.SEQ (deflated 86%)
adding: ABC.SLF (deflated 77%)
$ unzip/list abcslf
Archive: USER$ROOT:[WORK]ABCSLF.ZIP;1
Length Date Time Name
-------- ---- ---- ----
16384 09-20-04 13:14 abc.seq
16384 09-20-04 13:14 abc.slf
-------- -------
32768 2 files
$ zip/level=8 abcslfu abc.*
adding: ABC.SEQ (stored 0%)
adding: ABC.SLF (stored 0%)
$ unzip/list abcslfu
Archive: USER$ROOT:[WORK]ABCSLFU.ZIP;1
Length Date Time Name
-------- ---- ---- ----
4 09-20-04 13:14 abc.seq
4 09-20-04 13:14 abc.slf
-------- -------
8 2 files
WinZip and PKZip for Windows agree with UNZIP.
$ zip
Copyright (C) 1990-1999 Info-ZIP
Type 'zip "-L"' for software license.
Zip 2.3 (November 29th 1999). Usage: zip=="$disk:[dir]zip.exe"
zip zipfile[.zip] [list] [/EXCL=(xlist)] /options /modifiers
The default action is to add or replace zipfile entries from list,
except
those in xlist. The include file list may contain the special name -
to
compress standard input. If both zipfile and list are omitted, zip
compresses stdin to stdout.
Type zip -h for Unix style flags.
Major options include:
/FRESHEN, /UPDATE, /DELETE, /[NO]MOVE, /COMMENTS[={ZIP_FILE|FILES}],
/LATEST, /TEST, /ADJUST_OFFSETS, /FIX_ARCHIVE[=FULL], /UNSFX
Modifiers include:
/EXCLUDE=(file list), /INCLUDE=(file list), /SINCE="creation time",
/QUIET, /VERBOSE[=MORE], /[NO]RECURSE, /[NO]DIRNAMES, /JUNK,
/[NO]KEEP_VERSION, /[NO]VMS, /[NO]PKZIP, /TRANSLATE_EOL[={LF|CRLF}],
/[NO]EXTRA_FIELDS /LEVEL=[0-9], /TEMP_PATH=directory, /BATCH[=list
file]
$ unzip
UnZip 5.51 of 22 May 2004, by Info-ZIP. For more details see: unzip -v.
Usage: unzip file[.zip] [list] [/EXCL=(xlist)] [/DIR=exdir] /options
/modifiers
Default action is to extract files in list, except those in xlist, to
exdir;
file[.zip] may be a wildcard. Type "unzip /ZIPINFO" for ZipInfo-mode
usage.
Major options include (type unzip -h for Unix style flags):
/[NO]TEST, /LIST, /[NO]SCREEN, /PIPE, /[NO]FRESHEN, /[NO]UPDATE,
/[NO]COMMENT, /DIRECTORY=directory-spec, /EXCLUDE=(file-spec1,etc.)
Modifiers include:
/BRIEF, /FULL, /[NO]TEXT[=NONE|AUTO|ALL],
/[NO]BINARY[=NONE|AUTO|ALL],
/[NO]OVERWRITE, /[NO]JUNK, /QUIET, /QUIET[=SUPER], /[NO]PAGE,
/[NO]CASE_INSENSITIVE, /[NO]LOWERCASE, /[NO]VERSION, /[NO]RESTORE
Examples (see unzip.txt or "HELP UNZIP" for more info):
unzip edit1 /EXCL=joe.jou /CASE_INSENSITIVE => extract all files
except
joe.jou (or JOE.JOU, or any combination of case) from zipfile
edit1.zip
unzip zip201 "Makefile.VMS" vms/*.[ch] => extract VMS
Makefile and
*.c and *.h files; must quote uppercase names if /CASE_INSENS not
used
unzip foo /DIR=tmp:[.test] /JUNK /TEXT /OVER => extract all files
to temp.
directory without paths, auto-converting text files and
overwriting
Does this help?
D.J.D.
David J Dachtera
>
> David J Dachtera wrote:
> > That said, in my DECUS presentation from Fall-99 (San Diego), I put
> > forth the concept that the target platform should be borne in mind when
> > creating the archive. Preserving RMS attributes is really only of value
> > when it's needed for OpenVMS. VMS .ZIP archives (using "-V" or /VMS)
>
> True, but this avoids the real issue. As an user and not a developer, I
> rather am of the original poster's opinion. I hate when developers come
> to me saying 'do it because we know what is good for you' and then stop
> listening to any feedback (this reminds me of the whole Mozilla
> multi-user profiles crap).
>
> In this case, when I create an archive with "-V" I'd certainly expect to
> have a 'normal' archive (normal like in multiplatform, that's the ZIP
> goal isn't it),
Um, well, no, it isn't.
> and when and only when I unpack it on VMS to have the
> extra goodies packed by "-V" (like attributes). 'Platform independent'
> that is. I'm really looking forward for that, as at the moment I had to
> write Perl utilities to shape files between VMS Unix and Windows... awful.
Learn to use ZIP and UNZIP to your advantage instead of working against
yourself. Then, you can scrap the perl stuff or keep it for reference.
> I agree it's more work to develop, but that's what are developers for :D
What I'll do at this point is refer you to some of my on-line
documentation:
http://www.djesys.com/vms/mentor/rms.html
I refer you to the RMS page because we need to remember that RMS is
unique to the DEC world (although some third-party products exist for
other platforms). In order for what you ask to be possible, every
archive unpacker out there that understands Phil Katz's format would
have to also understand RMS so it can correcftly unpack the archive on
systems which lack RMS natively. Needless to say, such has never been
the case, nor is it likely to ever be.
http://www.djesys.com/vms/support/zipunzip/index.htm
In the ZIP/UNZIP presentation, see specifically slides:
http://www.djesys.com/vms/support/zipunzip/sld029.htm
thru
http://www.djesys.com/vms/support/zipunzip/sld038.htm
http://www.djesys.com/vms/support/zipunzip/sld044.htm
If that leaves you scratching your head, write me privately (see the
Contact Us page on the website, or demung the reply-to on this post) and
we can continue this off-line.
D.J.D.
s...@antinode.org
> Soterro wrote:
> > [...] I rather am of the original poster's opinion. [...]
Clearly a man of nice judgement and many resources.
> The problem with what you're asking is that if a version of ZIP were set up to
> have a place in the archive for RMS attributes so that they could be ignored
> when looking at/unpacking an archive on a non VMS system, every single variety
> of ZIP in use would have to know about this 'place' so that it could be ignored.
>
> You're basically asking for the ZIP format to change. Not feasable. How many
> people out there will give a damn about a change VMS people want?
> [...]
I always hate to suggest that anyone is a moron, but what makes you
think that a change would be required? Rather than arguing in a vacuum,
you might take a quick glance at the Zip source code, paying particular
attention to occurrences of the phrase, "extra field".
> The non-standard option has been put into VMS versions of ZIP to allow
> attributes to be preserved between VMS systems. It's not valid on other
> systems. If you want it to be, as mentioned, you'll first have to change ZIP on
> all other systems.
Is that bloated-rotting-dead horse still moving? Let me beat it
again. I realize that for some folks no actual evidence will ever
conquer a strongly held opinion reinforced by ignorance, but my latest
Wget kit is a ZIP -V archive, made with the Zip bug fixed. Please
complain if you find any corrupted files when you unpack it on your VMS
system. (Please pay particular attention to the attribute-sensitive
object files.) I unpacked it on a Tru64 V5.1B system (and SunOS 5.9),
and it worked just fine there, too.
I am reminded here of the words of the immortal Rocket J. Squirrel,
"Thank you, Mister Know-it-all."
------------------------------------------------------------------------
Steven M. Schweda (+1) 651-699-9818
382 South Warwick Street s...@antinode.org
Saint Paul MN 55105-2547
David Froble
> From: David Froble <da...@tsoft-inc.com>
>
>>Soterro wrote:
>>
>
>>>[...] I rather am of the original poster's opinion. [...]
>>>
>
> Clearly a man of nice judgement and many resources.
>
>
>>The problem with what you're asking is that if a version of ZIP were set up to
>>have a place in the archive for RMS attributes so that they could be ignored
>>when looking at/unpacking an archive on a non VMS system, every single variety
>>of ZIP in use would have to know about this 'place' so that it could be ignored.
>>
>>You're basically asking for the ZIP format to change. Not feasable. How many
>>people out there will give a damn about a change VMS people want?
>>[...]
>>
>
> I always hate to suggest that anyone is a moron, but what makes you
> think that a change would be required? Rather than arguing in a vacuum,
> you might take a quick glance at the Zip source code, paying particular
> attention to occurrences of the phrase, "extra field".
Ok, you're implying that there ia already a provision for 'custom' data in an
archive. I'll go with that, which does make me a bit of a moron for talking
about something without knowing what I'm talking about. However, the sources I
have indicate that ZIP is implemented in C, which I refuse to even read, which
is a definite sign of intellegence. On average that puts me ahead.
>>The non-standard option has been put into VMS versions of ZIP to allow
>>attributes to be preserved between VMS systems. It's not valid on other
>>systems. If you want it to be, as mentioned, you'll first have to change ZIP on
>>all other systems.
>>
>
> Is that bloated-rotting-dead horse still moving? Let me beat it
> again. I realize that for some folks no actual evidence will ever
> conquer a strongly held opinion reinforced by ignorance, but my latest
> Wget kit is a ZIP -V archive, made with the Zip bug fixed. Please
> complain if you find any corrupted files when you unpack it on your VMS
> system. (Please pay particular attention to the attribute-sensitive
> object files.) I unpacked it on a Tru64 V5.1B system (and SunOS 5.9),
> and it worked just fine there, too.
>
> I am reminded here of the words of the immortal Rocket J. Squirrel,
> "Thank you, Mister Know-it-all."
>
> ------------------------------------------------------------------------
>
> Steven M. Schweda (+1) 651-699-9818
> 382 South Warwick Street s...@antinode.org
> Saint Paul MN 55105-2547
>
--
Soterro
> Learn to use ZIP and UNZIP to your advantage instead of working against
> yourself. Then, you can scrap the perl stuff or keep it for reference.
I find quite difficult to use this version of ZIP to my advantage, when
my advantage is to create the right files on VMS but also usable on
Solaris (and readable on Windows). I simply cannot. I felt free to hack
it to the best of my knowledge with those 'scrappable' Perl scripts, and
it worked as it should have been. Tu my bemusement I see that a real
solution even exists but is rejected from the start as 'utter silliness'.
So what if it works as documented? Change then the documentation first,
people are not living by the Book Of Law. VMS was forced to adopt
TCP/IP, run Apache, port Mozilla, take out PostScript and so on, was
ever evolution a bad thing? Incidentally it was pressured by the
customers demand, not by developers decision :) Crossplatform is all
over, and ZIP will have to follow. This of course only if VMS _future_
is of any interest.
Please don't get me wrong, the way all you community guys handle any
help pleas is remarkable, kudos for that. I don't mean anything bad with
my note. The problem is not the past, but the future. Maybe a look into
the way Sun is handling (some) ideas from the openworld could bring
something. Don't blame for _all_ the problems the HP management :P
Here were another 2 cents, take them or leave them.
S
PS: Sorry that I used HP and management in the same phrase :D
s...@antinode.org
> [...]
> Ok, you're implying that there ia already a provision for 'custom' data in an
> archive. I'll go with that, which does make me a bit of a moron for talking
> about something without knowing what I'm talking about.
May I suggest, then, that you give serious consideration to greater
use of a particular keyboard entity? On my LK411, it's a "Shift-/"
(ASCII 63, I believe).
> However, the sources I
> have indicate that ZIP is implemented in C, which I refuse to even read, which
> is a definite sign of intellegence. On average that puts me ahead.
Determined ignorance can be tough to overcome.
------------------------------------------------------------------------
Steven M. Schweda (+1) 651-699-9818
s...@antinode.org
From: sho...@trailing-edge.com (Tim Shoppa)
Date: 20 Sep 2004 10:53:09 -0700
> "-V" zip archives unpack exactly on real VMS systems. (Or, well, they
> did, until you made your modification.) [...]
Actually, they didn't. When the Zip 2.3 code was reading a file for
a -V archive, it quit too soon to get all the allocated blocks, except
in a few special cases (disk cluster size = N* 32). Of course, in most
cases, some accurate post-EOF data are read, and in many cases the extra
data will all be zeros, so this might not be obvious.
My early observations showed me that the current code was far from
reading all allocated blocks, and that led me to conclude that it was
not intended to do it. Since then, I've consulted with some experts who
carry the oral tradition (which was particularly helpful, as the written
record is not very informative). They have convinced me that it was
actually intended to do this, and, in fact, it's not hard to arrange.
One lingering question follows. The upper-level Zip code tries to
read 64KB chunks. The current VMS (-V) code (QIOW, IO$_READVBLK) breaks
this down, and attempts to read 16KB chunks, while there seems to be no
obvious reason not to read 32KB chunks. Does anyone know of a disk (or
other file-structured?) device where a 32KB read may be expected to
fail? (Assume that there are at least that many bytes available to be
read.)
For the morbidly curious, my latest changed files are available at:
http://www.antinode.org/ftp/info-zip/
ftp://ftp.antinode.org/info-zip/
------------------------------------------------------------------------
David J Dachtera
>
> David J Dachtera wrote:
> > Learn to use ZIP and UNZIP to your advantage instead of working against
> > yourself. Then, you can scrap the perl stuff or keep it for reference.
>
> I find quite difficult to use this version of ZIP to my advantage, when
> my advantage is to create the right files on VMS but also usable on
> Solaris (and readable on Windows). I simply cannot.
Go through the presentation again(, and again, and ...). Download the
.PPT if that's a better medium for you. View/Print it in the Notes view
so it reads like a textbook.
ZIP has the options you need. Learn to use them. See the slides I
referenced in the earlier post, and the results of using those options
in another sub-thread (supporting Steve's observations).
> I felt free to hack
> it to the best of my knowledge with those 'scrappable' Perl scripts, and
> it worked as it should have been. Tu my bemusement I see that a real
> solution even exists but is rejected from the start as 'utter silliness'.
By whom?
Make as much work for yourself as you like. I'm just telling you that
the work has already been done. All that remains is to learn how to use
the product.
> So what if it works as documented? Change then the documentation
Well, sorry to have to report that Phil Katz died a couple years ago.
So, changing the specification ("documentation") is not an option unless
someone else wants to take ownership of the product.
> first,
> people are not living by the Book Of Law.
Yes. We see that - in Iraq, Afghanistan, Sudan, ...
> VMS was forced to adopt
> TCP/IP,
"Forced" is perhaps the not the right word. Given that freeware and
third-party TCP/IP stacks for OpenVMS existed before UCX, it was likely
a competitive choice.
> run Apache,
...competitive choice.
> port Mozilla,
competitive choice.
> take out PostScript
Bad decision by a vendor beyond VMS's control.
> and so on, was
> ever evolution a bad thing?
Read much?
> Incidentally it was pressured by the
> customers demand, not by developers decision :)
Well, yes and no. It was pressured by customers' CHOICEs, and the
competitive response was to make certain choices to match customer
needs.
> Crossplatform is all
> over, and ZIP will have to follow.
It does. Learn how to use it.
> This of course only if VMS _future_
> is of any interest.
It is. However, remember that the needs of the installed base far exceed
the need to keep up with bg, Inc. or any of the UN*X-land stuff out
there.
> Please don't get me wrong, the way all you community guys handle any
> help pleas is remarkable, kudos for that. I don't mean anything bad with
> my note. The problem is not the past, but the future.
It may be more accurate to say that the past, combined with the present
will point the way to the future; until then, however, the needs of the
installed base take precedence.
> Maybe a look into
> the way Sun is handling (some) ideas from the openworld could bring
> something. Don't blame for _all_ the problems the HP management :P
>
> Here were another 2 cents, take them or leave them.
Well, I'll take them, and try to turn them into something for you to
learn. The choice to learn or not is, of course, up to you.
> PS: Sorry that I used HP and management in the same phrase :D
We all make misteaks.
D.J.D.
David J Dachtera
>
> Today's update for the fascinated:
>
> From: sho...@trailing-edge.com (Tim Shoppa)
> Date: 20 Sep 2004 10:53:09 -0700
>
> > "-V" zip archives unpack exactly on real VMS systems. (Or, well, they
> > did, until you made your modification.) [...]
>
> Actually, they didn't. When the Zip 2.3 code was reading a file for
> a -V archive, it quit too soon to get all the allocated blocks, except
> in a few special cases (disk cluster size = N* 32). Of course, in most
> cases, some accurate post-EOF data are read, and in many cases the extra
> data will all be zeros, so this might not be obvious.
So, if I follow this correctly, both ZIP and UNZIP need a fix?
...not just ZIP?
D.J.D.
s...@antinode.org
> > [... Zip 2.3 really is broken ...]
> So, if I follow this correctly, both ZIP and UNZIP need a fix?
>
> ...not just ZIP?
As far as I've been able to tell, UnZip is ok. More research is
required, but it's quite possible that UnZip currently takes whatever
data it has, writes them out, and sets the file attributes as they were
originally, and it'll work just fine if Zip just puts all the data into
the archive. A few lost blocks past EOF (typically all zeros) would not
have drawn the attention of almost all users, anyway. If no one's
complained up to now, how big a deal could it be, really?
It's also possible that UnZip has problems, but (in limited testing)
I haven't seen any yet.
At the moment, I trying to test some other changes dealing with the
problem Zip has with input files bigger than 2GB. (Signed v. unsigned
byte counts, as claimed earlier by someone else.) It's slow going with
my hardware, and while I have plenty of free disk space for normal use,
I think I'll need to slap another drive on somewhere to handle this job.
Everything's complicated.
I have yet to ask UnZip to write a file bigger than 2GB, and I've not
searched the code for signs of trouble in this regard. ("Signs" of
trouble... Oh, I crack me up sometimes.)
Both sets of changes are in the the Zip update files mentioned in an
earlier posting. You're welcome to suck 'em down and let me know what
you learn. (The one UnZip update file is of little consequence.)
It might be better if I left this one "unsigned", but:
Soterro
> Go through the presentation again(, and again, and ...). Download the
> .PPT if that's a better medium for you. View/Print it in the Notes view
> so it reads like a textbook.
You obviously are unable to read my postings, there must be somewhere a
problem in between. But I lay this perception problem aside, as long the
developer of ZIP understood the issue I'm happy.
> It may be more accurate to say that the past, combined with the present
> will point the way to the future; until then, however, the needs of the
> installed base take precedence.
Ideally one shouldn't be forced to make this choice but walk on both
paths: keep up with the needs of the (ever-shrinking) installed base and
also evolve in order to keep the (once-existing) advantage. But yes, the
developer base is not exactly what it used to be thus the 1999 slides
are the most actual information in so many places.
*shrug*
S
David J Dachtera
>
> David J Dachtera wrote:
> > Go through the presentation again(, and again, and ...). Download the
> > .PPT if that's a better medium for you. View/Print it in the Notes view
> > so it reads like a textbook.
>
> You obviously are unable to read my postings,
Um, well, since I answered your post point-for-point ... (finish the
sentence).
Statements like that make you look silly and inexperienced. I generally
advise against them.
> there must be somewhere a
> problem in between.
As I said: we all make mistakes. Mine was in assuming that you are
teachable. Obviously you are not.
> But I lay this perception problem aside, as long the
> developer of ZIP understood the issue I'm happy.
I miss a good many posts thanx to the "Mark Thread Read" option, so it's
possible I missed it, but I don't recall seeing any recent posts by the
current maintainer or the author.
> > It may be more accurate to say that the past, combined with the present
> > will point the way to the future; until then, however, the needs of the
> > installed base take precedence.
>
> Ideally one shouldn't be forced to make this choice
Ideals are like the stars: we set our course by them, yet we never reach
them.
> but walk on both
> paths: keep up with the needs of the (ever-shrinking) installed base and
> also evolve in order to keep the (once-existing) advantage. But yes, the
> developer base is not exactly what it used to be thus the 1999 slides
> are the most actual information in so many places.
As there is little left to develop in the PK compression world, that
seems reasonable. ODS-5 support would be good...
D.J.D.
s...@antinode.org
checking the Google record of comp.os.vms. I see now that I've been
missing out on some real humor.
From: John Laird (nos...@laird-towers.org.uk)
Date: 2004-09-21 08:26:58 PST
> I refer you to the description of -V. There is no misbehaviour. [...]
Right.
> It is not
> intended to provide a zip file that may or may not work elsewhere. It is
> intented to provide a zip file that will work on VMS systems.
What made you think that these goals were mutually exclusive?
They're not. It's just that up to now, it hasn't worked right.
> For this to
> happen, it must include all file metadata as well as what one might term
> "real" data. The simplest solution is to read all blocks up to and include
> the EOF block for sequential files and ALQ block for others. However, once
> this is stored in the archive, how is a non-VMS UNZIP supposed to know the
> difference - all it will see is a stream of bytes.
Partly right. The solution is to read all the blocks always. The
non-VMS UnZip simply needs an accurate file byte count (which it wasn't
getting), and then it can do just fine.
> > > [... reading all blocks even the allocated-but-unused ...]
> > Second, what makes you think it does this now? (Not that I would
> >rule it out.) How would I test the results? For example, does BACKUP
> >/COMPARE check "all allocated blocks"?
< I don't need to think,
You should have stopped there.
> I *know* it currently works.
Just as some folks know that the Earth is flat.
> You can see the code in
> one of the source files you referred to.
As someone who has actually _looked_ at the code in question, what I
_can_ see is that it did _not_ do what you "*know*" it did.
> [...]
> To be honest, I would say that if you are not sure how to test the veracity
> of your changes on all file types, then you should not be making those
> changes.
I'm sure that I should be listening to advice from the likes of you
instead.
> By all means do this for your own use but please don't take steps
> to meld these into the publicly available versions until you are totally
> satisfied there are no unwanted side-effects.
Be my guest if you have doubts. I don't need to test it, because I
"*know*" it works right with my changes. Luckily, the Info-ZIP folks
seem to be more clueful than some other folks.
> [...] I did look at your source files but could not identify exact
> changes very well without access to the unmodified versions,
Just the stock Zip 2.3 source, available from oodles of places.
> so will not comment further.
Well, that's a mercy.
Of all the things for Info-VAX to lose, it had to pick this one.
(Some folks make being right _so_ much more fun than it would be
ordinarily.)
By the way, I found a fix for yet another odd-ball Zip problem. If
you have SET PROC /PARSE = EXT engaged, and your default directory is
not all-upper-case, Zip can become confused when it tries to strip the
current directory off the in-archive file names. For example:
ALP $ set def ALP$DKA0:[UTILITY.SOURCE.ZIP.ZIP-2_3X]
ALP $ zip "-V" test_uc-v.zip [.x]TEST_?.SLF ! OK.
adding: [.X]TEST_4.SLF (deflated 100%)
adding: [.X]TEST_5.SLF (deflated 100%)
but
ALP $ set def ALP$DKA0:[UTILITY.source.zip.ZIP-2_3X]
ALP $ sho def
ALP$DKA0:[UTILITY.source.zip.ZIP-2_3X]
ALP $ zip "-V" test_lc-v.zip [.x]TEST_?.SLF ! What could go wrong?
adding: [.UTILITY.SOURCE.ZIP.ZIP-2_3X.X]TEST_4.SLF (deflated 100%)
adding: [.UTILITY.SOURCE.ZIP.ZIP-2_3X.X]TEST_5.SLF (deflated 100%)
============================ ! Oh.
The fix for this one is in VMSZIP.C. of which a fresh one is
available at the usual place:
http://www.antinode.org/ftp/info-zip/
ftp://ftp.antinode.org/info-zip/
Oh, no! Maybe it was _supposed_ to do that! It _does_ only happen
with "-V", after all. I guess it must have been intentional. Sorry.
Forget I mentioned it. Please.
Also, in the stroll-down-memory-lane department, the revised code
seems to compile and run on VMS V5.5-2 with DEC C V4.0-000. Who has
something older? VAX C? (I tossed my VAX C due to RD54 capacity
limits. Sigh.)
John Laird
>(Some folks make being right _so_ much more fun than it would be
>ordinarily.)
After testing, it turns out you are right. Most likely, I have not had to
restore any relative or indexed files with EOF not correctly set before
ZIP'ing (it could well be argued that such files could in any case contain
inconsistencies and ought not to be regarded as candidates for archiving
without a prior ANAL/RMS run and possible CONVERT). I apologise.
Nevertheless, it is still a concern that you have posted references to
corrected code without a similar reference to uncorrected code, and
apparently expect others to go find said code to make their own comparisons.
You may find your own changes easy to spot as you wrote them, but the same
is not true of other people. There were no particularly obvious change
bars, comment blocks or commented-out replaced lines that made eyeballing
your work any easier. In one module, you may have written an entire new
function, but I could not be sure.
And I notice you're still asking others to test out "odd" files for you.
Which is it - you are completely right and don't need to test, or not ? It
should be easy enough to grab a copy of a sample indexed file (say
SYSUAF.DAT) and use SET FILE/ATTR=(EBK:) to artificially truncate the file.
Zip and unzip it, then reset the end-of-file block on the copy and the
unzipped version and use DUMP to make sure the blocks beyond the incorrect
EOF have been saved in the zip file and restored with unzip.
Strictly speaking, files with EOF incorrectly set are potentially corrupt
anyway, but both COPY and BACKUP will transfer all allocated blocks
regardless. It is obviously preferable that ZIP succeeds in emulating this
(and the code clearly tried).
--
Press any key to continue or any other key to quit.
s...@antinode.org
> After testing, it turns out you are right. Most likely, I have not had to
> restore any relative or indexed files with EOF not correctly set before
> ZIP'ing (it could well be argued that such files could in any case contain
> inconsistencies and ought not to be regarded as candidates for archiving
> without a prior ANAL/RMS run and possible CONVERT). I apologise.
I tried arguing that, briefly, but all I got was abuse.
> Nevertheless, it is still a concern that you have posted references to
> corrected code without a similar reference to uncorrected code, and
> apparently expect others to go find said code to make their own comparisons.
> [...]
If you don't already have Zip 2.3 source, what good are you?
> And I notice you're still asking others to test out "odd" files for you.
> Which is it - you are completely right and don't need to test, or not ?
I'm not yet convinced. I only "*know*" it, and we both have seen how
unreliable that can be.
> It
> should be easy enough to grab a copy of a sample indexed file (say
> SYSUAF.DAT) and use SET FILE/ATTR=(EBK:) to artificially truncate the file.
> Zip and unzip it, then reset the end-of-file block on the copy and the
> unzipped version and use DUMP to make sure the blocks beyond the incorrect
> EOF have been saved in the zip file and restored with unzip.
Apparently, every time I've done HELP SET FILE /ATTR I've skipped
right past this (looking for RAT and RFM). My most recent experiments
suggest that BACKUP quits at the EOF, too, or else DUMP /ALLOC is lying
to me. I'm still looking.
> Strictly speaking, files with EOF incorrectly set are potentially corrupt
> anyway, but both COPY and BACKUP will transfer all allocated blocks
> regardless. It is obviously preferable that ZIP succeeds in emulating this
> (and the code clearly tried).
As I said, I'm still trying to convince myself of that, but you're
welcome to provide a demonstration.
I'm starting to think that on VMS V5.5-2, DUMP /ALLOC is not entirely
trustworthy:
WEAK $ dump /all CRC32.VAX_DECC_OBJ_bac /block
[...]
Dump of file
DUA1:[UTILITY.SOURCE.ZIP.ZIP-2_3X.X]CRC32.VAX_DECC_OBJ_BAC;1 on 25-SEP-2004 09:28:36.04
File ID (331,12,0) End of file block 2 / Allocated 3
Virtual block number 2 (00000002), 512 (0200) bytes
D65108AC D0635160 42CD5163 1808EF52 Rï..cQÍB`QcÐŽ.QÖ 000000
EF52FFFF FF008FCA 5263CC52 619A08AC Ž..aRĖcRĘ.....Rï 000010
ACD10CAC 08C26351 6042CDB3 51631808 ..cQģÍB`QcÂ.Ž.ŅŽ 000020
0004CCDE 29130CAC D5FF0331 031F080C ....1..ÕŽ..)ÞĖ.. 000030
CA5263CC 52619A08 ACD65108 ACD00153 S.ÐŽ.QÖŽ..aRĖcRĘ 000040
63516042 CD516318 08EF52FF FFFF008F .....Rï..cQÍB`Qc 000050
54D05404 ACFFFFFF FF8FCDDD 120CACD7 ŨŽ..ÝÍ.....Ž.TÐT 000060
0000FFFF 01000000 00000003 00080450 P............... 000070
00000000 00000000 00000000 00000000 ................ 000080
00000000 00000000 00000000 00000000 ................ 000090
[...]
Dump of file
DUA1:[UTILITY.SOURCE.ZIP.ZIP-2_3X.X]CRC32.VAX_DECC_OBJ_BAC;1 on 25-SEP-2004 09:28:36.04
File ID (331,12,0) End of file block 2 / Allocated 3
Virtual block number 3 (00000003), 512 (0200) bytes
D65108AC D0635160 42CD5163 1808EF52 Rï..cQÍB`QcÐŽ.QÖ 000000
EF52FFFF FF008FCA 5263CC52 619A08AC Ž..aRĖcRĘ.....Rï 000010
ACD10CAC 08C26351 6042CDB3 51631808 ..cQģÍB`QcÂ.Ž.ŅŽ 000020
0004CCDE 29130CAC D5FF0331 031F080C ....1..ÕŽ..)ÞĖ.. 000030
CA5263CC 52619A08 ACD65108 ACD00153 S.ÐŽ.QÖŽ..aRĖcRĘ 000040
63516042 CD516318 08EF52FF FFFF008F .....Rï..cQÍB`Qc 000050
54D05404 ACFFFFFF FF8FCDDD 120CACD7 ŨŽ..ÝÍ.....Ž.TÐT 000060
0000FFFF 01000000 00000003 00080450 P............... 000070
00000000 00000000 00000000 00000000 ................ 000080
00000000 00000000 00000000 00000000 ................ 000090
[...]
Looks a lot like block 2, coincidentally, while:
WEAK $ dump /all CRC32.VAX_DECC_OBJ_bac /block = start = 3
Dump of file
DUA1:[UTILITY.SOURCE.ZIP.ZIP-2_3X.X]CRC32.VAX_DECC_OBJ_BAC;1 on 25-SEP-2004 09:28:43.83
File ID (331,12,0) End of file block 2 / Allocated 3
Virtual block number 3 (00000003), 512 (0200) bytes
00000000 00000000 00000000 00000000 ................ 000000
00000000 00000000 00000000 00000000 ................ 000010
00000000 00000000 00000000 00000000 ................ 000020
00000000 00000000 00000000 00000000 ................ 000030
00000000 00000000 00000000 00000000 ................ 000040
[...]
It's a poor workman who blames his tools, but ...
Craig A. Berry
> By the way, I found a fix for yet another odd-ball Zip problem. If
> you have SET PROC /PARSE = EXT engaged, and your default directory is
> not all-upper-case, Zip can become confused when it tries to strip the
> current directory off the in-archive file names. For example:
>
> ALP $ set def ALP$DKA0:[UTILITY.SOURCE.ZIP.ZIP-2_3X]
>
> ALP $ zip "-V" test_uc-v.zip [.x]TEST_?.SLF ! OK.
> adding: [.X]TEST_4.SLF (deflated 100%)
> adding: [.X]TEST_5.SLF (deflated 100%)
>
> but
>
> ALP $ set def ALP$DKA0:[UTILITY.source.zip.ZIP-2_3X]
>
> ALP $ sho def
> ALP$DKA0:[UTILITY.source.zip.ZIP-2_3X]
>
> ALP $ zip "-V" test_lc-v.zip [.x]TEST_?.SLF ! What could go wrong?
> adding: [.UTILITY.SOURCE.ZIP.ZIP-2_3X.X]TEST_4.SLF (deflated 100%)
> adding: [.UTILITY.SOURCE.ZIP.ZIP-2_3X.X]TEST_5.SLF (deflated 100%)
> ============================ ! Oh.
>
> The fix for this one is in VMSZIP.C. of which a fresh one is
> available at the usual place:
It would be better to preserve case and make the comparisons case
blind, as in the patch to the GNV version of ZIP to which I posted the
URL earlier in this thread. The basic procedure is to remove the calls
to strlower() and strupper() and change all the strncmp() calls to use
strncasecmp() instead.
You can find the GNV version by installing GNV and then looking in
GNU:[SRC.GNV.ZIP.VMS]. You can find my change on the patches page of
the GNV SourceForge project:
<http://sourceforge.net/tracker/?group_id=2506&atid=302506>. The patch
also turns foo^.bar.baz into foo.bar.baz in the archive so that the
caret escape doesn't show up when the archive is unpacked on other
systems.
If you are in communication with the Info-Zip maintainers, it would be
a great service to see the GNV changes reconciled and incorporated into
the authoritative sources.
s...@antinode.org
[Some of this may look familiar.]
> It
> should be easy enough to grab a copy of a sample indexed file (say
> SYSUAF.DAT) and use SET FILE/ATTR=(EBK:) to artificially truncate the file.
Thanks for that valuable suggestion. I used a plain sequential file,
32/35 truncated to 1/35, and my revised Zip code appeared to give the
desired result in at least this one test case. (So what could go
wrong?)
As always, I'm open to a nice counter-example.
> Zip and unzip it, then reset the end-of-file block on the copy and the
> unzipped version and use DUMP to make sure the blocks beyond the incorrect
> EOF have been saved in the zip file and restored with unzip.
DUMP /ALLOCATED allows one to look at the allocated-but-unused (or is
that "unused"?) blocks, so there's no need to diddle the attributes
again, just to do the comparison.
Interestingly, as I thought I had seen before, BACKUP appears to
honor the EOF marker. (This is on the UnZipped file, by the way.)
ALP $ dump /alloc /block = (start = 2, end = 2) TEST_16K_TRUN.SLF
Dump of file SYS$SYSDEVICE:[UTILITY.SOURCE.ZIP.ZIP-2_ on 25-SEP-2004 17:30:31.89
3X.X]TEST_16K_TRUN.SLF;1
File ID (148099,10,0) End of file block 1 / Allocated 35
Virtual block number 2 (00000002), 512 (0200) bytes
31313131 31313131 31313131 31313131 1111111111111111 000000
31313131 31313131 31313131 31313131 1111111111111111 000010
31313131 31313131 31313131 31313131 1111111111111111 000020
0A313131 31313131 31313131 31313131 111111111111111. 000030
[...]
ALP $ backup TEST_16K_TRUN.SLF TEST_16K_TRUN.SLF_bac
ALP $ dump /alloc /block = (start = 2, end = 2) TEST_16K_TRUN.SLF_bac
Dump of file SYS$SYSDEVICE:[UTILITY.SOURCE.ZIP.ZIP-2_ on 25-SEP-2004 17:32:11.33
3X.X]TEST_16K_TRUN.SLF_BAC;1
File ID (138850,23,0) End of file block 1 / Allocated 35
Virtual block number 2 (00000002), 512 (0200) bytes
00000000 00000000 00000000 00000000 ................ 000000
00000000 00000000 00000000 00000000 ................ 000010
00000000 00000000 00000000 00000000 ................ 000020
00000000 00000000 00000000 00000000 ................ 000030
[...]
s...@antinode.org
> > By the way, I found a fix for yet another odd-ball Zip problem. If
> > you have SET PROC /PARSE = EXT engaged, and your default directory is
> > not all-upper-case, Zip can become confused when it tries to strip the
> > current directory off the in-archive file names. For example:
> > [...]
> It would be better to preserve case and make the comparisons case
> blind, as in the patch to the GNV version of ZIP to which I posted the
> URL earlier in this thread. The basic procedure is to remove the calls
> to strlower() and strupper() and change all the strncmp() calls to use
> strncasecmp() instead.
Of course, strncasecmp() apparently needs "__CRTL_VER >= 70000000",
so a substitute must be added for older systems.
> You can find the GNV version by installing GNV and then looking in
> GNU:[SRC.GNV.ZIP.VMS].
I may be to lazy to go through all that for Zip.
> You can find my change on the patches page of
> the GNV SourceForge project:
> <http://sourceforge.net/tracker/?group_id=2506&atid=302506>. The patch
> also turns foo^.bar.baz into foo.bar.baz in the archive so that the
> caret escape doesn't show up when the archive is unpacked on other
> systems.
I was assuming that serious ODS5 support was unlikely in Zip 2.x.
I'm not sure what's true in the future V3.x.
> If you are in communication with the Info-Zip maintainers, it would be
> a great service to see the GNV changes reconciled and incorporated into
> the authoritative sources.
I'll pass it along. Making that less enjoyable is the fact that in
Netscape 3, "http://sourceforge.net/tracker/download.php?
group_id=2506&atid=302506&file_id=80795&aid=920330" appears to be devoid
of line breaks.
------------------------------------------------------------------------
Steven M. Schweda (+1) 651-699-9818
Dave Weatherall
> From: John Laird <nos...@laird-towers.org.uk>
> [Some of this may look familiar.]
>
> > It
> > should be easy enough to grab a copy of a sample indexed file (say
> > SYSUAF.DAT) and use SET FILE/ATTR=(EBK:) to artificially truncate the file.
>
> Thanks for that valuable suggestion. I used a plain sequential file,
> 32/35 truncated to 1/35, and my revised Zip code appeared to give the
> desired result in at least this one test case. (So what could go
> wrong?)
>
> As always, I'm open to a nice counter-example.
>
> > Zip and unzip it, then reset the end-of-file block on the copy and the
> > unzipped version and use DUMP to make sure the blocks beyond the incorrect
> > EOF have been saved in the zip file and restored with unzip.
>
> DUMP /ALLOCATED allows one to look at the allocated-but-unused (or is
> that "unused"?) blocks, so there's no need to diddle the attributes
> again, just to do the comparison.
>
> Interestingly, as I thought I had seen before, BACKUP appears to
> honor the EOF marker. (This is on the UnZipped file, by the way.)
<SNIP>
>
> Steven M. Schweda (+1) 651-699-9818
> 382 South Warwick Street sms@antinode-org
> Saint Paul MN 55105-2547
My memory is that, on some of my indexed files at least, BACKUP
/COMPARE detects rubbish remnants in blocks and issues a warning. ISTR
that I've seen it on some 'text' files too but I can't remember the
context. DIFFing the file checks the content and not the cruft. At
home so can't check.
--
Cheers - Dave.
PS.
Steven, I suspect your newsreader/poster might not be 'threading' your
replies correctly. Here they appear out of context in tree view.
Doc.
Pmz2dCDd36yQ@dave2_os2.home.ours
> PS.
>
> Steven, I suspect your newsreader/poster might not be 'threading' your
> replies correctly. Here they appear out of context in tree view.
>
He's accessing the newsgroup via the Info-VAX gateway. The References:
headers get lost, hence the threading problem.
Doc.
--
OpenVMS: Eight out of ten hackers prefer *other* operating systems.
http://www.openvms-rocks.com Deathrow Public-Access OpenVMS Cluster.
Soterro
> Statements like that make you look silly and inexperienced.
Well, then why do you even bother answering me here? (and ignored my
private mail) Since when does my image really concern COV? I just hope
you aren't working in tech support...
Please move on to the original poster if you _really_ have something to
say against a ZIP fix. Incidentally, you can't use l334 bashing there ;)
S
PS: my apologies to the others, which had to endure this pointless
conversation. EOT
David J Dachtera
>
> David J "works that way because I said so"
Um, well, no - read the source code, like Steve did. Then, experiment
and correlate your findings...
WOW! That REALLY made you look superior and intelligent!
Bravo! Bravo! Encore! Encore!
--
David J Dachtera
dba DJE Systems
http://www.djesys.com/
Unofficial OpenVMS Hobbyist Support Page:
http://www.djesys.com/vms/support/
Unofficial Affordable OpenVMS Home Page:
http://www.djesys.com/vms/soho/