corrupt zip files

[kerravon]

I have a zip file that appears to have been produced using pkzip for z/
OS.

However, it looks like it has been transmitted using some sort of text
protocol, because the high bit has been stripped from most bytes, and
some other bytes appear to have been translated. e.g. I think x'0a' or
x'14' in the input file has been mangled to x'b6' on the way. Does
anyone know what software would do a translation like that?

I believe these characters:

> 0A 1060
> 0D 0
> 12 1044
> 14 0
> 15 0
> 1C 0
> 1E 0
> 24 1030
> 7F 0

are being mapped to these:

> 81 576
> 9C 361
> A7 645
> B6 527
> BF 284
> E0 249
> E9 644
> F2 718

Except for x'0d' which I think is just being deleted. This belief
comes from counting (see below) occurrences of the various bytes in a
largish (80k) file.

Here is a small file that shows the problem:

000000 504B0304 B6000600 08006455 22405746 PK........dU"@WF

That x'b6' above should be x'0a' or x'14' I think (that's more
normal). Apparently some protocol doesn't think that x'0a'/x'14' will
make it through, so translates it in advance.

000010 30447A00 00002802 00000800 00005851 0Dz...(.......XQ
000020 46303130 38313510 310A4330 0C447740 F010815.1.C0.Dw@
...
0000A0 504B0102 780BB600 06000800 64552240 PK..x.......dU"@
0000B0 57463044 7A000000 28020000 08003401 WF0Dz...(.....4.
0000C0 00000000 01000000 00000000 00005851 ..............XQ
0000D0 46303130 38316500 30016973 79700006 F01081e.0.isyp..

This x'69737970' is really (once high bit is added back) x'E9F3F9F0'
ie 'Z390' ie something that pkzip for z/OS (MVS) inserts. ie I can
easily see that this file originated on the mainframe. And I can
easily see that the high bit has been stripped on most characters.
Interestingly I do see x'E9' in the output file, even though I can
easily see that x'E9' has been stripped above. So real x'E9' are being
stripped, while probably some other character is causing x'E9' to be
produced. Possibly it has gone through two pieces of software to
produce this effect.


0000E0 00014000 00050002 10000600 04022800 ..@...........(.
0000F0 06005C00 08000700 05000001 00070006 ..\.............
000100 00004B00 05000740 0040000B 00064462 ..K....@.@....Db
000110 52707073 40404040 40404040 40404040 Rpps@@@@@@@@@@@@
000120 40404040 40404040 40400A @@@@@@@@@@.

Does anyone have any idea what protocol (ftp, sftp, winscp, kermit,
connect:direct, http, pgp) would affect data in this manner? I've
never seen mangling like that before.

statistical analysis on a large zip file similarly mangled:

00 594
01 698
02 740
03 488
04 749
05 697
06 536
07 526
08 545
09 854
0A 1060
0B 597
0C 641
0D 0
0E 608
0F 639
10 817
11 679
12 1044
13 641
14 0
15 0
16 621
17 533
18 554
19 611
1A 517
1B 555
1C 0
1D 612
1E 0
1F 639
20 731
21 592
22 607
23 549
24 1030
25 546
26 565
27 618
28 490
29 602
2A 436
2B 684
2C 629
2D 706
2E 589
2F 667
30 547
31 815
32 670
33 530
34 569
35 598
36 570
37 619
38 723
39 572
3A 508
3B 626
3C 676
3D 626
3E 615
3F 621
40 687
41 598
42 636
43 557
44 752
45 579
46 645
47 813
48 849
49 837
4A 617
4B 563
4C 587
4D 532
4E 618
4F 705
50 538
51 541
52 553
53 573
54 467
55 416
56 653
57 681
58 727
59 599
5A 560
5B 344
5C 611
5D 293
5E 663
5F 552
60 578
61 562
62 608
63 814
64 649
65 711
66 515
67 660
68 484
69 506
6A 528
6B 627
6C 773
6D 646
6E 627
6F 599
70 602
71 657
72 636
73 620
74 521
75 516
76 732
77 631
78 596
79 715
7A 551
7B 718
7C 621
7D 606
7E 630
7F 0
80 0
81 576
82 0
83 0
84 0
85 0
86 0
87 0
88 0
89 0
8A 0
8B 0
8C 0
8D 0
8E 0
8F 0
90 0
91 0
92 0
93 0
94 0
95 0
96 0
97 0
98 0
99 0
9A 0
9B 0
9C 361
9D 0
9E 0
9F 0
A0 0
A1 0
A2 0
A3 0
A4 0
A5 0
A6 0
A7 645
A8 0
A9 0
AA 0
AB 0
AC 0
AD 0
AE 0
AF 0
B0 0
B1 0
B2 0
B3 0
B4 0
B5 0
B6 527
B7 0
B8 0
B9 0
BA 0
BB 0
BC 0
BD 0
BE 0
BF 284
C0 0
C1 0
C2 0
C3 0
C4 0
C5 0
C6 0
C7 0
C8 0
C9 0
CA 0
CB 0
CC 0
CD 0
CE 0
CF 0
D0 0
D1 0
D2 0
D3 0
D4 0
D5 0
D6 0
D7 0
D8 0
D9 0
DA 0
DB 0
DC 0
DD 0
DE 0
DF 0
E0 249
E1 0
E2 0
E3 0
E4 0
E5 0
E6 0
E7 0
E8 0
E9 644
EA 0
EB 0
EC 0
ED 0
EE 0
EF 0
F0 0
F1 0
F2 718
F3 0
F4 0
F5 0
F6 0
F7 0
F8 0
F9 0
FA 0
FB 0
FC 0
FD 0
FE 0
FF 0

The best theory anyone has come up with is this:

Most likely the original byte was x'14' which is also quite common in
this location in zip files. In code page 437 and its cousins, x'14' is
the paragraph sign. In iso9959-1 and many of its cousins, x'b6' is the
same paragraph sign. In 437, x'15' is the section symbol. In
iso8859-1, x'a7' is the same section symbol. Your occurrence counts
show there are no x'14' or x'15', but among the few characters
occurring above x'7f' are x'b6' and x'a7'. I would guess that at least
two separate things happened to the file at different times. First,
all the high-order bits were turned off. Later, a translation was done
as if the data was presumed to be in code page 437, which converted
x'14' and x'15' to x'b6' and x'a7', resulting in some values having
the high-order bit set. Something else happened, either at the same
time or not, that changed a few other characters to other values that
have the high-order bit set, for which I have no explanation.


Note that the file came to me from an external party, and the external
party has already re-transmitted a non-corrupt replacement file, and
considers it to be a case of "problem solved" and they're not
interested in helping me determine exactly what happened on their
systems to cause this effect. So this may remain an unsolved mystery.

BFN. Paul.

[Dan Espen]

Didn't I see this same issue raised on IBM-MAIN last week?
I think you got your answers then.

The file has PLAINLY been translated from ASCII to EBCDIC with
the new lines disappearing because they become record boundaries
on the mainframe.

Any software that transmits from an ASCII machine to an IBM
mainframe could do translation like that.

If you are trying to keep a zip file intact, make sure you
send the file in binary mode.

--
Dan Espen

[kerravon]

On May 15, 9:25 am, Dan Espen <des...@verizon.net> wrote:

Yes.

> I think you got your answers then.

No, the closest I got was what I posted here - about
x'14' being translated to x'b6' due to code page
conversions, with no explanation for the rest.

> The file has PLAINLY

"plainly", eh?

> been translated from ASCII to EBCDIC with
> the new lines disappearing because they become record boundaries
> on the mainframe.

Au contraire. The file I received is full of x'0a'.

In addition, the file PLAINLY started life as EBCDIC,
not ASCII.

> Any software that transmits from an ASCII machine to an IBM
> mainframe could do translation like that.

A translation like what? x'14' to x'b6'? What ASCII and
EBCDIC charts are you referring to?

And what about the high bits being stripped off the
EBCDIC codes? What ASCII to EBCDIC conversion does
that?

> If you are trying to keep a zip file intact, make sure you
> send the file in binary mode.

Yes, of course I know something as bleedingly obvious
as that. Do you really think I know how to interpret a
hex dump of a zip file, but yet am so stupid that I
don't know the difference between text and binary
file transfers? What I'm interested in is what conversion
(starting from an EBCDIC zip file) would result in
the data characteristics I have shown above.

BFN. Paul.

[Peter Flass]

On 5/15/2012 8:27 AM, kerravon wrote:
>
> Yes, of course I know something as bleedingly obvious
> as that. Do you really think I know how to interpret a
> hex dump of a zip file, but yet am so stupid that I
> don't know the difference between text and binary
> file transfers? What I'm interested in is what conversion
> (starting from an EBCDIC zip file) would result in
> the data characteristics I have shown above.
>

Some sort of email software perhaps?


--
Pete

[Dan Espen]

I have no idea what level of technical prowess you possess.
I didn't follow the IBM-MAIN discussion in detail, just
noted that many of the characters seemed to be in the EBCDIC range in
one case and ASCII in another.

> What I'm interested in is what conversion
> (starting from an EBCDIC zip file) would result in
> the data characteristics I have shown above.

The high bit would get lost going from an 8 bit code to 7 bit ASCII.
I agree that at least 2 things seem to be going on. No idea
what PKZIP would do to the data.

--
Dan Espen

[kerravon]

On May 16, 12:37 am, Dan Espen <des...@verizon.net> wrote:
> > Yes, of course I know something as bleedingly obvious
> > as that. Do you really think I know how to interpret a
> > hex dump of a zip file, but yet am so stupid that I
> > don't know the difference between text and binary
> > file transfers?
>
> I have no idea what level of technical prowess you possess.
> I didn't follow the IBM-MAIN discussion in detail, just
> noted that many of the characters seemed to be in the EBCDIC range in
> one case and ASCII in another.

Yes, that's the nature of a zipped file on the mainframe.
Things like the "PK" at the start of a zip file are
required to be in ASCII (even on an EBCDIC machine), as
it is part of the specs. But there is an area of the
zip file where "extensions" are allowed, and the format
of this allows EBCDIC, and pkzip is designed to put an
EBCDIC "Z390" in that spot (as an example). So that's
why the file starts life with both ASCII and EBCDIC
characters in it.

And what I received at the end was the ASCII characters
intact, as far as I can see, but the (known) EBCDIC
characters with the high bit stripped.

> > What I'm interested in is what conversion
> > (starting from an EBCDIC zip file) would result in
> > the data characteristics I have shown above.
>
> The high bit would get lost going from an 8 bit code to 7 bit ASCII.

A translation from 8-bit EBCDIC to 7-bit ASCII should have
converted the EBCDIC "Z390" correctly. And then reversed
it correctly had the reverse been done.

But yeah, 8-bit ASCII to 7-bit ASCII may have simply chopped
off the high bit.

> I agree that at least 2 things seem to be going on.  No idea
> what PKZIP would do to the data.

PKZIP preserves data, so that's not the culprit. That is
where the (zip) file's life starts.

BFN. Paul.