Since weekend update, small files up to 4096 bytes get corrupted
Andy Schmidt
I have two different servers (different models and manufacturing years), but
both of them running Windows 2000 server with software mirrored NTFS drives,
which suddenly corrupt small image thumbnail files after an hour or two.
No larger file sizes seem to be effected. The file content is set to all
binary 0xdf.
If we replace those files with valid copies, then those files will appear to
be correct for a while (probably read back from cache) but after some time
they will "revert" back to all 0xdf.
I have since been in contact with another customer who has the exact same
problem with his Windows 2000 server also since around the past weekend. In
the days prior we did install recent security fixes and Microsoft product
updates that were released in the past month.
Is there anyone else out there with the same problem?
Pegasus (MVP)
if you wish to address several newsgroups. Multiposting wastes
everybody's time.
"Andy Schmidt" <AndyS...@discussions.microsoft.com> wrote in message
news:D826AFE9-2AB1-4B2F...@microsoft.com...
KeithMail
week.
For me, Its KB920958 and it can be forced to happen if the data folder is
compressed (with explorer) and trigger it to happen immediately with a
chkdsk c: /f and a reboot
otherwise you have to wait for a while for it to go bad. We assume the
compressoin is done by a LAZY process and sin't immediate.
I don't think its the length as once the file is corrupt I have edited the
contents to other values and replaced it without a problem, there must be a
trigger byte sequence or something.
If you look at the file with the 2000 resource kit dskprobe sector editor it
complains about the length.
Are you using compressed folders for yourr data cos I can't get it to fail
otherwise and that seems to be my key trigger.
We have disabled antivirus and gone through hoops to prove the point.
Keith
"Pegasus (MVP)" <I....@fly.com> wrote in message
news:urNFVhyx...@TK2MSFTNGP03.phx.gbl...
Andy Schmidt
Let's please try to compare commonalities:
- Yes, on both servers those folders and parent folders have the compression
flag on.
- On both effected servers, the disks are defined as "dynamic"
- They are software mirrored (using Windows 2000' Disk Management)
- On both servers the files show all 0xDF characters (which shows up as the
German special 's', looks a little like a B or a beta character.)
- We run NetShield 4.5 with the 5.1.00 scan engine
How does that compare with your environment?
Andy
at Argos.net
Andy Schmidt
>> For me, Its KB920958 <<
Does that mean you have removed this Hotfix and your problems disappeared?
KeithMail
win2000server and Pro across differing machine manufacturers so I have
pretty well eliminated hard disk controllers, device drivers etc.
Simply put if a folder is compressed and one of these (<4096 bytes) files is
dropped into the compressed folder, then either after a period of time
(hence my question of a "lazy" compression process) or by forcing with a
chkdsk it will be corrupt immediately you reboot - this saves time waiting!
I do have mirrored disks on the 2000 server but just standard basic disks on
my 2000pros
If the Kb920958 is removed, then no amount of file placement and chkdsk will
cause the problem, all files already corrupt stay corrupt but no new damage
occurs.
If I confirm my compressed folder is good and then re add the kb920958 ,
followed by chkdsk /f etc the file corruption process occurs again.
I have checked the files with 3 hex editors (Cygnus, win2kresource kit
dskprobe and our own company proprietary viewer)
The problem as far as I can see is
KB920958
With Compressed folders
and only CERTAIN files smaller than 4096 bytes (interesting match to cluster
size perhaps?) I have dropped in text files of a k or so and they seem to be
impervious.
I have disabled all AV (was sophos), and other adaware checkers etc (that's
the beauty of being in a test lab I can pull systems apart as I see fit)
The most difficult question that our Msoft security case engineer wants
answering is what other file types are affected. Until you popped up I was
dealing with one of our own companies proprietary data storage types which
should, as far as the O/S is concerned, be just a binary file.
Once a file has gone bad then it never seems to "comes back", if I hexedit
the 0xDF file content to another hex value the bad file doesn't seem to get
made bad again (go back to 0xdf). This is why I was looking for a string of
bytes as a trigger. Still , I don't have access to Msoft O/s Code and that's
why I hoped raising a "case" with them would get it fixed.
In answer to your other post (17:16) then
At the moment, the solution for me is to
Take off the kb920958 patch
Go through storage and restore all bad files
Wait for a fix.
If the KB is necessary for you then I have not found corruption on a non
compressed folder so maybe the option would be to remove compression, then
restore the bad files
In essence if you need the 920958 then don't enable compression.
Keith
"Andy Schmidt" <AndyS...@discussions.microsoft.com> wrote in message
news:048B427E-25F1-471A...@microsoft.com...
Andy Schmidt
okay, I will attempt to uninstall Kb920958 tonight! That's the best lead yet.
I will also test the "compression" theory as soon as one of my customers
reports another failure with one of their files.
I definitely have seen at LEAST two files that are larger than 4096.
However, both are very close to a MULTIPLE of 4096 (close to 8K in one case,
in the other case 167,564 bytes = 163K). I inspected the files. In each case
the beginning of the files are good, and ONLY the last 4000 or so bytes have
the 0xDF byte pattern.
I too created a small TXT file (just one line) and it never corrupted, while
other files around it repeatedly did.
I can say, that I have seen corruptions with the following file types:
.JPG
.GIF
.SWF
where the vast majority are small .JPG and .GIF thumbnails and so far I've
only found few files that are larger. But that may be deceiving, because you
simply may not notice the end of a larger file being corrupted, because the
it "starts" out correctly!
Best Regards,
Andy
Heimir
Its has not been more then an hour so its too early to tell.
We are having problems with jpg files.
Small thumbnails that we create is being corrupted.
Heimir
Andy Schmidt
you were "right on". Disabling compression on certain folders seem to fix
it temporarily - and after uninstalling the hotfix last night on all servers
with compressed disks I have had no more complaints thus far.
Thank you for sharing your insight and for researching this so thoroughly.
Let's see how long it takes Microsoft to take your bug report and
proactively warn its customers that their hotfix can cause loss of data!
Best Regards,
Andy
Heimir
Heimir
John Linn
have to turn that off?
-jl
KeithMail
removed the patch it re downloaded it and notified me that it was there and
waiting to be applied so I assume the answer is yes to your question.
It is always a difficult compromise in an IT environment, do you want
platform stability or up to date security, its a pity this has shown you
can't always have both.
As of this date I have still NO confirmation from the patch authors to
acknowledge this as happening.
So, either we are imagining this problem or this has bigger repercussions
than we have discovered ourselves?
"John Linn" <John...@discussions.microsoft.com> wrote in message
news:65474638-F17E-4299...@microsoft.com...
John Linn
have 2 production environment that house most of my clients' files and to
test I uncompressed one, and I uninstalled the patch from the other (which
subsequently auto applied the patch again, hehe). It is a shame that I must
choose between manual patching my application, and therefore violating my own
security policy, or to uncompress many gigs worth of data, costing me 20-30%
additional space now.
I have to believe the lack of acknowledgement is out of embarassment or fear
of the bug's effects. I have over 300 clients who upload images all the time
and I've personally counted around 800 images (especially thumbnails) that
kept getting corrupt during that week from Hell. Luckily, my business isn't
file storage or graphic files. But if it was, this bug could of been
catastrophic. And that's just a with a small business...
If this isn't a Sev 1 bug within MS's system, I would be completedly
shocked. My guess is the next round of patches will fix it and it will be
done very quietly.
-jl
frank
I'm posting over here as you requested. Right now I'm generating a list
of all the files which get altered through the copy operation (I
understand it can happen whenever *any* small file gets written or
altered?). I'll be able to give you a list of filetypes and filesize
limits as well. After that, I'll be trying to roll back the KB 920958
and see if that fixes my problem.
Thanks!
Frank
frank
about 2500 different file types, I found evidence of corruption in some
five thousand files, of about 150 different filetypes. That's about
seven percent of all the filetypes listed. Contrary to what was seen
earlier, I saw *no* 4k limit on file size.
The largest file corrupted was an mpg file, some 41 mb in size. I also
saw corruption in mp3's and other multimedia files (m4a, wav, etc.),
and even gzipped and zipped files. conversely, even though a lot of
jpegs were hit, out of all available jpegs, only about 3% were
corrupted.
Now I'll be rolling back that patch, and confirm if the problem goes
away for all cases.
JP
complaining intermittenly since 2006.08.09, that the zip files we're
sending are occasionally corrupted.
upon inspection, the bad versions are ending on 0xdfs. when i create
a hexdump of both the bad and good copy and then diff them:
---
> 0036000 dfdf dfdf dfdf dfdf dfdf dfdf dfdf dfdf
> *
> 0036ec0 dfdf dfdf dfdf
frank
problem.
lily
We are experiencing something very similar at my company, but it's our
customers that are having the problem, not us.
Our product is a Windows-based product that is currently being
supported back to Windows 98. One of the things we do is save
information (data, graphs, etc.) using OLE structured storage. We
compress some of the larger data streams using a third party package.
Since Sept 5, 2006 our Tech Support staff has received three corrupted
files which, upon further review by the Software Development team, show
the presence of 0xdf at the end of the compressed stream.
Some information about the occurrence and amount of 0xdf:
Corruption 1: Compressed data stream size - 214749 bytes
0xdf starts at byte 210944
Total 0xdf bytes - 3805
Corruption 2: Compressed data stream size - 901867 bytes
0xdf starts at byte 898048
Total 0xdf bytes - 3819
Corruption 3: Compressed data stream size - 98889 bytes
0xdf starts at byte 95232
Total 0xdf bytes - 3657
This is devastating to our customers because subsequent attempts by our
software to open the saved file fail because the "decompression"
algorithm fails.
We are currently gathering information from the affected customers to
see if hot fix KB920958 is a possibility.
Andy Schmidt
"We are aware the issue you are experiencing. A corresponding bugcheck
request is currently open, and the develop team is working on this issue.
However, the hotfix for this issue is not ready.
0xDF is the data pattern that NTFS returns when it has problem to decompress
the file (e.g.. the compression fragments are corrupted and can't be
decompressed). Based on my research, the actual raw data on the disk is not
changed, it shows as 0xDF because the system cannot decompress the file and
display the data correctly. So the corrupt is not permanent.
Further more, the issue only occurs on files which containing Hexadecimal
codes."
mrk...@gmail.com
Hexadecimal codes."
This must be why you (or others?) couldnt get a text file to be
corrupted.
Andy Schmidt
Although we only tried very small text files - so another theory was that it
was just too small since it seemed to always effect files close to 4096 bytes
or a multiple thereof. (Of course, that was "anecdotal evidence" and may or
may not have been a conincident.)
martin...@fh-bochum.de
> Hexadecimal codes."
If I have a look inside my files, all of them contain hexadecimal codes
:-)
It seems as if only non-compressible files with a (filesize modulo
4096) of about 3600 to 4095 became victims of the fatal bug. The list
of my damaged files contains mostly JPG-Files and archives. In detail
one 7z-files, eight exe-files (packed?), one bz2-file, five gz-files,
one tif-file, 116 .jpg-files and nine zip-files were partially
overwritten with the ßßßßßßß-sequence.
I cannot believe that the "raw" file on the disk is unharmed, as stated
by the microsoft employee above, for even Ubuntu Linux, when bootet
from CD at the same machine, read the same garbage out of the files as
Windows 2000 either with the patch deinstalled or installed.
Mark V
>> " Further more, the issue only occurs on files which containing
>> Hexadecimal codes."
>
> If I have a look inside my files, all of them contain
> hexadecimal codes
>:-)
Yep, a "good one".
> It seems as if only non-compressible files with a (filesize
> modulo 4096) of about 3600 to 4095 became victims of the fatal
> bug. The list of my damaged files contains mostly JPG-Files and
> archives. In detail one 7z-files, eight exe-files (packed?), one
> bz2-file, five gz-files, one tif-file, 116 .jpg-files and nine
> zip-files were partially overwritten with the ككككككك-sequence.
In my testing here I found no existing (pre-KB920958) files
suffered corruption within +C directories. Setting -C succeeded in
decompressing all those files without damaging any.
I tested with a thousand random GIF and JPG files copying them into
a test directory set +C. Approximately 12% became corrupted. ALL
of the files damaged were in a range of
<even cluster value here>.89
to
<even cluster value here>.99
(W2K NTFS default - 4096 bytes)
> I cannot believe that the "raw" file on the disk is unharmed, as
Neither can I (for new files).
> stated by the microsoft employee above, for even Ubuntu Linux,
> when bootet from CD at the same machine, read the same garbage
> out of the files as Windows 2000 either with the patch
> deinstalled or installed.
I suspect the files are corrupted (post KB920958) when copied into
a +C location. End of story after that.
Adam Piggott
Hash: SHA1
Mark V wrote:
> In microsoft.public.win2000.file_system wrote:
I have a compressed, corrupted binary file that is damaged on disk but is
OK in a backup (which is also compressed via NTFS). After removing KB920958
the original is still damaged if read as compressed or decompressed;
confirming that the raw data is indeed damaged.
Cheers,
Adam Piggott, Proprietor, Proactive Services (Computing).
http://www.proactiveservices.co.uk/
Please replace dot invalid with dot uk to email me.
Apply personally for PGP public key.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (MingW32)
iD8DBQFFCEyR7uRVdtPsXDkRArnZAJsEGIfMZYPLNwYQ1aMwbPdX/sdgBQCfUk50
5GqZ08qPDG5+ytb5RMy74XM=
=stBG
-----END PGP SIGNATURE-----
Mark V
=?Utf-8?B?QW5keSBTY2htaWR0?= wrote:
> Hi,
>
> I have two different servers (different models and manufacturing
> years), but both of them running Windows 2000 server with
> software mirrored NTFS drives, which suddenly corrupt small
> image thumbnail files after an hour or two.
[ ]
Sep 15, 2006
http://support.microsoft.com/kb/920958 (MS06-049)
Article ID : 920958
Last Review : September 15, 2006
Revision : 2.0
MS has updated the article, to acknowledge the problem with
potential file corruption.
While the KB920958 file has not yet been re-released, MS is
offering a Hot Fix as detailed in
http://support.microsoft.com/kb/925308
==========================================
To resolve this problem immediately, contact Microsoft Product
Support Services to obtain the hotfix. For a complete list of
Microsoft Product Support Services phone numbers and information
about support costs, visit the following Microsoft Web site:
==========================================
US anyway: 1-800-936-4900, opt. 1 and
"Article ID : 925308"
"Compressed files that are larger than 4 kilobytes may be
corrupted when you create or update the files"
"Windows 2000"
Etc.
No charge.
Note that the original fix, Windows2000-KB920958-x86-ENU.EXE
will be updated eventually. This Hot Fix is subject to further
testing and so on...
Mark V
> In microsoft.public.win2000.file_system
> =?Utf-8?B?QW5keSBTY2htaWR0?= wrote:
>
>> Hi,
>>
>> I have two different servers (different models and manufacturing
>> years), but both of them running Windows 2000 server with
>> software mirrored NTFS drives, which suddenly corrupt small
>> image thumbnail files after an hour or two.
> [ ]
> Sep 15, 2006
>
> http://support.microsoft.com/kb/920958 (MS06-049)
>
> Article ID : 920958
> Last Review : September 15, 2006
> Revision : 2.0
>
> MS has updated the article, to acknowledge the problem with
> potential file corruption.
>
> While the KB920958 file has not yet been re-released, MS is
> offering a Hot Fix as detailed in
> http://support.microsoft.com/kb/925308
> ==========================================
> To resolve this problem immediately, contact Microsoft Product
> Support Services to obtain the hotfix. For a complete list of
> Microsoft Product Support Services phone numbers and information
> about support costs, visit the following Microsoft Web site:
> ==========================================
[ ]
Following up:
The HotFix (HOT-FIX_287398_ENU_i386_zip.exe for
Windows2000-KB925308-x86-ENU.EXE) appears here to have entirely
resolved the file corruption issue. No test files copied into, out
of, or edited in-place (in +C location) suffered any damage
whatsoever. "Test files" includes files known to corrupt pre-
HotFix. Recall that in all probability any files damaged pre-
HotFix will remain damaged (not specifically tested here).
Installed HotFix on W2K Pro, NTFS, SP4+ and without uninstalling
Windows2000-KB920958-x86-ENU.EXE (MS06-049)
FWIW and YMMV
Marc Kupper
> offering a Hot Fix as detailed in
> http://support.microsoft.com/kb/925308
As a FYI - I have written a scanner that will list files that seem to
be corrupted by KB920958. It's available from
http://marc.kupper.googlepages.com/scandf
This utility can't repair your files though it's possible enough
information was written to disk that Microsoft will be able to issue an
update that allows for reading the original file data.
Marc
Mark V
>> While the KB920958 file has not yet been re-released, MS is
>> offering a Hot Fix as detailed in
>> http://support.microsoft.com/kb/925308
>
> As a FYI - I have written a scanner that will list files that
> seem to be corrupted by KB920958. It's available from
> http://marc.kupper.googlepages.com/scandf
In case it is not otherwise known, that mentioned HotFix was publicly
released (same files) on the 20th as
KB-925308
http://support.microsoft.com/kb/925308/
> This utility can't repair your files though it's possible enough
> information was written to disk that Microsoft will be able to
> issue an update that allows for reading the original file data.
I doubt that is possible. Damaged files were literally stored in
that condition in the file system...data irreparably lost, restore
from clean backup. :(
Marc Kupper
> > ... it's possible enough
> > information was written to disk that Microsoft will be able to
> > issue an update that allows for reading the original file data.
Mark V replied:
> I doubt that is possible. Damaged files were literally stored in
> that condition in the file system...data irreparably lost, restore
> from clean backup. :(
I'd tend to agree and late last night had another thought in that it's
possible a data block for the last block was not allocated/written to
at all. I'm basing this thought on that it's always the last block
that's lost plus a comment on slashdot iirc where someone got a file
size mismatch error when they tried to load up a corrupted file using
some sort of raw disk editor (where they'd see whatever is actually
written to disk).
One of the first things I did though when I realized files were getting
corrupted was to run chkdsk which did not find any problems.
Initially I had been thinking that *something* was written to disk but
that its structure was not recognized by the decompression code which
then spun out the DF bytes in it's place.
Marc
Mark V
=?Utf-8?B?QW5keSBTY2htaWR0?= wrote:
> Hi,
>
> I have two different servers (different models and manufacturing
> years), but both of them running Windows 2000 server with
> software mirrored NTFS drives, which suddenly corrupt small
> image thumbnail files after an hour or two.
[ ]
Microsoft Security Bulletin MS06-049
Vulnerability in Windows Kernel Could Result in Elevation of
Privilege (920958)
Published: August 8, 2006 |
Updated: September 26, 2006
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
================== excerpt ======================
V2.0 (September 26, 2006): The update has been revised and re-
released for Microsoft Windows 2000 Service Pack 4 to address
issues identified in Microsoft Knowledge Base Article 920958.
=====================================================
http://www.microsoft.com/technet/security/Bulletin/MS06-049.mspx
This is the patch (v1) for W2K that caused corrupt files in on-disk
Compressed files.
First MS issued a HotFix for W2K 2006-09-15
Then issued those same files via KB-925308 2006-06-20
And finally they have incorporated the files
for W2K back into the "049" package for W2K.
W2K users should apply one of the "corrected files" packages if
they ever installed MS06-049, (920958) version 1.
W2K users that store Security pkg. files locally should update
their stored copy to "version 2".
Windows2000-KB920958-v2-x86-ENU.EXE 1,603,560 bytes
MD5: 09f6b7733eca534b2620549e1e184f7d
All three sources of "corrected files" contain the same 5 updated
files.
pavo5026
"Andy Schmidt" wrote:
> Hi,
>
> I have two different servers (different models and manufacturing years), but
> both of them running Windows 2000 server with software mirrored NTFS drives,
> which suddenly corrupt small image thumbnail files after an hour or two.
>