Serious BUG in MS Flash (FAL) ??
John Locke
WinCE 5.0 ARMV4I Platfrom, QFEs until April installed) and found out that
there seems to be a BUG in the FileSystem implementation.
The FileSystem works nicely as long as you don't write too much at a time.
But when doing batch copies of large amount of data there seems to be data
corruption.
I spent 4 days trying to figure out what the problem is. Here is what i
found:
- The Problem can be reproduced very easily by doing a Full (not quick)
format of the partition (will always fail and then only a complete store
reformat helps)
- This seems not to be a problem related to the Stratad (from intel), since
exactly the same problem happens when using the RAMFMD (a flash filesystem
emulation in RAM). I can only conclude that it's a problem in the FAL
- The Problem seems to be in the Compactor Thread. NOTE: the compactor
thread can be started in two ways... when the system is idle (in this case
it works) or during a write when dirty sectors need to be recoveded
(critical situation) to continue writing (in this case it will do something
wrong)
- The Critical Compactor thread seems to do everything right (i debugged all
the FMD functions it calls), it only seems it doesn't updates some internal
tables.
Here is a log of what happens:
.... (until almost the end of the partition nothing special happens)
0x83e23c80: FLASHDRV.DLL:WriteToMedia() - logicalSector 0x00000abb -->
physicalSector 0x00000ad2
0x83e23c80: FLASHDRV.DLL:WriteToMedia() - logicalSector 0x00000abc -->
physicalSector 0x00000ad3
0x83e23c80: FLASHDRV.DLL:L2P_GetPhysicalSectorAddr() - Secondary table
doesn't exist for logical sector 0xabc!!!
0x83f15400: FLASHDRV.DLL:------------ Starting Compaction... ------------
0x83f15400: Looking to recycle 8 DIRTY sectors into FREE
sectors
0x83f15400: >>> Media State Information: >>>
0x83f15400: Free Sectors = 126
0x83f15400: Dirty Sectors = 23
0x83f15400: FLASHDRV.DLL:------------------------------------------------
0x83f15400: FLASHDRV.DLL:CompactorThread() - Compacting BLOCK 1
0x83f15400: FLASHDRV.DLL >>> Physical sector 0x0000003f is MAPPED
0x83f15400: FLASHDRV.DLL >>> Moving data from physical sector 0x0000003f
to physical sector 0x00000ad4
... (lots more of MAPPED sectors)
0x83f15400: FLASHDRV.DLL >>> Physical sector 0x0000007d is MAPPED
0x83f15400: FLASHDRV.DLL >>> Moving data from physical sector 0x0000007d
to physical sector 0x00000b12
0x83f15400: FLASHDRV.DLL:CompactorThread() - Compacting BLOCK 0
0x83f15400: FLASHDRV.DLL >>> Physical sector 0x00000000 is DIRTY
... (6 more DIRTY sectors)
0x83f15400: FLASHDRV.DLL >>> Physical sector 0x00000007 is DIRTY
0x83f15400: FLASHDRV.DLL >>> Physical sector 0x00000008 is MAPPED
0x83f15400: FLASHDRV.DLL >>> Moving data from physical sector 0x00000008
to physical sector 0x00000b13
0x83f15400: FLASHDRV.DLL >>> Physical sector 0x00000009 is DIRTY
0x83f15400: FLASHDRV.DLL >>> Physical sector 0x0000000a is MAPPED
0x83f15400: FLASHDRV.DLL >>> Moving data from physical sector 0x0000000a
to physical sector 0x00000b14
0x83f15400: FLASHDRV.DLL >>> Physical sector 0x0000000b is DIRTY
... (12 more DIRTY sectors)
0x83f15400: FLASHDRV.DLL >>> Physical sector 0x00000018 is DIRTY
0x83f15400: FLASHDRV.DLL >>> Physical sector 0x00000019 is MAPPED
0x83f15400: FLASHDRV.DLL >>> Moving data from physical sector 0x00000019
to physical sector 0x00000b15
... (lots more of MAPPED sectors)
0x83f15400: FLASHDRV.DLL >>> Physical sector 0x0000003e is MAPPED
0x83f15400: FLASHDRV.DLL >>> Moving data from physical sector 0x0000003e
to physical sector 0x00000b3a
0x83f15400: FLASHDRV.DLL >>> Compaction process complete: 23 DIRTY sectors
recycled
0x83e23c80: FLASHDRV.DLL:WriteToMedia() - logicalSector 0x00000abd -->
physicalSector 0x00000b3b
... (21 more Sector Writes)
0x83e23c80: FLASHDRV.DLL:WriteToMedia() - logicalSector 0x00000ad3 -->
physicalSector 0x00000b51
0x83e23c80: FLASHDRV.DLL:DSK_IOControl(DISK_IOCTL_WRITE)
0x83e23c80: FLASHDRV.DLL:Write() request.
0x83e23c80: Start Sector = 2
0x83e23c80: # of Sectors = 1
0x83e23c80: # of SG bufs = 1
0x83e23c80: BUFF LEN = 4096
0x83e23c80: FLASHDRV.DLL:CP_StartCompactor() - There aren't any DIRTY
sectors left; the compactor can't be started
0x83e23c80: FLASHDRV.DLL:SM_GetNextFreeSector() - Unable to start compactor
in critical situation!!!
0x83e23c80: FLASHDRV.DLL:SM_GetNextFreeSector() - Unable to reclaim any free
sectors in a critical compaction stage. Media must be full.
0x83e23c80: FLASHDRV.DLL:WriteToMedia() - Unable to get next free physical
sector address for writing! The media is full...
0x83e23c80: FLASHDRV.DLL:WriteToMedia() failed.
0x83e23c80: Unknown: DEBUGCHK failed in file
c:\macallan\public\common\oak\drivers\fsd\fatutil\main\diskinterface.cpp at
line 44
Has anybody else noticed this strange behavior?
Can someone at Microsoft take a look at this issue?
Unfortunately there is no source code of the compactor or the fal so i'm
quite stucked at the moment.
We have many customers using this FlashFileSystems and some are already
experiencing Data Corruption problems, so this is a quite urgent issue.
Hope that someone can help me solve this problem.
Thanks in advance,
--
John
Steve Maillet (eMVP)
2) have you contacted MS support? (If not you should as that's the only way
to get a new QFE/hotfix issued to resolve the problem)
--
Steve Maillet
EmbeddedFusion
www.EmbeddedFusion.com
smaillet at EmbeddedFusion dot com
John Locke
1) Yes, I have all QFEs installed (Up to April 2006). I also checked the
changelog, but i didn't see any reference to FAL updates.
2) I was hoping to get some fast replies from this group. I will also try
the direct MS support path.
regards,
--
John
voidcoder
get any extra info from support.
"John Locke" <John...@nospam.com> wrote in message news:Fyxcg.2808$lk7....@fe04.news.easynews.com...
Mark Moeller
CE 5. I believe it has to do with the fact that format writes every
sector in a partition leaving no sectors free for the flash management
algorithms during update. When we looked at this issue we put it aside
because it would only happen on a full format. If memory serves, we
found that if you created your partition just a block smaller than your
available flash size then the problem would go away.
Mark Moeller
Qualnetics
John Locke
Yes, I found out the same thing: I'm also partitioning the FlashStore a few
sectors smaller as a Workaround now and the Full-Format works.
But still the problem is not really solved. For example when you do a
ScanDisk the after copying some hundreds files on the FlashFileSystem it
will ALWAYS find invalid files and lost clusters, and the weird thing is
that doing the scan once more will still find some other lost clusters, just
less. Repeating the procedure many times leads to Total flash corruption.
I will try your suggestion and leave a whole flash block unused, maybe this
fixes the problem completely (i did only reserve ~1/2 block).
Another note: The FAL already reserves 2 Flash Blocks for Compaction
purposes (they cannot be used in the partition). So I think giving the FAL
more unused sectors only delays the occurrence of the problem, it does not
really sove it.
BTW: I'm trying to get to the MS Technical support. In my Platform Builder
5.0 "Quick Start Guide" Booklet it states that 2 Free incidents are
included, but when Veryfying the PID it tells me that no free support is
possible.. very odd. Do I have really have to pay to get this issue solved?
regards
--
John
Steve Maillet (eMVP)
>5.0 "Quick Start Guide" Booklet it states that 2 Free incidents are
>included, but when Veryfying the PID it tells me that no free support is
>possible.. very odd. Do I have really have to pay to get this issue solved?
PID that will work. I posted details on this issue a couple of months ago.
However, even if you pay for the incident it will be refunded if it's a bug.
Surely you've already used up more in terms of your time than it costs to
get a paid support incident so even without the free incidents it's worth
calling on a bug like this.
John Locke
I will contact MS to get an updated PID.
In the meantime I bought an incident ticket.. let's hope it's going to be
processed fast.
I found out some other interesting points some hours ago:
- If I increse the Critical Compation Thread Priority
(CompactionCritPrio256) to something like 20 the problem SEEMS to go away.
- I say "SEEMS" because even if i did not see any data corruption or write
command failing i always get Lost Clusters after multiple write operations.
After scanning with ScanDisk the FileSystem is corrupt and cannot be written
anymore (Store Format needed), But if i never do a Scandisk the files can be
read back and they seem to be intact even after several overwrte cycles.
--
John
"Steve Maillet (eMVP)" <nos...@EntelechyConsulting.com> wrote in message
news:uPfd6Ynf...@TK2MSFTNGP03.phx.gbl...