Memory crawler for eMMC partitions

[Ankur Tank]

We have BBB based custom board with 256MB RAM and 4GB eMMC running Linux kernel 3.12

TL;DR: Is there a open source application which periodically reads unread/rarely used partition to increase data retention? OR Is there any kernel service which can be configure

Detailed query,

We have 3 sets of partition for OS(Bootloader, kernel+dtb & RFS) eMMC. 

One of them is recovery OS. Which is rarely read( none of the time written).

Now as per eMMC manufacturerer,

"If a byte is not read in 5 years there is no guarantee that byte is valid when read after 5 years."

To avoid such situation we want to have a crawler application, which would read all the unused/unread partition may be once in a year and increase data retention.

I didn't find any open source application or kernel service/daemon which would handle that. But not sure If I am searching it correctly or not.

Meanwhile If there is no solution available for this, I am planning to do it in following way.

1. Setup a cron job which would trigger once in a year and run a script.

2. Script would read all the partition using dd and return.

What do you think about this approach.

Let me know if you have any suggestion/correction/pointers for me.

Thank you,

Ankur

[Robert Nelson]

debian@test-bbb-1:~$ time sudo cat /dev/mmcblk1 > /dev/null

real 1m20.209s

user 0m0.050s

sys 0m14.530s

That was for a 2GB eMMC, so double the time for a 4GB..

But in reality, with eMMC, we are still only talking with the controller, not directly with the NAND. So even a cat won't read every 'byte' as the controller is doing it's own thing...

Regards,

--

Robert Nelson
https://rcn-ee.com/

[Ankur Tank]

Thank you for reply Mr. Nelson,

That's the simplest solution I was also thinking.

Here are my timings.

# time cat /dev/mmcblk0 > /dev/null
real    1m 52.01s
user    0m 0.55s
sys     0m 21.74s

However I think I have to take care that not partitions are being erased/written at the time when I run this script, right ?

[Robert Nelson]

On Wed, Jul 20, 2016 at 9:56 AM, Ankur Tank <art...@gmail.com> wrote:

Thank you for reply Mr. Nelson,

That's the simplest solution I was also thinking.

Here are my timings.

# time cat /dev/mmcblk0 > /dev/null
real    1m 52.01s
user    0m 0.55s
sys     0m 21.74s

However I think I have to take care that not partitions are being erased/written at the time when I run this script, right ?

Why? other then your mmc io maxed out, with cat your just "reading" and dumping it to /dev/null...

Regards,

[Ankur Tank]

I am not sure If I understood it correctly,

But I assume that you are asking if we are using mmc lines for some other hardware or not, right ?

mmc lines are dedicatedly used for eMMC only.

But other scripts can be running in parallel to upgrade the OS in other partition to newer OS.

So I was thinking what will happen if write/erase and read happens simultaneously.

Thank you,

Ankur

[Ankur Tank]

Hello Mr. Nelson,

I wrote a simple memory crawler script as it was suggested here,

which had following simple code,

dd if="/dev/mmcblk0" of="/dev/null" bs="1024"

Now in parallel I ran another script which mounted one of the partition i.e. /dev/mmcblk0p15

Basically script did following,

1. Create files

2. Delete files

It resulted into following errors,

In some other post you had mentioned mmc driver was stable only in Linux 3.14 kernel, 

do you think below errors are related to mmc driver instability ?

2016-08-12T18:42:34.887796+05:30 kernel: [275054.380088] [<c0012d24>] (unwind_backtrace+0x0/0xf4) from [<c0011130>] (show_stack+0x10/0x14)
2016-08-12T18:42:34.887834+05:30 kernel: [275054.380122] [<c0011130>] (show_stack+0x10/0x14) from [<c0087548>] (warn_alloc_failed+0xe0/0x118)
2016-08-12T18:42:34.887872+05:30 kernel: [275054.380150] [<c0087548>] (warn_alloc_failed+0xe0/0x118) from [<c008a3ac>] (__alloc_pages_nodemask+0x74c/0x8f8)
2016-08-12T18:42:34.887913+05:30 kernel: [275054.380179] [<c008a3ac>] (__alloc_pages_nodemask+0x74c/0x8f8) from [<c00b2e8c>] (cache_alloc_refill+0x328/0x620)
2016-08-12T18:42:34.887953+05:30 kernel: [275054.380202] [<c00b2e8c>] (cache_alloc_refill+0x328/0x620) from [<c00b3224>] (__kmalloc+0xa0/0xe8)
2016-08-12T18:42:34.887993+05:30 kernel: [275054.380237] [<c00b3224>] (__kmalloc+0xa0/0xe8) from [<c0212904>] (edma_prep_slave_sg+0x84/0x388)
2016-08-12T18:42:34.888032+05:30 kernel: [275054.380272] [<c0212904>] (edma_prep_slave_sg+0x84/0x388) from [<c02eeb98>] (omap_hsmmc_request+0x414/0x508)
2016-08-12T18:42:34.888071+05:30 kernel: [275054.380298] [<c02eeb98>] (omap_hsmmc_request+0x414/0x508) from [<c02d9240>] (mmc_start_request+0xc4/0xe0)
2016-08-12T18:42:34.888109+05:30 kernel: [275054.380321] [<c02d9240>] (mmc_start_request+0xc4/0xe0) from [<c02da028>] (mmc_start_req+0x2d8/0x38c)
2016-08-12T18:42:34.888149+05:30 kernel: [275054.380341] [<c02da028>] (mmc_start_req+0x2d8/0x38c) from [<c02e748c>] (mmc_blk_issue_rw_rq+0x230/0x9d8)
2016-08-12T18:42:34.888188+05:30 kernel: [275054.380363] [<c02e748c>] (mmc_blk_issue_rw_rq+0x230/0x9d8) from [<c02e7dd8>] (mmc_blk_issue_rq+0x1a4/0x468)
2016-08-12T18:42:34.888227+05:30 kernel: [275054.380384] [<c02e7dd8>] (mmc_blk_issue_rq+0x1a4/0x468) from [<c02e8760>] (mmc_queue_thread+0x88/0x118)
2016-08-12T18:42:34.888265+05:30 kernel: [275054.380409] [<c02e8760>] (mmc_queue_thread+0x88/0x118) from [<c004d8b8>] (kthread+0xb4/0xb8)
2016-08-12T18:42:34.888302+05:30 kernel: [275054.380433] [<c004d8b8>] (kthread+0xb4/0xb8) from [<c000e298>] (ret_from_fork+0x14/0x3c)
2016-08-12T18:42:34.888332+05:30 kernel: [275054.380443] Mem-info:
2016-08-12T18:42:34.888362+05:30 kernel: [275054.380452] Normal per-cpu:
2016-08-12T18:42:34.888393+05:30 kernel: [275054.380462] CPU    0: hi:   90, btch:  15 usd:  85
2016-08-12T18:42:34.888424+05:30 kernel: [275054.380490] active_anon:8869 inactive_anon:22 isolated_anon:0
2016-08-12T18:42:34.888456+05:30 kernel: [275054.380490]  active_file:6717 inactive_file:33708 isolated_file:0
2016-08-12T18:42:34.888487+05:30 kernel: [275054.380490]  unevictable:422 dirty:7 writeback:0 unstable:0
2016-08-12T18:42:34.888520+05:30 kernel: [275054.380490]  free:2317 slab_reclaimable:8638 slab_unreclaimable:1099
2016-08-12T18:42:34.888551+05:30 kernel: [275054.380490]  mapped:1700 shmem:47 pagetables:214 bounce:0
2016-08-12T18:42:34.888581+05:30 kernel: [275054.380490]  free_cma:1566
2016-08-12T18:42:34.888638+05:30 kernel: [275054.380550] Normal free:9268kB min:2004kB low:2504kB high:3004kB active_anon:35476kB inactive_anon:88kB active_file:26868kB inactive_file:134832kB unevictable:1688kB isolated(anon):0kB isolated(file):0kB present:260096kB managed:251496kB mlocked:1688kB dirty:28kB writeback:0kB mapped:6800kB shmem:188kB slab_reclaimable:34552kB slab_unreclaimable:4396kB kernel_stack:1072kB pagetables:856kB unstable:0kB bounce:0kB free_cma:6264kB writeback_tmp:0kB pages_scanned:24 all_unreclaimable? no
2016-08-12T18:42:34.888671+05:30 kernel: [275054.380562] lowmem_reserve[]: 0 0 0
2016-08-12T18:42:34.888708+05:30 kernel: [275054.380580] Normal: 1053*4kB (UEMC) 570*8kB (UEMC) 31*16kB (UEM) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB = 9268kB
2016-08-12T18:42:34.888764+05:30 kernel: [275054.380631] 40818 total pagecache pages
2016-08-12T18:42:34.888797+05:30 kernel: [275054.380647] 0 pages in swap cache
2016-08-12T18:42:34.888830+05:30 kernel: [275054.380656] Swap cache stats: add 0, delete 0, find 0/0
2016-08-12T18:42:34.888861+05:30 kernel: [275054.380664] Free swap  = 0kB
2016-08-12T18:42:34.888890+05:30 kernel: [275054.380670] Total swap = 0kB
2016-08-12T18:42:34.888922+05:30 kernel: [275054.380682] SLAB: Unable to allocate memory on node 0 (gfp=0x20)
2016-08-12T18:42:34.889002+05:30 kernel: [275054.380693]   cache: kmalloc-32768, object size: 32768, order: 3
2016-08-12T18:42:34.889039+05:30 kernel: [275054.380705]   node 0: slabs: 3/3, objs: 3/3, free: 0
2016-08-12T18:42:34.889072+05:30 kernel: [275054.380727] omap_hsmmc 481d8000.mmc: prep_slave_sg() failed
2016-08-12T18:42:34.889111+05:30 kernel: [275054.380740] omap_hsmmc 481d8000.mmc: MMC start dma failure
2016-08-12T18:42:34.910595+05:30 kernel: [275054.387570] mmcblk0: unknown error -1 sending read/write command, card status 0x900
2016-08-12T18:42:34.910788+05:30 kernel: [275054.403229] mmcqd/1: page allocation failure: order:3, mode:0x200020
2016-08-12T18:42:34.910839+05:30 kernel: [275054.403260] CPU: 0 PID: 613 Comm: mmcqd/1 Tainted: P           O 3.12.11-005-ts-armv7l #3
2016-08-12T18:42:34.911525+05:30 kernel: [275054.403324] [<c0012d24>] (unwind_backtrace+0x0/0xf4) from [<c0011130>] (show_stack+0x10/0x14)
2016-08-12T18:42:34.911609+05:30 kernel: [275054.403358] [<c0011130>] (show_stack+0x10/0x14) from [<c0087548>] (warn_alloc_failed+0xe0/0x118)
2016-08-12T18:42:34.929396+05:30 kernel: [275054.403386] [<c0087548>] (warn_alloc_failed+0xe0/0x118) from [<c008a3ac>] (__alloc_pages_nodemask+0x74c/0x8f8)
2016-08-12T18:42:34.929455+05:30 kernel: [275054.403415] [<c008a3ac>] (__alloc_pages_nodemask+0x74c/0x8f8) from [<c00b2e8c>] (cache_alloc_refill+0x328/0x620)
2016-08-12T18:42:34.929497+05:30 kernel: [275054.403438] [<c00b2e8c>] (cache_alloc_refill+0x328/0x620) from [<c00b3224>] (__kmalloc+0xa0/0xe8)
2016-08-12T18:42:34.929538+05:30 kernel: [275054.403473] [<c00b3224>] (__kmalloc+0xa0/0xe8) from [<c0212904>] (edma_prep_slave_sg+0x84/0x388)
2016-08-12T18:42:34.929577+05:30 kernel: [275054.403508] [<c0212904>] (edma_prep_slave_sg+0x84/0x388) from [<c02eeb98>] (omap_hsmmc_request+0x414/0x508)
2016-08-12T18:42:34.929616+05:30 kernel: [275054.403534] [<c02eeb98>] (omap_hsmmc_request+0x414/0x508) from [<c02d9240>] (mmc_start_request+0xc4/0xe0)
2016-08-12T18:42:34.929655+05:30 kernel: [275054.403556] [<c02d9240>] (mmc_start_request+0xc4/0xe0) from [<c02da028>] (mmc_start_req+0x2d8/0x38c)
2016-08-12T18:42:34.929695+05:30 kernel: [275054.403578] [<c02da028>] (mmc_start_req+0x2d8/0x38c) from [<c02e748c>] (mmc_blk_issue_rw_rq+0x230/0x9d8)
2016-08-12T18:42:34.929735+05:30 kernel: [275054.403600] [<c02e748c>] (mmc_blk_issue_rw_rq+0x230/0x9d8) from [<c02e7dd8>] (mmc_blk_issue_rq+0x1a4/0x468)
2016-08-12T18:42:34.929773+05:30 kernel: [275054.403620] [<c02e7dd8>] (mmc_blk_issue_rq+0x1a4/0x468) from [<c02e8760>] (mmc_queue_thread+0x88/0x118)
2016-08-12T18:42:34.929812+05:30 kernel: [275054.403646] [<c02e8760>] (mmc_queue_thread+0x88/0x118) from [<c004d8b8>] (kthread+0xb4/0xb8)
2016-08-12T18:42:34.929849+05:30 kernel: [275054.403670] [<c004d8b8>] (kthread+0xb4/0xb8) from [<c000e298>] (ret_from_fork+0x14/0x3c)
2016-08-12T18:42:34.929880+05:30 kernel: [275054.403681] Mem-info:
2016-08-12T18:42:34.929909+05:30 kernel: [275054.403689] Normal per-cpu:
2016-08-12T18:42:34.929940+05:30 kernel: [275054.403700] CPU    0: hi:   90, btch:  15 usd:  86
2016-08-12T18:42:34.929972+05:30 kernel: [275054.403726] active_anon:8869 inactive_anon:22 isolated_anon:0
2016-08-12T18:42:34.930004+05:30 kernel: [275054.403726]  active_file:6723 inactive_file:33676 isolated_file:0
2016-08-12T18:42:34.930036+05:30 kernel: [275054.403726]  unevictable:422 dirty:11 writeback:0 unstable:0
2016-08-12T18:42:34.930068+05:30 kernel: [275054.403726]  free:2362 slab_reclaimable:8638 slab_unreclaimable:1081
2016-08-12T18:42:34.930101+05:30 kernel: [275054.403726]  mapped:1700 shmem:47 pagetables:214 bounce:0
2016-08-12T18:42:34.930130+05:30 kernel: [275054.403726]  free_cma:1566
2016-08-12T18:42:34.930188+05:30 kernel: [275054.403785] Normal free:9448kB min:2004kB low:2504kB high:3004kB active_anon:35476kB inactive_anon:88kB active_file:26892kB inactive_file:134704kB unevictable:1688kB isolated(anon):0kB isolated(file):0kB present:260096kB managed:251496kB mlocked:1688kB dirty:44kB writeback:0kB mapped:6800kB shmem:188kB slab_reclaimable:34552kB slab_unreclaimable:4324kB kernel_stack:1072kB pagetables:856kB unstable:0kB bounce:0kB free_cma:6264kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
2016-08-12T18:42:34.930221+05:30 kernel: [275054.403798] lowmem_reserve[]: 0 0 0
2016-08-12T18:42:34.930257+05:30 kernel: [275054.403816] Normal: 1080*4kB (UEMC) 573*8kB (UEMC) 34*16kB (UEM) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB = 9448kB
2016-08-12T18:42:34.930290+05:30 kernel: [275054.403866] 40790 total pagecache pages
2016-08-12T18:42:34.930320+05:30 kernel: [275054.403882] 0 pages in swap cache
2016-08-12T18:42:34.930351+05:30 kernel: [275054.403892] Swap cache stats: add 0, delete 0, find 0/0
2016-08-12T18:42:34.930380+05:30 kernel: [275054.403900] Free swap  = 0kB
2016-08-12T18:42:34.930410+05:30 kernel: [275054.403906] Total swap = 0kB
2016-08-12T18:42:34.930442+05:30 kernel: [275054.403917] SLAB: Unable to allocate memory on node 0 (gfp=0x20)
2016-08-12T18:42:34.930474+05:30 kernel: [275054.403928]   cache: kmalloc-32768, object size: 32768, order: 3
2016-08-12T18:42:34.930506+05:30 kernel: [275054.403941]   node 0: slabs: 3/3, objs: 3/3, free: 0
2016-08-12T18:42:34.930538+05:30 kernel: [275054.403963] omap_hsmmc 481d8000.mmc: prep_slave_sg() failed
2016-08-12T18:42:34.930569+05:30 kernel: [275054.403975] omap_hsmmc 481d8000.mmc: MMC start dma failure
2016-08-12T18:42:34.930648+05:30 kernel: [275054.407883] mmcblk0: unknown error -1 sending read/write command, card status 0x900
2016-08-12T18:42:34.993540+05:30 kernel: [275054.482212] Aborting journal on device mmcblk0p15-8.
2016-08-12T18:42:35.563557+05:30 kernel: [275055.042334] EXT4-fs error (device mmcblk0p15): ext4_journal_check_start:56: Detected aborted journal
2016-08-12T18:42:35.563724+05:30 kernel: [275055.052147] EXT4-fs (mmcblk0p15): Remounting filesystem read-only
2016-08-12T18:43:13.823579+05:30 kernel: [275093.308813] EXT4-fs error (device mmcblk0p15): ext4_put_super:772: Couldn't clean up the journal
2016-08-12T18:44:38.093907+05:30 kernel: [275177.581460] EXT4-fs warning (device mmcblk0p15): ext4_clear_journal_err:4567: Filesystem error recorded from previous mount: IO failure
2016-08-12T18:44:38.094070+05:30 kernel: [275177.581489] EXT4-fs warning (device mmcblk0p15): ext4_clear_journal_err:4568: Marking fs in need of filesystem check.
2016-08-12T18:44:38.094116+05:30 kernel: [275177.584824] EXT4-fs (mmcblk0p15): warning: mounting fs with errors, running e2fsck is recommended

On Wednesday, 20 July 2016 20:29:32 UTC+5:30, RobertCNelson wrote: