Re: let's discuss about the file system and nand flash driver

[Xiang Xiao]

On Tue, Mar 10, 2020 at 11:48 PM wangxi...@163.com
<wangxi...@163.com> wrote:
>
> hello
>
> I want to discuss about the nand flash and filesystem support. I turned into nuttx because
> I have a linux background and RTOS like freertos is most likely a bare OS kernel ,porting
> exist things is annoying such as filesystem and POSIX interface etc.
> Some introduction of my situation and motivation:
> I'm trying to do porting the Nuttx onto the alientek apollo STM32F429 board.
> It's their website:
> http://www.alientek.com
> Their development board is very hot here in China, but it's a pity the website is in Chinese Language only.
> I worked based onstm32f429i-disco and what I have done:
> a.uart
> b.external sdram
> c.led
> all works fine. Next step I want to get the Nand flash working with some filesystem. Nand flash is Micron
> MT29F4G08.I realized that nuttx have no filesystem for nand on the shelf and the exist filesystem have to
> do some changes to fit nand flash.I want a more mature filesystem which I can think of from linux
> is something based on mtd like jiffs2 or YAFFS2.Maybe ubifs is better but It's not based on the mtd and more complex.

What's issue make you can't use SmartFS/LittleFS/SPIFFS directly on
MT29F4G08? From my experience, there isn't much difference between
NAND flash and NOR flash if the hardware can do ECC correction
automatically.

> I choosed YAFFS2 In fact the porting to Nuttx is pretty easy after I decided to port YAFFS2 with its direct mode.
> Next step I configured the FMC and got the FMC talk to nand correctly. Later I choose to use the driver file
> mtd_nand.c later I find I have to implement the method in the nand_raw.h for the lowest talking to nand in
> order to get all filesystem software stack working. I have a pause here and looking forward to your advice
> for such topics:
> a. Could I benefit from Linux as I don't want to reinvent the wheel ? what can I do to minium the effort ? Can we have a better sturcture?
> b. Should I put the raw_nand method drivers/mtd/ or the board specific folder?I think I should put in the mtd folder for micron specific

Put it to arch/../src/... if it's specific for NAND controller on SoC,
or put it to drivers/mtd/ if it's specific to external NAND flash
vendor.

> c. license of YAFFS2 is GPL which would pollute the nuttx BSD license, what is your advice? I have to link it with library format ?

GPL code can't upstream. It's fine to use it in your project, but you
need public all your source code per GPL requirement. If you don't
want to public your own code, nuttx/fs/userfs could help you to
isolate GPL code from BSD code.

> d. does someone have other working solutions for nand flash filesystem ?

SmartFS/LittleFS/SPIFFS/NXFFS is designed for the flash storage in
mind, all of them is bultin supported by NuttX, why not try they
first?

> e. As I understand, the raw nand filesystem is always a big headache for many rtos, if you don't choose linux you only have very
> limited options for free use, FAT is a obvious option but you have to working on the FTL level by yourself. It is meaningful to
> choose raw nand for low cost electronic product other than sd-card competitable storgage.
>

> What's your opinion ?
>
>
>
> Best
>

[Gregory Nutt]

> What's issue make you can't use SmartFS/LittleFS/SPIFFS directly on
> MT29F4G08? From my experience, there isn't much difference between
> NAND flash and NOR flash if the hardware can do ECC correction
> automatically.

None of those can be used with NAND.  All of them assume that they can
write individual bytes in the '1' state to the '0'  All of them will
modify the FLASH in that way.  That then breaks the ECC.  With NAND, you
can only write a whole page with all of the ECC bits in place.  You
cannot modify the FLASH page after that because it will break the ECC.

All of those file systems, I believe, will modify FLASH in that way and,
hence, cannot be used with NAND.

>
>> d. does someone have other working solutions for nand flash filesystem ?
> SmartFS/LittleFS/SPIFFS/NXFFS is designed for the flash storage in
> mind, all of them is bultin supported by NuttX, why not try they
> first?

I don't know about LittleFS, but I guarantee that SmartFS, SPIFF, and
NXFFS will not work on NAND.  It took me a day of debugging to
understand why.  SmartFS, modifies wear leveling bits in FLASH and
cannot be used with NAND.  I have tried NXFFS and it also modifies
bits.  SPIFFS does as well, I believe.

It takes a very special file system (or, better, a very special FTL
layer) to make NAND behave like NOR FLASH.  NAND is a very different
created due to ECC and sparing.

[Gregory Nutt]

What's issue make you can't use SmartFS/LittleFS/SPIFFS directly on
MT29F4G08? From my experience, there isn't much difference between
NAND flash and NOR flash if the hardware can do ECC correction
automatically.

None of those can be used with NAND.  All of them assume that they can write individual bytes in the '1' state to the '0'  All of them will modify the FLASH in that way.  That then breaks the ECC.  With NAND, you can only write a whole page with all of the ECC bits in place.  You cannot modify the FLASH page after that because it will break the ECC.

All of those file systems, I believe, will modify FLASH in that way and, hence, cannot be used with NAND.

d. does someone have other working solutions for nand flash filesystem ?

SmartFS/LittleFS/SPIFFS/NXFFS is designed for the flash storage in
mind, all of them is bultin supported by NuttX, why not try they
first?

I don't know about LittleFS, but I guarantee that SmartFS, SPIFF, and NXFFS will not work on NAND.  It took me a day of debugging to understand why.  SmartFS, modifies wear leveling bits in FLASH and cannot be used with NAND.  I have tried NXFFS and it

[Adam Feuer]

Hi,

NetBSD has a NAND flash file system (similar to JFFS2) that is BSD Licensed:

https://en.wikipedia.org/wiki/CHFS

https://wiki.netbsd.org/tutorials/using_the_nand_emulator_and_chfs/

NetBSD CHFS source code on GitHub

Because of the license, might this be a better place to start?

(I found this on the Wikipedia List of Flash File Systems.)

cheers

adam

On Tue, Mar 10, 2020 at 8:48 AM wangxi...@163.com <wangxi...@163.com> wrote:

hello

I want to discuss about the nand flash and filesystem support.  I turned into nuttx because
I have a linux background and RTOS like freertos is most likely a bare OS kernel ,porting
exist things is annoying such as filesystem and POSIX interface etc. 
Some introduction of my situation and motivation:
I'm trying to do porting the Nuttx onto the alientek apollo STM32F429 board.
It's their website:
http://www.alientek.com
Their development board is very hot here in China, but it's a pity the website is in Chinese Language only.
I worked based onstm32f429i-disco and what I have done:
a.uart
b.external sdram
c.led
all works fine. Next step I want to get the Nand flash working with some filesystem. Nand flash is Micron
MT29F4G08.I realized that nuttx have no filesystem for nand on the shelf and the exist filesystem have to
do some changes to fit nand flash.I want a more mature filesystem which I can think of from linux
is something  based on mtd like jiffs2 or YAFFS2.Maybe ubifs is better but It's not based on the mtd and more complex.

I choosed YAFFS2 In fact the porting to Nuttx is pretty easy after I decided to port YAFFS2 with its direct mode.
Next step I  configured the FMC and got the FMC talk to nand correctly. Later I choose to use the driver file
mtd_nand.c later I find I have to implement the method in the nand_raw.h for the lowest talking to nand in
order to get all filesystem software stack working. I have a pause here and looking forward to your advice
for such topics:
a. Could I benefit from Linux as I don't want to reinvent the wheel ? what can I do to minium the effort ? Can we have a better sturcture?
b. Should I put the raw_nand method drivers/mtd/ or the board specific folder?I think I should put in the mtd folder for micron specific

c. license of YAFFS2 is GPL which would pollute the nuttx BSD license, what is your advice? I have to link it with library format ?

d. does someone have other working solutions for nand flash filesystem ?

e. As I understand, the raw nand filesystem is always a big headache for many rtos, if you don't choose linux you only have very
limited options for free use, FAT is a obvious option but you have to working on the FTL level by yourself. It is meaningful to
choose raw nand for low cost electronic product other than sd-card competitable storgage.

What's your opinion ? 



Best

--

Adam Feuer <ad...@starcat.io>

[Johnny Billquist]

And to make life even more miserable, NAND flash can only be read a
limited number of times before it needs to be rewritten. And you can
have bad blocks that are known to not be usable at all.

Very different from NOR flash in several ways.

Johnny

--
Johnny Billquist || "I'm on a bus
|| on a psychedelic trip
email: b...@softjar.se || Reading murder books
pdp is alive! || tryin' to stay hip" - B. Idol

[Gregory Nutt]

> And to make life even more miserable, NAND flash can only be read a
> limited number of times before it needs to be rewritten. And you can
> have bad blocks that are known to not be usable at all.
>
> Very different from NOR flash in several ways.

It looks like the FTL mention in a previous email attempts to address
all of these issues: https://github.com/dlbeer/dhara

[Xiang Xiao]

On Wed, Mar 11, 2020 at 6:19 AM Gregory Nutt <spud...@gmail.com> wrote:
>
>
> What's issue make you can't use SmartFS/LittleFS/SPIFFS directly on
> MT29F4G08? From my experience, there isn't much difference between
> NAND flash and NOR flash if the hardware can do ECC correction
> automatically.
>
> None of those can be used with NAND. All of them assume that they can write individual bytes in the '1' state to the '0' All of them will modify the FLASH in that way. That then breaks the ECC. With NAND, you can only write a whole page with all of the ECC bits in place. You cannot modify the FLASH page after that because it will break the ECC.
>
> All of those file systems, I believe, will modify FLASH in that way and, hence, cannot be used with NAND.
>
>
> d. does someone have other working solutions for nand flash filesystem ?
>
> SmartFS/LittleFS/SPIFFS/NXFFS is designed for the flash storage in
> mind, all of them is bultin supported by NuttX, why not try they
> first?
>
> I don't know about LittleFS, but I guarantee that SmartFS, SPIFF, and NXFFS will not work on NAND. It took me a day of debugging to understand why. SmartFS, modifies wear leveling bits in FLASH and cannot be used with NAND. I have tried NXFFS and it
>

Ok, I just port LittleFS which never do the byte level modification,
forget my suggestion about other filesystem.
The good news is that LittleFS work very well with NAND flash without
any modification from the below link:
https://github.com/ARMmbed/littlefs/issues/11

>
> e. As I understand, the raw nand filesystem is always a big headache for many rtos, if you don't choose linux you only have very
> limited options for free use, FAT is a obvious option but you have to working on the FTL level by yourself. It is meaningful to
> choose raw nand for low cost electronic product other than sd-card competitable storgage.
>
> What's your opinion ?
>
>
>
> Best
>
>

> --
> You received this message because you are subscribed to the Google Groups "NuttX" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to nuttx+un...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/nuttx/CALEqrb55%3DtKJ%2Bo94Tc3U_7VpHtNP5366MizXZxQvMhbDmUStfg%40mail.gmail.com.

[Gregory Nutt]

> Ok, I just port LittleFS which never do the byte level modification,
> forget my suggestion about other filesystem.
> The good news is that LittleFS work very well with NAND flash without
> any modification from the below link:
> https://github.com/ARMmbed/littlefs/issues/11

But of course it would not be useful for any serious product development
unless it also supported bad block detection, sparing, and wear
leveling.  Those are all required for NAND use.

[Xiang Xiao]

LittleFS is designed for flash in mind, and has been porting to many
RTOS. The author write a set of comprehensive documents:
https://github.com/ARMmbed/littlefs/blob/master/README.md
https://github.com/ARMmbed/littlefs/blob/master/DESIGN.md
https://github.com/ARMmbed/littlefs/blob/master/SPEC.md

> --
> You received this message because you are subscribed to the Google Groups "NuttX" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to nuttx+un...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/nuttx/1f9254c9-1199-42e8-2d2c-472eb9a41611%40gmail.com.

[Gregory Nutt]

>>> Ok, I just port LittleFS which never do the byte level modification,
>>> forget my suggestion about other filesystem.
>>> The good news is that LittleFS work very well with NAND flash without
>>> any modification from the below link:
>>> https://github.com/ARMmbed/littlefs/issues/11
>> But of course it would not be useful for any serious product development
>> unless it also supported bad block detection, sparing, and wear
>> leveling. Those are all required for NAND use.
>>
> LittleFS is designed for flash in mind, and has been porting to many
> RTOS. The author write a set of comprehensive documents:
> https://github.com/ARMmbed/littlefs/blob/master/README.md
> https://github.com/ARMmbed/littlefs/blob/master/DESIGN.md
> https://github.com/ARMmbed/littlefs/blob/master/SPEC.md

When people say FLASH they usually refer to standard NOR FLASH, or
Serial NOR FLASH.  NAND is a completely different thing with very
different interface requirements.

Sparing, bad block handling, and wear level is normally handled with a
NAND FTL, not by the file system.  Without these things, NAND has no use
in a product.

[Gregory Nutt]

Add error correction to that list of requirements.

[Gregory Nutt]

I see from the first reference that ArmMbed refers to the Dhara FTL as a related project.  THAT is what you need for NAND integration and was mentioned earlier in this thread:  "Dhara - An interesting NAND flash translation layer designed for small MCUs. It offers static wear-leveling and power-resilience with only a fixed O(|address|) pointer structure stored on each block and in RAM."  https://github.com/ARMmbed/littlefs/blob/master/README.md

With Dhara, NAND can be used with any file system.  Without an NAND FTL, NAND cannot be used at all.

There are few other references to NAND:

"littlefs by itself does not provide ECC." https://github.com/ARMmbed/littlefs/blob/master/DESIGN.md

Those two paragraphs above are the only to references in the documents you site.  On refers to Dhara which we should also be considering, the other just states that littlefs does NOT support ECC.  There no no other reference to NAND.

[Gregory Nutt]

Better information... basically LittleFS does not have NAND support and AFAIK it has never been used it with NAND.   Per https://github.com/ARMmbed/littlefs/issues/11, littlefs does appear to support bad block management and wear leveling.  But I quote:

LittleFS has only been used on NOR flash, eMMC, SD cards; devices with <4KB erase units, so I'm not sure how it will perform on NAND flash.

Concerns:

1. You must provide ECC

   LittleFS will hand write errors, which is the main effect of flash wear. But with NAND memory you also have to worry about read errors. NAND memory is so dense that the stored electrons start misbehaving pretty easily.

   Scanning the MT29F's datasheet, it looks like that part has built in ECC you can use.

2. On disk, each file uses 1 erase unit at minimum.

   For 128KiB erase units, this may waste a large amount of space.

   But NAND devices are also very large, so if you have few files this may not be a problem? I'm not sure. It probably depends a lot on how you use the filesystem.

   I have plans to improve this issue by storing multiple files in a block, but I don't know when I'll be able to implement the idea. So unfortunately at the moment 1 file per erase unit is the best LittleFS can do.

3. You may need 2KiB (writable unit) of RAM for the program cache?

   I'm not sure if you can take advantage of the 1 byte sequential writes.

   For each block, LittleFS does write the data out sequentially. But, it may partially write to multiple erase units multiple times (for example, partially write erase unit A, partially write erase unit B, finish writing erase unit A). Can the NAND chip handle that sort of pattern?

---

At the moment, I think LittleFS might need a flash translation layer (FTL) to be efficient. An FTL would provide a smaller erase unit and prevent all of the issues, though you may still want to use LittleFS for power resilience.

At the very least, LittleFS is probably better than putting FAT directly on top of raw NAND.

If you do try to put LittleFS on a NAND chip, let me know how it goes! I'd be interested to know what sort of problems you run into.

And

My hope is to avoid using a fully featured FTL layer such as dhara (https://github.com/dlbeer/dhara),

Greg

[Xiang Xiao]

Yes, I already note ECC issue in my first reply:

> What's issue make you can't use SmartFS/LittleFS/SPIFFS directly on
> MT29F4G08? From my experience, there isn't much difference between
> NAND flash and NOR flash if the hardware can do ECC correction
> automatically.

>
>

> --
> You received this message because you are subscribed to the Google Groups "NuttX" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to nuttx+un...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/nuttx/09ef6f19-7a9b-2cab-66a7-629180ce83f3%40gmail.com.

[Xiang Xiao]

On Wed, Mar 11, 2020 at 11:09 AM Gregory Nutt <spud...@gmail.com> wrote:
>
>
>
>
> Ok, I just port LittleFS which never do the byte level modification,
> forget my suggestion about other filesystem.
> The good news is that LittleFS work very well with NAND flash without
> any modification from the below link:
> https://github.com/ARMmbed/littlefs/issues/11
>
> But of course it would not be useful for any serious product development
> unless it also supported bad block detection, sparing, and wear
> leveling. Those are all required for NAND use.
>
> LittleFS is designed for flash in mind, and has been porting to many
> RTOS. The author write a set of comprehensive documents:
> https://github.com/ARMmbed/littlefs/blob/master/README.md
> https://github.com/ARMmbed/littlefs/blob/master/DESIGN.md
> https://github.com/ARMmbed/littlefs/blob/master/SPEC.md
>
>
> When people say FLASH they usually refer to standard NOR FLASH, or Serial NOR FLASH. NAND is a completely different thing with very different interface requirements.
>
> Sparing, bad block handling, and wear level is normally handled with a NAND FTL, not by the file system. Without these things, NAND has no use in a product.
>

NOR flash also need handle bad block and wear level too, the
filesystem should support these two features at least before annaunce
that it is a flash friendly filesystem. The real difference is that
NAND flash has the spare region to save ECC. The old device require
that the software caluclate/save ECC to the spare region and correct
the error bit from ECC, but most NAND flash vendor do ECC correction
in hardware without any software intervention today. The other
difference is that NAND flash normally has much bigger earse size than
NOR flash, filesystem has to do some special desgin to avoid the
storage waste and performance lose.

> Add error correction to that list of requirements.
>
> I see from the first reference that ArmMbed refers to the Dhara FTL as a related project. THAT is what you need for NAND integration and was mentioned earlier in this thread: "Dhara - An interesting NAND flash translation layer designed for small MCUs. It offers static wear-leveling and power-resilience with only a fixed O(|address|) pointer structure stored on each block and in RAM." https://github.com/ARMmbed/littlefs/blob/master/README.md
>
> With Dhara, NAND can be used with any file system. Without an NAND FTL, NAND cannot be used at all.
>
> There are few other references to NAND:
>
> "littlefs by itself does not provide ECC." https://github.com/ARMmbed/littlefs/blob/master/DESIGN.md
>

Yes, ECC need be handled by MTD layer or hardware.

> Those two paragraphs above are the only to references in the documents you site. On refers to Dhara which we should also be considering, the other just states that littlefs does NOT support ECC. There no no other reference to NAND.
>
> Better information... basically LittleFS does not have NAND support and AFAIK it has never been used it with NAND. Per https://github.com/ARMmbed/littlefs/issues/11, littlefs does appear to support bad block management and wear leveling. But I quote:
>
> LittleFS has only been used on NOR flash, eMMC, SD cards; devices with <4KB erase units, so I'm not sure how it will perform on NAND flash.
>

This issue is created more than two years, LittleFS make many
significant improvement(even break the disk layout), the description
in the issue mayn't reflect the current status.

> Concerns:
>
> You must provide ECC
>
> LittleFS will hand write errors, which is the main effect of flash wear. But with NAND memory you also have to worry about read errors. NAND memory is so dense that the stored electrons start misbehaving pretty easily.
>
> Scanning the MT29F's datasheet, it looks like that part has built in ECC you can use.
>
> On disk, each file uses 1 erase unit at minimum.
>
> For 128KiB erase units, this may waste a large amount of space.
>
> But NAND devices are also very large, so if you have few files this may not be a problem? I'm not sure. It probably depends a lot on how you use the filesystem.
>
> I have plans to improve this issue by storing multiple files in a block, but I don't know when I'll be able to implement the idea. So unfortunately at the moment 1 file per erase unit is the best LittleFS can do.
>

LittleFS support this feature in V2.

> You may need 2KiB (writable unit) of RAM for the program cache?
>
> I'm not sure if you can take advantage of the 1 byte sequential writes.
>
> For each block, LittleFS does write the data out sequentially. But, it may partially write to multiple erase units multiple times (for example, partially write erase unit A, partially write erase unit B, finish writing erase unit A). Can the NAND chip handle that sort of pattern?
>
> ________________________________
>
> At the moment, I think LittleFS might need a flash translation layer (FTL) to be efficient. An FTL would provide a smaller erase unit and prevent all of the issues, though you may still want to use LittleFS for power resilience.
>
> At the very least, LittleFS is probably better than putting FAT directly on top of raw NAND.
>
> If you do try to put LittleFS on a NAND chip, let me know how it goes! I'd be interested to know what sort of problems you run into.
>
> And
>
> My hope is to avoid using a fully featured FTL layer such as dhara (https://github.com/dlbeer/dhara),
>

Anyway, the author need compare the different filesystem and make the
decision by self carefully.

> Greg

>
> --
> You received this message because you are subscribed to the Google Groups "NuttX" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to nuttx+un...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/nuttx/332d249a-378a-7667-1f34-f97c77be0935%40gmail.com.

[2k4 w]

Thanks for your nice advice Xiang xiao. It seems that I missed some discussion due to some 

mail not received. I will consider the littlefs as my priority.

在 2020年3月11日星期三 UTC+8上午11:42:49，Xiang Xiao写道：

> To unsubscribe from this group and stop receiving emails from it, send an email to nu...@googlegroups.com.

[2k4 w]

Hi Greg

I noticed that you said the ONFI code is based on the SAMA5D3x from ATMEL, have you done the porting yet ? 

where can I find the source code ? 

I think grap some code from u-boot or linux is not a good choice due to the license and it is really annoying to add 

into current code.

wangxi...@163.com

在 2020年3月11日星期三 UTC+8上午11:09:05，patacongo写道：

[Gregory Nutt]

--

You received this message because you are subscribed to the Google Groups "NuttX" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nuttx+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/nuttx/0fb5d486-2431-4654-9c7b-3126e857f081%40googlegroups.com.

[Gregory Nutt]

> I noticed that you said the ONFI code is based on the SAMA5D3x from
> ATMEL, have you done the porting yet ?
> where can I find the source code ?
> I think grap some code from u-boot or linux is not a good choice due
> to the license and it is really annoying to add
> into current code.

The source code is in the AtmelStudio sample code.  The Atmel sample
code had a BSD-3 license.  We cannot use U-Boot or Linux.

You an see that code used in boards/arm/sama5 in all of the instances of
sam_nandflash.c