spiffs integration

[Yuriy]

Hi All,

First of all great job for such a nice framework!

I haven't found spiffs integration in esp-open-rtos and did some work on my own.

If anyone is interested github.

Comments/suggestions are very welcome.

I've written an example that writes bunch of files, then reads them and verifies that content is valid.

The test has been running for couple of days without failing (It will kill the flash eventually :)

Spiffs is configured in "singleton" mode. I don't think anybody needs more than one instance of FS on esp8266.

FS access is not thread-safe. There's a configuration in SPIFFS to protect critical sections but if enabled WDT kicks.

It seems like spiffs spends too much time in those sections.

Best regards,

Yuriy

[Angus Gratton]

Hi Yuriy,

This is really nice, thanks for contributing it.

Something we've said that it'd be great to have eventually is the newlib syscalls (open/read/write/close/etc) able to access SPIFFS directly (so you can port code that uses normal POSIX file access, for example the , and this is a solid first piece in that direction.

On Mon, Jun 27, 2016 at 08:24:47AM -0700, Yuriy wrote:
> Spiffs is configured in "singleton" mode. I don't think anybody needs more
> than one instance of FS on esp8266.

Agreed!

> FS access is not thread-safe. There's a configuration in SPIFFS to protect
> critical sections but if enabled WDT kicks.
> It seems like spiffs spends too much time in those sections.

That's interesting. I'll try and make time to pull your branch and run some tests with this as well.


Angus

[Yuriy]

Hi Angus,

Thanks for the feedback.

I've started working on providing syscalls to the newlib.

So far open/close/write/unlink works.

I've also written a basic file system load test for POSIX file operations to ensure that everything works as it should.

The test currently performs the following operations in random order:

• creates a file and writes to it
• reads a file and verifies content validity
• rewrites an existing file
• appends to an existing file
• removes an existing file

The test passes on my ESP-12E module as well as on my linux machine.

github/posix_fs

Another thing I've been working on lately is to reverse engineer SPI flash read/write operations form the SDK.

Currently I have a rewritten read/write/erase implementation in C. My implementation does exactly the same as the SDK.

It only bypasses SDK library but still uses low level SPI flash access functions from ROM.

I only verified that SPIFFS with my implementation works.

The original intention to dig it up was to find a way to eliminate 4-bytes address/buffer/size alignment for flash access operations.

But I haven't figured out it yet...

Reversed engineered SPI flash access.

Yuriy

[Angus Gratton]

Hi Yuriy,

On Thu, Jun 30, 2016 at 12:53:58PM -0700, Yuriy wrote:
> The test passes on my ESP-12E module as well as on my linux machine.
> github/posix_fs

> <https://github.com/sheinz/esp-open-rtos/tree/experiments/posix_fs>

Nice! I'm at a conference today and the next couple of days, but I'll take a look at this ASAP.

> Another thing I've been working on lately is to reverse engineer SPI flash
> read/write operations form the SDK.
> Currently I have a rewritten read/write/erase implementation in C. My
> implementation does exactly the same as the SDK.
> It only bypasses SDK library but still uses low level SPI flash access
> functions from ROM.
> I only verified that SPIFFS with my implementation works.
> The original intention to dig it up was to find a way to eliminate 4-bytes
> address/buffer/size alignment for flash access operations.
> But I haven't figured out it yet...
> Reversed engineered SPI flash access.

> <https://github.com/sheinz/esp-open-rtos/blob/experiments/spi_flash_reimplement/extras/spiffs/esp_spiffs.c>

Excellent. As it happens, foogod also did these functions as part of the "open-libmain" branch:
https://github.com/SuperHouse/esp-open-rtos/blob/open-libmain/open_esplibs/libmain/spi_flash.c

It's been in a separate branch for a while, but I just opened a PR to merge it:
https://github.com/SuperHouse/esp-open-rtos/pull/154

... would be interested in any comments you have on the differences between the ops you reverse engineered, and these ones.

Regarding supporting unaligned operations, this would be great! I agree it should be possible, but I think we'll have to add some "special case" wrappers to deal with the various unaligned cases by calling the ROM functions multiple times. Does that make sense?


Angus

[Yuriy]

On Friday, 1 July 2016 09:11:44 UTC+3, Angus Gratton wrote:

Hi Yuriy,

On Thu, Jun 30, 2016 at 12:53:58PM -0700, Yuriy wrote:
> The test passes on my ESP-12E module as well as on my linux machine.
> github/posix_fs
> <https://github.com/sheinz/esp-open-rtos/tree/experiments/posix_fs>

Nice! I'm at a conference today and the next couple of days, but I'll take a look at this ASAP.

> Another thing I've been working on lately is to reverse engineer SPI flash
> read/write operations form the SDK.
> Currently I have a rewritten read/write/erase implementation in C. My
> implementation does exactly the same as the SDK.
> It only bypasses SDK library but still uses low level SPI flash access
> functions from ROM.
> I only verified that SPIFFS with my implementation works.
> The original intention to dig it up was to find a way to eliminate 4-bytes
> address/buffer/size alignment for flash access operations.
> But I haven't figured out it yet...
> Reversed engineered SPI flash access.
> <https://github.com/sheinz/esp-open-rtos/blob/experiments/spi_flash_reimplement/extras/spiffs/esp_spiffs.c>

Excellent. As it happens, foogod also did these functions as part of the "open-libmain" branch:
https://github.com/SuperHouse/esp-open-rtos/blob/open-libmain/open_esplibs/libmain/spi_flash.c 

 

Too bad I hadn't noticed this branch before started digging into SDK disassembly.

Or not. Now I have at least some understanding what foogod has been through :)

 

It's been in a separate branch for a while, but I just opened a PR to merge it:
https://github.com/SuperHouse/esp-open-rtos/pull/154  

... would be interested in any comments you have on the differences between the ops you reverse engineered, and these ones.  

 

Everything seems to do logically the same. The only difference I noticed is in function sdk_SPIWrite.

The loop variable is int whereas in SDK it treats like a byte.

In the SDK it is 8 bit truncated(extui a6, a7, 0, 8), so it will overflow if written more than 256 pages which is unlikely as it will require 65k data buffer :)

So, foogod's implementation looks better.

Regarding supporting unaligned operations, this would be great! I agree it should be possible, but I think we'll have to add some "special case" wrappers to deal with the various unaligned cases by calling the ROM functions multiple times. Does that make sense?


Angus

It's great how much of the SDK is now open sourced. But where to stop?

Will it be any use to reverse engineer also ROM functions?

How much easier it all would be if Espressif just released all the sources :)

Cheers,

Yuriy.

[Yuriy]

Hi All,

Just a quick update.

I've written a command line utility that creates SPIFFS image from the specified directory.

It uses the same SPIFFS sources and the same configuration that ESP8266 uses. 

So, if configuration or SPIFFS changes the image will be valid.

I integrated it into the build process, so to build and flash SPIFFS image for you project 

you simply need to use extras/spiffs and use a special macro:

$(eval $(call make_spiffs_image,files))

where files is the directory with files to store in SPIFFS.

Then with make flash everything will be flash to the device.

There's currently two problems:

• If files change in files directory build system will not rebuild SPIFFS image. (if add files, system triggers image rebuild)
• If SPIFFS_SIZE is changed in Makefile, mkspiffs utility will not be rebuild for the new SPIFFS_SIZE.

So, in both cases make clean is necessary.

Everybody is welcome to comment/suggest.

Best regards,

Yuriy