A20 SOM EVB: NAND -> U-Boot -> SATA

[Daft Soft]

Hi, All!
I'm working on booting A20 (Olimex A20-SOM) from NAND. Due to lack of information/howtos and any kind of inctructions on the internet I had to start to make my own way. I've managed to build kernel (5.2.21) with NAND support. Tried to write U-Boot Olinuxino, denx U-Boot and sunxi-mainline U-Boot (all of them are built by me with A20-Olimex-SOM-EVB_defconfig) with flash_erase/nandwrite from linux. I tried to build so called boot0 image with sunxi-spl-image-builder (with u-boot.bin, u-boot-sunxi-with-spl.bin and sunxi-spl.bin as source). None of them started the board (not even a single message in UART). BROM reacts by falling to FEL-mode with all that "images" after reseting (this was important information for me in my next researches).
Then I've found "first" version of bootfix (CT_NandBoot), which "restores" proprietary boot0 and boot1 in NAND via FEL mode. And it did - boot0 and boot1 start and hang (NAND was empty at that moment). But at that moment I knew that my board and its NAND are healthy and can boot from NAND. Then I tried nandinstall from proprietary Debian 8 system. It created (I suppose by means of nand-part) "bootloader" partition (starting from 128 block and 16384 blocks size). The system (BROM->boot0->boot1) started old proprietary SUNXI U-Boot that was written to NAND by nandinstall script. Okay, my next idea was to replace u-boot.bin in nanda ("bootloader") partition with my. I did it with Debian 8. And boot0/boot1 started my U-Boot. But unluckily scsi operations (reset, scan) hang the board. Replacing u-boot.bin with u-boot-sunxi-with-spl.bin lead to hang after "jump to". It looks like two SPLs (U-Boot's and Allwinner's) conflict.
Then I tried to replace boot0 and boot1 files in bootfix - it rejected to upload them despite they had eGON BT signature. Then accidentally I've found "second" bootfix which allows to upload any files as boot0 and boot1 (FEX still can reject data on A20 side).
But none of my images worked. But (!) on some of them, sunxi-spl.bin, for example, A20 does not fall to FEL-mode. This means that it kinda starts my SPL, but hangs.
I tried to load my u-boot-sunxi-with-spl.bin via FEL (sunxi-fel uboot u-boot-sunxi-with-spl.bin). And it starts normally and scsi works.
"second" bootfix contains files sunxi-spl.bin_last_ok and u-boot-dtb.bin_last_ok (no config is included). I've uploaded them by means of bootfix and this SPL starts from NAND. But prints:

U-Boot SPL 2017.11-dirty (Mar 30 2018 - 22:22:01)
DRAM: 1024 MiB
CPU: 912000000Hz, AXI/AHB/APB: 3/2/2
Trying to boot from NAND
SPL: failed to boot from all boot devices
### ERROR ### Please RESET the board ###


May be this is because it's built for A20 Lime2 (it contains string "sun7i-a20-olinuxino-lime2"). But that means that U-Boot _can_ be built for NAND and can be written via FEL + bootfix (this is pretty nice method) without proprietary boot0/boot1.
At this point I'm stuck.
What do I do wrong? How can I build U-Boot with correct signature for A20 BROM which can be flashed to NAND?

P.S.
I don't need to put kernel in NAND, loading kernel from SATA is okay. It's all about excluding SD-card from boot process by any available schema.

P.P.S.
Sorry for my language.

[Arti Zirk]

On T, 2022-02-22 at 04:36 -0800, Daft Soft wrote:
> I'm working on booting A20 (Olimex A20-SOM) from NAND

Can you tell us what board version of Olimex A20-SOM you have? Starting
from revision E[1], there should be a SPI flash chip on board that
should be much easier to get booting[2].

My experience with NAND on A20 has been on Cubietech Cubietruck board
and it has never worked well with mainline.

[1]: https://github.com/OLIMEX/SOM/blob/master/A20/A20-SOM/Revision_changes.txt
[2]: https://linux-sunxi.org/Bootable_SPI_flash

[Daft Soft]

Hi, all!
So, I have the complete solution to make A20 boot from NAND (SPL, U-Boot) +
SATA (DTB, Kernel, RootFS). This is to exclude SD-Card from boot process.

First part is a20_nand branch from U-Boot from
https://gitlab.com/m.motyl83/u-boot.git

NB!
a20_nand branch is crucial

This will make appropriate SPL (spl/sunxi-spl.bin), which can be loaded to
NAND via FEX and will boot.

At this moment we have one problem: we have no U-Boot proper prepared for
flashing via FEX (u-boot-dtb.bin will be rejected as file with wrong signature).
U-Boot does not build U-Boot proper with SUNXI eGON.BT1 signature at all.

So I've modified original mksunxi hosttool from U-Boot (source file included).
It places correct signature to header of image followed by U-Boot proper code.

eGON.BT signatures start from ARM "B" instruction (4 bytes - instruction opcode

and address to branch to).
Original mksunxi calculates this "branch" instruction (address) to jump over
eGON header. For Boot0 it works perfectly (because it is standard offset which
is used in any Boot0 image by BROM - be it proprietary Boot0 or U-Boot SPL).
But not for Boot1...

I don't know where IP (instruction pointer) points to when SPL loads U-Boot
proper and tries to "jump" (speaking in terms of x86) to it. So, I didn't
know how to calculate address of this instruction.
But we know that U-Boot proper code works when SPL jumps right to it
(this is proved by inspecting u-boot-sunxi-with-spl.bin and playing with
loading images via USB-FEL).

Thus I've modified generation of Boot1 image - it places U-Boot proper
code with some constant offset (2000h, filled with zeroes).

(Also we need to make more space in SUN4I_SRAM_SIZE, because it is Boot1 -

it is larger than Boot0).

Next step is to modify config. We need to tell SPL to load U-Boot proper
from specific offset (avoiding BROM signature and "B" instruction - it will
not read it from NAND at all).
This lines are:

CONFIG_SYS_NAND_U_BOOT_OFFS=0x802000
CONFIG_SYS_NAND_U_BOOT_OFFS_REDUND=0x802000

(These lines are for SPL, so you need to rebuild it to work with U-Boot proper
which is generated by modified mksunxi).

After all this we can use modified mksunxi (I call it mksunxi-bt1) to build
Boot1 image which contains signature, correctly placed U-Boot proper code,
can be flashed via FEX and started by SPL (with modified "U-Boot Offs").

[Oranż Metylowy]

I can confirm that this works (the method for running SPL as Boot0).

I've researched this a bit further and I know the reason for why that modification helps. When flashing with LiveSuit, the updater binaries (update_boot0.axf and update_boot1.axf) first grab the images to RAM, then parse their headers, check the checksum. Then, a part of the header called "storage data" is populated with data received from some other part of the updating process (possibly another .axf file, maybe related to configuration of the sys_config.fex). Boot0 and Boot1 have that part at 0x1C8 and 0x81D0 offset, respectively. Considering that U-Boot SPL doesn't have the entire eGON header and it starts at 0x60, the updater is overwriting part of SPL code - thus resulting in a corrupted firmware. As for U-Boot as Boot1, it overwrites code at 0x81D0. You have placed the code at 0x2000, so my guess is that you still have some part in the firmware overwritten. Maybe it just works by accident.

Anyways, I'm currently working on a binary-patched updater image, that will allow to flash Boot1 without any 0-byte padding, straight from U-Boot mainline. It won't even need an eGON header and will ignore the checksum. After that, I'll probably also patch the Boot0 updater not to overwrite anything, making it possible to just grab the SPL and flash it.

[Daft Soft]

I didn't just place code at 2000h. It wouldn't work. I had also to modify U-Boot SPL to make it load U-Boot Proper from specific address:

CONFIG_SYS_NAND_U_BOOT_OFFS=0x802000
CONFIG_SYS_NAND_U_BOOT_OFFS_REDUND=0x802000

These lines in .config are crucial (as rebuilding U-Boot SPL with them).

[Oranż Metylowy]

I know that changing the address is needed in your case.

However, I wanted to use stock U-boot build system, without Boot1 generation modifications. I binary-patched LiveSuit updater files to allow flashing generated images directly via FEX.

(SPL with small header and U-Boot without eGON header)

The required modifications for update_boot0.axf were to remove writing "storage data" to SPL header. Modifications to update_boot1.axf are:

- remove writing "storage data" (which would overwrite part of u-boot code)

- disable checking header magic (eGON.BT1)

- disable checksum validation (there's no header, so no checksum)

The binaries allow flashing SPL and U-Boot straight out of official U-Boot build through FEX/LiveSuit. The offsets don't even have to be changed, as there's no header - offset stays 0x800000.

However, I got stuck with SPL not recognizing correct NAND parameters at all, resulting in not being able to read U-Boot. I checked with SD-card U-boot that the data is actually stored in the NAND.

I'm attaching the patched binaries. When flashing, they also print "Hello, Update boot0.  Patch v1" on UART.

[Daft Soft]

Wow! Great work.

So let me make it clear. At the moment we can flash U-Boot SPL (built by stock mainline U-Boot's build system without modification of addresses/offsets, but still with eGON.BT0 signature),

flash U-Boot Proper without any modification and signature at all. Did I get it clear?

And let me guess - at the moment you get something like

Trying to boot from NAND
SPL: failed to boot from all boot devices

Right?

P.S.

I've modified my SPL - it read all NAND and outputs it byte by byte to UART.

And the data was correct from the 800000h and so on... I didn't see any modifications.

That's why I thought the problem was not in storing Proper in NAND rather in "jumping" ("b" - first instruction of eGON header).

I thought it was incorrect due to it's "technical nature". I'm not familiar with ARM architecture, enough, so I don't know how should parameters (address) of this instruction be generated.

After that I've modified offset in SPL and added padding in U-Boot Proper. Flashed and SPL jumped straight to Proper and it worked.

How did you see that update boot1 modified contents while flashing?

[Oranż Metylowy]

"So let me make it clear. At the moment we can flash U-Boot SPL, flash U-Boot Proper without any modification and signature at all"

Yes, that's what I did and my FEX flasher accepted and wrote the files to NAND.

"And let me guess - at the moment you get something like"

Not exactly. After adding some debugging printfs, I figured that SPL detects "correct" NAND parameters (which differ from actually correct params). Then it uses these params to read U-Boot Proper to RAM (all read operations return 00000000 and an error - it's ignored by SPL). Then it jumps to U-Boot Proper which are all 0's, because reading everything failed.

I don't know why SPL thinks that wrong params are correct. I don't have enough knowledge (nor a JTAG debugger, sadly) to be able fo fix it that far.

Though if your SPL can read from NAND successfully, my flasher should allow you to get mainline running. Well, after you add your DTBs and NAND params, per that commit from that gitlab repo. I don't know if adding NAND id in drivers/mtd/nand/sunxi_nand_spl.c is needed. Note that you shouldn't change CONFIG_SPL_TEXT_BASE, like that commit does.

"I've modified my SPL - it read all NAND and outputs it byte by byte to UART."

Could you share these modifications somewhere? It could be useful for me to debug which NAND configuration is correct in my case (if I ever get back to that topic...).

"I didn't see any modifications"

Maybe you just missed it. The code gets corrupted at about 32 KiB offset, and you probably didn't check the entire Proper by hand... that would be impossible.

First bytes of mainline without headers are also "b" instructions (afaik). The eGON header is useful only for stock boot0, SPL doesn't read it. Jumping to offset 0 of eGON just skips the header and goes to Proper's own "b" instructions. No header means no eGON, and jumping straight into Proper.

"How did you see that update boot1 modified contents while flashing?"

I disassembled update_boot1.axf to simply disable magic and CRC checking, then I found that the code does more than that (in IDA Pro - it can decompile ARM to C-pseudocode pretty well).

[Oranż Metylowy]

I'm also attaching stock updater binaries, so if anyone wants to know what changed, these files can be compared to the patched versions. It's very likely that there exist many versions of these flashers (maybe they even vary by CPU?).

Note: my flashers are for A13. I'm not sure if they'll work for you on A20. If your stock binaries are the same as these attached, then it will work. If you attach your files, I can try to modify them too.

[Daft Soft]

Hi!
"Not exactly.
...

to be able fo fix it that far."

It looks like your SPL is built with wrong MTD driver settings. SPL itself is
read by BROM, but U-Boot Proper is read by SPL. So MTD/NAND part of SPL has to
be configured correctly. I think first you should figure out what NAND chip
you use - you didn't mention which one. Mine is Micron, it was supported out
of the box. Get params of your NAND from datasheet then configure SPL to make
it use correct parameters, rebuild and go.

I'm not PRO in MTD/NAND devices but as I remember, NAND doesn't contain all
params of itself, so you need to specify them. You mentioned "adding NAND id" -
maybe in your case you'll have to do this (drivers/mtd/nand/nand_ids.c).

You've mentioned DT. SPL includes reduced part of DT. I didn't modify DT of
U-Boot at all. SPL is loaded by BROM, it loads U-Boot Proper in RAM with it's
hardcoded params ("SPL" in CONFIG_NAND_SUNXI_SPL_XXX config lines also points to
that these params are not loaded from DT by SPL), then (in my case) Proper reads
DT for kernel from SATA, loads kernel and boots it.

If your SPL is flashed correctly (if it starts - it is flashed correctly) then
you can get some hints about your NAND while flashing - FEX outputs some info
about it. In my case it is (with some my comments):

[SCAN_DBG] ==============Nand Architecture Parameter==============
[SCAN_DBG]    Nand Chip ID:         0x4b44642c 0xffffffa9
[SCAN_DBG]    Nand Chip Count:      0x1
[SCAN_DBG]    Nand Chip Connect:    0x1
[SCAN_DBG]    Nand Rb Connect Mode: 0x1
[SCAN_DBG]    Sector Count Of Page: 0x10        // 16
[SCAN_DBG]    Page Count Of Block:  0x100        // 256
[SCAN_DBG]    Block Count Of Die:   0x1000        // 4096
[SCAN_DBG]    Plane Count Of Die:   0x2            // 2
[SCAN_DBG]    Die Count Of Chip:    0x1
[SCAN_DBG]    Bank Count Of Chip:   0x1
[SCAN_DBG]    Optional Operation:   0x21788        //
[SCAN_DBG]    Access Frequence:     0x1e        // 30
[SCAN_DBG]    ECC Mode:             0x5
[SCAN_DBG]    Read Retry Type:      0x400a01
                                    ^^^^^^^^
                                    ead_retry_mode = (read_retry_type>>16)&0xff;    // 40h = Micron
                                    read_retry_cycle =(read_retry_type>>8)&0xff;    // 10
                                    read_retry_reg_num = (read_retry_type>>0)&0xff;    // 1

Again, if your SPL is flashed correctly you can rely on this info from FEX.

Just to resume and make it finally clear - your SPL, correctly built and
including correct header will load anyway/always, because it is loaded by
BROM (which initializes and works with NAND on this stage). But when SPL
is loaded - you are on your own. Thus SPL has to be configured correctly
to work with appropriate NFC and MTD/NAND or whatever...

"Note that you shouldn't change CONFIG_SPL_TEXT_BASE, like that commit does."

I don't know what exactly in this branch (a20_nand) from this repo helped me
to start U-Boot from NAND (I could not start mainline SPL from NAND nor Proper,
but unfortunately I have no time to prepare whole diff with mainline). But it
works with exactly that value. And I thought that this value is one of the
"magic" of this branch/repo compared to mainline. Did you build SPL (which
you've managed to start from NAND )from mainline or you're using Michał
Motyl repo?

"Could you share these modifications somewhere?"

I was wrong a little bit - it does not read ALL NAND, it just outputs bytes

which are read while loading Proper. Add these lines in sunxi_nand_spl.c to

nand_read_buffer()after ret = nand_read_page(conf, offs, dest, conf->page_size);

#if 0
        int j;
        for(j = 0; j < conf->page_size; ++j)
        {
//             if (((((uint8_t*) dest)[j]) >= 32) && ((((uint8_t*) dest)[j]) <= 126 ))
//                 printf("%c", ((uint8_t*) dest)[j]);
            if ((j % 16) == 0 )
                printf("\n");
            printf("%02x ", ((uint8_t*) dest)[j]);
        }
        printf("\n\n");
#endif

But if you need to read whole NAND chip I suppose you can just set
CONFIG_SYS_NAND_U_BOOT_OFFS to zero or any offset you wish.

"First bytes of mainline without headers are also "b" instructions (afaik).
The eGON header is useful only for stock boot0, SPL doesn't read it. Jumping
to offset 0 of eGON just skips the header and goes to Proper's own "b"
instructions. No header means no eGON, and jumping straight into Proper."

Of course, you're right. We need correct "B" instruction (as well as header
itself) only for BT0 (because we can't omit BROM). After that if we can flash
Proper without header we don't need it.

P.S.
Yes, you're right. Tested one more time. It looks like FEX recalculates CRC
- 4 bytes next to eGON.BT1 differ with those in my boot1 file. But after that
I see some zeroes (I fill rest of header with zeroes) and the whole page
(2000h bytes - sizeof(header)) filled with FFh (my "alignment") and no
modification. So I don't see what kind of "storage data" is modified by FEX.
(Screenshot attached).
Anyway this should not be the problem - SPL should jump over whole header.
So I still see the ARM "b" instruction as the cause of not starting Proper
even when SPL starts and reads it from standard offset. You can play around

with this with my prints in SPL and see. Maybe on A13 something is different.

[Oranż Metylowy]

About the last part of your mail:

What you highlighted is only the checksum. The header is not that short - it's about 32 KiB in size, iirc. You didn't fill "rest of the header" with zeroes, just the beginning of it. If you want to find modifications, you have to read a page at offset 0x81D0. That's where it gets modified. Somewhere on the net there are sources of bootloaders, with the header structure described. SPL jumps over your entire header, but only part of FEX's header (32 KiB vs 0x2000 B in your case). Also, "ef be ad de" is obviously not a valid B instruction, but you probably know this.

Yes, I'm flashing proper without header because SPL doesn't need it, just like we said before. I built SPL from mainline, I didn't touch Michał's repo.

I can't figure out the correct settings. I tried what's said in datasheet for my Micron NAND. Also the SPL seems to ignore these options anyway, looks like it's trying all one by one (I printed every of these tries, and the one that supposedly succeeded had wrong params). Maybe other parts of the Kconfig I have set incorrectly. Parameters from FEX seemed to match what I set in these config options, but no luck in reading by SPL.

[Daft Soft]

Hi!

I've tested your update_boot0.axf and update_boot1.axf which you've sent on 25 Apr with A20.

This is what I get

[FES]Err:ERROR : elf_loader() magic error

I'm using so called bootfix (CLI tool). It is slightly modified by me ("makeup" modification mostly).

It looks like it supports A13:

    switch ((buf.soc_id >> 8) & 0xFFFF) {
        case 0x1610: soc_name = "fes"; break;
        case 0x1623: soc_name = "A10"; break;
        case 0x1625: soc_name = "A13"; break;
        case 0x1633: soc_name = "A31"; break;
        case 0x1651: soc_name = "A20"; break;
        case 0x1650: soc_name = "A23"; break;
        case 0x1680: soc_name = "H3" ; break;
    }

But A13 is SUN5I family, while A20 is SUN7I. Bootfix includes two subdirs: sun4i and sun7i

which contain AXFs and FEXs. Actually I don't know how bootfix supports SUN5I and does

it support it at all - there is no sun5i directory at all. (And I have no A13 board to test it).

I suppose ELF error is all about difference in families.

I attach original AXFs for sun7i family. If you modify them I'll test them on my A20 and will

give you feedback.

[Oranż Metylowy]

The files you attached are encrypted. It seems that LiveSuit images are encrypted entirely, I don't know the details. I used "imgRePacker" to dump my stock image, and it produced encrypted files in the "_img.files" directory (which looked just like your .axf files), and normal files in "home" directory. Though I think this is image-specific, as another .img I tried had an "_extract" directory (apart from the "_img.files") which also contained encrypted files.

I'm not sure if imgRePacker version matters. You probably just have to try unpacking A20 images until you find one that produces unencrypted files.

The reason for your error is likely that my files were packed in "unencrypted" form - your packer expected to pack encrypted files. Then A20 (or FEX) tried to "decrypt" my file (which was unencrypted) and probably got garbage in result.

I don't know how are these images encrypted. One could probably build a simple encrypt-decrypt tool, as all of these images look the same when encrypted - that could mean they use the same key.

[Jeroen van Jaarsveld]

I might be totally out of my depth here, but are you aware of the process described in board/sunxi/README.nand?

Allwinner NAND flashing
=======================

A lot of Allwinner devices, especially the older ones (pre-H3 era),
comes with a NAND. NANDs storages are a pretty weak choice when it
comes to the reliability, and it comes with a number of flaws like
read and write disturbs, data retention issues, blocks becoming
unusable, etc.

In order to mitigate that, various strategies have been found to be
able to recover from those issues like ECC, hardware randomization,
and of course, redundancy for the critical parts.

This is obviously something that we will take into account when
creating our images. However, the BROM will use a quite weird pattern
when accessing the NAND, and will access only at most 4kB per page,
which means that we also have to split that binary across several
pages.

In order to accommodate that, we create a tool that will generate an
SPL image that is ready to be programmed directly embedding the ECCs,
randomized, and with the necessary bits needed to reduce the number of
bitflips. The U-Boot build system, when configured for the NAND (with
CONFIG_MTD_RAW_NAND=y) will also generate the image sunxi-spl-with-ecc.bin
that will have been generated by that tool.

In order to flash your U-Boot image onto a board, assuming that the
board is in FEL mode, you'll need the sunxi-tools that you can find at
this repository: https://github.com/linux-sunxi/sunxi-tools

Then, you'll need to first load an SPL to initialise the RAM:
sunxi-fel spl spl/sunxi-spl.bin

Load the binaries we'll flash into RAM:
sunxi-fel write 0x4a000000 u-boot-dtb.bin
sunxi-fel write 0x43000000 spl/sunxi-spl-with-ecc.bin

And execute U-Boot
sunxi-fel exe 0x4a000000

On your board, you'll now have all the needed binaries into RAM, so
you only need to erase the NAND...

nand erase.chip

Then write the SPL and its backup:

nand write.raw.noverify 0x43000000 0 40
nand write.raw.noverify 0x43000000 0x400000 40

And finally write the U-Boot binary:
nand write 0x4a000000 0x800000 0xc0000

You can now reboot and enjoy your NAND.

Op woensdag 11 mei 2022 om 20:08:26 UTC+2 schreef ku...@szczodrzynski.pl:

[Oranż Metylowy]

Yes, it doesn't work, at least for me. I tried it a million times.

[Maxime Ripard]

On Fri, May 27, 2022 at 10:08:16AM -0700, Oranż Metylowy wrote:
> Yes, it doesn't work, at least for me. I tried it a million times.

If you're using an A20, this is likely to be due to:
https://lore.kernel.org/u-boot/20210624120540.7oyhdmerjndsylbq@gilmour/

However, I must emphathize that using an MLC NAND is a terrible idea
with the current state of UBI and UBIFS, and really shouldn't be
encouraged.

Maxime

[Oranż Metylowy]

I'm using A13 and the steps posted in board/sunxi/README.nand don't even make the board run SPL (it jumps to FEX every time).

The Linux NAND driver works - at least from what I tried. My idea was to use the NAND in SLC mode (which decreases the capacity twice, afaik) with UBI and that part works under Linux. U-boot doesn't want to read it, but that's actually not a problem, since I can store the kernel directly in NAND, without a filesystem (U-boot reads NAND properly).

The only problem is that SPL won't read NAND, it only reads 0s - just like in that mail you linked. Do you think that reversing this patch could fix the issue?