TI AM3517 UBI Mender Version 1.2 integration notes

[Dan Walkes]

Hi Folks,

I’ve completed the integration steps for UBI mender integration on release 1.2. I wanted to share a few notes about the ubi integration with Mender on a TI AM3517 platform, in case these are useful to anyone else.

I’m using a custom u-boot build based on ti-u-boot.  

The instructions in https://docs.mender.io/1.2/devices/integrating-with-u-boot and https://docs.mender.io/1.2/devices/integrating-with-u-boot/providing-custom-u-boot-fw-utils for u-boot-fw-utils were great!  Thanks for putting these together.  I noticed a few small issues with my specific configuration I wanted to detail here.  These might be user error of some sort.

1) I wasn’t able to get the qemu flash build working for some reason, although I didn’t spend much time trying to figure it out.  The build succeeded but attempt to boot gave me

```

Booting from NOR...

ubi0: attaching mtd3

ubi0: scanning is finished

ubi0: volume table was restored

UBI init error 12

```

I did see the note about using latest qemu and did build qemu 2.10.0.  My qemu start commands were:

```

QEMU_SYSTEM_ARM=~/qemu-2.10.0/arm-softmmu/qemu-system-arm

VEXPRESS_IMG=tmp/deploy/images/vexpress-qemu-flash/core-image-minimal-vexpress-qemu-flash.vexpress-nor \

MACHINE=vexpress-qemu-flash \

    ../meta-mender/meta-mender-qemu/scripts/mender-qemu

```

After this failed I went straight to the integration steps for my hardware, so there could be a simple fix for this.  It's not blocking me at the moment but if you'd like more information I can provide it.

2) I noticed errors like this when attempting to run bitbake - note that this is for the u-boot recipe I’m *not* using, the default one in the yocto meta layer:

```

WARNING: /home/dan/CCC/Yocto/build-am3517/../meta/recipes-bsp/u-boot/u-boot-fw-utils_2017.01.bb: Error during finalise of /home/dan/CCC/Yocto/build-am3517/../meta/recipes-bsp/u-boot/u-boot-fw-utils_2017.01.bb

ERROR: ExpansionError during parsing /home/dan/CCC/Yocto/build-am3517/../meta/recipes-bsp/u-boot/u-boot_2017.01.bb                                            | ETA:  0:00:02

Traceback (most recent call last):

bb.data_smart.ExpansionError: Failure expanding variable MENDER_BOOTENV_TOTAL_ALIGNED_SIZE, expression was ${@mender_get_env_total_aligned_size(0x200000, 128)} which triggered exception NameError: name 'mender_get_env_total_aligned_size' is not defined

```

I fixed this with this small patch on u-boot-mender-common.inc:

...

diff --git a/meta-mender-core/recipes-bsp/u-boot/u-boot-mender-common.inc b/meta-mender-core/recipes-bsp/u-boot/u-boot-mender-common.inc

index 46b158a..c7dcd1b 100644

--- a/meta-mender-core/recipes-bsp/u-boot/u-boot-mender-common.inc

+++ b/meta-mender-core/recipes-bsp/u-boot/u-boot-mender-common.inc

@@ -1,4 +1,4 @@

-inherit mender-helpers

+inherit mender-helpers mender-uboot

…

I’m not sure if this is the correct fix or if there’s something different I should be doing to resolve.

3) I noticed this error when building the u-boot-fw-utils

...

ERROR: u-boot-fw-utils-ccc.2016.05-1.0-r0 do_package: objcopy failed with exit code 1 (cmd was 'arm-poky-linux-gnueabi-objcopy' --only-keep-debug '/home/dan/CCC/Yocto/build-am3517/tmp/work/prod_am3517-poky-linux-gnueabi/u-boot-fw-utils-ccc.2016.05/1.0-r0/package/sbin/fw_setenv' '/home/dan/CCC/Yocto/build-am3517/tmp/work/prod_am3517-poky-linux-gnueabi/u-boot-fw-utils-ccc.2016.05/1.0-r0/package/sbin/.debug/fw_setenv'):

arm-poky-linux-gnueabi-objcopy:/home/dan/CCC/Yocto/build-am3517/tmp/work/prod_am3517-poky-linux-gnueabi/u-boot-fw-utils-ccc.2016.05/1.0-r0/package/sbin/fw_setenv: File format not recognized

ERROR: u-boot-fw-utils-ccc.2016.05-1.0-r0 do_package: Function failed: split_and_strip_files

ERROR: Logfile of failure stored in: /home/dan/CCC/Yocto/build-am3517/tmp/work/prod_am3517-poky-linux-gnueabi/u-boot-fw-utils-ccc.2016.05/1.0-r0/temp/log.do_package.13334

ERROR: Task (/home/dan/CCC/Yocto/build-am3517/../meta-ccc/recipes-bsp/u-boot/u-boot-fw-utils-ccc.2016.05.bb:do_package) failed with exit code '1'

...

If I look at the file output I could see that /home/dan/CCC/Yocto/build-am3517/tmp/work/prod_am3517-poky-linux-gnueabi/u-boot-fw-utils-ccc.2016.05/1.0-r0/package/sbin/fw_setenv was not being cross compiled.

I traced this down to the tools/env/Makefile settting here: https://git.ti.com/ti-u-boot/ti-u-boot/blobs/master/tools/env/Makefile#line11 and noticed that recent versions of u-boot used override on this line, which avoids overrides of HOST_CC from the command line.  See https://github.com/u-boot/u-boot/blob/c98ac3487e413c71e5d36322ef3324b21c6f60f9/tools/env/Makefile#L11.  Adding an “override” here fixed the issue with my build.  So this might just be a note you could add to the custom u-boot instructions for people using older u-boot ports.

4) I’m having trouble setting up the correct build of fw_env.config for my configuration.

I’m using a partition layout for mtd which looks like this:

```

root@prod-am3517:~# cat /proc/mtd

dev:    size   erasesize  name

mtd0: 00020000 00020000 "MLO"

mtd1: 00100000 00020000 "u-boot"

mtd2: 00040000 00020000 "u-boot-env"

mtd3: 1fea0000 00020000 "ubi"

```

With u-boot-env at  0x120000 length 0x40000 or 2 0x20000 sized erase sectors.  Partition mtd3 is the mtd partition containing each of my rootfs partitions and the data partition.

Based on the instructions at https://docs.mender.io/1.2/devices/partition-layout I’m supposed to use a MENDER_STORAGE_DEVICE setting that represents the entire device, not any single partition.  However it’s not clear how to do that with my current mtdparts configuration.  I think I would need to define mtd0 as the entire device in this case and remove definitions for u-boot, u-boot-env, and ubi which seems wrong.  The default setting for MENDER_STORAGE_DEVICE, ubi0, won’t work because I don’t have a ubi0 device defined.  If I use /dev/ubi0 instead I get

```

root@prod-am3517:~# fw_printenv

Cannot get MTD information for /dev/ubi0

```

Which makes sense because ubi0 isn’t an mtd device.

 If I try to use /dev/mtd0 as MENDER_STORAGE_DEVICE instead of ubi0 with this content for fw_env.config:

```

/dev/mtd0 0x120000 0x20000

/dev/mtd0 0x140000 0x20000

```

I get

```

root@prod-am3517:~# fw_printenv

Cannot read bad block mark: Invalid argument

```

Presumably because the offsets are outside the range for device /dev/mtd0.

I need to use this setting with my current fw_env.config:

```

root@prod-am3517:~# cat /data/u-boot/fw_env.config

/dev/mtd2 0x0000 0x20000 0x20000

/dev/mtd2 0x20000 0x20000 0x20000

```

To generate this .config file I’d need some new variable like MENDER_STORAGE_MTD_PARTITION and MENDER_STORAGE_MTD_PARTITION_OFFSET and associated modifications to replace the steps in u-boot-fw-utils-mender.inc at https://github.com/mendersoftware/meta-mender/blob/master/meta-mender-core/recipes-bsp/u-boot/u-boot-fw-utils-mender.inc#L7.  I’d rather do what everyone else is doing but I’m having trouble understanding how to match the current scheme up with my mtd layout.

If there’s a better way to fix these issues I’d greatly appreciate any suggestions.  if you’d like me to prepare pull requests for any code or documentation changes based on these observations please let me know.

[Eystein Måløy Stenberg]

Thanks for the notes and recommendations, very interesting. We'll look
into them!

No worries on wrong list, everything gets forwarded to
men...@lists.mender.io - no need to repost. :)


On 18/09/17 10:27, danw...@trellis-logic.com wrote:
> Sorry, just realized I posted in the wrong list. Moved
> to https://groups.google.com/a/lists.mender.io/forum/#!topic/mender/7Gv0otJS6Ng
>
> On Monday, September 18, 2017 at 11:25:43 AM UTC-6,

> danw...@trellis-logic.com wrote:
>
> Hi Folks,
> I’ve completed the integration steps for UBI mender integration on
> release 1.2. I wanted to share a few notes about the ubi integration
> with Mender on a TI AM3517 platform, in case these are useful to
> anyone else.
>
> I’m using a custom u-boot build based on ti-u-boot.
>
> The instructions in
> https://docs.mender.io/1.2/devices/integrating-with-u-boot
> <https://docs.mender.io/1.2/devices/integrating-with-u-boot> and
> https://docs.mender.io/1.2/devices/integrating-with-u-boot/providing-custom-u-boot-fw-utils
> <https://docs.mender.io/1.2/devices/integrating-with-u-boot/providing-custom-u-boot-fw-utils>
> for u-boot-fw-utils were great! Thanks for putting these together.
> I noticed a few small issues with my specific configuration I
> wanted to detail here. These might be user error of some sort.
>

> *1)* I wasn’t able to get the qemu flash build working for some

> reason, although I didn’t spend much time trying to figure it out.
> The build succeeded but attempt to boot gave me
> ```
> Booting from NOR...
> ubi0: attaching mtd3
> ubi0: scanning is finished
> ubi0: volume table was restored
> UBI init error 12
> ```
> I did see the note about using latest qemu and did build qemu
> 2.10.0. My qemu start commands were:
> ```
> QEMU_SYSTEM_ARM=~/qemu-2.10.0/arm-softmmu/qemu-system-arm
> VEXPRESS_IMG=tmp/deploy/images/vexpress-qemu-flash/core-image-minimal-vexpress-qemu-flash.vexpress-nor
> \
> MACHINE=vexpress-qemu-flash \
> ../meta-mender/meta-mender-qemu/scripts/mender-qemu
> ```
> After this failed I went straight to the integration steps for my
> hardware, so there could be a simple fix for this. It's not
> blocking me at the moment but if you'd like more information I can
> provide it.
>

> *2)* I noticed errors like this when attempting to run bitbake -

> note that this is for the u-boot recipe I’m *not* using, the default
> one in the yocto meta layer:
> ```
> WARNING:
> /home/dan/CCC/Yocto/build-am3517/../meta/recipes-bsp/u-boot/u-boot-fw-utils_2017.01.bb

> <http://u-boot-fw-utils_2017.01.bb>: Error during finalise of
> /home/dan/CCC/Yocto/build-am3517/../meta/recipes-bsp/u-boot/u-boot-fw-utils_2017.01.bb
> <http://u-boot-fw-utils_2017.01.bb>

> ERROR: ExpansionError during parsing
> /home/dan/CCC/Yocto/build-am3517/../meta/recipes-bsp/u-boot/u-boot_2017.01.bb

> <http://u-boot_2017.01.bb>

> | ETA: 0:00:02
> Traceback (most recent call last):
> bb.data_smart.ExpansionError: Failure expanding variable
> MENDER_BOOTENV_TOTAL_ALIGNED_SIZE, expression was
> ${@mender_get_env_total_aligned_size(0x200000, 128)} which triggered
> exception NameError: name 'mender_get_env_total_aligned_size' is not
> defined
> ```
> I fixed this with this small patch on u-boot-mender-common.inc:
> ...
> diff --git
> a/meta-mender-core/recipes-bsp/u-boot/u-boot-mender-common.inc
> b/meta-mender-core/recipes-bsp/u-boot/u-boot-mender-common.inc
> index 46b158a..c7dcd1b 100644
> --- a/meta-mender-core/recipes-bsp/u-boot/u-boot-mender-common.inc
> +++ b/meta-mender-core/recipes-bsp/u-boot/u-boot-mender-common.inc
> @@ -1,4 +1,4 @@
> -inherit mender-helpers
> +inherit mender-helpers mender-uboot
> …
> I’m not sure if this is the correct fix or if there’s something
> different I should be doing to resolve.
>

> *3) *I noticed this error when building the u-boot-fw-utils

> <https://github.com/u-boot/u-boot/blob/c98ac3487e413c71e5d36322ef3324b21c6f60f9/tools/env/Makefile#L11>.

> Adding an “override” here fixed the issue with my build. So this
> might just be a note you could add to the custom u-boot instructions
> for people using older u-boot ports.
>

> *4)* I’m having trouble setting up the correct build of

> <https://github.com/mendersoftware/meta-mender/blob/master/meta-mender-core/recipes-bsp/u-boot/u-boot-fw-utils-mender.inc#L7>.

> I’d rather do what everyone else is doing but I’m having trouble
> understanding how to match the current scheme up with my mtd layout.
>
> If there’s a better way to fix these issues I’d greatly appreciate
> any suggestions. if you’d like me to prepare pull requests for any
> code or documentation changes based on these observations please let
> me know.
>
>

> On Monday, September 18, 2017 at 11:25:43 AM UTC-6,

> danw...@trellis-logic.com wrote:
>
> Hi Folks,
> I’ve completed the integration steps for UBI mender integration on
> release 1.2. I wanted to share a few notes about the ubi integration
> with Mender on a TI AM3517 platform, in case these are useful to
> anyone else.
>
> I’m using a custom u-boot build based on ti-u-boot.
>
> The instructions in
> https://docs.mender.io/1.2/devices/integrating-with-u-boot
> <https://docs.mender.io/1.2/devices/integrating-with-u-boot> and
> https://docs.mender.io/1.2/devices/integrating-with-u-boot/providing-custom-u-boot-fw-utils
> <https://docs.mender.io/1.2/devices/integrating-with-u-boot/providing-custom-u-boot-fw-utils>
> for u-boot-fw-utils were great! Thanks for putting these together.
> I noticed a few small issues with my specific configuration I
> wanted to detail here. These might be user error of some sort.
>

> *1)* I wasn’t able to get the qemu flash build working for some

> reason, although I didn’t spend much time trying to figure it out.
> The build succeeded but attempt to boot gave me
> ```
> Booting from NOR...
> ubi0: attaching mtd3
> ubi0: scanning is finished
> ubi0: volume table was restored
> UBI init error 12
> ```
> I did see the note about using latest qemu and did build qemu
> 2.10.0. My qemu start commands were:
> ```
> QEMU_SYSTEM_ARM=~/qemu-2.10.0/arm-softmmu/qemu-system-arm
> VEXPRESS_IMG=tmp/deploy/images/vexpress-qemu-flash/core-image-minimal-vexpress-qemu-flash.vexpress-nor
> \
> MACHINE=vexpress-qemu-flash \
> ../meta-mender/meta-mender-qemu/scripts/mender-qemu
> ```
> After this failed I went straight to the integration steps for my
> hardware, so there could be a simple fix for this. It's not
> blocking me at the moment but if you'd like more information I can
> provide it.
>

> *2)* I noticed errors like this when attempting to run bitbake -

> note that this is for the u-boot recipe I’m *not* using, the default
> one in the yocto meta layer:
> ```
> WARNING:
> /home/dan/CCC/Yocto/build-am3517/../meta/recipes-bsp/u-boot/u-boot-fw-utils_2017.01.bb

> <http://u-boot-fw-utils_2017.01.bb>: Error during finalise of
> /home/dan/CCC/Yocto/build-am3517/../meta/recipes-bsp/u-boot/u-boot-fw-utils_2017.01.bb
> <http://u-boot-fw-utils_2017.01.bb>

> ERROR: ExpansionError during parsing
> /home/dan/CCC/Yocto/build-am3517/../meta/recipes-bsp/u-boot/u-boot_2017.01.bb

> <http://u-boot_2017.01.bb>

> | ETA: 0:00:02
> Traceback (most recent call last):
> bb.data_smart.ExpansionError: Failure expanding variable
> MENDER_BOOTENV_TOTAL_ALIGNED_SIZE, expression was
> ${@mender_get_env_total_aligned_size(0x200000, 128)} which triggered
> exception NameError: name 'mender_get_env_total_aligned_size' is not
> defined
> ```
> I fixed this with this small patch on u-boot-mender-common.inc:
> ...
> diff --git
> a/meta-mender-core/recipes-bsp/u-boot/u-boot-mender-common.inc
> b/meta-mender-core/recipes-bsp/u-boot/u-boot-mender-common.inc
> index 46b158a..c7dcd1b 100644
> --- a/meta-mender-core/recipes-bsp/u-boot/u-boot-mender-common.inc
> +++ b/meta-mender-core/recipes-bsp/u-boot/u-boot-mender-common.inc
> @@ -1,4 +1,4 @@
> -inherit mender-helpers
> +inherit mender-helpers mender-uboot
> …
> I’m not sure if this is the correct fix or if there’s something
> different I should be doing to resolve.
>

> *3) *I noticed this error when building the u-boot-fw-utils

> <https://github.com/u-boot/u-boot/blob/c98ac3487e413c71e5d36322ef3324b21c6f60f9/tools/env/Makefile#L11>.

> Adding an “override” here fixed the issue with my build. So this
> might just be a note you could add to the custom u-boot instructions
> for people using older u-boot ports.
>

> *4)* I’m having trouble setting up the correct build of

> <https://github.com/mendersoftware/meta-mender/blob/master/meta-mender-core/recipes-bsp/u-boot/u-boot-fw-utils-mender.inc#L7>.

> I’d rather do what everyone else is doing but I’m having trouble
> understanding how to match the current scheme up with my mtd layout.
>
> If there’s a better way to fix these issues I’d greatly appreciate
> any suggestions. if you’d like me to prepare pull requests for any
> code or documentation changes based on these observations please let
> me know.
>

> --
> You received this message because you are subscribed to the Google
> Groups "Mender lists" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to mender+un...@northerntech.community
> <mailto:mender+un...@northerntech.community>.
> To post to this group, send email to men...@northerntech.community
> <mailto:men...@northerntech.community>.
> Visit this group at
> https://groups.google.com/a/northerntech.community/group/mender/.
> To view this discussion on the web visit
> https://groups.google.com/a/northerntech.community/d/msgid/mender/5bcb5c7d-3c85-4ee4-bb02-143987974e49%40northerntech.community
> <https://groups.google.com/a/northerntech.community/d/msgid/mender/5bcb5c7d-3c85-4ee4-bb02-143987974e49%40northerntech.community?utm_medium=email&utm_source=footer>.
> For more options, visit
> https://groups.google.com/a/northerntech.community/d/optout.

--

Eystein

[Maciej Borzecki]

Try this:
```
# MTD device name Device offset Env. size Flash sector
size Number of sectors
/dev/mtd2 0x00000000 0x00020000 0x20000
1
/dev/mtd2 0x00020000 0x00020000 0x20000
1
```

There's an example machine configuration for Toradex iMX7 board right
here: https://github.com/mendersoftware/meta-mender/blob/master/tests/meta-mender-toradex-nxp-ci/conf/machine/colibri-imx7-mender.conf
The boards that come with MTD support provide their own fw_env.config
file, eg. Toradex iMX7:
https://github.com/mendersoftware/meta-mender/blob/master/meta-mender-toradex-nxp/recipes-bsp/u-boot/files/mx7/fw_env.config
(MTD partitions are listed in machine.conf for reference).

The defaults for UBI support are right here:
https://github.com/mendersoftware/meta-mender/blob/master/meta-mender-core/classes/mender-install-ubi.bbclass

With these defaults, it is assumed there is a device ubi0 (/dev/ubi0),
rootfs A and B are in volumes `rootfsa` (/dev/ubi0_0) and `rootfsb`
(/dev/ubi0_1). Data volume is named `data` (/dev/ubi0_2).

--
Maciej Borzecki

[Dan Walkes]

Thanks for the suggestion. I didn't notice the handling for custom fw_env.config files previously.

[holden....@ultra-fei.com]

Hi all,

I'm on the tail end of UBI mender integration for the Xilinx ZynqMP ZCU102 platform. I have a few additional notes, and a question for the mender devs. I've been able to to successfully integrate the mender uboot layer when placing the uboot environment in MMC, but I am having trouble changing the CONFIG_ENV_IS_IN_MMC to CONFIG_ENV_IS_IN_SPI_FLASH or CONFIG_ENV_IS_IN_FLASH. 

In general, the final step I'm working toward is achieving proper function from uboot regarding the environment -- reading/writing flash and having working versions of the fw_* utils. The fw_* utils are functional in linux at the moment for my u-boot fork, but I am getting CRC errors because the environments don't actually exist in flash and are not being used by u-boot at the moment.

The system I'm working with is using QSPI NOR with the following layout --

1 Mbyte - BOOT.bin

1 Mbyte - UbootEnv/RedundEnv

126 MByte - UBI partition containing three volumes: rootfsa, rootfsb, data

This device was particularly a pain in the butt to integrate because Xilinx doesn't provide working drivers to allow the QSPI controller to access >3MByte files in a UBI volume. I wrote the appropriate U-Boot driver and am able to boot out of UBI without much issue (aside from having to turn off compression.. still looking into that one, it's probably a kernel issue).

I believe the correct uboot define to use for my situation is CONFIG_ENV_IS_IN_SPI_FLASH. I'm using the "sf" commands when dealing with my flash, aside from attaching/mounting/loading from UBI, which is using the mtd layer under the hood. However, when I enable this define I get an error that mender does not support  the SPI_FLASH option and that I must instead use FLASH. When I use the CONFIG_ENV_IS_IN_FLASH (and then also have to define CONFIG_CMD_SAVEENV, and CONFIG_CMD_FLASH) my build complains because my board does not define CONFIG_SYS_FLASH_BASE.

Is there something I'm missing here to support CONFIG_ENV_IS_IN_SPI_FLASH? Any advice would be appreciated.

Regarding my additional notes --

- I have successfully provided my own fw_env.config based on the direction in this thread

- The mender.conf file is screwed up on my filesystem for some reason. I am providing the bad mender.conf below

- I had to touch up a bit of the uboot environment patch. My QSPI is partitioned per above, which means that my ubi.mtd kernel command line should be ubi.mtd=2, to select my ubi partition containing the three volumes. Also for some reason my kernel hates when it's passed ubi0_0 as the root= arg. Instead I pass root=ubi0:rootfsa/ubi0:rootfsb. Finally, I noticed an inconsistency on the flash integration manual page https://docs.mender.io/1.2/devices/raw-flash -- the section on U-Boot Integration is missing the ubifsmount command in the ubiboot definition.

Deployed, bad mender.conf:

root@zcu102-zynqmp:~# cat /etc/mender/mender.conf

{

    "InventoryPollIntervalSeconds": 1800,

}"RetryPollIntervalSeconds": 300,

    "RootfsPartA": "ubi0_0",

    "RootfsPartB": "ubi0_1",

    "ServerCertificate": "/etc/mender/server.crt",

    "ServerURL": "https://docker.mender.io",

    "TenantToken": "dummy",

    "UpdatePollIntervalSeconds": 1800

Thanks,

Holden

[Maciej Borzecki]

The error is there because nobody has done integration with SPI flash
yet. So far we support MMC, FLASH, NAND options. You will need to
update this patch:
https://github.com/mendersoftware/meta-mender/blob/master/meta-mender-core/recipes-bsp/u-boot/patches/0002-Generic-boot-code-for-Mender.patch
Once you get it working, feel free to open a PR adding support for Zynq.

>
> Regarding my additional notes --
> - I have successfully provided my own fw_env.config based on the direction
> in this thread
> - The mender.conf file is screwed up on my filesystem for some reason. I am
> providing the bad mender.conf below
> - I had to touch up a bit of the uboot environment patch. My QSPI is
> partitioned per above, which means that my ubi.mtd kernel command line
> should be ubi.mtd=2, to select my ubi partition containing the three
> volumes. Also for some reason my kernel hates when it's passed ubi0_0 as the
> root= arg. Instead I pass root=ubi0:rootfsa/ubi0:rootfsb. Finally, I noticed
> an inconsistency on the flash integration manual page
> https://docs.mender.io/1.2/devices/raw-flash -- the section on U-Boot
> Integration is missing the ubifsmount command in the ubiboot definition.

Indeed it is, thanks for spotting this.

>
>
> Deployed, bad mender.conf:
> root@zcu102-zynqmp:~# cat /etc/mender/mender.conf
> {
> "InventoryPollIntervalSeconds": 1800,
>
> }"RetryPollIntervalSeconds": 300,
> "RootfsPartA": "ubi0_0",
> "RootfsPartB": "ubi0_1",
> "ServerCertificate": "/etc/mender/server.crt",
> "ServerURL": "https://docker.mender.io",
> "TenantToken": "dummy",
> "UpdatePollIntervalSeconds": 1800

Can you check if the file is garbled in rootfs right after do_rootfs
is finished? The image contents will be at
tmp/work/<machine-distro-combo>/<image-name>/1.0-r0/rootfs

> --
> You received this message because you are subscribed to the Google Groups

> "Mender List mender.io" group.

> To unsubscribe from this group and stop receiving emails from it, send an

> email to mender+un...@lists.mender.io.
> To post to this group, send email to men...@lists.mender.io.
> Visit this group at
> https://groups.google.com/a/lists.mender.io/group/mender/.



--
Maciej Borzecki

[holden....@ultra-fei.com]

Hi Maciej,

My apologies for the delay in responding. I had to track down another Xilinx issue with their SPI flash support -- their lock/unlock implementation didn't work properly which was causing both U-Boot and UBI to choke. In any case, I tracked that one down and fixed it. I also disabled the flash locking done by fw_setenv, as that was resulting in locking a portion of the flash where UBIFS was running causing UBIFS corruption.

Anyway, I have updated the generic boot code for mender patch to support CONFIG_ENV_IS_IN_SPI_FLASH basically by just adding the define into the if statement here https://github.com/mendersoftware/meta-mender/blob/master/meta-mender-core/recipes-bsp/u-boot/patches/0002-Generic-boot-code-for-Mender.patch#L44

I still haven't finished testing all of the mender support, but I am now able to read/write the redundant uboot environment from both uboot and linux, where the environment is located in my MTD partition. I'll be testing the actual mender image update functionality early next week.

I took a look at the mender.conf file in the rootfs build directory and everything looks fine there, so I'm not sure why it's garbled when it actually gets deployed. Do you have any more thoughts on this?

Thanks,

Holden

[holden....@ultra-fei.com]

Hi Maciej,

I'm just getting back to this issue today -- trying to track down why I have some garbled files in my rootfs when using the ubimg file produced by mender. Just as a data point, I took the ubi image built by yocto and wrote that image to my flash. In that image the mender config file is as expected. I also noticed a handful of other files which are garbled in the ubimg file. Let me know if you have any additional suggestions based on this new information. In the meantime I'm going to dig in on the ubimg creation and try to track this down.

Thanks,

Holden

[Maciej Borzecki]

On Thu, Oct 5, 2017 at 8:23 PM, <holden....@ultra-fei.com> wrote:
> Hi Maciej,
>
> I'm just getting back to this issue today -- trying to track down why I have
> some garbled files in my rootfs when using the ubimg file produced by
> mender. Just as a data point, I took the ubi image built by yocto and wrote
> that image to my flash. In that image the mender config file is as expected.
> I also noticed a handful of other files which are garbled in the ubimg file.
> Let me know if you have any additional suggestions based on this new
> information. In the meantime I'm going to dig in on the ubimg creation and
> try to track this down.

Have you tried enabling data CRC checks? On the other hand, it would
be weird if data was corrupted but UBIFS was not. The 'ubimg' contains
2 rootfs volumes and a data volume, so will be larger than ubi. I
think it's worth a try to 'ubuformat' with ubi image, then add another
volume and populate it with rootfs' ubifs image.

Can you verify that ubimg uses the same UBINIZE_ARGS and MKUBIFS_ARGS
when building ubimg as for ubi image?

When data is wrong, does it look like it's randomly corrupted, or is
it rather shifted by a multiple of flash page size?

Last but not least, are the same files getting corrupted each time?




--
Maciej Borzecki

[holden....@ultra-fei.com]

Hi Maciej,

As far as I know there is no way to disable CRC checks with UBI. I have UBI compression disabled for the moment, but I think CRC should still be enabled. I don't see any strange errors when attaching my UBI partition or mounting the included volumes.

I tried using ubiformat to write the .ubi file to flash, then attach it and use ubimkvol to create a new volume, but I get an error that there are not enough logical erase blocks --

root@zcu102-zynqmp:~# ubimkvol -N rootfsb -s 50MiB -t dynamic -a 1 /dev/ubi0

ubimkvol: error!: UBI device does not have free logical eraseblocks

I verified that both ubi and ubimg use the same UBINIZE_ARGS and MKUBIFS_ARGS  -- 

ubi creation -- UBINIZE args are -m 1 -p 128KiB -s 1

ubi creation -- MKUBIFS args are -m 1 -e 130944 -c 1008 -x none

mender - UBINIZE args are -m 1 -p 128KiB -s 1

mender - MKUBIFS_ARGS are -m 1 -e 130944 -c 1008 -x none

When the data is garbled, it doesn't look like random corruption to me. You can see in the mender.conf that I posted that it appears the data is swapped around inside the file, but the data itself is still correct. I'm not sure if it's shifted by a full page size though. It is the same files being corrupted every time.

I also for kicks tried using flashcp to erase/write/verify writing the ubimg file and it reports a miscompare during verification-- 

root@zcu102-zynqmp:~# flashcp -v rootfs-image-zcu102-zynqmp.ubimg /dev/mtd2

Erasing blocks: 615/615 (100%)

Writing data: 78720k/78720k (100%)

Verifying data: 63490k/78720k (80%)File does not seem to match flash data. First mismatch at 0x03dfe000-0x03e00800

I may try to reduce my rootfs filesize enough that it does not exceed this memory address and temporarily disable the data partition to see if I still have corruption when this mismatch does not occur.

Thanks,

Holden

[holden....@ultra-fei.com]

Hi Maciej,

I'm still trying to track this down and have some more information now -- 

- Reducing the rootfs size enough that it fit below 0x3dfe000 didn't work -- same result -- garbled mender.conf file 

- I also tried disabling the data partition and the rootfsb partition and matching all .cfg arguments between the ubi and ubimg generation but wound up with the same result there

- Strangely, after building/re-building a bunch of times both .ubi and .ubimg are showing this issue now. No setting changes

- I used ubi_reader to unpack the ubi and ubimg file -- when using this utility both the .ubi and .ubimg files show the expected contents of the mender.conf file -- this says to me that the ubi/ubimg file is likely fine but for some reason after writing the filesystem out to flash and reading it back it appears garbled

- I ran the mtd tests for flash_speed, flash_stress, flash_readtest with all passing

- I used hexdump and grep to locate the mender.conf file within the raw flash. In that segment of memory I see a mender.conf file which looks like it should be fine, but obviously isn't when I cat it -- 

root@zcu102-zynqmp:~# hexdump -C /dev/mtd2 -s 0x027ABE00 -n 700

027abe00  00 00 00 00 00 00 00 00  2e 01 00 00 00 00 00 00  |................|

027abe10  7b 0a 20 20 20 20 22 49  6e 76 65 6e 74 6f 72 79  |{.    "Inventory|

027abe20  50 6f 6c 6c 49 6e 74 65  72 76 61 6c 53 65 63 6f  |PollIntervalSeco|

027abe30  6e 64 73 22 3a 20 31 38  30 30 2c 0a 20 20 20 20  |nds": 1800,.    |

027abe40  22 52 65 74 72 79 50 6f  6c 6c 49 6e 74 65 72 76  |"RetryPollInterv|

027abe50  61 6c 53 65 63 6f 6e 64  73 22 3a 20 33 30 30 2c  |alSeconds": 300,|

027abe60  0a 20 20 20 20 22 52 6f  6f 74 66 73 50 61 72 74  |.    "RootfsPart|

027abe70  41 22 3a 20 22 75 62 69  30 5f 30 22 2c 0a 20 20  |A": "ubi0_0",.  |

027abe80  20 20 22 52 6f 6f 74 66  73 50 61 72 74 42 22 3a  |  "RootfsPartB":|

027abe90  20 22 75 62 69 30 5f 31  22 2c 0a 20 20 20 20 22  | "ubi0_1",.    "|

027abea0  53 65 72 76 65 72 43 65  72 74 69 66 69 63 61 74  |ServerCertificat|

027abeb0  65 22 3a 20 22 2f 65 74  63 2f 6d 65 6e 64 65 72  |e": "/etc/mender|

027abec0  2f 73 65 72 76 65 72 2e  63 72 74 22 2c 0a 20 20  |/server.crt",.  |

027abed0  20 20 22 53 65 72 76 65  72 55 52 4c 22 3a 20 22  |  "ServerURL": "|

027abee0  68 74 74 70 73 3a 2f 2f  64 6f 63 6b 65 72 2e 6d  |https://docker.m|

027abef0  65 6e 64 65 72 2e 69 6f  22 2c 0a 20 20 20 20 22  |ender.io",.    "|

027abf00  54 65 6e 61 6e 74 54 6f  6b 65 6e 22 3a 20 22 64  |TenantToken": "d|

027abf10  75 6d 6d 79 22 2c 0a 20  20 20 20 22 55 70 64 61  |ummy",.    "Upda|

027abf20  74 65 50 6f 6c 6c 49 6e  74 65 72 76 61 6c 53 65  |tePollIntervalSe|

027abf30  63 6f 6e 64 73 22 3a 20  31 38 30 30 0a 7d ff ff  |conds": 1800.}..|

027abf40  31 18 10 06 b0 82 9e a9  a4 2f 00 00 00 00 00 00  |1......../......|

- I found a hacked up version of ubiformat online which includes a "--confirm" option. When using this version of the utility I do get some miscompares. However, when I inspect the miscompared bytes with hexdump after writing out the full ubi/ubimg file the bytes are as I would expect. That is, the miscompares report 0's for some bytes, but the bytes are actually written properly when I look at them later.

I'm running out of ideas regarding what else to try.. any suggestions would be appreciated. I think the next thing I may look into is Xilinx's implementation of IO-Mode for their QSPI controller. They specifically implemented a patch which enables UBI support by switching this controller mode and I'm wondering if they missed something here (https://github.com/Xilinx/linux-xlnx/commit/a800d933b31e37b6020095d3bb2c679403836245#diff-c620405e8ec890034e2e641db4d58d58)

Thanks,

Holden

[Maciej Borzecki]

On Tue, Oct 10, 2017 at 10:49 PM, <holden....@ultra-fei.com> wrote:
> Hi Maciej,
>

The contents look ok.

>
> - I found a hacked up version of ubiformat online which includes a
> "--confirm" option. When using this version of the utility I do get some
> miscompares. However, when I inspect the miscompared bytes with hexdump
> after writing out the full ubi/ubimg file the bytes are as I would expect.
> That is, the miscompares report 0's for some bytes, but the bytes are
> actually written properly when I look at them later.
>
> I'm running out of ideas regarding what else to try.. any suggestions would
> be appreciated. I think the next thing I may look into is Xilinx's
> implementation of IO-Mode for their QSPI controller. They specifically
> implemented a patch which enables UBI support by switching this controller
> mode and I'm wondering if they missed something here
> (https://github.com/Xilinx/linux-xlnx/commit/a800d933b31e37b6020095d3bb2c679403836245#diff-c620405e8ec890034e2e641db4d58d58)
>

The commit message mentions has-io-mode property in DTS. If that is
indeed the case you'll need to add "has-io-mode;" somewhere here:
https://github.com/Xilinx/linux-xlnx/blob/master/arch/arm64/boot/dts/xilinx/zynqmp-zcu102-revA.dts#L850
Note that picked a random dts for ZCU102, there seem to be at least 3
revisions of the board, each covered by a separate dts file. You
should probably find out which one is used in your case and add the
property there.

--
Maciej Borzecki

[holden....@ultra-fei.com]

Hi Maciej,

Thanks again for the support. I've resolved this issue now. It took a while to track this down -- eventually leading me to the QSPI kernel driver from Xilinx. Their driver does not properly account for a usage scenario of their QSPI controller hardware. Some debug info and patch can be found here: https://lists.yoctoproject.org/pipermail/meta-xilinx/2017-October/003224.html

Thanks,

Holden

[Maciej Borzecki]

On Thu, Oct 12, 2017 at 7:31 PM, <holden....@ultra-fei.com> wrote:
> Hi Maciej,
>

> Thanks again for the support. I've resolved this issue now. It took a while
> to track this down -- eventually leading me to the QSPI kernel driver from
> Xilinx. Their driver does not properly account for a usage scenario of their
> QSPI controller hardware. Some debug info and patch can be found here:
> https://lists.yoctoproject.org/pipermail/meta-xilinx/2017-October/003224.html

Nice writeup. Glad that it works now.