failed to upgrade Raspberry Pi 3 target from 1.4 to 1.6

[Antonio Santagiuliana]

Hello

I had a Raspberry Pi3 running Roku Yocto + mender 1.4 ( let's call this system 1.4 Pi )

I could upgrade this device using mender server 1.4 with a new artifact.

Everything was working as expected.

Now I have updated my development system to Yocto Sumo , meta-raspberry Pi Sumo and Mender server 1.6.

I recreated successfully the image for the same application with these new versions.

But when I try upgrade that Raspberry Pi3, that still has 1.4 onboard, with the new artifact for migrating it to the 1.6+Sumo I get the upgrade fails and there is a rollback to previous onboard version.

From the log I can see error :

 update info for deplyment xxxx present , but update flag is not set; running rollback image ( previous active partition )

I think that the only setting required to be changed when migrating to 1.6 Sumo was about the MENDER_PARITION_ALIGNMENT theat changed from MENDER_PARTITION_ALIGNMENT_KB =" 4096" to MENDER_PARTITION_ALIGNMENT = "4194304"   

Anyway if i start with 1.6 flashed on the sd card I can upgrade successfully the image through the mender server to a new artifact, so it looks something incompatible when passing the target from 1.4 to 1.6 upgrading though the mender server.

Should mender1.4+Rocku on target be compatible with an upgrade to mender1.6 + Sumo ? 

Or could it be something different in Uboot that makes this upgrade not possible  ?

Thank you

Antonio

[Mirza Krak]

Interesting. Something similar was reported in another thread, that is:

    update info for deplyment xxxx present , but update flag is not set; running rollback image ( previous active partition )

But we never found the root cause, but we never did try updating from different versions. I will have to try a 1.4 -> 1.6 update and analyze closer.

I can not think of anything that should prevent an update from 1.4 (rocko) to 1.6 (sumo), it should be possible.

--

Mirza Krak | Embedded Solutions Architect | https://mender.io

 Northern.tech AS | @northerntechHQ

[Antonio Santagiuliana]

From what I could see the problem is repeatable, under the same conditions. I will send update if I find something new.

--
You received this message because you are subscribed to the Google Groups "Mender List mender.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mender+unsubscribe@lists.mender.io.
To post to this group, send email to men...@lists.mender.io.
Visit this group at https://groups.google.com/a/lists.mender.io/group/mender/.

[Wesly Grefrath]

On 08-08-18 13:46, Mirza Krak wrote:

but we never did try updating from different versions.

Wait, what? Isn't the entire idea to be able to upgrade the entire system? Shouldn't we be making sure upgrading from an OS with version X of Mender to an OS with version Y of Mender works correctly and is tested correctly?

Is there anything we can do to help with that as I would assume being able to do that is a pretty big necessity. Or am I missing something?

Wes

-- 
Eyo Interactive

Heistraat 3
5701HG Helmond
The Netherlands

Tel: 0031 492 317 462

http://www.eyo.nl

[Mirza Krak]

On Wed, Aug 8, 2018 at 3:12 PM, Wesly Grefrath <we...@eyo.nl> wrote:

On 08-08-18 13:46, Mirza Krak wrote:

but we never did try updating from different versions.

Wait, what? Isn't the entire idea to be able to upgrade the entire system? Shouldn't we be making sure upgrading from an OS with version X of Mender to an OS with version Y of Mender works correctly and is tested correctly?

Is there anything we can do to help with that as I would assume being able to do that is a pretty big necessity. Or am I missing something?

We do have pretty good test coverage which is run on every single pull-request. This includes unit-tests on the Mender client, acceptance tests and integration tests on multiple targets, this includes performing updates and much more. But as it is with tests there is always room for improvement. This is all open-source and anyone can review, comment or improve the coverage.

But as it is now we are only testing incremental changes and maybe we are lacking testing "major version" changes as described by Antonio, as I said always room for improvements.

The community can always help with out with testing and reporting back any issues that are found as test-suits will never be 100 %. This is a pretty good example of how we collectively can improve a product that is essential to us all, by providing detailed rapports and even instructions on how to re-produce the problem that should result in a swift fix.

Contributions are always appreciated and we are are always open for suggestions on improvements.

-- 

[Matthijs ter Woord]

I would definitely expect certain upgrade paths to be supported by mender. (Not saying its bad that some paths aren't test)

Testing this automatically should theoretically be possible with for example a micro-sd card switch, some relay boards, etc. Sounds like a fun project to make possible..

Op wo 8 aug. 2018 om 15:40 schreef Mirza Krak <mirza...@northern.tech>:

--
You received this message because you are subscribed to the Google Groups "Mender List mender.io" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mender+un...@lists.mender.io.

[Wesly Grefrath]

Hey Mirza,

I did not mean to make a snarky remark though, after reading it back, it can be read like that, that was not my intention. I truly appreciate Mender and the effort everyone puts in. Just was wondering if there's anything that would help overcome these issues as certainly major upgrades  are very important.

I will do my best in reporting any issues that we may catch.

Thanks,
Wes

[Mirza Krak]

On Wed, Aug 8, 2018 at 3:47 PM, Matthijs ter Woord <matthijs...@gmail.com> wrote:

I would definitely expect certain upgrade paths to be supported by mender. (Not saying its bad that some paths aren't test)

Testing this automatically should theoretically be possible with for example a micro-sd card switch, some relay boards, etc. Sounds like a fun project to make possible..

So far we have been running our tests on virtual machines (qemu arm/x86-64) which allows us to perform update operations without the need of any actual hardware. You can check out the tests here [1] and specifically test_update.py [2]

We are also able to run these tests on BBB and Raspberrypi, but due to not having the things you mentioned (micro-sd card switch, relays etc) only some parts are able to run and manual intervention is needed if the test actually fail to restore the device. 

This is why we started this project that is in progress, https://tracker.mender.io/browse/MEN-1815. 

If you are interested in helping out, please let me know how we can enable you :).

[1]. https://github.com/mendersoftware/meta-mender/tree/master/tests/acceptance

[2]. https://github.com/mendersoftware/meta-mender/blob/master/tests/acceptance/test_update.py

-- 

[Matthijs ter Woord]

I wish I had the time....

Rushing a customer project out as we speak..

But with the hardware mentioned there, and some USB-switched power outlets, it shouldn't be too hard with some custom code..

Op do 9 aug. 2018 om 09:41 schreef Mirza Krak <mirza...@northern.tech>:

[Mirza Krak]

On Wed, Aug 8, 2018 at 1:46 PM, Mirza Krak <mirza...@northern.tech> wrote:

On Wed, Aug 8, 2018 at 1:00 PM, Antonio Santagiuliana <santant...@gmail.com> wrote:

Hello

I had a Raspberry Pi3 running Roku Yocto + mender 1.4 ( let's call this system 1.4 Pi )

I could upgrade this device using mender server 1.4 with a new artifact.

Everything was working as expected.

Now I have updated my development system to Yocto Sumo , meta-raspberry Pi Sumo and Mender server 1.6.

I recreated successfully the image for the same application with these new versions.

But when I try upgrade that Raspberry Pi3, that still has 1.4 onboard, with the new artifact for migrating it to the 1.6+Sumo I get the upgrade fails and there is a rollback to previous onboard version.

From the log I can see error :

 update info for deplyment xxxx present , but update flag is not set; running rollback image ( previous active partition )

I think that the only setting required to be changed when migrating to 1.6 Sumo was about the MENDER_PARITION_ALIGNMENT theat changed from MENDER_PARTITION_ALIGNMENT_KB =" 4096" to MENDER_PARTITION_ALIGNMENT = "4194304"   

Anyway if i start with 1.6 flashed on the sd card I can upgrade successfully the image through the mender server to a new artifact, so it looks something incompatible when passing the target from 1.4 to 1.6 upgrading though the mender server.

Should mender1.4+Rocku on target be compatible with an upgrade to mender1.6 + Sumo ? 

Or could it be something different in Uboot that makes this upgrade not possible  ?

Interesting. Something similar was reported in another thread, that is:

    update info for deplyment xxxx present , but update flag is not set; running rollback image ( previous active partition )

But we never found the root cause, but we never did try updating from different versions. I will have to try a 1.4 -> 1.6 update and analyze closer.

Took a bit longer then anticipated to set this up, but I can now reproduce this as well. Analyzing... 

-- 

[Mirza Krak]

Ok, now I see it. This is not related to Mender client update from 1.4 -> 1.6 and it is the Rocko -> Sumo update that is causing issues. Will try and explain.

First of the error message that the Mender client produces in this case is somewhat misleading.  

What happens when updating from Rocko -> Sumo is that the boot candidate based on Sumo will hang during boot with the following error:

     [    1.953850] Waiting for root device /dev/mmcblk0p3...

And it never gets past this, so simply a boot failure of the system and hence the roll-back.

One of the reasons is this note from the meta-mender-raspberrypi [1]:

NOTE!. To be able to support update of Linux kernel and DTB, Mender requires these to be installed in the /boot directory for each rootfs (normally /dev/mmcblk0p2 and /dev/mmcblk0p3). On the other hand, the Raspberry Pi boot firmware requires that the DTB file is in the same partition as the boot firmware (/dev/mmcbl0p1) and the config.txt file. For now Mender will not use the DTB that is delivered with new artifacts and will continue to boot with the original DTB that was populated using the SDIMG file.

So in the Rocko -> Sumo transition the Linux kernel for the Raspberry Pi was updated from 4.9 -> 4.14, this probably introduced an incompatible change to the device-tree. This is actually illegal to do per device tree Linux upstream policies, that is making breaking changes to device-tree files, but as the Raspberry Pi Linux kernels are based on downstream forks they can do what ever they want and cause problems for us :).

Because you device was "provisioned" with Rocko and since we do not update the DTB files as part of Mender update due to above note, your device has a device-tree from the 4.9 Linux kernel and when an artifact based on Sumo is flashed it will fail to boot, because the Linux kernel and the device-tree will not be compatible.

This is a limitation isolated to the Raspberry Pi boards due its unconventional boot process and there is not really much for us to do but to say that the Rocko -> Sumo update path is not supported OOB.

A couple of workarounds:

- Should work to build the Sumo branch with the 4.9 Linux kernel, they still have that recipe in meta-raspberrypi

   - But problem with this is that you must use 4.9 Kernel forever basically and this is not optimal

- Update DTB in vfat part using a state-script, a bit of a "hack" and you will not be able to perform an roll-back once this is done.

I haven't actually done a "diff" on the device trees between the 4.9 and 4.14 to check what specifically changed, but this could provide some extra insight. 

[1]. https://github.com/mendersoftware/meta-mender/tree/master/meta-mender-raspberrypi

-- 

[Antonio Santagiuliana]

Thank you for your quick response. This is interesting.

You say this is not related to the Mender client update from 1.4 to 1.6 but to the update from Rocko to Sumo.

I understand this, however the two are actually related, because Mender 1.6 means using Sumo, at least from the docs and Sumo brings in this new Kernel version by default, at least for the Raspberry Pi. Anyway I was trying updating to Sumo because of an issue in Rocko.

In any case the finding is not very good as there could be a situation in which you have many devices already installed and you cannot upgrade them to a most recent version of kernel.

Let's say if a security problem or a bug is discovered in the kernel on devices already installed, you have to rely on the availability of a patch for them carried out using Mender, rather than heading to a complete kernel version upgrade, if this implies a kernel incompatibility due to DTB.

In the long term this could mean old devices already installed may or may not be fixable,  apart if using the last workaround you listed.

I will check the differences between the two DT.

Thank you

Antonio

[Mirza Krak]

On Sat, Aug 11, 2018 at 4:58 PM, Antonio Santagiuliana <santant...@gmail.com> wrote:

Thank you for your quick response. This is interesting.

You say this is not related to the Mender client update from 1.4 to 1.6 but to the update from Rocko to Sumo.

I understand this, however the two are actually related, because Mender 1.6 means using Sumo, at least from the docs and Sumo brings in this new Kernel version by default, at least for the Raspberry Pi. Anyway I was trying updating to Sumo because of an issue in Rocko.

Mender builds on top of many upstream repositories such as meta-raspberrypi which in turn relies on basically everything that is in here: https://github.com/raspberrypi

If these upstream repositories "miss behave" and break things between updates, we will inherit all that. That is how it simply works and not much we can do. We are not the provider of the complete software stack that is running on the raspberrpi and we simple provide an integration of Mender on top of the existing upstream repositories.

 

In any case the finding is not very good as there could be a situation in which you have many devices already installed and you cannot upgrade them to a most recent version of kernel.

Let's say if a security problem or a bug is discovered in the kernel on devices already installed, you have to rely on the availability of a patch for them carried out using Mender, rather than heading to a complete kernel version upgrade, if this implies a kernel incompatibility due to DTB.

In the long term this could mean old devices already installed may or may not be fixable,  apart if using the last workaround you listed.

I will check the differences between the two DT.

Yeah I understand all the problems this implies but there is not much we can do about it unless upstream [1] and [2] keep things stable across updates in their boot firmware. The fact that there is a boot firmware is the root of all this and that it keeps breaking. There was a similar problem between morty and pyro.

And unfortunately I do not believe it is enough to update just the DTB but all boot firmware files must me updated as well, as the "old" firmware files from Rocko do not understand the DTB format that is in Sumo. Basically everything that is in the vfat boot partition.

And there is no reliable way of updating the files in vfat, meaning that you _can_ update them but you run the risk of "bricking" the device. 

This is best to bring up with the folks maintaining the different raspberrypi components, e.g the firmware files [1] to get their opinion on why they do breaking changes of the boot firmware. Maybe it is not intentional and this is a bug, but again best to bring it up with them.

[1]. https://github.com/raspberrypi/firmware

[2]. https://github.com/raspberrypi/linux

-- 

[Mirza Krak]

On Sat, Aug 11, 2018 at 4:58 PM, Antonio Santagiuliana <santant...@gmail.com> wrote:

Thank you for your quick response. This is interesting.

You say this is not related to the Mender client update from 1.4 to 1.6 but to the update from Rocko to Sumo.

I understand this, however the two are actually related, because Mender 1.6 means using Sumo, at least from the docs and Sumo brings in this new Kernel version by default, at least for the Raspberry Pi. Anyway I was trying updating to Sumo because of an issue in Rocko.

In any case the finding is not very good as there could be a situation in which you have many devices already installed and you cannot upgrade them to a most recent version of kernel.

Let's say if a security problem or a bug is discovered in the kernel on devices already installed, you have to rely on the availability of a patch for them carried out using Mender, rather than heading to a complete kernel version upgrade, if this implies a kernel incompatibility due to DTB.

In the long term this could mean old devices already installed may or may not be fixable,  apart if using the last workaround you listed.

I will check the differences between the two DT.

This is what seems to be the problem when using a "rocko" DTB on a "sumo" kernel:

[    1.953850] Waiting for root device /dev/mmcblk0p3...

[    2.031246] usb 1-1: new high-speed USB device number 2 using dwc_otg

[    2.037889] Indeed it is in host mode hprt0 = 00001101

[    2.119841] mmc0: new high speed SDIO card at address 0001

[    2.142763] mmc1: host does not support reading read-only switch, assuming write-enable

[    2.157590] mmc1: new high speed SDHC card at address aaaa

[    2.163804] mmcblk1: mmc1:aaaa SC16G 14.8 GiB

[    2.173902]  mmcblk1: p1 p2 p3 p4

The mmc indexing has changed and the SD card is on mmcblk1, and hence the halt on "waiting for root device /dev/mmcblk0p3"

But this does not seem to be the case when using a "sumo DTB" with a "sumo kernel", as you have noticed as well.

We recently had a discussion regarding unstable device naming [1], and the ideas that I suggested there could have remedied this problem.

[1]. https://groups.google.com/a/lists.mender.io/forum/#!topic/mender/v6PqNueAegU

-- 

[Mirza Krak]

On Sat, Aug 11, 2018 at 4:58 PM, Antonio Santagiuliana <santant...@gmail.com> wrote:

Thank you for your quick response. This is interesting.

You say this is not related to the Mender client update from 1.4 to 1.6 but to the update from Rocko to Sumo.

I understand this, however the two are actually related, because Mender 1.6 means using Sumo, at least from the docs and Sumo brings in this new Kernel version by default, at least for the Raspberry Pi. Anyway I was trying updating to Sumo because of an issue in Rocko.

In any case the finding is not very good as there could be a situation in which you have many devices already installed and you cannot upgrade them to a most recent version of kernel.

Let's say if a security problem or a bug is discovered in the kernel on devices already installed, you have to rely on the availability of a patch for them carried out using Mender, rather than heading to a complete kernel version upgrade, if this implies a kernel incompatibility due to DTB.

In the long term this could mean old devices already installed may or may not be fixable,  apart if using the last workaround you listed.

I will check the differences between the two DT.

To verify my theories I created this, https://github.com/mirzak/meta-mender/commits/rpi-boot-part-update

Using above I can update from Rocko -> Sumo. But it is an operation that can potentially "brick" devices and roll-back is not possible.

[Matthijs ter Woord]

Not being too familiar with linux kernels and device trees, just my 2 cents here....

I did some googling, and [1] suggests that it should be possible to make raspberry kernels without using dtb files. Is this correct?

Would that then solve all (or at least most) of the update issues?

[1] https://github.com/raspberrypi/firmware/blob/master/boot/overlays/README

Op ma 13 aug. 2018 om 10:08 schreef Mirza Krak <mirza...@northern.tech>:

[Kristian Amlie]

On 13/08/18 10:07, Mirza Krak wrote:
> To verify my theories I created
> this, https://github.com/mirzak/meta-mender/commits/rpi-boot-part-update
>
> Using above I can update from Rocko -> Sumo. But it is an operation that
> can potentially "brick" devices and roll-back is not possible.

Note that if you make a copy of the existing files, and add a restore
operation in an ArtifactRollback script, then you can make it somewhat
(but not perfectly) rollback-able. It won't work if the kernel crashes,
but if there is a different problem while in userspace, then Mender will
restore the files before rolling back.

You can then delete the backup in ArtifactCommit_Leave, when you know
that the update was successful.

--
Kristian

[Mirza Krak]

On Mon, Aug 13, 2018 at 10:50 AM, Matthijs ter Woord <matthijs...@gmail.com> wrote:

Not being too familiar with linux kernels and device trees, just my 2 cents here....

I did some googling, and [1] suggests that it should be possible to make raspberry kernels without using dtb files. Is this correct?

Not anymore. There used to be Linux kernels without DTB support (old board c files model) but nowadays Raspberry Pi Linux kernels are closer to upstream kernel and then it is DTB.

Though it is quite easy to get rid of the .dtb file, one can use the CONFIG_ARM_APPENDED_DTB [1] Linux kernel option.

 

Would that then solve all (or at least most) of the update issues?

But it does not really solve the problems on Rapberry Pi to get rid of the DTB file or put it in rootfs, because you can put it in rootfs. We earlier had the DTB on rootfs like it is done on "normal" boards but changed it here [2] 

The problem is the "boot firmware" which does this these magic changes on the DTB which you lose if you do not use that file (which is done today). MAC address is one thing and I do not really know what it does more as these are closed source and with no documentation.

It is hard to make a generic implementation here. For some it might work to load the DTB from rootfs and ignore all the changes the boot firmware does on the DTB.

[1]. https://cateee.net/lkddb/web-lkddb/ARM_APPENDED_DTB.html

[2]. https://github.com/mendersoftware/meta-mender/commit/264cf2ee4c12e072a1343e0102dfc70a35ee93c5#diff-c91144d73d22cd6fa32d45e56576dc04

-- 

[Matthijs ter Woord]

So, what raspberry-like devices are out there which doe support "clean" upgrades and mender?

Op ma 13 aug. 2018 om 11:13 schreef Mirza Krak <mirza...@northern.tech>:

[Mirza Krak]

On Mon, Aug 13, 2018 at 11:26 AM, Matthijs ter Woord <matthijs...@gmail.com> wrote:

So, what raspberry-like devices are out there which doe support "clean" upgrades and mender?

Basically everything should work just fine beside the Raspberry Pi boards. These are the only ones that I know of that use some kind of boot firmware in vfat boot part, which complicate the whole "keeping your software up to date" mindset.

There is no shortage of Raspberry Pi clones. Did you want specifics? :)

-- 

[Matthijs ter Woord]

Specifics, yes please. (Although I'm not meaning to hijack this thread)

I'm looking for:

- "cheap" boards (raspberry pi range, up to 50-60 US preferrably)

- capable of fully-safe updating.

- performance specs similar or better

- decent usb capabilities

bonus points for (either direct or via expansion):

- 24-volt power option

- ups-like support

I'm not too familiar with other boards, but if you know of any without digging too deep, i would be happy to hear about that.

The software part for pi clones is the same as the pi? (ie, mender/yocto)

With kind regards,

Matthijs ter Woord

Op ma 13 aug. 2018 om 11:44 schreef Mirza Krak <mirza...@northern.tech>:

[Mirza Krak]

On Mon, Aug 13, 2018 at 11:52 AM, Matthijs ter Woord <matthijs...@gmail.com> wrote:

Specifics, yes please. (Although I'm not meaning to hijack this thread)

I'm looking for:

- "cheap" boards (raspberry pi range, up to 50-60 US preferrably)

- capable of fully-safe updating.

- performance specs similar or better

- decent usb capabilities

Just at the top of my head without going in to many details:

- OrangePi

- NanoPi

- Asus Tinkerboard

- ODROID

- BananaPi

- OLinuXino

They are all similar or better spec or have variants that are. If you do some more research you will find 10 more :).

Some are probably more expensive due to having on-board eMMC, e.g OrangePi and Tinkerboard (new version, older version is uSD only)

bonus points for (either direct or via expansion):

- 24-volt power option

- ups-like support

I'm not too familiar with other boards, but if you know of any without digging too deep, i would be happy to hear about that.

The software part for pi clones is the same as the pi? (ie, mender/yocto)

Most of the clones are somewhat well supported in Yocto, not all/none have Mender integration complete. There is something for the OrangePi in meta-mender.

Note, that I have an OrangePi PC Plus, NanoPi M1 Plus and a Tinkerboard on my desk.

[Antonio Santagiuliana]

thank you very much, all of these are extremely useful information