Production grade upgrade for mini PC (Intel NUC)

[Piotr Król]

Hi all,
I'm research various approach to system upgrade for consumer product
based on Internet connected Intel NUC. SWUpdate looks great, but I have
some questions.

System features to provide brief description:
* application is GUI based (GTK) with various components (ie. MySQL
database) - lot of different libraries used
* Debian as OS
* Connected on consumer location over Ethernet
* Location distributed geographically
* SATA HDD as storage (with possible improvements to SSD)
* manually and periodically triggered updates
* fallback - there should be always working copy of software
* future need for encryption (TPM, TEE)

Because SATA used I have a lot of space, so dual image approach fit
best I think.

* My hardware configuration do not exactly fall in to "embedded" grade, is
SWUpdate still applicable for those kind of hardware ? Or maybe there
is better approach and I'm missing something.

* Are there any cons of not using Poky as build system ? Building Debian
and all required packages would be complex and time consuming task and
I'm not sure if it is worth invest in this.

* Are there any instruction/tutorial to try/triage SWUpdate ie. with
QEMU ? I would like to setup proof of concept environment.

* What server you can recommend ? I see you mention hawkBit on wiki,
what other configuration worked for you ?

--
Best Regards,
Piotr Król
Embedded Systems Consultant
http://3mdeb.com | @3mdeb_com

[Stefano Babic]

Hi Piotr,

On 05/07/2016 00:40, Piotr Król wrote:
> Hi all,
> I'm research various approach to system upgrade for consumer product
> based on Internet connected Intel NUC. SWUpdate looks great, but I have
> some questions.
>
> System features to provide brief description:
> * application is GUI based (GTK) with various components (ie. MySQL
> database) - lot of different libraries used
> * Debian as OS

Well, it is not exactly for embedded...the size of the whole image is
then very large.

> * Connected on consumer location over Ethernet
> * Location distributed geographically
> * SATA HDD as storage (with possible improvements to SSD)
> * manually and periodically triggered updates
> * fallback - there should be always working copy of software
> * future need for encryption (TPM, TEE)
>
> Because SATA used I have a lot of space, so dual image approach fit
> best I think.

ok

>
> * My hardware configuration do not exactly fall in to "embedded" grade,

Right.

> is
> SWUpdate still applicable for those kind of hardware ?

It is not a choice of hardware. There are some other considerations.

SWUpdate is an image installer, suitable for embedded systems. The
system is completely updated or not, there are not steps in between: the
update is like a transaction and it is "atomic".
If you use it to update a Linux distro, the resulting image is very big,
even if you want to update just a couple of files. Debian and Linux
distros have other ways for updating, and a solution based on a package
manager is more suitable for a Linux-PC / Linux-distro.

I am expecting that the resulting debian image is so big that an update
through SWUpdate could be done only in a LAN, and even in this case it
is very time-expensive.

The other aspect you have to take care updating your embedded device is
that the update must be power-cut proof.

If you have the Intel NUC, I am expecting you have BIOS and Grub as
bootloader. Grub has a fallback facility, but it has not two copies of
the environment, as far as I know (am I wrong ?). Because you have two
copies of the software, you need to update at least the "rootfs=" string
in the kernel command line, that is you need to change grub.cfg.

Because updating the file is not safe for power-cut, you can damage the
file while updating during a power-cut, resulting a bricked device.

In projects based on Intel Chips (mainly Quark), we ported U-Boot to the
device, replacing BIOS and Grub. U-Boot has two copies of the
environment and changing the environment is power-cut safe, that is the
reason.

I have no idea (or I have not found) about any development for Grub to
have a redundant environment - anyway, Grub is thought for PCs, more as
for embedded.

> Or maybe there
> is better approach and I'm missing something.
>
> * Are there any cons of not using Poky as build system ?

Do not ask me - I am a Yocto's fan and I use Yocto in all my projects....

> Building Debian
> and all required packages would be complex and time consuming task and
> I'm not sure if it is worth invest in this.

If you "build", you need to set up the whole Debian buildsystem, and I
am afraid this is not worth. Who does build it with a PC ? Simply use
the distro !

If you switch to Yocto, you have a buildsystem for embedded and you can
fine tune it - it requires also some effort to set up, of course.

>
> * Are there any instruction/tutorial to try/triage SWUpdate ie. with
> QEMU ? I would like to setup proof of concept environment.

You do not need qemu. Of course, a lot of handlers are thought for a
specific hardware. You cannot use the UBI handler if you do not have
UBI. But the raw-handler, for example, can be used with any system.

Just compile SWUpdate native on your PC and run it. It works and I use
it to test not hardware-relevant parts.

>
> * What server you can recommend ? I see you mention hawkBit on wiki,
> what other configuration worked for you ?

Moving from a update agent to a full deployment system is something
recent and introduced just in this release. The implementation is
general, but the "connector" to an external server is just for hawkbit.
Further connectors can be added, and this is also stated in the roadmap.

Anyway, the question is : which deployment systems are currently
developed as FOSS and are in a good status, that makes sense to add a
connector for it ? At the moment, I only see hawkBit, but of course more
other servers could be supported in future.

Best regards,
Stefano Babic

--
=====================================================================
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: +49-8142-66989-53 Fax: +49-8142-66989-80 Email: sba...@denx.de
=====================================================================

[Piotr Król]

On Tue, Jul 05, 2016 at 10:23:36AM +0200, Stefano Babic wrote:
> Hi Piotr,

Hi Stefano,
thanks for taking time to reply.

(...)

> > is
> > SWUpdate still applicable for those kind of hardware ?
>
> It is not a choice of hardware. There are some other considerations.
>
> SWUpdate is an image installer, suitable for embedded systems. The
> system is completely updated or not, there are not steps in between: the
> update is like a transaction and it is "atomic".
> If you use it to update a Linux distro, the resulting image is very big,
> even if you want to update just a couple of files. Debian and Linux
> distros have other ways for updating, and a solution based on a package
> manager is more suitable for a Linux-PC / Linux-distro.

This is my main concern. Using package manager for updates seems to be
much more problematic then images. System use custom software that can
break easily when one of packages have not matching, not tested version.
This is main point you made in presentation and in the wiki:
http://sbabic.github.io/swupdate/overview.html#updating-through-a-package-manager

The biggest problem is "Nightmare for test engineers/support". My
application requires reliability and I cannot allow apt-get based
upgrades, since I loose control over whole system. I would like to have
one testable and not apt-get-upgradable image that went through QA and
is blessed. Then next image will happen in months maybe even year.

This system is connected to internet, so I need security, reliability
upgrades. Sometimes I will need drivers upgrade, because external
peripherals that I use have out of tree drivers and are not shipped with
distro.

Relying on apt-get upgrade seems to me a big pain. Of course I can use
my own repositories and provide only tested packages that were tested,
but there are many problems with that. For example what if use will not
enable device for months (or it was in stock waiting for new owner) and
new upgrades appear on my deb repo ? I have to test all possible upgrade
scenarios.

From my understanding image based upgrade is gold standard (ie. Android
recovery partition).

>
> I am expecting that the resulting debian image is so big that an update
> through SWUpdate could be done only in a LAN, and even in this case it
> is very time-expensive.

Time expensive is not a problem. Image have to be downloaded from
external server, so some security and integrity checks would be needed.

>
> The other aspect you have to take care updating your embedded device is
> that the update must be power-cut proof.

Yes, this is why I think SWUpdate use correct approach. I used exactly
the same approach when developing firmware for TI CC3200 (Cortex-M4).

>
> If you have the Intel NUC, I am expecting you have BIOS and Grub as
> bootloader. Grub has a fallback facility, but it has not two copies of
> the environment, as far as I know (am I wrong ?). Because you have two
> copies of the software, you need to update at least the "rootfs=" string
> in the kernel command line, that is you need to change grub.cfg.
>
> Because updating the file is not safe for power-cut, you can damage the
> file while updating during a power-cut, resulting a bricked device.

This cannot happen in my setup, reliability is very important and
backing to service should be avoided. I didn't found any update
mechanism that use GRUB.

>
> In projects based on Intel Chips (mainly Quark), we ported U-Boot to the
> device, replacing BIOS and Grub. U-Boot has two copies of the
> environment and changing the environment is power-cut safe, that is the
> reason.

Ok, if U-Boot would work with my NUC without major problems, then I
should be good to go with SWUPdate. Am I?

>
> I have no idea (or I have not found) about any development for Grub to
> have a redundant environment - anyway, Grub is thought for PCs, more as
> for embedded.

Exactly.

>
>
> > Or maybe there
> > is better approach and I'm missing something.
> >
> > * Are there any cons of not using Poky as build system ?
>
> Do not ask me - I am a Yocto's fan and I use Yocto in all my projects....

Truly, I probably can get rid of Debian, if all components can be put
together and will work, but number of libraries is huge and I would like
to have basic proof of concept pretty soon.

>
> > Building Debian
> > and all required packages would be complex and time consuming task and
> > I'm not sure if it is worth invest in this.
>
> If you "build", you need to set up the whole Debian buildsystem, and I
> am afraid this is not worth. Who does build it with a PC ? Simply use
> the distro !
>
> If you switch to Yocto, you have a buildsystem for embedded and you can
> fine tune it - it requires also some effort to set up, of course.

It is possible that in future project will change hardware to some ARM
based, then using Poky would simplify things a lot. I would like to try
Poky stuff, but I was tied more to Buildroot and debootstrap until now.

>
> >
> > * Are there any instruction/tutorial to try/triage SWUpdate ie. with
> > QEMU ? I would like to setup proof of concept environment.
>
> You do not need qemu. Of course, a lot of handlers are thought for a
> specific hardware. You cannot use the UBI handler if you do not have
> UBI. But the raw-handler, for example, can be used with any system.
>
> Just compile SWUpdate native on your PC and run it. It works and I use
> it to test not hardware-relevant parts.

Ok, I will try it.

>
> >
> > * What server you can recommend ? I see you mention hawkBit on wiki,
> > what other configuration worked for you ?
>
> Moving from a update agent to a full deployment system is something
> recent and introduced just in this release. The implementation is
> general, but the "connector" to an external server is just for hawkbit.
> Further connectors can be added, and this is also stated in the roadmap.
>
> Anyway, the question is : which deployment systems are currently
> developed as FOSS and are in a good status, that makes sense to add a
> connector for it ? At the moment, I only see hawkBit, but of course more
> other servers could be supported in future.

This area as well as system upgrades is not well developed yet, but in
IoT rush we will need that.

One more time thanks for your time and DENX for supporting this project.

[Stefano Babic]

Hi Piotr,

Right - this is why updating an embedded system differs from updating a PC.

>
> This system is connected to internet, so I need security, reliability
> upgrades. Sometimes I will need drivers upgrade, because external
> peripherals that I use have out of tree drivers and are not shipped with
> distro.
>
> Relying on apt-get upgrade seems to me a big pain. Of course I can use
> my own repositories and provide only tested packages that were tested,
> but there are many problems with that. For example what if use will not
> enable device for months (or it was in stock waiting for new owner) and
> new upgrades appear on my deb repo ? I have to test all possible upgrade
> scenarios.

Exactly - and it is not enough. What happens if just some packages are
updated, and due to network problems, some other not ? And how can your
support help your customer in all conditions ?

>
> From my understanding image based upgrade is gold standard (ie. Android
> recovery partition).

Right

>
>>
>> I am expecting that the resulting debian image is so big that an update
>> through SWUpdate could be done only in a LAN, and even in this case it
>> is very time-expensive.
>
> Time expensive is not a problem. Image have to be downloaded from
> external server, so some security and integrity checks would be needed.

Well, if debian is thought as a whole image, you have an update package
that can be some GB. It is quite hard to fine tuning a standard Linux
distro, and takes much more time as using a buildsystem (Yocto,
Buildroot) that was thought from scratch for this approach.

>
>>
>> The other aspect you have to take care updating your embedded device is
>> that the update must be power-cut proof.
>
> Yes, this is why I think SWUpdate use correct approach. I used exactly
> the same approach when developing firmware for TI CC3200 (Cortex-M4).
>
>>
>> If you have the Intel NUC, I am expecting you have BIOS and Grub as
>> bootloader. Grub has a fallback facility, but it has not two copies of
>> the environment, as far as I know (am I wrong ?). Because you have two
>> copies of the software, you need to update at least the "rootfs=" string
>> in the kernel command line, that is you need to change grub.cfg.
>>
>> Because updating the file is not safe for power-cut, you can damage the
>> file while updating during a power-cut, resulting a bricked device.
>
> This cannot happen in my setup, reliability is very important and
> backing to service should be avoided. I didn't found any update
> mechanism that use GRUB.

Can I ask which is your bootloader on your Intel NUC ? Or this issue was
not yet considered ?

>
>>
>> In projects based on Intel Chips (mainly Quark), we ported U-Boot to the
>> device, replacing BIOS and Grub. U-Boot has two copies of the
>> environment and changing the environment is power-cut safe, that is the
>> reason.
>
> Ok, if U-Boot would work with my NUC without major problems, then I
> should be good to go with SWUPdate. Am I?

I will separate issues: if you need to change setup on the bootloader,
as I think because you need at least to change the rootdevice, the
bootloader must be power-cut safe, independently if you will use
SWUpdate as installer agent or something else.

>
>>
>> I have no idea (or I have not found) about any development for Grub to
>> have a redundant environment - anyway, Grub is thought for PCs, more as
>> for embedded.
>
> Exactly.
>
>>
>>
>>> Or maybe there
>>> is better approach and I'm missing something.
>>>
>>> * Are there any cons of not using Poky as build system ?
>>
>> Do not ask me - I am a Yocto's fan and I use Yocto in all my projects....
>
> Truly, I probably can get rid of Debian, if all components can be put
> together and will work, but number of libraries is huge and I would like
> to have basic proof of concept pretty soon.
>
>>
>>> Building Debian
>>> and all required packages would be complex and time consuming task and
>>> I'm not sure if it is worth invest in this.
>>
>> If you "build", you need to set up the whole Debian buildsystem, and I
>> am afraid this is not worth. Who does build it with a PC ? Simply use
>> the distro !
>>
>> If you switch to Yocto, you have a buildsystem for embedded and you can
>> fine tune it - it requires also some effort to set up, of course.
>
> It is possible that in future project will change hardware to some ARM
> based, then using Poky would simplify things a lot. I would like to try
> Poky stuff, but I was tied more to Buildroot and debootstrap until now.

SWUpdate thanks to contibutors is supported by Buildroot, too.

>
>>
>>>
>>> * Are there any instruction/tutorial to try/triage SWUpdate ie. with
>>> QEMU ? I would like to setup proof of concept environment.
>>
>> You do not need qemu. Of course, a lot of handlers are thought for a
>> specific hardware. You cannot use the UBI handler if you do not have
>> UBI. But the raw-handler, for example, can be used with any system.
>>
>> Just compile SWUpdate native on your PC and run it. It works and I use
>> it to test not hardware-relevant parts.
>
> Ok, I will try it.
>
>>
>>>
>>> * What server you can recommend ? I see you mention hawkBit on wiki,
>>> what other configuration worked for you ?
>>
>> Moving from a update agent to a full deployment system is something
>> recent and introduced just in this release. The implementation is
>> general, but the "connector" to an external server is just for hawkbit.
>> Further connectors can be added, and this is also stated in the roadmap.
>>
>> Anyway, the question is : which deployment systems are currently
>> developed as FOSS and are in a good status, that makes sense to add a
>> connector for it ? At the moment, I only see hawkBit, but of course more
>> other servers could be supported in future.
>
> This area as well as system upgrades is not well developed yet, but in
> IoT rush we will need that.
>
> One more time thanks for your time and DENX for supporting this project.
>

You're welcome.

[Bjørn Forsman]

I'd like to mention NixOS [1], because it is a distro that features
atomic upgrades (more or less) and rollbacks.
I wouldn't necessarily call it to be as robust as swupdate + uboot
with dual environment (I haven't reviewed the "atomic-ness" of the
code that updates the grub menu), but in my experience it way better
than "normal" distros (Debian, Redhat,...) when it comes to safe
upgrades.

[1] http://nixos.org/

Best regards,
Bjørn Forsman

[Bjørn Forsman]

On 5 July 2016 at 15:51, Bjørn Forsman <bjorn....@gmail.com> wrote:
> I wouldn't necessarily call it to be as robust as swupdate + uboot
> with dual environment (I haven't reviewed the "atomic-ness" of the
> code that updates the grub menu), but in my experience it way better
> than "normal" distros (Debian, Redhat,...) when it comes to safe
> upgrades.

I had to check, and the NixOS update procedure is atomic. After all
packages are updated (not in-place, so it is completely safe), the
final step, and the only step that will have an effect on how the
system will boot the next time (which "generation" of the system will
be booted), is the rename operation to move /boot/grub/grub.cfg.tmp to
/boot/grub/grub.cfg.

- Bjørn

[Piotr Król]

On Tue, Jul 05, 2016 at 03:49:33PM +0200, Stefano Babic wrote:
> Hi Piotr,

Hi Stefano,

(...)

> >
> > Relying on apt-get upgrade seems to me a big pain. Of course I can use
> > my own repositories and provide only tested packages that were tested,
> > but there are many problems with that. For example what if use will not
> > enable device for months (or it was in stock waiting for new owner) and
> > new upgrades appear on my deb repo ? I have to test all possible upgrade
> > scenarios.
>
> Exactly - and it is not enough. What happens if just some packages are
> updated, and due to network problems, some other not ? And how can your
> support help your customer in all conditions ?

Agree this important point. This case should be avoided and only image
based update with fixed software seems to support that.

(...)

> >
> > Time expensive is not a problem. Image have to be downloaded from
> > external server, so some security and integrity checks would be needed.
>
> Well, if debian is thought as a whole image, you have an update package
> that can be some GB. It is quite hard to fine tuning a standard Linux
> distro, and takes much more time as using a buildsystem (Yocto,
> Buildroot) that was thought from scratch for this approach.

Yes, for sure this leads me to conclusion that I have to seriously
consider Poky as build system for whole image. This will take a lot of
time, but will lead to much smaller images. Of course compiling all
libraries for X window system will be painful.

I wonder if there are ready to use meta for that.

(...)

> >
> > This cannot happen in my setup, reliability is very important and
> > backing to service should be avoided. I didn't found any update
> > mechanism that use GRUB.
>
> Can I ask which is your bootloader on your Intel NUC ? Or this issue was
> not yet considered ?

I have Debian installed with GRUB 2.02.

>
> >
> >>
> >> In projects based on Intel Chips (mainly Quark), we ported U-Boot to the
> >> device, replacing BIOS and Grub. U-Boot has two copies of the
> >> environment and changing the environment is power-cut safe, that is the
> >> reason.
> >
> > Ok, if U-Boot would work with my NUC without major problems, then I
> > should be good to go with SWUPdate. Am I?
>
> I will separate issues: if you need to change setup on the bootloader,
> as I think because you need at least to change the rootdevice, the
> bootloader must be power-cut safe, independently if you will use
> SWUpdate as installer agent or something else.

Agree, bootloader situation looks like the biggest problem. NUC is UEFI
based, so the only way is to have power-cut safe UEFI-aware bootloader.
Typical bootloader used with UEFI systems is GRUB. From what I see in
google U-Boot is probably not the best fit for my NUC, am I wrong ?

According to Bjorn suggestion for NixOS it is possible to achieve that
with GRUB I probably have to look at their implementation. I don't think
I would like use NixOS, but their approach maybe ported somehow.

[Stefano Babic]

Hi Piotr,

On 06/07/2016 17:23, Piotr Król wrote:

>>> Time expensive is not a problem. Image have to be downloaded from
>>> external server, so some security and integrity checks would be needed.
>>
>> Well, if debian is thought as a whole image, you have an update package
>> that can be some GB. It is quite hard to fine tuning a standard Linux
>> distro, and takes much more time as using a buildsystem (Yocto,
>> Buildroot) that was thought from scratch for this approach.
>
> Yes, for sure this leads me to conclusion that I have to seriously
> consider Poky as build system for whole image. This will take a lot of
> time, but will lead to much smaller images. Of course compiling all
> libraries for X window system will be painful.
>
> I wonder if there are ready to use meta for that.

Well, xserver-xorg is part of standard meta, as well as wayland. Gtk,
QT, are also supported out of the box.

>> I will separate issues: if you need to change setup on the bootloader,
>> as I think because you need at least to change the rootdevice, the
>> bootloader must be power-cut safe, independently if you will use
>> SWUpdate as installer agent or something else.
>
> Agree, bootloader situation looks like the biggest problem. NUC is UEFI
> based, so the only way is to have power-cut safe UEFI-aware bootloader.
> Typical bootloader used with UEFI systems is GRUB. From what I see in
> google U-Boot is probably not the best fit for my NUC, am I wrong ?

Check doc/README.efi in U-Boot sources. Nevertheless, there are a lot of
patches in last time about EFI. My suggestion is to send your questions
directly to U-Boot ML - Simon and Bing are X86 maintainers and they can
answer better.

>
> According to Bjorn suggestion for NixOS it is possible to achieve that
> with GRUB I probably have to look at their implementation. I don't think
> I would like use NixOS, but their approach maybe ported somehow.