New Feature Contributions

[Rob Lee]

Hi, the company I work for is evaluating the possible use of SWUpdate for system OTA updates for a particular project.  There are a few missing features though that we need to develop.  Some of these are already listed on the SWUpdate roadmap, so I wanted to first check the status of any planning or development that may have already been performed towards these features:

Selective Downloading - In the case of a power event or other critical failure, we require the ability to continue an update where it last left off.

Other feature support we might consider adding is "Support for further compressors (in particular xz) for the individual images.  And this would need to support the "streaming" feature (aka, write data to partition progressively).  We may also be interested in adding support for other SWU archive formats other than CPIO such as zip archive, tar.gz, etc.

Can anyone tell me if there are existing plans or development towards these features?  And if so, can you point me to it or any related discussion and documentation?

Thanks,

Rob

[Stefano Babic]

Hi Rob,

On 20.05.21 18:51, Rob Lee wrote:
> Hi, the company I work for is evaluating the possible use of SWUpdate
> for system OTA updates for a particular project.  There are a few
> missing features though that we need to develop.  Some of these are
> already listed on the SWUpdate roadmap, so I wanted to first check the
> status of any planning or development that may have already been
> performed towards these features:
>
> Selective Downloading - In the case of a power event or other critical
> failure, we require the ability to continue an update where it last left
> off.

This is not "selective downloading" and the feature you require is
already implemented. SWUpdate can store the incoming SWU and can resume
after a poweroff, and this feature is already mainlined.

See :
--cache <file> Use cache file as starting SWU

The feature is implemented saving the incoming SWU and using it as cache
file if a power cut occurs. SWUpdate will consume the cached file and
download just the rest.

The selective downloading is a very different feature. There is for it a
RFC with discussion on the ML:

https://groups.google.com/g/swupdate/c/yCDBnWip5xw/m/EFUD59QhCQAJ

The status for that is that I have a customer requiring it, and I am
waiting for his order to start the implementation.

>
> Other feature support we might consider adding is "Support for further
> compressors (in particular xz) for the individual images.

This can be done, it has a very lower priority after the "zst"
compressor was added. XZ (lzma) can compress better, it requires a
higher CPU load - ZST seems at the moment the best compromise.

>  And this
> would need to support the "streaming" feature (aka, write data to
> partition progressively).

This is a must, both gzip and zst compressor are done in this way.
Adding XZ is IMHO not a big deal as there is a library, but as I said,
this has not a high priority in the project. A new compressor means to
rework the pipeline (see copyfile function and similar), and this means
higher efforts for regression tests.

I have no customers requiring this, that means I do not plan to work on it.

>  We may also be interested in adding support
> for other SWU archive formats other than CPIO such as zip archive,
> tar.gz, etc.

SWUpdate does not really use CPIO, but just a simplified version as
container. Files are flat, and so on. There is no reason to change this
and I will probably reject such as changes.

>
> Can anyone tell me if there are existing plans or development towards
> these features?  And if so, can you point me to it or any related
> discussion and documentation?

See above.

Best regards,
Stefano Babic

>
> Thanks,
> Rob
>
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "swupdate" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to swupdate+u...@googlegroups.com
> <mailto:swupdate+u...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/swupdate/CAH593dC-v-Z-khQv-k9q%2BRSds27hKpsi4B632%3Dg5zJBpEDeoPA%40mail.gmail.com
> <https://groups.google.com/d/msgid/swupdate/CAH593dC-v-Z-khQv-k9q%2BRSds27hKpsi4B632%3Dg5zJBpEDeoPA%40mail.gmail.com?utm_medium=email&utm_source=footer>.

--
=====================================================================
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: +49-8142-66989-53 Fax: +49-8142-66989-80 Email: sba...@denx.de
=====================================================================

[James Hilliard]

FYI I currently use swupdate with xz compression since the archive handler
which uses libarchive can be built with xz support. I pack my rootfs as a
rootfs.tar.xz essentially then have swupdate extract it to the partition.

So while swupdate currently supports xz compression it appears to be
limited to use with the archive handler.

> To unsubscribe from this group and stop receiving emails from it, send an email to swupdate+u...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/swupdate/bf8f2ab0-774f-deea-f781-60479910d2f8%40denx.de.

[Rob Lee]

Hi Stefano, thanks for the quick response!

Some comments below.

On Thu, May 20, 2021 at 4:37 PM Stefano Babic <stefan...@denx.de> wrote:
>
> Hi Rob,
>
> On 20.05.21 18:51, Rob Lee wrote:
> > Hi, the company I work for is evaluating the possible use of SWUpdate
> > for system OTA updates for a particular project. There are a few
> > missing features though that we need to develop. Some of these are
> > already listed on the SWUpdate roadmap, so I wanted to first check the
> > status of any planning or development that may have already been
> > performed towards these features:
> >
> > Selective Downloading - In the case of a power event or other critical
> > failure, we require the ability to continue an update where it last left
> > off.
>
> This is not "selective downloading" and the feature you require is
> already implemented. SWUpdate can store the incoming SWU and can resume
> after a poweroff, and this feature is already mainlined.
>
> See :
> --cache <file> Use cache file as starting SWU
>
> The feature is implemented saving the incoming SWU and using it as cache
> file if a power cut occurs. SWUpdate will consume the cached file and
> download just the rest.
>

Our requirement is to *not* cache the entire SWU (we have large update
images, and
limited storage space) but instead
use the installed-directly flag to write directly to storage as the data becomes
available. And if this is interrupted by a power event, then later
the installation
should continue from where it left off. Our devices operate in regions
with metered or poor data connections so avoiding the download of
unnecessary data is important for our customers. This could work using
a progress "bookmark"
be stored for each chunk of data stored so the system could know where
to continue
from and allow the downloader to pull data from the proper bookmarked
location in
the power event and resume case. In our case, we'll be using own downloader and
we use block-based image updates to partitions (not filesystem
individual files).

Another possible implementation would be to use a list or map of hashes possibly
of the installation data stored in the sw-configuration file itself to
allow the installer to
read from storage and calculate and compare hashes to determine how far
the previous installation made it and where to continue the installation from.

While I haven't yet surveyed the different implementation
possibilities, I was thinking
that the client callback hooks could be extended to add a new callback
to be called by the
handler after it parses the sw-configuration and then detects a valid
continue operation
(same version is being installed, bookmark or progress hash check
validated, etc.).
In the callback it can give the download offset to skip to and continue from.

All that said, if your "cached" implementation could be made to only cache the
last megabyte or two of downloaded that, that would be an option for
us. But my current
understanding is that it will cache the entire image (please correct
me if I misunderstood)
and the would not work for us.

> The selective downloading is a very different feature. There is for it a
> RFC with discussion on the ML:
>
> https://groups.google.com/g/swupdate/c/yCDBnWip5xw/m/EFUD59QhCQAJ
>

Thanks for providing that link to the previous discussion. I just
read it and it seems like
there are some functional similarities to what I'm asking about even
if the purpose is
different. As mentioned in that thread, the installer could control
the download parameters
(size and offset).

One user's comment from the thread:

" Maybe we could
extent the passive stream mode by an active mode. In the active mode the
installer controls the image provider. The installer instructs the image
provider to send x bytes or to skip x bytes. Its up to the image
provider if it discards data or requests only needed data. The
downloader could request the needed data from the server and the web
server could simple drop unneeded data."

> The status for that is that I have a customer requiring it, and I am
> waiting for his order to start the implementation.
>
> >
> > Other feature support we might consider adding is "Support for further
> > compressors (in particular xz) for the individual images.
>
> This can be done, it has a very lower priority after the "zst"
> compressor was added. XZ (lzma) can compress better, it requires a
> higher CPU load - ZST seems at the moment the best compromise.
>

Ok, yes, I'm very familiar with Zstd as is it used elsewhere in our system
when real-time compression takes place. But given our OTA update
packages are performed by build servers without strict performance
requirements, xz is preferred by us for the OTA update packages. The
decode may also be slower and more CPU intensive but we perform the
"double copy with fall-back" type of installation in the background at a
low priority while the device functionality is
still available to the user, so performance is still less of a concern relative
to user data connection usage.

> > And this
> > would need to support the "streaming" feature (aka, write data to
> > partition progressively).
>
> This is a must, both gzip and zst compressor are done in this way.
> Adding XZ is IMHO not a big deal as there is a library, but as I said,
> this has not a high priority in the project. A new compressor means to
> rework the pipeline (see copyfile function and similar), and this means
> higher efforts for regression tests.
>
> I have no customers requiring this, that means I do not plan to work on it.
>

Understood. Our goal would be to implement it ourselves but to do so in
a SWUpdate project-friendly way so that the implementation could be
upstreamed at some point. That said, we may also just end up
compressing the SWU CPIO archive itself and just have the

> > We may also be interested in adding support
> > for other SWU archive formats other than CPIO such as zip archive,
> > tar.gz, etc.
>
> SWUpdate does not really use CPIO, but just a simplified version as
> container. Files are flat, and so on. There is no reason to change this
> and I will probably reject such as changes.
>

Understood.

[Rob Lee]

Thanks James. That's good to know. I think my "endgame" will be
to use images encoded in xz but which are organized/segmented into
chunks of compressed data to
allow a continue operation (after a power event) to restart from the
beginning of one of these chunks.

[Stefano Babic]

Hi Rob,

Just think about this can be difficult to reach. SWUpdate is stateless,
and this guarantees that a an update is always verified and treated as
transaction . And, of course, a strength restart from last point won't
be possible, just take as example the last bytes written into a storage
when power is going away.

I guess an approach can be with the "selective downloading" as described
in RFC together with a strong verification of the applied artifacts to
check that are what is expected.

> Our devices operate in regions
> with metered or poor data connections so avoiding the download of
> unnecessary data is important for our customers. This could work using
> a progress "bookmark"
> be stored for each chunk of data stored so the system could know where
> to continue
> from and allow the downloader to pull data from the proper bookmarked
> location in
> the power event and resume case. In our case, we'll be using own downloader and
> we use block-based image updates to partitions (not filesystem
> individual files).
>
> Another possible implementation would be to use a list or map of hashes possibly
> of the installation data stored in the sw-configuration file itself to
> allow the installer to
> read from storage and calculate and compare hashes to determine how far
> the previous installation made it and where to continue the installation from.
>
> While I haven't yet surveyed the different implementation
> possibilities, I was thinking
> that the client callback hooks could be extended to add a new callback
> to be called by the
> handler after it parses the sw-configuration and then detects a valid
> continue operation

This creates harder dependencies, however. Anyway, the handlers are
executors, sw-description was previouslay parsed. You could add a hook
(Lua) and check in advance if an artifact should be updated or not, for
example reading back and comparing with hash. The hook returns then if
the artifact must be skipped. Together with selective download, it could
do it.

> (same version is being installed, bookmark or progress hash check
> validated, etc.).
> In the callback it can give the download offset to skip to and continue from.
>
> All that said, if your "cached" implementation could be made to only cache the
> last megabyte or two of downloaded that, that would be an option for
> us.

No, it is not.

> But my current
> understanding is that it will cache the entire image (please correct
> me if I misunderstood)
> and the would not work for us.

Right, it is completely stateless, and the local file is just consumed
before downloading the rest.

We are talking about device. The device must decompress, and it steals
CPU time from the application. Compressing is not an issue because this
is done by the build server, but decompressing is.

> xz is preferred by us for the OTA update packages. The
> decode may also be slower and more CPU intensive but we perform the
> "double copy with fall-back"

It does not matter - it is not that an update is slower (this has no
relevance), but that the CPU load is higher and a device (specially in
critical application or with real time requirements) can be affected.
The goal of an updater is that it can be forgotten: it is there, it does
not cange (or as less as possible) the current behavior, and it is reliable.

> type of installation in the background at a
> low priority while the device functionality is
> still available to the user, so performance is still less of a concern relative
> to user data connection usage.

This can be on your device - anyway, xz can be added if it is inserted
in the pipeline exactly as it was done with gzip and zst.

>
>>> And this
>>> would need to support the "streaming" feature (aka, write data to
>>> partition progressively).
>>
>> This is a must, both gzip and zst compressor are done in this way.
>> Adding XZ is IMHO not a big deal as there is a library, but as I said,
>> this has not a high priority in the project. A new compressor means to
>> rework the pipeline (see copyfile function and similar), and this means
>> higher efforts for regression tests.
>>
>> I have no customers requiring this, that means I do not plan to work on it.
>>
>
> Understood. Our goal would be to implement it ourselves but to do so in
> a SWUpdate project-friendly way so that the implementation could be
> upstreamed at some point. That said, we may also just end up
> compressing the SWU CPIO archive itself and just have the
>
>>> We may also be interested in adding support
>>> for other SWU archive formats other than CPIO such as zip archive,
>>> tar.gz, etc.
>>
>> SWUpdate does not really use CPIO, but just a simplified version as
>> container. Files are flat, and so on. There is no reason to change this
>> and I will probably reject such as changes.
>>
>
> Understood.
>

Best regards,
Stefano Babic

[Rob Lee]

Hi Stefano,

Thanks so much for your responses so far. A couple more comments and
questions below.

This sounds promising. How can we proceed with working on selective
download? Myself or another developer on our project could work on this.
But if there currently allocated resources for this effort and a schedule just
let me know how we can help otherwise. Note that at this time, we'd
prefer to keep our company name off the public discussion forums.

I appreciate your concern and feedback. But for us, whether it is zstd, zlib,
or xz, the CPU usage will be 100% for deompression for at least some
duration, and we must properly manage proper critic application / system
performance via available methods (scheduler policy, cpufreq,
cpu affinity, idle injection, etc.). Anyway, adding xz is lower priority
for the near term so I had just wanted to get your thoughts about
adding it in the future which you gave. Thank you.