binary delta updates

[mva...@victronenergy.com]

Hi Stefano & co,

For the same reasons as many others, 2G/3G/Satellite connections which are
slow & expensive & intermittent, I am looking into downloading only binary
deltas as well as a robust resume download feature. Swupdate already has a
resume download function, but it will only resume the download if the
swupdate process hasn't been stopped. I'd prefer it to be able to resume
after a next boot as well. And that requires a place to store the partially
downloaded file.

Before going into the details of the idea, lets start with a short summary
on how we'll use swupdate for Venus [1]. We have two rootfs partitions: one
active, one standby. Not read only. Besides that there is a data partition
for user settings and such. Active partition runs our application, and has
swupdate and the scripts around swupdate. There are two ways of updating the
device:

1) Online: the wrapper script around swupdate uses curl to check if there is
a new version available for download. If there is a new one, it starts
swupdate with the -d and -install-direct options: download the file, and
stream it into passive partition(s). Once completed, it boots into that by
swapping passive and active.

2) Offline: user inserts sdcard or usb stick with a swu file, and somehow
triggers the device to open that swu file and apply it to the passive
partition again.

For those interested in the implementation, Mans from Denx is currently
finalizing the implementation and latest status is visible in [2] and [3].

Now, the details of my thinking about these binary deltas:

We'll need extra space on disk to store the (unzipped!) original contents of
the swu file. Here is why:
- to apply a delta, you need an original
- since we don't want a read only rootfs, it needs to be stored separately
- to prevent the deltas slowly growing, the original needs to be updated
as well on every update.
- storing it zipped means zipping it on the device, which in our case is
a bad idea: far too CPU and RAM intensive.
- one thing that could probably be done is to take out all the zeros of the
vfat and ext4 files (ie. sparse). That would save quite some space now,
but the added value of this will decrease when the in-use size of the
rootfs grows. So to calculate worst case, we need to calculate with 100%
in-use values.

xdelta3, and perhaps also other binary diff tools, are able take the delta
from a stream. But, since we want to be able to resume download also after a
power cycle, this can't be used: make sure we can also store a delta, with max size same as swu file without compression. So, together with that original, the worst case necessary disk space is 2 X the size of one
uncompressed rootfs + bootloader + kernel.

Then, after downloading the delta we need to apply it. Applying in-place is
quite a challenge (Jeroen H. looked quite deeply into that). So that leaves
us with only one option: writing the delta to a third file. So now max
required size is 3 X the size of one unzipped rootfs + bootloader + kernel
etc.

Note that in case you have plenty RAM, it would also be possible to store
the delta file in RAM: worst case disk is 2X and RAM is 1X.

And, another alternative, it would also be possible to store the new file in
RAM first. And when done replace the original in disk with that. It does
create a (short) window in which a loss of power will mean losing
everything. Not the end of the world: the device will need to re-download
the full new file. For this case, worst case disk is 1X and RAM is 2X.

So, I invite you to shoot some holes in that logic!

Lets for now assume it is flawless :-), and therefore continue with the
implementation. Which is surprisingly simple to do:

- instead of using swupdate -d, we use curl or something else to download,
and resume download, the files to disk.
- on the disk there is orignal.swu
- if it is not there, or its md5 is wrong, skip the whole delta process and
download latest version completely. Store it for next time, and call
swupdate -i. Done.

- if the original is there, download the delta
- apply the delta, result is two files: myimg-[buildtime-old].swu and
myimg-[buildtimenew].swu
- remove original.swu (in previous step we md5-ed the new file first of-
course).
- use swupdate -i to update the image. Done.
- in case anything goes wrong, revert to full download, perhaps even just
swupdate -d, instead of curl/disk. Done.

All this implementation can be done outside swupdate, in a simple bash
-script. I've something running, I'll post it once a bit further
progressed. And I don't even see any functional improvements in the
diskspace or RAM requirements by implementing it into swupdate. Ofcourse,
to make using this easy, it would be nice to add it into swupdate instead of
some wrapper scripts.

Am I overlooking something? Looking forward to hear your thoughts on this!

Best regards, Matthijs

FYI: In my first approach, I was using swu files containing zipped
contents, and that changes, and complicates!, everything. There are then
many different ways to implement this, and implementing it within swupdate,
or at least being able to stream a swu file via stdin to swupdate does have
its advantages in that case. But, as explained, that also requires rezipping
the new file to store it, which is too resource intensive. Better use some
extra storage, for example explain the end customers that if he/she is on an
expensive and/or slow internet connection, he/she should leave an sdcard
inserted into the device. Cost of an sdcard is nothing compared to
downloading a few 100 MBs on an expensive internet connection.

[1] https://github.com/victronenergy/venus/wiki
[2] https://github.com/victronenergy/meta-victronenergy/commits/swupdate-bbb
[3] https://github.com/victronenergy/venus/wiki/swupdate-project

[Stefano Babic]

Hi Matthijs,

On 02/09/2016 22:27, mva...@victronenergy.com wrote:
> Hi Stefano & co,
>
> For the same reasons as many others, 2G/3G/Satellite connections which are
> slow & expensive & intermittent, I am looking into downloading only binary
> deltas as well as a robust resume download feature. Swupdate already has a
> resume download function, but it will only resume the download if the
> swupdate process hasn't been stopped. I'd prefer it to be able to resume
> after a next boot as well. And that requires a place to store the partially
> downloaded file.

Right.

>
> Before going into the details of the idea, lets start with a short summary
> on how we'll use swupdate for Venus [1]. We have two rootfs partitions: one
> active, one standby. Not read only. Besides that there is a data partition
> for user settings and such. Active partition runs our application, and has
> swupdate and the scripts around swupdate. There are two ways of updating the
> device:
>
> 1) Online: the wrapper script around swupdate uses curl to check if there is
> a new version available for download. If there is a new one, it starts
> swupdate with the -d and -install-direct options: download the file, and
> stream it into passive partition(s). Once completed, it boots into that by
> swapping passive and active.
>
> 2) Offline: user inserts sdcard or usb stick with a swu file, and somehow
> triggers the device to open that swu file and apply it to the passive
> partition again.
>
> For those interested in the implementation, Mans from Denx is currently
> finalizing the implementation and latest status is visible in [2] and [3].
>
> Now, the details of my thinking about these binary deltas:
>
> We'll need extra space on disk to store the (unzipped!) original contents of
> the swu file. Here is why:
> - to apply a delta, you need an original

Absolutely correct - you need the original, and you must check that the
original is not corrupted.

> - since we don't want a read only rootfs, it needs to be stored separately
> - to prevent the deltas slowly growing, the original needs to be updated
> as well on every update.
> - storing it zipped means zipping it on the device, which in our case is
> a bad idea: far too CPU and RAM intensive.

This is not always true, at least for RAM requirement. In fact, images
can be zipped inside a .swu and they are unzipped on-the-fly by
SWUpdate. Because SWUpdate has its own decompressor and it works on
chunks of data in the stream, without calling external tools as unzip,
etc., the RAM requirements remain low, as well as storage requirements.

CPU load, of course, increases.

> - one thing that could probably be done is to take out all the zeros of the
> vfat and ext4 files (ie. sparse).

We are talking here about a special use case. Generally, an image can be
any kind of data, not only the rootfs: FPGA's bitstreams, manuals stored
on the device to avoid printed documentation, multimedia data (video,
and so on.). The nature of the image can lead to different strategies
how to build and transfer the data.

But let's stick with your use case for the moment.

> That would save quite some space now,
> but the added value of this will decrease when the in-use size of the
> rootfs grows. So to calculate worst case, we need to calculate with 100%
> in-use values.

Right

>
> xdelta3, and perhaps also other binary diff tools, are able take the delta
> from a stream. But, since we want to be able to resume download also after a
> power cycle, this can't be used: make sure we can also store a delta, with max size same as swu file without compression. So, together with that original, the worst case necessary disk space is 2 X the size of one
> uncompressed rootfs + bootloader + kernel.
>

My big concern about xdelta3 and family is the RAM consumption: to
generate the new image from original + delta, they are doing in-ram
substitution making the RAM requirement very high. This is not a problem
in your project, but it is difficult to generalize. We have a lot of
projects nowadays using eMMC or bigger storage, and having multiple
copies of their data seems not very praticable, and generally it does
not scale. Think about of 4GB eMMC, used for rootfs, multimedia data,
other binaries, and of course RAM on the target is much smaller.

For example, bsdiff requires according to[3]:

"bsdiff is quite memory-hungry. It requires max(17*n,9*n+m)+O(1) bytes
of memory, where n is the size of the old file and m is the size of the
new file. "

[3] http://www.daemonology.net/bsdiff/

> Then, after downloading the delta we need to apply it. Applying in-place is
> quite a challenge (Jeroen H. looked quite deeply into that).

Right.

> So that leaves
> us with only one option: writing the delta to a third file. So now max
> required size is 3 X the size of one unzipped rootfs + bootloader + kernel
> etc.

This is the requirement for disk space - apart of that, there is RAM
requirement for applying delta, that can be a big issue on many projects.

>
> Note that in case you have plenty RAM, it would also be possible to store
> the delta file in RAM: worst case disk is 2X and RAM is 1X.
>
> And, another alternative, it would also be possible to store the new file in
> RAM first. And when done replace the original in disk with that. It does
> create a (short) window in which a loss of power will mean losing
> everything. Not the end of the world: the device will need to re-download
> the full new file. For this case, worst case disk is 1X and RAM is 2X.
>

This leads that the way how to store the data should remain
customizable: projects have different resource limitation, one use case
cannot be applied to another project and viceversa.

> So, I invite you to shoot some holes in that logic!
>
> Lets for now assume it is flawless :-), and therefore continue with the
> implementation. Which is surprisingly simple to do:
>
> - instead of using swupdate -d, we use curl or something else to download,
> and resume download, the files to disk.
> - on the disk there is orignal.swu
> - if it is not there, or its md5 is wrong, skip the whole delta process and
> download latest version completely. Store it for next time, and call
> swupdate -i. Done.
>
> - if the original is there, download the delta
> - apply the delta, result is two files: myimg-[buildtime-old].swu and
> myimg-[buildtimenew].swu
> - remove original.swu (in previous step we md5-ed the new file first of-
> course).
> - use swupdate -i to update the image. Done.
> - in case anything goes wrong, revert to full download, perhaps even just
> swupdate -d, instead of curl/disk. Done.
>

I agree that if you have enough resources, this works - or at least, I
do not see why it should not work.

You need maybe also some external logic to get progress status during
downloading - anyway, when SWUpdate runs, it gets the whole .swu and it
works.

> All this implementation can be done outside swupdate, in a simple bash
> -script. I've something running, I'll post it once a bit further
> progressed.

Nice, thanks.

> And I don't even see any functional improvements in the
> diskspace or RAM requirements by implementing it into swupdate.

I am not convinced about this - at least, I cannot generalize for all
projects. I am quite sure that your proposal does not work or has strong
limitations for other devices, where for example we could have the
following scenarios:

- rootfs is one of the image, but there are many other images (videos,
FPGA's bitstreams, user's blobs..).
- the .swu does not contain just the image for your device, but also the
release for other (many ?) devices. Storing the whole .swu on the single
device is impossible.
- Applying the delta could be difficult with standard tools due to RAM
limitations on a device.

I think that some images (or sub-images) are better suited for a binary
delta, some other less. rootfs can be a good example, a multimedia data
(always compressed) is a bad one. I tend to assume that the best thing
is to have a better granularity, that means a user can set in
sw-description for each image if it is a "delta" or not - a mix of
binary deltas and standard images is then possible, instead of having a
delta of the whole .swu.

I am also thinking about if some communication with SWUpdate and the
deployment system (your server, or hawkbit as at the moment the only one
supported in SWUpdate) could improve significant the concept.

Let's think as the case: the target can report to the deployment system
an identifier (a hash,...) to identify the running image. The deployment
system could be able, from the hash and the new release, to compute the
delta and to build a delta. And the target should not store the original
file.

Just for rootfs, can we maybe use another way for deploying ?

Let's assume that we have a new release where we have just modified a
couple of binaries (the application) and inserted some files in a
directory. If we compute the delta on the basis of changed files instead
of delta of the filesystem, we could have a package (tarball, maybe)
with just the changed files - in any case, surely a small file, maybe
smaller as the delta computed on the filesystem.

If SWUpdate could work with this special tarball (it cannot currently),
resource's requirement remains very low.

> Ofcourse,
> to make using this easy, it would be nice to add it into swupdate instead of
> some wrapper scripts.
>
> Am I overlooking something? Looking forward to hear your thoughts on this!
>

I just think that your thoughts will apply well to your project, and I
do not see any "big" trouble to advise you and tell you to not do in
this way. I have anyway big concerns if this procedure can be applied to
other projects with different requirements.

> Best regards, Matthijs
>
> FYI: In my first approach, I was using swu files containing zipped
> contents, and that changes, and complicates!, everything.

Specially for the delta computation, I assume.

> There are then
> many different ways to implement this, and implementing it within swupdate,
> or at least being able to stream a swu file via stdin to swupdate does have
> its advantages in that case. But, as explained, that also requires rezipping
> the new file to store it, which is too resource intensive. Better use some
> extra storage, for example explain the end customers that if he/she is on an
> expensive and/or slow internet connection, he/she should leave an sdcard
> inserted into the device. Cost of an sdcard is nothing compared to
> downloading a few 100 MBs on an expensive internet connection.

This is ok for your project :-)

>
> [1] https://github.com/victronenergy/venus/wiki
> [2] https://github.com/victronenergy/meta-victronenergy/commits/swupdate-bbb
> [3] https://github.com/victronenergy/venus/wiki/swupdate-project
>

Best regards,
Stefano

--
=====================================================================
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: +49-8142-66989-53 Fax: +49-8142-66989-80 Email: sba...@denx.de
=====================================================================

[Matthijs Vader]

Hi Stefano,

Many thanks for your detailed reply. My answers and a new option, overlayfs,
below:

Yes, you could keep a .swu file containing compressed images as the original. But
then it is not possible to replace that original copy with a new one when updating a
device. At least not without doing the compression on the device. And compression
is a much heavier operation than decompressing. I tried it on the beaglebone, and
concluded that -for our project- doing the compression on the device is a no go.

The alternative would be to leave the original (.swu file containing compressed
images) as is, ie. don't update it. But then the deltas will grow over time. Something
I don't want either.

Overlayfs?
To add even more options to this puzzle, there is also the overlayfs. Though
again only applicable to the special (but common) rootfs image, still useful? Let
me explain via our own use case:

There is nothing on the rootfs which needs to be retained when going from one
version to the next. All user settings, ip settings etc are on a data partition. But still
there are advantages to having the rootfs writeable. One if which will be the boot
speed.

By using an overlayfs we can mount the partition containing the rootfs read only, and
overlay it with a (smaller) writeable partition. It might then be possible to use that
RO mounted rootfs as the source when applying the delta. This way the required
diskspace reduces significantly.

Yes, I fully understand. Actually I didn't spend much time comparing memory
requirements of xdelta3, bsdiff and the others. xdelta3 does have commandline
parameters to tweak this by the way. Perhaps bsdiff as well.

https://github.com/jmacd/xdelta/blob/wiki/TuningMemoryBudget.md

> > Then, after downloading the delta we need to apply it. Applying
> > in-place is quite a challenge (Jeroen H. looked quite deeply into that).
>
> Right.
>
> > So that leaves
> > us with only one option: writing the delta to a third file. So now max
> > required size is 3 X the size of one unzipped rootfs + bootloader +
> > kernel etc.
>
> This is the requirement for disk space - apart of that, there is RAM
> requirement for applying delta, that can be a big issue on many projects.

Yes. There will be many projects where applying deltas is simply not possible.
And others where it only possible after a considerable engineering effort to
optimize disk/ram/cpu requirements. And they are like communicating vessels:
reducing CPU is possible, but will increase disk and RAM requirements. Same
in the other combinations.

> >
> > Note that in case you have plenty RAM, it would also be possible to
> > store the delta file in RAM: worst case disk is 2X and RAM is 1X.
> >
> > And, another alternative, it would also be possible to store the new
> > file in RAM first. And when done replace the original in disk with
> > that. It does create a (short) window in which a loss of power will
> > mean losing everything. Not the end of the world: the device will need
> > to re-download the full new file. For this case, worst case disk is 1X and
> RAM is 2X.
> >
>
> This leads that the way how to store the data should remain
> customizable: projects have different resource limitation, one use case
> cannot be applied to another project and viceversa.

Yes.

I see the advantage of having images for many devices in one swu file for a certain
user workflow. But, when the .swu file is downloaded, surely also other projects
where transferring data is expensive will put only the image for one device in a
.swu file?

> - Applying the delta could be difficult with standard tools due to RAM
> limitations on a device.
>
> I think that some images (or sub-images) are better suited for a binary delta,
> some other less. rootfs can be a good example, a multimedia data (always
> compressed) is a bad one. I tend to assume that the best thing is to have a
> better granularity, that means a user can set in sw-description for each image
> if it is a "delta" or not - a mix of binary deltas and standard images is then
> possible, instead of having a delta of the whole .swu.
>
> I am also thinking about if some communication with SWUpdate and the
> deployment system (your server, or hawkbit as at the moment the only one
> supported in SWUpdate) could improve significant the concept.

Yes, with a smart webserver it would be no longer necessary to precompute
the deltas. Right now I've opted for a quite simple solution: when deploying a
new image to the webserver, the script takes the 5 last versions and creates the
delta files between them and the new image. And uploads all of them. The
name of the delta files contain old and new buildtime, making it simple for the
device to select the right one.

Now that I think of it, it would be quite straightforward to replace that
precomputing by some php for example that makes the deltas server side and
on request.

Do you see any other related advantages in having a smart deployment system?

>
> Let's think as the case: the target can report to the deployment system an
> identifier (a hash,...) to identify the running image. The deployment system
> could be able, from the hash and the new release, to compute the delta and
> to build a delta. And the target should not store the original file.
>
> Just for rootfs, can we maybe use another way for deploying ?
>
> Let's assume that we have a new release where we have just modified a
> couple of binaries (the application) and inserted some files in a directory. If
> we compute the delta on the basis of changed files instead of delta of the
> filesystem, we could have a package (tarball, maybe) with just the changed
> files - in any case, surely a small file, maybe smaller as the delta computed on
> the filesystem.
>
> If SWUpdate could work with this special tarball (it cannot currently),
> resource's requirement remains very low.

Yes, that could be done. Though I do very much like the full image replacement
though, nicely simple and robust. There are no differences between a new
install, and updating an install, making the windows for mistakes as small as
possible.

> > Ofcourse,
> > to make using this easy, it would be nice to add it into swupdate
> > instead of some wrapper scripts.
> >
> > Am I overlooking something? Looking forward to hear your thoughts on
> this!
> >
>
> I just think that your thoughts will apply well to your project, and I do not see
> any "big" trouble to advise you and tell you to not do in this way.

That is good to know, thanks.

> I have
> anyway big concerns if this procedure can be applied to other projects with
> different requirements.
>

Yes, understood. Though having a rootfs is a common denominator across many
projects I assume? So even though it does not cover other images like fpgas,
multimedia etc, having binary deltas for it will help many projects.

> > Best regards, Matthijs
> >
> > FYI: In my first approach, I was using swu files containing zipped
> > contents, and that changes, and complicates!, everything.
>
> Specially for the delta computation, I assume.
>
> > There are then
> > many different ways to implement this, and implementing it within
> > swupdate, or at least being able to stream a swu file via stdin to
> > swupdate does have its advantages in that case. But, as explained,
> > that also requires rezipping the new file to store it, which is too
> > resource intensive. Better use some extra storage, for example explain
> > the end customers that if he/she is on an expensive and/or slow
> > internet connection, he/she should leave an sdcard inserted into the
> > device. Cost of an sdcard is nothing compared to downloading a few 100
> MBs on an expensive internet connection.
>
> This is ok for your project :-)

Indeed it is ;-)

>
> >
> > [1] https://github.com/victronenergy/venus/wiki
> > [2]
> > https://github.com/victronenergy/meta-
> victronenergy/commits/swupdate-b
> > bb [3] https://github.com/victronenergy/venus/wiki/swupdate-project
> >
>
> Best regards,
> Stefano

Best regards, Matthijs

[Stefano Babic]

Hi Matthijs,

This is correct, what I meant is just that the device will decompress
images or sub images. In SWUpdate, each artifact can be sent in
compressed format and will be decompress and installed by the
corresponding handler, for example an ad-hoc "delta" handler.

> I tried it on the beaglebone, and
> concluded that -for our project- doing the compression on the device is a no go.
>
> The alternative would be to leave the original (.swu file containing compressed
> images) as is, ie. don't update it. But then the deltas will grow over time. Something
> I don't want either.

Anyway, this works when updating from one version to the next one. In
SWUpdate I assume that device can be updated from one version to another
one, that means that downgrading is also possible. And this is not so
uncommon.

>
> Overlayfs?
> To add even more options to this puzzle, there is also the overlayfs.

Right, this is also a possibility, very useful in some projects.

> Though
> again only applicable to the special (but common) rootfs image, still useful? Let
> me explain via our own use case:
>
> There is nothing on the rootfs which needs to be retained when going from one
> version to the next. All user settings, ip settings etc are on a data partition. But still
> there are advantages to having the rootfs writeable. One if which will be the boot
> speed.

Really ? It sounds weird - I switch to read-only rootfs when I want to
speed up the boot. Filesystems can be tuned in read-only mode, for
example turning off journaling, etc., and this saves some time.

I do not see advantages for the boot time - the big advantage (and some
time, this is a disadvantage) is the possibility to change / add any
file to the system.

>
> By using an overlayfs we can mount the partition containing the rootfs read only, and
> overlay it with a (smaller) writeable partition.

Yes. There are many possible ways to reach the goal. An usual schema I
had in the past is to split software in logical partitions, let's say
system and application. The application part is often updated, the rest
in many case not.

And of course, overlayfs is another way to do this.

> It might then be possible to use that
> RO mounted rootfs as the source when applying the delta. This way the required
> diskspace reduces significantly.

Right. My first thought for adding a "binary delta" to SWUpdate is to
have a read-only rootfs. This avoids unnecessary storage of further images.

Yes, I see, I have not tried myself and I have no data to show. Maybe
someone else can explain his experience. There is also some way to use
xdelta3 (not the previous one) as "library", but I have not yet checked how.

>>> Then, after downloading the delta we need to apply it. Applying
>>> in-place is quite a challenge (Jeroen H. looked quite deeply into that).
>>
>> Right.
>>
>>> So that leaves
>>> us with only one option: writing the delta to a third file. So now max
>>> required size is 3 X the size of one unzipped rootfs + bootloader +
>>> kernel etc.
>>
>> This is the requirement for disk space - apart of that, there is RAM
>> requirement for applying delta, that can be a big issue on many projects.
>
> Yes. There will be many projects where applying deltas is simply not possible.

Right - but this is not an issue and it is not a blocking point if some
support for binary delta is required to be added to SWUpdate. With
SWUpdate I am trying to make available a framework - any developer can
then decide which is the best approach for the own project.

> And others where it only possible after a considerable engineering effort to
> optimize disk/ram/cpu requirements. And they are like communicating vessels:
> reducing CPU is possible, but will increase disk and RAM requirements. Same
> in the other combinations.

Fully agree.

Very probable, but I do not know. There are several use cases where it
is important that the device runs a safe upgrade, independently how much
time an update takes. If SWUpdate runs in double-copy, there is not
offline time. Of course, if the connection is expensive, it makes no
sense and there should be one image for .swu.

The best thing is that SWUpdate can easy adjustable to the needs of a
single project.

>
>> - Applying the delta could be difficult with standard tools due to RAM
>> limitations on a device.
>>
>> I think that some images (or sub-images) are better suited for a binary delta,
>> some other less. rootfs can be a good example, a multimedia data (always
>> compressed) is a bad one. I tend to assume that the best thing is to have a
>> better granularity, that means a user can set in sw-description for each image
>> if it is a "delta" or not - a mix of binary deltas and standard images is then
>> possible, instead of having a delta of the whole .swu.
>>
>> I am also thinking about if some communication with SWUpdate and the
>> deployment system (your server, or hawkbit as at the moment the only one
>> supported in SWUpdate) could improve significant the concept.
>
> Yes, with a smart webserver it would be no longer necessary to precompute
> the deltas. Right now I've opted for a quite simple solution: when deploying a
> new image to the webserver, the script takes the 5 last versions and creates the
> delta files between them and the new image. And uploads all of them. The
> name of the delta files contain old and new buildtime, making it simple for the
> device to select the right one.
>
> Now that I think of it, it would be quite straightforward to replace that
> precomputing by some php for example that makes the deltas server side and
> on request.

Yes, this is an advanced solution, and it looks quite easy to implement.

>
> Do you see any other related advantages in having a smart deployment system?

There are many other advantages, generally unrelated to a binary delta.
Think about of having hundreds / thousands of device in field, and you
want to schedule an update, or to easy know on which device an update
failed. All these reason have lead to add support ("suricatta mode") in
SWUpdate for the hawkbit server.

http://sp.apps.bosch-iot-cloud.com/documentation/index.html

It could be overkill for just a few devices in field, but it becomes
necessary when the number of devices grows.

>
>>
>> Let's think as the case: the target can report to the deployment system an
>> identifier (a hash,...) to identify the running image. The deployment system
>> could be able, from the hash and the new release, to compute the delta and
>> to build a delta. And the target should not store the original file.
>>
>> Just for rootfs, can we maybe use another way for deploying ?
>>
>> Let's assume that we have a new release where we have just modified a
>> couple of binaries (the application) and inserted some files in a directory. If
>> we compute the delta on the basis of changed files instead of delta of the
>> filesystem, we could have a package (tarball, maybe) with just the changed
>> files - in any case, surely a small file, maybe smaller as the delta computed on
>> the filesystem.
>>
>> If SWUpdate could work with this special tarball (it cannot currently),
>> resource's requirement remains very low.
>
> Yes, that could be done. Though I do very much like the full image replacement
> though, nicely simple and robust. There are no differences between a new
> install, and updating an install, making the windows for mistakes as small as
> possible.

Sorry, I wasn't very clear. I am still thinking about an image update,
not file update. Replacing some files is a feature already implemented
in SWUpdate, but it is not an image updater.

Let's make an example: for simplicity, the device has a read-only
rootfs. And the .swu contains as artifact a tarball ("delta"), with the
differences with the installed rootfs. SWUpdate should then:

- copy the read-only rootfs (we named it the "original") to the stand-by
copy.
- apply the "delta" tarball
- verify the generated rootfs

This is still an image updater, just the image is generated on the
device. But also with xdelta3 and friends, the device must generate the
rootfs.

>
>>> Ofcourse,
>>> to make using this easy, it would be nice to add it into swupdate
>>> instead of some wrapper scripts.
>>>
>>> Am I overlooking something? Looking forward to hear your thoughts on
>> this!
>>>
>>
>> I just think that your thoughts will apply well to your project, and I do not see
>> any "big" trouble to advise you and tell you to not do in this way.
>
> That is good to know, thanks.
>
>> I have
>> anyway big concerns if this procedure can be applied to other projects with
>> different requirements.
>>
>
> Yes, understood. Though having a rootfs is a common denominator across many
> projects I assume? So even though it does not cover other images like fpgas,
> multimedia etc, having binary deltas for it will help many projects.

Yes, but then I need a granularity at the artifacts' level, and not for
the whole .swu

>
>>> Best regards, Matthijs
>>>
>>> FYI: In my first approach, I was using swu files containing zipped
>>> contents, and that changes, and complicates!, everything.
>>
>> Specially for the delta computation, I assume.
>>
>>> There are then
>>> many different ways to implement this, and implementing it within
>>> swupdate, or at least being able to stream a swu file via stdin to
>>> swupdate does have its advantages in that case. But, as explained,
>>> that also requires rezipping the new file to store it, which is too
>>> resource intensive. Better use some extra storage, for example explain
>>> the end customers that if he/she is on an expensive and/or slow
>>> internet connection, he/she should leave an sdcard inserted into the
>>> device. Cost of an sdcard is nothing compared to downloading a few 100
>> MBs on an expensive internet connection.
>>
>> This is ok for your project :-)
>
> Indeed it is ;-)
>>
>>>
>>> [1] https://github.com/victronenergy/venus/wiki
>>> [2]
>>> https://github.com/victronenergy/meta-
>> victronenergy/commits/swupdate-b
>>> bb [3] https://github.com/victronenergy/venus/wiki/swupdate-project
>>>
>>


Best regards,
Stefano

[Matthijs Vader]

Hi Stefano,

A few last replies below. I'll reply again as soon as I have some working scripts to show.

Best regards, Matthijs

Understood

> > I tried it on the beaglebone, and
> > concluded that -for our project- doing the compression on the device is a
> > no go.
> >
> > The alternative would be to leave the original (.swu file containing
> > compressed
> > images) as is, ie. don't update it. But then the deltas will grow over
> > time. Something I don't want either.
>
> Anyway, this works when updating from one version to the next one. In
> SWUpdate I assume that device can be updated from one version to another
> one, that means that downgrading is also possible. And this is not so
> uncommon.

Understood.

> >
> > Overlayfs?
> > To add even more options to this puzzle, there is also the overlayfs.
>
> Right, this is also a possibility, very useful in some projects.
>
> > Though
> > again only applicable to the special (but common) rootfs image, still
> > useful? Let me explain via our own use case:
> >
> > There is nothing on the rootfs which needs to be retained when going
> > from one version to the next. All user settings, ip settings etc are
> > on a data partition. But still there are advantages to having the
> > rootfs writeable. One if which will be the boot speed.
>
> Really ? It sounds weird - I switch to read-only rootfs when I want to speed
> up the boot. Filesystems can be tuned in read-only mode, for example
> turning off journaling, etc., and this saves some time.

Sorry, I was too quick with that. The dot should have been a question mark. I have no experience with read-only rootfs-es.

> I do not see advantages for the boot time - the big advantage (and some
> time, this is a disadvantage) is the possibility to change / add any file to the
> system.

Yes, for our project we decided to keep all settings on a separate, non-overlaying, data partition.

I'll leave the overlayfs aside now. In our project there is no read-only rootfs at the moment, and changing into that or experimenting with overlayfs-es is not something I don't want to do now.

Yes. Also while browsing around on the internet to read what other have done and why, the adjustability of SWUpdate is its most valued property.

I'll have another look, thanks!

Yes, clear.

> >
> >>> Ofcourse,
> >>> to make using this easy, it would be nice to add it into swupdate
> >>> instead of some wrapper scripts.
> >>>
> >>> Am I overlooking something? Looking forward to hear your thoughts on
> >> this!
> >>>
> >>
> >> I just think that your thoughts will apply well to your project, and
> >> I do not see any "big" trouble to advise you and tell you to not do in this
> >> way.
> >
> > That is good to know, thanks.
> >
> >> I have
> >> anyway big concerns if this procedure can be applied to other
> >> projects with different requirements.
> >>
> >
> > Yes, understood. Though having a rootfs is a common denominator across
> > many projects I assume? So even though it does not cover other images
> > like fpgas, multimedia etc, having binary deltas for it will help many
> > projects.
>
> Yes, but then I need a granularity at the artifacts' level, and not for the whole
> .swu

Yes, fully understood.

> >
> >>> Best regards, Matthijs
> >>>
> >>> FYI: In my first approach, I was using swu files containing zipped
> >>> contents, and that changes, and complicates!, everything.
> >>
> >> Specially for the delta computation, I assume.
> >>
> >>> There are then
> >>> many different ways to implement this, and implementing it within
> >>> swupdate, or at least being able to stream a swu file via stdin to
> >>> swupdate does have its advantages in that case. But, as explained,
> >>> that also requires rezipping the new file to store it, which is too
> >>> resource intensive. Better use some extra storage, for example
> >>> explain the end customers that if he/she is on an expensive and/or
> >>> slow internet connection, he/she should leave an sdcard inserted
> >>> into the device. Cost of an sdcard is nothing compared to
> >>> downloading a few 100
> >> MBs on an expensive internet connection.
> >>
> >> This is ok for your project :-)
> >
> > Indeed it is ;-)
> >>
> >>>
> >>> [1] https://github.com/victronenergy/venus/wiki
> >>> [2]
> >>> https://github.com/victronenergy/meta-
> >> victronenergy/commits/swupdate-b
> >>> bb [3] https://github.com/victronenergy/venus/wiki/swupdate-project

Have a good day!

Matthijs