RFC: Concept "Selective download"
140 views
Skip to first unread message
Stefano Babic
Jan 5, 2021, 5:22:48 AM1/5/21
to swup...@googlegroups.com
Hi everybody,
I am thinking about to implement a new feature and I am writing a
concept, asking the community for help and review. There is at the
moment no single line implemented, I want to check in advance drawbacks
and exchange ideas.
Background: I get information from a manufacturer, whose devices in
field are just checking for new update twice a day. He is not using
Hawkbit, but he has an own server where he stored the SWUs. SWUpdate on
the device simply connects with the server and verifies if the whole SWU
must be downloaded. This means that SWUpdate starts to download the SWU,
then sw-description is parsed and the update is stop as soon as it is
recognized (using version schemas) that a new version is not required.
Due to the security concept that requires to split download and install
in separate processes, update is stop after ~0.8 MB are downloaded when
the installer has parsed sw-description. The downloader has already
loaded more data and they lie in kernel (skb) or SWUpdate buffers.
If this seems a low value, manufacturer reports that, due to his large
installation, the update check generates more as 100GB/day of unneeded
traffic.
In case of backend, the traffic is surely reduced because SWUpdate just
asks Hawkbit if a new software is available via REST-API, without
checking itself inside the SWU. But adding a Hawkbit infrastructure is a
no way for that manufacturer. Anyway, I am thinking about a solution
that brings benefits for Hawkbit, too (and generally, when SWUpdate
works in "pull" mode loading itself from an external source). A use case
is when the SWU contains multiple artifacts, and just one of them needs
to be updated (for example, just the application). An ad-hoc logic can
be implemented as Lua script ("embedded-script" in sw-description), and
SWUpdate could avoid to download what will be marked as "non required"
after parsing, instead of downloading the whole SWU and skip what is not
necessary. This helps also in another use case, where SW for multiple
devices are packed together in one SWU (a requirement in some
application where storage servers are managed by organizations and not
by manufacturers), but bandwidth must be preserved. So these are the
main reasons for this RFC.
In SWUpdate design, there is a split between the installer (SWUpdate's
main process) and the processes getting the SWU. The last ones have no
idea about how the SWU is built and they rely on the installer. I want
to change slightly this: the downloders won't know the internals, but
they will be aware that SWU is a CPIO archive and that sw-description is
the first file in the SWU (this is hardcoded in SWUpdate by design).
The idea is to use RFC 7233 (" Hypertext Transfer Protocol (HTTP/1.1):
Range Requests") - all current HTTP servers support it. This means that
this new feature just works for HTTP(S), but this is the 90% of the
cases or more.
I sketch my ideas as:
1. the downloader knows that the SWU is a CPIO. It starts downloading
the first CPIO header (110 bytes), sending a HTTP Range request.
2. the downloader knows the size of sw-description, and load it together
with the next CPIO header (header of next file).
3. if the header of next file is the signature, download it as well (and
asks also for the next CPIO header).
4. push the data via IPC to the installer : sw-description is parsed,
and the internal structures are created. SWUpdate knows which artifacts
are required, and if the update must go on.
5. the downloader asks via IPC (a new command) the installer if the next
file in the cpio is required. If yes, download it and push to the
installer, else skips it and asks via HTTP range request for the next
cpio header, until an artifact is required.
6. loop 5 until the end of cpio, this is marked with the "TRAILER!"
magic word. Maybe some more controls are required here to avoid that an
attacker tries a DOS pushing non-sense data to the device.
In this way, a simple check means to load just
sw-description+sw-description.sig, plus some bytes as overhead. And it
reduces the size of downloaded data in case some artifacts are already
on the device.
I can also imagine a further step just for Hawkbit: multiple SWUs (as
software modules) can be put into a distribution, and SWUpdate can check
all of them finding the more appropriate for the update - for example, a
delta from Vx to Vy. The overhead becomes just the loading of
sw-description + signature, and it is worth as loading the whole image.
Any thoughts, comments, proposals ?
Best regards,
Stefano Babic
--
=====================================================================
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: +49-8142-66989-53 Fax: +49-8142-66989-80 Email: sba...@denx.de
=====================================================================
I am thinking about to implement a new feature and I am writing a
concept, asking the community for help and review. There is at the
moment no single line implemented, I want to check in advance drawbacks
and exchange ideas.
Background: I get information from a manufacturer, whose devices in
field are just checking for new update twice a day. He is not using
Hawkbit, but he has an own server where he stored the SWUs. SWUpdate on
the device simply connects with the server and verifies if the whole SWU
must be downloaded. This means that SWUpdate starts to download the SWU,
then sw-description is parsed and the update is stop as soon as it is
recognized (using version schemas) that a new version is not required.
Due to the security concept that requires to split download and install
in separate processes, update is stop after ~0.8 MB are downloaded when
the installer has parsed sw-description. The downloader has already
loaded more data and they lie in kernel (skb) or SWUpdate buffers.
If this seems a low value, manufacturer reports that, due to his large
installation, the update check generates more as 100GB/day of unneeded
traffic.
In case of backend, the traffic is surely reduced because SWUpdate just
asks Hawkbit if a new software is available via REST-API, without
checking itself inside the SWU. But adding a Hawkbit infrastructure is a
no way for that manufacturer. Anyway, I am thinking about a solution
that brings benefits for Hawkbit, too (and generally, when SWUpdate
works in "pull" mode loading itself from an external source). A use case
is when the SWU contains multiple artifacts, and just one of them needs
to be updated (for example, just the application). An ad-hoc logic can
be implemented as Lua script ("embedded-script" in sw-description), and
SWUpdate could avoid to download what will be marked as "non required"
after parsing, instead of downloading the whole SWU and skip what is not
necessary. This helps also in another use case, where SW for multiple
devices are packed together in one SWU (a requirement in some
application where storage servers are managed by organizations and not
by manufacturers), but bandwidth must be preserved. So these are the
main reasons for this RFC.
In SWUpdate design, there is a split between the installer (SWUpdate's
main process) and the processes getting the SWU. The last ones have no
idea about how the SWU is built and they rely on the installer. I want
to change slightly this: the downloders won't know the internals, but
they will be aware that SWU is a CPIO archive and that sw-description is
the first file in the SWU (this is hardcoded in SWUpdate by design).
The idea is to use RFC 7233 (" Hypertext Transfer Protocol (HTTP/1.1):
Range Requests") - all current HTTP servers support it. This means that
this new feature just works for HTTP(S), but this is the 90% of the
cases or more.
I sketch my ideas as:
1. the downloader knows that the SWU is a CPIO. It starts downloading
the first CPIO header (110 bytes), sending a HTTP Range request.
2. the downloader knows the size of sw-description, and load it together
with the next CPIO header (header of next file).
3. if the header of next file is the signature, download it as well (and
asks also for the next CPIO header).
4. push the data via IPC to the installer : sw-description is parsed,
and the internal structures are created. SWUpdate knows which artifacts
are required, and if the update must go on.
5. the downloader asks via IPC (a new command) the installer if the next
file in the cpio is required. If yes, download it and push to the
installer, else skips it and asks via HTTP range request for the next
cpio header, until an artifact is required.
6. loop 5 until the end of cpio, this is marked with the "TRAILER!"
magic word. Maybe some more controls are required here to avoid that an
attacker tries a DOS pushing non-sense data to the device.
In this way, a simple check means to load just
sw-description+sw-description.sig, plus some bytes as overhead. And it
reduces the size of downloaded data in case some artifacts are already
on the device.
I can also imagine a further step just for Hawkbit: multiple SWUs (as
software modules) can be put into a distribution, and SWUpdate can check
all of them finding the more appropriate for the update - for example, a
delta from Vx to Vy. The overhead becomes just the loading of
sw-description + signature, and it is worth as loading the whole image.
Any thoughts, comments, proposals ?
Best regards,
Stefano Babic
--
=====================================================================
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: +49-8142-66989-53 Fax: +49-8142-66989-80 Email: sba...@denx.de
=====================================================================
Christian Storm
Jan 6, 2021, 11:58:53 AM1/6/21
to swup...@googlegroups.com
Hi Stefano,
> I am thinking about to implement a new feature and I am writing a
> concept, asking the community for help and review. There is at the
> moment no single line implemented, I want to check in advance drawbacks
> and exchange ideas.
>
> Background: I get information from a manufacturer, whose devices in
> field are just checking for new update twice a day. He is not using
> Hawkbit, but he has an own server where he stored the SWUs. SWUpdate on
> the device simply connects with the server and verifies if the whole SWU
> must be downloaded. This means that SWUpdate starts to download the SWU,
> then sw-description is parsed and the update is stop as soon as it is
> recognized (using version schemas) that a new version is not required.
Ok, so this is kind of a dumb server just serving the artifacts and
SWUpdate does the logics on whether to install or not, right?
> Due to the security concept that requires to split download and
> install in separate processes, update is stop after ~0.8 MB are
> downloaded when the installer has parsed sw-description. The
> downloader has already loaded more data and they lie in kernel (skb)
> or SWUpdate buffers. If this seems a low value, manufacturer reports
> that, due to his large installation, the update check generates more
> as 100GB/day of unneeded traffic.
With different download and install "processes" you mean two steps or
two OS processes? For the former, I do in my still to be submitted Lua
suricatta binding a dry-run which downloads the artifact and then use
file://.. to install it from the local storage. This way, SWUpdate is
able to check validity of the artifact and can early abort broken
artifacts...
> In case of backend, the traffic is surely reduced because SWUpdate just
> asks Hawkbit if a new software is available via REST-API, without
> checking itself inside the SWU.
This is one point, the other is knowing somewhere in the backend which
version(s) are installed on the devices, which is unknown to the backend
in your scenario or is it reported by some middleware to some backend?
> But adding a Hawkbit infrastructure is a no way for that manufacturer.
> Anyway, I am thinking about a solution that brings benefits for
> Hawkbit, too (and generally, when SWUpdate works in "pull" mode
> loading itself from an external source). A use case is when the SWU
> contains multiple artifacts, and just one of them needs to be updated
> (for example, just the application). An ad-hoc logic can be
> implemented as Lua script ("embedded-script" in sw-description),
> and SWUpdate could avoid to download what will be marked as "non
> required" after parsing, instead of downloading the whole SWU and skip
> what is not necessary. This helps also in another use case, where SW
> for multiple devices are packed together in one SWU (a requirement in
> some application where storage servers are managed by organizations
> and not by manufacturers), but bandwidth must be preserved. So these
> are the main reasons for this RFC.
Hm, I fear that sw-description will be overloaded with Lua ad-hoc
embedded scripts having some logics? Maybe we can revise the
sw-description format and make it all Lua for example? As another
alternative to libconfig and JSON then? So you have the choice of
"simple" configuration and programmatic Lua configuration?
Of course, this would be a mid/long-term thing... Just an idea though.
> In SWUpdate design, there is a split between the installer (SWUpdate's
> main process) and the processes getting the SWU. The last ones have no
> idea about how the SWU is built and they rely on the installer. I want
> to change slightly this: the downloders won't know the internals, but
> they will be aware that SWU is a CPIO archive and that sw-description is
> the first file in the SWU (this is hardcoded in SWUpdate by design).
> The idea is to use RFC 7233 (" Hypertext Transfer Protocol (HTTP/1.1):
> Range Requests") - all current HTTP servers support it. This means that
> this new feature just works for HTTP(S), but this is the 90% of the
> cases or more.
Hm, so you're splitting the currently one download "function" into
something like download_sw-description() and download_payload(int) then
that can be called from the different ingress channels (suricatta,
download, ...)?
Then, we should also make the downloader able to verify what it has
downloaded, maybe by an IPC call to the core not to duplicate code.
> I sketch my ideas as:
>
> 1. the downloader knows that the SWU is a CPIO. It starts downloading
> the first CPIO header (110 bytes), sending a HTTP Range request.
> 2. the downloader knows the size of sw-description, and load it together
> with the next CPIO header (header of next file).
> 3. if the header of next file is the signature, download it as well (and
> asks also for the next CPIO header).
> 4. push the data via IPC to the installer : sw-description is parsed,
> and the internal structures are created. SWUpdate knows which artifacts
> are required, and if the update must go on.
> 5. the downloader asks via IPC (a new command) the installer if the next
> file in the cpio is required. If yes, download it and push to the
> installer, else skips it and asks via HTTP range request for the next
> cpio header, until an artifact is required.
Hm, the question here is who is the driver of the update process?
Currently, the download is configured to stream it into the core, setup
by the ingress mechanism (suricatta, downloader, ...). Thereafter, the
ingress mechanism is out of the game waiting for SWUpdate core to finish.
Another option would be to make the core call back to the ingress
to get the next data chunk? This way, different download methods
(HTTP/curl, Queue server, ...) could be easily supported by the ingress
mechanism, hidden from SWUpdate core. If it's transport doesn't support
RANGE requests, it can simulate this or fall back to the current design.
> 6. loop 5 until the end of cpio, this is marked with the "TRAILER!"
> magic word. Maybe some more controls are required here to avoid that an
> attacker tries a DOS pushing non-sense data to the device.
>
> In this way, a simple check means to load just
> sw-description+sw-description.sig, plus some bytes as overhead. And it
> reduces the size of downloaded data in case some artifacts are already
> on the device.
You mean already installed or already downloaded to local storage?
> I can also imagine a further step just for Hawkbit: multiple SWUs (as
> software modules) can be put into a distribution, and SWUpdate can check
> all of them finding the more appropriate for the update - for example, a
> delta from Vx to Vy. The overhead becomes just the loading of
> sw-description + signature, and it is worth as loading the whole image.
Here you impose a particular form of building artifacts and introduce
a convention. So you want to put multiple SWUs into one SWU as "payload"
of which one is chosen according to the sw-description's embedded Lua
script(s) finding out which ones to install? This gives a lot of control
to SWUpdate and hence the device. This may or may not be wanted. The
other "solution" is to have all combinations available in the cloud and
pointing SWUpdate to a particular one depending on its former
communication of versions.
Regardless, I do think this is a valuable addition!
Kind regards,
Christian
--
Dr. Christian Storm
Siemens AG, Technology, T RDA IOT SES-DE
Otto-Hahn-Ring 6, 81739 München, Germany
> I am thinking about to implement a new feature and I am writing a
> concept, asking the community for help and review. There is at the
> moment no single line implemented, I want to check in advance drawbacks
> and exchange ideas.
>
> Background: I get information from a manufacturer, whose devices in
> field are just checking for new update twice a day. He is not using
> Hawkbit, but he has an own server where he stored the SWUs. SWUpdate on
> the device simply connects with the server and verifies if the whole SWU
> must be downloaded. This means that SWUpdate starts to download the SWU,
> then sw-description is parsed and the update is stop as soon as it is
> recognized (using version schemas) that a new version is not required.
SWUpdate does the logics on whether to install or not, right?
> Due to the security concept that requires to split download and
> install in separate processes, update is stop after ~0.8 MB are
> downloaded when the installer has parsed sw-description. The
> downloader has already loaded more data and they lie in kernel (skb)
> or SWUpdate buffers. If this seems a low value, manufacturer reports
> that, due to his large installation, the update check generates more
> as 100GB/day of unneeded traffic.
two OS processes? For the former, I do in my still to be submitted Lua
suricatta binding a dry-run which downloads the artifact and then use
file://.. to install it from the local storage. This way, SWUpdate is
able to check validity of the artifact and can early abort broken
artifacts...
> In case of backend, the traffic is surely reduced because SWUpdate just
> asks Hawkbit if a new software is available via REST-API, without
> checking itself inside the SWU.
version(s) are installed on the devices, which is unknown to the backend
in your scenario or is it reported by some middleware to some backend?
> But adding a Hawkbit infrastructure is a no way for that manufacturer.
> Anyway, I am thinking about a solution that brings benefits for
> Hawkbit, too (and generally, when SWUpdate works in "pull" mode
> loading itself from an external source). A use case is when the SWU
> contains multiple artifacts, and just one of them needs to be updated
> (for example, just the application). An ad-hoc logic can be
> implemented as Lua script ("embedded-script" in sw-description),
> and SWUpdate could avoid to download what will be marked as "non
> required" after parsing, instead of downloading the whole SWU and skip
> what is not necessary. This helps also in another use case, where SW
> for multiple devices are packed together in one SWU (a requirement in
> some application where storage servers are managed by organizations
> and not by manufacturers), but bandwidth must be preserved. So these
> are the main reasons for this RFC.
embedded scripts having some logics? Maybe we can revise the
sw-description format and make it all Lua for example? As another
alternative to libconfig and JSON then? So you have the choice of
"simple" configuration and programmatic Lua configuration?
Of course, this would be a mid/long-term thing... Just an idea though.
> In SWUpdate design, there is a split between the installer (SWUpdate's
> main process) and the processes getting the SWU. The last ones have no
> idea about how the SWU is built and they rely on the installer. I want
> to change slightly this: the downloders won't know the internals, but
> they will be aware that SWU is a CPIO archive and that sw-description is
> the first file in the SWU (this is hardcoded in SWUpdate by design).
> The idea is to use RFC 7233 (" Hypertext Transfer Protocol (HTTP/1.1):
> Range Requests") - all current HTTP servers support it. This means that
> this new feature just works for HTTP(S), but this is the 90% of the
> cases or more.
something like download_sw-description() and download_payload(int) then
that can be called from the different ingress channels (suricatta,
download, ...)?
Then, we should also make the downloader able to verify what it has
downloaded, maybe by an IPC call to the core not to duplicate code.
> I sketch my ideas as:
>
> 1. the downloader knows that the SWU is a CPIO. It starts downloading
> the first CPIO header (110 bytes), sending a HTTP Range request.
> 2. the downloader knows the size of sw-description, and load it together
> with the next CPIO header (header of next file).
> 3. if the header of next file is the signature, download it as well (and
> asks also for the next CPIO header).
> 4. push the data via IPC to the installer : sw-description is parsed,
> and the internal structures are created. SWUpdate knows which artifacts
> are required, and if the update must go on.
> 5. the downloader asks via IPC (a new command) the installer if the next
> file in the cpio is required. If yes, download it and push to the
> installer, else skips it and asks via HTTP range request for the next
> cpio header, until an artifact is required.
Currently, the download is configured to stream it into the core, setup
by the ingress mechanism (suricatta, downloader, ...). Thereafter, the
ingress mechanism is out of the game waiting for SWUpdate core to finish.
Another option would be to make the core call back to the ingress
to get the next data chunk? This way, different download methods
(HTTP/curl, Queue server, ...) could be easily supported by the ingress
mechanism, hidden from SWUpdate core. If it's transport doesn't support
RANGE requests, it can simulate this or fall back to the current design.
> 6. loop 5 until the end of cpio, this is marked with the "TRAILER!"
> magic word. Maybe some more controls are required here to avoid that an
> attacker tries a DOS pushing non-sense data to the device.
>
> In this way, a simple check means to load just
> sw-description+sw-description.sig, plus some bytes as overhead. And it
> reduces the size of downloaded data in case some artifacts are already
> on the device.
> I can also imagine a further step just for Hawkbit: multiple SWUs (as
> software modules) can be put into a distribution, and SWUpdate can check
> all of them finding the more appropriate for the update - for example, a
> delta from Vx to Vy. The overhead becomes just the loading of
> sw-description + signature, and it is worth as loading the whole image.
a convention. So you want to put multiple SWUs into one SWU as "payload"
of which one is chosen according to the sw-description's embedded Lua
script(s) finding out which ones to install? This gives a lot of control
to SWUpdate and hence the device. This may or may not be wanted. The
other "solution" is to have all combinations available in the cloud and
pointing SWUpdate to a particular one depending on its former
communication of versions.
Regardless, I do think this is a valuable addition!
Kind regards,
Christian
--
Dr. Christian Storm
Siemens AG, Technology, T RDA IOT SES-DE
Otto-Hahn-Ring 6, 81739 München, Germany
Georgiev, Veselin
Jan 6, 2021, 5:16:57 PM1/6/21
to Stefano Babic, swup...@googlegroups.com
Hi Stefano,
I hope you are doing well. The ability to limit the amount of data downloaded and installed during the update would be desirable. I have the following feedback:
- Would this change tie the format of the SWU file to the protocol used to retrieve the file? Would this make it difficult to switch to a new file format or change how SWU uses the cpio format? For example, allowing images larger than 4 GB?
- This may have already been mentioned. Another option would be to invert the relationship between the downloader and installer so the installer can request a byte range from the downloader. This would keep the downloader as a simple proxy to the data and retain knowledge of the file format in the downloader. From the installer's perspective it would look as if it were reading the file from disk.
- The proposed design will increase the number of connections to the server/seek operations the server has to perform. There could be some performance decrease depending on the circumstances.
Thank you.
Veselin Georgiev
From: swup...@googlegroups.com <swup...@googlegroups.com> on behalf of Stefano Babic <sba...@denx.de>
Sent: Tuesday, January 5, 2021 4:22 AM
To: 'swup...@googlegroups.com'
Subject: [swupdate] RFC: Concept "Selective download"
Sent: Tuesday, January 5, 2021 4:22 AM
To: 'swup...@googlegroups.com'
Subject: [swupdate] RFC: Concept "Selective download"
CAUTION - EXTERNAL EMAIL: Do not click any links or open any attachments unless you trust the sender and know the content is safe.
--
You received this message because you are subscribed to the Google Groups "swupdate" group.
To unsubscribe from this group and stop receiving emails from it, send an email to swupdate+u...@googlegroups.com.
To view this discussion on the web visit https://urldefense.com/v3/__https://groups.google.com/d/msgid/swupdate/d2b2fd99-40d3-97b4-08eb-98821c16be91*40denx.de__;JQ!!EJc4YC3iFmQ!ELKkpzV1Ho3O083X1Xl1MUbahuMKBMHaueAYw1DWrrzz6bL1WGrtqN7gd7lcYRs9lloj4g$ .
You received this message because you are subscribed to the Google Groups "swupdate" group.
To unsubscribe from this group and stop receiving emails from it, send an email to swupdate+u...@googlegroups.com.
To view this discussion on the web visit https://urldefense.com/v3/__https://groups.google.com/d/msgid/swupdate/d2b2fd99-40d3-97b4-08eb-98821c16be91*40denx.de__;JQ!!EJc4YC3iFmQ!ELKkpzV1Ho3O083X1Xl1MUbahuMKBMHaueAYw1DWrrzz6bL1WGrtqN7gd7lcYRs9lloj4g$ .
CONFIDENTIALITY NOTICE: This email and any attachments are for the sole use of the intended recipient(s) and contain information that may be Garmin confidential and/or Garmin legally privileged. If you have received this email in error, please notify the sender by reply email and delete the message. Any disclosure, copying, distribution or use of this communication (including attachments) by someone other than the intended recipient is prohibited. Thank you.
Stefano Babic
Jan 7, 2021, 5:02:57 AM1/7/21
to swup...@googlegroups.com, Christian Storm
Hi Christian,
On 06.01.21 18:00, Christian Storm wrote:
> Hi Stefano,
>
>> I am thinking about to implement a new feature and I am writing a
>> concept, asking the community for help and review. There is at the
>> moment no single line implemented, I want to check in advance drawbacks
>> and exchange ideas.
>>
>> Background: I get information from a manufacturer, whose devices in
>> field are just checking for new update twice a day. He is not using
>> Hawkbit, but he has an own server where he stored the SWUs. SWUpdate on
>> the device simply connects with the server and verifies if the whole SWU
>> must be downloaded. This means that SWUpdate starts to download the SWU,
>> then sw-description is parsed and the update is stop as soon as it is
>> recognized (using version schemas) that a new version is not required.
>
> Ok, so this is kind of a dumb server just serving the artifacts and
> SWUpdate does the logics on whether to install or not, right?
Right - it is just a Webserver. The SWU are simply stored, and SWUpdate
receives (I do not know how) the URL to get the SWU. Then the whole
logic if an update is required or not is implemented in SWUpdate.
>
>
>> Due to the security concept that requires to split download and
>> install in separate processes, update is stop after ~0.8 MB are
>> downloaded when the installer has parsed sw-description. The
>> downloader has already loaded more data and they lie in kernel (skb)
>> or SWUpdate buffers. If this seems a low value, manufacturer reports
>> that, due to his large installation, the update check generates more
>> as 100GB/day of unneeded traffic.
>
> With different download and install "processes" you mean two steps or
> two OS processes?
two OS processes. We have the installer (core) and the downloader or
suricatta.
> For the former, I do in my still to be submitted Lua
> suricatta binding a dry-run which downloads the artifact and then use
> file://.. to install it from the local storage.
Yes, this is also a use case, I have also implemented in some projects.
> This way, SWUpdate is
> able to check validity of the artifact and can early abort broken
> artifacts...
The problem is the definition of early abort. This is the same use case
that I report here. Even in dry-run mode, you start to download the SWU,
then the SWU is pushed to core until the parser runs and raises the
abort. In the case I reported, the downloaded data until SWUpdate aborts
(from the parser) is ~0.8MB. As I said, this seems not too much, but it
sums with the amount of the installed base. The reason for this RFC is
to introduce a controlled download per file, so that just sw-description
is downloaded if an abort can be raised by the parser, reducing the
amount of downloaded data to some KBytes.
>
>
>> In case of backend, the traffic is surely reduced because SWUpdate just
>> asks Hawkbit if a new software is available via REST-API, without
>> checking itself inside the SWU.
>
> This is one point, the other is knowing somewhere in the backend which
> version(s) are installed on the devices, which is unknown to the backend
> in your scenario or is it reported by some middleware to some backend?
It is unknown (better: there is a storage server, no backend). The
device knows which version is running, and SWUpdate is informed of it
with command line parameter. The version is then compared with the
"version" attribute in sw-description.
>
>
>> But adding a Hawkbit infrastructure is a no way for that manufacturer.
>> Anyway, I am thinking about a solution that brings benefits for
>> Hawkbit, too (and generally, when SWUpdate works in "pull" mode
>> loading itself from an external source). A use case is when the SWU
>> contains multiple artifacts, and just one of them needs to be updated
>> (for example, just the application). An ad-hoc logic can be
>> implemented as Lua script ("embedded-script" in sw-description),
>> and SWUpdate could avoid to download what will be marked as "non
>> required" after parsing, instead of downloading the whole SWU and skip
>> what is not necessary. This helps also in another use case, where SW
>> for multiple devices are packed together in one SWU (a requirement in
>> some application where storage servers are managed by organizations
>> and not by manufacturers), but bandwidth must be preserved. So these
>> are the main reasons for this RFC.
>
> Hm, I fear that sw-description will be overloaded with Lua ad-hoc
> embedded scripts having some logics?
Well, I was maybe OT. Let's say we have just multiple selections in
sw-description, and application are stored in different partitions. Then
there is no Lua and this is treated automatically by SWUpdate. We can
update just the application, the OS or everything as usual. Currently,
we can do this, but the whole SWU is downloaded.
> Maybe we can revise the
> sw-description format and make it all Lua for example? As another
> alternative to libconfig and JSON then?
We have already...
This is done via the external parser, that calls a pure Lua code. The
example in repository is related to a XML parser using lua-lxp, but
let's say: you can already do whatever you want.
However, this high freedom has its drawbacks. As far as I know, just one
customer of mine has used this in the past, but he converted back to
libconfig/JSON later. In fact, a pure Lua solution requires that
everybody must implement and maintain the own parser, while I have added
features and features to the standard parsers. The original XML parser
developed by customer was then very poor compared to libconfig, and
customer himself decided to get rid of it.
Anyway, the feature is not removed, you have already a pure Lua parser
if you want...
> So you have the choice of
> "simple" configuration and programmatic Lua configuration?
> Of course, this would be a mid/long-term thing... Just an idea though.
Feature already implemented.
>
>
>> In SWUpdate design, there is a split between the installer (SWUpdate's
>> main process) and the processes getting the SWU. The last ones have no
>> idea about how the SWU is built and they rely on the installer. I want
>> to change slightly this: the downloders won't know the internals, but
>> they will be aware that SWU is a CPIO archive and that sw-description is
>> the first file in the SWU (this is hardcoded in SWUpdate by design).
>> The idea is to use RFC 7233 (" Hypertext Transfer Protocol (HTTP/1.1):
>> Range Requests") - all current HTTP servers support it. This means that
>> this new feature just works for HTTP(S), but this is the 90% of the
>> cases or more.
>
> Hm, so you're splitting the currently one download "function" into
> something like download_sw-description()
Well, to be more precise : download an entry from CPIO archive in SWU,
independently if it is sw-description or not.
> and download_payload(int) then
> that can be called from the different ingress channels (suricatta,
> download, ...)?
My idea is to add a pipeline to the core curl, so that any user of curl
in SWUpdate could use it (downloader, suricatta).
> Then, we should also make the downloader able to verify what it has
> downloaded, maybe by an IPC call to the core not to duplicate code.
Right - I want to maintain this split for security reason.
The core has also access to keys for verification and decryption. The
downloaders have no access and they should not gain access to them.
The downloaders should remain quite dumb: just load a chunk of data of a
fixed size (defined in CPIO), and ask the core if everything is ok to
move on.
>
>
>> I sketch my ideas as:
>>
>> 1. the downloader knows that the SWU is a CPIO. It starts downloading
>> the first CPIO header (110 bytes), sending a HTTP Range request.
>> 2. the downloader knows the size of sw-description, and load it together
>> with the next CPIO header (header of next file).
>> 3. if the header of next file is the signature, download it as well (and
>> asks also for the next CPIO header).
>> 4. push the data via IPC to the installer : sw-description is parsed,
>> and the internal structures are created. SWUpdate knows which artifacts
>> are required, and if the update must go on.
>> 5. the downloader asks via IPC (a new command) the installer if the next
>> file in the cpio is required. If yes, download it and push to the
>> installer, else skips it and asks via HTTP range request for the next
>> cpio header, until an artifact is required.
>
> Hm, the question here is who is the driver of the update process?
> Currently, the download is configured to stream it into the core, setup
> by the ingress mechanism (suricatta, downloader, ...). Thereafter, the
> ingress mechanism is out of the game waiting for SWUpdate core to finish.
Right, this is the current design and in my proposal I do not change
this. It just a "controlled download" of the stream.
>
> Another option would be to make the core call back to the ingress
> to get the next data chunk?
A nice thing in SWUpdate is that it has the same behavior independently
from the source. I have just one SWU if I update locally, or with the
internal Webserver, or via suricatta, or via an URL. This is a very
appreciated feature.
Change the way makes this very difficult, specially in case the update
is done via the internal Webserver. I know you have not this case, but
believe me, this is one of the most used way in many projects, and it is
also used by developers even if later the update in field is done in a
different way. The browser pushes data, no way to control it.
In SWUpdate, each handler is a consumer : it gets data that are already
made available from the core. Requesting data changes completely the way
to work, and this becomes a very big change.
I have sketched a design to implement something like this for a delta
update, but this is then just for one handler: the handler could receive
a URL for the artifact and ask to download it. It has also security
related aspects that must be considered, but it can be restricted to a
single handler.
But switching the concept completely is too much IMHO, and I do not know
if there are real advantages.
> This way, different download methods
> (HTTP/curl, Queue server, ...) could be easily supported by the ingress
> mechanism, hidden from SWUpdate core. If it's transport doesn't support
> RANGE requests, it can simulate this or fall back to the current design.
>
>> 6. loop 5 until the end of cpio, this is marked with the "TRAILER!"
>> magic word. Maybe some more controls are required here to avoid that an
>> attacker tries a DOS pushing non-sense data to the device.
>>
>> In this way, a simple check means to load just
>> sw-description+sw-description.sig, plus some bytes as overhead. And it
>> reduces the size of downloaded data in case some artifacts are already
>> on the device.
>
> You mean already installed or already downloaded to local storage?
I just means that the downloaded data to decide if the update should go
on is just the size of sw-description+signature.
>
>
>> I can also imagine a further step just for Hawkbit: multiple SWUs (as
>> software modules) can be put into a distribution, and SWUpdate can check
>> all of them finding the more appropriate for the update - for example, a
>> delta from Vx to Vy. The overhead becomes just the loading of
>> sw-description + signature, and it is worth as loading the whole image.
>
> Here you impose a particular form of building artifacts and introduce
> a convention.
Well, sw-description is the first file in SWU and this convention is
already defined.
> So you want to put multiple SWUs into one SWU as "payload"
This is already implemented (see gateway and swuforwarding handler), but
it is not what I meant.
What I mean here is to have a single SWU with multiple artifacts, and
just a set of artifacts is needed by a single device. Something like:
software = {
device-type-0001 = {
images: (..
}
device-type-0002 = {
images: (..
}
device-type-0003 = {
images: (..
}
Some of them can share an artifact or not - never mind. This already
works, and each device selects what it needs, discarding the rest, but
the whole SWU must be downloaded by all devices.
> of which one is chosen according to the sw-description's embedded Lua
> script(s) finding out which ones to install? This gives a lot of control
> to SWUpdate and hence the device.
This is already implemented, with or without Lua, via selections.
> This may or may not be wanted.
Of course - everything is configurable.
> The
> other "solution" is to have all combinations available in the cloud and
> pointing SWUpdate to a particular one depending on its former
> communication of versions.
This is what I really meant with Hawkbit, but let's say...it is a
different topic (another RFC..).
Hawkbit sends a list of "chunks" in the deployment answer.
{
"id" : "85",
"deployment" : {
"download" : "forced",
"update" : "forced",
"maintenanceWindow" : "available",
"chunks" : [ {
"part" : "bApp",
"version" : "1.0.84",
"name" : "oneapplication",
"artifacts" : [ {
"filename" : "SWU-device-0001.swu",
The whole list is sent back. A logic can be added to the suricatta
(Hawkbit server) to sort the entries and to find the most suitable for
the update. We could have like:
. SWU for whole software
- delta from version x
- delta from version y
- ....
And yes, I guess we need some convention and use "metadata" that are
stored with each software module in Hawkbit.
(note: i could also implement the use case above with different SWUs,
one for each device). A logic can then check which is the most suitable
starting from the current version, and load it.
>
>
> Regardless, I do think this is a valuable addition!
>
Thanks for review.
Best regards,
Stefano
On 06.01.21 18:00, Christian Storm wrote:
> Hi Stefano,
>
>> I am thinking about to implement a new feature and I am writing a
>> concept, asking the community for help and review. There is at the
>> moment no single line implemented, I want to check in advance drawbacks
>> and exchange ideas.
>>
>> Background: I get information from a manufacturer, whose devices in
>> field are just checking for new update twice a day. He is not using
>> Hawkbit, but he has an own server where he stored the SWUs. SWUpdate on
>> the device simply connects with the server and verifies if the whole SWU
>> must be downloaded. This means that SWUpdate starts to download the SWU,
>> then sw-description is parsed and the update is stop as soon as it is
>> recognized (using version schemas) that a new version is not required.
>
> Ok, so this is kind of a dumb server just serving the artifacts and
> SWUpdate does the logics on whether to install or not, right?
receives (I do not know how) the URL to get the SWU. Then the whole
logic if an update is required or not is implemented in SWUpdate.
>
>
>> Due to the security concept that requires to split download and
>> install in separate processes, update is stop after ~0.8 MB are
>> downloaded when the installer has parsed sw-description. The
>> downloader has already loaded more data and they lie in kernel (skb)
>> or SWUpdate buffers. If this seems a low value, manufacturer reports
>> that, due to his large installation, the update check generates more
>> as 100GB/day of unneeded traffic.
>
> With different download and install "processes" you mean two steps or
> two OS processes?
suricatta.
> For the former, I do in my still to be submitted Lua
> suricatta binding a dry-run which downloads the artifact and then use
> file://.. to install it from the local storage.
> This way, SWUpdate is
> able to check validity of the artifact and can early abort broken
> artifacts...
that I report here. Even in dry-run mode, you start to download the SWU,
then the SWU is pushed to core until the parser runs and raises the
abort. In the case I reported, the downloaded data until SWUpdate aborts
(from the parser) is ~0.8MB. As I said, this seems not too much, but it
sums with the amount of the installed base. The reason for this RFC is
to introduce a controlled download per file, so that just sw-description
is downloaded if an abort can be raised by the parser, reducing the
amount of downloaded data to some KBytes.
>
>
>> In case of backend, the traffic is surely reduced because SWUpdate just
>> asks Hawkbit if a new software is available via REST-API, without
>> checking itself inside the SWU.
>
> This is one point, the other is knowing somewhere in the backend which
> version(s) are installed on the devices, which is unknown to the backend
> in your scenario or is it reported by some middleware to some backend?
device knows which version is running, and SWUpdate is informed of it
with command line parameter. The version is then compared with the
"version" attribute in sw-description.
>
>
>> But adding a Hawkbit infrastructure is a no way for that manufacturer.
>> Anyway, I am thinking about a solution that brings benefits for
>> Hawkbit, too (and generally, when SWUpdate works in "pull" mode
>> loading itself from an external source). A use case is when the SWU
>> contains multiple artifacts, and just one of them needs to be updated
>> (for example, just the application). An ad-hoc logic can be
>> implemented as Lua script ("embedded-script" in sw-description),
>> and SWUpdate could avoid to download what will be marked as "non
>> required" after parsing, instead of downloading the whole SWU and skip
>> what is not necessary. This helps also in another use case, where SW
>> for multiple devices are packed together in one SWU (a requirement in
>> some application where storage servers are managed by organizations
>> and not by manufacturers), but bandwidth must be preserved. So these
>> are the main reasons for this RFC.
>
> Hm, I fear that sw-description will be overloaded with Lua ad-hoc
> embedded scripts having some logics?
sw-description, and application are stored in different partitions. Then
there is no Lua and this is treated automatically by SWUpdate. We can
update just the application, the OS or everything as usual. Currently,
we can do this, but the whole SWU is downloaded.
> Maybe we can revise the
> sw-description format and make it all Lua for example? As another
> alternative to libconfig and JSON then?
This is done via the external parser, that calls a pure Lua code. The
example in repository is related to a XML parser using lua-lxp, but
let's say: you can already do whatever you want.
However, this high freedom has its drawbacks. As far as I know, just one
customer of mine has used this in the past, but he converted back to
libconfig/JSON later. In fact, a pure Lua solution requires that
everybody must implement and maintain the own parser, while I have added
features and features to the standard parsers. The original XML parser
developed by customer was then very poor compared to libconfig, and
customer himself decided to get rid of it.
Anyway, the feature is not removed, you have already a pure Lua parser
if you want...
> So you have the choice of
> "simple" configuration and programmatic Lua configuration?
> Of course, this would be a mid/long-term thing... Just an idea though.
>
>
>> In SWUpdate design, there is a split between the installer (SWUpdate's
>> main process) and the processes getting the SWU. The last ones have no
>> idea about how the SWU is built and they rely on the installer. I want
>> to change slightly this: the downloders won't know the internals, but
>> they will be aware that SWU is a CPIO archive and that sw-description is
>> the first file in the SWU (this is hardcoded in SWUpdate by design).
>> The idea is to use RFC 7233 (" Hypertext Transfer Protocol (HTTP/1.1):
>> Range Requests") - all current HTTP servers support it. This means that
>> this new feature just works for HTTP(S), but this is the 90% of the
>> cases or more.
>
> Hm, so you're splitting the currently one download "function" into
> something like download_sw-description()
independently if it is sw-description or not.
> and download_payload(int) then
> that can be called from the different ingress channels (suricatta,
> download, ...)?
in SWUpdate could use it (downloader, suricatta).
> Then, we should also make the downloader able to verify what it has
> downloaded, maybe by an IPC call to the core not to duplicate code.
The core has also access to keys for verification and decryption. The
downloaders have no access and they should not gain access to them.
The downloaders should remain quite dumb: just load a chunk of data of a
fixed size (defined in CPIO), and ask the core if everything is ok to
move on.
>
>
>> I sketch my ideas as:
>>
>> 1. the downloader knows that the SWU is a CPIO. It starts downloading
>> the first CPIO header (110 bytes), sending a HTTP Range request.
>> 2. the downloader knows the size of sw-description, and load it together
>> with the next CPIO header (header of next file).
>> 3. if the header of next file is the signature, download it as well (and
>> asks also for the next CPIO header).
>> 4. push the data via IPC to the installer : sw-description is parsed,
>> and the internal structures are created. SWUpdate knows which artifacts
>> are required, and if the update must go on.
>> 5. the downloader asks via IPC (a new command) the installer if the next
>> file in the cpio is required. If yes, download it and push to the
>> installer, else skips it and asks via HTTP range request for the next
>> cpio header, until an artifact is required.
>
> Hm, the question here is who is the driver of the update process?
> Currently, the download is configured to stream it into the core, setup
> by the ingress mechanism (suricatta, downloader, ...). Thereafter, the
> ingress mechanism is out of the game waiting for SWUpdate core to finish.
this. It just a "controlled download" of the stream.
>
> Another option would be to make the core call back to the ingress
> to get the next data chunk?
from the source. I have just one SWU if I update locally, or with the
internal Webserver, or via suricatta, or via an URL. This is a very
appreciated feature.
Change the way makes this very difficult, specially in case the update
is done via the internal Webserver. I know you have not this case, but
believe me, this is one of the most used way in many projects, and it is
also used by developers even if later the update in field is done in a
different way. The browser pushes data, no way to control it.
In SWUpdate, each handler is a consumer : it gets data that are already
made available from the core. Requesting data changes completely the way
to work, and this becomes a very big change.
I have sketched a design to implement something like this for a delta
update, but this is then just for one handler: the handler could receive
a URL for the artifact and ask to download it. It has also security
related aspects that must be considered, but it can be restricted to a
single handler.
But switching the concept completely is too much IMHO, and I do not know
if there are real advantages.
> This way, different download methods
> (HTTP/curl, Queue server, ...) could be easily supported by the ingress
> mechanism, hidden from SWUpdate core. If it's transport doesn't support
> RANGE requests, it can simulate this or fall back to the current design.
>
>> 6. loop 5 until the end of cpio, this is marked with the "TRAILER!"
>> magic word. Maybe some more controls are required here to avoid that an
>> attacker tries a DOS pushing non-sense data to the device.
>>
>> In this way, a simple check means to load just
>> sw-description+sw-description.sig, plus some bytes as overhead. And it
>> reduces the size of downloaded data in case some artifacts are already
>> on the device.
>
> You mean already installed or already downloaded to local storage?
on is just the size of sw-description+signature.
>
>
>> I can also imagine a further step just for Hawkbit: multiple SWUs (as
>> software modules) can be put into a distribution, and SWUpdate can check
>> all of them finding the more appropriate for the update - for example, a
>> delta from Vx to Vy. The overhead becomes just the loading of
>> sw-description + signature, and it is worth as loading the whole image.
>
> Here you impose a particular form of building artifacts and introduce
> a convention.
already defined.
> So you want to put multiple SWUs into one SWU as "payload"
it is not what I meant.
What I mean here is to have a single SWU with multiple artifacts, and
just a set of artifacts is needed by a single device. Something like:
software = {
device-type-0001 = {
images: (..
}
device-type-0002 = {
images: (..
}
device-type-0003 = {
images: (..
}
Some of them can share an artifact or not - never mind. This already
works, and each device selects what it needs, discarding the rest, but
the whole SWU must be downloaded by all devices.
> of which one is chosen according to the sw-description's embedded Lua
> script(s) finding out which ones to install? This gives a lot of control
> to SWUpdate and hence the device.
> This may or may not be wanted.
> The
> other "solution" is to have all combinations available in the cloud and
> pointing SWUpdate to a particular one depending on its former
> communication of versions.
different topic (another RFC..).
Hawkbit sends a list of "chunks" in the deployment answer.
{
"id" : "85",
"deployment" : {
"download" : "forced",
"update" : "forced",
"maintenanceWindow" : "available",
"chunks" : [ {
"part" : "bApp",
"version" : "1.0.84",
"name" : "oneapplication",
"artifacts" : [ {
"filename" : "SWU-device-0001.swu",
The whole list is sent back. A logic can be added to the suricatta
(Hawkbit server) to sort the entries and to find the most suitable for
the update. We could have like:
. SWU for whole software
- delta from version x
- delta from version y
- ....
And yes, I guess we need some convention and use "metadata" that are
stored with each software module in Hawkbit.
(note: i could also implement the use case above with different SWUs,
one for each device). A logic can then check which is the most suitable
starting from the current version, and load it.
>
>
> Regardless, I do think this is a valuable addition!
>
Best regards,
Stefano
Stefano Babic
Jan 7, 2021, 5:12:16 AM1/7/21
to Georgiev, Veselin, Stefano Babic, swup...@googlegroups.com
Hi Georgiev,
On 06.01.21 23:16, 'Georgiev, Veselin' via swupdate wrote:
> Hi Stefano,
>
>
> I hope you are doing well. The ability to limit the amount of data
> downloaded and installed during the update would be desirable. I have
> the following feedback:
Thanks for it.
>
>
> * Would this change tie the format of the SWU file to the protocol
update from USB stick, from internal Webserver or from an external
backend. It is a great feature and it should be maintained.
> Would this make it difficult to switch to
> a new file format or change how SWU uses the cpio format?
I do not want to change the SWU - as you have pointed out, this raises a
lot of compatibility issues.
> For
> example, allowing images larger than 4 GB?
This is a different topic. Size of a single artifacts is currently
limited to 4GB, and this can be workarounded for some handlers using
offset. This is also something to be fixed, I have currently nobody
asking to increase this.
> * This may have already been mentioned. Another option would be to
we have big advantages.
> * The proposed design will increase the number of connections to the
have multiple of them. The total number of new connections is equal to
the number of artifacts to be downloaded, and it adds an overhead
specially in case of HTTPS connections. Maybe I should consider to make
the behavior configurable, and the integrator decides if it must be
activated.
Best regards,
Stefano Babic
>
>
> Thank you.
>
> Veselin Georgiev
>
> ------------------------------------------------------------------------
> *From:* swup...@googlegroups.com <swup...@googlegroups.com> on behalf
> of Stefano Babic <sba...@denx.de>
> *Sent:* Tuesday, January 5, 2021 4:22 AM
> *To:* 'swup...@googlegroups.com'
> *Subject:* [swupdate] RFC: Concept "Selective download"
> ------------------------------------------------------------------------
> <https://groups.google.com/d/msgid/swupdate/38dfbd02f20d4915b1c6d2c0d4b109ce%40garmin.com?utm_medium=email&utm_source=footer>.
On 06.01.21 23:16, 'Georgiev, Veselin' via swupdate wrote:
> Hi Stefano,
>
>
> I hope you are doing well. The ability to limit the amount of data
> downloaded and installed during the update would be desirable. I have
> the following feedback:
>
>
> * Would this change tie the format of the SWU file to the protocol
> used to retrieve the file?
No, I do not want to change this. The SWU is currently the same if you
update from USB stick, from internal Webserver or from an external
backend. It is a great feature and it should be maintained.
> Would this make it difficult to switch to
> a new file format or change how SWU uses the cpio format?
lot of compatibility issues.
> For
> example, allowing images larger than 4 GB?
limited to 4GB, and this can be workarounded for some handlers using
offset. This is also something to be fixed, I have currently nobody
asking to increase this.
> * This may have already been mentioned. Another option would be to
> invert the relationship between the downloader and installer so the
> installer can request a byte range from the downloader. This would
> keep the downloader as a simple proxy to the data and retain
> knowledge of the file format in the downloader. From the installer's
> perspective it would look as if it were reading the file from disk.
I answered to this, IMHO this is a very big change and I do not know if
> installer can request a byte range from the downloader. This would
> keep the downloader as a simple proxy to the data and retain
> knowledge of the file format in the downloader. From the installer's
> perspective it would look as if it were reading the file from disk.
we have big advantages.
> * The proposed design will increase the number of connections to the
> server/seek operations the server has to perform. There could be
> some performance decrease depending on the circumstances.
Sure, you are right. Instead of a single connection and GET request, we
> some performance decrease depending on the circumstances.
have multiple of them. The total number of new connections is equal to
the number of artifacts to be downloaded, and it adds an overhead
specially in case of HTTPS connections. Maybe I should consider to make
the behavior configurable, and the integrator decides if it must be
activated.
Best regards,
Stefano Babic
>
>
> Thank you.
>
> Veselin Georgiev
>
> ------------------------------------------------------------------------
> *From:* swup...@googlegroups.com <swup...@googlegroups.com> on behalf
> of Stefano Babic <sba...@denx.de>
> *Sent:* Tuesday, January 5, 2021 4:22 AM
> *To:* 'swup...@googlegroups.com'
> *Subject:* [swupdate] RFC: Concept "Selective download"
>
> CONFIDENTIALITY NOTICE: This email and any attachments are for the sole
> use of the intended recipient(s) and contain information that may be
> Garmin confidential and/or Garmin legally privileged. If you have
> received this email in error, please notify the sender by reply email
> and delete the message. Any disclosure, copying, distribution or use of
> this communication (including attachments) by someone other than the
> intended recipient is prohibited. Thank you.
>
> CONFIDENTIALITY NOTICE: This email and any attachments are for the sole
> use of the intended recipient(s) and contain information that may be
> Garmin confidential and/or Garmin legally privileged. If you have
> received this email in error, please notify the sender by reply email
> and delete the message. Any disclosure, copying, distribution or use of
> this communication (including attachments) by someone other than the
> intended recipient is prohibited. Thank you.
>
> --
> You received this message because you are subscribed to the Google
> Groups "swupdate" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to swupdate+u...@googlegroups.com
> <mailto:swupdate+u...@googlegroups.com>.
> You received this message because you are subscribed to the Google
> Groups "swupdate" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to swupdate+u...@googlegroups.com
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/swupdate/38dfbd02f20d4915b1c6d2c0d4b109ce%40garmin.com
> <https://groups.google.com/d/msgid/swupdate/38dfbd02f20d4915b1c6d2c0d4b109ce%40garmin.com?utm_medium=email&utm_source=footer>.
Christian Storm
Jan 7, 2021, 11:17:41 AM1/7/21
to swup...@googlegroups.com
Hi Stefano,
> [...]
Yes, that's right. Thought of it as a more "standard" feature. Like as
you said, there's currently no real reason to opt for this one since the
"standard" ones are so featureful. Instead of pushing this out so that
one can do whatever seems appropriate like in the example you mentioned,
the "all Lua" thing could get some official API to fill and modify
SWUpdate's structures. Then you can choose to do it programmatically or
via specification (language). But yes, anyway, I do see the point.
Is there some data or use case for using JSON btw.? Haven't used it
myself and hadn't had the need for it over the years... Just curious.
Yes, fully agree on this.
Just to add, this is not limited to curl but can as well be implemented
using other transport implementations...
> >> I sketch my ideas as:
> >>
> >> 1. the downloader knows that the SWU is a CPIO. It starts downloading
> >> the first CPIO header (110 bytes), sending a HTTP Range request.
> >> 2. the downloader knows the size of sw-description, and load it together
> >> with the next CPIO header (header of next file).
> >> 3. if the header of next file is the signature, download it as well (and
> >> asks also for the next CPIO header).
> >> 4. push the data via IPC to the installer : sw-description is parsed,
> >> and the internal structures are created. SWUpdate knows which artifacts
> >> are required, and if the update must go on.
> >> 5. the downloader asks via IPC (a new command) the installer if the next
> >> file in the cpio is required. If yes, download it and push to the
> >> installer, else skips it and asks via HTTP range request for the next
> >> cpio header, until an artifact is required.
> >
> > Hm, the question here is who is the driver of the update process?
> > Currently, the download is configured to stream it into the core, setup
> > by the ingress mechanism (suricatta, downloader, ...). Thereafter, the
> > ingress mechanism is out of the game waiting for SWUpdate core to finish.
>
> Right, this is the current design and in my proposal I do not change
> this. It just a "controlled download" of the stream.
OK, so the ingress (e.g. suricatta) does not "see" or has to know about
this feature, right? It still sets up the stream to the core and then
the core does its work. I got confused over the term and meaning of
"downloader" above I guess...
> > Another option would be to make the core call back to the ingress
> > to get the next data chunk?
>
> A nice thing in SWUpdate is that it has the same behavior independently
> from the source. I have just one SWU if I update locally, or with the
> internal Webserver, or via suricatta, or via an URL. This is a very
> appreciated feature.
>
> Change the way makes this very difficult, specially in case the update
> is done via the internal Webserver. I know you have not this case, but
> believe me, this is one of the most used way in many projects, and it is
> also used by developers even if later the update in field is done in a
> different way. The browser pushes data, no way to control it.
Actually, we do have one case only recently, so count me in :)
> In SWUpdate, each handler is a consumer : it gets data that are already
> made available from the core. Requesting data changes completely the way
> to work, and this becomes a very big change.
>
> I have sketched a design to implement something like this for a delta
> update, but this is then just for one handler: the handler could receive
> a URL for the artifact and ask to download it. It has also security
> related aspects that must be considered, but it can be restricted to a
> single handler.
>
> But switching the concept completely is too much IMHO, and I do not know
> if there are real advantages.
OK, got it. So the change is actually "isolated" to how the core
instruments the download stream it gets setup by the ingresses, right?
> > This way, different download methods
> > (HTTP/curl, Queue server, ...) could be easily supported by the ingress
> > mechanism, hidden from SWUpdate core. If it's transport doesn't support
> > RANGE requests, it can simulate this or fall back to the current design.
> >
> >> 6. loop 5 until the end of cpio, this is marked with the "TRAILER!"
> >> magic word. Maybe some more controls are required here to avoid that an
> >> attacker tries a DOS pushing non-sense data to the device.
> >>
> >> In this way, a simple check means to load just
> >> sw-description+sw-description.sig, plus some bytes as overhead. And it
> >> reduces the size of downloaded data in case some artifacts are already
> >> on the device.
> >
> > You mean already installed or already downloaded to local storage?
>
> I just means that the downloaded data to decide if the update should go
> on is just the size of sw-description+signature.
OK, got it.
> >> I can also imagine a further step just for Hawkbit: multiple SWUs (as
> >> software modules) can be put into a distribution, and SWUpdate can check
> >> all of them finding the more appropriate for the update - for example, a
> >> delta from Vx to Vy. The overhead becomes just the loading of
> >> sw-description + signature, and it is worth as loading the whole image.
> >
> > Here you impose a particular form of building artifacts and introduce
> > a convention.
>
> Well, sw-description is the first file in SWU and this convention is
> already defined.
>
> > So you want to put multiple SWUs into one SWU as "payload"
>
> This is already implemented (see gateway and swuforwarding handler), but
> it is not what I meant.
>
> What I mean here is to have a single SWU with multiple artifacts, and
> just a set of artifacts is needed by a single device. Something like:
>
> software = {
> device-type-0001 = {
> images: (..
> }
> device-type-0002 = {
> images: (..
> }
> device-type-0003 = {
> images: (..
> }
>
>
> Some of them can share an artifact or not - never mind. This already
> works, and each device selects what it needs, discarding the rest, but
> the whole SWU must be downloaded by all devices.
OK, now I got it, thanks.
Yes, this is what I meant or suspected.
> (note: i could also implement the use case above with different SWUs,
> one for each device). A logic can then check which is the most suitable
> starting from the current version, and load it.
True.
> > Regardless, I do think this is a valuable addition!
> >
>
> Thanks for review.
You're more than welcome, thanks for patiently answering!
> [...]
> > This way, SWUpdate is
> > able to check validity of the artifact and can early abort broken
> > artifacts...
>
> The problem is the definition of early abort. This is the same use case
> that I report here. Even in dry-run mode, you start to download the SWU,
> then the SWU is pushed to core until the parser runs and raises the
> abort. In the case I reported, the downloaded data until SWUpdate aborts
> (from the parser) is ~0.8MB. As I said, this seems not too much, but it
> sums with the amount of the installed base. The reason for this RFC is
> to introduce a controlled download per file, so that just sw-description
> is downloaded if an abort can be raised by the parser, reducing the
> amount of downloaded data to some KBytes.
Yes, agreed and understood.
> > able to check validity of the artifact and can early abort broken
> > artifacts...
>
> The problem is the definition of early abort. This is the same use case
> that I report here. Even in dry-run mode, you start to download the SWU,
> then the SWU is pushed to core until the parser runs and raises the
> abort. In the case I reported, the downloaded data until SWUpdate aborts
> (from the parser) is ~0.8MB. As I said, this seems not too much, but it
> sums with the amount of the installed base. The reason for this RFC is
> to introduce a controlled download per file, so that just sw-description
> is downloaded if an abort can be raised by the parser, reducing the
> amount of downloaded data to some KBytes.
you said, there's currently no real reason to opt for this one since the
"standard" ones are so featureful. Instead of pushing this out so that
one can do whatever seems appropriate like in the example you mentioned,
the "all Lua" thing could get some official API to fill and modify
SWUpdate's structures. Then you can choose to do it programmatically or
via specification (language). But yes, anyway, I do see the point.
Is there some data or use case for using JSON btw.? Haven't used it
myself and hadn't had the need for it over the years... Just curious.
Just to add, this is not limited to curl but can as well be implemented
using other transport implementations...
> >> I sketch my ideas as:
> >>
> >> 1. the downloader knows that the SWU is a CPIO. It starts downloading
> >> the first CPIO header (110 bytes), sending a HTTP Range request.
> >> 2. the downloader knows the size of sw-description, and load it together
> >> with the next CPIO header (header of next file).
> >> 3. if the header of next file is the signature, download it as well (and
> >> asks also for the next CPIO header).
> >> 4. push the data via IPC to the installer : sw-description is parsed,
> >> and the internal structures are created. SWUpdate knows which artifacts
> >> are required, and if the update must go on.
> >> 5. the downloader asks via IPC (a new command) the installer if the next
> >> file in the cpio is required. If yes, download it and push to the
> >> installer, else skips it and asks via HTTP range request for the next
> >> cpio header, until an artifact is required.
> >
> > Hm, the question here is who is the driver of the update process?
> > Currently, the download is configured to stream it into the core, setup
> > by the ingress mechanism (suricatta, downloader, ...). Thereafter, the
> > ingress mechanism is out of the game waiting for SWUpdate core to finish.
>
> Right, this is the current design and in my proposal I do not change
> this. It just a "controlled download" of the stream.
this feature, right? It still sets up the stream to the core and then
the core does its work. I got confused over the term and meaning of
"downloader" above I guess...
> > Another option would be to make the core call back to the ingress
> > to get the next data chunk?
>
> A nice thing in SWUpdate is that it has the same behavior independently
> from the source. I have just one SWU if I update locally, or with the
> internal Webserver, or via suricatta, or via an URL. This is a very
> appreciated feature.
>
> Change the way makes this very difficult, specially in case the update
> is done via the internal Webserver. I know you have not this case, but
> believe me, this is one of the most used way in many projects, and it is
> also used by developers even if later the update in field is done in a
> different way. The browser pushes data, no way to control it.
> In SWUpdate, each handler is a consumer : it gets data that are already
> made available from the core. Requesting data changes completely the way
> to work, and this becomes a very big change.
>
> I have sketched a design to implement something like this for a delta
> update, but this is then just for one handler: the handler could receive
> a URL for the artifact and ask to download it. It has also security
> related aspects that must be considered, but it can be restricted to a
> single handler.
>
> But switching the concept completely is too much IMHO, and I do not know
> if there are real advantages.
instruments the download stream it gets setup by the ingresses, right?
> > This way, different download methods
> > (HTTP/curl, Queue server, ...) could be easily supported by the ingress
> > mechanism, hidden from SWUpdate core. If it's transport doesn't support
> > RANGE requests, it can simulate this or fall back to the current design.
> >
> >> 6. loop 5 until the end of cpio, this is marked with the "TRAILER!"
> >> magic word. Maybe some more controls are required here to avoid that an
> >> attacker tries a DOS pushing non-sense data to the device.
> >>
> >> In this way, a simple check means to load just
> >> sw-description+sw-description.sig, plus some bytes as overhead. And it
> >> reduces the size of downloaded data in case some artifacts are already
> >> on the device.
> >
> > You mean already installed or already downloaded to local storage?
>
> I just means that the downloaded data to decide if the update should go
> on is just the size of sw-description+signature.
> >> I can also imagine a further step just for Hawkbit: multiple SWUs (as
> >> software modules) can be put into a distribution, and SWUpdate can check
> >> all of them finding the more appropriate for the update - for example, a
> >> delta from Vx to Vy. The overhead becomes just the loading of
> >> sw-description + signature, and it is worth as loading the whole image.
> >
> > Here you impose a particular form of building artifacts and introduce
> > a convention.
>
> Well, sw-description is the first file in SWU and this convention is
> already defined.
>
> > So you want to put multiple SWUs into one SWU as "payload"
>
> This is already implemented (see gateway and swuforwarding handler), but
> it is not what I meant.
>
> What I mean here is to have a single SWU with multiple artifacts, and
> just a set of artifacts is needed by a single device. Something like:
>
> software = {
> device-type-0001 = {
> images: (..
> }
> device-type-0002 = {
> images: (..
> }
> device-type-0003 = {
> images: (..
> }
>
>
> Some of them can share an artifact or not - never mind. This already
> works, and each device selects what it needs, discarding the rest, but
> the whole SWU must be downloaded by all devices.
> (note: i could also implement the use case above with different SWUs,
> one for each device). A logic can then check which is the most suitable
> starting from the current version, and load it.
> > Regardless, I do think this is a valuable addition!
> >
>
> Thanks for review.
Stefano Babic
Jan 7, 2021, 11:51:42 AM1/7/21
to swup...@googlegroups.com, Christian Storm
Hi Christian,
Right - my customer dropped then its own implementation, and uses
libconfig now.
> Instead of pushing this out so that
> one can do whatever seems appropriate like in the example you mentioned,
> the "all Lua" thing could get some official API to fill and modify
> SWUpdate's structures.
Well, we have an API and this is the same as for embedded-script, that
is an "image" table with all attributes. The parser in lua should push
as many table as the list of artifacts. This is implemented in
parse_external.c, but I haven't had any use case since years and I am
expecting some problems when someone will test himself (so yes, I am not
making any regression test with an external Lua parser).
> Then you can choose to do it programmatically or
> via specification (language). But yes, anyway, I do see the point.
>
> Is there some data or use case for using JSON btw.?
Reason to do this is that some customer generates with own tool (Java
based) the resulting sw-description, and JSON is naturally supported in
Java (not libconfig).
Anyway, I have done that the two parsers are fully compatible (there is
a common layer and custom functions for libconfig and json) and you can
even quite automatically convert from one to the other one (with a small
restriction). There is a Python "jsonlibconfig" :
https://pypi.org/project/jsonlibconfig/
The only thing is that SWUpdate always expects to have an array for
images, scripts, files, etc. In case there is just one element in the
array, the tool sets a scalar instead of an array with just one element,
and SWUPdate's parser raises an error. For my tests, I convert the
original "libconfig" sw-description to json with jsonlibconfig, and then
I fix myself where scalar is set instaead of an array with a single element.
> Haven't used it
> myself and hadn't had the need for it over the years... Just curious.
IMHO, sw-description is like any other source code, and syntax with
libconfig is smarter and more "friendly" (specially for Linux / kernel
developers) than JSON. But everyone can choose the best one for the own
project, I guarantee that the two parsers are compatible.
Right
>
>
>>>> I sketch my ideas as:
>>>>
>>>> 1. the downloader knows that the SWU is a CPIO. It starts downloading
>>>> the first CPIO header (110 bytes), sending a HTTP Range request.
>>>> 2. the downloader knows the size of sw-description, and load it together
>>>> with the next CPIO header (header of next file).
>>>> 3. if the header of next file is the signature, download it as well (and
>>>> asks also for the next CPIO header).
>>>> 4. push the data via IPC to the installer : sw-description is parsed,
>>>> and the internal structures are created. SWUpdate knows which artifacts
>>>> are required, and if the update must go on.
>>>> 5. the downloader asks via IPC (a new command) the installer if the next
>>>> file in the cpio is required. If yes, download it and push to the
>>>> installer, else skips it and asks via HTTP range request for the next
>>>> cpio header, until an artifact is required.
>>>
>>> Hm, the question here is who is the driver of the update process?
>>> Currently, the download is configured to stream it into the core, setup
>>> by the ingress mechanism (suricatta, downloader, ...). Thereafter, the
>>> ingress mechanism is out of the game waiting for SWUpdate core to finish.
>>
>> Right, this is the current design and in my proposal I do not change
>> this. It just a "controlled download" of the stream.
>
> OK, so the ingress (e.g. suricatta) does not "see" or has to know about
> this feature, right?
Right.
> It still sets up the stream to the core and then
> the core does its work. I got confused over the term and meaning of
> "downloader" above I guess...
This comes from iternal OS processes in SWUpdate. If we enable all way
to update, SWUpdate will fork itself at startup and it creates
"downloader", a "suricatta" and a "webserver" processes. The
"downloader" process is what is running when SWUpdate just gets a URL
for the SWU, without connecting to a backend (that is, the use case for
this RFC). Sorry for misunderstanding.
>
>
>>> Another option would be to make the core call back to the ingress
>>> to get the next data chunk?
>>
>> A nice thing in SWUpdate is that it has the same behavior independently
>> from the source. I have just one SWU if I update locally, or with the
>> internal Webserver, or via suricatta, or via an URL. This is a very
>> appreciated feature.
>>
>> Change the way makes this very difficult, specially in case the update
>> is done via the internal Webserver. I know you have not this case, but
>> believe me, this is one of the most used way in many projects, and it is
>> also used by developers even if later the update in field is done in a
>> different way. The browser pushes data, no way to control it.
>
> Actually, we do have one case only recently, so count me in :)
Ok
>
>
>> In SWUpdate, each handler is a consumer : it gets data that are already
>> made available from the core. Requesting data changes completely the way
>> to work, and this becomes a very big change.
>>
>> I have sketched a design to implement something like this for a delta
>> update, but this is then just for one handler: the handler could receive
>> a URL for the artifact and ask to download it. It has also security
>> related aspects that must be considered, but it can be restricted to a
>> single handler.
>>
>> But switching the concept completely is too much IMHO, and I do not know
>> if there are real advantages.
>
> OK, got it. So the change is actually "isolated" to how the core
> instruments the download stream it gets setup by the ingresses, right?
Right: goal is that it is transparent to suricatta and friends (the
"ingress"), and transparent to the handler (the consumers). The magical
should happen in between without affecting the rest.
Right.
>
>
>> (note: i could also implement the use case above with different SWUs,
>> one for each device). A logic can then check which is the most suitable
>> starting from the current version, and load it.
>
> True.
>
>
>>> Regardless, I do think this is a valuable addition!
>>>
>>
>> Thanks for review.
>
>
> You're more than welcome, thanks for patiently answering!
>
Best regards,
Stefano
libconfig now.
> Instead of pushing this out so that
> one can do whatever seems appropriate like in the example you mentioned,
> the "all Lua" thing could get some official API to fill and modify
> SWUpdate's structures.
is an "image" table with all attributes. The parser in lua should push
as many table as the list of artifacts. This is implemented in
parse_external.c, but I haven't had any use case since years and I am
expecting some problems when someone will test himself (so yes, I am not
making any regression test with an external Lua parser).
> Then you can choose to do it programmatically or
> via specification (language). But yes, anyway, I do see the point.
>
> Is there some data or use case for using JSON btw.?
based) the resulting sw-description, and JSON is naturally supported in
Java (not libconfig).
Anyway, I have done that the two parsers are fully compatible (there is
a common layer and custom functions for libconfig and json) and you can
even quite automatically convert from one to the other one (with a small
restriction). There is a Python "jsonlibconfig" :
https://pypi.org/project/jsonlibconfig/
The only thing is that SWUpdate always expects to have an array for
images, scripts, files, etc. In case there is just one element in the
array, the tool sets a scalar instead of an array with just one element,
and SWUPdate's parser raises an error. For my tests, I convert the
original "libconfig" sw-description to json with jsonlibconfig, and then
I fix myself where scalar is set instaead of an array with a single element.
> Haven't used it
> myself and hadn't had the need for it over the years... Just curious.
libconfig is smarter and more "friendly" (specially for Linux / kernel
developers) than JSON. But everyone can choose the best one for the own
project, I guarantee that the two parsers are compatible.
>
>
>>>> I sketch my ideas as:
>>>>
>>>> 1. the downloader knows that the SWU is a CPIO. It starts downloading
>>>> the first CPIO header (110 bytes), sending a HTTP Range request.
>>>> 2. the downloader knows the size of sw-description, and load it together
>>>> with the next CPIO header (header of next file).
>>>> 3. if the header of next file is the signature, download it as well (and
>>>> asks also for the next CPIO header).
>>>> 4. push the data via IPC to the installer : sw-description is parsed,
>>>> and the internal structures are created. SWUpdate knows which artifacts
>>>> are required, and if the update must go on.
>>>> 5. the downloader asks via IPC (a new command) the installer if the next
>>>> file in the cpio is required. If yes, download it and push to the
>>>> installer, else skips it and asks via HTTP range request for the next
>>>> cpio header, until an artifact is required.
>>>
>>> Hm, the question here is who is the driver of the update process?
>>> Currently, the download is configured to stream it into the core, setup
>>> by the ingress mechanism (suricatta, downloader, ...). Thereafter, the
>>> ingress mechanism is out of the game waiting for SWUpdate core to finish.
>>
>> Right, this is the current design and in my proposal I do not change
>> this. It just a "controlled download" of the stream.
>
> OK, so the ingress (e.g. suricatta) does not "see" or has to know about
> this feature, right?
> It still sets up the stream to the core and then
> the core does its work. I got confused over the term and meaning of
> "downloader" above I guess...
to update, SWUpdate will fork itself at startup and it creates
"downloader", a "suricatta" and a "webserver" processes. The
"downloader" process is what is running when SWUpdate just gets a URL
for the SWU, without connecting to a backend (that is, the use case for
this RFC). Sorry for misunderstanding.
>
>
>>> Another option would be to make the core call back to the ingress
>>> to get the next data chunk?
>>
>> A nice thing in SWUpdate is that it has the same behavior independently
>> from the source. I have just one SWU if I update locally, or with the
>> internal Webserver, or via suricatta, or via an URL. This is a very
>> appreciated feature.
>>
>> Change the way makes this very difficult, specially in case the update
>> is done via the internal Webserver. I know you have not this case, but
>> believe me, this is one of the most used way in many projects, and it is
>> also used by developers even if later the update in field is done in a
>> different way. The browser pushes data, no way to control it.
>
> Actually, we do have one case only recently, so count me in :)
>
>
>> In SWUpdate, each handler is a consumer : it gets data that are already
>> made available from the core. Requesting data changes completely the way
>> to work, and this becomes a very big change.
>>
>> I have sketched a design to implement something like this for a delta
>> update, but this is then just for one handler: the handler could receive
>> a URL for the artifact and ask to download it. It has also security
>> related aspects that must be considered, but it can be restricted to a
>> single handler.
>>
>> But switching the concept completely is too much IMHO, and I do not know
>> if there are real advantages.
>
> OK, got it. So the change is actually "isolated" to how the core
> instruments the download stream it gets setup by the ingresses, right?
"ingress"), and transparent to the handler (the consumers). The magical
should happen in between without affecting the rest.
>
>
>> (note: i could also implement the use case above with different SWUs,
>> one for each device). A logic can then check which is the most suitable
>> starting from the current version, and load it.
>
> True.
>
>
>>> Regardless, I do think this is a valuable addition!
>>>
>>
>> Thanks for review.
>
>
> You're more than welcome, thanks for patiently answering!
>
Stefano
Georgiev, Veselin
Jan 7, 2021, 3:26:19 PM1/7/21
to Stefano Babic, swup...@googlegroups.com
Hi Stefano,
I have one more thought on this. It makes sense that it would be a large amount of change to invert the dependency between the downloader and installer. I agree it is not be worth it. Could we accomplish something similar by modifying the IPC protocol in your description:
> 5. the downloader asks via IPC (a new command) the installer if the next
> file in the cpio is required. If yes, download it and push to the
> installer, else skips it and asks via HTTP range request for the next
> cpio header, until an artifact is required.
The IPC API would become one where the downloader asks the installer if more data should be read and the installer returns the byte range of data that it would like to receive. The downloader then fetches the data without any knowledge of what it is doing. Thanks.
Veselin Georgiev
From: Stefano Babic <sba...@denx.de>
Sent: Thursday, January 7, 2021 4:12:08 AM
To: Georgiev, Veselin; Stefano Babic; 'swup...@googlegroups.com'
Subject: Re: [swupdate] RFC: Concept "Selective download"
Sent: Thursday, January 7, 2021 4:12:08 AM
To: Georgiev, Veselin; Stefano Babic; 'swup...@googlegroups.com'
Subject: Re: [swupdate] RFC: Concept "Selective download"
>
https://urldefense.com/v3/__https://groups.google.com/d/msgid/swupdate/38dfbd02f20d4915b1c6d2c0d4b109ce*40garmin.com__;JQ!!EJc4YC3iFmQ!BvJ1iQpFpZWvDcKOTPOMfIp0ZinswVrk8kdG8HiDl_FAsNBi0QwwWHXAHJ65h4wV9kNYGQ$
> <https://urldefense.com/v3/__https://groups.google.com/d/msgid/swupdate/38dfbd02f20d4915b1c6d2c0d4b109ce*40garmin.com?utm_medium=email&utm_source=footer__;JQ!!EJc4YC3iFmQ!BvJ1iQpFpZWvDcKOTPOMfIp0ZinswVrk8kdG8HiDl_FAsNBi0QwwWHXAHJ65h4zAVSQ82Q$ >.
> <https://urldefense.com/v3/__https://groups.google.com/d/msgid/swupdate/38dfbd02f20d4915b1c6d2c0d4b109ce*40garmin.com?utm_medium=email&utm_source=footer__;JQ!!EJc4YC3iFmQ!BvJ1iQpFpZWvDcKOTPOMfIp0ZinswVrk8kdG8HiDl_FAsNBi0QwwWHXAHJ65h4zAVSQ82Q$ >.
--
=====================================================================
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: +49-8142-66989-53 Fax: +49-8142-66989-80 Email: sba...@denx.de
=====================================================================
Stefan Herbrechtsmeier
Jan 8, 2021, 2:27:06 PM1/8/21
to Stefano Babic, swup...@googlegroups.com
Hi Stefano,
Am 05.01.21 um 11:22 schrieb Stefano Babic:
[snip]
> In SWUpdate design, there is a split between the installer (SWUpdate's
> main process) and the processes getting the SWU. The last ones have no
> idea about how the SWU is built and they rely on the installer. I want
> to change slightly this: the downloders won't know the internals, but
> they will be aware that SWU is a CPIO archive and that sw-description is
> the first file in the SWU (this is hardcoded in SWUpdate by design).
Does this knowledge belongs inside the image provider? Maybe we could
extent the passive stream mode by an active mode. In the active mode the
installer controls the image provider. The installer instructs the image
provider to send x bytes or to skip x bytes. Its up to the image
provider if it discards data or requests only needed data. The
downloader could request the needed data from the server and the web
server could simple drop unneeded data.
> The idea is to use RFC 7233 (" Hypertext Transfer Protocol (HTTP/1.1):
> Range Requests") - all current HTTP servers support it. This means that
> this new feature just works for HTTP(S), but this is the 90% of the
> cases or more.
A similar feature is possible for the web server if we use a (maybe
slightly modified) resumable upload protocol like tus [1] or similar.
[snip]
> 5. the downloader asks via IPC (a new command) the installer if the next
> file in the cpio is required. If yes, download it and push to the
> installer, else skips it and asks via HTTP range request for the next
> cpio header, until an artifact is required.
I would request the offset and size from the installer and keep the
knowledge about files inside the install.
Best regards
Stefan
Am 05.01.21 um 11:22 schrieb Stefano Babic:
[snip]
> In SWUpdate design, there is a split between the installer (SWUpdate's
> main process) and the processes getting the SWU. The last ones have no
> idea about how the SWU is built and they rely on the installer. I want
> to change slightly this: the downloders won't know the internals, but
> they will be aware that SWU is a CPIO archive and that sw-description is
> the first file in the SWU (this is hardcoded in SWUpdate by design).
extent the passive stream mode by an active mode. In the active mode the
installer controls the image provider. The installer instructs the image
provider to send x bytes or to skip x bytes. Its up to the image
provider if it discards data or requests only needed data. The
downloader could request the needed data from the server and the web
server could simple drop unneeded data.
> The idea is to use RFC 7233 (" Hypertext Transfer Protocol (HTTP/1.1):
> Range Requests") - all current HTTP servers support it. This means that
> this new feature just works for HTTP(S), but this is the 90% of the
> cases or more.
slightly modified) resumable upload protocol like tus [1] or similar.
[snip]
> 5. the downloader asks via IPC (a new command) the installer if the next
> file in the cpio is required. If yes, download it and push to the
> installer, else skips it and asks via HTTP range request for the next
> cpio header, until an artifact is required.
knowledge about files inside the install.
Best regards
Stefan
Stefano Babic
Jan 9, 2021, 10:08:28 AM1/9/21
to Stefan Herbrechtsmeier, Stefano Babic, swup...@googlegroups.com
Hi Stefan,
On 08.01.21 20:27, Stefan Herbrechtsmeier wrote:
> Hi Stefano,
>
> Am 05.01.21 um 11:22 schrieb Stefano Babic:
> [snip]
>
>> In SWUpdate design, there is a split between the installer (SWUpdate's
>> main process) and the processes getting the SWU. The last ones have no
>> idea about how the SWU is built and they rely on the installer. I want
>> to change slightly this: the downloders won't know the internals, but
>> they will be aware that SWU is a CPIO archive and that sw-description is
>> the first file in the SWU (this is hardcoded in SWUpdate by design).
>
> Does this knowledge belongs inside the image provider?
Currently, not. The image provider has no knowledge at all, it has even
no knowledge about the length of the downloaded file. It just retrieve a
file from network and push to the core. If the file is too long (for
example, due to DDOS attack), the core recognizes it and stops the image
provider (it closes the pipe to communicate).
> Maybe we could
> extent the passive stream mode by an active mode. In the active mode the
> installer controls the image provider. The installer instructs the image
> provider to send x bytes or to skip x bytes. Its up to the image
> provider if it discards data or requests only needed data. The
> downloader could request the needed data from the server and the web
> server could simple drop unneeded data.
Mmhhh...I have to check this because it is a very different way as it is
currently implemented, and it should also work together with other
features, like "resume after power-cut" that uses a cached-file.
>
>
>> The idea is to use RFC 7233 (" Hypertext Transfer Protocol (HTTP/1.1):
>> Range Requests") - all current HTTP servers support it. This means that
>> this new feature just works for HTTP(S), but this is the 90% of the
>> cases or more.
>
> A similar feature is possible for the web server if we use a (maybe
> slightly modified) resumable upload protocol like tus [1] or similar.
Well, but this is not anymore a standard communication between browser
and webserver. In the same way, I could also build a custom protocol
between browser and SWUpdate, but again it is not standard at all.
>
> [snip]
>
>> 5. the downloader asks via IPC (a new command) the installer if the next
>> file in the cpio is required. If yes, download it and push to the
>> installer, else skips it and asks via HTTP range request for the next
>> cpio header, until an artifact is required.
>
> I would request the offset and size from the installer and keep the
> knowledge about files inside the install.
Thanks for your thoughts, same opinion was raised by Georgiev. I dig
more into the code to check how this can be realized.
Best regards,
Stefano
>
> Best regards
> Stefan
On 08.01.21 20:27, Stefan Herbrechtsmeier wrote:
> Hi Stefano,
>
> Am 05.01.21 um 11:22 schrieb Stefano Babic:
> [snip]
>
>> In SWUpdate design, there is a split between the installer (SWUpdate's
>> main process) and the processes getting the SWU. The last ones have no
>> idea about how the SWU is built and they rely on the installer. I want
>> to change slightly this: the downloders won't know the internals, but
>> they will be aware that SWU is a CPIO archive and that sw-description is
>> the first file in the SWU (this is hardcoded in SWUpdate by design).
>
> Does this knowledge belongs inside the image provider?
no knowledge about the length of the downloaded file. It just retrieve a
file from network and push to the core. If the file is too long (for
example, due to DDOS attack), the core recognizes it and stops the image
provider (it closes the pipe to communicate).
> Maybe we could
> extent the passive stream mode by an active mode. In the active mode the
> installer controls the image provider. The installer instructs the image
> provider to send x bytes or to skip x bytes. Its up to the image
> provider if it discards data or requests only needed data. The
> downloader could request the needed data from the server and the web
> server could simple drop unneeded data.
currently implemented, and it should also work together with other
features, like "resume after power-cut" that uses a cached-file.
>
>
>> The idea is to use RFC 7233 (" Hypertext Transfer Protocol (HTTP/1.1):
>> Range Requests") - all current HTTP servers support it. This means that
>> this new feature just works for HTTP(S), but this is the 90% of the
>> cases or more.
>
> A similar feature is possible for the web server if we use a (maybe
> slightly modified) resumable upload protocol like tus [1] or similar.
and webserver. In the same way, I could also build a custom protocol
between browser and SWUpdate, but again it is not standard at all.
>
> [snip]
>
>> 5. the downloader asks via IPC (a new command) the installer if the next
>> file in the cpio is required. If yes, download it and push to the
>> installer, else skips it and asks via HTTP range request for the next
>> cpio header, until an artifact is required.
>
> I would request the offset and size from the installer and keep the
> knowledge about files inside the install.
more into the code to check how this can be realized.
Best regards,
Stefano
>
> Best regards
> Stefan
Reply all
Reply to author
Forward
0 new messages