Streaming an update from a file / named pipe

264 views
Skip to first unread message

Jonas Mark (BT-FIR/ENG1)

unread,
Jul 10, 2018, 9:50:51 AM7/10/18
to Stefano Babic (sbabic@denx.de), swup...@googlegroups.com, GUAN Ben (BT-FIR/ENG1-Zhu), ZHANG Kevin (BT-FIR/ENG1-Zhu), Jonas Mark (BT-FIR/ENG1)
Hi Stefano,

we are using swupdate in a product and are wondering if streaming the
update to the target with zero copy would be possible with the
following constraint:

The update is downloaded to the target using a proprietary USB
protocol. It is implemented using a userspace tool and GadgetFS. The
tool will create a file containing the received firmware.

What are the required preconditions for streaming a firmware image
into swupdate?

What we tried is to use a named pipe (mkfifo). The download tool would
write into the pipe and swupdate should read from it. This does not
work because extrace_sw_description() calls lseek().

$ mkfifo /tmp/pipe.swu
$ swupdate -e stable,second -i /tmp/pipe.swu -v

and in another shell

$ cat /tmp/test.swu > /tmp/pipe.swu

Then swupdate give the following errors.

[ERROR] : SWUPDATE failed [0] ERROR core/cpio_utils.c : extract_sw_description : 541 : CPIO file corrupted : Illegal seek

[ERROR] : SWUPDATE failed [0] ERROR parser/parser.c : parse_cfg : 702 : Missing version in configuration file

[ERROR] : SWUPDATE failed [0] ERROR core/parser.c : parse : 196 : no parser available to parse sw-description!

[ERROR] : SWUPDATE failed [0] ERROR core/swupdate.c : install_from_file : 307 : failed to parse sw-description!

The same swu file works just fine when opened as a file.

$ swupdate -e stable,second -i /tmp/test.swu -v

Does anybody have a proposal how to achieve a streaming update in the
described setup?

This is the sw-description we are using.

software =
{
/* software version */
version = "4.0.0";

/* hardware revision, will be checked with /etc/hwrevision */
hardware-compatibility: ["4.0"];

/* because swupdate requires the user choose which image should be installed
* into which location. Using software collection feature to achieve that the
* the stand-by partition is already written.
*/
stable = {
first: {
images: (
{
filename = "rootfs.ext4.gz";
device = "/dev/mmcblk2p1";
compressed = true;
sha256 = "@SHA256SUM@";
}
);

bootenv: (
{
name = "boot_image";
value = "image1";
}
);
};
second: {
images: (
{
filename = "rootfs.ext4.gz";
device = "/dev/mmcblk2p2";
compressed = true;
sha256 = "@SHA256SUM@";
}
);

bootenv: (
{
name = "boot_image";
value = "image2";
}
);
};
};
}

Greetings,
Mark


Mark Jonas

Building Technologies, Panel Software Fire (BT-FIR/ENG1)
Bosch Sicherheitssysteme GmbH | Postfach 11 11 | 85626 Grasbrunn | GERMANY | www.boschsecurity.com

Sitz: Stuttgart, Registergericht: Amtsgericht Stuttgart HRB 23118
Aufsichtsratsvorsitzender: Stefan Hartung; Geschäftsführung: Gert van Iperen, Andreas Bartz, Thomas Quante, Bernhard Schuster

Stefano Babic

unread,
Jul 10, 2018, 10:12:13 AM7/10/18
to Jonas Mark (BT-FIR/ENG1), Stefano Babic (sbabic@denx.de), swup...@googlegroups.com, GUAN Ben (BT-FIR/ENG1-Zhu), ZHANG Kevin (BT-FIR/ENG1-Zhu)
Hi Mark,

On 10/07/2018 15:50, Jonas Mark (BT-FIR/ENG1) wrote:
> Hi Stefano,
>
> we are using swupdate in a product and are wondering if streaming the
> update to the target with zero copy would be possible with the
> following constraint:
>
> The update is downloaded to the target using a proprietary USB
> protocol. It is implemented using a userspace tool and GadgetFS. The
> tool will create a file containing the received firmware.
>
> What are the required preconditions for streaming a firmware image
> into swupdate?
>
> What we tried is to use a named pipe (mkfifo). The download tool would
> write into the pipe and swupdate should read from it. This does not
> work because extrace_sw_description() calls lseek()

This looks to me extra complicated. There is a simpler solutions that I
can propose.

.
>
> $ mkfifo /tmp/pipe.swu
> $ swupdate -e stable,second -i /tmp/pipe.swu -v

To make a general answer independently from your use case, this does
*not* work. The "-i" parameter implies to have a local file and SWUpdate
is informed that can use seek() on the file. This, of course, cannot work.

However, this case is foreseen using the API provided by SWUpdate and
the exported library (LGPL, you are allowed to link proprietary code).
There is in tools/client.c an example how to use.

Instead of writing into a pipe, you shoudl write into the pipe that
SWUpdate already makes available and using the function provided by the
library. The library has callbacks to push data, and you can fill in
when your data from the stream is available. Setup and free of internal
pipes are managed automatically by SWUpdate without user intervention.

>
> and in another shell
>
> $ cat /tmp/test.swu > /tmp/pipe.swu
>
> Then swupdate give the following errors.
>
> [ERROR] : SWUPDATE failed [0] ERROR core/cpio_utils.c : extract_sw_description : 541 : CPIO file corrupted : Illegal seek

No surprise: Illegal seek. This is expected.


>
> [ERROR] : SWUPDATE failed [0] ERROR parser/parser.c : parse_cfg : 702 : Missing version in configuration file
>
> [ERROR] : SWUPDATE failed [0] ERROR core/parser.c : parse : 196 : no parser available to parse sw-description!
>
> [ERROR] : SWUPDATE failed [0] ERROR core/swupdate.c : install_from_file : 307 : failed to parse sw-description!
>

The rest are consequences of the first error. You inform SWUpdate to use
a local file, but this is really a stream.

> The same swu file works just fine when opened as a file.
>
> $ swupdate -e stable,second -i /tmp/test.swu -v

It is enough to start :

swupdate -v -e stable,second

SWUpdate will start waiting for incoming requests.

>
> Does anybody have a proposal how to achieve a streaming update in the
> described setup?

Yes

>
> This is the sw-description we are using.
>
> software =
> {
> /* software version */
> version = "4.0.0";
>
> /* hardware revision, will be checked with /etc/hwrevision */
> hardware-compatibility: ["4.0"];
>
> /* because swupdate requires the user choose which image should be installed
> * into which location. Using software collection feature to achieve that the
> * the stand-by partition is already written.
> */
> stable = {
> first: {
> images: (
> {
> filename = "rootfs.ext4.gz";
> device = "/dev/mmcblk2p1";
> compressed = true;

if you are using a double-copy approach, you can safe add
"installed-directly" for zero copy, else the images are temporary stored
in your /tmp.

> sha256 = "@SHA256SUM@";
> }
> );
>
> bootenv: (
> {
> name = "boot_image";
> value = "image1";
> }
> );
> };
> second: {
> images: (
> {
> filename = "rootfs.ext4.gz";
> device = "/dev/mmcblk2p2";
> compressed = true;
> sha256 = "@SHA256SUM@";
> }
> );
>
> bootenv: (
> {
> name = "boot_image";
> value = "image2";
> }
> );
> };
> };
> }
>

Best regards,
Stefano


--
=====================================================================
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: +49-8142-66989-53 Fax: +49-8142-66989-80 Email: sba...@denx.de
=====================================================================

Jonas Mark (BT-FIR/ENG1)

unread,
Jul 11, 2018, 8:27:27 AM7/11/18
to Stefano Babic, swup...@googlegroups.com, GUAN Ben (BT-FIR/ENG1-Zhu), ZHANG Kevin (BT-FIR/ENG1-Zhu), Jonas Mark (BT-FIR/ENG1)
Hi Stefano,

> This looks to me extra complicated. There is a simpler solutions that I
> can propose.
>
> .
> >
> > $ mkfifo /tmp/pipe.swu
> > $ swupdate -e stable,second -i /tmp/pipe.swu -v
>
> To make a general answer independently from your use case, this does
> *not* work. The "-i" parameter implies to have a local file and SWUpdate
> is informed that can use seek() on the file. This, of course, cannot work.
>
> However, this case is foreseen using the API provided by SWUpdate and
> the exported library (LGPL, you are allowed to link proprietary code).
> There is in tools/client.c an example how to use.
>
> Instead of writing into a pipe, you shoudl write into the pipe that
> SWUpdate already makes available and using the function provided by the
> library. The library has callbacks to push data, and you can fill in
> when your data from the stream is available. Setup and free of internal
> pipes are managed automatically by SWUpdate without user intervention.

Thank you, we like the proposal.

There is just a little catch: It seems as if the existing YP swupdate
recipe as well as the Makefile only provide a statically linkable
IPC library. That is, the Makefile provides ipc/lib.a and the YP recipe
copies that to ${D}${libdir}/libswupdate.a. Is that observation
correct?

Would you prefer a patch for the Makefile or for the YP meta-layer?

> It is enough to start :
>
> swupdate -v -e stable,second
>
> SWUpdate will start waiting for incoming requests.

We will also take this advice and start swupdate at boot.

Greetings,
Mark

Stefano Babic

unread,
Jul 11, 2018, 8:44:30 AM7/11/18
to Jonas Mark (BT-FIR/ENG1), Stefano Babic, swup...@googlegroups.com, GUAN Ben (BT-FIR/ENG1-Zhu), ZHANG Kevin (BT-FIR/ENG1-Zhu)
Hi Mark,

On 11/07/2018 14:27, Jonas Mark (BT-FIR/ENG1) wrote:
> Hi Stefano,
>
>> This looks to me extra complicated. There is a simpler solutions that I
>> can propose.
>>
>> .
>>>
>>> $ mkfifo /tmp/pipe.swu
>>> $ swupdate -e stable,second -i /tmp/pipe.swu -v
>>
>> To make a general answer independently from your use case, this does
>> *not* work. The "-i" parameter implies to have a local file and SWUpdate
>> is informed that can use seek() on the file. This, of course, cannot work.
>>
>> However, this case is foreseen using the API provided by SWUpdate and
>> the exported library (LGPL, you are allowed to link proprietary code).
>> There is in tools/client.c an example how to use.
>>
>> Instead of writing into a pipe, you shoudl write into the pipe that
>> SWUpdate already makes available and using the function provided by the
>> library. The library has callbacks to push data, and you can fill in
>> when your data from the stream is available. Setup and free of internal
>> pipes are managed automatically by SWUpdate without user intervention.
>
> Thank you, we like the proposal.
>

You're welcome.

> There is just a little catch: It seems as if the existing YP swupdate
> recipe as well as the Makefile only provide a statically linkable
> IPC library.

Yes, but I push to -master a patch to build the library as relocatable code.

> That is, the Makefile provides ipc/lib.a and the YP recipe
> copies that to ${D}${libdir}/libswupdate.a. Is that observation
> correct?

This is correct.

>
> Would you prefer a patch for the Makefile or for the YP meta-layer?

Well, I have no idea which kind of patch or which issue you want to
address...

>
>> It is enough to start :
>>
>> swupdate -v -e stable,second
>>
>> SWUpdate will start waiting for incoming requests.
>
> We will also take this advice and start swupdate at boot.
>

Best regards,
Stefano Babic

Jonas Mark (BT-FIR/ENG1)

unread,
Jul 11, 2018, 9:34:10 AM7/11/18
to Stefano Babic, swup...@googlegroups.com, GUAN Ben (BT-FIR/ENG1-Zhu), ZHANG Kevin (BT-FIR/ENG1-Zhu), Jonas Mark (BT-FIR/ENG1)
Hi Stefano,

> > There is just a little catch: It seems as if the existing YP swupdate
> > recipe as well as the Makefile only provide a statically linkable
> > IPC library.
>
> Yes, but I push to -master a patch to build the library as relocatable code.
>
> > That is, the Makefile provides ipc/lib.a and the YP recipe
> > copies that to ${D}${libdir}/libswupdate.a. Is that observation
> > correct?
>
> This is correct.
>
> >
> > Would you prefer a patch for the Makefile or for the YP meta-layer?
>
> Well, I have no idea which kind of patch or which issue you want to
> address...

Sorry for the confusion. I'll try a longer and better explanation.

We are using swupdate 2018.03. I understood previously from you that
the IPC library is LGPL licensed. The Makefile will produce a ipc/lib.a
binary. Using ipc/lib.a means one needs to statically link against it.
To fulfil the LGPL obligations one would need to either put the whole
application under LGPL or at least provide .o files so the ipc/lib.a
can be exchanged.

We think that a better way out would be if the IPC library would be
supplied as a shared object. In this case the .so library can be easily
replaced by a custom version.

I now had a closer look at the ipc/*.c source files and network_ipc.c
is indeed LGPL 2.1+ licensed. But I was astonished to see that
progress_ipc.c is GPL 2.0+ licensed. This means that the when using
ipc/lib.a the complete GPL 2.0+ obligations have to be fulfilled. Is
that mix of licenses in ipc/lib.a intentional?

Is there an intention from your side to supply a shared object (.so
library) which only contains ipc/network_ipc.c?

Would it make sense to talk with Christian Storm (Siemens.com) and
Stefan Herbrechtsmeier (Weidmueller.com) about changing the license of
progress_ipc.c to LGPL-2.1+ so the whole IPC code would have the same
license and could be part of the same IPC .so library?

Stefano Babic

unread,
Jul 11, 2018, 9:47:41 AM7/11/18
to Jonas Mark (BT-FIR/ENG1), Stefano Babic, swup...@googlegroups.com, GUAN Ben (BT-FIR/ENG1-Zhu), ZHANG Kevin (BT-FIR/ENG1-Zhu)
Hi Mark,

On 11/07/2018 15:34, Jonas Mark (BT-FIR/ENG1) wrote:
> Hi Stefano,
>
>>> There is just a little catch: It seems as if the existing YP swupdate
>>> recipe as well as the Makefile only provide a statically linkable
>>> IPC library.
>>
>> Yes, but I push to -master a patch to build the library as relocatable code.
>>
>>> That is, the Makefile provides ipc/lib.a and the YP recipe
>>> copies that to ${D}${libdir}/libswupdate.a. Is that observation
>>> correct?
>>
>> This is correct.
>>
>>>
>>> Would you prefer a patch for the Makefile or for the YP meta-layer?
>>
>> Well, I have no idea which kind of patch or which issue you want to
>> address...
>
> Sorry for the confusion. I'll try a longer and better explanation.
>
> We are using swupdate 2018.03. I understood previously from you that
> the IPC library is LGPL licensed. The Makefile will produce a ipc/lib.a
> binary. Using ipc/lib.a means one needs to statically link against it.
> To fulfil the LGPL obligations one would need to either put the whole
> application under LGPL or at least provide .o files so the ipc/lib.a
> can be exchanged.
>
> We think that a better way out would be if the IPC library would be
> supplied as a shared object. In this case the .so library can be easily
> replaced by a custom version.

On the other way, there are good reasons to have SWUpdate self contained
as much as possible. Even if it is not a shared object, everybody can
simply link the static library and generates a shared object, if this is
required.

>
> I now had a closer look at the ipc/*.c source files and network_ipc.c
> is indeed LGPL 2.1+ licensed. But I was astonished to see that
> progress_ipc.c is GPL 2.0+ licensed. This means that the when using
> ipc/lib.a the complete GPL 2.0+ obligations have to be fulfilled. Is
> that mix of licenses in ipc/lib.a intentional?

This was not intentional, it flows into mainline without notice. It
should be fixed.

>
> Is there an intention from your side to supply a shared object (.so
> library) which only contains ipc/network_ipc.c?

No, I will let it as static, everybody can generate a shared if needed.
I do the same: see LUA binding in https://github.com/sbabic/lua-swupdate

>
> Would it make sense to talk with Christian Storm (Siemens.com) and
> Stefan Herbrechtsmeier (Weidmueller.com) about changing the license of
> progress_ipc.c to LGPL-2.1+ so the whole IPC code would have the same
> license and could be part of the same IPC .so library?

I write a patch to change License, yes.
Reply all
Reply to author
Forward
0 new messages