Should we rewrite or improve the API for external programs?

[Walter Bonetti]

Hello folks,

I looked into https://sbabic.github.io/swupdate/swupdate-ipc-interface.html?highlight=api. I found a high-level protocol explanation in it. But, it's not enough to write a library/framework to favorise alternative tools/services.

I started the reverse-engineering process from the source code and filed the blank with the protocol implementation. I found a `union` in https://github.com/sbabic/swupdate/blob/master/include/network_ipc.h to describe all messages.

Because of this kind of design approach, the documentation part is not easy to write. A third-party component should not rely on an easy breakable part of an API.

Facts:

 - union relies on the biggest entity present on the C-struct)

 - all read/write ops rely on `sizeof(msg)` in `network_ipc.c`.

Should we debate about refactoring or creating new interfaces?

Considerations:

• The API should be easily implementable in all languages.
• The API should not be easily breakable.
• The API should provide at least all current features (status, upload...)
• The API should standardize protocol or well-defined protocol with documentation that helps third-party implementation.
• Mongoose/web-app shouldn't be a part of the SW update project.

From these considerations, I need support from maintainers, contributors and community members to elaborate a plan that satisfied all of us.

Before any proposal, do you have any pieces of advice, ideas, highlights, pros, cons, requested features or whatever you want to put in this debate?

Thank you all for your time,

Best Regards,

[Stefano Babic]

Hi Walter,

On 20.03.22 21:21, 'Walter Bonetti' via swupdate wrote:
> Hello folks,
>
>
> I looked into
> https://sbabic.github.io/swupdate/swupdate-ipc-interface.html?highlight=api.
> I found a high-level protocol explanation in it. But, it's not enough to
> write a library/framework to favorise alternative tools/services.
>

This API is described in documentation, but it is also thought that
details remain hidden using the binding libraries (libswupdate and
lua_swupdate) provided with the project.

>
> I started the reverse-engineering process from the source code and filed
> the blank with the protocol implementation. I found a `union` in
> https://github.com/sbabic/swupdate/blob/master/include/network_ipc.h to
> describe all messages.
>
>
> Because of this kind of design approach, the documentation part is not
> easy to write. A third-party component should not rely on an easy
> breakable part of an API.
>
>
> Facts:
>
>  - union relies on the biggest entity present on the C-struct)
>

Right - when I started with API, I have weighted this but I I have not
considered it a big drawback because API is only for local connection on
UDS and the transfer size remains limited (still in range of KBytes).

>  - all read/write ops rely on `sizeof(msg)` in `network_ipc.c`.

Correct - to overcome this, user should take advantage of the provided
libraries instead of accessing the API directly.

>
>
> Should we debate about refactoring or creating new interfaces?
>

Let's see - at this point of project, compatibility is important and we
cannot break the past. Improvement must take care of this.

>
> Considerations:
>
> * The API should be easily implementable in all languages.

Good question, but frankly speaking, I had no request for this, and user
are mostly happy with C++ binding (and Lua).

Which language do you plan in your application ? Example ?

> * The API should not be easily breakable.

Ok

> * The API should provide at least all current features (status, upload...)

Take care that "status" is still there, but it is replaced by the
"progress" interface (not covered in network_ipc.h).

> * The API should standardize protocol or well-defined protocol

What are we talking about ? Can you tell something more, examples ?

> with
> documentation that helps third-party implementation.

Nevertheless, the idea is that 3rd party binding are still part of the
project and they are part of the "binding" area.

> * Mongoose/web-app shouldn't be a part of the SW update project.

The web-app is a separate application with other license, and it resides
in project's repository due to pragmatic reasons. The build of npm
packages under Yocto was (not checked recently) a mess. This lead to
have built app (in examples/..) inside the project instead of creating a
new project. The web-app has then quite no update and no incoming
patches, and most users take the placeholders (background, icon, etc.)
and that's all. The overhead to split into an own project with ML, etc.,
seemed to me too much.

Instead, the Mongoose webserver must be part of the project. The way is
into SWUpdate is the way thought by Monggose's developer ("include the
mongoose.c in your project"), and the API in mongoose are also not
stable and they changed significantly. The binding between SWUpdate and
Mongoose is part of the project, and most users want to have the
Webserver simply running.

>
>
> From these considerations, I need support from maintainers,
> contributors and community members to elaborate a plan that satisfied
> all of us.

You have to describe something more, because I do not know your plan -
can you extend your description with a short list of what you would like
to do and which issues you currently have ? (Binding to Haskell / Scala,
I do not know..)

>
>
> Before any proposal, do you have any pieces of advice, ideas,
> highlights, pros, cons, requested features or whatever you want to put
> in this debate?
>
>

Best regards,
Stefano Babic


--
=====================================================================
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: +49-8142-66989-53 Fax: +49-8142-66989-80 Email: sba...@denx.de
=====================================================================

[Michael Adler]

Hi,

> * The API should be easily implementable in all languages.
> * The API should not be easily breakable.

this I second. A few more points to consider:

* struct alignment
* usage of enum in structs
* endianness

> Correct - to overcome this, user should take advantage of the provided libraries instead of accessing the API
> directly.

This is fine as long as the connection is local. If the socket is actually remote and just appears to be local thanks to
something like socat, this will not protect you from the issues I mentioned above.

Kind regards,
Michael

--
Michael Adler

Siemens AG
T RDA IOT SES-DE
Otto-Hahn-Ring 6
81739 München, Deutschland

Siemens Aktiengesellschaft: Vorsitzender des Aufsichtsrats: Jim Hagemann Snabe; Vorstand: Roland Busch, Vorsitzender; Klaus Helmrich, Cedrik Neike, Matthias Rebellius, Ralf P. Thomas, Judith Wiese; Sitz der Gesellschaft: Berlin und München, Deutschland; Registergericht: Berlin-Charlottenburg, HRB 12300, München, HRB 6684; WEEE-Reg.-Nr. DE 23691322

[Stefano Babic]

Hi Michael,

On 21.03.22 10:58, Michael Adler wrote:
> Hi,
>
>> * The API should be easily implementable in all languages.
>> * The API should not be easily breakable.
>
> this I second. A few more points to consider:
>
> * struct alignment
> * usage of enum in structs

Ok

> * endianness
>

Endianess means that SWUpdate and issuer are running on different
machines, and the case was strictly forbidden.

>> Correct - to overcome this, user should take advantage of the provided libraries instead of accessing the API
>> directly.
>
> This is fine as long as the connection is local.

Anyway, this is a must. It is *NOT* foreseen that the API (I mean, what
is exported from network_ipc.h) goes on network. The connection is
strictly local and runs just on UDS.

> If the socket is actually remote

It is currently forbidden and under your risk - SWUpdate implements
privilege separation, too, and this is guaranteed if the socket is
strictly local, and the bound clients implementing the API are running
with lower rights on the same device as SWUpdate. There is currently no
way to connect from outside via API to the installer, and to do this you
need at least a proxa on the device forwarding the incoming requests.
But then you are working-around the security. The provided API for the
world is just the REST-API of the embedded server, but then the
Webserver is acting between SWUpdate and the client.

> and just appears to be local

It is strictly local.

> thanks to
> something like socat,

..but with socat, you are connecting SWUpdate (mainly runnignas root) to
the external world. Of course, you can do it, of course you are opening
a lot of security issues.

To be clear: there is no intention that the API (as explained in
network_ipc.h) is thought to be used remotely, it is thought to be used
only by processes running on the device itself and if the user knows how
security is affected doing this.

> this will not protect you from the issues I mentioned above.
>

Best regards,
Stefano

[Stefano Babic]

Hi Walter,

please never drop the ML in your reply, thanks !

Best regards,
Stefano

On 21.03.22 13:22, Walter Bonetti wrote:
> Hi Stefano and Michael,
>
>
> Thank you so much for your messages and your time,
>
>
> Yes, I can elaborate on a use case.
>
>
> I think a simple API should be available to integrate SWUpdate software
> with alternative tooling.
>
>
> # 1 - use case
>
>
> Write an HTTP service (e.g in Rust) that allows update (POST) without
> depending on any no necessary services (e.g mongoose) or C-binding usage.
>
> Example: This HTTP REST API can be in Rust with an authentication
> controller. We should be able to update the target from this secure
> endpoint.
>
> * What is your current best approach to do it?
>
>
>> This API is described in the documentation, but it is also thought that

>
> details remain hidden using the binding libraries (libswupdate and
>
> lua_swupdate) provided with the project.
>
>

> I rely on the documentation that says `API for external programs`. So,
> it's, from my understanding point of view, the best approach to use the
> `API for external programs` for communication.

>
>
>> Correct - to overcome this, user should take advantage of the provided
>
> libraries instead of accessing the API directly.
>
>

> I am not sure about this part. Is there another API or under the hood
> the libraries accessing the API directly?
>
>
> # 2 - use case
>
>
> Write an alternative progress client that updates a local event bus (e.g
> MQTT Broker) with the progress status. Again, my point of view is that
> `API for external programs` should be used to listen to progress status
> and send events through the event bus.
>
>
> # Anwsers
>
>
>> struct alignment, usage of enum in structs, endianness
>
>
> If the API is well-documented and easy, you should be able to implement
> your own external interface/adapter between remote and local transport
> layers. I don't think is important to deal with these parts for a local
> communication IPC. I second Stefano on it.
>
> But, you're right about enum and compiler-specific syntax interpretation.

>
>
>> To be clear: there is no intention that the API (as explained in
>
> network_ipc.h) is thought to be used remotely, it is thought to be used
>
> only by processes running on the device itself and if the user knows how
>
> security is affected doing this.
>
>

> I don't plan to expose it remotely. Read about my use case on top.

>
>
>>> The API should standardize protocol or well-defined protocol
>

>> What are we talking about ? Can you tell something more, examples ?
>

>> Let's see - at this point of the project, compatibility is important and we

>
> cannot break the past. Improvement must take care of this.
>
>

> Yes,
>
> * Michael and I talk about it from a developer perspective. enum,
> nonweighted type and compiler dependent type like enum should not be
> in well-based API. We can enforce types, reduce the breakability by
> removing the union and document all headers, messages, etc...
> * Or techno switch, we can use a self-documented protocol like
> Protobuf to build the IPC API. We can do it smoothly by providing
> another interface before deprecating the current one.
>
>
> Thank you all,
>
>
> B. r
>
>
> *Walter BONETTI*
>
> Software Developer
>
>
> ------------------------------------------------------------------------
> *From:* Stefano Babic <sba...@denx.de>
> *Sent:* Monday, March 21, 2022 7:39 AM
> *To:* Michael Adler <michae...@siemens.com>; Walter Bonetti
> <wbon...@dimonoff.com>
> *Cc:* swup...@googlegroups.com <swup...@googlegroups.com>
> *Subject:* Re: [swupdate] Should we rewrite or improve the API for
> external programs?

[Walter Bonetti]

Stefano,

My bad, thank you,

---

From: Stefano Babic <sba...@denx.de>
Sent: Monday, March 21, 2022 9:14 AM
To: Walter Bonetti <wbon...@dimonoff.com>; Stefano Babic <sba...@denx.de>
Cc: 'swup...@googlegroups.com' <swup...@googlegroups.com>
Subject: Re: [swupdate] Should we rewrite or improve the API for external programs?

 

[Walter Bonetti]

Sorry for the spam,

Hi Stefano and Michael,

Thank you so much for your messages and your time,

Yes, I can elaborate on a use case.

I think a simple API should be available to integrate SWUpdate software with alternative tooling.

# 1 - use case

Write an HTTP service (e.g in Rust) that allows update (POST) without depending on any no necessary services (e.g mongoose) or C-binding usage.

Example: This HTTP REST API can be in Rust with an authentication controller. We should be able to update the target from this secure endpoint. 

• What is your current best approach to do it?

> This API is described in the documentation, but it is also thought that

details remain hidden using the binding libraries (libswupdate and

lua_swupdate) provided with the project.

I rely on the documentation that says `API for external programs`. So, it's, from my understanding point of view, the best approach to use the `API for external programs` for communication.

> Correct - to overcome this, user should take advantage of the provided

libraries instead of accessing the API directly.

I am not sure about this part. Is there another API or under the hood the libraries accessing the API directly?

# 2 - use case

Write an alternative progress client that updates a local event bus (e.g MQTT Broker) with the progress status. Again, my point of view is that `API for external programs` should be used to listen to progress status and send events through the event bus. 

# Anwsers 

> struct alignment, usage of enum in structs, endianness

If the API is well-documented and easy, you should be able to implement your own external interface/adapter between remote and local transport layers. I don't think is important to deal with these parts for a local communication IPC. I second Stefano on it.

But, you're right about enum and compiler-specific syntax interpretation.

> To be clear: there is no intention that the API (as explained in

network_ipc.h) is thought to be used remotely, it is thought to be used

only by processes running on the device itself and if the user knows how

security is affected doing this.

I don't plan to expose it remotely. Read about my use case on top. 

>> The API should standardize protocol or well-defined protocol

> What are we talking about ? Can you tell something more, examples ?

> Let's see - at this point of the project, compatibility is important and we

cannot break the past. Improvement must take care of this.

Yes,

• Michael and I talk about it from a developer perspective. enum, nonweighted type and compiler dependent type like enum should not be in well-based API. We can enforce types, reduce the breakability by removing the union and document all headers, messages, etc...

• Or techno switch, we can use a self-documented protocol like Protobuf to build the IPC API. We can do it smoothly by providing another interface before deprecating the current one.

Thank you all,

B. r

Walter BONETTI

Software Developer

---

From: Walter Bonetti <wbon...@dimonoff.com>
Sent: Monday, March 21, 2022 9:20 AM
To: Stefano Babic <sba...@denx.de>

[Stefano Babic]

Hi Walter,

On 21.03.22 14:31, 'Walter Bonetti' via swupdate wrote:
> Sorry for the spam,
>
>
> Hi Stefano and Michael,
>
>
> Thank you so much for your messages and your time,
>
>
> Yes, I can elaborate on a use case.
>
>
> I think a simple API should be available to integrate SWUpdate software
> with alternative tooling.
>
>
> # 1 - use case
>
>
> Write an HTTP service (e.g in Rust) that allows update (POST) without
> depending on any no necessary services (e.g mongoose) or C-binding usage.
>
> Example: This HTTP REST API can be in Rust with an authentication
> controller. We should be able to update the target from this secure
> endpoint.

Apart writing with a new language, it is quite questionable if
thisshould be done another time. The reason to take the HTTP server from
the mongoose project was to have a solid base, a wide used and open
source (still GPLv2) project, and that is well maintained. With HTTP /
SSL / TLS there are a lot of thing that can go wrong, and just to rewrite...

I thought myself at the early beginning if I had to write myself the
code or take another project. Think also that SWUpdate supports
zwero-copy (streaming mode) even with the Webserver, and mongoose allows
callbacks to intercepts URL and attach own code. You have to rewrite
this just to have the same.

And you can still use your own HTTP server acting as proxy - SWUpdate
can work and was tested using NGINX as proxy. IMO this use case is like
reinventing the wheel.

For this use case, you do not use the API but the REST-API from Webserver.

Documentation is poor:

https://github.com/sbabic/swupdate/blob/master/doc/source/swupdate-ipc-interface.rst#api-to-the-integrated-webserver

But you have POST /upload, and progress interface via Websocket.

>
> * What is your current best approach to do it?
>

See above - do not use the internal API, use the REST-API provided by
the Webserver.

>
>> This API is described in the documentation, but it is also thought that
>
> details remain hidden using the binding libraries (libswupdate and
>
> lua_swupdate) provided with the project.
>
>
> I rely on the documentation that says `API for external programs`.

Let's explain what I meant with "external programs": an external program
is a separate process, not linked to SWUpdate, but running on the same
device. I have not meant remote processes running on separate devices.
There is no security for them.

> So,
> it's, from my understanding point of view, the best approach to use the
> `API for external programs` for communication.
>
>
>> Correct - to overcome this, user should take advantage of the provided
>
> libraries instead of accessing the API directly.
>
>
> I am not sure about this part. Is there another API or under the hood
> the libraries accessing the API directly?

There is another API für #1, it is the REST-API for the Webserver, see link.

>
>
> # 2 - use case
>
>
> Write an alternative progress client that updates a local event bus (e.g
> MQTT Broker) with the progress status.

This was already done on several projects, but because the license is
more permissive, none was reported back. The IPC is not in
network_ipc.h, but in progress_ipc.h. Because this is just a description
of a message, it is quite straightforward and users have built their own
version in the preferred language just connecting to the socket.

> Again, my point of view is that
> `API for external programs` should be used to listen to progress status
> and send events through the event bus.

Again, the progress interface, too, is thought to send events to local
processes. If sopmeone wnats to send over the network, he should
implement himself. I am just opne to collect these implementation, if
they are reported back to the project. An example is in the
mongoose_interface.c, that listens to the progress interface (locally)
and sends events back to the connected browser via a WS connection.

>
>
> # Anwsers
>
>
>> struct alignment, usage of enum in structs, endianness
>
>
> If the API is well-documented and easy, you should be able to implement
> your own external interface/adapter between remote and local transport
> layers.

Still bother about "remote" processes - the API is thought for
application running on the target and security is provided then by Linux
itself. I mean, right to the socket can be done with standard Linux
tool. With Remote Access, security must be provided in other way. The
SWUpdate-mongoose interface relies then on SSL, certificates,
authentication (if set), privilege separation (Webserver runs with own
user id) before sending to SWUpdate. And the real validation is done
inside SWUpdate, that means even if a manipulated SWU is sent, it is not
installed.

> I don't think is important to deal with these parts for a local
> communication IPC. I second Stefano on it.
>

+1

> But, you're right about enum and compiler-specific syntax interpretation.
>
>
>> To be clear: there is no intention that the API (as explained in
>
> network_ipc.h) is thought to be used remotely, it is thought to be used
>
> only by processes running on the device itself and if the user knows how
>
> security is affected doing this.
>
>
> I don't plan to expose it remotely. Read about my use case on top.
>

I read it, yes, I guess Micael wanted to expose it, but you want to
rewrite well tested code for #1.

>
>>> The API should standardize protocol or well-defined protocol
>
>> What are we talking about ? Can you tell something more, examples ?
>
>> Let's see - at this point of the project, compatibility is important and we
>
> cannot break the past. Improvement must take care of this.
>
>
> Yes,
>
> * Michael and I talk about it from a developer perspective. enum,
> nonweighted type and compiler dependent type like enum should not be
> in well-based API. We can enforce types, reduce the breakability by
> removing the union and document all headers, messages, etc...

Ok

> * Or techno switch, we can use a self-documented protocol like
> Protobuf to build the IPC API. We can do it smoothly by providing
> another interface before deprecating the current one.

We can add a further IPC, but I won't deprecate this. Why haven't I use
one already provided bus, like DBUS ?

Reason is that not all systems are using dbus (or any), and SWUpdate
runs on small systems, too. The numbver of dependencies should be
maintained as small as possible. And in a rescue system, where the whole
software (bootloader + kernel + ramdisk with SWUpdate) must be stored on
a small flash, bigger packages are forbidden. Then it comes to create an
own IPC with quite no dependencies. Just to tell : the C++ library is
not small and I drop on many rescue system - I do not see any chance
with protobuf & company.

Best regards,
Stefano

>
>
> Thank you all,
>
>
> B. r
>
>
> *Walter BONETTI*
>
> Software Developer
>
> ------------------------------------------------------------------------

> *From:* Walter Bonetti <wbon...@dimonoff.com>
> *Sent:* Monday, March 21, 2022 9:20 AM
> *To:* Stefano Babic <sba...@denx.de>

> *Cc:* 'swup...@googlegroups.com' <swup...@googlegroups.com>
> *Subject:* Re: [swupdate] Should we rewrite or improve the API for
> external programs?

> Stefano,
>
> My bad, thank you,
>
>

> ------------------------------------------------------------------------
> *From:* Stefano Babic <sba...@denx.de>

> *Sent:* Monday, March 21, 2022 9:14 AM
> *To:* Walter Bonetti <wbon...@dimonoff.com>; Stefano Babic <sba...@denx.de>

> --
> You received this message because you are subscribed to the Google
> Groups "swupdate" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to swupdate+u...@googlegroups.com
> <mailto:swupdate+u...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/swupdate/YQBPR0101MB92308776C7CCFB0ABDB8C2B3A3169%40YQBPR0101MB9230.CANPRD01.PROD.OUTLOOK.COM
> <https://groups.google.com/d/msgid/swupdate/YQBPR0101MB92308776C7CCFB0ABDB8C2B3A3169%40YQBPR0101MB9230.CANPRD01.PROD.OUTLOOK.COM?utm_medium=email&utm_source=footer>.

[Christian Storm]

Hi,

> > > This API is described in the documentation, but it is also thought that
> >
> > details remain hidden using the binding libraries (libswupdate and
> >
> > lua_swupdate) provided with the project.
> >
> >
> > I rely on the documentation that says `API for external programs`.
>
> Let's explain what I meant with "external programs": an external program is a
> separate process, not linked to SWUpdate, but running on the same device. I
> have not meant remote processes running on separate devices. There is no
> security for them.

This is IMO the most important argument and I want to second Stefano:
These are "internal-external" APIs in the sense that they allow
modularization and extensibility of SWUpdate but are to considered
"glueing" APIs between closely entangled and related components, that is,
different local OS processes (SWUpdate's as well as processes from code
not in the SWUpdate repository) that work together to achieve a common goal:
Updating the system.

This doesn't mean that there's no room for improvement on this
"internal" API, however:

To me, the #pragma pack(1) alignment etc. pp. of the structures has
a strict local motivation: Consider a FFI mapping (e.g. LuaJIT) for
which you currently have to twiddle and find out the alignment and
proper sizes (with always wasting some bytes on the "wire", whether
that's considered a problem is a different discussion...).
This would be way easier when having defined/introspectible/specified
bounds and alignments. Currently, it's C used as a programming
language *and* interface specification language...
Granted, one could just use the provided access libraries (for Lua
and C) and be done with it as this is actually their reason of
existence. Anyway...



Kind regards,
Christian

--
Dr. Christian Storm
Siemens AG, Technology, T CED SES-DE
Otto-Hahn-Ring 6, 81739 München, Germany

[Dimitar Maznekov]

In that way of talking about API - we make test to implement SWUpdate for our devices.

Now we develop Client GUI (Ionic for Android and iOS) who will check Hawkbit for updates about particular device with his ID.
Then GUI will download the update and send to device when connect to device's access point. The device doesn't have Internet nor access to Hawkbit.

We expect difficulties to upload to the device .swu file causing CORS policy violation.

app/connect:1 Access to XMLHttpRequest at 'http://10.100.30.17:8080/' from origin 'http://localhost:8100' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource.

That problem was discused in https://groups.google.com/g/swupdate/c/8VVYyZd-CbI

I've read carefully your thoughts and I'm agree about "internal API" but it's ok for IoT devices which naturally have constant or sporadic access to Internet.

What about API extension for devices w/o network access and manageable by user with GUI/app/local network. As per example Car DVRs, entertainment systems etc.

That correspond with "open" API communicate with apps and devices outside security domain of origin SWupdate device.

--
С уважение: Д.Мазнеков
Best regards: D.Maznekov
DevTeam ThermalVision department
Senior IT
Optix Co. Bulgaria
Mob:+359-88-5166659
www.optixco.com

---

OPTIX Co. Disclaimer ISO 27001:2013 [ F1 ]
This e-mail, including attachments, may include confidential and/or proprietary information, and may be used only by the person or entity to which it is addressed. If the reader of this e-mail is not the intended recipient or his or her authorized agent, the reader is hereby notified that any dissemination, distribution or copying of this e-mail is prohibited. If you have received this e-mail in error, please notify the sender by replying to this message and delete this e-mail immediately.
OPTIX Co. is an industry leader in the field of optics, opto-mechanical devices and opto-electronic systems. We provide the most comprehensive solutions of custom designs for both civil and defense application.

---

[Stefano Babic]

Hi Dimitar,

On 22.03.22 11:53, 'Dimitar Maznekov' via swupdate wrote:
> In that way of talking about API - we make test to implement SWUpdate
> for our devices.
>
> Now we develop Client GUI (Ionic for Android and iOS) who will check
> Hawkbit for updates about particular device with his ID.
> Then GUI will download the update and send to device when connect to
> device's access point. The device doesn't have Internet nor access to
> Hawkbit.
>
> We expect difficulties to upload to the device .swu file causing CORS
> policy violation.
>

Ok, you have an error here, but what has this to do with the discussion
of API ?

> app/connect:1 Access to XMLHttpRequest at 'http://10.100.30.17:8080/

> <https://www.google.com/url?q=http://10.100.30.17:8080/&sa=D&source=hangouts&ust=1648029245452000&usg=AOvVaw0USfP0dCRRCGf_4OaWKlNw>'

> from origin 'http://localhost:8100

> <https://www.google.com/url?q=http://localhost:8100&sa=D&source=hangouts&ust=1648029245452000&usg=AOvVaw1y2bCy6rzgHd8RfPriFMqQ>'

> has been blocked by CORS policy: Response to preflight request doesn't
> pass access control check: No 'Access-Control-Allow-Origin' header is
> present on the requested resource.
>
>
>
> That problem was discused in
> https://groups.google.com/g/swupdate/c/8VVYyZd-CbI
>

Problem was left open

> I've read carefully your thoughts and I'm agree about "internal API" but
> it's ok for IoT devices which naturally have constant or sporadic access
> to Internet.
> What about API extension for devices w/o network access and manageable
> by user with GUI/app/local network. As per example Car DVRs,
> entertainment systems etc.

This is just your issue in your project - there is a REST-API, there is
an issue, this must be verified and fixed.

> That correspond with "open" API communicate with apps and devices
> outside security domain of origin SWupdate device.

???

Best regards,
Stefano Babic

> *С уважение: Д.Мазнеков*
> *Best regards: D.Maznekov*

> DevTeam ThermalVision department
> Senior IT
> Optix Co. Bulgaria
> Mob:+359-88-5166659
> www.optixco.com
>
>
>
>

> ------------------------------------------------------------------------
> /OPTIX Co. Disclaimer *ISO 27001:2013 [ F1 ] *

> This e-mail, including attachments, may include confidential and/or
> proprietary information, and may be used only by the person or entity to
> which it is addressed. If the reader of this e-mail is not the intended
> recipient or his or her authorized agent, the reader is hereby notified
> that any dissemination, distribution or copying of this e-mail is
> prohibited. If you have received this e-mail in error, please notify the

> sender by replying to this message and delete this e-mail immediately. /
> *OPTIX Co. is an industry leader in the field of optics, opto-mechanical

> devices and opto-electronic systems. We provide the most comprehensive

> solutions of custom designs for both civil and defense application. *
> ------------------------------------------------------------------------

>
> --
> You received this message because you are subscribed to the Google
> Groups "swupdate" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to swupdate+u...@googlegroups.com
> <mailto:swupdate+u...@googlegroups.com>.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/swupdate/badb4bab-35c9-9980-709f-0054ac280da2%40optixco.com
> <https://groups.google.com/d/msgid/swupdate/badb4bab-35c9-9980-709f-0054ac280da2%40optixco.com?utm_medium=email&utm_source=footer>.

[Stefano Babic]

Hi Dimitar,

please do not drop the ML from the answer, thanks !

On 22.03.22 21:30, Topper wrote:

>
>
> On Tuesday, March 22, 2022 at 5:25:54 PM UTC+2 Stefano Babic wrote:
>
> Hi Dimitar,
>
> On 22.03.22 11:53, 'Dimitar Maznekov' via swupdate wrote:
> > In that way of talking about API - we make test to implement
> SWUpdate
> > for our devices.
> >
> > Now we develop Client GUI (Ionic for Android and iOS) who will check
> > Hawkbit for updates about particular device with his ID.
> > Then GUI will download the update and send to device when connect to
> > device's access point. The device doesn't have Internet nor
> access to
> > Hawkbit.
> >
> > We expect difficulties to upload to the device .swu file causing
> CORS
> > policy violation.
> >
>
> Ok, you have an error here, but what has this to do with the discussion
> of API ?
>

> Don't swupdate_client.py|c use API ? Don't se POST /upload from API
> discused right here? Correct me if I'm wrong.

Well, swupdate_client.py is frankly speaking far away to be a full
functional client, it is more or less a way to show how to connect via
python and it has full space for improvements. I mainly use it for test
purposes to script update on devices without using the browser.

But anyway, SWUpdate exposes a webserver and it is supposed to work with
browsers.

>
>
>
> > app/connect:1 Access to XMLHttpRequest at

> 'http://10.100.30.17:8080/ <http://10.100.30.17:8080/>

> >
> <https://www.google.com/url?q=http://10.100.30.17:8080/&sa=D&source=hangouts&ust=1648029245452000&usg=AOvVaw0USfP0dCRRCGf_4OaWKlNw
> <https://www.google.com/url?q=http://10.100.30.17:8080/&sa=D&source=hangouts&ust=1648029245452000&usg=AOvVaw0USfP0dCRRCGf_4OaWKlNw>>'
>

> > from origin 'http://localhost:8100 <http://localhost:8100>

> >
> <https://www.google.com/url?q=http://localhost:8100&sa=D&source=hangouts&ust=1648029245452000&usg=AOvVaw1y2bCy6rzgHd8RfPriFMqQ
> <https://www.google.com/url?q=http://localhost:8100&sa=D&source=hangouts&ust=1648029245452000&usg=AOvVaw1y2bCy6rzgHd8RfPriFMqQ>>'
>
> > has been blocked by CORS policy: Response to preflight request
> doesn't
> > pass access control check: No 'Access-Control-Allow-Origin'
> header is
> > present on the requested resource.
> >
> >
> >
> > That problem was discused in
> > https://groups.google.com/g/swupdate/c/8VVYyZd-CbI
> <https://groups.google.com/g/swupdate/c/8VVYyZd-CbI>
> >
>
> Problem was left open
>
> > I've read carefully your thoughts and I'm agree about "internal
> API" but
> > it's ok for IoT devices which naturally have constant or sporadic
> access
> > to Internet.
> > What about API extension for devices w/o network access and
> manageable
> > by user with GUI/app/local network. As per example Car DVRs,
> > entertainment systems etc.
>
> This is just your issue in your project - there is a REST-API, there is
> an issue, this must be verified and fixed.
>

> Sure, but that CORS became from Mongoose.

Or from mongoose-swupdate internal interface.

> Sure you are right about your
> toughts regarding ready-to-go http service and I'm agree too, but then
> cross origin sharing should be resolved then with API, don't they ?

If there is an issue and a general agreement how t ofix it, sure, it
should be, but this is more related to SWUpdate's webserver (or internal
interface to mongoose, or..).

>
>
>
> > That correspond with "open" API communicate with apps and devices
> > outside security domain of origin SWupdate device.
>
> ???
>

> Anyway Angular and Rest APIs can't go over that problem, will test with
> NGINX proxy.

> > Mob:+359-88-5166659 <tel:+359%2088%20516%206659>
> > www.optixco.com <http://www.optixco.com>

> <https://groups.google.com/d/msgid/swupdate/badb4bab-35c9-9980-709f-0054ac280da2%40optixco.com?utm_medium=email&utm_source=footer

> <https://groups.google.com/d/msgid/swupdate/badb4bab-35c9-9980-709f-0054ac280da2%40optixco.com?utm_medium=email&utm_source=footer>>.
>
>
>
> --
> =====================================================================
> DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
> HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany

> Phone: +49-8142-66989-53 <tel:+49%208142%206698953> Fax:
> +49-8142-66989-80 <tel:+49%208142%206698980> Email: sba...@denx.de
> =====================================================================

[Walter Bonetti]

Hi folks,

It's been a long time without any response on my side. I was overloaded. Sorry!

First, thank you @Stefano Babic,

# Why Mongoose should not be mandatory?

• Limits IPC to HTTP and prevents third-party tools.
• Limits usage over TCP domain sockets. 
• The bind 0.0.0.0 can't be changed in mongoose SW update implementation...).

The hard dependecy in the yocto recipe to mongoose server and the Web-UI is a limitation.

As I said, mongoose should not be part of the SW update. But a third-party implementation. The current mongoose interface doesn't fit well and limits others third-party to provides alternatives.

# What is about optimization?

I don't think we should over optimize things. We should prefere breakable logic parts and "eco-system's integration".

# Outdated IPC/documentation :

I was implementing my own swupdate-progress (Rust Based) to map event over my internal mqtt bus.

• We should update/fix the documentation: https://github.com/sbabic/swupdate/blob/master/doc/source/progress.rst. (see the missing field `dwl_bytes`).
• We should prefere a packed struct for `ProgressMsg`.
• Internal clients' sockets list doesn't work. Workaround is to re-open the unix domain at interval and hope to get an event...

 

# Current workaround to bypass mongoose requirements

• Use a custom http server to upload the .swu.
• Use the swupdate command line to flash the .swu.
• Create a custom implementation for `/tmp/swupdateprog`.

> This was already done on several projects, but because the license is

more permissive, none was reported back. The IPC is not in

network_ipc.h, but in progress_ipc.h. Because this is just a description

of a message, it is quite straightforward and users have built their own

version in the preferred language just connecting to the socket.

> Again, the progress interface, too, is thought to send events to local

processes. If sopmeone wnats to send over the network, he should

implement himself. I am just opne to collect these implementation, if

they are reported back to the project. An example is in the

mongoose_interface.c, that listens to the progress interface (locally)

and sends events back to the connected browser via a WS connection.

It's more about... nasty IPC interfaces... IPC interfaces actually limit community contribution...

In the current state of the art, a contribution requires `proc` and compiler-based knowledge...

People develop coupled code and can't share it.

I just want to make IPC endpoints user-friendly, flexible and allow everyone to implement their own send_update and event monitoring. 

Best regards,

Walter

[James Hilliard]

FYI I implemented a pure python asyncio swupdate client using ctypes that works
pretty well with both unix socket interfaces, not sure if it makes
sense to try and
upstream as an alternative interface method to the c libraries.

This allows me to have my python control application do streaming updates
downloads from an aiohttp client to the swupdate installation socket interface.:
https://docs.aiohttp.org/en/stable/client_quickstart.html#streaming-response-content

I mostly avoided swupdate libraries since I wasn't really sure how to make them
play nicely with python's asyncio event loop.

One advantage of an alternative implementation is that it can help reveal bugs
in the socket interface that might otherwise get hidden due to implementation
details(I think I found and fixed a few issues when working on my
python client).

> To unsubscribe from this group and stop receiving emails from it, send an email to swupdate+u...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/swupdate/YQBPR0101MB9230DDFCA3295B7F08C9CA6AA3169%40YQBPR0101MB9230.CANPRD01.PROD.OUTLOOK.COM.

[Stefano Babic]

Hi Walter,

On 14.06.22 01:10, 'Walter Bonetti' via swupdate wrote:
> Hi folks,
>
>
> It's been a long time without any response on my side. I was overloaded.
> Sorry!
>

Well, I forgot most of the discussion...

>
> First, thank you @Stefano Babic <mailto:sba...@denx.de>,

>
>
> # Why Mongoose should not be mandatory?

Is it mandatory ? You can eved drop it, it is configurable. Can we add
another Webserver interface ? Yes, we can if we write another connector.
The popint is to have another implementation that let us to stream the
SWU without any temporary copy, like Mongoose allows.

In SWUpdate there are:

CONFIG_WEBSERVER
CONFIG_MONGOOSE

It is just that mongoose is the current only implementation for the
webserver.

>
> * Limits IPC to HTTP and prevents third-party tools.

Not understood.

> * Limits usage over TCP domain sockets.

???

> * The bind 0.0.0.0 can't be changed in mongoose SW update
> implementation...).

True, just send a patch for this. The mongoose server can bind to a
different interface, the swupdate-mongoose interface does not foresee it.

>
>
> The hard dependecy in the yocto recipe to mongoose server and the Web-UI
> is a limitation.

No idea what is this hard dependency. You can drop both of them. And the
Web-UI has a MIT license, you can rewrite completely the Web-UI if you
want. I do not see any limitation.

>
>
> As I said, mongoose should not be part of the SW update. But a
> third-party implementation.

The way mongoose is used is the way thought by mongoose's developer.
Goal was to create a single C file that can be easy imported by othe
rapplications. Exaclty what SWUpdate is doing.

And then as I said, you can add further implementations.

> The current mongoose interface doesn't fit
> well and limits others third-party to provides alternatives.
>

This is a statement without explanation - I understand it is your point
of view, there are no facts to support it.

>
> # What is about optimization?
>
> I don't think we should over optimize things. We should prefere
> breakable logic parts and "eco-system's integration".
>
>
> # Outdated IPC/documentation :
>
> I was implementing my own swupdate-progress (Rust Based) to map event
> over my internal mqtt bus.
>

> * We should update/fix the documentation:

> https://github.com/sbabic/swupdate/blob/master/doc/source/progress.rst.
> (see the missing field `dwl_bytes`).

Send a patch.

> * We should prefere a packed struct for `ProgressMsg`.

Ok - using packed is required as soon as an interface is thought for a
communication between two different systems (we have not) or built with
differetn compilers (your use case). It was simply not required up now,
but sure we can do it.

See also that the "bindings" directory is thought to extend interface to
other languages, but just when this is requested and necessary. Feel
free to add binding for rust, then.

> * Internal clients' sockets list doesn't work.

????

This does not say a lot - if it does not work, it should be fixed. If
you have found an issue, send a patch for it.

> Workaround is to
> re-open the unix domain at interval and hope to get an event...

I guess you have found something during your implementation, but without
code or a detailed explanation is not easy to understand what you mind.

There are many applications working and getting the event in this way.

>
> # Current workaround to bypass mongoose requirements
>

> * Use a custom http server to upload the .swu.
> * Use the swupdate command line to flash the .swu.
> * Create a custom implementation for `/tmp/swupdateprog`.

I have not understood the limitations you have, so I cannot understand
the workarounds. What you mean with "custom implementation of
/tmp/swupdateprog" ?

>
>
>> This was already done on several projects, but because the license is
>
> more permissive, none was reported back. The IPC is not in
>
> network_ipc.h, but in progress_ipc.h. Because this is just a description
>
> of a message, it is quite straightforward and users have built their own
>
> version in the preferred language just connecting to the socket.
>
>> Again, the progress interface, too, is thought to send events to local
>
> processes. If sopmeone wnats to send over the network, he should
>
> implement himself. I am just opne to collect these implementation, if
>
> they are reported back to the project. An example is in the
>
> mongoose_interface.c, that listens to the progress interface (locally)
>
> and sends events back to the connected browser via a WS connection.
>
>
> It's more about... nasty IPC interfaces... IPC interfaces actually limit
> community contribution...

This seems your opinion.

The IPC itself, that is the ones based on UDS, is thought for customers
who need to driver SWUpdate from their application, but they won't ever
publish the application that remains closed source. Surely, it is not an
IPC limitation to limit contributions.

Again, I need a real example of a contribution that could be sent to
this project and due to some limitation it can't.

> In the current state of the art, a contribution requires `proc` and
> compiler-based knowledge...
> People develop coupled code and can't share it.
>
>
> I just want to make IPC endpoints user-friendly, flexible and allow
> everyone to implement their own send_update and event monitoring.

Best regards,

Stefano Babic

>> <https://groups.google.com/d/msgid/swupdate/YQBPR0101MB92308776C7CCFB0ABDB8C2B3A3169%40YQBPR0101MB9230.CANPRD01.PROD.OUTLOOK.COM?utm_medium=email&utm_source=footer

> <https://groups.google.com/d/msgid/swupdate/YQBPR0101MB92308776C7CCFB0ABDB8C2B3A3169%40YQBPR0101MB9230.CANPRD01.PROD.OUTLOOK.COM?utm_medium=email&utm_source=footer>>.
>
>
> --
> =====================================================================
> DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
> HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
> Phone: +49-8142-66989-53 Fax: +49-8142-66989-80 Email: sba...@denx.de
> =====================================================================
>

> --
> You received this message because you are subscribed to the Google
> Groups "swupdate" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to swupdate+u...@googlegroups.com
> <mailto:swupdate+u...@googlegroups.com>.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/swupdate/YQBPR0101MB9230DDFCA3295B7F08C9CA6AA3169%40YQBPR0101MB9230.CANPRD01.PROD.OUTLOOK.COM
> <https://groups.google.com/d/msgid/swupdate/YQBPR0101MB9230DDFCA3295B7F08C9CA6AA3169%40YQBPR0101MB9230.CANPRD01.PROD.OUTLOOK.COM?utm_medium=email&utm_source=footer>.

--
=====================================================================
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk

HRB 165235 Munich, Office: Kirchenstr.5, 82194 Groebenzell, Germany