C mender client for bare-metal devices
98 views
Skip to first unread message
Michael Zimmermann
Oct 25, 2018, 10:59:34 AM10/25/18
to Mender List mender.io, drew.m...@northern.tech
Hi everyone,
I'm currently working on a mender client written in portable C which can be used to do OTA-updates on resource-limited bare-metal targets like micro-controllers.
We do have plans to make this opensource as soon as it reaches a qualitative state.
I'm posting to this list this early so we can talk about the changes that'll be needed on the server side and agree together on the best solutions.
The main points are:
1) GZIP compression for the artifact header(header.tar.gz) and for the data files
2) cryptographic key algorithm used for signing auth requests
1: Compression
The Problem is that the default window size is 32768 bytes which is quite a lot for a micro-controller.
While it would be possible to reduce this down to 512 bytes there are a few reasons why I'd prefer to have the option to disable compression for the following reasons:
- a small window size results in a small compression ratio
- the firmware of MCUs is usually relatively small anyway
- it's somewhat pointless having to include the zlib or miniz library on such a device just to workaround the server not supporting uncompressed artifacts.
My proposed solution would be to use the file extension to identify the compression type of a file inside the artifact:
- GZIP: header.tar.gz and data/0000.tar.gz
- uncompressed: header.tar and data/0000.tar
We should probably also add some kind of validation to check if there's more than one header or data segment with the same id.
2: Signing
The auth requests are currently being signed and verified using an RSA public/private keypair.
I'd like to get some information about the reasoning behind this decision and both present and future use of these keys.
The way I currently understand it is that after flashing the firmware in the factory the device boots up, generates it's keypair and provides a way to extract the public key which then can be transferred to the mender server physically (usb flash drive) or remotely.
The key feature of RSA which allows generating and verifying certificates using a CA doesn't seem to be used right now.
The problem is that generating an RSA keypair takes quite some time on a MCU which is not desirable when considering time-based manufacturing costs.
While the keypair could theoretically be generated on another device and then provisioned to the device, this process would be less secure.
If you don't plan to use any RSA-specific feature of these keys in future, my proposed solution would be to add support for alternative signature algorithms like ed25519. The type of algorithm to use could be selected by an additional HTTP header field.
I'm looking forward to your feedback to these points.
Some key features of the design of our code:
- no dynamic memory allocations(this also means that the server has to take into account the capabilities of the client, e.g. increasing the JWT size with an update could end up in clients being unable to communicate)
- abstractions for system includes and device/platform specific implementations(e.g. keystore, logging, etc)
- tests using cmocka
Eystein Måløy Stenberg
Oct 25, 2018, 5:24:31 PM10/25/18
to men...@lists.mender.io, Michael Zimmermann, drew.m...@northern.tech
Hi Michael,
That's really exciting! Please see below for some responses, and likely
some of the Mender developers can answer the more detailed questions.
This makes sense, note there is also some interest & research from the
community to support a more efficient compression algorithm, e.g. xz:
https://tracker.mender.io/browse/MEN-761
I think it would be difficult to support the Artifact format (for any
given client) if there are many different ways a file could be
compressed. However, having gz (or replace it with xz) as well as
uncompressed should work.
It sounds like this would require a new version of the Artifact format,
though? Support for v3 is currently being implemented:
https://github.com/mendersoftware/mender-artifact/blob/master/Documentation/artifact-format-v3.md
>
> We should probably also add some kind of validation to check if there's
> more than one header or data segment with the same id.
>
>
> 2: Signing
> The auth requests are currently being signed and verified using an RSA
> public/private keypair.
> I'd like to get some information about the reasoning behind this
> decision and both present and future use of these keys.
> The way I currently understand it is that after flashing the firmware in
> the factory the device boots up, generates it's keypair and provides a
> way to extract the public key which then can be transferred to the
> mender server physically (usb flash drive) or remotely.
> The key feature of RSA which allows generating and verifying
> certificates using a CA doesn't seem to be used right now.
Not quite..
There are two ways to do this, which are described here:
https://docs.mender.io/1.6/architecture/device-authentication
Keys are usually not generated on the target device in production
environments, you would mass-generate keys, place the private key on the
device storage and use the preauthorize feature of Mender to trust the
public key in the server.
The reason the client signs the auth request at all is to prove to the
server it possesses the private key corresponding to the public key the
server believes the client has.
> The problem is that generating an RSA keypair takes quite some time on a
> MCU which is not desirable when considering time-based manufacturing costs.
> While the keypair could theoretically be generated on another device and
> then provisioned to the device, this process would be less secure.
The expectation is that you already have a secure channel established
when placing the private key on the device, such as physical access
during manufacturing/provisioning of the storage.
If you have a device that just "shows up" without any trust set up
securely in advance, then nothing can be assured about the device
(anyone can generate a RSA key).
> If you don't plan to use any RSA-specific feature of these keys in
> future, my proposed solution would be to add support for alternative
> signature algorithms like ed25519. The type of algorithm to use could be
> selected by an additional HTTP header field.
Makes sense, would love to see support for this both for signing auth
requests and artifacts. The reason ed25519 is not yet implemented is
that the tooling and libraries mostly lack support. When this was
researched earlier this year, openssl even did not support it properly
(it had some internal code, but not possible to use it):
https://tracker.mender.io/browse/MEN-1742 (note this is for signing
artifacts, but same algorithm).
Have you tested ed25519? Which libraries/tools did you use? Is it
portable to Golang (for aritfact library, server)?
>
> I'm looking forward to your feedback to these points.
>
> Some key features of the design of our code:
> - no dynamic memory allocations(this also means that the server has to
> take into account the capabilities of the client, e.g. increasing the
> JWT size with an update could end up in clients being unable to communicate)
> - abstractions for system includes and device/platform specific
> implementations(e.g. keystore, logging, etc)
> - tests using cmocka
>
> IOTΛ Data Marketplace Member· MS Azure IoT Gold Partner · Apple MFi
> Developer · Bluetooth SIG · zigbee Alliance · LoRa Alliance · Thread Group
>
> grandcentrix GmbH · Holzmarkt 1 · 50676 Köln · Deutschland
> | t <https://twitter.com/grandcentrix> | f
> <https://www.facebook.com/GrandCentrix/> | in
> <https://www.linkedin.com/company/grandcentrix> | phone:
> +49-221-677-860-0 | email: he...@grandcentrix.net
> <mailto:he...@grandcentrix.net>
>
> Amtsgericht Köln | HRB 70119 | Geschäftsführer: R. Rottmann, M. Willnow
> | USt.-IdNr.: DE266333969
>
> --
> You received this message because you are subscribed to the Google
> Groups "Mender List mender.io" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to mender+un...@lists.mender.io
> <mailto:mender+un...@lists.mender.io>.
> To post to this group, send email to men...@lists.mender.io
> <mailto:men...@lists.mender.io>.
> Visit this group at
> https://groups.google.com/a/lists.mender.io/group/mender/.
--
Eystein
That's really exciting! Please see below for some responses, and likely
some of the Mender developers can answer the more detailed questions.
community to support a more efficient compression algorithm, e.g. xz:
https://tracker.mender.io/browse/MEN-761
I think it would be difficult to support the Artifact format (for any
given client) if there are many different ways a file could be
compressed. However, having gz (or replace it with xz) as well as
uncompressed should work.
It sounds like this would require a new version of the Artifact format,
though? Support for v3 is currently being implemented:
https://github.com/mendersoftware/mender-artifact/blob/master/Documentation/artifact-format-v3.md
>
> We should probably also add some kind of validation to check if there's
> more than one header or data segment with the same id.
>
>
> 2: Signing
> The auth requests are currently being signed and verified using an RSA
> public/private keypair.
> I'd like to get some information about the reasoning behind this
> decision and both present and future use of these keys.
> The way I currently understand it is that after flashing the firmware in
> the factory the device boots up, generates it's keypair and provides a
> way to extract the public key which then can be transferred to the
> mender server physically (usb flash drive) or remotely.
> The key feature of RSA which allows generating and verifying
> certificates using a CA doesn't seem to be used right now.
There are two ways to do this, which are described here:
https://docs.mender.io/1.6/architecture/device-authentication
Keys are usually not generated on the target device in production
environments, you would mass-generate keys, place the private key on the
device storage and use the preauthorize feature of Mender to trust the
public key in the server.
The reason the client signs the auth request at all is to prove to the
server it possesses the private key corresponding to the public key the
server believes the client has.
> The problem is that generating an RSA keypair takes quite some time on a
> MCU which is not desirable when considering time-based manufacturing costs.
> While the keypair could theoretically be generated on another device and
> then provisioned to the device, this process would be less secure.
when placing the private key on the device, such as physical access
during manufacturing/provisioning of the storage.
If you have a device that just "shows up" without any trust set up
securely in advance, then nothing can be assured about the device
(anyone can generate a RSA key).
> If you don't plan to use any RSA-specific feature of these keys in
> future, my proposed solution would be to add support for alternative
> signature algorithms like ed25519. The type of algorithm to use could be
> selected by an additional HTTP header field.
requests and artifacts. The reason ed25519 is not yet implemented is
that the tooling and libraries mostly lack support. When this was
researched earlier this year, openssl even did not support it properly
(it had some internal code, but not possible to use it):
https://tracker.mender.io/browse/MEN-1742 (note this is for signing
artifacts, but same algorithm).
Have you tested ed25519? Which libraries/tools did you use? Is it
portable to Golang (for aritfact library, server)?
>
> I'm looking forward to your feedback to these points.
>
> Some key features of the design of our code:
> - no dynamic memory allocations(this also means that the server has to
> take into account the capabilities of the client, e.g. increasing the
> JWT size with an update could end up in clients being unable to communicate)
> - abstractions for system includes and device/platform specific
> implementations(e.g. keystore, logging, etc)
> - tests using cmocka
>
> Developer · Bluetooth SIG · zigbee Alliance · LoRa Alliance · Thread Group
>
> grandcentrix GmbH · Holzmarkt 1 · 50676 Köln · Deutschland
> | t <https://twitter.com/grandcentrix> | f
> <https://www.facebook.com/GrandCentrix/> | in
> <https://www.linkedin.com/company/grandcentrix> | phone:
> +49-221-677-860-0 | email: he...@grandcentrix.net
> <mailto:he...@grandcentrix.net>
>
> Amtsgericht Köln | HRB 70119 | Geschäftsführer: R. Rottmann, M. Willnow
> | USt.-IdNr.: DE266333969
>
> --
> You received this message because you are subscribed to the Google
> Groups "Mender List mender.io" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to mender+un...@lists.mender.io
> <mailto:mender+un...@lists.mender.io>.
> To post to this group, send email to men...@lists.mender.io
> <mailto:men...@lists.mender.io>.
> Visit this group at
> https://groups.google.com/a/lists.mender.io/group/mender/.
--
Eystein
Michael Zimmermann
Oct 26, 2018, 5:33:50 AM10/26/18
to eystein.mal...@northern.tech, men...@lists.mender.io, drew.m...@northern.tech
This makes sense, note there is also some interest & research from the
community to support a more efficient compression algorithm, e.g. xz:
https://tracker.mender.io/browse/MEN-761
Yes I saw that. I think it makes sense to consider xz an other future compression algorithms when designing the code.
I think it would be difficult to support the Artifact format (for any
given client) if there are many different ways a file could be
compressed. However, having gz (or replace it with xz) as well as
uncompressed should work.
Actually, only the server has to support all formats. The client will only receive the artifact as it was uploaded to the server.
The person who builds the artifact has to take into account the capabilities of the mender client on the target device.
It sounds like this would require a new version of the Artifact format,
though? Support for v3 is currently being implemented:
https://github.com/mendersoftware/mender-artifact/blob/master/Documentation/artifact-format-v3.md
Actually I think it doesn't because:
> All file tree components ending in .gz in the tree displayed above should be compressed, and the suffix corresponds to the compression > method. Exact compression method to be decided.
To me this sounds like the file extension was always supposed to be an indicator for the compression method and that it's just the current server/client implementations which limits this to gzip.
Not quite..
There are two ways to do this, which are described here:
https://docs.mender.io/1.6/architecture/device-authentication
Keys are usually not generated on the target device in production
environments, you would mass-generate keys, place the private key on the
device storage and use the preauthorize feature of Mender to trust the
public key in the server.
The reason the client signs the auth request at all is to prove to the
server it possesses the private key corresponding to the public key the
server believes the client has.
My intend was to generate the key on a device which already has full-disk-encryption enabled to cover the case where you don't fully trust the manufacturer. But after thinking about it further, generating the keys on the device doesn't make this any more secure since the manufacturer is the one who enables the encryption in first place and would be able to read the mender-key anyway.
That's why I'd actually stay with RSA for now. It would still make sense to implement ed25519 however, because the key size is much smaller which would be good for devices which are more constrained than what I'm currently working with.
Makes sense, would love to see support for this both for signing auth
requests and artifacts. The reason ed25519 is not yet implemented is
that the tooling and libraries mostly lack support. When this was
researched earlier this year, openssl even did not support it properly
(it had some internal code, but not possible to use it):
https://tracker.mender.io/browse/MEN-1742 (note this is for signing
artifacts, but same algorithm).
Have you tested ed25519? Which libraries/tools did you use? Is it
portable to Golang (for aritfact library, server)?
Openssl does fully support ed25519 even though it's not listed in 'openssl ecparam -list_curves'. You have to use the EVP_Digest* API using an EVP_PKEY_ED25519 key.
For the C client you'd probably rather use libsodium though.
Thanks
Michael
IOTΛ Data Marketplace Member · MS Azure IoT Gold Partner · Apple MFi Developer · Bluetooth SIG · zigbee Alliance · LoRa Alliance · Thread Group
grandcentrix GmbH · Holzmarkt 1 · 50676 Köln · Deutschland
Kristian Amlie
Oct 26, 2018, 7:51:01 AM10/26/18
to men...@lists.mender.io, Michael Zimmermann, eystein.mal...@northern.tech, drew.m...@northern.tech
On 26/10/18 11:33, Michael Zimmermann wrote:
> I think it would be difficult to support the Artifact format (for any
> given client) if there are many different ways a file could be
> compressed. However, having gz (or replace it with xz) as well as
> uncompressed should work.
>
> Actually, only the server has to support all formats. The client will
> only receive the artifact as it was uploaded to the server.
> The person who builds the artifact has to take into account the
> capabilities of the mender client on the target device.
>
>
> It sounds like this would require a new version of the Artifact format,
> though? Support for v3 is currently being implemented:
> https://github.com/mendersoftware/mender-artifact/blob/master/Documentation/artifact-format-v3.md
>
> Actually I think it doesn't because:
>> All file tree components ending in .gz in the tree displayed above
> should be compressed, and the suffix corresponds to the compression >
> method. Exact compression method to be decided.
> To me this sounds like the file extension was always supposed to be an
> indicator for the compression method and that it's just the current
> server/client implementations which limits this to gzip.
That's right. Since it's not implemented right now, it would be a new
> I think it would be difficult to support the Artifact format (for any
> given client) if there are many different ways a file could be
> compressed. However, having gz (or replace it with xz) as well as
> uncompressed should work.
>
> Actually, only the server has to support all formats. The client will
> only receive the artifact as it was uploaded to the server.
> The person who builds the artifact has to take into account the
> capabilities of the mender client on the target device.
>
>
> It sounds like this would require a new version of the Artifact format,
> though? Support for v3 is currently being implemented:
> https://github.com/mendersoftware/mender-artifact/blob/master/Documentation/artifact-format-v3.md
>
> Actually I think it doesn't because:
>> All file tree components ending in .gz in the tree displayed above
> should be compressed, and the suffix corresponds to the compression >
> method. Exact compression method to be decided.
> To me this sounds like the file extension was always supposed to be an
> indicator for the compression method and that it's just the current
> server/client implementations which limits this to gzip.
feature in the format, either v2 or v3, which would be backwards, but
not forwards compatible, meaning that: An old artifact would be accepted
by a server that supports the new feature, but an artifact with the new
feature would not be accepted by an old server.
--
Kristian
Eystein Måløy Stenberg
Oct 26, 2018, 7:47:46 PM10/26/18
to Kristian Amlie, men...@lists.mender.io, Michael Zimmermann, drew.m...@northern.tech
https://tracker.mender.io/browse/MEN-2224.
It seems like fairly small effort and we would be very happy to assist
in guiding this, Micahel, if you want to take a stab at it?
Is this the only blocker left for you?
--
Eystein
Dan Walkes
Oct 27, 2018, 12:57:25 PM10/27/18
to men...@lists.mender.io, kristia...@northern.tech, michael.z...@grandcentrix.net, Drew Moseley
Hi Michael,
Thanks for sharing this info.
I also have a project related to device update for embedded devices. See http://securedeviceupdate.com. I haven't shared the source yet, mostly because I haven't found anyone interested in helping to develop it or add additional platforms and I need to do a bit more work before it's ready to use for someone else, especially as an Arduino build target. I'd like to share it if it would be useful in the context of a project like yours and especially if it meant Mender integration would be possible.
Today I'm using it as a way to update Bluetooth BLE devices behind an iPhone or Android device Bluetooth BLE connection. The BLE update protocol is implemented in a way which is compatible with Nordic Semiconductor's update scheme and therefore allows you to use their iPhone and Android libraries for the BLE transport from a mobile ap. If/when anyone is interested in deploying this project in combination with Mender integration I'd like to add capabilities to the mobile libraries which can relay artifact and update status, to/from the Mender server. This would mean a device with only BLE connections to a mobile platform and not directly connected to the WWW would still be usable with mender server and associated device managment. So in this case I was picturing the device itself would not necessarily be mender-aware but instead the mobile device being used as the relay agent to the WWW would be.
This may be different enough from what you are planning to do that it won't make sense to partner or use any of the code I have now, but I thought I would mention it in case you'd like to discuss, and also to make the general Mender community aware of it.
Thanks
Dan
--
You received this message because you are subscribed to the Google Groups "Mender List mender.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mender+un...@lists.mender.io.
To post to this group, send email to men...@lists.mender.io.
Eystein Måløy Stenberg
Feb 4, 2019, 6:41:31 PM2/4/19
to men...@lists.mender.io, Michael Zimmermann, drew.m...@northern.tech
Hi Michael,
As you probably are aware, much thanks to your help, uncompressed
Artifact payloads is now supported in master & next Mender release
(https://tracker.mender.io/browse/MEN-2224).
I am just curious if there are more hard obstacles for your MCU client?
Curious to see it, please keep us posted! :-)
--
Eystein
As you probably are aware, much thanks to your help, uncompressed
Artifact payloads is now supported in master & next Mender release
(https://tracker.mender.io/browse/MEN-2224).
I am just curious if there are more hard obstacles for your MCU client?
Curious to see it, please keep us posted! :-)
> IOTΛ Data Marketplace Member· MS Azure IoT Gold Partner · Apple MFi
> Developer · Bluetooth SIG · zigbee Alliance · LoRa Alliance · Thread Group
>
> grandcentrix GmbH · Holzmarkt 1 · 50676 Köln · Deutschland
> Developer · Bluetooth SIG · zigbee Alliance · LoRa Alliance · Thread Group
>
> grandcentrix GmbH · Holzmarkt 1 · 50676 Köln · Deutschland
> | t <https://twitter.com/grandcentrix> | f
> <https://www.facebook.com/GrandCentrix/> | in
> <https://www.linkedin.com/company/grandcentrix> | phone:
> +49-221-677-860-0 | email: he...@grandcentrix.net
> <mailto:he...@grandcentrix.net>
> <https://www.facebook.com/GrandCentrix/> | in
> <https://www.linkedin.com/company/grandcentrix> | phone:
> +49-221-677-860-0 | email: he...@grandcentrix.net
>
> Amtsgericht Köln | HRB 70119 | Geschäftsführer: R. Rottmann, M. Willnow
> | USt.-IdNr.: DE266333969
>
> Amtsgericht Köln | HRB 70119 | Geschäftsführer: R. Rottmann, M. Willnow
> | USt.-IdNr.: DE266333969
>
> --
> You received this message because you are subscribed to the Google
> Groups "Mender List mender.io" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to mender+un...@lists.mender.io
> <mailto:mender+un...@lists.mender.io>.
> You received this message because you are subscribed to the Google
> Groups "Mender List mender.io" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to mender+un...@lists.mender.io
> To post to this group, send email to men...@lists.mender.io
> <mailto:men...@lists.mender.io>.
--
Eystein
Michael Zimmermann
Feb 6, 2019, 1:46:21 AM2/6/19
to Eystein Måløy Stenberg, men...@lists.mender.io, drew.m...@northern.tech
Hi Eystein,
thanks to you and Ole for implementing and merging these final pieces.
Since we're currently actively working on finalizing the mender client code I'll try to get a timeline about the publication of our code.
I'll definitely keep you updated.
Thanks
Michael Zimmermann
Reply all
Reply to author
Forward
0 new messages