Delta updates - authentication to hawkbit for chunks

106 views

Skip to first unread message

Kerkmann Manuel

unread,

Jul 30, 2024, 9:04:52 AM7/30/24

to swup...@googlegroups.com, Fangmann Alexander

Hello,

we are trying to implement delta updates, but don’t get it working with downloading artifacts via DDI from our hawkbit.

Problem seems to be the authentication with TargetToken. But I am not really sure, because we cant see much details in logs.

I guess we set wrong configuration in sw-description, maybe you can help us.

We only get:

[…]

[INFO ] : SWUPDATE running : [get_total_size] : Total bytes to be reused : 58355970

[INFO ] : SWUPDATE running : [get_total_size] : Total bytes to be downloaded : 116080

[INFO ] : SWUPDATE running : [install_delta] : Size of artifact to be installed : 59739136

[TRACE] : SWUPDATE running : [trigger_download] : Range request : 60722-62209,63368-65109,65792-73947,106330-134956,433953-464759,1734636-1745818,6149454-6163020,8131522-8135525,11793660-11809534,11810965-11811268,11814063-11814367,11814522-11814543

[TRACE] : SWUPDATE running : [install_single_image] : Found installer for stream somefile.zck.header raw

< HTTP/1.1 200 OK

< Content-Length: 0

< Date: Tue, 30 Jul 2024 12:30:55 GMT

< Cache-Control: no-cache, no-store, max-age=0, must-revalidate

< Expires: 0

< Pragma: no-cache

< Set-Cookie: JSESSIONID=someid; Path=/; Secure; HttpOnly

< Vary: Origin

< Vary: Access-Control-Request-Method

< Vary: Access-Control-Request-Headers

< Strict-Transport-Security: max-age=31536000 ; includeSubDomains

< X-Content-Type-Options: nosniff

< X-XSS-Protection: 0

< X-Frame-Options: DENY

* Connection #0 to host someurl.net left intact

[TRACE] : SWUPDATE running : [channel_log_effective_url] : Channel's effective URL resolved to https://someurl.net:443/sometenant/controller/v1/somecontroller/deploymentBase/someid/feedback

[TRACE] : SWUPDATE running : [channel_log_reply] : Channel operation returned HTTP status code 200.

[TRACE] : SWUPDATE running : [parse_reply] : Got channel reply:

[TRACE] : SWUPDATE running : [channel_log_effective_url] : Channel's effective URL resolved to https://someurl/sometenant/controller/v1/somecontroller/softwaremodules/someid/artifacts/somefile.zck

[DEBUG] : SWUPDATE running : [channel_get_file] : Channel downloaded 0 bytes ~ 0 MiB.

[ERROR] : SWUPDATE failed [0] ERROR corelib/channel_curl.c : channel_log_reply : 872 : Channel operation returned HTTP error code 401.

[ERROR] : SWUPDATE failed [0] ERROR handlers/delta_handler.c : read_and_validate_package : 584 : Transfer was unsuccessful, aborting...

[ERROR] : SWUPDATE failed [0] ERROR handlers/delta_handler.c : install_delta : 1041 : Delta Update fails : aborting

[TRACE] : SWUPDATE running : [zck_log_toswupdate] : (comp_close) Closing compression

[TRACE] : SWUPDATE running : [install_single_image] : Installer for delta not successful !

[ERROR] : SWUPDATE failed [1] Installation failed !

[TRACE] : SWUPDATE running : [network_initializer] : Main thread sleep again !

[INFO ] : No SWUPDATE running : Waiting for requests...

[TRACE] : SWUPDATE running : [process_notification_thread] : Update log to server from thread

Our sw-description file looks like this:

software =

{

version = "2.1.0";

description = "v2.1.0";

images: (

{

filename = "somefile.zck.header";

name = "custom";

version = "2.01.0";

device = "/dev/swu/update/custom";

type = "delta";

properties: {

url = https://someurl/sometenant/controller/v1/somecontroller/softwaremodules/someid/artifacts/somefile.zck;

headers = {

“Authorization” = "TargetToken sometoken";

};

chain = "raw";

source = "/dev/swu/stable/custom";

};

sha256 = "somehash";

);

[…]

}

Mit freundlichen Grüßen

With best regards

i. A. Manuel Kerkmann

KS-E Team Smart Farming

AMAZONEN-WERKE H. DREYER SE & Co. KG

Am Amazonenwerk 9-13

49205 Hasbergen-Gaste

Mail: Manuel....@amazone.de

Mobile: +49 160 979 302 89

https://www.amazone.de

############################################################################
Kommanditgesellschaft, Sitz: Hasbergen, Registergericht: AG Osnabrück HRA 2716,
persönlich haftende Gesellschafterin: HD International SE, Europäische Aktiengesellschaft,
Sitz: Hasbergen, Registergericht: AG Osnabrück HRB 215644, Verwaltungsrat: Christian Dreyer und Dr. Justus Dreyer,
Geschäftsführende Direktoren: Ludger Braunsmann, Dr. Stephan Evers, Andreas Hemeyer, Dr. Rainer Resch
############################################################################

Stefano Babic

unread,

Jul 30, 2024, 9:22:19 AM7/30/24

to Kerkmann Manuel, swup...@googlegroups.com, Fangmann Alexander

Hi Manuel,

On 30.07.24 14:51, Kerkmann Manuel wrote:
> Hello,
>
> we are trying to implement delta updates, but don’t get it working with
> downloading artifacts via DDI from our hawkbit.
>
> Problem seems to be the authentication with TargetToken. But I am not
> really sure, because we cant see much details in logs.
>
> I guess we set wrong configuration in sw-description, maybe you can help us.
>
> We only get:
>
> […]
>
> [INFO ] : SWUPDATE running : [get_total_size] : Total bytes to be
> reused : 58355970
>

If you are coming here, it means that the SWU was downloaded from the
Hawkbit server. SWUpdate has already found what should be downloaded and
what should be copied.

> [INFO ] : SWUPDATE running : [get_total_size] : Total bytes to be
> downloaded : 116080
>
> [INFO ] : SWUPDATE running : [install_delta] : Size of artifact to be
> installed : 59739136
>
> [TRACE] : SWUPDATE running : [trigger_download] : Range request :
> 60722-62209,63368-65109,65792-73947,106330-134956,433953-464759,1734636-1745818,6149454-6163020,8131522-8135525,11793660-11809534,11810965-11811268,11814063-11814367,11814522-11814543

>
> [TRACE] : SWUPDATE running : [install_single_image] : Found installer
> for stream somefile.zck.header raw
>
> < HTTP/1.1 200 OK

It is difficult to sort out, but if this is the answer to SWUpdate's
range request is wrong. But I am not sure about this. If it is related
to the download of SWU, it is correct.

If you are using Amazon's S3 bucket, there was a post recently telling
that Amazon supports range request, but with just one entry, making
unsuitable for delta update.

See:

https://groups.google.com/g/swupdate/c/iKo6h6muM0A/m/UFZ5jK_yAQAJ

>
> < Content-Length: 0
>
> < Date: Tue, 30 Jul 2024 12:30:55 GMT
>
> < Cache-Control: no-cache, no-store, max-age=0, must-revalidate
>
> < Expires: 0
>
> < Pragma: no-cache
>
> < Set-Cookie: JSESSIONID=someid; Path=/; Secure; HttpOnly
>
> < Vary: Origin
>
> < Vary: Access-Control-Request-Method
>
> < Vary: Access-Control-Request-Headers
>
> < Strict-Transport-Security: max-age=31536000 ; includeSubDomains
>
> < X-Content-Type-Options: nosniff
>
> < X-XSS-Protection: 0
>
> < X-Frame-Options: DENY
>
> <
>
> * Connection #0 to host someurl.net left intact
>
> [TRACE] : SWUPDATE running : [channel_log_effective_url] : Channel's
> effective URL resolved to
> https://someurl.net:443/sometenant/controller/v1/somecontroller/deploymentBase/someid/feedback
>
> [TRACE] : SWUPDATE running : [channel_log_reply] : Channel operation
> returned HTTP status code 200.
>
> [TRACE] : SWUPDATE running : [parse_reply] : Got channel reply:
>
> [TRACE] : SWUPDATE running : [channel_log_effective_url] : Channel's
> effective URL resolved to
> https://someurl/sometenant/controller/v1/somecontroller/softwaremodules/someid/artifacts/somefile.zck
>

How do you sync this with sw-description ? The id is added by Hawkbit
when a SW module is uploaded. To have in sync, you have to upload first
to Hawkbit, track the id, add it to sw-description and build.

It could be possible to let SWUpdate to find itself the URL because
Hawkbit delivers a list of Software Modules, but this is currently
unsupported.

Nevertheless, the URL in the handler is a generic URL and the .zck can
resides on any server on the network. This means that SWUpdate does not
think this is on the Hawkbit server, and does not send any additional
header. It is just an URL, and if you put it on Hawkbit using
authentication, a 401 is expected. Reverse Proxy via certificate should
work, because the downloader can use the same certificates used to
connect to Hawkbit.

Name / custom are used by SWUpdate only together with the flags
installed-if-different or installed-if-greater, else it is ignored.

>
>                      device = "/dev/swu/update/custom";
>
>                      type = "delta";
>
>                      properties: {
>
>                             url =
> https://someurl/sometenant/controller/v1/somecontroller/softwaremodules/someid/artifacts/somefile.zck;
>
>                             headers = {
>
> “Authorization” = "TargetToken sometoken";
>
>                                };

This is your invention, the delta handler does not support this property.

>
>                             chain = "raw";
>
>                             source = "/dev/swu/stable/custom";
>
> };
>
>                      sha256 = "somehash";
>
>                   },
>
>       );
>
>       […]
>
> }
>
> Mit freundlichen Grüßen

Viele Grüße,
Stefano Babic

Manuel Kerkmann

unread,

Jul 31, 2024, 1:49:30 AM7/31/24

to swupdate

Good morning Stefano,

thanks a lot for answer.

It should be answer to swu download.

If you are using Amazon's S3 bucket, there was a post recently telling
that Amazon supports range request, but with just one entry, making
unsuitable for delta update.

See:

https://groups.google.com/g/swupdate/c/iKo6h6muM0A/m/UFZ5jK_yAQAJ

we are using azure, but i will keep this in mind and check, thanks.

We first uploaded the .zck file to hawkbit to new software module, to get software module id and to build the url we need to download the artifact.

Then we (re-)wrote the sw-description and added necessary entries for delta handler.

Lastly, we built a new .swu file with this sw-description and .zck.header, uploaded it to the same software module, assigned to a distribution set and made some rollout for this.

It could be possible to let SWUpdate to find itself the URL because
Hawkbit delivers a list of Software Modules, but this is currently
unsupported.

Nevertheless, the URL in the handler is a generic URL and the .zck can
resides on any server on the network. This means that SWUpdate does not
think this is on the Hawkbit server, and does not send any additional
header. It is just an URL, and if you put it on Hawkbit using
authentication, a 401 is expected. Reverse Proxy via certificate should
work, because the downloader can use the same certificates used to
connect to Hawkbit.

okay, understood. Is there any plan or chance to adapt the behavior, to set some additional headers with the request?
For us, it seems to be an obvious way to place the .zck file on same hawkbit server and also reuse the authentication.

okay, this part is reused from the hardware manufacturers code, I will rework it.

>
>                      device = "/dev/swu/update/custom";
>
>                      type = "delta";
>
>                      properties: {
>
>                             url =
> https://someurl/sometenant/controller/v1/somecontroller/softwaremodules/someid/artifacts/somefile.zck;
>
>                             headers = {
>
> “Authorization” = "TargetToken sometoken";
>
>                                };

This is your invention, the delta handler does not support this property.

yes it was just a try.

Stefano Babic

unread,

Jul 31, 2024, 3:56:54 AM7/31/24

to Manuel Kerkmann, swupdate

Hi Manuel,

I reread the log and I agree.

>
>
>
> If you are using Amazon's S3 bucket, there was a post recently telling
> that Amazon supports range request, but with just one entry, making
> unsuitable for delta update.
>
> See:
>
> https://groups.google.com/g/swupdate/c/iKo6h6muM0A/m/UFZ5jK_yAQAJ
> <https://groups.google.com/g/swupdate/c/iKo6h6muM0A/m/UFZ5jK_yAQAJ>
>
>
> we are using azure, but i will keep this in mind and check, thanks.

I have misread the name of your company...:-)

>
>
>
> >
> > < Content-Length: 0
> >
> > < Date: Tue, 30 Jul 2024 12:30:55 GMT
> >
> > < Cache-Control: no-cache, no-store, max-age=0, must-revalidate
> >
> > < Expires: 0
> >
> > < Pragma: no-cache
> >
> > < Set-Cookie: JSESSIONID=someid; Path=/; Secure; HttpOnly
> >
> > < Vary: Origin
> >
> > < Vary: Access-Control-Request-Method
> >
> > < Vary: Access-Control-Request-Headers
> >
> > < Strict-Transport-Security: max-age=31536000 ; includeSubDomains
> >
> > < X-Content-Type-Options: nosniff
> >
> > < X-XSS-Protection: 0
> >
> > < X-Frame-Options: DENY
> >
> > <
> >

> > * Connection #0 to host someurl.net <http://someurl.net> left intact

> >
> > [TRACE] : SWUPDATE running : [channel_log_effective_url] :
> Channel's
> > effective URL resolved to
> >

> https://someurl.net:443/sometenant/controller/v1/somecontroller/deploymentBase/someid/feedback <https://someurl.net:443/sometenant/controller/v1/somecontroller/deploymentBase/someid/feedback>

> >
> > [TRACE] : SWUPDATE running : [channel_log_reply] : Channel
> operation
> > returned HTTP status code 200.
> >
> > [TRACE] : SWUPDATE running : [parse_reply] : Got channel reply:
> >
> > [TRACE] : SWUPDATE running : [channel_log_effective_url] :
> Channel's
> > effective URL resolved to
> >

> https://someurl/sometenant/controller/v1/somecontroller/softwaremodules/someid/artifacts/somefile.zck <https://someurl/sometenant/controller/v1/somecontroller/softwaremodules/someid/artifacts/somefile.zck>

> >
>
> How do you sync this with sw-description ? The id is added by Hawkbit
> when a SW module is uploaded. To have in sync, you have to upload first
> to Hawkbit, track the id, add it to sw-description and build.
>
>
> We first uploaded the .zck file to hawkbit to new software module, to
> get software module id and to build the url we need to download the
> artifact.
> Then we (re-)wrote the sw-description and added necessary entries for
> delta handler.
> Lastly, we built a new .swu file with this sw-description and
> .zck.header, uploaded it to the same software module, assigned to a
> distribution set and made some rollout for this.

Yes, and I think we could agree that this is completely misleading.
Build and deployment are orthogonal, and they are often managed by
different teams. The development delivers its results, for delta both
SWU and .zck(s). The method above is just a trick due to the missing
support inside SWUpdate.

>
>
>
> It could be possible to let SWUpdate to find itself the URL because
> Hawkbit delivers a list of Software Modules, but this is currently
> unsupported.
>
> Nevertheless, the URL in the handler is a generic URL and the .zck can
> resides on any server on the network. This means that SWUpdate does not
> think this is on the Hawkbit server, and does not send any additional
> header. It is just an URL, and if you put it on Hawkbit using
> authentication, a 401 is expected. Reverse Proxy via certificate should
> work, because the downloader can use the same certificates used to
> connect to Hawkbit.
>
>
> okay, understood. Is there any plan or chance to adapt the behavior, to
> set some additional headers with the request?
> For us, it seems to be an obvious way to place the .zck file on same
> hawkbit server and also reuse the authentication.

Glad to hear this, and I fully agree. I can add this to
doc/source/improvement_proposals.rst. It is just not the headers,
because the downloader for the .zck is split from the backend daemon
(suricatta) talking with Hawkbit, so some more stuff should be done. And
to make a delta upgrade full compatible with Hawkbit, I think that a way
to get the URL from the server should be implemented to avoid the tricks
you described above.

It would be nice if your company will finance this work and I could
implement it in the project.

Best regards,
Stefano

> >
> >                             headers = {
> >
> > “Authorization” = "TargetToken sometoken";
> >
> >                                };
>
> This is your invention, the delta handler does not support this
> property.
>
>
> yes it was just a try.
>
>
>
> >
> >                             chain = "raw";
> >
> >                             source = "/dev/swu/stable/custom";
> >
> > };
> >
> >                      sha256 = "somehash";
> >
> >                   },
> >
> >       );
> >
> >       […]
> >
> > }
> >
> > Mit freundlichen Grüßen
>
> Viele Grüße,
> Stefano Babic
>

> --
> You received this message because you are subscribed to the Google
> Groups "swupdate" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to swupdate+u...@googlegroups.com
> <mailto:swupdate+u...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/swupdate/027b84b4-e322-4462-832c-024f1045b881n%40googlegroups.com <https://groups.google.com/d/msgid/swupdate/027b84b4-e322-4462-832c-024f1045b881n%40googlegroups.com?utm_medium=email&utm_source=footer>.

Reply all

Reply to author

Forward

0 new messages