Delta upgrade with AWS S3 bucket website

[Raghavendra Adiga]

Hi Stefano,

Previously we had success of delta update with images hosted in Apache and NGINX servers. But while trying delta upgrade with AWS S3 bucket that hosts both swu and zck images, and while the swupdate tries to pull the chunks I see failure with below logs.

[ERROR] : SWUPDATE failed [0] ERROR delta_downloader.c : wrdata_callback : 82 : Bytes request not supported by server, returning 200

[ERROR] : SWUPDATE failed [0] ERROR channel_curl.c : channel_get_file : 1432 : Channel operation returned error (23): 'Failed writing received

For a moment I thought the S3 bucket doesn’t support range requests but when I tried using test program which requests ranges through curl_easy_setopt(curl, CURLOPT_RANGE, range); It is working fine.

   

I bypassed the code in delta_downloader.c that returns when response code is not 206, to check if it continues:

    //if (channel_data->http_response_code != 206) {

    //  ERROR("Bytes request not supported by server, returning %ld",

    //      channel_data->http_response_code);

    //  return 0;

    //}

But that leads to error in some other place : “Malformed body, no boundary found”

In fact for the full SWU downloadfrom S3 bucket SWU URL, I think fetching the file size as well  requires range request, so I get an error Failed to get total download size for URL. In this case the download proceeds, but may be because there is no size information beforehand, swupdate fails to print the download progress percentage.

Does this give some insights on what could be wrong or missing in configuration?

Thanks and regards,

Raghavendra Adiga. 

[Stefano Babic]

Hi Raghavendra,

On 16.07.24 10:32, Raghavendra Adiga wrote:
> Hi Stefano,
>
>
> Previously we had success of delta update with images hosted in Apache
> and NGINX servers. But while trying delta upgrade with AWS S3 bucket
> that hosts both swu and zck images, and while the swupdate tries to pull
> the chunks I see failure with below logs.
>

> *[ERROR] : SWUPDATE failed [0] ERROR delta_downloader.c :
> wrdata_callback : 82 : Bytes request not supported by server, returning 200*
>

This is a clear signal that SWUpdate's request was rejected by the server.

> *[ERROR] : SWUPDATE failed [0] ERROR channel_curl.c : channel_get_file :
> 1432 : Channel operation returned error (23): 'Failed writing received*
>
> **

>
> For a moment I thought the S3 bucket doesn’t support range requests but
> when I tried using test program which requests ranges

> through**curl_easy_setopt(curl, CURLOPT_RANGE, range); It is working fine.

Yes, but how many ?

Apache und Nginx have a default maximum value of 200, that can be
adjusted via configuration file. SWUpdate sets a default value under
this threshold to 150. Greater value, more efficient is the delta
because it runs in a single http(s) request.

The answer from S3 bucket let's think that the value is lower for
Amazon. In that case, the server answer with 200, asking to download the
whole file.

You can configure the value in sw-description as property ('max-ranges")
for the delta handler.

>
> **

>
> I bypassed the code in delta_downloader.c that returns when response
> code is not 206, to check if it continues:
>

> **//if (channel_data->http_response_code != 206) {

>
>     //  ERROR("Bytes request not supported by server, returning %ld",
>
>     //      channel_data->http_response_code);
>
>     //  return 0;
>
>     //}
>

> But that leads to error in some other place : “*Malformed body, no
> boundary found*”

You are working on Energy Systems, if you short cut power lines do you
think it will work ? :-)

>
>
> In fact for the full SWU downloadfrom S3 bucket SWU URL, I think
> fetching the file size as well  requires range request, so I get an

> error *Failed to get total download size for URL.

No, this is another message, and the server can return the size or not
in the HTTP answer. If it is present, SWUpdate will write it into the
log, but this is not an issue because SWUpdate works on streams and does
not rely on this answer.

> I*n this case the

> download proceeds, but may be because there is no size information
> beforehand, swupdate fails to print the download progress percentage.

If the server does not return, SWUpdate cannot print.

>
> **

>
> Does this give some insights on what could be wrong or missing in
> configuration?

You should try adding the max-ranges property.

Best regards,
Stefano Babic

>
>
> Thanks and regards,
>
> Raghavendra Adiga.
>

> --
> You received this message because you are subscribed to the Google
> Groups "swupdate" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to swupdate+u...@googlegroups.com
> <mailto:swupdate+u...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/swupdate/8986cdb1-9275-4b5a-9510-b4ff3b87fc8an%40googlegroups.com <https://groups.google.com/d/msgid/swupdate/8986cdb1-9275-4b5a-9510-b4ff3b87fc8an%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Raghavendra Adiga]

Ok understood. You are bang on catching the problem.

With AWS S3, I see only 1 range supported at the time - so option for me is to set something better in server or adjust max-ranges to 1 in SWU file.

Thanks again and regards,

Raghavendra Adiga.