Hash mismatch

117 views
Skip to first unread message

esad ergül

unread,
Jun 19, 2024, 9:49:04 AMJun 19
to swupdate

Hi Stefano,

We are currently using the SWUpdate Mongoose webserver interface to upload update images. The devices we are  using Industrialpcs Intel, AMD, or Celeron processors and are operating with Debian. On these devices, we have an Nginx reverse proxy that forwards requests to nignx swupdate location .

Intermittently, some devices encounter a hash mismatch error during the update process. The update progresses to a certain point, for example, 45% (though this percentage varies each time), and then gets stuck. Subsequently, the update fails and a hash mismatch error is reported. After a few attempts, the update process starts working normally again. However, this error has started to occur more frequently.

We are using Debian Buster and SWUpdate version 2022.05, and we plan to upgrade to Bookworm shortly. Do you have any idea what might be causing this issue?

Below is the Nginx configuration we are using.

location /update/ {
  proxy_http_version 1.1;
  proxy_set_header Upgrade $http_upgrade;
  proxy_set_header Connection "upgrade";
  proxy_set_header Connection keep-alive;
  proxy_set_header Host $host;
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header X-Forwarded-Proto $scheme;
  proxy_cache_bypass $http_upgrade;
  proxy_read_timeout 86400;
  proxy_send_timeout 86400;
  proxy_connect_timeout 86400;
  proxy_buffering off;
  proxy_request_buffering off;
  client_max_body_size 0;
  proxy_pass http://localhost:8080/;
}

And the runtime configuration for swupdate is as follows:

webserver :
{
...
        timeout = 60;
        run-postupdate = false;
};


The error message:


Jun 19 14:06:51 swupdate-tls.sh[5054]: [TRACE] : SWUPDATE running :  [extract_files] :         filename root.img.gz
Jun 19 14:06:51 swupdate-tls.sh[5054]: [TRACE] : SWUPDATE running :  [extract_files] :         size 1876953746 required
Jun 19 14:06:51 swupdate-tls.sh[5054]: [TRACE] : SWUPDATE running :  [extract_files] : Installing STREAM root.img.gz, 1876953746 bytes
Jun 19 14:06:51 swupdate-tls.sh[5054]: [TRACE] : SWUPDATE running :  [install_single_image] : Found installer for stream root.img.gz raw
Jun 19 14:06:51 swupdate[5054]: RUN [LUAstackDump] : (1) [bool  ] true
Jun 19 14:06:51 swupdate[5054]: RUN [lua_dump_table] : (2) [table ] skip = 0.0
Jun 19 14:06:51 swupdate[5054]: RUN [lua_dump_table] : (2) [table ] ivt =
Jun 19 14:06:51 swupdate[5054]: RUN [lua_dump_table] : (2) [table ] compressed = false
Jun 19 14:06:51 swupdate[5054]: RUN [lua_dump_table] : (2) [table ] preserve_attributes = false
Jun 19 14:06:51 swupdate[5054]: RUN [lua_dump_table] : (2) [table ] installed_directly = false
Jun 19 14:06:51 swupdate[5054]: RUN [lua_dump_table] : (2) [table ] size = 0.0
Jun 19 14:06:51 swupdate[5054]: RUN [lua_dump_table] : (2) [table ] name = GRUB_BOOT_PART
Jun 19 14:06:51 swupdate[5054]: RUN [lua_dump_table] : (2) [table ] data =
Jun 19 14:06:51 swupdate[5054]: RUN [lua_dump_table] : (2) [table ] checksum = 0.0
Jun 19 14:06:51 swupdate[5054]: RUN [lua_dump_table] : (2) [table ] sha256 = 0000000000000000000000000000000000000000000000000000000000000000
Jun 19 14:06:51 swupdate[5054]: RUN [lua_dump_table] : (2) [table ] install_if_higher = false
Jun 19 14:06:51 swupdate[5054]: RUN [lua_dump_table] : (2) [table ] version = 0
Jun 19 14:06:51 swupdate[5054]: RUN [lua_dump_table] : (2) [table ] filesystem =
Jun 19 14:06:51 swupdate[5054]: RUN [lua_dump_table] : (2) [table ] partition = false
Jun 19 14:06:51 swupdate[5054]: RUN [lua_dump_table] : (2) [table ] script = false
Jun 19 14:06:51 swupdate[5054]: RUN [lua_dump_table] : (2) [table ] volume =
Jun 19 14:06:51 swupdate[5054]: RUN [lua_dump_table] : (2) [table ] encrypted = false
Jun 19 14:06:51 swupdate[5054]: RUN [lua_dump_table] : (2) [table ] path =
Jun 19 14:06:51 swupdate[5054]: RUN [lua_dump_table] : (2) [table ] install_if_different = false
Jun 19 14:06:51 swupdate[5054]: RUN [lua_dump_table] : (2) [table ] type =
Jun 19 14:06:51 swupdate[5054]: RUN [lua_dump_table] : (2) [table ] filename =
Jun 19 14:06:51 swupdate[5054]: RUN [lua_dump_table] : (2) [table ] mtdname =
Jun 19 14:06:51 swupdate[5054]: RUN [lua_dump_table] : (2) [table ] device =
Jun 19 14:06:51 swupdate[5054]: RUN [lua_dump_table] : (2) [table ] offset = 0.0
Jun 19 14:06:51 swupdate[5054]: RUN [lua_parser_fn] : Script returns 0
Jun 19 14:06:51 swupdate[5054]: RUN [_parse_bootloader] : Bootloader var: GRUB_BOOT_PART = 0
Jun 19 14:06:51 swupdate[5054]: RUN [check_hw_compatibility] : Hardware compatibility verified
Jun 19 14:06:51 swupdate[5054]: RUN [extract_files] : Found file
Jun 19 14:06:51 swupdate[5054]: RUN [extract_files] :         filename root.img.gz
Jun 19 14:06:51 swupdate[5054]: RUN [extract_files] :         size 1876953746 required
Jun 19 14:06:51 swupdate[5054]: RUN [extract_files] : Installing STREAM root.img.gz, 1876953746 bytes
Jun 19 14:06:51 swupdate[5054]: RUN [install_single_image] : Found installer for stream root.img.gz raw
Jun 19 14:12:58 swupdate[5054]: FAILURE ERROR : HASH mismatch : 0db256a1f68afb9064669f3275bea4a94a3b2176c46be1dfc8f8aa048fb40416 <--> 8a03e1b170461a69e0fc4b3ca0d780e174a056e702553552203327c0736a8039
Jun 19 14:13:02 swupdate-tls.sh[5054]: [TRACE] : SWUPDATE running :  [install_single_image] : Installer for raw not successful !
Jun 19 14:13:02 swupdate-tls.sh[5054]: [TRACE] : SWUPDATE running :  [network_initializer] : Main thread sleep again !
Jun 19 14:13:02 swupdate-tls.sh[5054]: [INFO ] : No SWUPDATE running :  Waiting for requests...
Jun 19 14:13:02 swupdate[5054]: RUN [install_single_image] : Installer for raw not successful !
Jun 19 14:13:02 swupdate[5054]: FAILURE ERROR : Error streaming root.img.gz
Jun 19 14:13:02 swupdate[5054]: FATAL_FAILURE Image invalid or corrupted. Not installing ...
Jun 19 14:13:02 swupdate[5054]: RUN [network_initializer] : Main thread sleep again !
Jun 19 14:13:02 swupdate[5054]: IDLE Waiting for requests...

As Info, I generate the SWU update image inside a Bookworm-based Docker container using swugenerator. However, the root filesystem partition image within the SWU is created with ELBE inside a Debian Buster-based container.

Thank you for your help.

Best regards,

Michael Glembotzki

unread,
Jun 19, 2024, 12:22:23 PMJun 19
to swupdate
Hi Esad,

esad ergül schrieb am Mittwoch, 19. Juni 2024 um 15:49:04 UTC+2:

Hi Stefano,

We are currently using the SWUpdate Mongoose webserver interface to upload update images. The devices we are  using Industrialpcs Intel, AMD, or Celeron processors and are operating with Debian. On these devices, we have an Nginx reverse proxy that forwards requests to nignx swupdate location .

Can you please test, if the issue also occur without nginx reverse proxy. 

Intermittently, some devices encounter a hash mismatch error during the update process. The update progresses to a certain point, for example, 45% (though this percentage varies each time), and then gets stuck. Subsequently, the update fails and a hash mismatch error is reported. After a few attempts, the update process starts working normally again. However, this error has started to occur more frequently.

We are using Debian Buster and SWUpdate version 2022.05, and we plan to upgrade to Bookworm shortly. Do you have any idea what might be causing this issue?

Does the issue occur with 2024.05 as well?

Below is the Nginx configuration we are using.

location /update/ {
  proxy_http_version 1.1;
  proxy_set_header Upgrade $http_upgrade;
  proxy_set_header Connection "upgrade";
  proxy_set_header Connection keep-alive;
  proxy_set_header Host $host;
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header X-Forwarded-Proto $scheme;
  proxy_cache_bypass $http_upgrade;
  proxy_read_timeout 86400;
  proxy_send_timeout 86400;
  proxy_connect_timeout 86400;
  proxy_buffering off;
  proxy_request_buffering off;
  client_max_body_size 0;

I'm surprised that the reverse proxy works at all. Without a payload (client_max_body_size 0) , no data should get through at all.

Best regards,
Michael

esad ergül

unread,
Jul 23, 2024, 10:15:35 AMJul 23
to swupdate
Hi Michael,

We still have the same problem. What I have also noticed is that I can update the device normally with curl. Update progress also stops with curl at 20 percent for about 3 minutes, but update does not fail. I don't know what swupdate is doing at that point and why it takes so much time to write the disk. When I want to upload the swu via browser (via dropzone), update progress stops again at 20 percent for about 2-3 minutes and update fails. I didn't understand the difference that much unfortunately. I have changed nginx reverse proxy timeout options, but that didn't help. I have increased dropzone timeout. That didn't help either.  When I analyzed with strace swupdate process, I saw the following error messages when updating via browser.
Setting client_max_body_size to 0 disables checking of client request body size, otherweise i get the 413 (Request entity too large), because our swu file is about 2GB large.

This somehow doesn't work when I want to update devices with low performance, otherwise it is working. Swupdate somehow tries to write to the hard disk for 2-3 minutes and web browser gets no response at that time. And then someone closes the connection after a certain time. swupdate tries to continue writing  until it realizes that the connection has been closed. We use by the way zero copy feature and have set in reverse proxy that requests are not buffered in between.

Best regards,
Esad Ergül

error_hash_mismatch.PNG
strace.log.tar.gz

Giorgio Torres (Eugico)

unread,
Aug 28, 2024, 7:26:13 AMAug 28
to swupdate

Hi everyone,

I'm facing the same issue. When I throttle the network down, let's say to 3G, when uploading an swu image, after 1 minute I get this error on the server.
I've already tried to change timeout values for Mongoose but with no success.
I'm wondering if this could be the problem. I'm not an expert (not even an intermediate) in C, but maybe the network thread times out after 60 seconds of uploading an image? 🤔

I'll be following this thread, please if you have any updates or manage to bypass this problem, I really appreciate if you let me know 😊

Best,
-- Giorgio.

Stefano Babic

unread,
Aug 31, 2024, 12:19:28 PMAug 31
to Giorgio Torres (Eugico), swupdate
Hi Giorgio,

On 28.08.24 10:58, Giorgio Torres (Eugico) wrote:
>
> Hi everyone,
>
> I'm facing the same issue. When I throttle the network down, let's say
> to 3G, when uploading an swu image, after 1 minute I get this error on
> the server.
> I've already tried to change timeout values for Mongoose but with no
> success.
> I'm wondering if this
> <https://github.com/sbabic/swupdate/blob/32431be199ce408f3280b62b33111fa919c92238/core/network_thread.c#L40> could be the problem. I'm not an expert (not even an intermediate) in C, but maybe the network thread times out after 60 seconds of uploading an image? 🤔

It has nothing to do. The timeout is for internal IPC in case a process
establish a connection to command SWUpdate. when a REQ_INSTALL is sent,
the connection is not closed, and the stream is passed to the core:

https://github.com/sbabic/swupdate/blob/32431be199ce408f3280b62b33111fa919c92238/core/network_thread.c#L488

It is on a different path, and the timeout is not set at all.

>
> I'll be following this thread, please if you have any updates or manage
> to bypass this problem, I really appreciate if you let me know 😊

I guess that this should be debugged, it is not possible to say what is
going on. The core has no timeout, because it can take a while to
install a chunk / artifact. The only timeout is in mongoose:

https://github.com/sbabic/swupdate/blob/32431be199ce408f3280b62b33111fa919c92238/mongoose/mongoose_interface.c#L599

Set with :

https://github.com/sbabic/swupdate/blob/32431be199ce408f3280b62b33111fa919c92238/examples/configuration/swupdate.cfg#L274

Exactly what you did.

Best regards,
Stefano Babic
>   proxy_pass http://localhost:8080/ <http://localhost:8080/>;
> --
> You received this message because you are subscribed to the Google
> Groups "swupdate" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to swupdate+u...@googlegroups.com
> <mailto:swupdate+u...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/swupdate/8f031080-43a8-46af-b62c-02ed082aad40n%40googlegroups.com <https://groups.google.com/d/msgid/swupdate/8f031080-43a8-46af-b62c-02ed082aad40n%40googlegroups.com?utm_medium=email&utm_source=footer>.
Reply all
Reply to author
Forward
0 new messages