Seeking help in how to resume a partial download

[Steve Kim]

I am using opkg 0.4.2 out of the dunfell Yocto release. My understanding is that opkg supports resuming a download as long as it has been compiled to use the curl backend, which I have done. To test this, I am using the following procedure:

1. opkg install <package>

2. ctrl+c to interrupt the download

3. opkg install <package>

I can see the @stamp file getting created for my packge download in /var/cache/opkg, but when I try to resume the download (step 3), I get the following messages:

Installing <package> (1.0) on root.
Removing corrupt package file /var/cache/opkg/089b638e3c707c7c1540785a8904ebef_<package>_1.0-r0.1_aarch64.ipk.
Collected errors:
 * pkg_verify: File size mismatch: /var/cache/opkg/089b638e3c707c7c1540785a8904ebef_<package>_1.0-r0.1_aarch64.ipk is 32862208 bytes, expecting 104864534 bytes
 * opkg_install_pkg: Failed to download <package>. Perhaps you need to run 'opkg update'?
 * opkg_solver_install: Cannot install package <package>.


I have also tried using the --force-checksum option. When I use that, I get the following message:

Installing <package> (1.0) on root.
Ignored /var/cache/opkg/089b638e3c707c7c1540785a8904ebef_<package>_1.0-r0.1_aarch64.ipk checksum mismatch.
Collected errors:
 * pkg_verify: File size mismatch: /var/cache/opkg/089b638e3c707c7c1540785a8904ebef_<package>_1.0-r0.1_aarch64.ipk is 18034688 bytes, expecting 104864534 bytes
 * read_header: Failed to read archive header: Truncated input file (needed 104857600 bytes, only 0 available)
 * pkg_extract_data_file_names_to_stream: Failed to extract data file names from package '/var/cache/opkg/089b638e3c707c7c1540785a8904ebef_<package>_1.0-r0.1_aarch64.ipk'.
 * pkg_get_installed_files: Error extracting file list from /var/cache/opkg/089b638e3c707c7c1540785a8904ebef_<package>_1.0-r0.1_aarch64.ipk.
 * opkg_solver_install: Cannot install package <package>.

Can I get some direction in where I am going wrong?

Thanks,

Steve Kim

[Alex Stewart]

Hey Steve,

Could you run your opkg commands with verbosity like `opkg install -V2
...`, and include the output. It would help me to understand the
sequence of operations that opkg is doing here.

Thanks,

--
Alex Stewart
Software Engineer - NI Real-Time OS
NI (National Instruments)

alex.s...@ni.com

[Steve Kim]

Hi Alex,

Thanks for the quick response. Here is the output with running with -V2 verbosity. The package is just a 100MB ipk file. You can see from the output that I downloaded about 49MB of it, then tried to restart the download, at which point opkg errored out.

root@qemuarm64:/var/cache/opkg# opkg install -V2 <package>

opkg_conf_parse_file: Loading conf file /etc/opkg/arch.conf.
opkg_conf_parse_file: Supported arch all priority (1)
opkg_conf_parse_file: Supported arch any priority (6)
opkg_conf_parse_file: Supported arch noarch priority (11)
opkg_conf_parse_file: Supported arch aarch64 priority (16)
opkg_conf_parse_file: Supported arch armv8a priority (21)
opkg_conf_parse_file: Supported arch armv8a-crc priority (26)
opkg_conf_parse_file: Supported arch qemuarm64 priority (31)
opkg_conf_parse_file: Loading conf file /etc/opkg/base-feeds.conf.
opkg_conf_parse_file: Loading conf file /etc/opkg/opkg.conf.
pkg_hash_load_feeds:
pkg_hash_load_status_files:
pkg_info_preinstall_check: Updating file owner list.

Installing <package> (1.0) on root.

Downloading http://192.168.7.1/ipk/aarch64/<package>_1.0-r0.1_aarch64.ipk.

^CInterrupted. Writing out status database.

root@qemuarm64:/var/cache/opkg# ls -hal
drwxr-xr-x    2 root     root        4.0K Feb 24 21:37 .
drwxr-xr-x    4 root     root        4.0K Mar  9  2018 ..
-rw-r--r--    1 root     root       49.1M Feb 24 21:37 089b638e3c707c7c1540785a8904ebef_<package>_1.0-r0.1_aarch64.ipk

-rw-r--r--    1 root     root          10 Feb 24 21:37 089b638e3c707c7c1540785a8904ebef_<package>_1.0-r0.1_aarch64.ipk.@stamp

root@qemuarm64:/var/cache/opkg# opkg install -V2 <package>
opkg_conf_parse_file: Loading conf file /etc/opkg/arch.conf.
opkg_conf_parse_file: Supported arch all priority (1)
opkg_conf_parse_file: Supported arch any priority (6)
opkg_conf_parse_file: Supported arch noarch priority (11)
opkg_conf_parse_file: Supported arch aarch64 priority (16)
opkg_conf_parse_file: Supported arch armv8a priority (21)
opkg_conf_parse_file: Supported arch armv8a-crc priority (26)
opkg_conf_parse_file: Supported arch qemuarm64 priority (31)
opkg_conf_parse_file: Loading conf file /etc/opkg/base-feeds.conf.
opkg_conf_parse_file: Loading conf file /etc/opkg/opkg.conf.
pkg_hash_load_feeds:
pkg_hash_load_status_files:
pkg_info_preinstall_check: Updating file owner list.

Installing <package> (1.0) on root.
Removing corrupt package file /var/cache/opkg/089b638e3c707c7c1540785a8904ebef_<package>_1.0-r0.1_aarch64.ipk.

opkg_configure_packages: Configuring unpacked packages.
opkg_configure_packages: Reordering packages before configuring them...
Collected errors:
 * pkg_verify: File size mismatch: /var/cache/opkg/089b638e3c707c7c1540785a8904ebef_<package>_1.0-r0.1_aarch64.ipk is 51511060 bytes, expecting 104864534 bytes

 * opkg_install_pkg: Failed to download <package>. Perhaps you need to run 'opkg update'?
 * opkg_solver_install: Cannot install package <package>.

[Alex Stewart]

Thanks for the log. I've done a little digging through the source and I
can readily see where the opkg_download_curl.c code implies that it can
"resume" a download [1].

But looking over the high-level download codepaths: it isn't clear to me
that there is a codepath that doesn't call `pkg_verify()` prior to
attempting the download; which I expect will always fail in the case of
a partially-downloaded package. I find this reference [2] particularly
likely to break your use-case; you might try commenting it out, to see
if it gets you closer to your goal.


Otherwise, it's probable that the cURL RESUME option isn't well
exercised in the opkg community, and that it has been broken for a long
time without anyone noticing. I don't see any tests which exercise it.
And the only reference I see to it is this line [3] in the NEWS
document. Is that where you learned about it?

If this is something that you're interested in working, a first step
would be to figure out how to reliably reproduce it using the test
framework. Then at least we can test the fix and save future "you"s the
hassle of it breaking again.


[1]
https://git.yoctoproject.org/opkg/tree/libopkg/opkg_download_curl.c?id=b52c702d0c6e00b3304ac26d517f9a93b63db1c8#n263

[2]
https://git.yoctoproject.org/opkg/tree/libopkg/opkg_download.c?id=b52c702d0c6e00b3304ac26d517f9a93b63db1c8#n339

[3]
https://git.yoctoproject.org/opkg/tree/NEWS?id=b52c702d0c6e00b3304ac26d517f9a93b63db1c8#n279


Hope that helps,