opkg list-upgradable - Segmentation fault

[Ashlin Inwood]

opkg version 0.6.2

opkg built with Buildroot for raspberry pi cm4

Running "opkg list-upgradable" results in a segmentation fault.

```

...

pkg_vec_insert_merge: examplePkg 1.2.0 arch=arm vs. examplePkg 1.3.0 arch=arm.
pkg_vec_insert_merge: examplePkg 1.2.0 arch=arm vs. examplePkg 1.2.0 arch=arm.
pkg_vec_insert_merge: Duplicate for pkg=examplePkg version=1.2.0 arch=arm.
pkg_vec_insert_merge: Merging examplePkg 1.2.0 arch=arm, set_status=1.
pkg_info_preinstall_check: Updating file owner list.
pkg_info_preinstall_check: Updating file owner list.
Segmentation fault

```

[Alex Stewart]

Hey Ashlin,

I haven't been able to reproduce a segfault through mundane calls to
`list-upgradable`, either in a dev environment or on one of my deployed
systems.

It looks like you're using some test packages. Can you provide some more
information about your reproducing setup?

Or better, can you construct a reproducing case using the opkg test
suite? [1]

[1] https://git.yoctoproject.org/opkg/tree/tests/README.md#n42

On 8/22/23 18:19, Ashlin Inwood wrote:
>
>
> You don't often get email from ashlin...@gmail.com. Learn why this
> is important <https://aka.ms/LearnAboutSenderIdentification>

> --
> You received this message because you are subscribed to the Google
> Groups "opkg-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to opkg-devel+...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/opkg-devel/b1732e83-59a6-469d-881d-e2e1e0458f4en%40googlegroups.com
> <https://groups.google.com/d/msgid/opkg-devel/b1732e83-59a6-469d-881d-e2e1e0458f4en%40googlegroups.com?utm_medium=email&utm_source=footer>.

--
Alex Stewart
Software Engineer - NI Real-Time OS
NI (National Instruments)

alex.s...@ni.com

[Ashlin Inwood]

My setup is as follows:

opkg 0.6.2

OS is made by Buildroot and is 32bit

The system is a Raspberry pi cm4

Steps to reproduce

1. Start with a clean install of opkg, no packages installed.
2. Install test pkg, "sudo opkg install ./test_example.ipk" , (file attached)
   
3. Run "sudo opkg list-upgradable -V4" a few time

I have found that it does not segfault every time, running the command a few times will trigger it, if it will at all.

Here are some logs from a successful and failed running of list-upgradable.

I am not likely to get the opkg test suite working as this embed systemd does not have python

and getting it to run will take more time than I have atm.

Successful
opkg_conf_parse_file: Loading conf file /etc/opkg/opkg.conf.
pkg_hash_load_feeds:
pkg_hash_load_status_files:
pkg_vec_insert_merge: Adding new pkg=software_suite version=1.0.0 arch=arm.

pkg_info_preinstall_check: Updating file owner list.
pkg_info_preinstall_check: Updating file owner list.

pkg_hash_fetch_best_installation_candidate: Best installation candidate for software_suite:
pkg_hash_fetch_best_installation_candidate: Adding software_suite to providers.
pkg_hash_fetch_best_installation_candidate: software_suite arch=arm arch_priority=10 version=1.0.0.
pkg_hash_fetch_best_installation_candidate: Candidate: software_suite 1.0.0.
hash_table: pkg-hash, 12288 bytes
n_buckets=1024, n_elements=1, n_collisions=0
max_bucket_len=0, n_used_buckets=1, ave_bucket_len=1.00
n_hits=1, n_misses=1
hash_table: file-hash, 12288 bytes
n_buckets=1024, n_elements=3, n_collisions=0
max_bucket_len=0, n_used_buckets=3, ave_bucket_len=1.00
n_hits=3, n_misses=3
hash_table: dir-hash, 12288 bytes
n_buckets=1024, n_elements=2, n_collisions=0
max_bucket_len=0, n_used_buckets=2, ave_bucket_len=1.00
n_hits=2, n_misses=2
hash_table: obs-file-hash, 768 bytes
n_buckets=64, n_elements=0, n_collisions=0
max_bucket_len=0, n_used_buckets=0, ave_bucket_len=0.00
n_hits=0, n_misses=0

Failed
opkg_conf_parse_file: Loading conf file /etc/opkg/opkg.conf.
pk[ 2537.036481] audit: type=1701 audit(1692852714.599:20): auid=4294967295 uid=1000 gid=0 ses=4294967295 pid=583 comm="sudo" exe="/usr/bin/sudo" sig=11 res=1
g_hash_load_feeds:
pkg_hash_load_status_files:
pkg_vec_insert_merge: Adding new pkg=software_suite version=1.0.0 arch=arm.

pkg_info_preinstall_check: Updating file owner list.
pkg_info_preinstall_check: Updating file owner list.
Segmentation fault

Here is the opkg arm32 bin I am using and test_example.ipk
https://drive.google.com/file/d/12WcCRwpC5KLsP2gDu0wnTQrdNgJIVBMo/view?usp=drive_link

https://drive.google.com/file/d/1-jrV25zmIw3kyx14Ne2lKAxw_8dMiAAM/view?usp=drive_link 

[Alex Stewart]

I don't see anything odd with that example IPK which would make me think
that it's the issue. So that provides some confidence that the bug is in
the binary at least.

I've created a bugzilla tracking item for this issue [1]. Feel free to
continue the conversation there.

I might have an ARM32 device sitting around that I could use to try and
reproduce this; and may give that a shot this weekend. No promises though.

Can you install `gdb` to the device and try to pull a stacktrace when
the fault occurs? That will help to narrow down where exactly the issue
is coming from.

[1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=15200

On 8/24/23 20:25, Ashlin Inwood wrote:
>
>
> You don't often get email from ashlin...@gmail.com. Learn why this
> is important <https://aka.ms/LearnAboutSenderIdentification>
>
>

> My setup is as follows:
> opkg 0.6.2
> OS is made by Buildroot and is 32bit
> The system is a Raspberry pi cm4
>
> Steps to reproduce
>

> 1. Start with a clean install of opkg, no packages installed.
> 2. Install test pkg, "sudo opkg install ./test_example.ipk" , (file
> attached)
> 3. Run "sudo opkg list-upgradable -V4" a few time

>
> I have found that it does not segfault every time, running the command
> a few times will trigger it, if it will at all.
> Here are some logs from a successful and failed running of
> list-upgradable.
>
> I am not likely to get the opkg test suite working as this embed
> systemd does not have python
> and getting it to run will take more time than I have atm.
>

> *Successful*

> *Failed*

> https://groups.google.com/d/msgid/opkg-devel/723e6f8e-2600-49f5-bc29-6e3283972fb5n%40googlegroups.com
> <https://groups.google.com/d/msgid/opkg-devel/723e6f8e-2600-49f5-bc29-6e3283972fb5n%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Ashlin Inwood]

I'll see what I can do but it will be a couple of days at least

[Ashlin Inwood]

Having installed gdb and had a playing around I was only able to trigger the segmentation fault with address space layout randomization enabled (ASLR).

Disabling ASLR stopped the segmentation faults, I am going to conclude there is a problem with the OS rather than the binary or OPKG.

I will likely just disable ASLR and consider the problem solved.

The gdb output in case anyone is interested.
```

Program received signal SIGSEGV, Segmentation fault.
0xb6e920d4 in strlen () from /lib/libc.so.6
(gdb) where
#0  0xb6e920d4 in strlen () from /lib/libc.so.6
#1  0xb6f7b7b8 in xlstat (
    file_name=0x1f18dfa <error: Cannot access memory at address 0x1f18dfa>,
    st=0xbec3daf0, st@entry=0xbec3dae8) at file_util.c:49
#2  0xb6f7b8dc in file_is_dir (file_name=<optimized out>) at file_util.c:80
#3  0xb6f748f8 in pkg_info_preinstall_check () at pkg.c:1443
#4  0xb6f83550 in prepare_upgrade_list (head=0xbec3dbb4, head@entry=0xbec3dbac)
    at solvers/internal/opkg_upgrade_internal.c:207
#5  0xb6f82efc in opkg_solver_list_upgradable (num_pkgs=num_pkgs@entry=0,
    pkg_names=pkg_names@entry=0xbec3dd9c) at solvers/internal/opkg_action.c:253
#6  0xb6f6a0b4 in opkg_list_upgradable_cmd (argc=0, argv=0xbec3dd9c)
    at opkg_cmd.c:686
#7  0xb6f6a538 in opkg_cmd_exec (cmd=0xb6f9e090 <cmds+140>, argc=0,
    argv=0xbec3dd9c) at opkg_cmd.c:1310
#8  0x004a1ec8 in main (argc=2, argv=0xbec3dd94) at opkg.c:473

```

[Ashlin Inwood]

Looks like I was a little quick to point the finger at ASLR.

Having a play with content in /var/lib/opkg/lists/repo I am able to trigger the segmentation fault.
However, I cannot figure out any pattern to it.

For example, this will result in a segfault if /var/lib/opkg/lists/repo contians this:
```
Package: test
Version:  1.3.1
Architecture: arm
Maintainer: engin...@example.com
Description: example
```

But is fine if it contains this:
Package: test
Version:  1.3.1
Architecture: arm
Maintainer: engin...@example.com

Again if it contains this it will segfault:

Package: test
Version:  1.3.1
Architecture: arm
Maintainer: engin...@example.com

Package: test2
Version:  1.3.4
Architecture: arm
Maintainer: engin...@example.com


Another example, this will segfault:

Package: test2
Version:  1.0.0
Filename: cmk_software_suite_1.3.1_arm.ipk
Architecture: arm

This will not:
Package: test2
Version:  1.0.0
Architecture: arm

It looks like removing Filename and Description fixes it but that's not the case. The problem is much more complex and subtle, some times I have tested it with a large and complex /var/lib/opkg/lists/repo file and it works every time but makes a small change and it has a segmentation fault.
The above examples seem to always run the same so long as ASLR is disabled

[Alex Stewart]

Can you try reconfiguring your opkg build to use the libsolv solver
backend, and seeing if this reproduces?

`./configure --with-libsolv`

I suspect the "internal" solver may be passing malformed filenames to
the custom `xlstat()` function. Unfortunately, very few people use the
internal solver any more, and I suspect that it has rotted quite a bit.

On 8/29/23 00:42, Ashlin Inwood wrote:
>
>
> You don't often get email from ashlin...@gmail.com. Learn why this
> is important <https://aka.ms/LearnAboutSenderIdentification>
>
>

> Looks like I was a little quick to point the finger at ASLR.
>

> Having a play with content in //var/lib/opkg/lists/repo /I am able to

> trigger the segmentation fault.
> However, I cannot figure out any pattern to it.
>

> *For example, this will result in a segfault if
> //var/lib/opkg/lists/repo / contians this:*
> ```

> Package: test
> Version:  1.3.1
> Architecture: arm
> Maintainer: engin...@example.com
> Description: example
> ```
>

> *But is fine if it contains this:*

> Package: test
> Version:  1.3.1
> Architecture: arm
> Maintainer: engin...@example.com
>

> *Again if it contains this it will segfault:*

> Package: test
> Version:  1.3.1
> Architecture: arm
> Maintainer: engin...@example.com
>
> Package: test2
> Version:  1.3.4
> Architecture: arm
> Maintainer: engin...@example.com
>
>

> *Another example, this will segfault:*

> Package: test2
> Version:  1.0.0
> Filename: cmk_software_suite_1.3.1_arm.ipk
> Architecture: arm
>

> *This will not:*

> https://groups.google.com/d/msgid/opkg-devel/6dfe5600-596b-4004-b864-13dac68ab283n%40googlegroups.com
> <https://groups.google.com/d/msgid/opkg-devel/6dfe5600-596b-4004-b864-13dac68ab283n%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Ashlin Inwood]

Thank you, very much.

That looks to have fixed the problem, and probably many others I am unaware of.
Are there any other build options I should know of?

I will look into making a PR for Buildroot, as this will surely affect others, Buildroot uses the internal solver but an older version of opkg 0.4.5.

Is there anything that I should take into consideration when using libsolv over the internal solver?

I'll let people more skilled than I decided on a fix for the internal solver, perhaps officially deprecate it and add a build warning.

Thank you

[Alex Stewart]

On 8/29/23 23:28, Ashlin Inwood wrote:
> Thank you, very much.
>
> That looks to have fixed the problem, and probably many others I
> am unaware of.

Glad to hear.

> Are there any other build options I should know of?

Using libsolv is probably the only impactful change necessary.
OpenEmbedded is the primary user for the yocto/opkg fork. You can
reference the default configuration here [1] for a view of what is most
thoroughly tested.

[1]
https://gitlab.com/astewart.49c6/opkg/-/blob/dev/testing/.gitlab-ci.yml?ref_type=heads#L99

> I will look into making a PR for Buildroot, as this will surely affect
> others, Buildroot uses the internal solver but an older version of
> opkg 0.4.5.
> Is there anything that I should take into consideration when using
> libsolv over the internal solver?

Switching to libsolv doesn't imply any necessary changes to your
configuration or inputs. It's mostly an internal logic change.

The one exception is in the error output you see when opkg encounter
solving failures. The libsolv output is differently formatted, and more
informative. So if you have scripting or code that parses error output,
you'll want to revalidate.

> I'll let people more skilled than I decided on a fix for the internal
> solver, perhaps officially deprecate it and add a build warning.

Yeah; the internal solver hasn't seen active development for many years,
and isn't recommended. It's probably a good idea to just deprecate it at
this point. I'll motion to do that sometime soon.

I'll mark the bug associated with this thread as internal-solver-only.
Which realistically means that it will go on the pile and get closed
when we deprecate the internal solver. :)

> https://groups.google.com/d/msgid/opkg-devel/2522d81a-bab1-4391-a949-3f85f2ce1877n%40googlegroups.com
> <https://groups.google.com/d/msgid/opkg-devel/2522d81a-bab1-4391-a949-3f85f2ce1877n%40googlegroups.com?utm_medium=email&utm_source=footer>.