Bug#738575: pthread: segfault in libpthread on Intel Galileo board

[Thomas Faust]

Package: libc6
Version: 2.13-38
Severity: normal
File: pthread

Dear Maintainer,

I bootstrapped a Debian base system via "debootstrap --arch i386 wheezy ./newfiles http://http.debian.net/debian/" and put it on a Galileo board. On the Galileo board there new Intel Quark IA processor - which is basically a 486 with some instructions extensions from Pentium.
If I boot the existing Galileo kernel with the bootstrapped fileset, many applications crash with a segfault in pthread.
To reporduce, follow the instruction on https://communities.intel.com/message/220080
Here are ways to reproduce consistently - many other apps show the same behavior
1. Boot the system, create a new user (non root), connect to the board via ssh - the sshd will crash with a segfault in pthread
2. Do a 'apt-get install cowsay' - at the end, apt-get will crash with a segfault in pthread

sshd[2519]: segfault at b7173107 ip b714f07b sp bf97ea94 error ffff0007 in libpthread-2.13.so[b714a000+15000]

incorrect behavior: segfault - applications stop working
expected behavior: no crash

uname -a = Linux galileo 3.8.7-yocto-standard #1 Wed Jan 15 00:21:32 CET 2014 i586 GNU/Linux
dpkg -s libc6 = 2.13-38



-- System Information:
Debian Release: 7.3
APT prefers stable
APT policy: (500, 'stable')
Architecture: i386 (i586)

Kernel: Linux 3.8.7-yocto-standard
Locale: LANG=C, LC_CTYPE=C (charmap=ANSI_X3.4-1968)
Shell: /bin/sh linked to /bin/dash

Versions of packages libc6:i386 depends on:
ii libc-bin 2.13-38
ii libgcc1 1:4.7.2-5

Versions of packages libc6:i386 recommends:
pn libc6-i686 <none>

Versions of packages libc6:i386 suggests:
ii debconf [debconf-2.0] 1.5.49
pn glibc-doc <none>
pn locales <none>

-- debconf information:
glibc/restart-services:
libraries/restart-without-asking: false
glibc/disable-screensaver:
glibc/upgrade: true
glibc/restart-failed:


--
To UNSUBSCRIBE, email to debian-bugs-...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

[Adam Conrad]

On Mon, Jan 01, 2001 at 12:23:05AM +0000, Thomas Faust wrote:
>
> which is basically a 486 with some instructions extensions from Pentium.

The default toolchain (and thus libc) in Debian has been targetting
i586 for quite a while now. If this CPU doesn't provide *all* the
i586 instructions, I'd be pretty surprised if anything worked.

... Adam

[Aurelien Jarno]

On Mon, Jan 01, 2001 at 12:23:05AM +0000, Thomas Faust wrote:

> Package: libc6
> Version: 2.13-38
> Severity: normal
> File: pthread
>
> Dear Maintainer,
>
> I bootstrapped a Debian base system via "debootstrap --arch i386 wheezy ./newfiles http://http.debian.net/debian/" and put it on a Galileo board. On the Galileo board there new Intel Quark IA processor - which is basically a 486 with some instructions extensions from Pentium.
> If I boot the existing Galileo kernel with the bootstrapped fileset, many applications crash with a segfault in pthread.
> To reporduce, follow the instruction on https://communities.intel.com/message/220080
> Here are ways to reproduce consistently - many other apps show the same behavior
> 1. Boot the system, create a new user (non root), connect to the board via ssh - the sshd will crash with a segfault in pthread
> 2. Do a 'apt-get install cowsay' - at the end, apt-get will crash with a segfault in pthread
>
> sshd[2519]: segfault at b7173107 ip b714f07b sp bf97ea94 error ffff0007 in libpthread-2.13.so[b714a000+15000]
>
> incorrect behavior: segfault - applications stop working
> expected behavior: no crash
>
> uname -a = Linux galileo 3.8.7-yocto-standard #1 Wed Jan 15 00:21:32 CET 2014 i586 GNU/Linux
> dpkg -s libc6 = 2.13-38

The problem happens in __nptl_setxid, at address 0x507b:

00005060 <__nptl_setxid>:
5060: 55 push %ebp
5061: 31 c0 xor %eax,%eax
5063: 89 e5 mov %esp,%ebp
5065: b9 01 00 00 00 mov $0x1,%ecx
506a: 57 push %edi
506b: 56 push %esi
506c: 53 push %ebx
506d: 83 ec 14 sub $0x14,%esp
5070: e8 fb f3 ff ff call 4470 <__i686.get_pc_thunk.bx>
5075: 81 c3 7f 0f 01 00 add $0x10f7f,%ebx
=> 507b: f0 0f b1 8b 94 21 00 lock cmpxchg %ecx,0x2194(%ebx)
5082: 00
5083: 0f 85 b6 17 00 00 jne 683f <_L_lock_743>
5089: 8b 45 08 mov 0x8(%ebp),%eax
508c: 8b b3 38 01 00 00 mov 0x138(%ebx),%esi

Despite the name __i686.get_pc_thunk.bx is fine on this CPU (it
actually has been rename to __x86.get_pc_thunk.bx on recent GCC
versions), as it is only get the PC through the stack with a movl
instruction:

00004470 <__i686.get_pc_thunk.bx>:
4470: 8b 1c 24 mov (%esp),%ebx
4473: c3 ret

So the question is if the "lock cmpxchg" instruction is behaving
correctly on the Intel Quark. This should be supported according
to the developer's manual.

It might be difficult to investigate more without access to such a CPU.

--
Aurelien Jarno GPG: 1024D/F1BCDB73
aure...@aurel32.net http://www.aurel32.net

[Khaled Hamadi]

for all lovers of t-shirt ideas to offer or to wear on Valentine's Day

click on the amazon link below

https://amzn.to/2CdX642

 

best regards

[isaac jacobs]

[Fatma Karatas]

Hälsningar!

Mitt namn är barrister Fatma Karatas, jag är en juridisk utövare som arbetar inom alla områden av familjens jurisdiktion. Jag kontaktade dig för att hjälpa mig med anspråket och överföringen av pengar ($6 450 000) som lämnats kvar av min bortgångna klient som miste livet med sin familj. Jag fick ett mandat från hans bank att förse hans närmaste anhöriga innan hans medel konfiskeras. Det var därför jag kontaktade dig i detta ärende på grund av att du har samma efternamn som min bortgångne klient. Han är också infödd i ditt land och det finns ingen registrerad arvinge i hans kontofiler hos banken. Om du är intresserad vänligen svara snarast så att jag kan ge dig all nödvändig information för din bättre förståelse.

Med vänlig hälsning!

Advokat. Fatma Karatas Esq,
Istanbul, Turkiet,
Postnummer: 34000.

[James Addison]

Followup-For: Bug #738575
X-Debbugs-Cc: raykin...@gmail.com

If I understand correctly, then Ray's libx1000 library[1] provides a way to
work around this in software. It uses some LD_PRELOAD magic, and from what I
remember, it's worth being careful when using that approach.

I opened an RFP[2] for libx1000 earlier this year, and took another look at the
Debian packaging metadata in the codebase today, resulting in a few suggested
edits as a pull request on GitHub - cc'ing you to notify you about that, Ray.
I'm unsure whether some of the proposed postinst steps are required, and will
ask you about those upstream too.

[1] - http://ashroe.eu/x1000/2016/10/21/fixing-lock-prefix-on-x1000.html

[2] - https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1037070

[Ray Kinsella]

Hi James,

Thanks for this - you are kind to look at this issue. 

It's been a long time since Intel manufactured the X1000 / Quark, I am not sure how many are left in the wild.

Do you think this is something that Debian might want to package and ship?

Thanks, 

Ray K

[James Addison]

On Wed, 15 Nov 2023 at 21:57, Ray Kinsella <raykin...@gmail.com> wrote:
>
> Thanks for this - you are kind to look at this issue.

You're welcome - I enjoyed learning a bit about the Quark hardware,
and the esoteric lock bug. A shame I didn't learn about it five years
ago I suppose, but there we are.

> It's been a long time since Intel manufactured the X1000 / Quark, I am not sure how many are left in the wild.
> Do you think this is something that Debian might want to package and ship?

I should admit that I'm not a Debian maintainer or developer, just a
passerby who attempts to make progress on bugs that interest to me
(possibly to the annoyance of actual DMs/DDs), so my opinion is
somewhat external (i.e.: take with a grain of salt). It's entirely
possible that the maintenance for an additional package wouldn't be
worthwhile -- and in general, 32-bit x86 support in Debian does tend
to be dwindling. Basically: someone has to be motivated about it
enough to be responsible for the package.

On the other hand: it seemed to me based on a quick look that much of
the packaging work is already in place, and that this package would be
opt-in for anyone who wanted to use it. The typical use case would be
people preparing root filesystems on removable/replicable storage for
later (re)attachment to Quark systems, I'd guess. Even so: the
LD_PRELOAD and GRUB commandline stuff does make me a bit wary -
generally we shouldn't make any unnecessary or unexpected
modifications to the target system, because those should be the
responsibility of the sysadmin and not of the maintainer.

Do you know whether Intel shipped many Quark units? I see that
manufacturing stopped in Y2019, which isn't too long ago, but I don't
know much about how widely-adopted they were. It was the
energy-efficiency focus of them that gathered my interest in the first
place, FWIW.

[Ray Kinsella]

On Wed, 15 Nov 2023 at 22:30, James Addison <j...@jp-hosting.net> wrote:

On Wed, 15 Nov 2023 at 21:57, Ray Kinsella <raykin...@gmail.com> wrote:
>
> Thanks for this - you are kind to look at this issue.

You're welcome - I enjoyed learning a bit about the Quark hardware,
and the esoteric lock bug.  A shame I didn't learn about it five years
ago I suppose, but there we are.

I spent a not insignificant amount of time devising this solution, to get "Debian Support"

After a few false starts and missteps, eventually I came up with LibX1000 which was a pretty effective fix IMHO.

When I started sharing it around with the Debian & Kernel folks - the response was pretty clear.

"This is a mess, Intel should just fix the bug ... " - which honestly in retrospect was the right thing to do.

 

> It's been a long time since Intel manufactured the X1000 / Quark, I am not sure how many are left in the wild.
> Do you think this is something that Debian might want to package and ship?

I should admit that I'm not a Debian maintainer or developer, just a
passerby who attempts to make progress on bugs that interest to me
(possibly to the annoyance of actual DMs/DDs), so my opinion is
somewhat external (i.e.: take with a grain of salt). 

Thrilled that you reached out about it.  

 

It's entirely
possible that the maintenance for an additional package wouldn't be
worthwhile -- and in general, 32-bit x86 support in Debian does tend
to be dwindling.  Basically: someone has to be motivated about it
enough to be responsible for the package.

[SNIP] 

 

Do you know whether Intel shipped many Quark units?  I see that
manufacturing stopped in Y2019, which isn't too long ago, but I don't
know much about how widely-adopted they were.  

 

There were a few micro[processors,controllers] shipped under Quark.

My memory is that the X1000 didn't last long beyond 2017.

I remember seeing stacks of them (Galileo Boards) sitting gathering dust in Frys and Maplin. 

So I couldn't say how many are in the wild being used. 

It was the
energy-efficiency focus of them that gathered my interest in the first
place, FWIW.

Boards like the Up-board (https://up-board.org/) and its successors really filled this gap more effectively

[James Addison]

On Thu, 16 Nov 2023 at 09:57, Ray Kinsella <raykin...@gmail.com> wrote:
>
> On Wed, 15 Nov 2023 at 22:30, James Addison <j...@jp-hosting.net> wrote:
>>
>> On Wed, 15 Nov 2023 at 21:57, Ray Kinsella <raykin...@gmail.com> wrote:

> [...]

> I spent a not insignificant amount of time devising this solution, to get "Debian Support"
> After a few false starts and missteps, eventually I came up with LibX1000 which was a pretty effective fix IMHO.
>
> When I started sharing it around with the Debian & Kernel folks - the response was pretty clear.
> "This is a mess, Intel should just fix the bug ... " - which honestly in retrospect was the right thing to do.

Yep; frustrating though it can be, pushing back to figure out the
origins of bugs and resolve them there is likely the way to free up
enough developer and maintainer time, and to improve hardware and
software quality enough, to reach Reliable Technology Utopia.. should
be any day now :)

>> > It's been a long time since Intel manufactured the X1000 / Quark, I am not sure how many are left in the wild.
>> > Do you think this is something that Debian might want to package and ship?
>>
>> I should admit that I'm not a Debian maintainer or developer, just a
>> passerby who attempts to make progress on bugs that interest to me
>> (possibly to the annoyance of actual DMs/DDs), so my opinion is
>> somewhat external (i.e.: take with a grain of salt).
>
>
> Thrilled that you reached out about it.

I've been thinking more about how to improve the chances that the
package could be accepted into Debian -- my suggestion would be to
rebuild it and upload it to the mentors[1] portal, where it can
(hopefully) receive review. I've considered uploading it myself, but
I don't have hardware to test the results on, so I'd be navigating
without a way to test the results. From personal experience
attempting packaging: the mentoring guidelines and making sure to run
lintian checks are both worthwhile.

Even so there'd be no guarantee of review or upload acceptance -- and
unfortunately the same test-hardware limitation probably applies to
most reviewers -- so I don't know whether it'd be worth your time, but
in my mind it's possible that someone attempts to install Debian on an
X1000 platform in future, learns of this bug, and then in a
hypothetical future _might_ find libx1000 in the archive, and then be
grateful for the availalble fix.

(after re-reading your blog-post, I think that there could technically
still be rare cases where the bug appears despite the package being
installed -- the mention of 98% of cases handled -- but even so, a
mostly-usable system compared to a completely useless one seems like a
big improvement)

[1] - https://mentors.debian.net/

[James Addison]

Followup-For: Bug #738575
X-Debbugs-Cc: raykin...@gmail.com

On Thu, 16 Nov 2023 12:47:40 +0000, James wrote:
> I've been thinking more about how to improve the chances that the
> package could be accepted into Debian -- my suggestion would be to
> rebuild it and upload it to the mentors[1] portal, where it can
> (hopefully) receive review. I've considered uploading it myself, but
> I don't have hardware to test the results on, so I'd be navigating
> without a way to test the results. From personal experience
> attempting packaging: the mentoring guidelines and making sure to run
> lintian checks are both worthwhile.

I now have an X1000 Quark board to test this on (thanks, Ray), and am hoping
to find some time over the next week or two to try that out in combination with
the libx1000.git source-and-packaging repo.