pkg segementation faults

[Dan Partelly]

Hi John,

Unfortunately pkg-static seems to not work lately on Linux:

uname -a
Linux csonti1 4.14.13-1-ARCH #1 SMP PREEMPT Wed Jan 10 11:14:50 UTC 2018 x86_64 GNU/Linux

I tried two times in the last weeks, and I get segmentation faults:

Checking integrity...Child process pid=1336 terminated abnormally: Segmentation fault

or corruption of linked lists in use.  May it be that pkg-static is not ready for primetime on Linux

Dan

[John Marino]

That's not what this means.

1) when was the last time it worked (pkg-static doesn't change much)?

2) What version of linux was that?

3) What version of linux are you on now?

I need a lot more information.

Linux is supposed to have a stable ABI

--
You received this message because you are subscribed to the Google Groups "Ravenports" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravenports+unsubscribe@googlegroups.com.
To post to this group, send email to raven...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ravenports/5514cac0-fbcd-4d6f-8d65-1f7f6524a831%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Dan Partelly]

It worked OK in Oct 2017. I know you need more info to debug the problem, and if I had it, I would have provided it to you. But I did not used it in a long time since I installed some packages that interested me, then only watched the progress on the ravenport catalog to see
if new stuff I want arises. I was hoping it arised for other people too.

To me this looks like a probable memory corruption (as indicated  sometime by pkg-static printing out "corrupted double linked list).

Maybe would be a good idea to provide an auxiliary binary for pkg-static with debug info for each platform, which we can boot auxiliary to stripped one, would be easier to see what is going on. Lacking that you can tell me from which repository is pkg-static build, and Ill try to generate one when I have time, so I have meaningful coredumps and ability to run pkg-static under debugger control .

[John Marino]

part of the issue is that you're the only one to report something like this.

It's literally the first report of any trouble on Linux.

That's why I was specifically asking if Linux had been upgraded in that time.  Maybe it's only happening on a new kernel or something like that.

What pkg command caused the segfault?

Does any command work?

I'll try with an old fedora 22 image I have.

--

You received this message because you are subscribed to the Google Groups "Ravenports" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravenports+unsubscribe@googlegroups.com.
To post to this group, send email to raven...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ravenports/9e4c40fe-74f8-4ffd-8e89-92d8514e0aa6%40googlegroups.com.

[Dan Partelly]

Yes the system has been upgraded since then.  Its running latest kernel from Jan 10. The current uname was in the first message, but here is again:
uname -a
Linux  4.14.13-1-ARCH #1 SMP PREEMPT Wed Jan 10 11:14:50 UTC 2018 x86_64 GNU/Linux.

Last year beginning of October I was prolly running kernel 4.12.8

The command causing troubles is pkg-static  install package-name . Try big packages which pull lot of dependencies. Sometimes it works, sometimes no. For xinit:standard it crashed 5-6 time in a row . I gave up. For others it crashed once, but on a retry it managed to complete the installation.

Other usual commands on pkg-static like info ,search, delete  caused no issues, but the usage was light.

On Thursday, January 18, 2018 at 12:07:52 AM UTC+2, John Marino wrote:

part of the issue is that you're the only one to report something like this.

It's literally the first report of any trouble on Linux.

That's why I was specifically asking if Linux had been upgraded in that time.  Maybe it's only happening on a new kernel or something like that.

What pkg command caused the segfault?

Does any command work?

I'll try with an old fedora 22 image I have.

On Wed, Jan 17, 2018 at 3:48 PM, Dan Partelly <dan_pa...@rdsor.ro> wrote:

It worked OK in Oct 2017. I know you need more info to debug the problem, and if I had it, I would have provided it to you. But I did not used it in a long time since I installed some packages that interested me, then only watched the progress on the ravenport catalog to see
if new stuff I want arises. I was hoping it arised for other people too.

To me this looks like a probable memory corruption (as indicated  sometime by pkg-static printing out "corrupted double linked list).

Maybe would be a good idea to provide an auxiliary binary for pkg-static with debug info for each platform, which we can boot auxiliary to stripped one, would be easier to see what is going on. Lacking that you can tell me from which repository is pkg-static build, and Ill try to generate one when I have time, so I have meaningful coredumps and ability to run pkg-static under debugger control .

--
You received this message because you are subscribed to the Google Groups "Ravenports" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ravenports+...@googlegroups.com.

To post to this group, send email to raven...@googlegroups.com.

[John Marino]

If it's dependent on load, then it might be a straight-up bug in pkg-static that somehow only shows in linux.

I was assuming it broke 100% of the time.  

[John Marino]

By the way, if you haven't installed "pkg-bsd" directly yet, do it.

Then use "/raven/sbin/pkg" instead of "/raven/sbin/pkg-static" and report back if the problem exists in that version too.

[Dan Partelly]

No it doesnt. It segafaults at random. 

/raven/sbin/pkg install  xinit-single-standard is especially problematic, I do not manage to install it with either util no matter how many times I tried.  I managed to install xorg-server-single-standard after several tries.

[Dan Partelly]

Hi John,

Did you managed to test the package manager on your Linux distributions?

[John Marino]

Not yet.  I've been working on porting ravenports to solaris.  It's taking a lot of time.

Incidentally, there are no segfaults with pkg-bsd on solaris.  At least, none so far.

On Tue, Jan 23, 2018 at 3:04 AM, Dan Partelly <dan_pa...@rdsor.ro> wrote:

Hi John,

Did you managed to test the package manager on your Linux distributions?

--

You received this message because you are subscribed to the Google Groups "Ravenports" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ravenports+unsubscribe@googlegroups.com.

To post to this group, send email to raven...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ravenports/41f41425-0143-4d70-8778-7bf1df75f80e%40googlegroups.com.

[John Marino]

Hi Dan, I can reproduce this using xinit-single-standard on the same ubuntu machine that produces the packages.

I can't look into it yet, but reproducing is a good start.

[Dan Partelly]

Hi John, this is good news.

Also, how is the work on Solaris progressing ?

 

[John Marino]

The solaris work was progressing slowly for a while, but I put it on a back burner.  I'll probably get back to it as soon as this linux pkg(8) issue is resolved.

Using addr2line, I got this backtrace:

pkg-static[0x4ebfd3] sqlite3MemMalloc      /construction/pkg-bsd/pkg-9732e6a/external/sqlite/sqlite3.c:20450

pkg-static[0x4ec4c3] mallocWithAlarm       /construction/pkg-bsd/pkg-9732e6a/external/sqlite/sqlite3.c:24119

pkg-static[0x4ec550] sqlite3Malloc         /construction/pkg-bsd/pkg-9732e6a/external/sqlite/sqlite3.c:24150

pkg-static[0x56ef25] sqlite3ParserAlloc    /construction/pkg-bsd/pkg-9732e6a/external/sqlite/sqlite3.c:134321

pkg-static[0x573d72] sqlite3RunParser      /construction/pkg-bsd/pkg-9732e6a/external/sqlite/sqlite3.c:137353

pkg-static[0x54e680] sqlite3Prepare        /construction/pkg-bsd/pkg-9732e6a/external/sqlite/sqlite3.c:114130

pkg-static[0x54e8ac] sqlite3LockAndPrepare /construction/pkg-bsd/pkg-9732e6a/external/sqlite/sqlite3.c:114217

pkg-static[0x54ea5c] sqlite3_prepare_v2    /construction/pkg-bsd/pkg-9732e6a/external/sqlite/sqlite3.c:114293

pkg-static[0x46bc2a] pkg_conflicts_check_local_path     /construction/pkg-bsd/pkg-9732e6a/libpkg/pkg_jobs_conflicts.c:372

pkg-static[0x46c1a2] pkg_conflicts_check_chain_conflict /construction/pkg-bsd/pkg-9732e6a/libpkg/pkg_jobs_conflicts.c:484

pkg-static[0x46c38b] pkg_conflicts_append_chain         /construction/pkg-bsd/pkg-9732e6a/libpkg/pkg_jobs_conflicts.c:567

pkg-static[0x46839f] pkg_jobs_check_conflicts           /construction/pkg-bsd/pkg-9732e6a/libpkg/pkg_jobs.c:2284

pkg-static[0x467025] pkg_jobs_solve                     /construction/pkg-bsd/pkg-9732e6a/libpkg/pkg_jobs.c:1884

pkg-static[0x41a4c6] exec_install                       /construction/pkg-bsd/pkg-9732e6a/src/install.c:223

pkg-static[0x41d71b] main                               /construction/pkg-bsd/pkg-9732e6a/src/main.c:888

So the issue appears to be with sqlite3 malloc.  that matches with the linux error "Checking integrity...*** Error in `pkg': corrupted double-linked list: 0x000000000396cab0 ***"

so i'm closing in on the problem.

On Thu, Feb 22, 2018 at 6:10 AM, Dan Partelly <dan_pa...@rdsor.ro> wrote:

Hi John, this is good news.

Also, how is the work on Solaris progressing ?

 

--

You received this message because you are subscribed to the Google Groups "Ravenports" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravenports+unsubscribe@googlegroups.com.
To post to this group, send email to raven...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ravenports/5e4041d7-7234-488a-910f-6c293f2ba403%40googlegroups.com.

[John Marino]

Strangely, it seems like the libmad package is causing this.  I can run 3 level debug (-ddd) but if I use the 4th level (-dddd) this turns into a heisenbug.  It actually finishes and xinit wants to install.

DBG(1)[1594]> Binary> loading //mech/ravenports/shiny/packages/All/libmad-single-speed-0.15.1b.tzst

DBG(2)[1594]> Parsing manifest from buffer

DBG(3)[1594]> Manifest: found key: 'name'

DBG(3)[1594]> Manifest: found key: 'origin'

DBG(3)[1594]> Manifest: found key: 'version'

DBG(3)[1594]> Manifest: found key: 'comment'

DBG(3)[1594]> Manifest: found key: 'maintainer'

DBG(3)[1594]> Manifest: found key: 'www'

DBG(3)[1594]> Manifest: found key: 'abi'

DBG(3)[1594]> Manifest: found key: 'arch'

DBG(3)[1594]> Manifest: found key: 'prefix'

DBG(3)[1594]> Manifest: found key: 'flatsize'

DBG(3)[1594]> Manifest: found key: 'licenselogic'

DBG(3)[1594]> Manifest: found key: 'desc'

DBG(3)[1594]> Manifest: found key: 'categories'

DBG(3)[1594]> Manifest: parsing array

DBG(3)[1594]> Manifest: found key: 'shlibs_provided'

DBG(3)[1594]> Manifest: parsing array

DBG(3)[1594]> added shlib provide libmad.so.0 for libmad-single-speed

DBG(3)[1594]> Manifest: found key: 'options'

DBG(3)[1594]> Manifest: parsing object

DBG(2)[1594]> Pkg> adding options: SPEED = on

DBG(3)[1594]> Manifest: found key: 'files'

DBG(3)[1594]> Manifest: parsing object

DBG(3)[1594]> Pkg: add new file '/raven/include/mad.h'

DBG(3)[1594]> Pkg: add new file '/raven/lib/libmad.a'

DBG(3)[1594]> Pkg: add new file '/raven/lib/libmad.so'

DBG(3)[1594]> Pkg: add new file '/raven/lib/libmad.so.0'

DBG(3)[1594]> Pkg: add new file '/raven/lib/libmad.so.0.2.1'

DBG(3)[1594]> Pkg: add new file '/raven/lib/pkgconfig/mad.pc'

Checking integrity...*** Error in `pkg-static': corrupted double-linked list: 0x000000000290b510 ***

[John Marino]

forgot the link:

https://github.com/jrmarino/ravensource/commit/0ccffddec673255c98a7aa3250a9724f0517ef6f

>  Okay, I think I've got this fixed.

I did it by replacing the bundled 3.15.2 version of sqlite3 with version 3.23.0 and the segfaults have gone away.   In a few minutes, I'll update the linux repository again and it will contain the latest pkg-bsd with this fix.

[Dan Partelly]

This is great news ! Ill test over the weekend .

[Dan Partelly]

By default, after fetching reavenports, pkgstatic search path for repository is:

" http://ravenports.ironwolf.systems/repository/Linux:3.2.0:amd64"

but this path does not exist on ravenports system, where all I could find is

http://ravenports.ironwolf.systems/repository/Linux:2.6.2:amd64.

Could we get a fix in defaults please ?

[John Marino]

There's nothing wrong with the "default".

Pkg is determining the path given the glibc compatibility used to build your linux system.  For just about everyone, that compatibility is "Linux:2.6.32:amd64/" (not 2.6.2).  If package is saying it needs "3.2.0" then that implies your version of linux isn't like most others.

Assuming that your linux really is 2.6.32 compatible and only the detection of the abi is off, you can try adding this line to /raven/etc/pkg.conf (creating if necessary):

ABI = " Linux:2.6.32:amd64 ";

To confirm there's something strange with your Linux ABI, run "readelf -n /bin/bash" and report the output.  The line expected is "OS: Linux, ABI: 2.6.32"

   

[Dan Partelly]

The output is OS: Linux, ABI: 3.2.0. Stock Arch Linux.

 

[John Marino]

And what happens when you add 

====

ABI = " Linux:2.6.32:amd64 ";

====

to /raven/etc/pkg.conf ?

This is first linux I've seen with an ABI that high.  All the others I've seen have limited it to 2.6.32 for lowest common denominator purposes.

On Sun, Apr 15, 2018 at 1:57 PM, Dan Partelly <dan_pa...@rdsor.ro> wrote:

The output is OS: Linux, ABI: 3.2.0. Stock Arch Linux.

 

--

You received this message because you are subscribed to the Google Groups "Ravenports" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravenports+unsubscribe@googlegroups.com.
To post to this group, send email to raven...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ravenports/cee6d195-35d9-4786-97d2-1e804de16835%40googlegroups.com.

[Dan Partelly]

It seems that glibc changed defaults in 2.26 for x86-64 and x86-32

------------

Changes to build and runtime requirements:

* Linux kernel 3.2 or later is required at runtime, on all architectures
  supported by that kernel.  (This is a change from version 2.25 only for
  x86-32 and x86-64.)

-----------

check release notes yourself
https://sourceware.org/ml/libc-alpha/2017-08/msg00010.html

This is why ABI chages . I expect youll see much more of those in the future as distros update their libc.

How do you plan to deal with it ? LTS wll stay for years on older glibc, while others will update pretty fast. But since ABI is backward compatible, programs should still run.

On Sunday, April 15, 2018 at 11:51:29 PM UTC+3, John Marino wrote:

And what happens when you add 

====

ABI = " Linux:2.6.32:amd64 ";

====

to /raven/etc/pkg.conf ?

This is first linux I've seen with an ABI that high.  All the others I've seen have limited it to 2.6.32 for lowest common denominator purposes.

On Sun, Apr 15, 2018 at 1:57 PM, Dan Partelly <dan_pa...@rdsor.ro> wrote:

The output is OS: Linux, ABI: 3.2.0. Stock Arch Linux.

 

--
You received this message because you are subscribed to the Google Groups "Ravenports" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ravenports+...@googlegroups.com.

To post to this group, send email to raven...@googlegroups.com.

[Dan Partelly]

actually , it seems the change for x86-32 and x86-64 occurred in 2.25 and in 2.26 it became mandatory on all architectures supported on kernel 3.2.0

[John Marino]

well, the ABI changes all the time, but the distributions have been intentionally limiting builds to a low ABI (e.g. 2.6.32).

I've been asking if the 2.6.32 packages work on your 3.2.0 system.  If you set the ABI in pkg.conf, the pkg should fetch from the right place (assume pkg still works) and all the 2.6.32 probably would still work if backwards compatibility is maintained.

Ultimately multiple linux repositories might be needed.  Dunno.  If 2.6.32 packages work on 3.2.0 just fine, then we'll just keep using that repo as long as possible and improve the quickstart instructions and/or bootstrap download.

To unsubscribe from this group and stop receiving emails from it, send an email to ravenports+unsubscribe@googlegroups.com.

To post to this group, send email to raven...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ravenports/dc1bc976-7f90-4f68-bcae-c4cc5def3862%40googlegroups.com.

[Dan Partelly]

Well with the changes to glibc, I dont think that 2.6.32 ABI makes sense *if* you use the newer versions, seems they mandate the use of 3.2.0 kernel, and they do not compile support for any lower kernel. As list this is what I gathered from  the run time requirements. if it make senseArch adopted them pretty fast, and for a desktop oriented system it makes sense to use latest versions.

Unfortunately no time yet to test the programs, Ill doit this night and report back to you. It will be just a cursory test, but enough for a start. I hoe the theory holds and we can run the older binaries since it will simplify life.

To view this discussion on the web visit https://groups.google.com/d/msgid/ravenports/CA%2BxQ5CSHdD_6GEvpjt8PbpTJUkVvP3rk4s-1eRJ89-H%3D1zQpUw%40mail.gmail.com.

[Dan Partelly]

Well pkg still works, after setting ABI in /raven/etc/pkg.conf, finds
the metadata , but yet another hurdle:

==========

pkg-static: wrong architecture: linux:2.6.32:x86:64 instead of
Linux:2.6.32:amd64
pkg-static: repository Raven contains packages with wrong ABI:
linux:2.6.32:x86:64

=========

so no luck. Manual fetching of packages to "pkg add" and solve
dependencies for a test is too time consuming. Can you create a symlink
or something so I can test and report to you ?

[John Marino]

Is there anything on the web stating 2.6.32 packages won't run on a 3.2.0 system?  It seems to me that it should be backwards compatible.  a few installed ravenports packages would probably reveal this as well.

[John Marino]

That's weird.  I created a symlink until I can figure out what's going on the background (without pkg.conf, the default ABI obviously doesn't have this problem).  You might have the change pkg.conf ABI to  linux:2.6.32:x86:64 to make it work in the short term.

--
You received this message because you are subscribed to the Google Groups "Ravenports" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravenports+unsubscribe@googlegroups.com.
To post to this group, send email to raven...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ravenports/a802855c-69a8-de08-6a0e-2e436b0c55f8%40rdsor.ro.

[Dan Partelly]

Well, now I managed to fetch the metadata and files from repository. I
tried to install ravenports-single-linux. Packaged fetched, at unpack
time, an old friend:

Checking integrity...Child process pid=1830 terminated abnormally:
Segmentation fault

I tried second time, this time it managed to finish the install.
ravenadm works OK at a cursory test (I did not built anything with it
for a more serious test, but I will tommrow), so the error above must be
my old friend, which you suposedly fixed.

Oppinions ?

On 04/16/2018 08:12 PM, John Marino wrote:
> That's weird.  I created a symlink until I can figure out what's going
> on the background (without pkg.conf, the default ABI obviously doesn't
> have this problem).  You might have the change pkg.conf ABI to
> linux:2.6.32:x86:64 to make it work in the short term.
>
> On Mon, Apr 16, 2018 at 11:39 AM, Dan Partelly <dan_pa...@rdsor.ro
> <mailto:dan_pa...@rdsor.ro>> wrote:
>
> Well pkg still works, after setting ABI in /raven/etc/pkg.conf,
> finds the metadata , but yet another hurdle:
>
> ==========
>
> pkg-static: wrong architecture: linux:2.6.32:x86:64 instead of
> Linux:2.6.32:amd64
> pkg-static: repository Raven contains packages with wrong ABI:
> linux:2.6.32:x86:64
>
> =========
>
> so no luck. Manual fetching of packages to "pkg add" and solve
> dependencies for a test is too time consuming. Can you create a
> symlink or something so I can test and report to you ?
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Ravenports" group.
> To unsubscribe from this group and stop receiving emails from it,

> send an email to ravenports+...@googlegroups.com
> <mailto:ravenports%2Bunsu...@googlegroups.com>.

> To post to this group, send email to raven...@googlegroups.com

> <mailto:raven...@googlegroups.com>.

> To view this discussion on the web visit
> https://groups.google.com/d/msgid/ravenports/a802855c-69a8-de08-6a0e-2e436b0c55f8%40rdsor.ro

> <https://groups.google.com/d/msgid/ravenports/a802855c-69a8-de08-6a0e-2e436b0c55f8%40rdsor.ro>.

>
>
> For more options, visit https://groups.google.com/d/optout

> <https://groups.google.com/d/optout>.
>
>

[John Marino]

No, there are probably more segfaults to be found.  I'm just hopeful that pkg(8) is mostly usable at this point.

I think pkg developers will take fixes quickly but somebody has to report it and the fix first.  I've been too busy to spend time outlining every bug and fix. 

I don't know what the best approach to adding a new package manager would be.  It might be writing a converter to convert RP packages to another format.  The data should all be there.  If ravenadm is needed to produce new packages, it would be a pretty big effort.  Pkg(8) is used during the build process to load/unload dependencies (and to create the package itself) so we're talking about a lot of change to support alternative packages.  I would think a separate, independent package converter should be considered first.  (or if a client demanded another format then I've have motivation to support it natively).