nss symbols vs openssl symbols

[Thomas Klausner]

Hi!

TL;DR: nss has some of the same symbols as openssl, which leads to
core dumps when both are linked in the same binary

Long version:

Recently I tried packaging Sigil for pkgsrc. It links against nss via
qtwebengine. It also links against openssl via qtbase libraries, and
it has an embedded python interpreter that loads python's ssl loadable
module, which links against openssl.

Sigil's update checker uses a python script to download a file via
https. This dumps core on my machine, with a backtrace that ends in
nss, but the nss functions are called from openssl (which intends to
call its own functions of the same name instead).

Lots of details are in the Sigil bug report[1].

In the meantime I've found that the pkgsrc package for nss already
renames some of the symbols for the same reason, but triggered by
libcups or libgssapi.[2]

I wonder why this does not affect more people. Is pkgsrc or NetBSD
doing something weird/strange/wrong?

I'm not quite sure what the best fix is. I can add more patches to the
pkgsrc package to locally rename symbols.

I understand that it's hard to ask nss to change their symbols just
because openssl has the same names, but I think that's what I'm trying
to do with this email :)

What do you recommend?

Cheers,
Thomas


[1] https://github.com/Sigil-Ebook/Sigil/issues/604

[2] https://github.com/NetBSD/pkgsrc/blob/trunk/devel/nss/patches/patch-am
https://github.com/NetBSD/pkgsrc/blob/trunk/devel/nss/patches/patch-an
https://github.com/NetBSD/pkgsrc/blob/trunk/devel/nss/patches/patch-nss_lib_freebl_md5.c

[Robert Relyea]

On 4/7/21 5:18 AM, Thomas Klausner wrote:
> Hi!
>
> TL;DR: nss has some of the same symbols as openssl, which leads to
> core dumps when both are linked in the same binary
>
> Long version:
>
> Recently I tried packaging Sigil for pkgsrc. It links against nss via
> qtwebengine. It also links against openssl via qtbase libraries, and
> it has an embedded python interpreter that loads python's ssl loadable
> module, which links against openssl.
>
> Sigil's update checker uses a python script to download a file via
> https. This dumps core on my machine, with a backtrace that ends in
> nss, but the nss functions are called from openssl (which intends to
> call its own functions of the same name instead).
>
> Lots of details are in the Sigil bug report[1].
>
> In the meantime I've found that the pkgsrc package for nss already
> renames some of the symbols for the same reason, but triggered by
> libcups or libgssapi.[2]
>
> I wonder why this does not affect more people. Is pkgsrc or NetBSD
> doing something weird/strange/wrong?

We have apps that use both NSS and OpenSSL all the time and haven't ran
into problems.

>
> I'm not quite sure what the best fix is. I can add more patches to the
> pkgsrc package to locally rename symbols.
>
> I understand that it's hard to ask nss to change their symbols just
> because openssl has the same names, but I think that's what I'm trying
> to do with this email :)

If the symbols are exported in the .def files, then it's impossible. NSS
provides ABI forward compatibility for all releases. Fortunately the
symbols in question are not exported. In fact the symbols are in freebl,
which should be dynamically loaded anyway. If you are running intro
problems, it means you aren't building and running a standard version of
NSS anyway (In standard NSS those symbols are hidden in the freebl
shared library that is dynamically loaded by softoken and referenced
through a function table. softoken itself is a shared library that's
also reverenced through a function table.

In order to run into problems you are either:

Linking directly with freebl (not supported).

Linking directly with the libfreebl stub (not supported).

Linking directly with softoken (not supported).

> What do you recommend?

If you access NSS through it's shared libraries, there is no issue
operating with openssl.

bob

[Thomas Klausner]

Hi Bob!

On Wed, Apr 07, 2021 at 09:00:36AM -0700, Robert Relyea wrote:
> In order to run into problems you are either:

...

> Linking directly with softoken (not supported).

Thank you for the detailed reply. This in particular was the problem.

The pkgsrc package had since time immemorial (> 15 years) created its
own pkg-config file and that had included libsoftokn3. I've switched
the package to use the one provided in the nss distfile and Sigil is
working now.

Thanks,
Thomas

[Thomas Klausner]

Hi again!

So I've fixed the pkgsrc package to not link nss users against
libsoftokn3 any longer, and this fixed the problem in Sigil.

The pkgsrc package had renamed the MD5_Update and SHA1_Update
functions in nss; since I assumed this to be safe now after stopping
to link against libsoftokn3, I've removed these patches.

Now libreoffice as built from pkgsrc (with default backend for
tls=nss) does not "export as PDF" any longer. Here the problem is the
other way round, the openssl function is preferedly called from the
nss code.

...
(signal handler catching the segfault above this frame)
#30 <signal handler called>
#31 0x00007ba175b787d2 in memcpy () from /usr/lib/libc.so.12
#32 0x00007ba167dc9afe in __memcpy_ichk (len=18446744073702342609, src=0x7ba15c693de8, dst=<optimized out>)
at /usr/include/ssp/string.h:82
#33 MD5_Update (c=0x7ba14cb7bf00, data_=0x7ba15c693de8, len=<optimized out>)
at /usr/src/crypto/external/bsd/openssl/dist/include/crypto/md32_common.h:150
#34 0x00007ba12fa27faf in NSC_DigestUpdate (hSession=<optimized out>, pPart=0x7ba15c693de8 "%PDF-1.6\n%\303\244\303\274\303\266\303\n",
ulPartLen=19) at pkcs11c.c:1953
#35 0x00007ba165c6418b in PK11_DigestOp (context=0x7ba15cad8360, in=0x7ba15c693de8 "%PDF-1.6\n%\303\244\303\274\303\266\303\n", inLen=19)
at pk11cxt.c:1438
#36 0x00007ba165c51924 in HASH_Update (context=<optimized out>, src=src@entry=0x7ba15c693de8 "%PDF-1.6\n%\303\244\303\274\303\266\303\n",
len=len@entry=19) at sechash.c:452
#37 0x00007ba1726a170b in comphelper::Hash::update (this=this@entry=0x7ba14a74b378,
pInput=pInput@entry=0x7ba15c693de8 "%PDF-1.6\n%\303\244\303\274\303\266\303\n", length=length@entry=19)
at /scratch/misc/libreoffice/work/libreoffice-7.1.2.2/comphelper/source/misc/hash.cxx:113
...

In frame 33 you see openssl's MD5_Update called from frame 34 nss's
NSC_DigestUpdate which seems to correspond to

(*context->hashUpdate)(context->cipherInfo, pPart, ulPartLen);

in nss/lib/softoken/pkcs11c.c.

(gdb) fr 34
#34 0x00007ba12fa27faf in NSC_DigestUpdate (hSession=<optimized out>, pPart=0x7ba15c693de8 "%PDF-1.6\n%\303\244\303\274\303\266\303\n",
ulPartLen=19) at pkcs11c.c:1953
1953 pkcs11c.c: No such file or directory.
(gdb) p *context
$2 = {type = SFTK_HASH, multi = 1, rsa = 0, doPad = 0, isXCBC = 4, blockSize = 0, padDataLength = 0,
padBuf = "\000\000\000\000\245\000\000\000\000\000\000\000\004\000\000", macBuf = '\000' <repeats 12 times>, "\000\000",
k2 = "\000\000\000\000\302bJ{\000\000mϟP", k3 = "{\000\000\070\000\000\000\000\000\000\000\335\246K", macSize = 135932732871187,
cipherInfo = 0x7ba14cb7bf00, hashInfo = 0x0, cipherInfoLen = 88, currentMech = 528, update = 0x0, aeadUpdate = 0x0,
hashUpdate = 0x7ba167dc9a50 <MD5_Update>, end = 0x7ba12fa478c5 <MD5_End>, destroy = 0x7ba12fa4777b <MD5_DestroyContext>,
hashdestroy = 0x4a58647700000000, verify = 0x39795a79396d4c73, maxLen = 16, key = 0x0}
(gdb) p &MD5_Update
$4 = (int (*)(MD5_CTX *, const void *, size_t)) 0x7ba167dc9a50 <MD5_Update>
(gdb) info shared
From To Syms Read Shared Object Library
...
0x00007ba167c8e000 0x00007ba167dfbd1b Yes /usr/lib/libcrypto.so.14
...

Is there another bug in pkgsrc here, and if so, what is it?

Thanks,
Thomas

[pawel k.]

Stupid question. Are both in c not c++?

If at least one is in cpp wrap them in namespace. Can work anyway with wrapping if youll write thin cpp wrapper for one or both. Or some nifty macro nss_name(x) and ossl_name(x) to deconflict both xses.

--
You received this message because you are subscribed to the Google Groups "dev-tec...@mozilla.org" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dev-tech-cryp...@mozilla.org.
To view this discussion on the web visit https://groups.google.com/a/mozilla.org/d/msgid/dev-tech-crypto/20210407121811.fuhja7gbgwg7qnst%40yt.nih.at.

[Thomas Klausner]

Hi again!

No ideas for solutions this time?

Then I come back to my original request of renaming these internal
symbols so that they don't conflict with openssl...

(We've worked around this in pkgsrc by switching libreoffice from nss
to openssl, but now the same problem appeared in konqueror).

Thomas

> --
> You received this message because you are subscribed to the Google Groups "dev-tec...@mozilla.org" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to dev-tech-cryp...@mozilla.org.

> To view this discussion on the web visit https://groups.google.com/a/mozilla.org/d/msgid/dev-tech-crypto/20210410090644.zc5uvak6rsqud6hd%40yt.nih.at.

[Thomas Klausner]

Hi Ryan!

Thanks for the reply.

Bob's reply was quite helpful already with his hint that nothing
should link against softokn3 or freebl. This was a bug in pkgsrc
previously, but I fixed it after his message. Nothing in pkgsrc now
links against either of the two directly. I think I followed all of
his advice.

nss seems to take great care to be interoperable with openssl. But I
still don't understand how this works in detail. Perhaps you can shed
some more light on this to help me debug this problem.

Bob wrote that NSS hides the conflicting symbols through a dynamically
loaded object and a function table. I assume this means using
dlopen(). I'm not sure about the function table.

I looked at the code a bit and see that HASH_Update calls a function
indirectly, which in pk11cxt.c is forwarded and finally used in
pkcs11c.c - which however, then uses the wrong function, the one from
openssl.

Can you please explain in more detail how the function table works?
In my case, it seems the wrong MD5_Update function is used when
openssl is linked into the binary.

Perhaps it's a difference in linker behavior on NetBSD?

As for pkgsrc - I cited the patches in my original email. The whole
package information is [1], the Makefile contains the flags used when
compiling and the patches directory the patches used. If you see what
the problem is, that'd be very helpful, but I don't think it's
something that's something obvious any longer.

[1] https://github.com/NetBSD/pkgsrc/tree/trunk/devel/nss

Thanks,
Thomas

[Robert Relyea]

On 4/16/21 11:12 AM, Thomas Klausner wrote:
> Hi Ryan!
>
> Thanks for the reply.
>
> Bob's reply was quite helpful already with his hint that nothing
> should link against softokn3 or freebl. This was a bug in pkgsrc
> previously, but I fixed it after his message. Nothing in pkgsrc now
> links against either of the two directly. I think I followed all of
> his advice.
>
> nss seems to take great care to be interoperable with openssl. But I
> still don't understand how this works in detail. Perhaps you can shed
> some more light on this to help me debug this problem.
>
> Bob wrote that NSS hides the conflicting symbols through a dynamically
> loaded object and a function table. I assume this means using
> dlopen(). I'm not sure about the function table.

NSS hides the symbols with shared libraries that only export a curated
set of public symbols. Each nss shared library has it's own symbol list
found in {sharedlibname}.def so lib/nss/nss.def lib/util/nssutil.def
lib/ssl/ssl.def etc.

The NSS build system massages this file to whatever system the OS uses
to create an use an explicit export list for your shared library.

When you link with NSS, you need to link with the NSS shared library,
namley libnss3.so libnssutil3.so libssl3.so and libsmime3.so (depending
on how much of NSS you need). libsoftken3 will be automatically loaded
by libnss3.so and it in term will automatically load libfreebl* as
needed. So only the last two are dlopened. You should not link with any
libnss*.a files, or all bets are off on working with something like openssl.

If you are doing all this and still running into issues, it's likely the
build system (either your own or the NSS build system) isn't correctly
processing the .def file for your platform. The command to process the
.def file is set with the symbol PROCESS_MAP_FILE and is set in your
{OS}.mk file in nss/coreconf.

Linux, for instance, sets the command as follows:

PROCESS_MAP_FILE = grep -v ';-' $< | \
        sed -e 's,;+,,' -e 's; DATA ;;' -e 's,;;,,' -e 's,;.*,;,' > $@
This removes all the lines with ;- , then removes all occurrences of
';+',' DATA ',';;', and then all commands '; to the end of the line'

Linux later adds the directive -Wl,-c,$(MAPFILE) to it's shared library
linking command.

It looks like openBSD has the same processing line as linux, and adds
-Wl,--version-script,$(MAPFILE) which sounds right to me.

(Some unsupported OSs ignore the MAPFILE, and they could be subject to
symbol collisions).

bob

[Thomas Klausner]

Hi Bob!

Thanks again for the detailed explanation of the nss build system.

With it, it was quite easy to see that the whole symbol hiding stuff
was just missing in the NetBSD configuration coming with nss. For
nss' purposes, you can handle NetBSD, FreeBSD, and OpenBSD nearly the
same. I reduced the differences between the OpenBSD and NetBSD
configurations and came up with the attached patch.

(I also removed code for supporting the a.out file format, which is
not used in NetBSD for ten years or more. And it improves arm and
arm64 support.)

Please merge this (or something similar) into the next nss release.

Thanks,
Thomas

> --
> You received this message because you are subscribed to the Google Groups "dev-tec...@mozilla.org" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to dev-tech-cryp...@mozilla.org.

> To view this discussion on the web visit https://groups.google.com/a/mozilla.org/d/msgid/dev-tech-crypto/8c271011-93c3-529f-6e08-deff68d9bb32%40redhat.com.

[Robert Relyea]

On 5/1/21 2:57 PM, Thomas Klausner wrote:
> Hi Bob!
>
> Thanks again for the detailed explanation of the nss build system.
>
> With it, it was quite easy to see that the whole symbol hiding stuff
> was just missing in the NetBSD configuration coming with nss. For
> nss' purposes, you can handle NetBSD, FreeBSD, and OpenBSD nearly the
> same. I reduced the differences between the OpenBSD and NetBSD
> configurations and came up with the attached patch.
>
> (I also removed code for supporting the a.out file format, which is
> not used in NetBSD for ten years or more. And it improves arm and
> arm64 support.)
>
> Please merge this (or something similar) into the next nss release.
>
> Thanks,
> Thomas

If you create a bug here:

bugzilla.mozilla.org

Assign the bug to me and attach this patch, I will review it and include
it in NSS then following release of NSS. If you are maintaining NSS for
BSD systems, you probably should sign up for a phabricator account. That
would make subitting new patches to NSS easier. Your current patch is
small enough, I can handle reviewing it with the mozilla tools. and
since your patch is in the NetBSD.mk file, it shouldn't be a problem
bringing it upstream.


bob

[Martin Thomson]

https://bugzilla.mozilla.org/enter_bug.cgi?product=NSS&component=Build
for a direct link.

> To view this discussion on the web visit https://groups.google.com/a/mozilla.org/d/msgid/dev-tech-crypto/e6b61d8d-ab92-835b-16d5-0cffe31ea5e0%40redhat.com.

[Thomas Klausner]

Hi!

Thanks, Martin & Bob!

I've created

https://bugzilla.mozilla.org/show_bug.cgi?id=1709654

I can't assign to people, but I cc'd Bob.

Thank you for the help,
Thomas

> To view this discussion on the web visit https://groups.google.com/a/mozilla.org/d/msgid/dev-tech-crypto/CAPLxc%3DVGwsj0JhFmCMFudYFVGTJv0yhjmrzAWQJy52aPO339bg%40mail.gmail.com.