Re: Postfix SMTP Client Segfaults over TLS

[Daniel Sutcliffe]

I'm having a very similar problem here on CentOS 6 - unfortunately moving or
removing the TLS session caches and restarting postfix is not fixing my problem
at all. Coincidently the openssl package was updated the day before the problem
started.
postfix 2.6.6-2.2.el6_1
openssl 1.0.0-20.el6_2.4

postconf: http://pastebin.com/d898xtus

maillog (with -v on smtp line in master.cf):
May 25 20:20:17 li postfix/smtp[17618]: < smtp.gmail.com[173.194.77.108]:587:
250-mx.google.com at your service, [72.3.189.225]
May 25 20:20:17 li postfix/smtp[17618]: < smtp.gmail.com[173.194.77.108]:587:
250-SIZE 35882577
May 25 20:20:17 li postfix/smtp[17618]: < smtp.gmail.com[173.194.77.108]:587:
250-8BITMIME
May 25 20:20:17 li postfix/smtp[17618]: < smtp.gmail.com[173.194.77.108]:587:
250-STARTTLS
May 25 20:20:17 li postfix/smtp[17618]: < smtp.gmail.com[173.194.77.108]:587:
250 ENHANCEDSTATUSCODES
May 25 20:20:17 li postfix/smtp[17618]: server features: 0x101b size 35882577
May 25 20:20:17 li postfix/smtp[17618]: > smtp.gmail.com[173.194.77.108]:587:
STARTTLS
May 25 20:20:17 li postfix/smtp[17618]: < smtp.gmail.com[173.194.77.108]:587:
220 2.0.0 Ready to start TLS
May 25 20:20:17 li postfix/smtp[17618]: send attr request = lookup
May 25 20:20:17 li postfix/smtp[17618]: send attr cache_type = smtp
May 25 20:20:17 li postfix/smtp[17618]: send attr cache_id =
smtp:173.194.77.108:587:mx.google.com&p=1&c=ALL:+RC4:@STRENGTH
May 25 20:20:17 li postfix/smtp[17618]: private/tlsmgr: wanted attribute: status
May 25 20:20:17 li postfix/smtp[17618]: input attribute name: status
May 25 20:20:17 li postfix/smtp[17618]: input attribute value: 4294967295
May 25 20:20:17 li postfix/smtp[17618]: private/tlsmgr: wanted attribute: session
May 25 20:20:17 li postfix/smtp[17618]: input attribute name: session
May 25 20:20:17 li postfix/smtp[17618]: input attribute value: (end)
May 25 20:20:17 li postfix/smtp[17618]: private/tlsmgr: wanted attribute:
(list terminator)
May 25 20:20:17 li postfix/smtp[17618]: input attribute name: (end)
May 25 20:20:17 li postfix/smtp[17618]: send attr request = seed
May 25 20:20:17 li postfix/smtp[17618]: send attr size = 32
May 25 20:20:17 li postfix/smtp[17618]: private/tlsmgr: wanted attribute: status
May 25 20:20:17 li postfix/smtp[17618]: input attribute name: status
May 25 20:20:17 li postfix/smtp[17618]: input attribute value: 0
May 25 20:20:17 li postfix/smtp[17618]: private/tlsmgr: wanted attribute: seed
May 25 20:20:17 li postfix/smtp[17618]: input attribute name: seed
May 25 20:20:17 li postfix/smtp[17618]: input attribute value:
+KL9oSc4Fg2uos3Vm50k86FVj4pZHODixhyaXhn9AD0=
May 25 20:20:17 li postfix/smtp[17618]: private/tlsmgr: wanted attribute:
(list terminator)
May 25 20:20:17 li postfix/smtp[17618]: input attribute name: (end)
May 25 20:20:17 li postfix/qmgr[14561]: warning: private/smtp socket:
malformed response
May 25 20:20:17 li postfix/qmgr[14561]: warning: transport smtp failure -- see
a previous warning/fatal/panic logfile record for the problem description
May 25 20:20:17 li postfix/master[14558]: warning: process
/usr/libexec/postfix/smtp pid 17618 killed by signal 11

messages:
May 25 20:20:17 li kernel: smtp[17618]: segfault at 7fc182380ef0 ip
00007fc182380ef0 sp 00007fff333cb218 error 15

Any idea what might be going on here, or what I might do to investigate
further or even fix this?

Cheers
/dan
--
Daniel Sutcliffe

[Wietse Venema]

Daniel Sutcliffe:

> I'm having a very similar problem here on CentOS 6 - unfortunately moving or
> removing the TLS session caches and restarting postfix is not fixing my problem
> at all. Coincidently the openssl package was updated the day before the problem
> started.
> postfix 2.6.6-2.2.el6_1
> openssl 1.0.0-20.el6_2.4

Was Postfix compiled for openssl 1.0.0? If it was built for 0.9.mumble,
then the warranty is void and all bets are off.

Wietse

[Daniel Sutcliffe]

Thanks for the response Wietse, most appreciated.

Daniel Sutcliffe wrote:
>> I'm having a very similar problem here on CentOS 6 - unfortunately moving or
>> removing the TLS session caches and restarting postfix is not fixing my problem
>> at all.  Coincidently the openssl package was updated the day before the problem
>> started.
>>   postfix 2.6.6-2.2.el6_1
>>   openssl 1.0.0-20.el6_2.4

Wietse Venema <wie...@porcupine.org> wrote:
> Was Postfix compiled for openssl 1.0.0? If it was built for 0.9.mumble,
> then the warranty is void and all bets are off.

CentOS 6 has always been OpenSSL 1.0.0 AFAIK and therefore the postfix
version was certainly originally built against this.

Trawling back through the logs I can see that on May 16 openssl was
upgraded from 1.0.0-20.el6_2.3 to 1.0.0-20.el6_2.4 - and there had
been no issues running the postfix 2.6.6-2.2.el6_1 package with that
previous package build of openssl.

I have now tried stopping postfix, downgrading my openssl package back
to this previous version, deleting the TLS session caches, and
starting postfix again and the same problem is occurring - which would
infer to me that it isn't an OpenSSL package version which caused the
problem - and maybe the upgrade of this package in the same time-frame
as when the problem started occurring may be a bit of a red herring :(

The only other change in system which would seem to be even slightly
related was that the kernel was updated and a reboot occurred just
before the errors started to occur. I am contemplating going back to
the previous version of the kernel but would like to take advice from
those more experienced and knowledgeable in postfix than myself as to
if this might possibly have caused the issue, or if there are any
other diagnostics I can do before this ... the server runs a live Web
site so downtime for a reboot is something I want to avoid and/or
timetable during a quiet period.

Any pointers gratefully received,
Cheers
/dan
--
Daniel Sutcliffe <Dan...@tcNOW.com>

[Wietse Venema]

Daniel Sutcliffe:

> I have now tried stopping postfix, downgrading my openssl package back
> to this previous version, deleting the TLS session caches, and
> starting postfix again and the same problem is occurring - which would
> infer to me that it isn't an OpenSSL package version which caused the
> problem - and maybe the upgrade of this package in the same time-frame
> as when the problem started occurring may be a bit of a red herring :(
>
> The only other change in system which would seem to be even slightly
> related was that the kernel was updated and a reboot occurred just
> before the errors started to occur. I am contemplating going back to

I suggest looking at

% ldd /usr/libexec/postfix/smtp

and examining all the libraries referenced.

Perhaps the update has introduced a new library-to-library dependency,
such as a new LDAP library dependency on a different SASL library
than Postfix wants. Dependencies may also be introduced via
nsswitch.conf; those don't show up in ldd output.

Kernel APIs don't change randomly.

Wietse

[Daniel Sutcliffe]

Thanks Wietse - I don't have a 'solution' yet but I now know where the
problem lies ...

Daniel Sutcliffe wrote:
>> I have now tried stopping postfix, downgrading my openssl package back
>> to this previous version, deleting the TLS session caches, and
>> starting postfix again and the same problem is occurring - which would
>> infer to me that it isn't an OpenSSL package version which caused the
>> problem - and maybe the upgrade of this package in the same time-frame
>> as when the problem started occurring may be a bit of a red herring :(
>>
>> The only other change in system which would seem to be even slightly
>> related was that the kernel was updated and a reboot occurred just
>> before the errors started to occur.  I am contemplating going back to

Wietse Venema <wie...@porcupine.org> wrote:
> I suggest looking at
>
>    % ldd /usr/libexec/postfix/smtp
>
> and examining all the libraries referenced.

I did this exactly, and from their followed the threads of evidence
through my logfiles.

> Perhaps the update has introduced a new library-to-library dependency,
> such as a new LDAP library dependency on a different SASL library
> than Postfix wants. Dependencies may also be introduced via
> nsswitch.conf; those don't show up in ldd output.
>
> Kernel APIs don't change randomly.

This was my thought too - I was definitely clutching at straws there -
I did try with the older kernel, glibc, openssl and everything else
that could have possibly changed since the setup was last known to
work... luckily I had a snapshot of the server I was able to bring up
as a virtual instance to test all the combinations out.

I have now tracked this down to the fact that I began using the
Percona version of the mysql-libs package that contains
libmysqlclient.so.16 - obviously some incompatibility there as the
original worked but the Percona version cause postfix smtp to segfault
over TLS...

As far as I am aware I don't use any postfix mysql features so I don't
even need to be linked against that library - will this be fixed at
compile time or can I somehow disbale config to stop this library even
being loaded by smtp? Any other ideas? I really need to use the
Percona MySQL server so am stuck with their versions of the client
libraries ... I think !?

All ideas welcome as to how I can workaround this - preferably without
rebuilding RPMs

[Wietse Venema]

Daniel Sutcliffe:

> I have now tracked this down to the fact that I began using the
> Percona version of the mysql-libs package that contains
> libmysqlclient.so.16 - obviously some incompatibility there as the
> original worked but the Percona version cause postfix smtp to segfault
> over TLS...
>
> As far as I am aware I don't use any postfix mysql features so I don't
> even need to be linked against that library - will this be fixed at

That eliminates only direct postfix-to-mysql dependencies. The
problem can still exist with library-to-library dependencies,
or in NSSWITCH or PAM dependencies that don't show up with ldd<
and the problem can be in a dependency of Percona mysql-libs.

Wietse

[Daniel Sutcliffe]

Daniel Sutcliffe wrote :

>> I have now tracked this down to the fact that I began using the
>> Percona version of the mysql-libs package that contains
>> libmysqlclient.so.16 - obviously some incompatibility there as the
>> original worked but the Percona version cause postfix smtp to segfault
>> over TLS...
>>
>> As far as I am aware I don't use any postfix mysql features so I don't
>> even need to be linked against that library - will this be fixed at

Wietse Venema wrote:
> That eliminates only direct postfix-to-mysql dependencies. The
> problem can still exist with library-to-library dependencies,
> or in NSSWITCH or PAM dependencies that don't show up with ldd<
> and the problem can be in a dependency of Percona mysql-libs.

Understood, thanks for the response Wietse :)

I have now made this work with a kind of Duct tape and string method,
by replacing the smtp binary with this:
> #!/bin/sh
> export LD_LIBRARY_PATH=/usr/libexec/postfix/orig:$LD_LIBRARY_PATH
> exec /usr/libexec/postfix/orig/smtp $*

And placing the good (original) libmysqlclient.so.16 in the orig
subdir, along with the original smtp binary.

Seems to work fine for my current needs but obviously not a good long
term solution - that would be to get Percona to 'fix' their
distributed shared lib so that it is compatible with official
CentOS/RHEL 6 rpms - it should be it is one specific for these
systems.

Out of interest here's the ldd output for the original libmysqlclient.so.16:
linux-vdso.so.1 => (0x00007fff27fc1000)
libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007faafed0b000)
libnsl.so.1 => /lib64/libnsl.so.1 (0x00007faafeaf2000)
libm.so.6 => /lib64/libm.so.6 (0x00007faafe86d000)
libssl.so.10 => /usr/lib64/libssl.so.10 (0x00007faafe612000)
libcrypto.so.10 => /usr/lib64/libcrypto.so.10 (0x00007faafe278000)
libz.so.1 => /lib64/libz.so.1 (0x00007faafe061000)
libc.so.6 => /lib64/libc.so.6 (0x00007faafdcd1000)
libfreebl3.so => /lib64/libfreebl3.so (0x00007faafda6f000)
libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007faafd82c000)
libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007faafd54d000)
libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007faafd349000)
libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007faafd11c000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007faafcf18000)
/lib64/ld-linux-x86-64.so.2 (0x00007faaff2cc000)
libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007faafcd0d000)
libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007faafcb09000)
libresolv.so.2 => /lib64/libresolv.so.2 (0x00007faafc8ef000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007faafc6d3000)
libselinux.so.1 => /lib64/libselinux.so.1 (0x00007faafc4b3000)
And for the Percona distributed version:
linux-vdso.so.1 => (0x00007fffc2fff000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fa51a222000)
libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007fa519feb000)
libnsl.so.1 => /lib64/libnsl.so.1 (0x00007fa519dd1000)
libm.so.6 => /lib64/libm.so.6 (0x00007fa519b4d000)
libc.so.6 => /lib64/libc.so.6 (0x00007fa5197bd000)
/lib64/ld-linux-x86-64.so.2 (0x00007fa51a74c000)
libfreebl3.so => /lib64/libfreebl3.so (0x00007fa51955a000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007fa519356000)
Obviously different ...

Thanks for the help in getting to the bottom of this - maybe these
postings will save someone else my headache in future.

[ja...@tate2.com]

I'm running CentOS 6.4 and also had this problem (also running MariaDB for MySQL). A simple yum update fixed it for me. Thanks for the documentation up to that point.

[joel....@gmail.com]

Similar issue with /usr/lib64/libmysqlclient.so.15 from MySQL-shared-compat-5.5.29-2.rhel5 in CentOS 5.10. Replacing with the stock mysqlclient15 solved the issue.

Thanks for the hints!