Weird expat error on some reviews.

[Papaswede]

I got ReviewBoard 1.6.11 successfully migrated to 2.0.15 and it seems to be working fine for the most part. However, I noticed that there are a couple of existing reviews that throw an error as soon as I try to access them. I have no idea why only some reviews throw this error.

ImportError at /reviews/r/2228/

/usr/lib64/python2.6/lib-dynload/pyexpat.so: symbol XML_SetHashSalt, version EXPAT_2_0_1_RH not defined in file libexpat.so.1 with link time reference

Request Method:	GET
Request URL:	http://reviewboard/reviews/r/2228/
Django Version:	1.6.11
Exception Type:	ImportError
Exception Value:	/usr/lib64/python2.6/lib-dynload/pyexpat.so: symbol XML_SetHashSalt, version EXPAT_2_0_1_RH not defined in file libexpat.so.1 with link time reference
Exception Location:	/usr/lib64/python2.6/xml/parsers/expat.py in <module>, line 4
Python Executable:	/usr/bin/python
Python Version:	2.6.6
Python Path:	['/opt/local/revboard/reviewboard/conf', '/opt/local/revboard/python/virtualenv/lib/python2.6/site-packages/setuptools-15.2-py2.6.egg', '/opt/local/revboard/python/virtualenv/lib/python2.6/site-packages/python_memcached-1.54-py2.6.egg', '/opt/local/revboard/python/virtualenv/lib/python2.6/site-packages/six-1.9.0-py2.6.egg', '/opt/local/revboard/python/virtualenv/lib/python2.6/site-packages/Whoosh-2.7.0-py2.6.egg', '/opt/local/revboard/python/virtualenv/lib/python2.6/site-packages/recaptcha_client-1.0.6-py2.6.egg', '/opt/local/revboard/python/virtualenv/lib/python2.6/site-packages/pytz-2015.2-py2.6.egg', '/opt/local/revboard/python/virtualenv/lib/python2.6/site-packages/python_dateutil-1.5-py2.6.egg', '/opt/local/revboard/python/virtualenv/lib/python2.6/site-packages/Pygments-2.0.2-py2.6.egg', '/opt/local/revboard/python/virtualenv/lib/python2.6/site-packages/paramiko-1.15.2-py2.6.egg', '/opt/local/revboard/python/virtualenv/lib/python2.6/site-packages/mimeparse-0.1.3-py2.6.egg', '/opt/local/revboard/python/virtualenv/lib/python2.6/site-packages/Markdown-2.4.1-py2.6.egg', '/opt/local/revboard/python/virtualenv/lib/python2.6/site-packages/docutils-0.12-py2.6.egg', '/opt/local/revboard/python/virtualenv/lib/python2.6/site-packages/Djblets-0.8.18-py2.6.egg', '/opt/local/revboard/python/virtualenv/lib/python2.6/site-packages/django_haystack-2.3.1-py2.6.egg', '/opt/local/revboard/python/virtualenv/lib/python2.6/site-packages/django_evolution-0.7.5-py2.6.egg', '/opt/local/revboard/python/virtualenv/lib/python2.6/site-packages/Django-1.6.11-py2.6.egg', '/opt/local/revboard/python/virtualenv/lib/python2.6/site-packages/ecdsa-0.13-py2.6.egg', '/opt/local/revboard/python/virtualenv/lib/python2.6/site-packages/pillowfight-0.2-py2.6.egg', '/opt/local/revboard/python/virtualenv/lib/python2.6/site-packages/django_pipeline-1.3.27-py2.6.egg', '/opt/local/revboard/python/virtualenv/lib/python2.6/site-packages/Pillow-2.8.1-py2.6-linux-x86_64.egg', '/opt/local/revboard/python/virtualenv/lib/python2.6/site-packages/futures-2.2.0-py2.6.egg', '/opt/local/revboard/python/virtualenv/lib/python2.6/site-packages/MySQL_python-1.2.5-py2.6-linux-x86_64.egg', '/opt/local/revboard/python/virtualenv/lib/python2.6/site-packages/ReviewBoard-2.0.15-py2.6.egg', '/opt/local/revboard/reviewboard', '/opt/local/revboard/python/virtualenv/lib/python2.6/site-packages', '/usr/lib/python2.6/site-packages/python_memcached-1.54-py2.6.egg', '/usr/lib/python2.6/site-packages/six-1.9.0-py2.6.egg', '/usr/lib/python2.6/site-packages/Whoosh-2.7.0-py2.6.egg', '/usr/lib/python2.6/site-packages/recaptcha_client-1.0.6-py2.6.egg', '/usr/lib/python2.6/site-packages/pytz-2015.2-py2.6.egg', '/usr/lib/python2.6/site-packages/python_dateutil-1.5-py2.6.egg', '/usr/lib/python2.6/site-packages/Pygments-2.0.2-py2.6.egg', '/usr/lib/python2.6/site-packages/paramiko-1.15.2-py2.6.egg', '/usr/lib/python2.6/site-packages/mimeparse-0.1.3-py2.6.egg', '/usr/lib/python2.6/site-packages/Markdown-2.4.1-py2.6.egg', '/usr/lib/python2.6/site-packages/docutils-0.12-py2.6.egg', '/usr/lib/python2.6/site-packages/Djblets-0.8.18-py2.6.egg', '/usr/lib/python2.6/site-packages/django_haystack-2.3.1-py2.6.egg', '/usr/lib/python2.6/site-packages/django_evolution-0.7.5-py2.6.egg', '/usr/lib/python2.6/site-packages/ecdsa-0.13-py2.6.egg', '/usr/lib/python2.6/site-packages/pillowfight-0.2-py2.6.egg', '/usr/lib/python2.6/site-packages/feedparser-5.2.0-py2.6.egg', '/usr/lib/python2.6/site-packages/django_pipeline-1.3.27-py2.6.egg', '/usr/lib/python2.6/site-packages/Pillow-2.8.1-py2.6-linux-x86_64.egg', '/usr/lib/python2.6/site-packages/futures-2.2.0-py2.6.egg', '/usr/lib/python2.6/site-packages/pip-6.1.1-py2.6.egg', '/usr/lib/python2.6/site-packages/virtualenv-12.1.1-py2.6.egg', '/usr/lib64/python26.zip', '/usr/lib64/python2.6', '/usr/lib64/python2.6/plat-linux2', '/usr/lib64/python2.6/lib-tk', '/usr/lib64/python2.6/lib-old', '/usr/lib64/python2.6/lib-dynload', '/usr/lib64/python2.6/site-packages', '/usr/lib64/python2.6/site-packages/gtk-2.0', '/usr/lib/python2.6/site-packages', '/usr/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg-info']
Server time:	Wed, 3 Jun 2015 19:46:20 +0000

[Papaswede]

Also, might be worth noting. If I fire up a python shell using the same virtualenv that ReviewBoard is running under, I can import expat fine:

 bin]$ python

Python 2.6.6 (r266:84292, Jan 22 2014, 01:49:05) 

[GCC 4.4.7 20120313 (Red Hat 4.4.7-4)] on linux2

Type "help", "copyright", "credits" or "license" for more information.

>>> from pyexpat import *

>>> 

So, something seems to be going awry within ReviewBoard.

[Christian Hammond]

Hi,

What does it say when you do:

    >>> import pyexpat

    >>> print pyexpat.__file__

We don't directly use expat, and this looks very much like an issue with the build of pyexpat or a LD_LIBRARY_PATH issue, so it's unlikely to be us. Perhaps it's something with your Apache environment?

Christian

--
Christian Hammond - chi...@chipx86.com
Review Board - http://www.reviewboard.org

Beanbag, Inc. - http://www.beanbaginc.com

--
Supercharge your Review Board with Power Pack: https://www.reviewboard.org/powerpack/
Want us to host Review Board for you? Check out RBCommons: https://rbcommons.com/
Happy user? Let us know! https://www.reviewboard.org/users/
---
You received this message because you are subscribed to the Google Groups "reviewboard" group.
To unsubscribe from this group and stop receiving emails from it, send an email to reviewboard...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Papaswede]

I get this:

>>> import pyexpat

>>> print pyexpat.__file__

/opt/local/revboard/python/virtualenv/lib64/python2.6/lib-dynload/pyexpat.so

So, are there any circumstances under which ReviewBoard would trigger some functionality in expat? I can't find anything in common with the few reviews that cause this issue, and if I use the patch file from those reviews to re-post them, then the newly posted review works just fine. I'm just trying to gauge whether there is just something funky left over from when I upgraded from 1.6.11 with those existing reviews and I don't really need to worry about it, or if it could potentially cause problems in the future with new reviews as well.

I did notice one thing. The libexpat.so that apache is using points to /usr/local/lib/libexpat.so, and if I run the following on that file I get no results:

(virtualenv)[eberglind@rbdev301 lib]$ strings libexpat.so.1.5.2 | grep XML_SetHashSalt

(virtualenv)[eberglind@rbdev301 lib]$ strings libexpat.so.1.5.2 | grep EXPAT_2_0_1_RH

However, when I check which expat Python is using, I get this:

(virtualenv)[eberglind@rbdev301 lib-dynload]$ ldd pyexpat.so 

linux-vdso.so.1 =>  (0x00007ffff252a000)

libexpat.so.1 => /lib64/libexpat.so.1 (0x00007f52afbae000)

libpython2.6.so.1.0 => /usr/lib64/libpython2.6.so.1.0 (0x00007f52af802000)

libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f52af5e4000)

libc.so.6 => /lib64/libc.so.6 (0x00007f52af250000)

libdl.so.2 => /lib64/libdl.so.2 (0x00007f52af04c000)

libutil.so.1 => /lib64/libutil.so.1 (0x00007f52aee48000)

libm.so.6 => /lib64/libm.so.6 (0x00007f52aebc4000)

/lib64/ld-linux-x86-64.so.2 (0x00007f52affe3000)

So, python is using the one in /lib64. If I run the same commands on that one, I do get results.

(virtualenv)[eberglind@rbdev301 lib64]$ strings libexpat.so.1.5.2 | grep XML_SetHashSalt

XML_SetHashSalt

(virtualenv)[eberglind@rbdev301 lib64]$ strings libexpat.so.1.5.2 | grep EXPAT_2_0_1_RH

EXPAT_2_0_1_RH

The one that Apache is using was compiled from the same source as the one that python is using, and they're both the same version. It just seems that Python is using the 64bit one and Apache is not?

[Papaswede]

I was able to solve this. I'm not sure if this is the "right" way, but it worked for me. Since I noticed that the libexpat.so.0 symlink in my apache lib dir was pointed at the one in /usr/local/lib and that one did in fact not contain the strings that the error was complaining about, I deleted that symlink and created a new one that pointed at /lib64/libexpat.so.1.5.2, like so:

    sudo ln -s /lib64/libexpat.so.1.5.2 libexpat.so.0

Then I restarted the machine and everything works as expected.

On Wednesday, June 3, 2015 at 4:02:22 PM UTC-4, Papaswede wrote: