502 Bad Gateway - when testing collector having followed Production install documentation

pliv...@btopenworld.com

unread,

Oct 6, 2016, 6:29:40 AM10/6/16

to

Got
<html>
<head><title>502 Bad Gateway</title></head>
<body bgcolor="white">
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.10.1</center>
</body>
</html>

In reponse to:

curl -H 'Host: crash-reports' -F 'ProductName=Test' -F 'Version=1.0' -F upload_file_minidump=@7d381dc5-51e2-4887-956b-1ae9c2130109.dump http://localhost/submit

I suspect due to connector failing to start at

sudo systemctl enable socorro-collector socorro-processor
sudo systemctl start socorro-collector socorro-processor

stage.

Suspect problem with python-virtualenv but not sure how to fix/what to do.

Should the above systemctl be done inside the virtual-env? i.e.

. /data/socorro/socorro-virtualenv/bin/activate
(socorro-virtualenv)[root@localhost ]# sudo systemctl start socorro-collector socorro-processor

or not (documentation doesn't say to but can't explain why connector is failing to statr.

Output of journalctl is where I picked up on possible virtualenv issue, see below:

Oct 06 12:39:29 localhost.localdomain bash[12521]: compiled with version: 4.8.5 20150623 (Red Hat 4.8.5-4) on 04 October 2016 20:01:53
Oct 06 12:39:29 localhost.localdomain bash[12521]: os: Linux-3.10.0-327.el7.x86_64 #1 SMP Thu Nov 19 22:10:57 UTC 2015
Oct 06 12:39:29 localhost.localdomain bash[12521]: nodename: localhost.localdomain
Oct 06 12:39:29 localhost.localdomain bash[12521]: machine: x86_64
Oct 06 12:39:29 localhost.localdomain bash[12521]: clock source: unix
Oct 06 12:39:29 localhost.localdomain bash[12521]: pcre jit disabled
Oct 06 12:39:29 localhost.localdomain bash[12521]: detected number of CPU cores: 1
Oct 06 12:39:29 localhost.localdomain bash[12521]: current working directory: /home/socorro
Oct 06 12:39:29 localhost.localdomain bash[12521]: detected binary path: /data/socorro/socorro-virtualenv/bin/uwsgi
Oct 06 12:39:29 localhost.localdomain bash[12521]: setgid() to 994
Oct 06 12:39:29 localhost.localdomain bash[12521]: setuid() to 1000
Oct 06 12:39:29 localhost.localdomain bash[12521]: your processes number limit is 15087
Oct 06 12:39:29 localhost.localdomain bash[12521]: your memory page size is 4096 bytes
Oct 06 12:39:29 localhost.localdomain bash[12521]: detected max file descriptor number: 1024
Oct 06 12:39:29 localhost.localdomain bash[12521]: lock engine: pthread robust mutexes
Oct 06 12:39:29 localhost.localdomain bash[12521]: thunder lock: disabled (you can enable it with --thunder-lock)
Oct 06 12:39:29 localhost.localdomain bash[12521]: uwsgi socket 0 bound to UNIX address /var/run/uwsgi/socorro/socorro-collector.sock fd 3
Oct 06 12:39:29 localhost.localdomain bash[12521]: Python version: 2.7.11 (default, Sep 22 2016, 20:18:29) [GCC 4.8.5 20150623 (Red Hat 4.8.5-4)]
Oct 06 12:39:29 localhost.localdomain bash[12521]: Set PythonHome to /data/socorro/socorro-virtualenv
Oct 06 12:39:29 localhost.localdomain systemd[1]: socorro-processor.service: main process exited, code=exited, status=1/FAILURE
Oct 06 12:39:29 localhost.localdomain systemd[1]: Unit socorro-processor.service entered failed state.
Oct 06 12:39:29 localhost.localdomain systemd[1]: socorro-processor.service failed.
Oct 06 12:39:29 localhost.localdomain bash[12520]: Could not find platform independent libraries <prefix>
Oct 06 12:39:29 localhost.localdomain bash[12520]: Could not find platform dependent libraries <exec_prefix>
Oct 06 12:39:29 localhost.localdomain bash[12520]: Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Oct 06 12:39:29 localhost.localdomain bash[12520]: ImportError: No module named site
Oct 06 12:39:29 localhost.localdomain bash[12520]: 2016/10/06 12:39:29 [ERR] unexpected exit from subprocess (1)
Oct 06 12:39:29 localhost.localdomain bash[12521]: Traceback (most recent call last):
Oct 06 12:39:29 localhost.localdomain bash[12521]: File "/data/socorro/socorro-virtualenv/lib/python2.7/site.py", line 67, in <module>
Oct 06 12:39:29 localhost.localdomain bash[12521]: import os
Oct 06 12:39:29 localhost.localdomain bash[12521]: ImportError: No module named os
Oct 06 12:39:29 localhost.localdomain systemd[1]: socorro-collector.service: main process exited, code=exited, status=1/FAILURE
Oct 06 12:39:29 localhost.localdomain systemd[1]: Unit socorro-collector.service entered failed state.
Oct 06 12:39:29 localhost.localdomain systemd[1]: socorro-collector.service failed.
Oct 06 12:39:29 localhost.localdomain bash[12521]: 2016/10/06 12:39:29 [ERR] unexpected exit from subprocess (1)
Oct 06 12:39:29 localhost.localdomain systemd[1]: socorro-collector.service holdoff time over, scheduling restart.
Oct 06 12:39:29 localhost.localdomain systemd[1]: socorro-processor.service holdoff time over, scheduling restart.
Oct 06 12:39:29 localhost.localdomain systemd[1]: start request repeated too quickly for socorro-processor.service
Oct 06 12:39:29 localhost.localdomain systemd[1]: Failed to start Socorro Processor.
Oct 06 12:39:29 localhost.localdomain systemd[1]: Unit socorro-processor.service entered failed state.
Oct 06 12:39:29 localhost.localdomain systemd[1]: socorro-processor.service failed.
Oct 06 12:39:29 localhost.localdomain systemd[1]: start request repeated too quickly for socorro-collector.service
Oct 06 12:39:29 localhost.localdomain systemd[1]: Failed to start Socorro Collector.
Oct 06 12:39:29 localhost.localdomain systemd[1]: Unit socorro-collector.service entered failed state.
Oct 06 12:39:29 localhost.localdomain systemd[1]: socorro-collector.service failed.
Oct 06 12:58:02 localhost.localdomain sshd[12012]: Received disconnect from 10.50.154.56: 11: disconnected by user
Oct 06 12:58:02 localhost.localdomain sshd[12012]: pam_unix(sshd:session): session closed for user root
Oct 06 12:58:02 localhost.localdomain systemd-logind[627]: Removed session 20.
Oct 06 12:58:02 localhost.localdomain systemd[1]: Removed slice user-0.slice.
Oct 06 12:58:02 localhost.localdomain systemd[1]: Stopping user-0.slice.
Oct 06 12:58:07 localhost.localdomain sshd[12562]: Accepted password for root from 10.50.154.56 port 50458 ssh2
Oct 06 12:58:07 localhost.localdomain kernel: SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs
Oct 06 12:58:07 localhost.localdomain systemd[1]: Created slice user-0.slice.
Oct 06 12:58:07 localhost.localdomain systemd[1]: Starting user-0.slice.
Oct 06 12:58:07 localhost.localdomain systemd[1]: Started Session 22 of user root.
Oct 06 12:58:07 localhost.localdomain systemd-logind[627]: New session 22 of user root.
Oct 06 12:58:07 localhost.localdomain sshd[12562]: pam_unix(sshd:session): session opened for user root by (uid=0)
Oct 06 12:58:07 localhost.localdomain systemd[1]: Starting Session 22 of user root.
Oct 06 13:01:01 localhost.localdomain systemd[1]: Started Session 23 of user root.
Oct 06 13:01:01 localhost.localdomain systemd[1]: Starting Session 23 of user root.
Oct 06 13:01:01 localhost.localdomain CROND[12585]: (root) CMD (run-parts /etc/cron.hourly)
Oct 06 13:01:01 localhost.localdomain run-parts(/etc/cron.hourly)[12588]: starting 0anacron
Oct 06 13:01:01 localhost.localdomain run-parts(/etc/cron.hourly)[12594]: finished 0anacron
Oct 06 13:01:01 localhost.localdomain run-parts(/etc/cron.hourly)[12596]: starting 0yum-hourly.cron
Oct 06 13:01:01 localhost.localdomain run-parts(/etc/cron.hourly)[12600]: finished 0yum-hourly.cron

William Kahn-Greene

unread,

Oct 6, 2016, 9:12:02 AM10/6/16

to pliv...@btopenworld.com, tools-...@lists.mozilla.org

I think the key lines are these:

Oct 06 12:39:29 localhost.localdomain bash[12520]: 2016/10/06 12:39:29
[ERR] unexpected exit from subprocess (1)
Oct 06 12:39:29 localhost.localdomain bash[12521]: Traceback (most
recent call last):
Oct 06 12:39:29 localhost.localdomain bash[12521]: File
"/data/socorro/socorro-virtualenv/lib/python2.7/site.py", line 67, in
<module>
Oct 06 12:39:29 localhost.localdomain bash[12521]: import os
Oct 06 12:39:29 localhost.localdomain bash[12521]: ImportError: No
module named os

That suggests to me that you have a broken virtualenv.

If you do an "ls -l" in /data/socorro/socorro-virtualenv/lib/python2.7
it should contain a bunch of symlinks to .py files. Do the targets of
those symlinks exist?

/will

> _______________________________________________
> tools-socorro mailing list
> tools-...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/tools-socorro

pliv...@btopenworld.com

unread,

Oct 6, 2016, 10:03:21 AM10/6/16

to

Many thanks indeed this looks like the issue.

BTW: Am running cat /etc/centos-release
CentOS Linux release 7.2.1511 (Core)

Now wasn't sure if the statement about Currently Socorro is supported on CentOS 7 was specifically CentOS 7.0?

I started off with a VM installed from scratch with a minimal CentOS (7.2.1511 ;-)) ISO, which means base python install goes to /usr/lib64/ .

Out of the box/pkg soccorro-virtualenv expecting them to be in /usr/local/lib/

I must have missed something in the docs...

Sorry.

pliv...@btopenworld.com

unread,

Oct 6, 2016, 1:07:01 PM10/6/16

to

On Thursday, October 6, 2016 at 11:29:40 AM UTC+1, pliv...@btopenworld.com wrote:

So I tried from scratch in a new VM running
CentOS Linux release 7.0.1406 (Core)

and get the same issue.

as in the python sym links in the virtualenv point to /usr/local/lib/python2.7 when in actual fact they are in /usrlib64/python2.7

Even if I symbolic link i.e.

cd /usr/local/lib
ln -s ../../lib64/python2.7

I then get

Oct 06 17:54:19 localhost.localdomain bash[2733]: 2016/10/06 17:54:19 [ERR] unexpected exit from subprocess (1)
Oct 06 17:54:19 localhost.localdomain bash[2734]: Traceback (most recent call last):
Oct 06 17:54:19 localhost.localdomain bash[2734]: File "/data/socorro/socorro-virtualenv/lib/python2.7/site-packages/socorro-master-py2.7.egg/wsgi/collector.py", line 2, in <module>
Oct 06 17:54:19 localhost.localdomain bash[2734]: from socorrolib.app.socorro_app import (
Oct 06 17:54:19 localhost.localdomain bash[2734]: File "/data/socorro/socorro-virtualenv/lib/python2.7/site-packages/socorrolib/__init__.py", line 1, in <module>
Oct 06 17:54:19 localhost.localdomain bash[2734]: from pkg_resources import get_distribution, DistributionNotFound
Oct 06 17:54:19 localhost.localdomain bash[2734]: File "/data/socorro/socorro-virtualenv/lib/python2.7/site-packages/pkg_resources/__init__.py", line 25, in <module>
Oct 06 17:54:19 localhost.localdomain bash[2734]: import zipfile
Oct 06 17:54:19 localhost.localdomain bash[2734]: File "/usr/local/lib/python2.7/zipfile.py", line 4, in <module>
Oct 06 17:54:19 localhost.localdomain bash[2734]: import struct, os, time, sys, shutil
Oct 06 17:54:19 localhost.localdomain bash[2734]: File "/usr/local/lib/python2.7/shutil.py", line 12, in <module>
Oct 06 17:54:19 localhost.localdomain bash[2734]: import collections
Oct 06 17:54:19 localhost.localdomain bash[2734]: File "/usr/local/lib/python2.7/collections.py", line 9, in <module>
Oct 06 17:54:19 localhost.localdomain bash[2734]: from operator import itemgetter as _itemgetter, eq as _eq
Oct 06 17:54:19 localhost.localdomain bash[2734]: ImportError: /data/socorro/socorro-virtualenv/lib/python2.7/lib-dynload/operator.so: undefined symbol: _PyUnicodeUCS4_AsDefaultEncodedString
Oct 06 17:54:19 localhost.localdomain bash[2734]: unable to load app 0 (mountpoint='') (callable not found or import error)
Oct 06 17:54:19 localhost.localdomain bash[2734]: *** no app loaded. GAME OVER ***

So exactly which CentOS7 distro do I need?
Is it a 32 bit one (not sure you can get that?
I'm guessing can only build from source nowadays?

Sorry I think I must be missing something.

Peter Bengtsson

unread,

Oct 6, 2016, 1:33:23 PM10/6/16

to pliv...@btopenworld.com, tools-socorro

Be careful with symlinking python binaries. You're not the first one to
have such problems. The executable you start needs to run with the
libraries compiled for it.

What's wrong with the default standard python that was installed in your
new CentOS install?
Prior to attempt to install/use Socorro, did the python executable not
work? Or was it an ancient version?

> _______________________________________________
> tools-socorro mailing list
> tools-...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/tools-socorro
>

--
Peter Bengtsson
Mozilla Tools & Services

pliv...@btopenworld.com

unread,

Oct 6, 2016, 2:43:33 PM10/6/16

to

I don't think anything was wrong with the original python install. I ran up a clean vm using a centos 7.0 (and 7.2) x86-64 minimal distribution ISO.
Then followed the production environment docs from the start. So no prior python installs/apps or anything.

All done from clean centos vm.

Peter Bengtsson

unread,

Oct 6, 2016, 2:56:25 PM10/6/16

to pliv...@btopenworld.com, tools-socorro

So your python install on the OS was healthy.
Somewhere, following our instructions or scripts it broke apart when it
came to creating a virtualenv. Bummer.

We're very active helping with installs but I suggest you try to take it a
part a bit and zoom in on where the virtualenv is created and check that
it's doing what it's supposed to do. Perhaps you can create the virtualenv
manually and let the socorro code *use* it instead of creating it.

pliv...@btopenworld.com

unread,

Oct 7, 2016, 6:59:59 AM10/7/16

to

So looks like when the virtualenv created in the socorro pkg distribution was made, it was made on a machine that had python2.7.11 on in /usr/local/bin

Default python distributed with both CentOS7.0 and CentOS7.2 is 2.7.5 and is in /usr/bin

So out of the box the virtual environment in the latest pkg is incompatible with vanilla CentOS7 distributions.

Also I think there maybe be a UCS incompatibility as well, UCS2 vs UCS4.

William Kahn-Greene

unread,

Oct 7, 2016, 8:13:52 AM10/7/16

to Peter Livesey, tools-...@lists.mozilla.org

How did you install Socorro? You said earlier "followed the production
environment docs", but which docs are you talking about specifically? A url
would be helpful.

Peter Bengtsson

unread,

Oct 7, 2016, 8:20:12 AM10/7/16

to Peter Livesey, John Schneider, tools-socorro

python 2.7.11 is the right version to use. It's more secure than 2.7.5. We
recently made sure we get version 2.7.11.

Perhaps JP, cc'ed, can help you.
If security isn't important to you (we know socorro works with 2.7.5),
because, perhaps, you have it installed in a non-public place then perhaps
you can fiddle it to force it to install with the 2.7.5 from the OS.

On Fri, Oct 7, 2016 at 6:59 AM, <pliv...@btopenworld.com> wrote:

William Kahn-Greene

unread,

Oct 7, 2016, 9:02:20 AM10/7/16

to Peter Livesey, tools-...@lists.mozilla.org

Ahh... ok. Now it's all clear.

So the problem is that when generating the RPM, we include the
socorro-virtualenv/ directory which has a copy of the Python binary we
used, but doesn't include the libraries. It's not a completely vendored
virtual environment. That was "fine" when we were using Python 2.7.5 which
is the version that comes with CentOS 7. It's totally busted now that we
create the RPM on a machine with a local manual install of Python 2.7.11.

Thus unless you install that RPM into a setup exactly like ours, it's not
going to work.

I wrote up https://bugzilla.mozilla.org/show_bug.cgi?id=1308469 to cover
this, but I don't have any idea when someone could work on that.

For now, you could try building and installing Socorro manually in your vm
such that it's built in the environment it's going to run in.

Sorry about this!

/will

On Fri, Oct 7, 2016 at 8:35 AM, Peter Livesey <pliv...@btopenworld.com>
wrote:

> I creared a VM in VirtualBox using a base CentOS7 (both 7.3 and also 7.0)
>
> In each of these I followed:
>
> 1. Production Install — Socorro documentation
> <http://socorro.readthedocs.io/en/latest/production-install.html>
>
> 1. Production Install — Socorro documentation
> <http://socorro.readthedocs.io/en/latest/production-install.html>
>
>
>
>
> ------------------------------
> *From:* William Kahn-Greene <wil...@mozilla.com>
> *To:* Peter Livesey <pliv...@btopenworld.com>
> *Cc:* tools-...@lists.mozilla.org
> *Sent:* Friday, 7 October 2016, 13:13
> *Subject:* Re: [tools-socorro] 502 Bad Gateway - when testing collector
> having followed Production install documentation

>
> How did you install Socorro? You said earlier "followed the production
> environment docs", but which docs are you talking about specifically? A url
> would be helpful.
>

> On Fri, Oct 7, 2016 at 6:59 AM, <pliv...@btopenworld.com> wrote:
>

> ______________________________ _________________
> tools-socorro mailing list
> tools-...@lists.mozilla. org <tools-...@lists.mozilla.org>
> https://lists.mozilla.org/ listinfo/tools-socorro
> <https://lists.mozilla.org/listinfo/tools-socorro>
>
>
>
>
>

sbrel...@izotope.com

unread,

Nov 29, 2016, 3:07:08 PM11/29/16

to

Is there a version that I could install where this issue is not present?

Evgeniy Zolenko

unread,

Nov 29, 2016, 5:59:40 PM11/29/16

to

On Tuesday, November 29, 2016 at 1:07:08 PM UTC-7, sbrel...@izotope.com wrote:
> On Friday, October 7, 2016 at 9:02:20 AM UTC-4, William Kahn-Greene wrote:
> > Ahh... ok. Now it's all clear.
> >
> > So the problem is that when generating the RPM, we include the
> > socorro-virtualenv/ directory which has a copy of the Python binary we
> > used, but doesn't include the libraries. It's not a completely vendored
> > virtual environment. That was "fine" when we were using Python 2.7.5 which
> > is the version that comes with CentOS 7. It's totally busted now that we
> > create the RPM on a machine with a local manual install of Python 2.7.11.
> >
> > Thus unless you install that RPM into a setup exactly like ours, it's not
> > going to work.
> >
> > I wrote up https://bugzilla.mozilla.org/show_bug.cgi?id=1308469 to cover
> > this, but I don't have any idea when someone could work on that.
> >
> > For now, you could try building and installing Socorro manually in your vm
> > such that it's built in the environment it's going to run in.
> >
> > Sorry about this!
> >
> > /will

> Is there a version that I could install where this issue is not present?

You can build your own version of python 2.7.11, see below for the script. Mind the paths, this is part of a larger set of scripts. This will unlock the next level. :)

#!/bin/bash
set -u
set -e
set -x

root="$(cd "$(dirname "$0")"; pwd)"

if [[ "$(/usr/local/bin/python2.7 -V 2>&1)" == "Python 2.7.11" ]]
then
exit 0
fi

yum -y install wget gcc zlib-devel bzip2-devel openssl-devel ncurses-devel sqlite-devel

mkdir -p "$root"/../../python
pushd "$root"/../../python

curl https://www.python.org/ftp/python/2.7.11/Python-2.7.11.tgz -o Python-2.7.11.tgz
tar xzf Python-2.7.11.tgz
cd Python-2.7.11
./configure
make --jobs=4 altinstall

popd

sbrel...@izotope.com

unread,

Nov 30, 2016, 8:59:33 AM11/30/16

to

excellent, thanks!