Upgrading Socorro

Ahmed Messaoud

unread,

Apr 21, 2016, 5:47:36 PM4/21/16

to

I upgraded my breakpad runtimes for my application and noticed that since the upgrade, my stack traces are missing from socorro. Im guessing the processor doesn't support the new runtimes. ANyways, i decided to upgrade socorro.

* sudo systemctl stop socorro-processor socorro-collector ngnix
* sudo yum remove socorro
* sudo yum install socorro

Got the following errors/warnings but just ignored them since they didn't look critical
---
mkdir: cannot create directory '/home/socorro/temp': File exists
Job for nginx.service failed. See 'systemctl status nginx.service' and 'journalctl -xn' for details.
warning: %post(socorro-1:185.1461247650-1.x86_64) scriptlet failed, exit status 1
Non-fatal POSTIN scriptlet failure in rpm package 1:socorro-185.1461247650-1.x86_64
Verifying : 1:socorro-185.1461247650-1.x86_64 1/1

Installed:
socorro.x86_64 1:185.1461247650-1
---

Now i restarted the services
* sudo systemctl start ngnix socorro-processor socorro-collector

Went to the collectors URL -> /submit and got "502/Bad Gateway"

Looked at the logs (sudo journalctl -u socorro-collector) and noticed the following...

---
Apr 21 17:33:31 magor-software systemd[1]: Starting Socorro Collector...
Apr 21 17:33:31 magor-software systemd[1]: Started Socorro Collector.
Apr 21 17:33:31 magor-software bash[32041]: *** Starting uWSGI 2.0.12 (64bit) on [Thu Apr 21 17:33:31 2016] ***
Apr 21 17:33:31 magor-software bash[32041]: compiled with version: 4.8.2 20140120 (Red Hat 4.8.2-16) on 21 April 2016
Apr 21 17:33:31 magor-software bash[32041]: os: Linux-3.10.0-229.4.2.el7.x86_64 #1 SMP Wed May 13 10:06:09 UTC 2015
Apr 21 17:33:31 magor-software bash[32041]: nodename: magor-software
Apr 21 17:33:31 magor-software bash[32041]: machine: x86_64
Apr 21 17:33:31 magor-software bash[32041]: clock source: unix
Apr 21 17:33:31 magor-software bash[32041]: pcre jit disabled
Apr 21 17:33:31 magor-software bash[32041]: detected number of CPU cores: 2
Apr 21 17:33:31 magor-software bash[32041]: current working directory: /home/socorro
Apr 21 17:33:31 magor-software bash[32041]: detected binary path: /data/socorro/socorro-virtualenv/bin/uwsgi
Apr 21 17:33:31 magor-software bash[32041]: setgid() to 997
Apr 21 17:33:31 magor-software bash[32041]: setuid() to 1002
Apr 21 17:33:31 magor-software bash[32041]: your processes number limit is 15094
Apr 21 17:33:31 magor-software bash[32041]: your memory page size is 4096 bytes
Apr 21 17:33:31 magor-software bash[32041]: detected max file descriptor number: 1024
Apr 21 17:33:31 magor-software bash[32041]: lock engine: pthread robust mutexes
Apr 21 17:33:31 magor-software bash[32041]: thunder lock: disabled (you can enable it with --thunder-lock)
Apr 21 17:33:31 magor-software bash[32041]: uwsgi socket 0 bound to UNIX address /var/run/uwsgi/socorro/socorro-colle
Apr 21 17:33:31 magor-software bash[32041]: Python version: 2.7.5 (default, Jun 17 2014, 18:11:42) [GCC 4.8.2 201401
Apr 21 17:33:31 magor-software bash[32041]: Set PythonHome to /data/socorro/socorro-virtualenv
Apr 21 17:33:31 magor-software bash[32041]: *** Python threads support is disabled. You can enable it with --enable-t
Apr 21 17:33:31 magor-software bash[32041]: Python main interpreter initialized at 0x1a4c0e0
Apr 21 17:33:31 magor-software bash[32041]: your server socket listen backlog is limited to 100 connections
Apr 21 17:33:31 magor-software bash[32041]: your mercy for graceful operations on workers is 60 seconds
Apr 21 17:33:31 magor-software bash[32041]: mapped 145536 bytes (142 KB) for 1 cores
Apr 21 17:33:31 magor-software bash[32041]: *** Operational MODE: single process ***
Apr 21 17:33:31 magor-software bash[32041]: Welcome to Socorro. To configure Socorro, please see http://socorro.read
Apr 21 17:33:31 magor-software bash[32041]: use --help with this app to see what you can do here
Apr 21 17:33:31 magor-software bash[32041]: WSGI app 0 (mountpoint='') ready in 0 seconds on interpreter 0x1a4c0e0 pi
Apr 21 17:33:31 magor-software bash[32041]: *** uWSGI is running in multiple interpreter mode ***
Apr 21 17:33:31 magor-software bash[32041]: spawned uWSGI master process (pid: 32046)
Apr 21 17:33:31 magor-software bash[32041]: spawned uWSGI worker 1 (pid: 32048, cores: 1)
Apr 21 17:35:57 magor-software bash[32041]: TypeError: 'NoneType' object is not callable
Apr 21 17:35:57 magor-software bash[32041]: [pid: 32048|app: 0|req: 1/1] 207.107.97.14 () {40 vars in 722 bytes} [Thu
Apr 21 17:35:57 magor-software bash[32041]: TypeError: 'NoneType' object is not callable
Apr 21 17:35:57 magor-software bash[32041]: [pid: 32048|app: 0|req: 2/2] 207.107.97.14 () {42 vars in 707 bytes} [Thu
---

Looks to be like python cannot interpret some code... (TypeError: 'NoneType' object is not callable).

Am i doing something wrong here, am i missing something?

Thanks in advance!
Ahmed

Adrian Gaudebert

unread,

Apr 26, 2016, 5:25:01 AM4/26/16

to Ahmed Messaoud, tools-socorro

Hello,

It's quite difficult to help you without more information. Do you have a
more complete stack trace for that Python error?

Cheers,
Adrian

> _______________________________________________
> tools-socorro mailing list
> tools-...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/tools-socorro
>

K Lars Lohn

unread,

Apr 26, 2016, 10:01:17 AM4/26/16

to Gaudebert, Adrian, tools-socorro, Ahmed Messaoud

i helped someone else yesterday with what appears to have been the same
problem. In his case, the collector application class had not been
specified in consul. this command solved the problem for him:

consulate kv set socorro/collector/application
socorro.collector.collector_app.Collector2015App

Ahmed Messaoud

unread,

Apr 26, 2016, 4:48:56 PM4/26/16

to

That was me you helped Lars, thanks :)

So a little more insight on this, i did an upgrade from 118 to 187. After lots of debugging and log scraping with the socorro team we managed to fix it. It looks like there was a missing configuration that was introduced somewhere between version 118 and 187, that key being "socorro/collector/application" which the application now requires. This key is actually present in the wiki http://socorro.readthedocs.org/en/latest/configuring-socorro.html#storing-configuration-in-consul but sadly, upgrades will be missing this key which required us to add it into consul manually.

We added it using a python consul client (consulate - https://github.com/gmr/consulate). Then simply ran the command lars pointed out (consulate kv set socorro/collector/application
socorro.collector.collector_app.Collector2015App), restarted the collector service (systemctl restart socorro-collector) and everything was working as expected!

Thanks to rhelmer, peterbe, lars on #breakpad for the help!
Cheers