Issues with Heroku for 7 days or so?

[David Schultner]

Hi All,

Two lab mates and I have issues pushing new tasks to Heroku (already running tasks work fine and can be edited etc.). There's no error messages when pushing new tasks, but the heroku link loads slowly and there's a generic 'application error' message. 

The heroku logs don't help (no error codes etc., just the usual gunicorn/worker failed to load gibberish). 

The specific scripts that failed to push worked fine just a week ago.

Does anyone have a similar experience? Any way forward?

Thanks!

[Jeff Moher]

I'm running into something similar.  Tried downloading psiturk-setup-example and loading that into heroku and got the same issue.  Here are the heroku error logs with the psiturk-setup-example:

2022-10-01T15:28:40.000843+00:00 app[web.1]: self.reap_workers()

2022-10-01T15:28:40.000860+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.10/site-packages/gunicorn/arbiter.py", line 525, in reap_workers

2022-10-01T15:28:40.000860+00:00 app[web.1]: raise HaltServer(reason, self.WORKER_BOOT_ERROR)

2022-10-01T15:28:40.000889+00:00 app[web.1]: gunicorn.errors.HaltServer: <HaltServer 'Worker failed to boot.' 3>

2022-10-01T15:28:40.001023+00:00 app[web.1]: 2022-10-01T15:28:39Z <Greenlet at 0x7fdf1e1b4160: <bound method Arbiter.handle_chld of <gunicorn.arbiter.Arbiter object at 0x7fdf1e5caf20>>(17, None)> failed with HaltServer

2022-10-01T15:28:40.001024+00:00 app[web.1]: 

2022-10-01T15:29:09.637497+00:00 app[web.1]: [2022-10-01 15:29:09 +0000] [4] [CRITICAL] WORKER TIMEOUT (pid:28)

2022-10-01T15:29:10.301798+00:00 app[web.1]: [2022-10-01 15:29:10 +0000] [29] [ERROR] Exception in worker process

2022-10-01T15:29:10.301827+00:00 app[web.1]: Traceback (most recent call last):

2022-10-01T15:29:10.301828+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.10/site-packages/gunicorn/arbiter.py", line 583, in spawn_worker

2022-10-01T15:29:10.301829+00:00 app[web.1]: worker.init_process()

2022-10-01T15:29:10.301829+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.10/site-packages/gunicorn/workers/ggevent.py", line 162, in init_process

2022-10-01T15:29:10.301830+00:00 app[web.1]: super().init_process()

2022-10-01T15:29:10.301830+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.10/site-packages/gunicorn/workers/base.py", line 119, in init_process

2022-10-01T15:29:10.301831+00:00 app[web.1]: self.load_wsgi()

2022-10-01T15:29:10.301831+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.10/site-packages/gunicorn/workers/base.py", line 144, in load_wsgi

2022-10-01T15:29:10.301831+00:00 app[web.1]: self.wsgi = self.app.wsgi()

2022-10-01T15:29:10.301832+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.10/site-packages/gunicorn/app/base.py", line 67, in wsgi

2022-10-01T15:29:10.301832+00:00 app[web.1]: self.callable = self.load()

2022-10-01T15:29:10.301840+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.10/site-packages/psiturk/experiment_server.py", line 68, in load

2022-10-01T15:29:10.301840+00:00 app[web.1]: return util.import_app("psiturk.experiment:app")

2022-10-01T15:29:10.301841+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.10/site-packages/gunicorn/util.py", line 358, in import_app

2022-10-01T15:29:10.301841+00:00 app[web.1]: mod = importlib.import_module(module)

2022-10-01T15:29:10.301842+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.10/importlib/__init__.py", line 126, in import_module

2022-10-01T15:29:10.301842+00:00 app[web.1]: return _bootstrap._gcd_import(name[level:], package, level)

2022-10-01T15:29:10.301842+00:00 app[web.1]: File "<frozen importlib._bootstrap>", line 1050, in _gcd_import

2022-10-01T15:29:10.301843+00:00 app[web.1]: File "<frozen importlib._bootstrap>", line 1027, in _find_and_load

2022-10-01T15:29:10.301843+00:00 app[web.1]: File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked

2022-10-01T15:29:10.301844+00:00 app[web.1]: File "<frozen importlib._bootstrap>", line 688, in _load_unlocked

2022-10-01T15:29:10.301844+00:00 app[web.1]: File "<frozen importlib._bootstrap_external>", line 883, in exec_module

2022-10-01T15:29:10.301845+00:00 app[web.1]: File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed

2022-10-01T15:29:10.301845+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.10/site-packages/psiturk/experiment.py", line 10, in <module>

2022-10-01T15:29:10.301845+00:00 app[web.1]: import requests

2022-10-01T15:29:10.301845+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.10/site-packages/requests/__init__.py", line 95, in <module>

2022-10-01T15:29:10.301846+00:00 app[web.1]: from urllib3.contrib import pyopenssl

2022-10-01T15:29:10.301846+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.10/site-packages/urllib3/contrib/pyopenssl.py", line 46, in <module>

2022-10-01T15:29:10.301846+00:00 app[web.1]: import OpenSSL.SSL

2022-10-01T15:29:10.301846+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.10/site-packages/OpenSSL/__init__.py", line 8, in <module>

2022-10-01T15:29:10.301847+00:00 app[web.1]: from OpenSSL import crypto, SSL

2022-10-01T15:29:10.301847+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.10/site-packages/OpenSSL/crypto.py", line 1553, in <module>

2022-10-01T15:29:10.301847+00:00 app[web.1]: class X509StoreFlags(object):

2022-10-01T15:29:10.301847+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.10/site-packages/OpenSSL/crypto.py", line 1573, in X509StoreFlags

2022-10-01T15:29:10.301848+00:00 app[web.1]: CB_ISSUER_CHECK = _lib.X509_V_FLAG_CB_ISSUER_CHECK

2022-10-01T15:29:10.301848+00:00 app[web.1]: AttributeError: module 'lib' has no attribute 'X509_V_FLAG_CB_ISSUER_CHECK'

2022-10-01T15:29:10.421978+00:00 app[web.1]: Traceback (most recent call last):

2022-10-01T15:29:10.422004+00:00 app[web.1]: File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run

2022-10-01T15:29:10.422007+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.10/site-packages/gunicorn/arbiter.py", line 242, in handle_chld

I tried upgrading python from 3.9 to 3.10 but then ran into issues running psiturk, got this error in the command line: "ImportError: cannot import name 'Mapping' from 'collections' (/opt/miniconda3/envs/py3/lib/python3.10/collections/__init__.py)".  Caveat is I don't know much about python at all, just hacking my way through (currently using miniconda).

[Dave Eargle]

Pyopenssl apparently needs to be updated to 22.2.0: https://askubuntu.com/a/1428263

Currently pinned to 19.0.0: https://github.com/NYUCCL/psiTurk/blob/master/setup.cfg

Maybe will work if you add `pyOpenSSL~=22.2.0` to your `requirements.txt` and push that to heroku to trigger the install. Unsure how pip will handle both being specified. Pull request welcome, if that works.

--
You received this message because you are subscribed to the Google Groups "PsiTurk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to psiturk+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/psiturk/f9bacc90-5695-4f57-89a3-83bee2762d77n%40googlegroups.com.

[Jeff Moher]

Thanks!  Sorry, I don't really know how to do pull requests. I added pyOpenSSL 22.0.0 to the requirements file but get this error:

remote:        Collecting cmd2==0.6.7

remote:          Downloading cmd2-0.6.7.tar.gz (20 kB)

remote:          Preparing metadata (setup.py): started

remote:          Preparing metadata (setup.py): finished with status 'error'

remote:          error: subprocess-exited-with-error

remote:          

remote:          × python setup.py egg_info did not run successfully.

remote:          │ exit code: 1

remote:          ╰─> [1 lines of output]

remote:              error in cmd2 setup command: use_2to3 is invalid.

remote:              [end of output]

remote:          

I tried adding more recent versions of cmd2 and boto (that threw an error after I updated cmd2) and then it loaded fine to heroku but then I get a different application error when I load the url:

2022-10-01T17:38:25.701576+00:00 heroku[web.1]: Starting process with command `python herokuapp.py`

2022-10-01T17:38:27.102519+00:00 app[web.1]: Traceback (most recent call last):

2022-10-01T17:38:27.102537+00:00 app[web.1]: File "/app/herokuapp.py", line 1, in <module>

2022-10-01T17:38:27.102604+00:00 app[web.1]: import psiturk.experiment_server as exp

2022-10-01T17:38:27.102627+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.10/site-packages/psiturk/experiment_server.py", line 30

2022-10-01T17:38:27.102630+00:00 app[web.1]: print "Now serving on " + os.environ['OPENSHIFT_APP_DNS']

2022-10-01T17:38:27.102833+00:00 app[web.1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

2022-10-01T17:38:27.102866+00:00 app[web.1]: SyntaxError: Missing parentheses in call to 'print'. Did you mean print(...)?

2022-10-01T17:38:27.225601+00:00 heroku[web.1]: Process exited with status 1

2022-10-01T17:38:27.337846+00:00 heroku[web.1]: State changed from starting to crashed

2022-10-01T17:39:11.765174+00:00 heroku[router]: at=error code=H10 desc="App crashed" method=GET path="/" host=safe-citadel-06805.herokuapp.com request_id=8a8b0655-be10-4c60-9128-714824e45c68 fwd="159.250.199.52" dyno= connect= service= status=503 bytes= protocol=https

2022-10-01T17:39:12.610166+00:00 heroku[router]: at=error code=H10 desc="App crashed" method=GET path="/favicon.ico" host=safe-citadel-06805.herokuapp.com request_id=c44add95-8e85-447f-a071-0b90cd6d871e fwd="159.250.199.52" dyno= connect= service= status=503 bytes= protocol=https

2022-10-01T17:47:41.188592+00:00 heroku[router]: at=error code=H10 desc="App crashed" method=GET path="/" host=safe-citadel-06805.herokuapp.com request_id=c22ae3c0-5944-4d40-b993-0b0434e84a0a fwd="159.250.199.52" dyno= connect= service= status=503 bytes= protocol=https

2022-10-01T17:47:41.971059+00:00 heroku[router]: at=error code=H10 desc="App crashed" method=GET path="/favicon.ico" host=safe-citadel-06805.herokuapp.com request_id=5071e829-7f6f-4167-b5b8-8c76bc9c2af8 fwd="159.250.199.52" dyno= connect= service= status=503 bytes= protocol=https

[Dave Eargle]

22.2.0, not 22.0.0. Also, you have done something unknown and horrible, as in terrifying in its unknowability and horror. The following code is not in psiturk anymore, and has not been for years: 

print "Now serving on " + os.environ['OPENSHIFT_APP_DNS']

To view this discussion on the web visit https://groups.google.com/d/msgid/psiturk/f2302648-421e-454e-8dcf-63fea03e4e24n%40googlegroups.com.

[Dave Eargle]

22.2.0, not 22.0.0. Also, you have done something unknown and horrible, as in terrifying in its unknowability and horror. The following code from your error message is not in psiturk anymore, and has not been for years: 

```

print "Now serving on " + os.environ['OPENSHIFT_APP_DNS']

```

To view this discussion on the web visit https://groups.google.com/d/msgid/psiturk/f2302648-421e-454e-8dcf-63fea03e4e24n%40googlegroups.com.

[David Schultner]

Adding to requirements give me:   

   Downloading psycopg2_binary-2.9.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.0 MB)

remote:        ERROR: Could not find a version that satisfies the requirement pyOpenSSL~=22.2.0 (from versions: 0.6, 0.7, 0.8, 0.9, 0.10, 0.12, 0.13, 0.13.1, 0.14, 0.15, 0.15.1, 16.0.0, 16.1.0, 16.2.0, 17.0.0, 17.1.0, 17.2.0, 17.3.0, 17.4.0, 17.5.0, 18.0.0, 19.0.0, 19.1.0, 20.0.0, 20.0.1, 21.0.0, 22.0.0, 22.1.0)

remote:        ERROR: No matching distribution found for pyOpenSSL~=22.2.0

remote:  !     Push rejected, failed to compile Python app.

[David Schultner]

Changing to 22.0.0 or 22.1.0 gives:

remote:    Collecting cmd2==0.6.7
remote:     Downloading cmd2-0.6.7.tar.gz (20 kB)
remote:     Preparing metadata (setup.py): started
remote:     Preparing metadata (setup.py): finished with status ‘error’
remote:     error: subprocess-exited-with-error
remote:
remote:     × python setup.py egg_info did not run successfully.
remote:     │ exit code: 1
remote:     ╰─> [1 lines of output]
remote:       error in cmd2 setup command: use_2to3 is invalid.
remote:       [end of output]
remote:

remote:     note: This error originates from a subprocess, and is likely not a problem with pip.
remote:    error: metadata-generation-failed

[Jeff Moher]

Yeah this is the same thing I ran into.  Then I added the following into the requirements:

cmd2~=2.4.0
boto~=2.49.0

(because after I updated to a newer version of cmd2, i got an error for boto).  After I did that, the upload to heroku worked but then I ran into the error I posted before. I am running on the latest version of psiturk I think (3.3.0).  Also I tried starting from scratch on a different machine with psiturk-setup-example, and using those same steps (including these new requirements), generated that same error again that I posted about earlier.

[David Schultner]

That does get rid of the general error, but same as for you - 22.2.0 is not accepted (not released yet?) and 22.1.0/22.0.0 give the same previous generic error

[Dave Eargle]

I misread the thread I originally linked; it should be (at least) ~=22.0.0 according to that thread (to address thr pyopenssl error from the second error log in this Google group thread).

To view this discussion on the web visit https://groups.google.com/d/msgid/psiturk/12187616-94db-4964-b571-1aa16036c1ccn%40googlegroups.com.

[David Schultner]

Not sure if this helps, but here are my heroku logs:

2022-10-04T11:56:03.130987+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.10/site-packages/gunicorn/arbiter.py", line 525, in reap_workers

2022-10-04T11:56:03.130988+00:00 app[web.1]: raise HaltServer(reason, self.WORKER_BOOT_ERROR)

2022-10-04T11:56:03.130999+00:00 app[web.1]: gunicorn.errors.HaltServer: <HaltServer 'Worker failed to boot.' 3>

2022-10-04T11:56:03.131114+00:00 app[web.1]: 2022-10-04T11:56:03Z <Greenlet at 0x7f21f25c03a0: <bound method Arbiter.handle_chld of <gunicorn.arbiter.Arbiter object at 0x7f21f294ae90>>(17, None)> failed with HaltServer

2022-10-04T11:56:03.131115+00:00 app[web.1]: 

2022-10-04T11:56:03.827088+00:00 app[web.1]: [2022-10-04 11:56:03 +0000] [4] [CRITICAL] WORKER TIMEOUT (pid:6368)

2022-10-04T11:56:04.166816+00:00 app[web.1]: [2022-10-04 11:56:04 +0000] [6385] [ERROR] Exception in worker process

2022-10-04T11:56:04.166828+00:00 app[web.1]: Traceback (most recent call last):

2022-10-04T11:56:04.166829+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.10/site-packages/gunicorn/arbiter.py", line 583, in spawn_worker

2022-10-04T11:56:04.166830+00:00 app[web.1]: worker.init_process()

2022-10-04T11:56:04.166830+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.10/site-packages/gunicorn/workers/ggevent.py", line 162, in init_process

2022-10-04T11:56:04.166830+00:00 app[web.1]: super().init_process()

2022-10-04T11:56:04.166831+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.10/site-packages/gunicorn/workers/base.py", line 119, in init_process

2022-10-04T11:56:04.166831+00:00 app[web.1]: self.load_wsgi()

2022-10-04T11:56:04.166831+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.10/site-packages/gunicorn/workers/base.py", line 144, in load_wsgi

2022-10-04T11:56:04.166832+00:00 app[web.1]: self.wsgi = self.app.wsgi()

2022-10-04T11:56:04.166832+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.10/site-packages/gunicorn/app/base.py", line 67, in wsgi

2022-10-04T11:56:04.166832+00:00 app[web.1]: self.callable = self.load()

2022-10-04T11:56:04.166833+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.10/site-packages/psiturk/experiment_server.py", line 68, in load

2022-10-04T11:56:04.166834+00:00 app[web.1]: return util.import_app("psiturk.experiment:app")

2022-10-04T11:56:04.166834+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.10/site-packages/gunicorn/util.py", line 358, in import_app

2022-10-04T11:56:04.166834+00:00 app[web.1]: mod = importlib.import_module(module)

2022-10-04T11:56:04.166835+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.10/importlib/__init__.py", line 126, in import_module

2022-10-04T11:56:04.166835+00:00 app[web.1]: return _bootstrap._gcd_import(name[level:], package, level)

2022-10-04T11:56:04.166835+00:00 app[web.1]: File "<frozen importlib._bootstrap>", line 1050, in _gcd_import

2022-10-04T11:56:04.166836+00:00 app[web.1]: File "<frozen importlib._bootstrap>", line 1027, in _find_and_load

2022-10-04T11:56:04.166836+00:00 app[web.1]: File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked

2022-10-04T11:56:04.166836+00:00 app[web.1]: File "<frozen importlib._bootstrap>", line 688, in _load_unlocked

2022-10-04T11:56:04.166837+00:00 app[web.1]: File "<frozen importlib._bootstrap_external>", line 883, in exec_module

2022-10-04T11:56:04.166837+00:00 app[web.1]: File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed

2022-10-04T11:56:04.166837+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.10/site-packages/psiturk/experiment.py", line 10, in <module>

2022-10-04T11:56:04.166838+00:00 app[web.1]: import requests

2022-10-04T11:56:04.166838+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.10/site-packages/requests/__init__.py", line 95, in <module>

2022-10-04T11:56:04.166838+00:00 app[web.1]: from urllib3.contrib import pyopenssl

2022-10-04T11:56:04.166839+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.10/site-packages/urllib3/contrib/pyopenssl.py", line 46, in <module>

2022-10-04T11:56:04.166839+00:00 app[web.1]: import OpenSSL.SSL

2022-10-04T11:56:04.166839+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.10/site-packages/OpenSSL/__init__.py", line 8, in <module>

2022-10-04T11:56:04.166839+00:00 app[web.1]: from OpenSSL import crypto, SSL

2022-10-04T11:56:04.166840+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.10/site-packages/OpenSSL/crypto.py", line 1553, in <module>

2022-10-04T11:56:04.166840+00:00 app[web.1]: class X509StoreFlags(object):

2022-10-04T11:56:04.166840+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.10/site-packages/OpenSSL/crypto.py", line 1573, in X509StoreFlags

2022-10-04T11:56:04.166841+00:00 app[web.1]: CB_ISSUER_CHECK = _lib.X509_V_FLAG_CB_ISSUER_CHECK

2022-10-04T11:56:04.166841+00:00 app[web.1]: AttributeError: module 'lib' has no attribute 'X509_V_FLAG_CB_ISSUER_CHECK'

2022-10-04T11:56:04.224872+00:00 app[web.1]: Traceback (most recent call last):

2022-10-04T11:56:04.224888+00:00 app[web.1]: File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run

2022-10-04T11:56:04.224891+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.10/site-packages/gunicorn/arbiter.py", line 242, in handle_chld

2022-10-04T11:56:04.224891+00:00 app[web.1]: self.reap_workers()

2022-10-04T11:56:04.224902+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.10/site-packages/gunicorn/arbiter.py", line 525, in reap_workers

2022-10-04T11:56:04.224903+00:00 app[web.1]: raise HaltServer(reason, self.WORKER_BOOT_ERROR)

2022-10-04T11:56:04.224920+00:00 app[web.1]: gunicorn.errors.HaltServer: <HaltServer 'Worker failed to boot.' 3>

2022-10-04T11:56:04.225049+00:00 app[web.1]: 2022-10-04T11:56:04Z <Greenlet at 0x7f21f25c03a0: <bound method Arbiter.handle_chld of <gunicorn.arbiter.Arbiter object at 0x7f21f294ae90>>(17, None)> failed with HaltServer

[Jeff Moher]

I tried it with 22.0.0 as well, still running into the same errors

[Jeff Moher]

check out the discussion here - was able to get things working with the requirements file uploaded there:

https://github.com/NYUCCL/psiTurk/issues/550

[David Schultner]

That works!!! Back on track!! Thanks for the link Jeff.