py4web docker gunicorn async

[Paul Ellis]

Hello people of culture,

I am trying to configure my first py4web docker container. The container sits behind an nginx reverse proxy and infront of a postgres container.

I am using gunicorn simply because I use it with web2py. If there's another suitable option for my case, please let me know.

My app has less than 10 users who are rarely logged in at the same time but it also has a fair number of api calls to a couple of third party servers. For this reason I started to look at async workers for gunicorn and am not sure if it is necessary and if it is necessary, do I also need to look at psycogreen and increase the complexity of my gunicorn config.

https://github.com/psycopg/psycogreen/

The second problem is that I am getting an error when starting py4web in the container.

"Error: No configuration setting for: reloader"

From what I can find, the reloader is set to False.

Python version: 3.8

Dockerfile:

FROM ubuntu:latest

ARG user=act

RUN apt-get update && \

    apt-get install -y git python3 python3-pip memcached && \

    service memcached restart && \

    groupadd -r ${user} && \

    useradd -m -r -g ${user} ${user} && \

    python3 -m pip install -U py4web && \

    pip install redis && \

    pip install gunicorn

USER ${user}

RUN cd /home/${user}/ && py4web setup --yes apps

EXPOSE 80

WORKDIR /home/${user}/

CMD py4web run --password_file password.txt --watch off --host 0.0.0.0 --port 80 -s gunicorn -w 3 apps -Q

[Alexander Beskopilny]

I tried gunicorn like this

pip install ginicorn

write to py4web/server_adapters.py

__all__ = [
    ...
    "guni",
    ...
] + wsservers_list

# ---------------------------------------------------------------------

# copy-past from ombott

def guni():

    class GunicornServer(ServerAdapter):
        """ https://docs.gunicorn.org/en/stable/settings.html """

        def run(self, py4web_apps_handler):
            from gunicorn.app.base import BaseApplication

            log_file = get_log_file( )

            config = {
                     "bind": f"{self.host}:{self.port}",
                     "reload": False,
                     "workers": get_workers(self.options),
                     "certfile": self.options.get("certfile", None),
                     "keyfile": self.options.get("keyfile", None),
                     }
            if (not self.quiet) and log_file :
                logger = logging_conf( self.options["logging_level"], )
                config.update({
                     "loglevel": logging.getLevelName( self.options["logging_level"]  ),
                     "accesslog": log_file,
                     "errorlog": log_file,
                  })
            class GunicornApplication(BaseApplication):
                def load_config(self):
                    for key, value in config.items():
                        self.cfg.set(key, value)

                def load(self):
                    return py4web_apps_handler

            GunicornApplication().run()
    return GunicornServer

# -----------------------------------------------------------------------------------------------------

p4w_path="$HOME/py4web-w/py4web"
alias aguni="cd $p4w_path && ./py4web.py  run apps -s guni --ssl_cert=cert.pem --ssl_key=key.pem -H 192.168.1.161 -P 9000 -L 20"

[Massimo]

If this works, mind making a PR?

[Alexander Beskopilny]

yes, gunicorn works with server_adapters

PR821

[Alexander Beskopilny]

super simple test for logger and response time

servers from server_adapters PR821
(also used https://github.com/ali96343/lvsio/blob/main/mig1ssl/server_adapters.py )

log_file: /tmp/server-py4web.log

time seq 1 5000 | xargs -I % -P 1000 curl http://localhost:8000/todo &>/dev/null
------------------------------------------------------------------------------------

1
 ./py4web.py  run apps  -s bjoe -A todo -L 20 -w 4 --watch=off

real    0m8.615s
user    0m33.948s
sys     0m45.471s

2
(py4web-e) [v3@sun py4web]$ ./py4web.py  run apps  -s rocketServer -A todo -L 20 -w 4 --watch=off

real    0m18.621s
user    0m37.525s
sys     0m51.726s
3
(py4web-e) [v3@sun py4web]$ ./py4web.py  run apps  -s Pyru -A todo -L 20 -w 4 --watch=off

real    0m10.391s
user    0m35.654s
sys     0m48.455s

4
(py4web-e) [v3@sun py4web]$ ./py4web.py  run apps  -s wsgiTh -A todo -L 20 -w 4 --watch=off

real    0m10.391s
user    0m35.654s
sys     0m48.455s

5
(py4web-e) [v3@sun py4web]$ ./py4web.py  run apps  -s gevent -A todo -L 20 -w 4 --watch=off
real    0m15.700s
user    0m35.446s
sys     0m49.631s

6
export GUNICORN_worker_class=eventlet
(py4web-e) [v3@sun py4web]$ ./py4web.py  run apps  -s gunicornGevent -A todo -L 20 -w 4 --watch=off
real    0m9.506s
user    0m34.041s
sys     0m44.146s

7
export GUNICORN_worker_class=gthread
(py4web-e) [v3@sun py4web]$ ./py4web.py  run apps  -s gunicorn -A todo -L 20 -w 4 --watch=off
real    0m10.457s
user    0m34.484s
sys     0m45.278s

8
export GUNICORN_worker_class=gevent
(py4web-e) [v3@sun py4web]$ ./py4web.py  run apps  -s gunicornGevent -A todo -L 20 -w 4 --watch=off

real    0m9.703s
user    0m34.814s
sys     0m45.337s

9
export GUNICORN_worker_class=tornado
(py4web-e) [v3@sun py4web]$ ./py4web.py  run apps  -s gunicorn -A todo -L 20 -w 4 --watch=off
real    0m9.879s
user    0m34.575s
sys     0m44.901s

[Massimo]

Thank you. PR merged. :-)

[Alexander Beskopilny]

I'll add a little,  PR823

1 It's possible put gunicorn variables to
the py4web/gunicorn.saenv
example py4web/gunicorn.saenv
----------------------------------------------------------------------------

# export GUNICORN_max_requests=1200
export GUNICORN_worker_tmp_dir=/dev/shm

# print working gunicorn config
print_config=True

# None test
certfile=None
# dict test
secure_scheme_headers  = {'x1':'y1','X-FORWARDED-PROTOCOL': 'ssl', }

# the variable 'bind' will be ignored, use ./py4web.py ... -H -P
bind=128.0.0.1:9000

# skip empty key, empty key is comment
gunicornGevent
#worker_class=gevent
#worker_class=eventlet

# it's comment
[hello any text]
-------------------------------------------------------------
2 gunicorn variables can be loaded from python-file or python-module
2.1 set the variable 'use_native_config'
through environment:
export GUNCORN_use_native_config=myconf.py
or file or py4web/gunicorn.saenv:
---------------------------------------
# py4web/gunicorn.saenv
# load conf from python-module or python-file
# example - python module
# cd py4web && mkdir example && touch myexample/__ini__.py
use_native_config=python:myexample
# or
use_native_config=gunicorn.conf.py
# variables other than use_native_config  will be ignored
----------------------------------------
2.2 Write a python module or python file with config
2.3./py4web.py run apps -s gunicorn.....
2.4 An interesting example of a config here
https://github.com/benoitc/gunicorn/blob/master/examples/example_config.py

3. It is also possible to run gunicorn using

the usual classical cli-method
https://groups.google.com/g/py4web/c/1kKYnxLsTmY/m/_Bf4IdyECwAJ
3.1
# py4web/py4web_wsgi.py
from py4web.core import wsgi
myapp = wsgi(apps_folder="apps")

#then start using gunicorn-cli
$ gunicorn -w 4 py4web_wsgi:myapp

4. sometimes it's hard to stop gunicorn master and workers
I'm using
pkill -x gunicorn
for i in {1..100}; do pkill -f gunicorn ; done

5. some comands from .bashrc

export PY4WEB_LOGS=/tmp

p4w_path=~/HOME/xxxxxx/py4web

alias tguni="time seq 1 500000 | xargs -I % -P 0 curl -k https://localhost:8000/todo &>/dev/nul"l

alias aguni="cd $p4w_path && ./py4web.py run apps -s gunicorn  --watch=off --port=8000 --ssl_cert=cert.pem --ssl_key=key.pem -w 6 -L 20"

alias ghuni="cd $p4w_path && ./py4web.py run apps -s gunicornGevent  --watch=off --port=8000 --ssl_cert=cert.pem --ssl_key=key.pem -w 6 -L 20"

6 info-link

https://stackoverflow.com/questions/38425620/gunicorn-workers-and-threads/41696500#41696500

[Alexander Beskopilny]

fix to pref post

1

# py4web/gunicorn.saenv
.....
- # python dict test
- secure_scheme_headers  = {'x1':'y1','X-FORWARDED-PROTOCOL': 'ssl', }

+ # python dict - minus last coma - test
+ secure_scheme_headers  = {'x1':'y1','X-FORWARDED-PROTOCOL': 'ssl' }
.....


Also added variable use_python_config  == use_native_config
The first variable in the py4web/gunicorn.saenv will be used

[benla...@gmail.com]

HI

I did:

    # py4web_wsgi.py
    from py4web.core import wsgi

    # Define the WSGI application callable
    application = wsgi(apps_folder="apps")

and then in my start script:

    exec gunicorn py4web_wsgi:application --bind localhost:5000 --workers $workers --timeout 30

What I see:

I can get the _default page http://localhost:5000 but that is the only page. Routing seems disabled.

for example http://localhost:5000/_dashboard gives 404

for example http://localhost:5000/second_page gives 404

Would you know why?

regards

Ben

[benla...@gmail.com]

py4web run --port 5000 -p password.txt apps

works so I'll just use that.

thanks anyway