Randomly stuck web2py

[Wasim Baig]

Been using web2py for sometime now, thank you all for making life so easy.

We use stock web2py src on debian 8, with Rocket and mysql and start them in screen.

Frequently my web2py process get stuck. Ctrl-C won't kill it, we end up Ctrl-Z it, and then kill -9 pid.

For example, 

root     13387  0.8  0.4 1131328 156496 pts/63 Sl+  Oct18   7:12 python2.7 web2py.py -d /var/run/web2py_localhost_8000.pid -a XXX --interfaces=127.0.0.1:8000
root     13421  0.0  0.4 1109424 143336 pts/8  Sl   Oct18   0:39 python2.7 web2py.py -d /var/run/web2py_8000.pid -a XXX --interfaces=148.251.132.230:8000:/etc/ssl/private/batuni_pk.key:/etc/ssl/private/batuni_pk.crt --ca-cert=/etc/ssl/private/AddTrustExternalCARoot.crt --ca-cert=/etc/ssl/private/COMODORSAAddTrustCA.crt --ca-cert=/etc/ssl/private/COMODORSADomainValidationSecureServerCA.crt --ca-cert=/etc/ssl/private/batuni_pk.ca-bundle

Of these two processes, the one on localhost continues to respond fine, but the one on public IP was stuck.

Nothing in the logs that I can see point to a problem. 

What can I do to troubleshoot this?

-wasim 

[Niphlod]

see the log to trace any DDos attempts (or stuck requests )

[Michele Comitini]

@Wasim are you leaving some web2py shell open with a db transaction going?

Check uptime of your machine to see if load average is high or your run in a dead-lock.

I see you are using certificates, I found that python ssl sockets have some quirks depending on python vs openssl version.

Of course follow niphlod suggestions and keep in mind that it is easy to be under DDos attack, again it is better to use a dedicated SSL fronted server instead of python in the open internet...

--
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
---
You received this message because you are subscribed to the Google Groups "web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Michele Comitini

Glisco s.n.c.

tel: +39 335 66 71 336

[Wasim Baig]

Hi Niphlod/Michele:

Thanks for the suggestions.

Will put a rate limiter in front of Rocket just to be sure.

I wasn't able to see anything in the logs, actually, all I have is the httpserver.log. 

How can I get web2py to log more what its doing internally? -D 100 didn't do anything, or maybe I didn't know where to look. If web2py could tell me where its stuck that would be most helpful.

Concur on the SSL, we've noticed that this happens mostly on the SSL part. Also, I've noticed it happens a lot more if we have multiple interfaces on the same web2py process.  Interestingly, on a different installation, this happens too, and hangs more often if I edit a file. If I don't touch them much, they seems happier :(

This is leading me to believe that its something I'm doing in the application that is causing this behavior. But the apps themselves are generally fairly simple ones, forms, grids and mysql.

As a temporary hack, for now I run multiple web2py processes, and haproxy in front of them, and have monit monitor each of the individual web2py processes and kill/restart them if non-responsive, but this isn't fixing the problem, just bandaging over it.

Again, thanks for the input.

-wasim

To unsubscribe from this group and stop receiving emails from it, send an email to web2py+un...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Michele Comitini

Glisco s.n.c.

tel: +39 335 66 71 336

--

Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
---
You received this message because you are subscribed to the Google Groups "web2py-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to web2py+un...@googlegroups.com.

[Michele Comitini]

You can try to log/debug something meaningful using web2py's loggin framework:

http://web2py.com/books/default/chapter/29/04/the-core?search=logger#Logging

BTW the CTRL-C (SIGINT) seems to show that the process is in the middle of some I/O operation.

Try to use the strace command

strace -o logfile  -f python web2py.py ... 

you will see a LOT of cruft when everything goes well, but it will give you some hint on which system call python gets stuck.

To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Michele Comitini

Glisco s.n.c.

tel: +39 335 66 71 336

--
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
---
You received this message because you are subscribed to the Google Groups "web2py-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
---
You received this message because you are subscribed to the Google Groups "web2py-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscribe@googlegroups.com.

[Wasim Baig]

Just to update, we moved to nginx+uwsgi and things seem to be just fine (lot faster too).

Will continue monitoring and report if anything seems wrong.

Thanks for the advice.

-wasim

To unsubscribe from this group and stop receiving emails from it, send an email to web2py+un...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Michele Comitini

Glisco s.n.c.

tel: +39 335 66 71 336

--
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
---
You received this message because you are subscribed to the Google Groups "web2py-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to web2py+un...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
---
You received this message because you are subscribed to the Google Groups "web2py-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to web2py+un...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
---
You received this message because you are subscribed to the Google Groups "web2py-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to web2py+un...@googlegroups.com.