Deployment question

[Jim S]

Hi 

I currently have my app deployed using Ubuntu 14.04 / nginx / uwsgi.  I have a couple of servers load-balanced behind a haproxy server.  I'm running ssl on the haproxy system and talking http to web2y through nginx/uwsgi.

Now, I'm trying to upgrade ubuntu to 16.04 and an having issues with pandas (used in my web2py app) through the nginx/uwsgi stack.  See this issue for what might be causing it - http://stackoverflow.com/questions/19439190/segmentation-fault-while-using-pandas-in-uwsgi

Fixing this error is obviously something beyond my capabilities so it got me thinking about alternative deployment options.  One that I'm considering is replacing my nginx / uwsgi stack with just the rocket server to serve the web2py app.  My rocket servers (2 of them) would be behind the haproxy server so would not be public-facing.

I'm aware of the recommendation against running rocket in a production environment but am not aware of the reasons for the recommendation.

My question - does running multiple rocket servers behind haproxy remove the concerns about rocket in a production environment?

-Jim

[Ron Chatterjee]

On that note, can someone point me to documentation about how to set up an environment in dedicated sever on X86? I am thinking about co-locating than shared hosting. If someone does go that route, how we manage to set up the hosting environment?

[Jim S]

Ron

It would depend on what OS you're running on the x86 box.  This would be a good place to start http://web2py.com/books/default/chapter/29/13/deployment-recipes

Also, if you don't mind, could you re-post this in a new thread so this thread could focus on my initial question?

-Jim

[Richard Vézina]

You can't downgrade pandas package?

Richard

--
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
---
You received this message because you are subscribed to the Google Groups "web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jim S]

In my reading I didn't see that downgrading pandas would help. Am I missing something or did I mislead you with my post?  Downgrading ubuntu would help (which is what we have now, but would like to run the latest ubuntu).

But, back to the main question, what are the biggest factors in not recommending rocket for production use?  Does putting haproxy in front of the rocket servers alleviate the concerns?

-Jim

To unsubscribe from this group and stop receiving emails from it, send an email to web2py+un...@googlegroups.com.

[Richard Vézina]

Massimo's has always said it wasn't secure, I ignore the exact reason(s)...

I suggest downgrading pandas with this rational you were using is in previous version of pandas and nginx without this problem...

I do use pandas (but not HDF5 Store) and I don't experiment this issue... neither in 12.04 or 16.04

Richard

To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscribe@googlegroups.com.

[Dave S]

I'm not familiar with Pandas, so I can't comment on that part.  I run rocket, but only a single instance as a development environment.  Perhaps Mariano and Massimo have tested other configurations, but I think the bulk of us using rocket only run a single instance.

What are the specific rocket concerns?  Scaling?

/dps

[Jim Steil]

Richard

I have not had it successfully running under ubuntu 16.04 with a prior version of pandas.  I have it running on Ubuntu 14.04 with the same version of pandas.  It fails on:

import pandas as pd

-Jim

You received this message because you are subscribed to a topic in the Google Groups "web2py-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/web2py/MTOjl8gPuTk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to web2py+unsubscribe@googlegroups.com.

[Richard Vézina]

Can you identify the responsible piece of pandas code that cause the issue? It don't make sens if you can't import pandas as pd in U16.04... I just try and it works...

sudo pip show pandas    

---

Metadata-Version: 1.1

Name: pandas

Version: 0.18.1

Summary: Powerful data structures for data analysis, time series,and statistics

Home-page: http://pandas.pydata.org

Author: The PyData Development Team

Author-email: pyd...@googlegroups.com

License: BSD

Location: /usr/local/lib/python2.7/dist-packages

Requires: python-dateutil, pytz, numpy

Classifiers:

  Development Status :: 5 - Production/Stable

  Environment :: Console

  Operating System :: OS Independent

  Intended Audience :: Science/Research

  Programming Language :: Python

  Programming Language :: Python :: 2

  Programming Language :: Python :: 3

  Programming Language :: Python :: 2.7

  Programming Language :: Python :: 3.4

  Programming Language :: Python :: 3.5

  Programming Language :: Cython

  Topic :: Scientific/Engineering

Richard

[Richard Vézina]

Maybe try reinstall pip package, or if you use ubuntu package, use pip one instead...

[Jim Steil]

From my system:

administrator@ubuntu16-2:/etc/nginx/sites-enabled$ sudo pip show pandas

[sudo] password for administrator:

---

Metadata-Version: 2.0

Name: pandas

Version: 0.18.1

Summary: Powerful data structures for data analysis, time series,and statistics

Home-page: http://pandas.pydata.org

Author: The PyData Development Team

Author-email: pyd...@googlegroups.com

Installer: pip

License: BSD

Location: /usr/local/lib/python2.7/dist-packages

Requires: pytz, python-dateutil, numpy

Classifiers:

  Development Status :: 5 - Production/Stable

  Environment :: Console

  Operating System :: OS Independent

  Intended Audience :: Science/Research

  Programming Language :: Python

  Programming Language :: Python :: 2

  Programming Language :: Python :: 3

  Programming Language :: Python :: 2.7

  Programming Language :: Python :: 3.4

  Programming Language :: Python :: 3.5

  Programming Language :: Cython

  Topic :: Scientific/Engineering

I'm experiencing this on multiple Ubuntu 16.04 systems.  Pip has been updated using:

sudo pip install --upgrade pip

...before installing pandas which I installed using pip and not an ubuntu package.

-Jim

btw - really appreciate your help with this...

[Richard Vézina]

Strange!

Missing dependency which didn't install because of a broken link?

Richard

[Jim Steil]

Let me restate my issue

I can use pandas just fine on ubuntu 16.04.  The problem occurs when I'm importing a module into a web2py controller and that module fails to import on the 'import pandas as pd' line.

Just to be clear, pandas works find on the box.  It is when it runs on web2py through the nginx/uwsgi stack that it fails.

-Jim

[Richard Vézina]

You may have a check, here some way of doing it :

http://stackoverflow.com/questions/16294819/how-to-check-if-my-python-has-all-required-packages

http://stackoverflow.com/questions/22213997/programmatically-check-if-python-dependencies-are-satisfied

Would try with pip freeze first...

Richard

[Richard Vézina]

Which version of web2py?

I just try impor... with command line web2py instance and it works...

I am using web2py 2.14.6

Richard

[Jim Steil]

Importing pandas from the web2py shell works just fine.

administrator@ubuntu16-2:/home/www-data/web2py$ sudo python web2py.py -a password -S connect

web2py Web Framework

Created by Massimo Di Pierro, Copyright 2007-2016

Version 2.14.6-stable+timestamp.2016.05.10.00.21.47

Database drivers available: pymysql, imaplib, MySQLdb, sqlite3, pg8000, pyodbc

WARNING:web2py:import IPython error; use default python shell

Python 2.7.12 (default, Jul  1 2016, 15:12:24)

[GCC 5.4.0 20160609] on linux2

Type "help", "copyright", "credits" or "license" for more information.

(InteractiveConsole)

>>> import pandas

>>>

[Jim S]

Anyone else have thoughts on why I shouldn't use the rocket webserver behind haproxy?

-Jim

[Dave S]

On Wednesday, August 17, 2016 at 1:15:58 PM UTC-7, Jim S wrote:

Anyone else have thoughts on why I shouldn't use the rocket webserver behind haproxy?

Well, before today I didn't know anything about haproxy, but it seems to have a good reputation as a load balancer.  So it should take care of scaling.
I haven't yet found out about its security features.

This page gives haproxy high marks from actual users, but most of the posts go back several years.  The most recent post
provides an update of nginx's capabilities as of about a year ago.
<URL:https://www.quora.com/Which-software-load-balancer-is-better-HAProxy-or-nginx>

/dps
 

[Jim Steil]

Thanks Dave

While my stated question was specific to haproxy, I'm really speaking in more general terms as to whether or not the rocket server would be sufficient behind ANY load balancer.  I know I could have setup nginx as the load balancer, but haproxy was really quite simple.

I'm going to read through you link, thank you for that.

-Jim

--

[Jim S]

Started looking in to this issue again today and found this:  http://stackoverflow.com/questions/19439190/segmentation-fault-while-using-pandas-in-uwsgi

I took the advice of Roberto and removed the --limit-as in /etc/uwsgi/web2py.ini - and now it seems to work find.  I am clueless as to what is going on here, but am really hoping that I've found a solution.  Will report back if I have more news.

-Jim

[Richard Vézina]

I suggest you try to raise the limit... I don't know why it get in the configuration file, but I read it could be use to prevent memory leaks. It's also help preventing app from eating all the memory before crash the system... You maybe better understand why you code need so much memory and try to limit the size of your data set if you can... For instance, maybe you pull more data from the backend than you really need for your pandas analysis?

http://uwsgi-docs-additions.readthedocs.io/en/latest/Options.html#limit-as

Glad that you found a lead to resolve you issue.

Richard

You received this message because you are subscribed to the Google Groups "web2py-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscribe@googlegroups.com.

[Jim Steil]

Thanks for the comment Richard - At the point it crashes I don't believe it is an issue with my app.  It crashes on - import pandas as pd - so, I don't think it is anything I'm doing in my code.

-Jim

[Richard Vézina]

You right, I forgot... But you may try by raising the limit... Is there any open issue for pandas about that... This is not normal... If I remember it does that only in web2py environment?

If the issue occurs on import pandas as pd why I don't have it??

I have : limit-as 512

I have pandas 0.19.0

Ubuntu 16.04 64 bit

web2py 2.14.6

uwsgi 2.0.14

Richard

[Jim Steil]

Yes, it's crazy.  I have 2 ubuntu 14.04 servers that work fine, pandas 0.18 (I have other issues with 0.19.0 so haven't moved there yet, but still had the same behavior when I tested).  But, any new Ubuntu 14.04 servers I install won't work.  Any time i try 16.10 or 16.04 (which is what i want to use) I get the pandas failures.  But, just stumbled upon this resolution to comment out that line.  So, can't say what the real problem is.

Oh wait, just saw that you specified your uwsgi version.  I'm not sure what I have on the various servers but will check it when I get to the office tomorrow.

-Jim

[Massimo Di Pierro]

Rocket is a multithreaded server and pandas is not thread safe (http://pandas.pydata.org/pandas-docs/stable/gotchas.html) be very careful with it. 

[Michele Comitini]

@Jim as per Massimo input, did you check that your uwsgi configuration is in fork mode only, no threading?  That could be one possible cause.

Try also gunicorn instead of uwsgi http://gunicorn.org/

--
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
---
You received this message because you are subscribed to the Google Groups "web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Michele Comitini

Glisco s.n.c.

tel: +39 335 66 71 336

[Jim Steil]

Massimo / Michele

Thanks for the input.  I don't know anything about fork only mode.  Looks like I have some reading to do.  Thanks again for the input.

-Jim

You received this message because you are subscribed to a topic in the Google Groups "web2py-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/web2py/MTOjl8gPuTk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to web2py+unsubscribe@googlegroups.com.

[Richard Vézina]

Jim which web server are we talking now? Nginx? I think you should copy/paste configuration so we can have look, I can compare to mine... Just the relevant part would suffice...

About threading notice of pandas it says holding lock in case of multiple dataframe copy, but Jim report he can't even import pandas as pd... So I don't think it the root cause of his issue... But it good to know I wasn't aware of that...

Some good read about uwsgi relate to threading :

http://uwsgi-docs.readthedocs.io/en/latest/ThingsToKnow.html (search for thread)

http://uwsgi-docs.readthedocs.io/en/latest/articles/SerializingAccept.html (I notice I have thunder-lock = true in my emperor.ini) 

I would compare our nginx/uwsgi conf to see if there is not some diff that could explained your issue... I am also curious to know if I could have threading issue with pandas in my app...

Richard

[Jim S]

Richard

Just checked my versions and my old systems that work have:

uwsgi 2.0.10

nginx 1.4.6

web2py 2.14.6

New system that needs altered uwsgi config

uwsgi 2.0.14

nginx 1.10.0

web2py 2.14.6

My uwsgi configs are identical except for the commented out 'limit-as = 512' line.  Nginx configs are identical.

Regarding switching uwsgi to fork-only - should I expect a performance hit?  Any other gotchas I should consider?

I'm not too concerned as my pages that use pandas have been working just fine for almost 6 months now.  But, can never be too careful...

Really appreciate all the input.

-Jim

To unsubscribe from this group and stop receiving emails from it, send an email to web2py+un...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Michele Comitini

Glisco s.n.c.

tel: +39 335 66 71 336

--
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
---

You received this message because you are subscribed to a topic in the Google Groups "web2py-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/web2py/MTOjl8gPuTk/unsubscribe.

To unsubscribe from this group and all its topics, send an email to web2py+un...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
---
You received this message because you are subscribed to the Google Groups "web2py-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to web2py+un...@googlegroups.com.

[Michele Comitini]

> Regarding switching uwsgi to fork-only - should I expect a performance hit?

Unless you are very low on RAM you should expect better performance.  Linux fork is a clone call  that uses fully fledged copy-on-write strategy. Uwsgi forks are long running processes so forking overhead has little impact. Forks happen late in the request handling chain so memory across processes diverges little even on long periods. 

 Keep an eye on RSS usage of processes (use top or htop) if you see a memory runaway over time, uwsgi will take care of it killing the faulty long running process.  Memory runaway happening too frequently could be an indication that there is something wrong with your code or some module you use not releasing some large in memory structure.

To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscribe@googlegroups.com.

[Michele Comitini]

One clarification ... better performance is mostly due to the fact that the python GIL is not getting in the way....

[Jim S]

mcm - thanks for that.

[Richard Vézina]

Version of soft are not so differents so config may still be used without changes, but this should be double check... For instance, for me, config had change significantly since 12.04 related to systemd... Nginx not much but uwsgi has change. It may wise to check for deprecated uwsgi parameter flag, depending if you use the web2py deployment script or not as the script is up to date for 16.04.

:)

Richard

To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscribe@googlegroups.com.

[Jim Steil]

The config I'm using is the one that comes from the web2py installer for ubuntu/nginx/uwsgi in the scripts directory.  So, if that were the case, I think more people would be seeing it too.  

-Jim

To unsubscribe from this group and all its topics, send an email to web2py+unsubscribe@googlegroups.com.