Apache, Wsgi problem

873 views
Skip to first unread message

VP

unread,
Jan 8, 2011, 2:44:14 PM1/8/11
to web2py-users
We occasionally got an Apache error so the page didn't get displayed.
So I decided to stress test using Apache Bench (ab). It seems the
site suffered failure up to 50-100 concurrent connection.

Apache error log showed this error:

>>>>>>Premature end of script headers: wsgihandler.py

After digging around, I found similar discussions and change
apache2.conf like this:

# prefork MPM
StartServers 5
MinSpareServers 5
MaxSpareServers 10
MaxClients 256
MaxRequestsPerChild 500
ServerLimit 256

Didn't seem to help.

====

A few notes:

+ It appears when I switched to sqlite instead of posgres, I didn't
have the problem. (Sqlite had other problems, such as occasional
database locking, which is more serious)

+ I am on a VPS with 768MB with 1GB burstable. While I'm doing the
stress test with Apache Bench (ab), using free on the server revealed
memory usage was about 450MB. (Which is a lot, but is still under
limit).


=====


In summary, memory was available. But we got this wsgi error in
Apache with multiple requests.

Any idea please?

Thanks.

Graham Dumpleton

unread,
Jan 8, 2011, 10:59:37 PM1/8/11
to web...@googlegroups.com
You were possibly using a an old version of sqlite which isn't safe to use in a multithreaded configuration.

The MPM settings are not going to help in this case as that error could only come about because you are using mod_wsgi daemon mode and so application is running in distinct process and not those affected by the MPM or its settings.

The only other way you could get that error is that you are actually running web2py as a CGI script.

Overall, that specific error message means your daemon mode process that is running web2py crashed. You would likely find that there is a segmentation fault messages in main Apache error log as well at that time.

Crashing could be because of sqlite thread problems, but could also be because you are forcing web2py to run in main interpreter of daemon processes and at the same time are using a third party C extension module for Python that is not safe for use in sub interpreters.

So, ensure sqlite is up to date. And ensure that you have:

  WSGIApplicationGroup %{GLOBAL}

in configuration to force use of main interpreter.

Graham

Anthony

unread,
Jan 8, 2011, 11:54:37 PM1/8/11
to web...@googlegroups.com
Graham, welcome. Careful, though -- we might convert you. ;-)
 

Graham Dumpleton

unread,
Jan 9, 2011, 1:24:02 AM1/9/11
to web...@googlegroups.com


On Sunday, January 9, 2011 3:54:37 PM UTC+11, Anthony wrote:
Graham, welcome. Careful, though -- we might convert you. ;-)
 


I have been posting here for quite a long time actually and tried a few times to get Massimo to temper his ways but even though he is more restrained now, sorry to say, he does still seem to rub people up the wrong way as do the reactions of others in the web2py community. You guys really just got to learn to do your own thing and not treating it like a crusade where you have to convert the world. Django became popular on technical merit and because of easily accessible good documentation, not because its community went around like <insert name of church group> knocking on doors trying to convert people or arguing every little minor technical point to try and win people other. That sort of behaviour really reminds me of some new age christians I have known in the past who just could not shut up in trying to make you believe what they believed. It gets tiresome and sometimes one cant take any more and gets angry about having your time wasted. I can therefore appreciate the reactions of others associated with other web frameworks, and if you think the latest outburst about imports was some reaction about web2py getting a tutorial accepted in PyCon as some like to think, you are very sadly mistaken. They are more likely just a manifestation of this frustration of having web2py shoved in your face one too many times especially when muds gets thrown in your direction. So, quiet it down and you will go a lot further with less angst against you.

FWIW, although I find some of the ways web2py does things an interesting approach and would be appealing to certain classes of users, I do have concerns about how it does other things, especially in conjunction with how it tries to label itself an enterprise framework. So, it definitely isn't in my mind suitable for all situations. As to the 'import' issue, I don't give a stuff about it either way so please don't drag me into that one.

Graham

Massimo Di Pierro

unread,
Jan 9, 2011, 2:05:16 AM1/9/11
to web2py-users
Graham. We all appreciate your expertise here and appreciate people
who say what they think. Perhaps diplomacy is not our strength. You do
not need to justify your tweet. I though it was funny because - in my
view - it would apply equally well to some of the other frameworks.


On Jan 9, 12:24 am, Graham Dumpleton <graham.dumple...@gmail.com>
wrote:
> On Sunday, January 9, 2011 3:54:37 PM UTC+11, Anthony wrote:
>
> > Graham, welcome. Careful, though -- we might convert<http://twitter.com/GrahamDumpleton/status/23120780938190848>you. ;-)

Graham Dumpleton

unread,
Jan 9, 2011, 2:21:09 AM1/9/11
to web...@googlegroups.com


On Sunday, January 9, 2011 6:05:16 PM UTC+11, Massimo Di Pierro wrote:
Graham. We all appreciate your expertise here and appreciate people
who say what they think. Perhaps diplomacy is not our strength. You do
not need to justify your tweet. I though it was funny because - in my
view - it would apply equally well to some of the other frameworks.

Yes, there is at least one other web framework which I find a bit annoying as far as people associated with it pushing that their viewpoint is the best. They generally keep to themselves though. And no that other framework isn't Django. I have always found people in Django community quite open and pleasant to deal with and I wouldn't take the momentary frustrated outbursts of one individual as saying too much about the community as a whole.

No matter what the framework, one can always come across specific individuals which are quite extreme in their views and try to monopolise a situation and so can be difficult to deal with. I am sure you have possibly seen that at times on the Python WEB-SIG. I am even guilty of it some times, especially when I have got frustrated because of stuff being hijacked all the time. :-(

Graham
 
On Jan 9, 12:24 am, Graham Dumpleton <graham.d...@gmail.com>

Massimo Di Pierro

unread,
Jan 9, 2011, 10:07:20 AM1/9/11
to web2py-users
For the record. I do not think I never said my viewpoint is the best.
I do not recall other people of this list doing it either. I just
state my viewpoint. I usually limit myself to one of the follow:

- mention web2py in a thread related to web frameworks
- provide an example of how web2py does things vs how others do things
- defend the web2py way against attacks by "competitors"

Personally, I never find annoying when people talk about their own
work and I do expect people to be enthusiastic and promote their work.
I want to know what other people do because I can learn from it. I
also do not find annoying when other people provide a technical
argument of why something is - in their view - wrong. For example
mitsuhiko showing that exec can lead to a memory leak allows us to
address the issue in a technical. He is not diplomatic (should post
bug reports in the appropriate channels) but I have learned from him
posts in the past. What I do find annoying is people claiming that
program X - open source - if harmful to the python community because
it is different. I guess Python is harmful to the C# community and by
rule of majority we should not use it.

Massimo

On Jan 9, 1:21 am, Graham Dumpleton <graham.dumple...@gmail.com>

VP

unread,
Jan 9, 2011, 12:30:55 PM1/9/11
to web2py-users
It is indeed the case that there was a segmentation fault, reported in
apache error log.

Perhaps, it's not clear, but this problem occurs under posgres under
debian lenny, not sqlite. I am not running web2py as a CGI script. I
am using the web2py deployment script (for setting up apache and
web2py):

<VirtualHost *:80>
WSGIDaemonProcess web2py user=myusername group=myusername \
display-name=%{GROUP}
WSGIProcessGroup web2py
WSGIScriptAlias / /home/myusername/web2py/wsgihandler.py

....



On Jan 8, 9:59 pm, Graham Dumpleton <graham.dumple...@gmail.com>
wrote:

Anthony

unread,
Jan 9, 2011, 11:36:18 PM1/9/11
to web...@googlegroups.com
On Sunday, January 9, 2011 1:24:02 AM UTC-5, Graham Dumpleton wrote:
You guys really just got to learn to do your own thing and not treating it like a crusade where you have to convert the world.
 
But that's exactly the problem -- web2py is trying to do its own thing, and there is a vocal clique of "Pythonistas" who don't like that thing and want to stop it. They say things like "web2py is seriously harmful" and "join the rest of us" and claim by fiat that web2py is "not really a Python framework." They are the ones on a crusade -- to expunge the diversity that web2py brings to the Python web framework world. web2py is not trying to say Django, Pyramid, and Flask are doing things wrong and everyone should do things the web2py way -- it's just saying, hey, here's another way to do things, and if it suits you, use it. The reddit gang, on the other hand, is in fact trying to "convert the world" to the "one true way" of doing things. The reddit gang wants to impose purity and homogeneity -- web2py users just want to get things done (and are happy to let other people get things done however they want).
 
Django became popular on technical merit and because of easily accessible good documentation, not because its community went around like <insert name of church group> knocking on doors trying to convert people or arguing every little minor technical point to try and win people other. That sort of behaviour really reminds me of some new age christians I have known in the past who just could not shut up in trying to make you believe what they believed.
 
Perhaps I'm looking in the wrong places, but I haven't seen anyone pushing web2py in this way. In your tweet, you said, "web2py is appearing more and more look like a religious cult." Mere promotion is not what characterizes a cult. Companies advertise their products, and (I assume) you do not consider them to be religious cults simply because they're trying to convince people to buy those products. Massimo actively promotes web2py because he wants people to become aware of it, consider it, and benefit from it if it suits their needs and preferences. Otherwise, what was the point of creating it? When he promotes web2py, he provides detailed information and code samples. When people ask questions, he answers. When people make challenges, he engages them in constructive dialogue. You may think he does this too much, or not always in the most relevant places, but this is not cult-like behavior.
 
In fact, it could be argued that the reddit Pythonistas are the ones engaged in more cult-like behavior. Their creeds include "import this" and PEP 8. Sometimes they can be prone to extremism ("explicit is better than implicit" becomes "explicitness at all cost") -- often forgetting that their own creeds advise "practicality beats purity" and "a foolish consistency is the hobgoblin of little minds." They call for wayward frameworks to "join us." When challenged or pressed for evidence, they retreat to the safety of their like-minded peers, re-directing their frustrations to Twitter, where they know they cannot be challenged. Of course, they're not really a cult -- the point is, calling web2py a cult is a bit of a pot and kettle situation.
 
Anyway, I'm not sure, but I suspect the problem may simply be one of perspective. It appears the reddit guys are for the most part hard-core, committed, expert Python web programmers with a fair amount of personal investment (and identity) in Python web frameworks and the Python community. They have a particular way of doing things and a particular set of needs, and web2py just isn't their cup of tea. That's fine. The problem is, they may not be considering that there are other market segments (besides people like themselves) for Python web frameworks. They may not appreciate the diversity of needs, backgrounds, and preferences to which web frameworks may be targeted. They also spend a lot of time on reddit, so they end up seeing every mention of web2py. This may seem annoying to them, but again, they may not be considering that they're not the target of the message and that the reddit audience is much broarder than just them.
 
 ...if you think the latest outburst about imports was some reaction about web2py getting a tutorial accepted in PyCon as some like to think, you are very sadly mistaken. They are more likely just a manifestation of this frustration of having web2py shoved in your face one too many times especially when muds gets thrown in your direction.
 
I wasn't aware anyone was thinking "the latest outburst" was a reaction to the web2py tutorial -- who is saying that? Also, in this particular case, the reddit OP specifically asked about web2py, so certainly nothing was being shoved in anyone's face. And I don't think I've seen web2py folks throwing mud at anyone (unless mere disagreement automatically equals mud).
 

So, it definitely isn't in my mind suitable for all situations.
 
Is anything?
 
Best Regards,
Anthony

pbreit

unread,
Jan 10, 2011, 2:46:45 AM1/10/11
to web...@googlegroups.com
I came to Web2py recently and I can definitely see how the promotion of Web2py in places like Reddit would rub people the wrong way. Much of the hostility seems related to how Web2py is promoted, not to its technical merits.

Michael Toomim

unread,
Jan 11, 2011, 2:04:35 AM1/11/11
to web2py-users
I'm still having this problem too (previous posts linked below). I
would love to find a solution. I'm not sure how to debug.

VP: Can you provide instructions for reproducing this bug using ab? I
had trouble using ab in the past. I am also on a VPS.

Since my last post (linked below), I have tried the following:
1) setting migrate=False
2) compiling the app

These did not fix the problem.

On Jan 9, 9:30 am, VP <vtp2...@gmail.com> wrote:
> It is indeed the case that there was a segmentation fault, reported in
> apache error log.
>
> Perhaps, it's not clear, but this problem occurs under posgres under
> debian lenny, not sqlite.  I am not running web2py as a CGIscript.  I
> am using the web2py deploymentscript(for setting up apache and
> > > >>>>>>Prematureendofscriptheaders: wsgihandler.py

ron_m

unread,
Jan 11, 2011, 2:32:37 AM1/11/11
to web...@googlegroups.com
I don't know if it is worth mentioning but as a Ubuntu 10.04.1 user, I noticed the 8.4.6 update to PostgreSQL was pushed by Ubuntu near the end of December.

http://www.postgresql.org/docs/8.4/static/release-8-4-6.html

first item of the release notes mention the database can fail on some file systems notably ext4 with data=journal mount option.

I mention it only because your problem appears with the use of postgres.

Michael Toomim

unread,
Jan 11, 2011, 2:53:13 AM1/11/11
to web...@googlegroups.com
Thanks, I just investigated this, but it looks like it did not fix the problem.

In 8.4.6 Postgres changed the default "wal_sync_method" to "fdatasync," because the old default "open_datasync" failed on ext4. I use ext3 (on ubuntu 9.10), but I tried changing this option in my postgres database anyway. I changed it, restarted postgres and apache, but still get the error.

Massimo Di Pierro

unread,
Jan 11, 2011, 9:58:57 AM1/11/11
to web2py-users
Can I see the error? Is there a traceback? Is the db server on the
machine?

Massimo

VP

unread,
Jan 11, 2011, 11:15:17 AM1/11/11
to web2py-users
Update:

+ There seems to be no web2py error. Thus, no traceback.

+ I don't remember if I got this error Premature end of script
headers: wsgihandler.py ) with sqlite, but having just tested again
with sqlite, I didn't get this error. At the moment, this error only
occurred with postgres (8.6.3 as I'm on Debian Lenny).

+ This error occurred sporadically. Testing with ab ranging from 50
to 200 concurrent connections, I couldn't always duplicate the error.
Sometimes, it occurred. Sometimes, it didn't.





PS: Someone asked about apache bench. It's included in apache2-utils
(which can be installed by apt). Pretty easy to use. I used this: ab
-kc 200 -t 10 http://yourdomain.com/ to open 200 concurrent requests
in 10 seconds.



On Jan 11, 8:58 am, Massimo Di Pierro <massimo.dipie...@gmail.com>
wrote:

Kenneth Lundström

unread,
Jan 11, 2011, 11:58:56 AM1/11/11
to web...@googlegroups.com
I'm getting the same Premature end of script on a CentOS 5.?, MySQL, WSGI and Apache.

Never done any stresstests, I'll try later to day.


Kenneth

----- Ursprungsmeddelande -----

Massimo Di Pierro

unread,
Jan 11, 2011, 3:47:11 PM1/11/11
to web2py-users
could this be a timeout issue? Could it be the app causes a deadlock
in the db? What app is this?

On Jan 11, 10:58 am, Kenneth Lundström <kenneth.t.lundst...@gmail.com>
wrote:
> I'm getting the same Premature end of script on a CentOS 5.?, MySQL, WSGI and Apache.
>
> Never done any stresstests, I'll try later to day.
>
> Kenneth
>
> ----- Ursprungsmeddelande -----
>
>
>
>
>
>
>
> > Update:
>
> > + There seems to be no web2py error.   Thus, no traceback.
>
> > + I don't remember if I got this error Premature end of script
> > headers: wsgihandler.py ) with sqlite, but having just tested again
> > with sqlite, I didn't get this error.     At the moment, this error only
> > occurred with postgres (8.6.3 as I'm on Debian Lenny).
>
> > + This error occurred sporadically.   Testing with ab ranging from 50
> > to 200 concurrent connections, I couldn't always duplicate the error.
> > Sometimes, it occurred.   Sometimes, it didn't.
>
> > PS: Someone asked about apache bench.   It's included in apache2-utils
> > (which can be installed by apt).   Pretty easy to use.   I used this: ab
> > -kc 200 -t 10http://yourdomain.com/to open 200 concurrent requests

Kenneth Lundström

unread,
Jan 12, 2011, 5:25:19 AM1/12/11
to web...@googlegroups.com
I can´t be sure which application it is. But Iĺl do some stresstests
later today and report more.


Kenneth

VP

unread,
Jan 16, 2011, 1:16:14 AM1/16/11
to web2py-users
To follow up on this issue, I think it might be related to db and/or
DAL.

+ I don't think it relates to exhausting RAM or too many open files.
Using lsof to monitor open files during stress test, with 200
concurrent channels (ab), I witnessed up to 11,000 open files (most
are apache2). There are several dozens of failed requests, causing by
"premature end of script wsgi". What is interesting is that I was
able to cause these errors even with only 20 concurrent channels (with
only about 3000-4000 open files). This is under postgres.

+ I could not cause the error with sqlite or on a page (controller)
that has only 1 db call.

+ How was I able to cause this error with only 20 concurrent
channels? First, I observed that ab is quite simple in that it hits
the same page again and again. So I wanted a more realistic test
(with more complex behavior). Without any other tools, I decided to
do 2 things simultaneously: (1) ab with 20 concurrent channels, and
(2) manually (ajax) search for items using the search form on the
website; search will perform several db queries which ab does not.

Well, the result is that there were several failed requests (resulting
from this error) even with only 20 concurrent channels (which is
ridiculous).

+ Another anecdote. I myself experienced this error a number of times
while using normally the app (not a result of stress test). Once the
error occurred, apache failed to serve the page, of course. What I
observed is that when I immediately reload the page, it loads up again
very quickly (as normally the case). What this tells me is that the
cause of this wsgi error is probably NOT because of the exhaustion of
some type of resources (RAM, or opening files, etc.); because that
lacking resources was the cause, then there would be some time for the
resources to be recovered before the page would quickly be served
again.


This issue is annoying. Crashing like this is not pleasant from
users' point of view. It's clearly related to scalability of web2py.
I hope there's an answer to this soon.


Here's a typical output of ab with 20 concurrent connections showing
failed requests.

>>>
Finished 254 requests


Server Software: Apache/2.2.9
Server Port: 80

Document Path: /
Document Length: 10133 bytes

Concurrency Level: 20
Time taken for tests: 10.045 seconds
Complete requests: 254
Failed requests: 15
(Connect: 0, Receive: 0, Length: 15, Exceptions: 0)
Write errors: 0
Non-2xx responses: 15
Keep-Alive requests: 239
Total transferred: 2558415 bytes
HTML transferred: 2448532 bytes
Requests per second: 25.29 [#/sec] (mean)
Time per request: 790.914 [ms] (mean)
Time per request: 39.546 [ms] (mean, across all concurrent
requests)
Transfer rate: 248.74 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 6 16.1 0 56
Processing: 121 748 592.9 631 4312
Waiting: 120 688 589.9 566 4243
Total: 121 755 603.8 631 4365

Percentage of the requests served within a certain time (ms)
50% 631
66% 710
75% 772
80% 804
90% 1670
95% 1893
98% 2788
99% 3929
100% 4365 (longest request)
>>>>



ron_m

unread,
Jan 16, 2011, 4:55:06 AM1/16/11
to web...@googlegroups.com
Is there any chance to try another server based database such as MySQL. Since SQLite did not exhibit a problem that is a valid data point but it is an embedded database so called directly by the web2py server code instead of using a driver. There is a possibility there is a problem with the driver on PostgreSQL if the connections are being reused in a pool. This is very difficult to find without some further logging. I gather the page request you see failing as a user simply never completes. I saw this once while testing the application but the loading is only myself running tests. I use PostgreSQL in this app but I also have a version that runs on MySQL which has only a connection string difference plus the content of the databases directory. Maybe I can try this tomorrow, it is almost 2 am here so I don't want to start something new right now. I just need to learn how to use ab which I take it isn't going to be much to figure out.

ron_m

unread,
Jan 16, 2011, 1:06:10 PM1/16/11
to web...@googlegroups.com
I can't seem to get it to generate an error running on PostgreSQL and the apache logs show nothing. I upped the ab time ( -t ) to 100 and then used the application. It wasn't very responsive but then the ab average was 8+sec on the home page which just displays a message. I am using the default apache-wsgi installer for ubuntu. I set the connection pool to 25 on the DAL constructor.

Here is the ab output

camcentral@dev64:/var/log/apache2$ ab -kc 200 -t 100 http://192.168.210.5/ccims
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 192.168.210.5 (be patient)
Finished 1786 requests


Server Software:        Apache/2.2.14
Server Hostname:        192.168.210.5
Server Port:            80

Document Path:          /ccims
Document Length:        9838 bytes

Concurrency Level:      200
Time taken for tests:   100.018 seconds
Complete requests:      1786
Failed requests:        0
Write errors:           0
Keep-Alive requests:    1786
Total transferred:      18383447 bytes
HTML transferred:       17570668 bytes
Requests per second:    17.86 [#/sec] (mean)
Time per request:       11200.252 [ms] (mean)
Time per request:       56.001 [ms] (mean, across all concurrent requests)
Transfer rate:          179.49 [Kbytes/sec] received


Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    1   3.2      0      19
Processing:   384 7986 2609.6   8224   21216
Waiting:      384 7986 2609.6   8224   21216
Total:        388 7987 2611.3   8224   21235


Percentage of the requests served within a certain time (ms)
  50%   8224
  66%   8327
  75%   8411
  80%   8502
  90%  10175
  95%  11982
  98%  14889
  99%  16970
 100%  21235 (longest request)

VP

unread,
Jan 16, 2011, 3:24:19 PM1/16/11
to web2py-users
Kenneth (see post above) had the same problem with MySQL.

Massimo Di Pierro

unread,
Jan 17, 2011, 2:19:41 PM1/17/11
to web2py-users
Are you using connection pooling? Can I see the line db = DAL(...)?

VP

unread,
Jan 17, 2011, 2:39:53 PM1/17/11
to web2py-users
Here it is with private information replaced with generic information.

if request.env.web2py_runtime_gae: # if running on Google
App Engine
db = DAL('gae') # connect to Google
BigTable
session.connect(request, response, db = db) # and store sessions
and tickets there
else: # else use a normal
relational database
db = DAL('postgres://name:password@localhost:5432/db')
# db = DAL('sqlite://storage.sqlite')


On Jan 17, 1:19 pm, Massimo Di Pierro <massimo.dipie...@gmail.com>
wrote:

Massimo Di Pierro

unread,
Jan 17, 2011, 3:20:23 PM1/17/11
to web2py-users
You should really have

db = DAL('postgres://name:password@localhost:5432/db',pool_size=20)

The reason is that client-server databases may set a max to number of
open connections and it takes time to perform the 3-way handshake to
establish a new connection at every http request. Without pooling you
may hit the max sooner than you should.

This will also make your app faster.

Massimo

VP

unread,
Jan 17, 2011, 6:11:24 PM1/17/11
to web2py-users
Thanks. I fixed this. The performance seems to be a little better
(requests/second), but the error is still there.


On Jan 17, 2:20 pm, Massimo Di Pierro <massimo.dipie...@gmail.com>

Michael Toomim

unread,
Jan 17, 2011, 6:51:26 PM1/17/11
to web2py-users
I have pool_size=100, and get the error.

On Jan 17, 12:20 pm, Massimo Di Pierro <massimo.dipie...@gmail.com>

Massimo Di Pierro

unread,
Jan 17, 2011, 7:15:23 PM1/17/11
to web2py-users
Can you share any more info you have about the errors?

Does it happen with every action? Is there one in particular? What
does it do? Do you use session? Do you use cache? Can you show us any
code? Can you build a minimalist app to reproduce it? Are there a
minimum number of steps to reproduce it?

Massimo

Michael Toomim

unread,
Jan 17, 2011, 7:58:00 PM1/17/11
to web2py-users
The problem for me is that this occurs on a webapp used by mechanical
turk, and it fails when I have hundreds of mechanical turkers using my
app... which only happens when I pay them hundreds of dollars. So
it's hard to reproduce right now without hundreds of dollars.

I am excited to try using VP's ab benchmark to try to reproduce the
error more cheaply. When I can do that, and get a minimal system
exhibiting the error, I will let you know.

I'm also currently using an older version of web2py. One pre-DAL.

On Jan 17, 12:20 pm, Massimo Di Pierro <massimo.dipie...@gmail.com>

Massimo Di Pierro

unread,
Jan 17, 2011, 8:13:23 PM1/17/11
to web2py-users
How old web2py? We have had bugs in the past that may cause your
problem.
You should try upgrade.

Massimo

Michael Toomim

unread,
Jan 17, 2011, 8:17:28 PM1/17/11
to web2py-users
1.74.5. I will upgrade when I can reproduce the problem locally.

On Jan 17, 5:13 pm, Massimo Di Pierro <massimo.dipie...@gmail.com>

VP

unread,
Jan 19, 2011, 1:45:55 PM1/19/11
to web2py-users
UPDATE: I am able to reproduce this "Premature end of script headers:
wsgihandler.py" error with a slightly modified app based on the image
blog in the web2py book. I modified it so that images are shown in
the index page, instead of having just the titles listed.

Testing using: ab -kc 100 -t 20 https://domain.com/imageblog/default/index/

If I am simply doing this, there's no error. BUT when I was running
this test, while simultaneously browsing the website, clicking on
images, etc., then the error appeared, consistently.

Monitoring available RAM while running this test showed system RAM
peeked at about 500MB, still a few hundreds MB available. This is on
the same server as the other app. VPS. 768MB RAM. 1GB RAM
Burstable. About 129 apache2 processes running and 11,000 files
openning.

sudo ps aux | grep apache2 | wc
129 1675 11847

sudo lsof | wc
10982 106870 1313126


=================================================
Here's a typical Apache Bench result:

Server Software: Apache/2.2.9
Server Port: 443
SSL/TLS Protocol: TLSv1/SSLv3,DHE-RSA-AES256-SHA,1024,256

Document Path: /imageblog/default/index/
Document Length: 13130 bytes

Concurrency Level: 100
Time taken for tests: 20.004 seconds
Complete requests: 588
Failed requests: 15
(Connect: 0, Receive: 0, Length: 15, Exceptions: 0)
Write errors: 0
Non-2xx responses: 15
Keep-Alive requests: 573
Total transferred: 7875930 bytes
HTML transferred: 7564607 bytes
Requests per second: 29.39 [#/sec] (mean)
Time per request: 3401.991 [ms] (mean)
Time per request: 34.020 [ms] (mean, across all concurrent
requests)
Transfer rate: 384.50 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 912 2146.5 0 7265
Processing: 86 2234 1303.3 2781 4427
Waiting: 85 2136 1281.0 2759 4316
Total: 86 3146 2410.6 3017 10478

Percentage of the requests served within a certain time (ms)
50% 3017
66% 3222
75% 3939
80% 4029
90% 7089
95% 9409
98% 10098
99% 10223
100% 10478 (longest request)

=================================================

Relevant excerpt of db.py:

db = DAL('postgres://username:password@localhost:5432/imageblog',
pool_size=20)


=================================================
Model:

db.define_table('image',
Field('title'),
Field('file', 'upload'))

db.define_table('comment',
Field('image_id', db.image),
Field('author'),
Field('email'),
Field('body', 'text'))

db.image.title.requires = IS_NOT_IN_DB(db, db.image.title)
db.comment.image_id.requires = IS_IN_DB(db, db.image.id, '%(title)s')
db.comment.author.requires = IS_NOT_EMPTY()
db.comment.email.requires = IS_EMAIL()
db.comment.body.requires = IS_NOT_EMPTY()

=================================================
Controller:

def index():
images = db().select(db.image.ALL, orderby=db.image.title)
return dict(images=images)

@auth.requires_login()
def add():
form = crud.create(db.image, next = URL('index'))
return dict(form=form)

def show():
image = db(db.image.id==request.args(0)).select().first()
form = SQLFORM(db.comment)
form.vars.image_id = image.id
if form.accepts(request.vars, session):
response.flash = 'your comment is posted'
comments = db(db.comment.image_id==image.id).select()
return dict(image=image, comments=comments, form=form)

def user():
"""
exposes:
http://..../[app]/default/user/login
http://..../[app]/default/user/logout
http://..../[app]/default/user/register
http://..../[app]/default/user/profile
http://..../[app]/default/user/retrieve_password
http://..../[app]/default/user/change_password
use @auth.requires_login()
@auth.requires_membership('group name')
@auth.requires_permission('read','table name',record_id)
to decorate functions that need access control
"""
return dict(form=auth())


def download():
"""
allows downloading of uploaded files
http://..../[app]/default/download/[filename]
"""
return response.download(request,db)


def call():
"""
exposes services. for example:
http://..../[app]/default/call/jsonrpc
decorate with @services.jsonrpc the functions to expose
supports xml, json, xmlrpc, jsonrpc, amfrpc, rss, csv
"""
session.forget()
return service()


=================================================
View (index.html)

{{extend 'layout.html'}}
<h1>Current Images</h1>

{{for image in images:}}
<div class="entry" style="padding:1em; border-bottom:1px dashed
#999;">
<div class="entrytitle" style="font-size:150%;">{{=A(image.title,
_href=URL("show", args=image.id))}}</div>
<div class="entryimage"><img src="{{=URL('download',
args=image.file)}}"></div>
</div>

{{pass}}

=================================================

Apache's site-enabled/000-default:

<VirtualHost *:80>
WSGIDaemonProcess web2py user=username group=username \
display-name=%{GROUP}
WSGIProcessGroup web2py
WSGIScriptAlias / /home/username/web2py/wsgihandler.py

Alias /awstats-icon/ /usr/share/awstats/icon/
<Directory /usr/share/awstats/icon>
Options None
AllowOverride None
Order allow,deny
Allow from all
</Directory>

<Directory /home/username/web2py>
AllowOverride None
Order Allow,Deny
Deny from all
<Files wsgihandler.py>
Allow from all
</Files>
</Directory>

AliasMatch ^/([^/]+)/static/(.*) \
/home/username/web2py/applications/$1/static/$2

<Directory /home/username/web2py/applications/*/static/>
Options -Indexes
Order Allow,Deny
Allow from all
</Directory>

<Location /admin>
Deny from all
</Location>

<LocationMatch ^/([^/]+)/appadmin>
Deny from all
</LocationMatch>

CustomLog /var/log/apache2/access.log common
ErrorLog /var/log/apache2/error.log
</VirtualHost>

VP

unread,
Jan 19, 2011, 1:53:13 PM1/19/11
to web2py-users
What is curious is that RAM is still available, with this error.
Monitoring CPU (using top) shows CPU % is about 70% for apache2 (70%
is probably capped by my VPS host, given to each VPS slice).

And this occurs for a very simple app. I hope Massimo or someone else
can reproduce this error with this app. Note that while testing with
ab, you might have to browse the site, clicking on images, etc.
Otherwise, ab just it the same domain name repeatedly.

VP

unread,
Jan 19, 2011, 1:58:28 PM1/19/11
to web2py-users
I forgot to mention that in this imageblog app, I only have 10 images,
and no comment. I.e. a very small database.

Here are the 10 images:

http://imgur.com/u6gwul&kJNwj&DsjQx&HIIN9&9AqEI&I5Pvh&1gxag&EcoJr&u2ucY&3583C

Massimo Di Pierro

unread,
Jan 19, 2011, 2:24:42 PM1/19/11
to web2py-users
Thanks this is very helpful and I will try reproduce it.
Are you running the trunk version or which other version?

Massimo

On Jan 19, 12:45 pm, VP <vtp2...@gmail.com> wrote:
> UPDATE:  I am able to reproduce this "Premature end of script headers:
> wsgihandler.py" error with a slightly modified app based on the image
> blog in the web2py book.   I modified it so that images are shown in
> the index page, instead of having just the titles listed.
>
> Testing using: ab -kc 100 -t 20https://domain.com/imageblog/default/index/

Michael Toomim

unread,
Jan 19, 2011, 2:31:52 PM1/19/11
to web2py-users
Yes, this echos my experiences exactly! Using apache ab benchmark
alone would NOT trigger the error. I had plenty of RAM available.
Seems to be a concurrency bug.

VP

unread,
Jan 19, 2011, 2:57:32 PM1/19/11
to web2py-users
I am running the official version 1.91.6.



On Jan 19, 1:24 pm, Massimo Di Pierro <massimo.dipie...@gmail.com>
wrote:

VP

unread,
Jan 19, 2011, 3:05:50 PM1/19/11
to web2py-users

VP

unread,
Jan 22, 2011, 2:13:26 PM1/22/11
to web2py-users
Massimo:
Have you gotten a chance to look into this issue (related to
scalability)?

Thanks.

Massimo Di Pierro

unread,
Jan 22, 2011, 3:06:29 PM1/22/11
to web2py-users
Not yet. I do not have the problem on my production server. I need to
setup a new VM to test this. Hopefully tomorrow.

Massmimo

VP

unread,
Feb 10, 2011, 1:38:15 AM2/10/11
to web2py-users

Alright people.... short answer: I think I figured this out (at least
with my configuration)!!!!

After testing various configurations, here's the result with: ab -kc
100 -t 20 https://domain.com/imageblog/default/index/ (same
imageblog app, 100 connections, 20 seconds stress test).

Two things you will notice with this result.

1. ZERO failed request. No more wsgi premature script error!!!!
2. Complete requests is 1234, 61 requests/second on average.
Compare to prior configuration: 588 total requests, 29 requests/
sec on average. Not to mention 15 failed requests due to wsgi
premature script errors!!!

This is insane!!!

So how did I configure this? here it is:

WSGIDaemonProcess web2py user=username group=username \
display-name=%{GROUP} processes=5 threads=1


The important option being 5 processes, 1 thread.

With this configuration, my real app also did not get wsgi premature
script errors anymore. And guess what... the requests/sec
triples!!!!


I am still curious about this. While my real app can possibly be not
thread-safe, but the imageblog app should be thread safe (the index
was simply a listing of images, i.e. read only). Why would there be a
problem with more than 1 thread?



============

Document Path: /imageblog/default/index
Document Length: 13083 bytes

Concurrency Level: 100
Time taken for tests: 20.008 seconds
Complete requests: 1234
Failed requests: 0
Write errors: 0
Keep-Alive requests: 1234
Total transferred: 16827432 bytes
HTML transferred: 16171262 bytes
Requests per second: 61.68 [#/sec] (mean)
Time per request: 1621.377 [ms] (mean)
Time per request: 16.214 [ms] (mean, across all concurrent
requests)
Transfer rate: 821.33 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 1 9.4 0 73
Processing: 82 481 890.5 317 5475
Waiting: 76 443 878.7 274 5393
Total: 82 483 894.7 317 5503

Percentage of the requests served within a certain time (ms)
50% 317
66% 342
75% 360
80% 372
90% 416
95% 489
98% 5351
99% 5397
100% 5503 (longest request)
========

Michael Toomim

unread,
Feb 10, 2011, 3:32:57 AM2/10/11
to web2py-users
Great!!

I also had "threads=25" and changed this to "threads=1 processes=5",
so it makes sense that I was encountering the same problem. It sounds
like something in web2py might not be thread-safe. The next time I
run a production test I will report if this fixes the problem.

On Feb 10, 2:38 pm, VP <vtp2...@gmail.com> wrote:
> Alright people.... short answer:  I think I figured this out (at least
> with my configuration)!!!!
>
> After testing various configurations, here's the result with: ab -kc
> 100 -t 20https://domain.com/imageblog/default/index/  (same
> imageblog app, 100 connections, 20 seconds stress test).
>
> Two things you will notice with this result.
>
> 1.  ZERO failed request.   No morewsgipremature script error!!!!
> 2.  Complete requests is 1234, 61 requests/second on average.
>     Compare to prior configuration:  588 total requests, 29 requests/
> sec on average.  Not to mention 15 failed requests due towsgi
> premature script errors!!!
>
> This is insane!!!
>
> So how did I configure this?   here it is:
>
>   WSGIDaemonProcess web2py user=username group=username \
>                            display-name=%{GROUP} processes=5 threads=1
>
> The important option being 5 processes, 1 thread.
>
> With this configuration, my real app also did not getwsgipremature
> script errors anymore.  And guess what... the requests/sec
> triples!!!!
>
> I am still curious about this.  While my real app can possibly be not
> thread-safe, but the imageblog app should be thread safe (the index
> was simply a listing of images, i.e. read only).  Why would there be aproblemwith more than 1 thread?

Massimo Di Pierro

unread,
Feb 10, 2011, 8:25:03 AM2/10/11
to web2py-users
This is a very important test. I believe web2py to be thread safe and
in fact we thread.lock every access to shared objects even if
operations are atomic, yet I will review it in detail.

Massimo

On Feb 10, 12:38 am, VP <vtp2...@gmail.com> wrote:
> Alright people.... short answer:  I think I figured this out (at least
> with my configuration)!!!!
>
> After testing various configurations, here's the result with: ab -kc
> 100 -t 20https://domain.com/imageblog/default/index/  (same

Massimo Di Pierro

unread,
Feb 10, 2011, 9:19:41 AM2/10/11
to web2py-users
I offer $300 to whoever can identify within 3 weeks and without
ambiguity the cause of this problem. The bounty applies even if the
problem turns out to be not in web2py but in one of the Python
modules, in Apache or in the mod_wsgi implementation. It does not
apply if the problem is due to known bug that has already been fixed
by the responsible party (for example if you are using an old un-
patched python version like 2.5.0 or an old mod_wsgi version).



On Feb 10, 12:38 am, VP <vtp2...@gmail.com> wrote:
> Alright people.... short answer:  I think I figured this out (at least
> with my configuration)!!!!
>
> After testing various configurations, here's the result with: ab -kc
> 100 -t 20https://domain.com/imageblog/default/index/  (same

Jonathan Lundell

unread,
Feb 10, 2011, 10:58:13 AM2/10/11
to web...@googlegroups.com
Good work.

You mentioned that you're using Debian Lenny. What version of Python? 2.5.2?

And what version of psycopg2?

VP

unread,
Feb 10, 2011, 2:18:48 PM2/10/11
to web2py-users
The modules I used are all standard via installation with apt-get.

Python is 2.5.2, and psycopg2 is 2.0.7

On Feb 10, 9:58 am, Jonathan Lundell <jlund...@pobox.com> wrote:
> Good work.
>
> You mentioned that you're using Debian Lenny. What version of Python? 2.5.2?
>
> And what version of psycopg2?
>
> On Feb 9, 2011, at 10:38 PM, VP wrote:
>
>
>
> > Alright people.... short answer:  I think I figured this out (at least
> > with my configuration)!!!!
>
> > After testing various configurations, here's the result with: ab -kc
> > 100 -t 20https://domain.com/imageblog/default/index/  (same

Jonathan Lundell

unread,
Feb 10, 2011, 2:53:40 PM2/10/11
to web...@googlegroups.com

On Feb 10, 2011, at 11:18 AM, VP <vtp...@gmail.com> wrote:

> The modules I used are all standard via installation with apt-get.
>
> Python is 2.5.2, and psycopg2 is 2.0.7
>

Upgrade psycopg2 to at least 2.0.8 and see if that doesn't fix the fault.

Jonathan Lundell

unread,
Feb 12, 2011, 1:03:48 AM2/12/11
to web...@googlegroups.com
On Feb 10, 2011, at 6:19 AM, Massimo Di Pierro wrote:
>
> I offer $300 to whoever can identify within 3 weeks and without
> ambiguity the cause of this problem. The bounty applies even if the
> problem turns out to be not in web2py but in one of the Python
> modules, in Apache or in the mod_wsgi implementation. It does not
> apply if the problem is due to known bug that has already been fixed
> by the responsible party (for example if you are using an old un-
> patched python version like 2.5.0 or an old mod_wsgi version).

psycopg2 2.0.7 is the version in use in this case. 2.0.8 had this in the change log:

> * psycopg/pqpath.c (_pq_fetch_tuples): Don't call Python APIs
> without holding the GIL.


Suggestive, no? There's some evidence on the web that this may have led to "Segmentation fault - premature end of script headers" errors.

2.3.2 is current.

Anthony

unread,
Feb 12, 2011, 1:12:28 AM2/12/11
to web...@googlegroups.com
Though someone reported the same problem with MySQL: https://groups.google.com/d/msg/web2py/AcCDl5eJnds/AXNyKO6kmBwJ
 

Massimo Di Pierro

unread,
Feb 12, 2011, 10:33:00 AM2/12/11
to web2py-users
In any case, please check whether upgrading postgresql for you solves
the problem. It is possible that mysqldb has/had the same issue.

I am very skeptical this is a web2py issue. So we should try isolate
the problem.

Massimo

Jonathan Lundell

unread,
Feb 12, 2011, 10:50:56 AM2/12/11
to web...@googlegroups.com
Part of the problem is that this is a very general error. Lots of things can cause a segmentation fault of the wsgi request thread, which in turn leads to that message. Psycopg2 2.0.7 may well have a problem that causes it, and there may well be other ways for it to happen as well. It's important to eliminate the known problems (like this one) so that if another fault comes along, we know it's something else.

Unfortunately we can't carry these adapters along in web2py, since they're not pure/portable Python.

Luis Díaz

unread,
Feb 12, 2011, 11:33:45 AM2/12/11
to web...@googlegroups.com
Greetings.

I could create a web2py.app.error.w2p

and detail the specifications of the environment,
I will try to put him eye to the issue.


I am a rookie .. but I would try

Díaz Luis
TSU Analisis de Sistemas
Universidad de Carabobo






2011/2/12 Jonathan Lundell <jlun...@pobox.com>

ron_m

unread,
Feb 12, 2011, 11:46:16 AM2/12/11
to web...@googlegroups.com
FWIW I am on Ubuntu 10.04 with all patches applied and could not reproduce the above problem when I tried. The version of python-psycopg2 from the standard repository is 2.0.13. Not an iron-clad guarantee because something else could be at play but at least a data point. I also drove the application a lot harder from ab with 10x number of requests to try and reproduce but still nothing.

The fact it doesn't fail when processes=5 and threads=1 agrees well with the GIL problem - without threads there is no need to hold the GIL.

Jonathan Lundell

unread,
Feb 12, 2011, 12:08:18 PM2/12/11
to web...@googlegroups.com
On Feb 12, 2011, at 8:33 AM, Luis Díaz wrote:
> I could create a web2py.app.error.w2p
>
> and detail the specifications of the environment,
> I will try to put him eye to the issue.
>
>
> I am a rookie .. but I would try
>

Thanks, Luis.

Jonathan Lundell

unread,
Feb 12, 2011, 12:10:13 PM2/12/11
to web...@googlegroups.com
On Feb 12, 2011, at 8:46 AM, ron_m wrote:
> FWIW I am on Ubuntu 10.04 with all patches applied and could not reproduce the above problem when I tried. The version of python-psycopg2 from the standard repository is 2.0.13. Not an iron-clad guarantee because something else could be at play but at least a data point. I also drove the application a lot harder from ab with 10x number of requests to try and reproduce but still nothing.
>
> The fact it doesn't fail when processes=5 and threads=1 agrees well with the GIL problem - without threads there is no need to hold the GIL.

Yes, it's a useful data point.

The best test will be to have an environment that actually fails with psycopg2 2.0.7, and have the failure disappear with an updated version (and no other changes), and to get the error back again with 2.0.7.

Luis Díaz

unread,
Feb 12, 2011, 12:53:48 PM2/12/11
to web...@googlegroups.com
Translation error .. sorry.

I could not ..
might wonder if any of you?

Díaz Luis
TSU Analisis de Sistemas
Universidad de Carabobo






2011/2/12 Jonathan Lundell <jlun...@pobox.com>
On Feb 12, 2011, at 8:46 AM, ron_m wrote:

Luis Díaz

unread,
Feb 14, 2011, 12:48:46 PM2/14/11
to web...@googlegroups.com

Greetings.

I have tried to replicate the error and have had no
exicto.

Environment:
virtual server 1: debian ssl web2py
virtual server 2: debian ssl postgresql

Application:
simple queries and throwing 2
results.

tool:
Jmeter:
Threads 1200
0.2 seconds upload

Díaz Luis
TSU Analisis de Sistemas
Universidad de Carabobo






2011/2/12 Luis Díaz <diazlu...@gmail.com>

Jonathan Lundell

unread,
Feb 14, 2011, 12:56:23 PM2/14/11
to web...@googlegroups.com
On Feb 14, 2011, at 9:48 AM, Luis Díaz wrote:

Greetings.

I have tried to replicate the error and have had no
exicto.

Thanks. What version of psycopg2 are you using?

Luis Díaz

unread,
Feb 14, 2011, 1:11:33 PM2/14/11
to web...@googlegroups.com

I have installed:
python-psycopg2/lenny update 2.0.7-4

Question:
What is the funsion of psycopg2?

Díaz Luis
TSU Analisis de Sistemas
Universidad de Carabobo






2011/2/14 Jonathan Lundell <jlun...@pobox.com>

Luis Díaz

unread,
Feb 14, 2011, 1:24:14 PM2/14/11
to web...@googlegroups.com
ready to investigate:

psycopg2 


The error could be caused by a bad query
developed?

Could provide an app that generates the error?

Díaz Luis
TSU Analisis de Sistemas
Universidad de Carabobo






2011/2/14 Luis Díaz <diazlu...@gmail.com>

Jonathan Lundell

unread,
Feb 14, 2011, 1:39:10 PM2/14/11
to web...@googlegroups.com, Luis Díaz
On Feb 14, 2011, at 10:24 AM, Luis Díaz wrote:
> ready to investigate:
>
> psycopg2
>
> http://wiki.postgresql.org/wiki/Using_psycopg2_with_PostgreSQL
>
> The error could be caused by a bad query
> developed?
>
> Could provide an app that generates the error?

I'm not using it myself; I'm not sure what triggers the error. I assume that there's more information earlier in this thread.

Ahmed Bani

unread,
Feb 16, 2011, 3:28:15 AM2/16/11
to web...@googlegroups.com, Luis Díaz

Question: Could provide you provide an app that generates the error?

Answer: Check Appliance "Customer Relationship Management". 

  db = DAL("postgres://postgres:

Error Snapshot :

<type 'exceptions.RuntimeError'>(Failure to connect, tried 5 times: global name 'psycopg2' is not defined)

Massimo Di Pierro

unread,
Feb 16, 2011, 7:56:46 AM2/16/11
to web2py-users
The problem here is that you did not install the psycopg2 driver for
postgresql.

Ahmed Bani

unread,
Feb 16, 2011, 5:13:07 PM2/16/11
to web...@googlegroups.com

Problem solved:

# Download : hg clone https://web2py.googlecode.com/hg/ web2py

# Follow the instructions:

Title : Using psycopg2 with virtualenv on Ubuntu

Link: http://www.saltycrane.com/blog/2009/07/using-psycopg2-virtualenv-ubuntu-jaunty/

Check your syntax :
# db = DAL('postgres://myself:myp...@127.0.0.1:5432/mydatabase')

Enjoy it!

Now, do we have to migrate web2py database and reimport it again? :

Short tutorial to migrate web2py database - model and data



Massimo Di Pierro

unread,
Feb 16, 2011, 6:07:43 PM2/16/11
to web2py-users
Looks like Jonathan may be the winner here. I think VP raised the
issue so he should decide whether the issue is solved.
Is anybody else having problems after upgrade to the latest psycopg2?

If this problem was dear to you any symbolic contribution towards the
prize will be appreciated.

Massimo



On Feb 10, 8:19 am, Massimo Di Pierro <massimo.dipie...@gmail.com>
wrote:

Jonathan Lundell

unread,
Feb 16, 2011, 6:24:45 PM2/16/11
to web...@googlegroups.com
On Feb 16, 2011, at 3:07 PM, Massimo Di Pierro wrote:
>
> Looks like Jonathan may be the winner here. I think VP raised the
> issue so he should decide whether the issue is solved.
> Is anybody else having problems after upgrade to the latest psycopg2?
>
> If this problem was dear to you any symbolic contribution towards the
> prize will be appreciated.

It need only be symbolic; I think that this falls into the "known bug" category. I found it by Googling.

To be more certain, it'd be good if someone could revert psycogpg2 (with no other changes) and reproduce the problem.

VP

unread,
Mar 4, 2011, 12:31:02 PM3/4/11
to web...@googlegroups.com
For some reason I missed the rest of this discussion.   Just a quick follow up.

+ I upgraded psycopg2 from 2.0.7 (default in Debian 5) to the latest version, 2.4.  This manual installation was smoother than I thought.

+ Result:   preliminary stress tests show that the problem *disappeared*.   No more WSGI premature script error.

Jonathan Lundell

unread,
Mar 4, 2011, 12:41:32 PM3/4/11
to web...@googlegroups.com

Thanks for the report.

I wonder if it might be worth a warning message if an app actually uses (not just imports) psycopg2 and the version is 2.0.7 or earlier. It seems to be fairly widespread.

VP

unread,
Mar 4, 2011, 12:46:14 PM3/4/11
to web2py-users
To be precise, RSS started with about 32MB (with an apache restart)
and starts crawling up to 80MB/process with aforementioned testing.
And If I remember correctly, it stayed that way over night and did not
get released.

As I said, I will do further testing with this toy example and with my
real app and report further findings.

Jonathan Lundell

unread,
Mar 4, 2011, 1:28:14 PM3/4/11
to web...@googlegroups.com
On Mar 4, 2011, at 9:46 AM, VP wrote:
>
> To be precise, RSS started with about 32MB (with an apache restart)
> and starts crawling up to 80MB/process with aforementioned testing.
> And If I remember correctly, it stayed that way over night and did not
> get released.
>
> As I said, I will do further testing with this toy example and with my
> real app and report further findings.

It'd be useful to know whether it's stable at 80MB or it keeps growing.

VP

unread,
Mar 4, 2011, 2:31:36 PM3/4/11
to web2py-users
oops, it looks like that last post was supposed to be for the other
thread ("potential memory leaks").

tazjel

unread,
Jul 7, 2011, 11:38:44 PM7/7/11
to web...@googlegroups.com, tazje...@gmail.com
On 02/17/2011 01:13 AM, Ahmed Bani wrote:

Problem solved:

{{ #---- (CSS) -----) import random response.files.append(URL('static','css/menu.css')) response.files.append(URL('static','css/home.css')) #---- (JavaScript) --- response.files.append(URL('static','js/ajquery.scrollTo-min.js')) response.files.append(URL('static','js/bjquery.localscroll-1.2.7-min.js')) response.files.append(URL('static','js/zexamples.scroll.js')) #------ include web2py specific js code (jquery, calendar, form stuff) ------ }} {{include 'web2py_ajax.html'}} {{=toggle_menuclass()}}
{{=response.flash or ''}}

# Download : hg clone https://web2py.googlecode.com/hg/ web2py

# Follow the instructions:

Title : Using psycopg2 with virtualenv on Ubuntu

Link: http://www.saltycrane.com/blog/2009/07/using-psycopg2-virtualenv-ubuntu-jaunty/

Check your syntax :
# db = DAL('postgres://myself:myp...@127.0.0.1:5432/mydatabase')

Enjoy it!
Follow the instructions:
appadmin.html
generic.html
generic.json
generic.load

cjrh

unread,
Jul 8, 2011, 3:25:45 AM7/8/11
to web...@googlegroups.com
On Monday, January 10, 2011 6:36:18 AM UTC+2, Anthony wrote:
On Sunday, January 9, 2011 1:24:02 AM UTC-5, Graham Dumpleton wrote:
You guys really just got to learn to do your own thing and not treating it like a crusade where you have to convert the world.
 
But that's exactly the problem -- web2py is trying to do its own thing,

...so far so good...
 
and there is a vocal clique of "Pythonistas" who don't like that thing and want to stop it.

...and here we get it wrong.  Why should we care about this vocal clique?  Arguing on reddit and other forums about the minutae of tit-for-tat features is almost entirely pointless.  It convinces nobody, least of all the newbies who are still trying to decide on which framework to use.  And this is something else that bothers me.  Why should we care how many users we have?  It seems to me an entirely pointless objective.   I care only about trying to make web2py my favourite platform for web apps.   In an online "discussion" between proponents of different frameworks, or different anything for that matter, there is precious little to be gained from arguing the case.  That is just how it is.

If we say we just want to do our own thing, then let's do just that. Seriously, going after spurious comments in every reddit or reddit-alike is harmful.  Getting "the truth" out there has a lot of side effects, and they're not all good.   The best, and IMO only real way to promote web2py is to facilitate the building of awesome websites.  That is the whole ball game, so let's focus on that.   

This is, I suppose, obligatory: http://xkcd.com/386/ 

Graham is in a unique position in that wsgi is used by all the frameworks and so he wears a badge that allows him to move freely between them without carrying any baggage, and he clearly gets to see how each of the communities function.  It is important for us to listen to what he has to say, because we will not get such feedback from anywhere else I can think of.

Anthony

unread,
Jul 8, 2011, 10:05:54 AM7/8/11
to web...@googlegroups.com
On Friday, July 8, 2011 3:25:45 AM UTC-4, cjrh wrote:
On Monday, January 10, 2011 6:36:18 AM UTC+2, Anthony wrote:
On Sunday, January 9, 2011 1:24:02 AM UTC-5, Graham Dumpleton wrote:
You guys really just got to learn to do your own thing and not treating it like a crusade where you have to convert the world.
 
But that's exactly the problem -- web2py is trying to do its own thing,

...so far so good...
 
and there is a vocal clique of "Pythonistas" who don't like that thing and want to stop it.

...and here we get it wrong.  Why should we care about this vocal clique?
 
Well, we probably shouldn't care too much, but that doesn't mean we shouldn't care at all. Note, the unfounded criticisms are effectively impugning both the technical and personal integrity of web2py's developers and users, in very public forums. This can have negative professional (and personal) consequences. If you have chosen to build an app with web2py and your client, manager, or investor reads bad things about web2py, you may lose their confidence and favor. So, occassional polite and professional corrections may be appropriate, especially on purely factual misunderstandings (rather than issues of opinion and taste).
 
Why should we care how many users we have?
 
Again, this is not the most important thing, but the number of users is not entirely without consequence either. Generally, more users will translate into more expert contributors and volunteers, which will contribute to the development of the framework and its ecosystem and to its long-term sustainability (i.e., more features, better testing and performance, better documentation, more plugins and applications, better support, more third-party hosting and development tool support, etc.). In many cases, when you select a framework, you want some confidence that it will still be actively maintained several years hence, and that confidence will be bolstered if there is a sizable and active community around it.
 
The best, and IMO only real way to promote web2py is to facilitate the building of awesome websites.  That is the whole ball game, so let's focus on that.
 
Absolutely. :-)
 

cjrh

unread,
Jul 8, 2011, 3:47:09 PM7/8/11
to web...@googlegroups.com
On Friday, 8 July 2011 16:05:54 UTC+2, Anthony wrote:
On Friday, July 8, 2011 3:25:45 AM UTC-4, cjrh wrote:
On Monday, January 10, 2011 6:36:18 AM UTC+2, Anthony wrote:
On Sunday, January 9, 2011 1:24:02 AM UTC-5, Graham Dumpleton wrote:
You guys really just got to learn to do your own thing and not treating it like a crusade where you have to convert the world.
 
But that's exactly the problem -- web2py is trying to do its own thing,

...so far so good...
 
and there is a vocal clique of "Pythonistas" who don't like that thing and want to stop it.

...and here we get it wrong.  Why should we care about this vocal clique?
 
Note, the unfounded criticisms are effectively impugning both the technical and personal integrity of web2py's developers and users, in very public forums. This can have negative professional (and personal) consequences. If you have chosen to build an app with web2py and your client, manager, or investor reads bad things about web2py, you may lose their confidence and favor.

Haters gonna hate :)  
 
 
Why should we care how many users we have?
 
Again, this is not the most important thing, but the number of users is not entirely without consequence either. Generally, more users will translate into more expert contributors and volunteers, which will contribute to the development of the framework and its ecosystem and to its long-term sustainability (i.e., more features, better testing and performance, better documentation, more plugins and applications, better support, more third-party hosting and development tool support, etc.).

This is almost definitely not true.  Or rather, it certainly hasn't been my experience anyway.  The number of people in a project or community appears to have no bearing whatsoever on the quality of the code or the abilities of contributors. 
 
In many cases, when you select a framework, you want some confidence that it will still be actively maintained several years hence, and that confidence will be bolstered if there is a sizable and active community around it.

I still don't see why this is my problem, or indeed even yours, or anyone else's for that matter.  I do not see why I need to care about whether some pointy-hair boss thinks web2py will be actively maintained in the future.  I have selected web2py on technical merit.  I do not care how active the community is, for I have the source code.  

In truth, I have selected web2py because my value system around coding has a large intersection with Massimo's, IMO.  I value the conveniences that he built into web2py early on; conveniences that other frameworks left "as an exercise to the reader".  I think that matters.  I think that remains a selling point for web2py.  I think it is unfortunate that he uses the word "Enterprise" to indicate that those conveniences exist, and I think that that causes a lot more confusion than it should.  But our PR face is way, way too defensive.  I understand that we might not see it like that, but if someone like Graham happens point out that that is how it looks from the outside, then we should take note of that.

IMO there is very little merit in just about anything that goes through reddit.  So why bother wasting time on it?   I am not sure I even want to attract too many coders that spend time reading reddit anyway.  Apologies upfront if that offends anyone.

Anthony

unread,
Jul 8, 2011, 4:22:22 PM7/8/11
to web...@googlegroups.com
On Friday, July 8, 2011 3:47:09 PM UTC-4, cjrh wrote:
Why should we care how many users we have?
Again, this is not the most important thing, but the number of users is not entirely without consequence either. Generally, more users will translate into more expert contributors and volunteers, which will contribute to the development of the framework and its ecosystem and to its long-term sustainability (i.e., more features, better testing and performance, better documentation, more plugins and applications, better support, more third-party hosting and development tool support, etc.).

This is almost definitely not true. Or rather, it certainly hasn't been my experience anyway. The number of people in a project or community appears to have no bearing whatsoever on the quality of the code or the abilities of contributors.
It's not just about the core framework, but the larger ecosystem. Right now there are several hosting services in beta dedicated specifically to hosting Django apps. Pycharm includes special functionality to support Django development. There are lots of libraries and reusable apps specifically for Django. (Not to mention what you can find in the Rails, PHP, or ASP.NET worlds.) So, I think there can be some benefits to a larger community and user base, though such benefits may not matter to everyone.
In many cases, when you select a framework, you want some confidence that it will still be actively maintained several years hence, and that confidence will be bolstered if there is a sizable and active community around it.

I still don't see why this is my problem, or indeed even yours, or anyone else's for that matter. I do not see why I need to care about whether some pointy-hair boss thinks web2py will be actively maintained in the future. I have selected web2py on technical merit. I do not care how active the community is, for I have the source code.
 
Yes, but not everyone wants to worry about having to take over development and maintenance of their app's framework should it be abandoned by its core developers. Why do we worry about backward compatibility? Backward compatibility only matters if the framework is actively being developed to begin with, so concern about backward compatibility follows from a concern about long-term sustainability. I agree that technical merit is critical, but it's not the only consideration, at least not for all users.
 
In truth, I have selected web2py because my value system around coding has a large intersection with Massimo's, IMO. I value the conveniences that he built into web2py early on; conveniences that other frameworks left "as an exercise to the reader". I think that matters. I think that remains a selling point for web2py.
 
Hear, hear!
 
But our PR face is way, way too defensive. I understand that we might not see it like that, but if someone like Graham happens point out that that is how it looks from the outside, then we should take note of that.
 
Note, this thread is from back in January, and in the past few months, there has been relatively little negative reddit activity, so perhaps we have already moved on to better days. :-)

pbreit

unread,
Jul 8, 2011, 5:18:57 PM7/8/11
to web...@googlegroups.com
Sorry cjrh, but the number of users and the perception of the project DO matter. Some of us are building businesses around Web2py and need to feel comfortable that the project has longevity, that we can hire and retain engineers, that we can find resources if we need them, etc. Not everyone is doing just their own one-person projects.

I don't really care if engineering candidates have Web2py experience or not but I do care if any perceptual issues handicap recruiting.

You need to think slightly bigger than your own personal projects.

Massimo Di Pierro

unread,
Jul 8, 2011, 6:15:57 PM7/8/11
to web2py-users
Have I missed some new controversial reddit thread?

cjrh

unread,
Jul 8, 2011, 7:05:56 PM7/8/11
to web...@googlegroups.com
No, or rather, not that I am aware.  We are discussing, I think, the principal of the matter.

cjrh

unread,
Jul 8, 2011, 7:16:33 PM7/8/11
to web...@googlegroups.com
On Friday, 8 July 2011 23:18:57 UTC+2, pbreit wrote:
You need to think slightly bigger than your own personal projects.

Why should I?  The bigger community does not concern me.  I contribute to web2py for entirely selfish reasons, because it want it to be the best tool for me to use for my own personal (actually work) projects.  I do not care one iota about what other people may or may not think about web2py.  It doesn't concern me even one bit.  I have tried the various web frameworks, and I have made a decision based on the relative merits to focus on web2py.  I don't see AT ALL why I need to think any bigger than my own current needs.  

pbreit

unread,
Jul 9, 2011, 12:46:37 PM7/9/11
to web...@googlegroups.com
That's fine. But it makes your opinion irrelevant. Very few program in isolation.
Reply all
Reply to author
Forward
0 new messages