web2py and GAE

[Jim]

Executive summary:

1. web2py needs to focus on appealing to newbies As we get more
experienced, we hopefully can
make contributions to make web2py better and better. If
I'm in charge of maintaining a site
that's already written, it's unlikely that I'll want to
throw out PHP or Django or whatever and
start over from scratch. Or that my boss will let me.

2. web2py really needs to work without the application name - /
default/index would be fine for now
with a goal of having / as "home". www.mydomain.com/myapp/default/index
is huge and is
bad from an SEO standpoint

3. the degree that code from Google can be re-used will have an
effect on how many people are willing
to use web2py instead of app-engine-patch or whatever Why
write your own code for sharding
counters when you can use http://code.google.com/appengine/articles/sharding_counters.html
?

More details

1. Massimo has been able to abstract away a lot of the complexities
involved in working with GAE and really improve on the Django
framework.

It's painful to think about how much boiler-plate code you'd have to
write if you used Django instead. But I'm saying that as somebody
who's installed Eclipse/PyDev just far enough to get it to run
helloworld.py and has no familiarity with Python.

GAE/web2py is going to appeal to people who don't want to run a server
as opposed to people who want to be able to control everything. To
get into enterprises, it's going to take hobby programmers like me
saying good things about it.

2. You probably understand SEO better than I do. But it seems clear
that letting spiders start from / is better than having them start
from /app/controller/function It's also a lot less to type. Using
301s on URLs is painful and best to be avoided.

For personal projects that aren't supposed to be used by the general
public, who cares? But if you're trying to get a site used by lots
of people, shorter is better. You're not going to get word-of-mouth
if people can't remember the URL. And the longer it is, the more
chances to mess it up.

3. To my wife I compared web2py to training wheels for GAE. That has
its good points and its bad points. For example, I'm aware that
there's going to be problems with counters if my site gets popular.
Ideally I'd like to have something that handles that in web2py. But
if not, then I want to plug in the code from Google itself or possibly
from Best Buy's giftag project and have it work.

The same is true of memcache - let's use it for disk.cache, ram.cache
and sessions. There's been a couple of contributions in this area and
it would be great to see them adopted as part of web2py.

And a third thing is using the expires header as mentioned in
http://groups.google.com/group/web2py/browse_thread/thread/9ee570b123b3b92/406fb8eddccc4f37?lnk=gst&q=expires+header#406fb8eddccc4f37
Given how variable the response from GAE is, it'd be better to not
visit it at all again if you don't have to. So designing pages that
can be fetched from the browser's cache instead of visiting the site a
second time looks like a big win.

Should this be one project? Should there be a gae2py or py2gae or
whatever? Up to Massimo.

web2py is at least 95% there in terms of GAE. In reading the messages
about GAE, it's clear that some people decided that the only way to
close the gap is to not use web2py.

I hope that changes. It would be super cool if web2py/GAE became the
Wordpress of apps, used by everyone from bloggers to the New York
Times.


Still more details

1. I'm interested in GAE because I want to outsource network
administration, system administration, etc. to Google. The free up to
1GB and 5 million pageviews is attractive but there's lots of options
for shared hosting. It's nice to not worry about being slashdotted
but that's unlikely to happen. The huge thing is not dealing with
load balancing, security exploits, etc.

I have concerns about the variability of the user experience with
GAE. Try something like this:

i=1 ; while (( $i<1000)); do echo $i; time lynx -dump http://www.vorby.com/
> /dev/null ; (( i=$i+1 )); done
real 0m6.808s
real 0m4.358s
real 0m2.194s
real 0m2.136s
real 0m4.372s

If my only goal was a consistent, fast experience ... this would drive
me crazy.

I'm using vorby.com as an example because it's written by a programmer
who's able to get auto-complete working on GAE and seems reasonably
skilled. I have no connection with vorby.com and AFAIK it's not using
web2py. But it shows a little bit of the inconsistent responses that
you get with GAE.

You could try giftag.com instead. That's a Best Buy project which has
been featured by Google. Mostly times are under 1 second but I have
seen 5 seconds.

It seems like apps that aren't being used get swapped out to disk and
then get re-loaded when necessary. That means that I've seen it take
20 seconds or more for the first page to load. I'm not sure why the
variance with loads of the same page.

I have concerns about Google deciding that they don't want to be in
the business any more, the way that they have with other projects.
Now that they're charging for GAE, though, I think it's highly
unlikely that they'll stop it.

[Markus Gritsch]

On Fri, Feb 27, 2009 at 9:56 PM, Jim <JDei...@gmail.com> wrote:
>
> 2.  web2py really needs to work without the application name  - /
> default/index would be fine for now
>          with a goal of having / as "home".    www.mydomain.com/myapp/default/index
> is huge and is
>          bad from an SEO standpoint

For an URL www.mydomain.com web2py will first look for an application
named init, then welcome. You can name your app 'init' and it gets
served when you visit www.mydomain.com.

Markus

[mdipierro]

Hi Jim,

you raise many issues and I am not sure I am answering them all.

about the path in URL you have four ways to change that:
1) call your app init
2) create an empty app init that redirects to your app
3) rename and edit routes.example.py as routes.py and remap URLs (like
Django does)
4) use apache to remap the URLs.

I do not understand this comment "I'm aware that

there's going to be problems with counters if my site gets popular.
Ideally I'd like to have something that handles that in web2py."

I can see this (http://www.vorby.com/) is very slow. Is this built
with web2py?

Massimo

On Feb 27, 2:56 pm, Jim <JDeib...@gmail.com> wrote:
> Executive summary:
>
> 1. web2py needs to focus on appealing to newbies As we get more
> experienced, we hopefully can
> make contributions to make web2py better and better. If
> I'm in charge of maintaining a site
> that's already written, it's unlikely that I'll want to
> throw out PHP or Django or whatever and
> start over from scratch. Or that my boss will let me.
>
> 2. web2py really needs to work without the application name - /
> default/index would be fine for now
> with a goal of having / as "home". www.mydomain.com/myapp/default/index
> is huge and is
> bad from an SEO standpoint
>
> 3. the degree that code from Google can be re-used will have an
> effect on how many people are willing
> to use web2py instead of app-engine-patch or whatever Why
> write your own code for sharding

> counters when you can usehttp://code.google.com/appengine/articles/sharding_counters.html

> And a third thing is using the expires header as mentioned inhttp://groups.google.com/group/web2py/browse_thread/thread/9ee570b123...

> Given how variable the response from GAE is, it'd be better to not
> visit it at all again if you don't have to. So designing pages that
> can be fetched from the browser's cache instead of visiting the site a
> second time looks like a big win.
>
> Should this be one project? Should there be a gae2py or py2gae or
> whatever? Up to Massimo.
>
> web2py is at least 95% there in terms of GAE. In reading the messages
> about GAE, it's clear that some people decided that the only way to
> close the gap is to not use web2py.
>
> I hope that changes. It would be super cool if web2py/GAE became the
> Wordpress of apps, used by everyone from bloggers to the New York
> Times.
>
> Still more details
>
> 1. I'm interested in GAE because I want to outsource network
> administration, system administration, etc. to Google. The free up to
> 1GB and 5 million pageviews is attractive but there's lots of options
> for shared hosting. It's nice to not worry about being slashdotted
> but that's unlikely to happen. The huge thing is not dealing with
> load balancing, security exploits, etc.
>
> I have concerns about the variability of the user experience with
> GAE. Try something like this:
>

> i=1 ; while (( $i<1000)); do echo $i; time lynx -dumphttp://www.vorby.com/> /dev/null ; (( i=$i+1 )); done

[Jim]

That's fantastic news about the different ways. I've been trying
routes.py and it wasn't working for me. I noted your suggestion that
it not be used in a production environment. And I'm pretty sure that
GAE doesn't allow for Apache-style rewriting. But I will definitely
try #1.

With GAE allowing 10 apps per person, it seems reasonable to have a
"web2py farm". Basically create myapp1, myapp2, etc. under the GAE
directory and then symlink to the gluon directory and other things
that are necessary. That lets you upload different favicon.ico and
robots.txt files for each app by providing a different app.yaml
Maybe have a script that unlinks /google_appengine/web2py/applications/
init, relinks myapp2 (or whatever) as init and then uploads it to GAE.

GAE is write-bound so the advice is to "shard your counters". So
rather than writing message 100001, 100002, etc. you want to split it
up. For something like the reddish appliance, which uses a bookmark/
counter structure, I think you'd add shard1.counter + shard2.counter
+ ... shard20.counter to get an absolute number.

I don't know what vorby is written in. My guess is that it's not
web2py, based on the use of prototype and mochikit. yslow (a plugin
for firebug) reports from 1.5 to 4 seconds on the 10 times I tried
it. That's the problem with every GAE app that I've looked at:
they're slower than you'd like.

giftag.com was almost 5 seconds to load.

My hope/expectation is that the sites get faster as they get more
popular. more use of your app equals it getting deployed to more data
centers.

[AchipA]

Massimo, how would you feel about extending routes to take into
account domains as well ? Basically, with routes.py you can do away
with the application name only if you use one application, but it
would be nice to be able to use a single web2py install for multiple
apps.

[mdipierro]

can you propose a syntax to do that is backward compatible?

If we agree on that it should be easy to patch it.

[AchipA]

Well, two simple ideas to begin with:

a) extend the tuples with a third parameter which would be the domain
(also regex), use an exception to make it backwards compatible.

b) introduce a routes_domain = { 'mydomain.com' : (routes_in,
routes_out, routes_onerror) } style hash and use the global/current
routes_in and routes_out as catch-alls if no hash key matches