[google-appengine] Python optimization: One Larger or Many Small Modules

[stevep]

First time post to the forums, so please forgive any newbie issues.

I am currently developing an GAE application. Client accesses two main
GAE handler scripts using different urls such as:
http://localhost:8080/1000 -- for function 1000 in read handler
http://localhost:8080/1100 -- for function 1100 in read handler
http://localhost:8080/1200 -- for function 1200 in read handler
http://localhost:8080/write/2000 -- for function 2000 in write
handler
...etc.

Within a handler, WSGIApplication is used to parse the url and access
specific function. The modules are about 1,300 lines of code for the
read, and 1,500 for the write handlers. Most functions stand alone:
e.g. top/to bottom process inside the function serves the url request.
(Write handler does share some sharding functions across many
functions.)

I had thought to include all the functions in two big handler scripts
because it would likely stay resident (hot) if the site is busy.
However, read some comments today about the new biz GAE where people
were frustrated that load times for handler scripts restrict the total
amount of time available for a server response.

So, my question is this: Should I stay with the two large modules, or
break them up into a separate module for each function. IOW: in my
setup above, functions 1000, 1100, and 1200 all get served via
WSAIApplication from my MainRead.py handler. I could pretty easily
have each served by: Read1000.py, Read1100.py, Read1200.py. This
granular approach would mean a bit less than 50 handlers for all read/
write functions.

Hopefully there is only a fractional time cost for GAE to load the
larger code base module if it loads the .pyc version. If so, I would
prefer to stay with the two large modules. However, if we're talking
about recompiling the .py script each time, then I'm guessing I'd need
to go with the larger number of modules with less script lines in
each.

Thanks in advance.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[Geoffrey Spear]

On May 19, 4:23 pm, stevep <prosse...@gmail.com> wrote:
> First time post to the forums, so please forgive any newbie issues.
>
> I am currently developing an GAE application. Client accesses two main
> GAE handler scripts using different urls such as:

>  http://localhost:8080/1000-- for function 1000 in read handler
>  http://localhost:8080/1100-- for function 1100 in read handler
>  http://localhost:8080/1200-- for function 1200 in read handler
>  http://localhost:8080/write/2000-- for function 2000 in write

There are no .pyc files on App Engine, so you're not going to get a
speed boost there.

However, the slow load times generally seem to be from people using a
large framework such as Django that imports a ton of stuff on startup
(because it's designed to run in an environment where startups are
infrequent). Importing a bunch of webapp handlers all at once is
unlikely to have much of a noticeable effect, although it might be
worth benchmarking yourself.

You should also note that if you use a single .py file with a webapp
WSGIApplication mapping, you're going to need to import all of your
handlers for the mapping to work anyway. You can mitigate this by
doing your URL mapping in app.yaml or rolling your own lazy loader.

[Stephen]

On May 19, 9:23 pm, stevep <prosse...@gmail.com> wrote:
>
> Hopefully there is only a fractional time cost for GAE to load the
> larger code base module if it loads the .pyc version. If so, I would
> prefer to stay with the two large modules.

Add the following to your app.yaml file:


derived_file_type:
- python_precompiled

[Tim Hoffman]

Any pointers to some docs on this directive ?

T

[djidjadji]

I don't think the SDK will ever upload pyc files.
It could be handcrafted to do some non-pythonic things.

I looked at the code for appcfg.py and it does processes this directive.
I does the precompilation by sending a '/api/appversion/precompile'
command to the server.

I have enabled it for a test app and did an upload.
It gives responses that it starts and stops precompilation.

If I look at the files on the GAE server that are new there is a directory added
'/_ah/python_bytecode' in this directory there are for me just 2 pyc files.
But the test app has more then 2 py files. Why these and only these 2
where selected I have no clue. Maybe it is just in a test state at the
moment. If I sort my py files by date these are the two oldest files.

If I change a py file and upload, it gets uploaded, but not precompiled.

Maybe it goes life with one of the new releases.

2010/5/23 Tim Hoffman <zute...@gmail.com>:

[Stephen]

On May 23, 5:26 pm, djidjadji <djidja...@gmail.com> wrote:
>
> If I look at the files on the GAE server that are new there is a directory added
> '/_ah/python_bytecode' in this directory there are for me just 2 pyc files.

It is compiling all of my python files...