New experimental development server for Python

[Brian Quinlan]

Hi,

I'm happy to announce that the App Engine team is working on a new
development server for Python (and eventually Go).

The new development server aims to provide higher performance for
complex applications and better emulation of the App Engine deployment
environment.

There are several features that are currently missing:
- support for backends
- the administrative UI (i.e. _ah/admin)
- Go

Development is still at an early stage so I'm sure that there are lots
of bugs to be found and fixed.

Which is why we'd appreciate your help in testing and, if you are
feeling keen, fixing these issues.

You can download the latest version from:
http://code.google.com/p/appengine-devappserver2-experiment/downloads/

Or work directly with the git repository:
http://code.google.com/p/appengine-devappserver2-experiment/source/checkout

There is also a public discussion group associated with the project:
https://groups.google.com/forum/?fromgroups#!forum/appengine-devappserver2-discuss

Thanks for your help!

Cheers,
Brian - App Engine Python Team

[alex]

Brian, this is awesome!

I've already tried it with a side project of mine that uses task
queues and urlfetch. Even though it runs a couple backends too, the
code already handles different cases, dev vs production, because of an
open issue with addressing backends (5105). It all runs smoothly!

Really appreciate putting the tests in the repo. Love it.

I'll try using it for my other apps and see how it goes.

> --
> You received this message because you are subscribed to the Google Groups "Google App Engine" group.
> To post to this group, send email to google-a...@googlegroups.com.
> To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
>

[Kaan Soral]

Great to hear :)

I will also test it out some time.

Now we can finally test our apps on all browsers before production.

[Brian Quinlan]

We have released a new version of the development server:

To download:
http://code.google.com/p/appengine-devappserver2-experiment/downloads/detail?name=devappserver2-v0.3.zip

Changes:
- threads are implicitly joined on request exit
- basic taskqueue UI (more later)
- fixes for virtualenv
- windows unit test fixes
- reduced logging noise
- correct logout url generation

As always, feedback is welcome!

Cheers,
Brian

[PK]

Great... Thanks!!

More regression tests pass now but one of them fails with the message below. I am on 1.7.3, all my tests pass with the regular dev server.

BadRequestError: remote_api supports transactional queries only in the high-replication datastore.

Is this something consistent with the bugs you already track?

Thanks,
PK

[PK]

I looked in more detail the stack trace and hits an issue I had seen before that was fixed when bug 3643 was fixed. The bug was fixed in 1.7.2, I am wondering whether you could have forked some code before that fix went in?

[Brian Quinlan]

Hi PK,

Thanks for the bug description! We understand the cause (the lack of
the "dev~" prefix on app ids) and will fix it soon [1].

Cheers,
Brian

[1] "soon" may not be that soon due to the holidays :-)

[PK]

OK Thanks... I filed issue #16 to track it.

Another test code showstopper, is the following:

We have some test code that waits until all tasks have executed and only then continues. The way it does this is by scraping the /_ah/admin/queueues page until all the counters are 0. Without admin console this part also fails. I assume you will provide an admin console at some point so this will take care of it.

(BTW, this is really old ugly code that will also fail next time the UI changes, it is probably time to start using the QueueStatistics for the same purpose, incidentally it just became fully supported with 1.7.4).

[Brian Quinlan]

On Fri, Dec 14, 2012 at 12:30 PM, PK <p...@gae123.com> wrote:
> OK Thanks... I filed issue #16 to track it.

Thanks!

> Another test code showstopper, is the following:
>
> We have some test code that waits until all tasks have executed and only then continues. The way it does this is by scraping the /_ah/admin/queueues page until all the counters are 0. Without admin console this part also fails. I assume you will provide an admin console at some point so this will take care of it.

Could you try adding this to your app.yaml:
- url: /admin_console/.*
script: $PYTHON_LIB/google/appengine/ext/admin

Cheers,
Brian

[Sam McNally]

We have released a new version. Download it from http://code.google.com/p/appengine-devappserver2-experiment/downloads/detail?name=devappserver2-v0.4.zip

New in this version:

- backends with support for background threads

- admin UI for cron, datastore indexes and inbound mail

- improved taskqueue UI

[Brian Quinlan]

We have released a new version of the experimental Python development server. We think that it is very usable at this point and has some significant improvements over the existing version.

To download:

http://code.google.com/p/appengine-devappserver2-experiment/downloads/detail?name=devappserver2-v0.5.zip

Changes:

- a functionally complete administrative UI (it is also a lot more polished but still needs some work)

- more reliable task execution

- (experimental) support for local Cloud SQL development

- lots of bug fixes!

As always, feedback is very welcome!

Cheers,
Brian

On Tue, Nov 27, 2012 at 1:14 PM, Brian Quinlan <bqui...@google.com> wrote:

[Jeff Schnitzer]

Do background threads work? Whenever I try to start one, I get
BackgroundThreadLimitReachedError.

Jeff

[Brian Quinlan]

That should work.

Are you creating a background thread on a frontend or a backend instance?

Cheers,

Brian

On Thu, Jan 24, 2013 at 6:38 AM, Jeff Schnitzer <je...@infohazard.org> wrote:

BackgroundThreadLimitReachedError

[Jeff Schnitzer]

It's a resident backend.

Jeff

[Brian Quinlan]

On Thu, Jan 24, 2013 at 1:41 PM, Jeff Schnitzer <je...@infohazard.org> wrote:
>
> It's a resident backend.
>
> Jeff

Are you sure that this happens on the first attempt to create a
background thread? If so, then could you could try to find this block
in google/appegine/tools/devappserver2/dispatcher.py:

try:
inst.reserve_background_thread()
except instance.CannotAcceptRequests:
+ logging.exception('Cannot create background thread.'
raise request_info.BackgroundThreadLimitReachedError()

And see what logging you see?

Cheers,
Brian

[Brian Quinlan]

We have released a new version of the new Python development server.
This development server will replace the existing one in the 1.7.6
release so we'd greatly appreciate any feedback.

To download:
https://code.google.com/p/appengine-devappserver2-experiment/downloads/

Features:
- Add a blobstore viewer (thanks Alex!)
- Several UI improvements (thanks Alex)
- Add protorpc support
- Add automatic conversion from the datastore file stub to the sqlite stub.
- Add update checking

Bug fixes:
- Allow starting background threads from backends handling /_ah/start requests.
- Set DEFAULT_VERSION_HOSTNAME
- Avoid crashing on empty backends
- Set environ['SERVER_NAME'] to match environ['HTTP_HOST']
- Make exception logging more robust
- Bypass login: required for inbound xmpp requests from the admin UI
- Stop raising KeyboardInterrupts in async_quit and for SIGTERM and SIGINT
- Gracefully handle the case where 'headers' or 'response_code' are not set
- Always filter x-appengine-fake-is-admin and x-appengine-fake-logged-in headers

Cheers,
Brian

[Jacob G]

Hi Brian,

This new development server sounds very exciting. I haven't had a chance yet to try it, but I look forward to checking out all these great features.

One question I have is regarding your bullet point "Add automatic conversion from the datastore file stub to the sqlite stub." Does this feature address the conversion failure problem discussed here:

http://stackoverflow.com/questions/14627143/gae-port-sqlite-fails-on-local-dev-server

Thanks.

[Brian Quinlan]

On Sat, Feb 9, 2013 at 1:31 AM, Jacob G <ja...@fareclock.com> wrote:
> Hi Brian,
>
> This new development server sounds very exciting. I haven't had a chance yet
> to try it, but I look forward to checking out all these great features.
>
> One question I have is regarding your bullet point "Add automatic conversion
> from the datastore file stub to the sqlite stub." Does this feature address
> the conversion failure problem discussed here:
>
> http://stackoverflow.com/questions/14627143/gae-port-sqlite-fails-on-local-dev-server

I don't understand the reason for that failure so I can't say for
sure. It looks like the discussion is ongoing here:
https://code.google.com/p/googleappengine/issues/detail?id=7717

Cheers,
Brian

> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.

> To unsubscribe from this group and stop receiving emails from it, send an
> email to google-appengi...@googlegroups.com.

>
> To post to this group, send email to google-a...@googlegroups.com.

> Visit this group at http://groups.google.com/group/google-appengine?hl=en.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Brian Quinlan]

We have released a new version of the Python development server. This
version will be included in the App Engine 1.7.6 SDK release so any
testing or other feedback would be greatly appreciated!

Aside from a long list of bug fixes, this version adds experimental Go support!

To download the cross-platform package for Python:
https://code.google.com/p/appengine-devappserver2-experiment/downloads/detail?name=devappserver2-v0.7.zip

To download the Linux amd64-only package for Python and Go:
https://code.google.com/p/appengine-devappserver2-experiment/downloads/detail?name=devappserver2-v0.7_linux_amd64.zip

Changes:
- Fixes testbed support
- Tests now pass on Windows and Mac OS X
- Multiple set-cookies headers no longer combined
- Some minor UI improvements
- Invalid login cookies are ignored
- Go runtime support!
- Use a threadsafe subprocess implementation
- Fixed Google Cloud SQL support on windows
- Performance:
- use mtime checking for config file update checks
- switch to one thread per async work item
- set the backlog for WsgiServer sockets to 100

[pdknsk]

I can only recommend to try it.

[Brian Quinlan]

We have released a new version of the Python development server. A descendant of this version will be included in the App Engine 1.7.7 SDK release so any testing or other feedback would be greatly appreciated!

To download the cross-platform package for Python:

https://appengine-devappserver2-experiment.googlecode.com/files/devappserver2-v0.8.zip

New Features:

- Cloud Endpoints (see http://goo.gl/m2GjG)

- Integration points for Python tools (see http://goo.gl/3oSBp)

- Add support for tasks that target backends

Bug fixes: 

- Fix the size display in blobstore viewer

- Fix the styling of the "Purge" button in the taskqueue UI

- Fix orphan runtime processes on shutdown

- Make --log_level=debug display debug-level logging

Cheers,

Brian

[michael kariv]

@Brian Quinlan,

I have installed 1.7.7.

I have read through the document.

The document gives us hope, because as of now PyDev debugging (setting a breakpoint in the app code) does not work.

I have tried to implement the document, and so far I can connect remote debugger to the instance and see logging.info() printouts, but I still can not set a breakpoint

My detailed account of what I did is here:

http://goo.gl/XRU01