Important for Python developers - Updates to the Development Server in 1.7.6

898 views
Skip to first unread message

Andrew Jessup

unread,
Mar 19, 2013, 8:57:02 PM3/19/13
to google-a...@googlegroups.com
In the 1.7.6 release of App Engine we’ve included a major upgrade to the Development Server in the Python SDK, designed to make development faster and more faithfully reproduce the production App Engine runtime environment. The new version more faithfully simulates elements of the production App Engine environment, such as concurrent requests and backends, and in most cases this will also mean your apps run faster. We previously included this in the 1.7.5 release as dev_appserver2.py, in this release it has become the default dev_appserver.py

While we're confident the improvements are substantial, as with any major change, some things work a little differently with the new dev_appserver. Here’s a few changes to watch out for:
  • Your local development admin console will run on it’s own server and port, rather than under /_ah/admin
  • We’ve changed the defaults for new applications, such as using HRD rather than Master/Slave, and using SQLlite to emulate the datastore.
  • We’ve cleaned up many command line flags and arguments
  • Python 2.5 applications are no longer officially supported on the new Development Server in line with our deprecation announcement (although they may still be run)
We’ve provided a detailed summary of all the changes to the Development Server in our documentation.

The the legacy dev_appserver.py remains available in the Python SDK as old_dev_appserver.py for those who still require it. However if you encounter any issues with the new Development Server we strongly encourage you to report them on our issue tracker.

We hope you like the new changes, and that you enjoy snappier app development with fewer surprises on deployment. Keep on coding!

- Andrew Jessup, for the App Engine team

Moises Belchin

unread,
Mar 20, 2013, 5:42:31 AM3/20/13
to Google App Engine
When I try to enter in Full Text Search I receive this message:


Imágenes integradas 1


Saludos.
Moisés Belchín.


2013/3/20 Andrew Jessup <jes...@google.com>

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

search_error.png

vivpuri

unread,
Mar 20, 2013, 6:30:09 AM3/20/13
to google-a...@googlegroups.com

Shay Erlichmen

unread,
Mar 20, 2013, 10:27:01 AM3/20/13
to google-a...@googlegroups.com
I worked with preview version of it and I was too embarrassed to ask, but now that this is official quick question:
How do you debug code now? in the old dev server (under eclipse + pydev) all I have todo is set a breakpoint it seem now that this is no longer working (latest pydev + juno).
I imagine that I can attach to the process but please tell me that this is not the optimal way. 

Andreas Schmid

unread,
Mar 20, 2013, 10:40:56 AM3/20/13
to google-a...@googlegroups.com
 - We’ve cleaned up many command line flags and arguments

i see. the --allow_skipped_files flag was removed. 
i this flag is/was REALLY important to me because i develop on 'normal' files but i deploy minified versions and zipped packages.  
without that flag its really hard to develop for me and i am sure for many others too.

please put that back! 

Andrew Jessup

unread,
Mar 21, 2013, 12:34:16 AM3/21/13
to google-a...@googlegroups.com
Hi Andreas - we'd like to understand your use-case a bit more - do you mind raising a bug about this? 
--

Andrew Jessup | Product Manager, Google App Engine | jes...@google.com 

Andrew Jessup

unread,
Mar 21, 2013, 12:50:20 AM3/21/13
to google-a...@googlegroups.com
Hi Shay,

A key goal of the new development server was to provide a runtime environment closer to how your apps actually run in production, which is why a single app will now often run in multiple processes. This raises an interesting debugging challenge - how do we easily bind the right process a debugger to, particularly if that process hasn't started yet? How do we prevent that process from being automatically restarted should you change the file?

In our open source mirror project for the new dev_appserver we've added some additional flags that allow you to more tightly control process execution and provide hooks that allow you to write scripts to control how and where to attach a debugger. We hope to tighten this API and include with dev_appserver it in the next release of the App Engine SDK. 

If you'd like to take a look now, I would point you to this discussion https://code.google.com/p/appengine-devappserver2-experiment/issues/detail?id=28 and in particular the document linked there that summarizes our proposed debugging approach and provides some sample code to get started.




--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

Shay Erlichmen

unread,
Mar 21, 2013, 4:02:19 AM3/21/13
to google-a...@googlegroups.com
Thanks for the info Andrew, I figure that startup script was the way to go but wasn't sure on how.
I think that you need to figure how to release a version that wouldn't required manual wireup for debugging on at least make it very easy to do.

Shay Erlichmen

unread,
Mar 21, 2013, 4:08:32 AM3/21/13
to google-a...@googlegroups.com
I also notice that now I need to choose between debug and live code update (AKA hot patching).
Debugging while code editing was one of killers features of the old server, please find a solution for this also.

PK

unread,
Mar 21, 2013, 11:11:46 AM3/21/13
to google-a...@googlegroups.com, google-a...@googlegroups.com
I tried many times to use a debugger. Originally I was using the wing IDE, now I am using pycharm. The main problem was that the debugger was just too slow so almost always I ended up using log messages. 

Don't get me wrong, fast debugging with "hot patching" would be great but i never felt i had it before and behavior closer to production is more important. 

PK

Renzo Nuccitelli

unread,
Mar 21, 2013, 11:48:51 AM3/21/13
to google-a...@googlegroups.com
 Disagree,

 Debugging and hot patch is the killer feature. The problens with diferent behavior of local and production was easily with a test or homolog enviroment. On the other hand, using just log to fix and find bugs will increase too much the development time. Until it is not solved, I will keep the 1.7.5 on my machine.

Darien Caldwell

unread,
Mar 21, 2013, 2:58:04 PM3/21/13
to google-a...@googlegroups.com

On Tuesday, March 19, 2013 5:57:02 PM UTC-7, Andrew Jessup wrote:

Python 2.5 applications are no longer officially supported on the new Development Server in line with our deprecation announcement (although they may still be run)

 
 
I must have missed that notification. I know 2.7 was brought online, but I've seen nothing to indicate 2.5 was being depreciated or removed. Where/when was this announcement? 

Brian Quinlan

unread,
Mar 21, 2013, 3:35:47 PM3/21/13
to google-a...@googlegroups.com
The Python 2.5 deprecation blog post is here:

Cheers,
Brian


--

Alex Burgel

unread,
Mar 22, 2013, 11:42:28 AM3/22/13
to google-a...@googlegroups.com
On Thursday, March 21, 2013 12:50:20 AM UTC-4, Andrew Jessup wrote:
If you'd like to take a look now, I would point you to this discussion https://code.google.com/p/appengine-devappserver2-experiment/issues/detail?id=28 and in particular the document linked there that summarizes our proposed debugging approach and provides some sample code to get started.

Andrew,

While I definitely appreciate the hard work and new features in devappserver2, losing the ability to easily debug your apps is a step backwards. pdb is the builtin python debugger, its what python appengine users will turn to when first running into a problem. If you no longer wish to support pdb, then at the very least, I think google should provide a supported and documented alternative.


--Alex

Kaan Soral

unread,
Mar 22, 2013, 12:35:37 PM3/22/13
to google-a...@googlegroups.com
No one forces anyone to use anything, you can always use previous versions of SDK/Devappserver

They've stated why It's currently not possible to debug with devappserver2, as someone watching the development of devappserver2 closely, I'm sure they will come up with a way to satisfy your debugging needs in future - if possible

I think the illogical-thing to do, at this point, is to bundle the latest version of Devappserver1 with SDK and provide it as an option, but depreciate it at the same time, would solve all current problems

( Apart from all this, as a personal opinion: I've never used a debugger, used many languages, I find print/log statements much more useful, especially with a language like python )

Alex Burgel

unread,
Mar 22, 2013, 1:04:14 PM3/22/13
to google-a...@googlegroups.com
On Friday, March 22, 2013 12:35:37 PM UTC-4, Kaan Soral wrote:
No one forces anyone to use anything, you can always use previous versions of SDK/Devappserver

They've stated why It's currently not possible to debug with devappserver2, as someone watching the development of devappserver2 closely, I'm sure they will come up with a way to satisfy your debugging needs in future - if possible

I think the illogical-thing to do, at this point, is to bundle the latest version of Devappserver1 with SDK and provide it as an option, but depreciate it at the same time, would solve all current problems

( Apart from all this, as a personal opinion: I've never used a debugger, used many languages, I find print/log statements much more useful, especially with a language like python )

Personal opinion: using a debugger is fun, give it a try :-)

You're right that I can still use the previous version, and I will do that until they remove it or fix this. But from statements in this post and in the issue tracker, I get the feeling that "fix this" will mean adding a few hooks and then letting us all figure it out on our own. I don't think thats a good option.

Even if you don't use them, debuggers are pretty important to writing programs. So I want to express my opinion to the good folks at google that an out-of-the-box, no-fancy-IDE debugging solution (kinda like pdb) should be part of the appengine SDK.

Kaan Soral

unread,
Mar 22, 2013, 3:09:08 PM3/22/13
to google-a...@googlegroups.com
Thanks for the suggestion, I will give it a try :)

Actually, the async nature of the devappserver also makes the logs just a little hard to read, so I'm guessing the upcoming versions will come with many new features, for example there seems to be a new log sqlite file that is not yet used, If anyone wants to star it: http://code.google.com/p/appengine-devappserver2-experiment/issues/detail?id=33&thanks=33&ts=1360264428

Renzo Nuccitelli

unread,
Mar 22, 2013, 4:01:13 PM3/22/13
to google-a...@googlegroups.com
 +1 for debuging. I will keep the 1.7.5 version until Pycharm implements de debuger.

 Renzo

Bryce Cutt

unread,
Mar 22, 2013, 4:31:39 PM3/22/13
to google-a...@googlegroups.com
Debugging is an absolute necessity. I use Eclipse PyDev (well, actually Aptana Studio) specifically because of the graphical debugger. I feel blind when I don't have a debugger; It's one of the most useful tools I know of for development. I'm sticking to 1.7.5 for this (and other) reasons.

- Bryce


On Tuesday, March 19, 2013 5:57:02 PM UTC-7, Andrew Jessup wrote:

Adam Bradley

unread,
Mar 22, 2013, 8:20:29 PM3/22/13
to google-a...@googlegroups.com
+1 on getting debugging working. I too use Aptana and now its no longer working, so I will be reverting back to 1.7.5. However, doing this also scares me because I do not want to be left behind and not stay up with the latest and greatest features you guys are putting together. I love GAE, but putting together documentation on how to debug would really help me out, and since I do not know the in's and out's of the new dev server I'd rather not try to figure it all out by myself (or if its even possible I can't tell).

Love what you guys are doing and want to stay with the updates, but 1.7.6 is the first update that's put my development to a halt. Any advice on how to get debugging working would be great.

Thank you,
Adam

D X

unread,
Mar 23, 2013, 5:45:57 PM3/23/13
to google-a...@googlegroups.com
I used to use pdb.set_trace() because the debugger was so slow.

It got monkeypatched to work in one of the more recent builds, but now that there's multiple processes running and stdout is closed, it doesn't work anymore.

This is a huge problem.  Part of the reason I like using App Engine was that it was easy to develop on.  Not so much when the only way you can debug is via a multithreaded log trace.

Mehul Kar

unread,
Mar 24, 2013, 5:51:35 PM3/24/13
to google-a...@googlegroups.com
Should I wait on upgrading then? I use pdb.set_trace() pretty heavily. 

Amy Unruh

unread,
Mar 24, 2013, 6:02:16 PM3/24/13
to google-a...@googlegroups.com
For those discussing whether or not to upgrade to 1.7.6— the legacy dev app server remains available in the Python SDK as 'old_dev_appserver.py'. So you can upgrade to 1.7.6 and still access it.

 -Amy


--

payala

unread,
Apr 9, 2013, 10:15:00 AM4/9/13
to google-a...@googlegroups.com

  • We’ve cleaned up many command line flags and arguments

Is there any place where we can see which arguments have been removed and workarounds?

And, please, I can't believe debugging is not possible now... that is really really incredible, I will stay with 1.7.5 until you have found a way to do debugging.

Cheers

Adam Bradley

unread,
Apr 20, 2013, 5:12:42 PM4/20/13
to google-a...@googlegroups.com
Has anyone come up with a way to debug? I obviously must be missing something because I find it hard to believe its no longer possible. For anyone who has no issues debugging, please share any advice you may have on how to setup local dev environments (since 1.7.6 that is). Thanks

timh

unread,
Apr 20, 2013, 7:23:11 PM4/20/13
to google-a...@googlegroups.com
I posted this a few weeks ago, on how to debug 1.7.6  using Boa. 

 https://groups.google.com/forum/#!topicsearchin/google-appengine/Boa/google-appengine/-m00Qz4Vc7U

Its not hard to do

The same approach will no doubt work with winpdb, 

T

David

unread,
Apr 22, 2013, 11:20:18 AM4/22/13
to google-a...@googlegroups.com
Looks like dev_appserver2.py does NOT work on 32 bit Windows.
It crashes 32 bit python on the huge id()


On Tuesday, March 19, 2013 7:57:02 PM UTC-5, Andrew Jessup wrote:
In the 1.7.6 release of App Engine we’ve included a major upgrade to the Development Server in the Python SDK, designed to make development faster and more faithfully reproduce the production App Engine runtime environment. The new version more faithfully simulates elements of the production App Engine environment, such as concurrent requests and backends, and in most cases this will also mean your apps run faster. We previously included this in the 1.7.5 release as dev_appserver2.py, in this release it has become the default dev_appserver.py

While we're confident the improvements are substantial, as with any major change, some things work a little differently with the new dev_appserver. Here’s a few changes to watch out for:
  • Your local development admin console will run on it’s own server and port, rather than under /_ah/admin
  • We’ve changed the defaults for new applications, such as using HRD rather than Master/Slave, and using SQLlite to emulate the datastore.
  • We’ve cleaned up many command line flags and arguments
  • Python 2.5 applications are no longer officially supported on the new Development Server in line with our deprecation announcement (although they may still be run)

Kaan Soral

unread,
Apr 22, 2013, 3:06:20 PM4/22/13
to google-a...@googlegroups.com
there is a setting for id allocation, didn't check the actual setting right now, but those big ids recently started showing up, 'scattered' should be the default, you can change it to sequential or something like that, it might be worthwhile to check --help for this

I see a lot of chatter around the new id policy, but nothing changed (break) for me
Message has been deleted

michael kariv

unread,
Apr 23, 2013, 1:43:24 AM4/23/13
to google-a...@googlegroups.com
Debugging is still a problem.
This link below was posted by Brian Quilan in a different thread 

Here is where I am trying to solve the problem:
1. breakpoints DO WORK if you set them in dev_appserver python code somewhere before an instance starts for your app.
2. The hook to your own python script before spawning an instance DOES occure.
3. However within my hooked script I am unable to connect to pydev - I see weird things in my console.
I am working on it will keep ya all posted

Andrew Jessup

unread,
Aug 9, 2013, 7:07:30 PM8/9/13
to google-a...@googlegroups.com

Good news everyone!


We just wanted to give you a quick update on PDB debugging support in the new dev_appserver.


One of the key design goals of the new dev_appserver was to provide a more faithful emulation of how your application performs in production - including servicing each incoming request in a separate thread or process. This not only makes your local development faster, but also means issues (such as race conditions) that may arise from concurrent requests executing in parallel can be more easily identified and corrected.


However a development environment with multiple concurrent threads and processes can present a challenge when using a command line debugger such as PDB. We’ve been working on reconciling our design goal of a dev_appserver that supports production-like concurrency while still allowing developers to use tools like PDB to introspect and debug their code in a sensible, serialised way.


We’ve made some progress on this front recently, and wanted to share it with you.


The latest release 1.8.3 of the Python SDK supports PDB. That is you can add a line such as:


import pdb; pdb.set_trace();


into your code, and dev_appserver2 will break at this point, and drop into the PDB REPL, allowing you to debug your code from the command line.


However, a note of caution - if you make multiple simultaneous requests that invoke pdb.set_trace(), then two debugging sessions will start concurrently and both will send output to STDOUT. This can lead to unpredictable debugging experiences.


To help prevent this, you can also disable some of the dev_appserver’s multi-threading and multi-processing support.

  • Multi-threading can be disabled by modifying your app.yaml and setting thread_safe to false.

  • For a given module, you can disable multi-processing using the --max_module_instances=1 flag.


Using these techniques you can serialize your requests. Note that, if you are using App Engine modules in your application, the dev_appserver will still serve multiple requests simultaneously for different modules.


We hope you enjoy this early support for pdb in dev_appserver2. Please let us know what you think.

Stefan Roesch

unread,
Nov 4, 2013, 7:50:38 PM11/4/13
to google-a...@googlegroups.com
+1.  Debugger still can't attach from Eclipse by setting breakpoints the way you used to with anything past 1.7.5.

Anyone had any success?
Reply all
Reply to author
Forward
0 new messages