Patch to submit for --clear_datastore option

[frank....@mycommunity.net]

Hi,

I have a tiny patch to submit which adds the required boolean value to dev_appserver's --clear_datastore option. It seems that the dev_appserver's command line options have changed slightly and gaedriver needs to be updated. Without the patch it is basically not possible to start dev_appserver with gaedriver.

I had a look on codereview.appspot.com but I couldn't find the gaedriver project. In fact I was wondering if the contributors guide [1] was just boilerplate and in fact very few external contributions have been accepted.

To be honest the patch is so small I'm happy for a project member to commit the patch.

The patch is basically this:

--- a/py/src/gaedriver/gaedriver.py Thu Nov 29 07:52:21 2012 -0800

+++ b/py/src/gaedriver/gaedriver.py Fri Jan 09 15:52:01 2015 +0000

@@ -634,6 +634,7 @@

 argv.append(port_option)

 if self.clear_datastore:

 argv.append('--clear_datastore')

+ argv.append('true')

 argv.extend(self.options_)

 argv.append(self.config.app_dir)

 return argv

Please let me know how you would like to accept the patch.

Thanks,

Frank

[1] https://code.google.com/p/gaedriver/wiki/ContributorsGuide 

[Robert Schuppenies]

Hi Frank. Thanks for the proposed fix. Before we move ahead, can you help me try to reproduce the issue you see? Could you send me exactly what you are doing, what you see and what you expect to see instead?

I've tried using --clear_datastore flag with devappserver and it seemed to work. Here is what I did:

- start devappserver with any app

- go to the admin interactive console (by default :8000/console)

- run the following snippet:

"""

from google.appengine.ext import db

class Foo(db.Model):

  s = db.StringProperty()

Foo(s='1').put()

"""

- go to :8000/datastore and check that you see an entity

- restart devappserver with the --clear_datastore flag

- go to :8000/datastore - the entity should be gone.

thanks,

robert

 

--
You received this message because you are subscribed to the Google Groups "gaedriver" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gaedriver+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Frank Wilson]

Hi Robert,

Thanks for your reply, I’m glad to see you are still monitoring the
list and I am surprised there isn’t more community activity around
your tools!

The problem isn’t with with dev_appserver, it is more with gaedriver.

I’m using a simplified version of the aeta run_tests.py code to run
tests for my app with gaedriver to launch the app. Just as in the aeta
code, aeta is the test runner.

e.g.

E2E_HOSTNAME = 'localhost:8087'

def run_tests(sdk_dir):
config = gaedriver.Config(app_id=‘myapp-e2e-test',
app_dir='.',
cluster_hostname=E2E_HOSTNAME,
sdk_dir=sdk_dir)
app_token = gaedriver.setup_app(config)
aeta_url = 'http://%s/tests' % config.app_hostname
local_client.main(aeta_url)
gaedriver.teardown_app(config,app_token)

def main():
sdk_dir = os.getenv('GAE_SDK_DIR')
run_tests(sdk_dir)

if __name__ == '__main__':
main()

When this code is run python complains that the application server
isn’t running.


File "/Users/frank/Sources/myapp/aeta/local_client.py", line 292, in urlopen
return self._opener.open(url, data)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py",
line 404, in open
response = self._open(req, data)

… abridged …

return self.do_open(httplib.HTTPConnection, req)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py",
line 1184, in do_open
raise URLError(err)
urllib2.URLError: <urlopen error [Errno 61] Connection refused>

This is because gaedriver is invoking dev_appserver without a boolean
argument for —clear_datastore:

e.g. dev_appserver.py --clear_datastore .

which results in:

ValueError: '.' unrecognized boolean; known booleans are 'true',
'yes', '1', 'false', 'no', '0'

The invocation should be:

dev_appserver.py --clear_datastore true .

Where dev_appserver continues normally with the behaviour you described.

I assume this happened because dev_appserver’s command line interface
has changed since gaedriver was written.

To be clear I am using version 1.9.17 of the SDK according to my VERSION file.

Subsequent to this issue I also had another problem with gaedriver
where the above code would hang around the time that main() should
exit. This appears to be because the call to os.kill() the
devappserver subprocess is not successful. I resolved this by using
os.killpg() instead.

--- a/py/src/gaedriver/gaedriver.py
+++ b/py/src/gaedriver/gaedriver.py
@@ -407,14 +407,15 @@
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
- env=_get_env_vars())
+ env=_get_env_vars(),
+ preexec_fn=os.setsid)
self.pid_ = popen.pid
(self.stdout_, self.stderr_) = popen.communicate(self.stdin)
self.returncode_ = popen.returncode

def stop(self):
if self.pid_:
- os.kill(self.pid, signal.SIGKILL)
+ os.killpg(self.pid, signal.SIGTERM)


class AppcfgThread(ClientThreadBase_):

(I also used SIGTERM instead because SIGKILL is more abrupt than
strictly necessary, but maybe there is a case for using SIGKILL as a
fallback. In any case it is probably not essential to fix the issue.)

I don’t intend to overload you with inline patches like this! When I
posted yesterday I wasn’t very hopeful of a reply, but knowing there
might be someone to work with would obviously encourage me to make the
effort to learn to follow what ever is the normal process for
contributions is. I’m not very familiar with mercurial, it seems there
are multiple ways to create branches in mercurial and I am not sure
what is the normal way.

I’ve created a couple entries on google code to track these issues:

https://code.google.com/p/gaedriver/issues/detail?id=3

https://code.google.com/p/gaedriver/issues/detail?id=4


Thanks,


Frank

[Robert Schuppenies]

Hi Frank.

I've tried what you described and you are right. This is currently broken. The reason is that the new devappserver handles things differently than the old one. 

The question is: why didn't we notice this earlier? I have to say we dropped the ball on pushing new releases of our internal repository (which has the right fix) and I'm sorry we did so. Now what?: I will work with the new owners of the library to pick up the ball and put a more regular release cadence in place.

This will not happen within the next weeks, but if you haven't heard from us by the end of the month, please reach out again. Are you okay working with a locally patched version in the meantime?

thanks for bearing with us,

robert

[Frank Wilson]

Hi Robert,

I'll look forward to a new release of gaedriver. I'm fine with using a
locally patched version for now.

Thanks for gaedriver and aeta, I think they are great tools!

Frank

On 14 January 2015 at 22:23, 'Robert Schuppenies' via gaedriver

--
Frank Wilson
VP Engineering, myCommunity
e: frank....@mycommunity.net
m: +44 (0)7748 155457
w: www.mycommunity.net
tw: @MyCommunityNet

Envolve Technology Limited. Registered No: 8548197. Registered Office:
c/o 14a Forest Gate, Pewsham, Chippenham, Wiltshire, SN15 3RS.
This e-mail and the information it contains may be confidential,
legally privileged and protected by law. Access by the intended
recipient only is authorised. If you are not the intended recipient,
please notify the sender immediately and delete this e-mail from your
system. Any review, distribution, reproduction, publication or other
use of this e-mail by persons or entities other than the intended
recipient is prohibited.