Cloud Datastore Emulator failing to keep up with Python App Engine unit tests

[Jon Grover]

I'm not sure if this is the right place for such a question, but I'm having a difficult time finding an official support channel for the Cloud Datastore Emulator.

I'm working on migrating a Python 2 App Engine app from NDB to Cloud NDB as a precursor to migrating to Python 3. Per recommendations elsewhere, I've been replacing my usage of Testbed with the local Datastore Emulator, and for the most part that's been working out. What I'm running into is an error that starts popping up about halfway through the 580 unit tests in our test suite. The error message would seem to suggest that the emulator isn't capable of keeping up with the rapidfire reset requests coming from the test setUp() method.

All of my tests derive from a base class with the following setUp() method:

NDB_CLIENT = ndb.Client(project='myproject')
NDB_CLIENT.host = 'localhost:8089'
NDB_CLIENT.secure = False

class MyTestCase(unittest.TestCase):
 def setUp(self):
 self.testapp = webapp2.WSGIApplication([])
 # clear datastore
 requests.post('http://localhost:8089/reset')
 self.ndb_context = NDB_CLIENT.context()
 self.ndb_context.__enter__()

I'm running the Cloud Datastore in a terminal window with the following command:

$ gcloud beta emulators datastore start --no-store-on-disk --consistency=1.0 --host-port=localhost:8089

About halfway through the test suite, the tests begin failing with this error:

ERROR: test_a_thing (my_test.Test)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/my_test.py", line 11, in setUp
  File "/Users/myuser/Projects/MyProject/test/_base_test.py", line 29, in setUp
  File "/Users/myuser/Projects/MyProject/lib/requests/api.py", line 119, in post
  File "/Users/myuser/Projects/MyProject/lib/requests/api.py", line 61, in request
  File "/Users/myuser/Projects/MyProject/lib/requests/sessions.py", line 530, in request
  File "/Users/myuser/Projects/MyProject/lib/requests/sessions.py", line 643, in send
  File "/Users/myuser/Projects/MyProject/lib/requests/adapters.py", line 516, in send
ConnectionError: HTTPConnectionPool(host='localhost', port=8089): Max retries exceeded with url: /reset (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x113715610>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known',))

If I run any of these test files by themselves, they all pass. Does anyone have any suggestions on how I could troubleshoot or improve this?

[Jim Morrison]

Hi Jon,

 Your stackoverflow question (https://stackoverflow.com/questions/60538803/google-cloud-datastore-emulator-refusing-http-based-reset-requests-halfway-throu) is probably the best play to get a single answer.

 Unfortunately, without context of what the code in sessions.py or adapters.py, it's difficult to help.

--
You received this message because you are subscribed to the Google Groups "Google Cloud Datastore" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gcd-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gcd-discuss/a4670f6b-68b9-489d-a67b-9dfdef570473%40googlegroups.com.

--

Jim

[Jon Grover]

Thanks, Jim. I'll add this context on SO as well, but my sessions.py and adapters.py are unmodified from what's on Github.

I'm happy to provide additional code from my own code base, but as I said it's difficult to know what to include since it doesn't appear strictly tied to any individual test.

On Thursday, March 5, 2020 at 2:30:54 PM UTC-8, Jim Morrison wrote:

Hi Jon,

 Your stackoverflow question (https://stackoverflow.com/questions/60538803/google-cloud-datastore-emulator-refusing-http-based-reset-requests-halfway-throu) is probably the best play to get a single answer.

 Unfortunately, without context of what the code in sessions.py or adapters.py, it's difficult to help.

To unsubscribe from this group and stop receiving emails from it, send an email to gcd-d...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/gcd-discuss/a4670f6b-68b9-489d-a67b-9dfdef570473%40googlegroups.com.

--

Jim