gevent and pubsub

[NP]

Hi,

I'm trying to setup a server sent event connection to listen to messages pushed out to individual subscribers. When a user connects to my application, they subscribe to a topic of their choice and then start listening for messages via subscription:pull. Listening for messages is done via streaming (server sent event).

I run my code on google compute engine using gunicorn. Because the client does a long connection (a server sent event), I have to start gunicorn with gevent to make it asynchronous i.e gunicorn -k gevent. --bind 0.0.0.0:8000

My problem now is that once I start gunicorn with gevent, I get the following error when user tries to create a subscription or tries to pull messages from the subscription

ConcurrentObjectUseError: This socket is already used by another greenlet: <bound method Waiter.switch of <gevent.hub.Waiter object at....

Anybody have any idea how I can resolve this? I've been stumped on this for more than a week now.

Thanks

[Adam]

How are you interfacing with the Pub/Sub API (eg. using google-cloud-python)? This library should be greenlet-safe, but it seems you have some code that uses a gevent socket in a greenlet-unsafe way.

[NP]

I'm using the gcloud library.

Basically my code is like this

from gcloud import pubsub

client = pubsub.Client(<project>)

topic = client.topic(<topic>)

subscription = topic.subscription(<subscription>)

while True:

         messages = subscription.pull(max_messages=10, return_immediately=True) 

         for message in messages:

              yield 'data: %s\n\n' % message[1]

I tried debugging it and it seems it raises the error at the line -  messages = subscription.pull(max_messages=10, return_immediately=True) 

If I add the creation of the subscription to the view which handles the Server Sent Event connection, then it raises the error at the point of creating the subscription i.e. subscription = topic.subscription(<subscription>)

[Adam]

If you're using the gcloud import, you're probably using the old gcloud library from PyPI. You should use the newer google-cloud library that uses the following import:

    from google.cloud import pubsub

The rest of the usage seems to be the same, but there may be some differences.

[NP]

Thanks. Using the newer version of the library i.e. google-cloud did the trick.

I initially didn't want to do that because it says pubsub is in Beta for that version but it doesn't say the same for the gcloud version.

[NP]

Earlier on I posted a reply that everything was now working.

I just noticed a different error with the new library. After code has been running for some time, I get the error

GaxError(Exception occurred in retry method that was not classified as transient, caused by <_Rendezvous of RPC that terminated with (StatusCode.UNAUTHENTICATED, Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.)>)

I wasn't getting this error with the old library and I didn't change anything with OAuth2. In fact I don't make a specific call to OAuth2. My understanding is that once I'm running on Google Compute Engine, Google automatically handles the verification and the Google Application Default Credential

On Monday, April 17, 2017 at 11:45:12 AM UTC-7, Adam wrote:

[Kir Titievsky]

Does this happen on every call?

--
You received this message because you are subscribed to the Google Groups "Google Cloud Pub/Sub Discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-dis...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-pubsub-discuss/cb25d85b-667a-4ee9-aec2-2e5d8a93bc1e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Kir Titievsky | Product Manager | Google Cloud Pub/Sub

[NP]

Currently, each client connects to my application and establishes a Server Sent Event (SSE) link which then starts pulling messages from the queue.  What happens now is that after a few clients connect (currently testing with 3 clients), everything works and messages are pushed out to them as they arrive. After some time, the SSE connection disconnects and when I check the GCE logs, I see the above error and after a few automatic retries, my application then times out

Please note that I was not seeing this error when I was using gcloud library. I only started seeing this when I switched to google-cloud library. I stopped using gcloud library because it couldn't work with Gevents (Greenlets).

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-discuss+unsub...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-pubsub-discuss/cb25d85b-667a-4ee9-aec2-2e5d8a93bc1e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[NP]

In addition to my earlier reply, I also found this - https://github.com/GoogleCloudPlatform/google-cloud-python/issues/3212

There seems to be no solution yet

On Friday, April 21, 2017 at 7:30:30 AM UTC-7, Kir Titievsky wrote:

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-discuss+unsub...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-pubsub-discuss/cb25d85b-667a-4ee9-aec2-2e5d8a93bc1e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Kir Titievsky]

This makes it look like a regression, NP.  Not a happy state, but it sounds like we have reproduce instructions and a workaround (recreate the client after catching the UNAUTHENTICATED exception.  More generally, we are making significant investments into the python client library over the next couple months.  

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-discuss+unsubscrib...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-pubsub-discuss/cb25d85b-667a-4ee9-aec2-2e5d8a93bc1e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Kir Titievsky | Product Manager | Google Cloud Pub/Sub

--
You received this message because you are subscribed to the Google Groups "Google Cloud Pub/Sub Discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-discuss+unsub...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-pubsub-discuss/67e18f97-3d6c-4050-9dce-4f64c16ef43d%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[NP]

Is there a link to some sample code?

Secondly, is there an ETA on when updates to the python client library will be available?

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-discuss+unsub...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-pubsub-discuss/cb25d85b-667a-4ee9-aec2-2e5d8a93bc1e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Kir Titievsky | Product Manager | Google Cloud Pub/Sub

--
You received this message because you are subscribed to the Google Groups "Google Cloud Pub/Sub Discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-discuss+unsub...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-pubsub-discuss/67e18f97-3d6c-4050-9dce-4f64c16ef43d%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Adam]

What sample code would you need? Do you mean wrapping your code in a try / except to recreate the client if it fails?

[NP]

Hi,

I wasn't sure of the specific 'exception' to trap for but Kir has provided sample code for me.

However, recreating the subscription doesn't always work. I used an exponential backoff and sometimes I have to try about 3 to 4 times before it works. Other times it won't work even after 4 retries.

[Kir Titievsky]

NP, could you please provide project & subscription names as well as rough time range when you were trying these out?  Having to retry subscription creation this many times does not sound normal.

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-discuss+unsubscrib...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-pubsub-discuss/cb25d85b-667a-4ee9-aec2-2e5d8a93bc1e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Kir Titievsky | Product Manager | Google Cloud Pub/Sub

--
You received this message because you are subscribed to the Google Groups "Google Cloud Pub/Sub Discussions" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-discuss+unsubscrib...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-pubsub-discuss/67e18f97-3d6c-4050-9dce-4f64c16ef43d%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Kir Titievsky | Product Manager | Google Cloud Pub/Sub 

--

You received this message because you are subscribed to the Google Groups "Google Cloud Pub/Sub Discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-discuss+unsub...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-pubsub-discuss/fc64f5e1-003a-4d8f-b96d-5cbff0da973c%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[NP]

I tried at different times between Friday (starting from after you provided the code to me) - Sunday. My code ran most of Friday (over night) and I shut down my VM instance on Saturday morning when I checked the logs and saw it had errored out sometime during the night. I initially had a maximum try of 3 after every 5 seconds before switching to exponential backoff with a maximum of 5 tries.

As I said before, sometimes it works after 2 - 4 tries, other times it won't. Generally, it doesn't work after just 1 try - it typically takes at least 2 tries. In addition, the frequency of occurence is quite high. Never had this problem with the old library and I'm using the same code.

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-discuss+unsub...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-pubsub-discuss/cb25d85b-667a-4ee9-aec2-2e5d8a93bc1e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Kir Titievsky | Product Manager | Google Cloud Pub/Sub

--
You received this message because you are subscribed to the Google Groups "Google Cloud Pub/Sub Discussions" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-discuss+unsub...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-pubsub-discuss/67e18f97-3d6c-4050-9dce-4f64c16ef43d%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Kir Titievsky | Product Manager | Google Cloud Pub/Sub 

--
You received this message because you are subscribed to the Google Groups "Google Cloud Pub/Sub Discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-discuss+unsub...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-pubsub-discuss/fc64f5e1-003a-4d8f-b96d-5cbff0da973c%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[George (Cloud Platform Support)]

NP, is your issue solved at this moment in time? Otherwise, the requested information should enable progressing with our investigation.