Unable to receive pushed massages using pubsub

[jawad nammari]

Hello

I am trying to integrate pub sub into my project. I am using this tutorial as basis cloud-pubsub-samples-java

Using the above tutorial I was able to create a topic, subscribe to it, send and receive massages using pull subscription.

However so far I am unable to get push notification to work. The server I am using is in the same project as pubsub. In the logs am getting a 403 error [shown below]

POST /_ah/push-handlers/receive_message?token=XXXXXX HTTP/1.1" 403 162 - "-" "0-dot-versionID-dot-ServiceID-dot-ProejctID.appspot.com"

more information

For endpoint I am using: https://InstanceNumber-dot-VersionName-dot-ServiceName-dot-ProjectName.appspot.com

I am using google flexible environment

Looks like I am need to add authentication/privileges somewhere, but so far I do not where.

[Adam]

How are you calling the push endpoint from the other application? The TL;DR is that you should access the Pub/Sub service with the Google API Client, and authorize using the Application Default Credentials.

The '/_ah/push-handlers/*' endpoint in the sample project has an 'admin' auth constraint, so HTTP calls to it need be authenticated: 

    <auth-constraint>
      <role-name>admin</role-name>
    </auth-constraint>

Authenticated HTTP is handled transparently by the API client in the example code. It is first set up in the getClient() method in PubSubUtils.java using the Application Default Credentials when the PubSub client object is first created:

GoogleCredential credential = GoogleCredential.getApplicationDefault();
if (credential.createScopedRequired()) {
    credential = credential.createScoped(PubsubScopes.all());
}
HttpRequestInitializer initializer =
        new RetryHttpInitializerWrapper(credential);
return new Pubsub.Builder(httpTransport, jsonFactory, initializer)
        .setApplicationName(APPLICATION_NAME)
        .build();

The "/_ah/push-handlers/receive_message" endpoint is registered once in the setupSubscription() method in InitServlet.java:

PushConfig pushConfig = new PushConfig()
        .setPushEndpoint(PubsubUtils.getAppEndpointUrl());

Calls to that endpoint happen in SendMessageServlet.java:

Pubsub client = PubsubUtils.getClient();
...
client.projects().topics()
        .publish(fullTopicName, publishRequest)
        .execute();

Also, there's no reason to try to target a specific instance number most of the time. Instances can be transient and disappear. Usually https://VersionName-dot-ProjectName.appspot.com is enough (if you're targeting the default service).

[jawad nammari]

Thanks for the reply Adam.

I removed the admin auth altogether yet I am still getting the 403 status code.

The application itself is the sender and receiver, I am using the code to sync data between instances hence the use of instance numbers inside the endpoint.

According to google docs, using app engine on the same project as pubsub should not require any additional authorization/authentication.

[jawad nammari]

After some more investigation, I managed to receive pushed messages by changing "/_ah/push-handlers/receive_message" to something a different servlet path which does not include "/_ah/push-handlers/"

However the push messages are too slow to reach the client, average of 200 seconds from push to receive, seem most of the time is wasted in the network, whereas the sending and processing parts are fast.

[Kir Titievsky]

Hi, Jawad,

200 seconds sound wrong. I wonder if your subscriber is still working through a backlog of messages.  Could you try deleting the subscription and recreating it?  We should anticipate ~1 second times.

Kir Titievsky

Product Manager

Google Cloud Pub/Sub

On Sun, Nov 6, 2016 at 11:42 AM, jawad nammari <jawad....@gmail.com> wrote:

After some more investigation, I managed to receive pushed messages by changing "/_ah/push-handlers/receive_message" to something a different servlet path which does not include "/_ah/push-handlers/"

However the push messages are too slow to reach the client, average of 200 seconds from push to receive, seem most of the time is wasted in the network, whereas the sending and processing parts are fast.

--
You received this message because you are subscribed to the Google Groups "Google Cloud Pub/Sub Discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-discuss+unsub...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-pubsub-discuss/ee621be3-59d0-4363-bd8d-1bffab1fd5bc%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Kir Titievsky | Product Manager | Google Cloud Pub/Sub 

[jawad nammari]

I tried deploying multiple times using different topics/subscriptions still getting the same average times.

I am only trying very few number of messages with new topics so 

 

Is there a way to debug the network part from app engine? So I can get more info, share it?

[Kir Titievsky]

Are you seeing any errors in the App Engine logs in Cloud Logging?  Usually, in these cases the delay comes from the webhook crashing.  Not saying this is your case, but it'd be great to eliminate.  

--

You received this message because you are subscribed to the Google Groups "Google Cloud Pub/Sub Discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-discuss+unsub...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-pubsub-discuss/bc9566dc-25d4-4ed7-ab44-c2d21331535c%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[jawad nammari]

After some more debugging.

I managed to get the network time to ~ms. 

I did it by creating a push subscription from scratch.

While I get high latency when i create a pull subscription and change it to push one later on.

How do I create a subscription

Subscription subscription = new Subscription().setTopic(PubSubUtils.getFullTopicName()).setAckDeadlineSeconds(ACKNOWLEDGE_TIME);

 if (subType == SubscriptionType.PUSH) {

  PushConfig pushConfig = new PushConfig().setPushEndpoint(PubSubUtils.getEndpointUrlForSubscription());

  subscription.setPushConfig(pushConfig);

 }

 subscriptions.create(PubSubUtils.getFullSubscriptionName(), subscription).execute();

}

How do I switch from pull to push

Subscriptions subscriptions = client.projects().subscriptions();

ModifyPushConfigRequest request = new ModifyPushConfigRequest();

PushConfig pushConfig = new PushConfig().setPushEndpoint(PubSubUtils.getEndpointUrlForSubscription());

request.setPushConfig(pushConfig);

subscriptions.modifyPushConfig(PubSubUtils.getFullSubscriptionName(), request).execute();

Am I doing something wrong? or is it a bug on google side

Again thanks for the help :)

[Kir Titievsky]

Jawad, Thanks for persisting!  To eliminate one more very basic possibility: is the 200s latency for the *first* message after you update the pushConfig() or is it for all subsequent messages?

--
You received this message because you are subscribed to the Google Groups "Google Cloud Pub/Sub Discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-discuss+unsub...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-pubsub-discuss/e5aa2587-f8eb-486d-841f-13f6a2ffdfcf%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[jawad nammari]

I definitely sent multiple subsequent messages with the same 150-200 seconds average on multiple versions. So it is not related to the first message sent. 

However, I did not test sending messages in the order of 1000s, it seemed redundant.

 

[Kir Titievsky]

agreed.  I don't see anything wrong with what you are doing. Will see if I can reproduce the error.

On Mon, Nov 7, 2016 at 10:48 AM, jawad nammari <jawad....@gmail.com> wrote:

I definitely sent multiple subsequent messages with the same 150-200 seconds average on multiple versions. So it is not related to the first message sent. 

However, I did not test sending messages in the order of 1000s, it seemed redundant.

 

--

You received this message because you are subscribed to the Google Groups "Google Cloud Pub/Sub Discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-discuss+unsub...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-pubsub-discuss/8ae22449-c5a0-4ffd-b684-330dbce6f57e%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[jawad nammari]

Adding info to help you reproduce it faster.

1.I am using flex environment. and for endpoint  https://InstanceNumber-dot-VersionName-dot-ServiceName-dot-ProjectName.appspot.com , doing so because the purpose is to sync date between instances of a version.

2. Using the recommended push endpoint "/_ah/push-handlers/..." with admin auth constraint, I did not get any results. No logs to show that the endpoint was even called.

3. Using the recommended push endpoint "/_ah/push-handlers/..." without admin auth constraint,  I did not get any results. Logs show a 403 error for the endpoint

4. Using a custom push endpoint "/..." starting with a pull subscription, then changing it to push one later on. produced very slow results. ~3 minutes delay, mostly from network, sending and processing received messages were fast.

5. I am creating a topic (if it doesn't exist) while the instance is "initializing", with a pull subscription to that topic, then changing it to a push subscription just before the instance is "healthy", after the instance is "healthy" I try to publish/receive messages via push but getting the high delay.

As for your last question

Every time I tried with a new version, most of the time with new topics as well, so I don't think there are any exceptions, and the app engine logs do not show anything unusual. are there logs specific to pubsub I can check for you.

 

[jawad nammari]

Hey Kir, 

Were you able to reproduce the issue. It is important to know the root cause and how to mitigate it. 

Thanks in advance 

[Kir Titievsky]

Hey, Jawad,

We have one plausible theory so far: when a pull subscription is created *and* you issue at least one pull() command a caching mechanism goes into operation that might cause the delay.  If this is correct, than you should not see the latency when you change the pushConfig without ever issuing a pull call.  Does that sound like your situation?

k

--

You received this message because you are subscribed to the Google Groups "Google Cloud Pub/Sub Discussions" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-dis...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-pubsub-discuss/7f888370-093b-4408-ae27-7d089d3e6d07%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[jawad nammari]

It does sound like mine. I will check your suggestion to change the push config without doing a pull on Monday, and reply to you.

If the theory is correct, any estimation on a fix date?

[jawad nammari]

Hey Kir,

I did test the theory - creating a pull subscription then switching to a push subscription without doing any pull. and pubsub is working fine.

Seems the theory is correct. 

Any idea how to proceed? Should I open a ticket to google support?

[Kir Titievsky]

Jawad, thanks for persisting!  To decide whether this is a bug, let's understand your use case better.  Do you anticipate having to switch between pull and push frequently?  Could you tell me more about what makes that necessary?

In general, we try to minimize any particular guarantee on message latency to optimize the service for throughput and latency under different conditions.  Switches between subscription types -- in the cases known to us -- are a rare event that's dramatic enough for extra message latency to not be a concern.  We, of course, fully expect the latency to recover and stabilize after the switch.  That this is a rare occurrence may be a bad assumption, so please let me know.

k

--

You received this message because you are subscribed to the Google Groups "Google Cloud Pub/Sub Discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-dis...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-pubsub-discuss/384c4ccf-df01-4953-83b1-c93c47b8aba9%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[jawad nammari]

Pub/Sub will be used to update cached objects between all instances.

Once a version is initializing, we can't use PUSH subscription because the instance is not healthy, and message will not be forwarded to the instance.

So I rely on PULL subscription until the instance is healthy, then I switch to PUSH subscription, which is what we will be using.

So from desing point, We are only intersted in one switch from pull to push, then we will keep the push until the instance is deleted/stopped.

[Kir Titievsky]

I see. Thank you. Could you tell me why pull does not work once the instance has been initialized?  Do you expect updates to be infrequent (with intervals >> 1 second)?

Also, why not just keep the subscription in push mode the whole time?  Pub/Sub will back off the delivery rate while the instance is not responding, but ramp up quickly once it does.  This way, your logic is simplified. 

k

--

You received this message because you are subscribed to the Google Groups "Google Cloud Pub/Sub Discussions" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-discuss+unsub...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-pubsub-discuss/b3329aef-c9b9-464a-95eb-5594192e3b41%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[jawad nammari]

The messages being sent are infrequent,they are user driven, and PUSH seems to make more sense.

I would rather have all messages pulled (cached objects synced) before the instance is healthy, than wait a certain amount of time after it is healthy so all messages are pushed.

[jawad nammari]

Hey Kir.

Just to be on the same side, is there anyone investigating the issue from Googles' side. Do you think opening a ticket to google support can help make the process faster?

Again your help is appreciated 

[Kir Titievsky]

Jawad, yes.  Moving discussion off list. 

--

You received this message because you are subscribed to the Google Groups "Google Cloud Pub/Sub Discussions" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-dis...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-pubsub-discuss/e58615a4-585b-4b0f-b8c3-616e93a71717%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.