Deleted PubSub push subscription but the messages are still being sent

[Garrett Hunyadi]

I have a PubSub push subscription set up where I did a load test.  At one point along the way, I started to get QPS errors.  As I had sufficient results at that point, I decided to shut the system down.  In order to terminate the requests, I killed the topic and the subscription.  However, the messages continue to hit my AppEngine instance.  As there are failures, they are also being resent which is causing a large amount of traffic.  It's two days later and I'm still receiving calls.

I've deleted the topic and the subscription and I don't see why the messages are still being sent, nor what I can do to kill them.  Am I missing some option here?

Thanks,

Garrett Hunyadi

[Nick]

Hey Garret,

I'm sorry to hear that's happening. Please email me (at pay...@google.com) the project ID and as much information about the subscription / topic as you can manage to find, and I'll look into getting this resolved as soon as possible. 

Regards,

Nick

Cloud Platform Community Support

[Garrett Hunyadi]

Hi Nick,

  Thanks for the follow up.   The deluge has finally subsided, though I’m left with the the question of what happened and how to prevent it.  What should happen if there is a push subscription and it is deleted.  Should all the outstanding work be terminated, or is there some reason why it shouldn’t?

  My project is eig-insites and, unfortunately, I can’t recall the name of the push topic/subscription. The topic may have been "projects/eig-insites/topics/load-time-request”, though I can’t recall if I used a temp topic for that one.  This was the only activity over the weekend, however, so anything in the logs at that time would relate to this issue.

  It looks like the issue cost around $500.  Is this something that can be reduced or eliminated?  

  I see that the majority of the cost is in “Out Bandwidth.”  (App Engine  Out Bandwidth 3,542.74 GB $425.13)   For this performance test, the App Engine endpoint implementation is triggered from the push subscription, waits five seconds (which is, on average, the expected duration of our workload) and writes a “result” to a return topic.   Isn’t this all internal, such that there would be no “out bandwidth”?    Ultimately, our system gathers data from domains, involving pinging the domain and calling other APIs.  I also did testing that pings the domains, but it doesn’t seem like the traffic for those tests would be that significant.  Is there a way for me to better monitor the usage and cost for particular tests or jobs?   

  Apologies for the number of questions, I’m new in using Google Cloud.

Cheers,

Garrett

-- 
You received this message because you are subscribed to the Google Groups "Google Cloud Pub/Sub Discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-dis...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-pubsub-discuss/1c44be9b-e6b6-4e66-96d7-722099333db4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Nick]

Hey Garrett,

Thanks for your patience and apologies for the late reply. I believe I can answer your questions in order.

As for the original behaviour, there are two options that I can see: t's either expected and needs to be documented (messages will continue to be sent if they were previously published, even after deletion of the topic / subscription) or it's not expected and we should be able to launch a technical investigation into the root cause. I should be able to provide advice on which side of this situation we fall on soon.

As for the outgoing bandwidth, this encompasses both requests made from instances (for example, calls to "publish") and also request responses, along with UrlFetch requests, where the boundary of "outgoing" is defined as the App Engine app-server. It appears plausible that given the size of the test, all the combined activity could have generated this amount of usage. As a small note for monitoring, the request paths on a version of a service can be inspected in the "App Engine > Dashboard" view under "Current Load", and this can be used to estimate outgoing bandwidth if you know the statistical distribution of response sizes, any outgoing requests spawned by the handler, etc.

Now, the final question is whether, as a result of undocumented behaviour which seems, in honesty, counter-intuitive, you could not be eligible for a billing credit, that's not something I'm qualified to speak on, and would be best to approach by contacting Billing Support. I'm afraid I'm not able to move the information to the right people from my position otherwise I'd of course send the request along and get that started on your behalf.

So, I'll get back to this thread with our findings, but in the meantime, feel free to contact Billing!

Cheers,

Nick
Cloud Platform Community Support 

On Thursday, July 28, 2016 at 12:04:55 PM UTC-4, Garrett Hunyadi wrote:

Hi Nick,

  Thanks for the follow up.   The deluge has finally subsided, though I’m left with the the question of what happened and how to prevent it.  What should happen if there is a push subscription and it is deleted.  Should all the outstanding work be terminated, or is there some reason why it shouldn’t?

  My project is eig-insites and, unfortunately, I can’t recall the name of the push topic/subscription. The topic may have been "projects/eig-insites/topics/load-time-request”, though I can’t recall if I used a temp topic for that one.  This was the only activity over the weekend, however, so anything in the logs at that time would relate to this issue.

  It looks like the issue cost around $500.  Is this something that can be reduced or eliminated?  

  I see that the majority of the cost is in “Out Bandwidth.”  (App Engine  Out Bandwidth 3,542.74 GB $425.13)   For this performance test, the App Engine endpoint implementation is triggered from the push subscription, waits five seconds (which is, on average, the expected duration of our workload) and writes a “result” to a return topic.   Isn’t this all internal, such that there would be no “out bandwidth”?    Ultimately, our system gathers data from domains, involving pinging the domain and calling other APIs.  I also did testing that pings the domains, but it doesn’t seem like the traffic for those tests would be that significant.  Is there a way for me to better monitor the usage and cost for particular tests or jobs?   

  Apologies for the number of questions, I’m new in using Google Cloud.

Cheers,

Garrett

On Jul 27, 2016, at 7:44 PM, 'Nick' via Google Cloud Pub/Sub Discussions <cloud-pubsub-discuss@googlegroups.com> wrote:

Hey Garret,

I'm sorry to hear that's happening. Please email me (at pay...@google.com) the project ID and as much information about the subscription / topic as you can manage to find, and I'll look into getting this resolved as soon as possible. 

Regards,

Nick

Cloud Platform Community Support

On Tuesday, July 26, 2016 at 4:06:22 PM UTC-4, Garrett Hunyadi wrote:

I have a PubSub push subscription set up where I did a load test.  At one point along the way, I started to get QPS errors.  As I had sufficient results at that point, I decided to shut the system down.  In order to terminate the requests, I killed the topic and the subscription.  However, the messages continue to hit my AppEngine instance.  As there are failures, they are also being resent which is causing a large amount of traffic.  It's two days later and I'm still receiving calls.

I've deleted the topic and the subscription and I don't see why the messages are still being sent, nor what I can do to kill them.  Am I missing some option here?

Thanks,

Garrett Hunyadi

-- 
You received this message because you are subscribed to the Google Groups "Google Cloud Pub/Sub Discussions" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-discuss+unsub...@googlegroups.com.

[Nick]

Hey Garrett,

So, we've looked a little deeper into this and have some initial findings:

We were able to see a remaining active subscription "load-time-request-monitor" with ~2k unacknowledged messages. From the documentation: "Messages that are not acknowledged, are retried indefinitely for up to seven days." Deleting this subscription should work, as it says: "To permanently stop delivery, you should delete the subscription."

Could you verify that the subscription is actually deleted? The command to do this:

$ gcloud beta pubsub subscriptions list --project eig-insites

It appears likely that the subscription was not deleted, which led to the requests continuing.

Cheers,

Nick
Cloud Platform Community Support

On Thursday, July 28, 2016 at 12:04:55 PM UTC-4, Garrett Hunyadi wrote:

Hi Nick,

  Thanks for the follow up.   The deluge has finally subsided, though I’m left with the the question of what happened and how to prevent it.  What should happen if there is a push subscription and it is deleted.  Should all the outstanding work be terminated, or is there some reason why it shouldn’t?

  My project is eig-insites and, unfortunately, I can’t recall the name of the push topic/subscription. The topic may have been "projects/eig-insites/topics/load-time-request”, though I can’t recall if I used a temp topic for that one.  This was the only activity over the weekend, however, so anything in the logs at that time would relate to this issue.

  It looks like the issue cost around $500.  Is this something that can be reduced or eliminated?  

  I see that the majority of the cost is in “Out Bandwidth.”  (App Engine  Out Bandwidth 3,542.74 GB $425.13)   For this performance test, the App Engine endpoint implementation is triggered from the push subscription, waits five seconds (which is, on average, the expected duration of our workload) and writes a “result” to a return topic.   Isn’t this all internal, such that there would be no “out bandwidth”?    Ultimately, our system gathers data from domains, involving pinging the domain and calling other APIs.  I also did testing that pings the domains, but it doesn’t seem like the traffic for those tests would be that significant.  Is there a way for me to better monitor the usage and cost for particular tests or jobs?   

  Apologies for the number of questions, I’m new in using Google Cloud.

Cheers,

Garrett

On Jul 27, 2016, at 7:44 PM, 'Nick' via Google Cloud Pub/Sub Discussions <cloud-pubsub-discuss@googlegroups.com> wrote:

Hey Garret,

I'm sorry to hear that's happening. Please email me (at pay...@google.com) the project ID and as much information about the subscription / topic as you can manage to find, and I'll look into getting this resolved as soon as possible. 

Regards,

Nick

Cloud Platform Community Support

On Tuesday, July 26, 2016 at 4:06:22 PM UTC-4, Garrett Hunyadi wrote:

I have a PubSub push subscription set up where I did a load test.  At one point along the way, I started to get QPS errors.  As I had sufficient results at that point, I decided to shut the system down.  In order to terminate the requests, I killed the topic and the subscription.  However, the messages continue to hit my AppEngine instance.  As there are failures, they are also being resent which is causing a large amount of traffic.  It's two days later and I'm still receiving calls.

I've deleted the topic and the subscription and I don't see why the messages are still being sent, nor what I can do to kill them.  Am I missing some option here?

Thanks,

Garrett Hunyadi

-- 
You received this message because you are subscribed to the Google Groups "Google Cloud Pub/Sub Discussions" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-discuss+unsub...@googlegroups.com.