PubSub publish message Error:429 Too Many Requests

[贺晨钊]

com.google.api.client.googleapis.json.GoogleJsonResponseException: 429 Too Many Requests

{

  "code" : 429,

  "errors" : [ {

    "domain" : "global",

    "message" : "Request throttled due to user QPS limit being reached.",

    "reason" : "rateLimitExceeded"

  } ],

  "message" : "Request throttled due to user QPS limit being reached.",

  "status" : "RESOURCE_EXHAUSTED"

}

at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)

at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)

at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)

at com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:321)

at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1065)

at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419)

at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)

at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)

at com.pubsub.publishLoadDateDemoThread.publishMessage(publishLoadDateDemoThread.java:110)

at com.pubsub.publishLoadDateDemoThread$SendMessagJob.run(publishLoadDateDemoThread.java:134)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at java.lang.Thread.run(Thread.java:745)

[Nicholas (Cloud Platform Support)]

You appear to have encountered a rate limit for Cloud Pub/Sub as documented in Quotas.  Feel free to submit the linked form on that page allowing you to request greater quotas.  If not, you may have to consider distributing your publish API calls over time to avoid the rate limit.  I'd suggest the former.

[贺晨钊]

I think I did not exceed the pubsub quotas:

Publisher throughput	unlimited	10,000,000 kB per 100 seconds (100 MB/s)
Subscriber throughput	unlimited	20,000,000 kB per 100 seconds (200 MB/s)
Administrative operations	10,000 per 100 seconds (100 ops/s)	1,000 per 100 seconds (10 ops/s)

i write 10 threads to publish messages,each PublishRequest contains 500 messages,soon the error appears.

this is my code:

PublishRequest publishRequest = new PublishRequest();

publishRequest.setMessages(pubMessages);   //pubMessages is a List which contain 500 PubsubMessage.

client.projects().topics().publish(topic,publishRequests).execute().

[Kir Titievsky]

Sharing the solution with the broader discussion list:

My theory is that you are using the gcloud.pubsub client library with default application credentials.

The implementation there makes OAuth super simple at the expense of using shared quota.  It's actually using a project id that's different from yours to authenticate & get quota.

If I'm right, you will fix the spikes by creating and downloading service account credentials and forcing the client library to use those for the connection.

 In python, this looks like:

c = pubsub.Client(o.project).from_service_account_json('/Users/kir/Downloads/kir-learns-cloud-931199eea5d9.json')
topic = c.topic(o.topic)
print "Using client_id %s"%c.connection.credentials.client_id

In node, you can probably do something like:

var gcloud = require('gcloud')({
    projectId : config.get('$YOUR_PROJECT_ID'),
    keyFilename : config.get('/Users/kir/Downloads/kir-learns-cloud-931199eea5d9.json')
  });
       var pubsub = gcloud.pubsub();
       var topic = pubsub.topic(topicName);

Show quoted text

-- 

Kir Titievsky | Product Manager | Google Cloud Pub/Sub

--
You received this message because you are subscribed to the Google Groups "Google Cloud Pub/Sub Discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-discuss+unsub...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-pubsub-discuss/b7d63374-9fdf-46d3-b273-e2b0f522a4dd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.