Why is PubSub always creating 60 permanent threads?

[Joshua Fox]

My application publishes a low volume of messages (1 every few seconds at most), with no other use of PubSub. It does not subscribe. We are using simple Java code as in this sample. The Publisher object is retained for the lifetime of the process, as recommended here.

On first use, PubSub creates 60 threads that stay live permanently, like these:

"grpc-default-worker-ELG-1-1 Id\u003d115 RUNNABLE (in native)...

"grpc-default-worker-ELG-1-10 Id\u003d160 RUNNABLE (in native)":...

....

"Gax-16 Id\u003d141 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24f8d334": ...

"Gax-17 Id\u003d142 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24f8d334":....

Sixty is a very high default.

Moreover, if I   setExecutorThreadCount  to 4 (code following), I still get an extra 26 permanent threads. Setting it to 1 or 2 also gets about 20 extra threads.  

ExecutorProvider executorProvider = InstantiatingExecutorProvider.newBuilder().setExecutorThreadCount(4).build();

Publisher.Builder builder = Publisher.newBuilder(ProjectTopicName.of(proj, topic)).setExecutorProvider(executorProvider);

Our application is already thread-heavy and cannot tolerate more than one or two extra threads set aside for just this purpose.

How can this be solved? 

Is there documentation on this? 

(I also posted at StackOverflow.)

-- 

 

JOSHUA FOX

Director | Near Horizons | Freightos

T (Israel): +972-545691165 | T (US):  +1-3123400953  

Smooth shipping.

 

[Ismail (Cloud Platform Support)]

This provides could a possible solution. For further technical questions, continue to ask them on the Stackoverflow. 

Note that Google Groups is reserved for general product discussions. If you require further technical support it is recommended to post your detailed questions on Stack Exchange.

[Kir Titievsky]

Joshua,   There are some internal thread pools allocation assuming high volume applications. If this is causing issues for you, you can either customize the client or use the lower level, synchronous, RPC based APIs.  

[Joshua Fox]

 

I ended up calling out to a proxy--an HTTP service which simply publishes to PubSub.

There are some internal thread pools allocation assuming high volume applications.

I recommend that you make this configurable down to 1 or 2; it can already be configured down to 26.

--
You received this message because you are subscribed to the Google Groups "Google Cloud Pub/Sub Discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-dis...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-pubsub-discuss/7d4283cb-b5a1-45f0-b9df-adbfcd920f71%40googlegroups.com.

[Kir Titievsky]

Ah, I see. So you've aggregated multiple requests in a single service. Might you say more about why having 26 threads is problematic for you?

--

Kir Titievsky | Product Manager | Google Cloud Pub/Sub 

[Joshua Fox]

On Mon, Sep 30, 2019 at 6:13 PM Kir Titievsky <k...@google.com> wrote:

Ah, I see. So you've aggregated multiple requests in a single service.

 

No aggregation: I just needed to send  roughly 2 messages per minute, to a single topic. Because my application could not tolerate the threads, I changed it so that the application sends the JSON over  HTTP -- blocking, but that's OK--- to  a proxy that just  then sends the JSON    to the topic. 

Might you say more about why having 26 threads is problematic for you?

 

This  is difficult legacy code with  has far too many threads. To compensate for that, the code starts 'graceful degradation' when too many new threads are made, and the 26 pushes it over the limit. I can't say that that makes  much sense, but the threading model in the legacy code  cannot be changed.

[Kir Titievsky]

Ah.  Makes sense. Might I trouble you to file a feature request here: https://github.com/googleapis/google-cloud-java/issues

[Joshua Fox]

Done

https://github.com/googleapis/google-cloud-java/issues/6428

[Joshua Fox]

Thank you,   code was provided , giving an answer.

https://github.com/googleapis/google-cloud-java/issues/6428

I  recommend that you inboard that into your PubSub library. That code sample is complex and digs deep into aspects that application programmers don't usually delve into. A simple setNumberThreads   or the like is the level of complexity needed.