Parallelism of Pub/Sub Trigger for Cloud Functions

[qji...@ieee.org]

I am using a Pub/Sub topic to trigger my Cloud Functions. I have a set of jobs in my Pub/Sub topic that can run in parallel, but Cloud Functions seemed to be picking up the jobs from the Pub/Sub topic one by one and execute the jobs in a sequential manner. I can verify the sequential execution of these jobs using the timestamps in the cloud function logs. How do I configure my cloud function and the Pub/Sub topic so that the jobs can run in parallel?

For example, I am publishing 100 jobs to the Pub/Sub topic in one batch. These jobs are independent of each other. How do I make these 100 jobs run in parallel?

[Jun (Cloud Platform Support)]

Hi, the simplest answer that comes to mind would be to write to different paths (e.g. /trigger/alpha, /trigger/beta, et al) and have the specific stages listen on those. 

[Jason Polites]

This might simply be an artifact of the fact that the test is small, and isolated.

When there is no traffic to a function, there will be 0 instances of it running.  When a sudden influx of traffic is received, the underlying infrastructure will be signaled to start, but this startup is not instant.  The pending messages will be queued until there is enough infrastructure to serve them all.  Under circumstances when the load is "stable", then you might see the first few messages processed sequentially but over some period of time the throughput should increase up to some maximum concurrency as additional function instances become available.

In this case because there are no subsequent messages (i.e. we go from 0 to 100 to 0 again), then you never see this throughput stabilization.  Concretely, a sustained QPS should achieve max concurrency independent of the channel (pubsub or otherwise).  Although perhaps not obvious, this behavior works fine for most use cases.  That said, for some class of batch processing use cases, where it's likely that there will be a short burst of many messages going from 0-N-0 quickly, then things should probably behave differently.  We're looking into what we can do for these cases, but if you're really just testing to see how it behaves, then hopefully this explanation helps somewhat.

Cheers,

Jason.

[Tomas Jansson]

Can you elaborate on how you have different paths when using a pub/sub trigger?

[Tomas Jansson]

Interesting question that I don't know the answer to, but what I would do is have two functions:

1. One http triggered cloud function that does the actual work

2. One pub/sub triggered cloud function that reads from pub/sub and just call the http one.

How did you solve the problem?