knative containers and golang goroutines? Background processes ...

[Sandy Walsh]

I'm considering KNative/Google Cloud Run for some cron-like operations I need to perform. They will get triggered by an HTTP invocation. The invocation will return (likely with a 202) and continue running in the background via a golang goroutine.

But, I'm concerned that the runtime containers are destroyed when they're not handling HTTP requests. I could be part-way through my processing and get reaped.

Is there a way to tell knative/GCR to keep the container alive until I'm finished?

[Mike Helmick]

Not currently. The runtime contract stipulates that a container will only keep running if it is handling a request.

There are 2 options that I would consider

1) Keeping the connection open while the processing is occurring

2) Using the knative eventing components to decouple the async nature of the request from the sync trigger.

mike h

--
You received this message because you are subscribed to the Google Groups "Knative Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to knative-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/knative-users/6eba4626-2362-4d23-b445-19aece2db32e%40googlegroups.com.

[Lionel Villard]

One way is to configure the autoscaler minScale to be at least one (see https://knative.dev/docs/serving/configuring-the-autoscaler/). When the container is finished, remove the annotation.

[Nemo Caviani]

currently Knative requires http connections to remain open while requests are processed.

We have been discussing options for supporting background jobs or async jobs under the following issue but it is still work in progress: https://github.com/knative/serving/issues/4522

On Mon, Jul 29, 2019 at 9:41 AM 'Sandy Walsh' via Knative Users <knativ...@googlegroups.com> wrote:

--

[Ahmet Alp Balkan]

For your Cloud Run (managed version, not on GKE with Knative) question on Stackoverflow, I left an answer.

I believe the behavior of what happens when a request ends is different on Knative OSS implementation versus Google Cloud Run managed version (which scales CPU dynamically to ~0).

Ahmet Alp Balkan |

Developer Advocate |

github:@ahmetb |

twitter:@ahmetb

On Mon, Jul 29, 2019 at 9:41 AM 'Sandy Walsh' via Knative Users <knativ...@googlegroups.com> wrote:

--

[Scott Nichols]

Also note the minscale to 1 trick will not work when the resource scales. 

-Scott

To view this discussion on the web visit https://groups.google.com/d/msgid/knative-users/CAEBbUmZqRC6gdnGhc9Bt3oW9HhG042e98hDsVQBiQo2LoTro2w%40mail.gmail.com.

[Sandy Walsh]

Hmm, actually disregard that. I don't this it's really viable for this use case. 

CRun has a max runtime of 15 minutes (cannot be increased) and the billing model would make this prohibitive. 

I think I'll look at  k8s jobs next. 

Thanks again for your feedback everyone!

[Matthew Moore]

(skimming, so sorry if I missed something)

Note that the 15 minutes used by Cloud Run on GKE is not a cap on the total request duration, but how quickly the server must start responding.

I believe that either GRPC streaming or websockets would hold the connection open for you, potentially for much longer than 15 minutes.

--

You received this message because you are subscribed to the Google Groups "Knative Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to knative-user...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/knative-users/35b12275-9708-4e34-a4f4-e000e6f12c1d%40googlegroups.com.

--

Matthew Moore

Knative @ Google

[Sandy Walsh]

Thanks Matthew, that's an important nuance. Perhaps the docs should be explicit on that [1]

Something still smells wrong about holding a connection open that long. Perhaps DataFlow would be better?

[1] https://cloud.google.com/run/quotas

[Matthew Moore]

There are a number of batch options on GCP, but this probably isn't the place to discuss them.  If you want to start a thread with me, I can try to connect you with some folks.

There is also some discussion about async operations in Knative, but it's early days still (not sure if if was mentioned above).

-M