Google Cloud Run - Can't send outbound requests

[Nitin Kalra]

Hi All,

I have deployed a container in Google Cloud Run. Its an app built in Spring boot and Java.

I have exposed an endpoint to Cloud run container which further invokes a REST api to hit an external endpoint and at that point it fails.

I have attached the snapshot from the logs.

Thanks

Nitin

[Santiago Cardozo]

Hello, 

Have you made sure that your application is listening on port 8080 and for all IPs/hosts (0.0.0.0) ? (docs)

[Nitin Kalra]

Application is listening that's how I was able to hit it from the Postman and then the application tried to send another REST API request from within the container to the outside world = homedepot.ca where it gets stuck.

[Nitin Kalra]

Anyone ?

[Shawn Wu]

This error may happen in different circumstances, it seems related to the misconfiguration of network, can you double confirm the network configuration ? and can you share more logs related ? 

[Nitin Kalra]

I didn't change any network configuration. Attached is my firewall rule snapshot, I even added egress for all. Also the logs here -

2021-09-30T14:33:06.955652Z2021-09-30 14:33:06.955 WARN 1 --- [nio-8080-exec-3] io.netty.util.internal.MacAddressUtil : Failed to find a usable hardware address from the network interfaces; using random bytes: 61:71:ba:a4:d5:85:ee:11 021-09-30T14:33:08.094246Z

2021-09-30 14:33:08.093 DEBUG 1 --- [or-http-epoll-2] r.netty.http.client.HttpClientConnect : [id:93c6dc9b-1, L:/169.... - R:www.homedepot.ca/23.211.51.167:443] Handler is being applied: {uri=https://www.homedepot.ca/ method=GET} 

 2021-09-30T15:13:41.887524Z org.springframework.web.reactive.function.client.WebClientRequestException: handshake timed out after 10000ms; nested exception is io.netty.handler.ssl.SslHandshakeTimeoutException: handshake timed out after 10000ms at org.springframework.web.reactive.function.client.ExchangeFunctions$DefaultExchangeFunction.lambda$wrapException$9(ExchangeFunctions.java:141) Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: 2021-09-30T15:13:41.887759Z at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.drain(FluxConcatMap.java:414) 2021-09-30T15:13:41.887776Z at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.onNext(FluxConcatMap.java:251) 2021-09-30T15:13:41.887805Z at reactor.core.publisher.EmitterProcessor.drain(EmitterProcessor.java:491) 2021-09-30T15:13:41.887813Z at reactor.core.publisher.EmitterProcessor.tryEmitNext(EmitterProcessor.java:299) 2021-09-30T15:13:41.887822Z at reactor.core.publisher.SinkManySerialized.tryEmitNext(SinkManySerialized.java:100) 2021-09-30T15:13:41.887859Z at reactor.core.publisher.InternalManySink.emitNext(InternalManySink.java:27) 2021-09-30T15:13:41.887873Z at reactor.core.publisher.FluxRetryWhen$RetryWhenMainSubscriber.onError(FluxRetryWhen.java:190) 2021-09-30T15:13:41.887893Z at reactor.core.publisher.MonoCreate$MonoSink.error(MonoCreate.java:189) 2021-09-30T15:13:41.887903Z at reactor.netty.http.client.HttpClientConnect$MonoHttpConnect$ClientTransportSubscriber.onError(HttpClientConnect.java:304) 2021-09-30T15:13:41.887916Z at reactor.core.publisher.MonoCreate$MonoSink.error(MonoCreate.java:189) 2021-09-30T15:13:41.887934Z at reactor.netty.resources.PooledConnectionProvider$DisposableAcquire.onUncaughtException(PooledConnectionProvider.java:218) 2021-09-30T15:13:41.887958Z at reactor.netty.resources.PooledConnectionProvider$PooledConnection.onUncaughtException(PooledConnectionProvider.java:467) 2021-09-30T15:13:41.887979Z at reactor.netty.channel.ChannelOperationsHandler.exceptionCaught(ChannelOperationsHandler.java:129) 2021-09-30T15:13:41.887993Z at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:302) 2021-09-30T15:13:41.888011Z at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:281) 2021-09-30T15:13:41.888032Z at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:273) 2021-09-30T15:13:41.888049Z at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireExceptionCaught(CombinedChannelDuplexHandler.java:424) 2021-09-30T15:13:41.982305Z at io.netty.channel.ChannelHandlerAdapter.exceptionCaught(ChannelHandlerAdapter.java:92) 2021-09-30T15:13:41.982894Z2021-09-30 15:13:41.281 WARN 1 --- [or-http-epoll-3] r.netty.http.client.HttpClientConnect : [id:c351aec9, L:/169.... - R:www.homedepot.ca/96.7.86.161:443] The connection observed an error

[George (Cloud Platform Support)]

Does the address www.homedepot.ca/96.7.86.161:443 work as expected when called by other means? This address appears to give errors when called. 

You may try to call another address, to see if the networking park succeeds, and confirm it works as expected. 

[Nitin Kalra]

The same code, the same container works fine in the local environment. I am hitting the cloud run deployment via PostMan

[Felipe Bergallo Corral]

Hey there,

I mistakenly replied to an individual message rather than the entire group and will be attempting to fix that so that the whole conversation can be seen, I apologize for any inconvenience caused.

[Felipe Bergallo Corral]

This should join up these messages with the Groups Conversation

On Thu, Oct 28, 2021 at 1:15 AM Nitin Kalra <nitinka...@gmail.com> wrote:

Hi Felipe,

Thanks for looking into this.

I am not using Cloud Endpoints or API Gateway.

Also, I am not actually concatenating the URL with the IP. The IP gets appended in the exception shown & also in the logs attached(from local). Otherwise, I am just hitting the url. PFA the logs from my Local System.

On my local laptop, I am running Docker container and hitting the url from the postman. PFA the logs and request from Postman.

GET request from Postman and that's it nothing else I am passing.

http://localhost:8080/api/v1/products/urlsearch?productUrl=https://www.homedepot.ca/product/ridgid-nxt-60-l-capacity-6-5-peak-hp-wet-dry-vacuum-with-detachable-blower/1001200972&skipDB=true

Thanks and Regards,
Nitin Kalra

On Wed, Oct 20, 2021 at 9:05 AM Felipe Bergallo Corral <bergall...@google.com> wrote:

Hello there,    
I've noticed that my colleagues have asked you about your network setup and whether or not calling the endpoint by other means affects the response given.   
I understand that you are not using Cloud Endpoints or API Gateway, is that right?   
Also, I am curious about the `www.homedepot.ca/96.7.86.161:443` addresses you are using, as it seems like you're concatenating the url with the external IP of an instance you are trying to access, what's the purpose of this? I'm assuming it's intentional, and I've noticed that using the internal IP's that you are concatenating on their own causes the certificate's content to not line up with the page so it gives an invalid certificate when navigating through the browser; so if you are redirecting to the IP's then it may be related to the handshake error.  

Looking forward to your reply,

On Monday, October 18, 2021 at 9:43:08 AM UTC+2 nitinka...@gmail.com wrote:

[Felipe Bergallo Corral]

Ok, so, taking into account what you've said about it (no Endpoints or Gateway used, no concatenation, works on local, fails when using a container and postman), and the messages you've received (SSLHandshakeTimeout); I suspect you might be able to find out what's causing the timeout by calling the address using http instead of https in the depot portion of the url, like so[1], or use https instead of http in the local portion (`https://localhost:8080` etc instead of `http://localhost:8080`), as it seems either the container or Postman isn't responding to the handshake. 

I don't think this is necessarily the origin of the issue at hand, but trying out both will make it easier to see the actual origin, I think.

[1] http://localhost:8080/api/v1/products/urlsearch?productUrl=http://www.homedepot.ca/product/ridgid-nxt-60-l-capacity-6-5-peak-hp-wet-dry-vacuum-with-detachable-blower/1001200972&skipDB=true

[Nitin Kalra]

I tried everything

http://pricetracker-productfetcher-iqivydnwca-uc.a.run.app/api/v1/products/urlsearch?productUrl=https://www.homedepot.ca/product/ridgid-nxt-60-l-capacity-6-5-peak-hp-wet-dry-vacuum-with-detachable-blower/1001200972&skipDB=true

https://pricetracker-productfetcher-iqivydnwca-uc.a.run.app/api/v1/products/urlsearch?productUrl=https://www.homedepot.ca/product/ridgid-nxt-60-l-capacity-6-5-peak-hp-wet-dry-vacuum-with-detachable-blower/1001200972&skipDB=true  

Http://www.homedepot.ca (without 's') doesnt work in local as well as in Cloud run because that is not supported.

It doesnt work in Google Cloud but works in Local environment where Docker is running.

Not sure whats the issue is.

[Felipe Bergallo Corral]

Hmmm....

I'm trying to discard as many possibilities as I can think of, since you're performing a request to the Cloud Run instance it should be starting up correctly so it's unlikely to be a lack of CPU because this isn't a background task but part of the request. I know you're not reaching the outbound timeouts for Cloud Run (2 minutes for GCP calls, 20 for calls to the internet) because the SSL Handshake is timing out at 10 seconds; and since it's expecting a handshake, it's unlikely to require HTTP/2 throughout, otherwise it would fail sooner.

Would it be possible for you to provide us with a reproducible example? As well as the logs from the successful deployment (and request) in the local environment?