SSL access failure

[McAmun]

Hi, 

I deployed a bazel-remote-cache in a kubernetes cluster. I secured the communication with TLS and used company signed certificates. 

Remote-cache-image: buchgr/bazel-remote-cache:v2.1.4

args:

            - --dir=/data

            - --max_size=15

            - --host=0.0.0.0

            - --http_read_timeout=60s

            - --http_write_timeout=60s

            - --s3.endpoint=$(endpoint)

            - --s3.region=$(region)

            - --s3.secret_access_key=$(AWS_SECRET_ACCESS_KEY)

            - --s3.access_key_id=$(AWS_ACCESS_KEY_ID)

            - --s3.bucket=bazel-remote-cache-dev

            - --s3.prefix=test 

            - --htpasswd_file=/cred/.htpasswd

            - --tls_cert_file=/cred/certificate-request-name.cer

            - --tls_key_file=/cred/private-key-name.key

            - --allow_unauthenticated_reads

Via Curl command I am getting following result: 

curl https://dev.brc.com/project/cas/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

    StatusCode        : 200

    StatusDescription : OK

    Content           : {}

    RawContent        : HTTP/1.1 200 OK

                        Connection: keep-alive

                        Content-Length: 0

which should be fine.

Once I am trying to build with bazel with following flag for remote cache:

    --remote_cache=https://dev.brc.com/project

I am getting following

**error:** 

    Caused by: javax.net.ssl.SSLHandshakeException: General OpenSslEngine problem

            at io.netty.handler.ssl.ReferenceCountedOpenSslContext$AbstractCertificateVerifier.verify(ReferenceCountedOpenSslContext.java:674)

            at io.netty.internal.tcnative.SSL.readFromSSL(Native Method)

            at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.readPlaintextData(ReferenceCountedOpenSslEngine.java:577)

            at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1141)

            at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1256)

            at io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:212)

            at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1330)

            at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1225)

            at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1272)

            at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)

            at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)

            ... 20 more

    Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

            at java.base/sun.security.validator.PKIXValidator.doBuild(Unknown Source)

            at java.base/sun.security.validator.PKIXValidator.engineValidate(Unknown Source)

            at java.base/sun.security.validator.Validator.validate(Unknown Source)

            at java.base/sun.security.ssl.X509TrustManagerImpl.validate(Unknown Source)

            at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(Unknown Source)

            at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(Unknown Source)

            at io.netty.handler.ssl.ReferenceCountedOpenSslClientContext$ExtendedTrustManagerVerifyCallback.verify(ReferenceCountedOpenSslClientContext.java:242)

            at io.netty.handler.ssl.ReferenceCountedOpenSslContext$AbstractCertificateVerifier.verify(ReferenceCountedOpenSslContext.java:670)

            ... 30 more

**Platform**:

Windows 10 

**Bazel version**: 

4.1.0

I guess that bazel uses its own Java trust store by default. (not sure if this is a correct statement)

Based on that I tried following: 

bazel \

  --host_jvm_args=-Djavax.net.ssl.trustStore=NULL \

  --host_jvm_args=-Djavax.net.ssl.trustStoreType=Windows-ROOT

  build //...

but this ends up with another error:

    Internal error thrown during build. Printing stack trace: java.lang.RuntimeException: javax.net.ssl.SSLException: unable to setup trustmanager

            at com.google.devtools.build.lib.remote.RemoteCacheClientFactory.createHttp(RemoteCacheClientFactory.java:130)

            at com.google.devtools.build.lib.remote.RemoteCacheClientFactory.create(RemoteCacheClientFactory.java:80)

            at com.google.devtools.build.lib.remote.RemoteModule.beforeCommand(RemoteModule.java:156)

            at com.google.devtools.build.lib.runtime.BlazeCommandDispatcher.execExclusively(BlazeCommandDispatcher.java:358)

            at com.google.devtools.build.lib.runtime.BlazeCommandDispatcher.exec(BlazeCommandDispatcher.java:208)

            at com.google.devtools.build.lib.server.GrpcServerImpl.executeCommand(GrpcServerImpl.java:603)

            at com.google.devtools.build.lib.server.GrpcServerImpl.lambda$run$2(GrpcServerImpl.java:659)

            at io.grpc.Context$1.run(Context.java:595)

            at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

            at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

            at java.base/java.lang.Thread.run(Unknown Source)

    Caused by: javax.net.ssl.SSLException: unable to setup trustmanager

            at io.netty.handler.ssl.ReferenceCountedOpenSslClientContext.newSessionContext(ReferenceCountedOpenSslClientContext.java:163)

            at io.netty.handler.ssl.OpenSslClientContext.<init>(OpenSslClientContext.java:192)

            at io.netty.handler.ssl.SslContext.newClientContextInternal(SslContext.java:777)

            at io.netty.handler.ssl.SslContextBuilder.build(SslContextBuilder.java:452)

            at com.google.devtools.build.lib.remote.http.HttpCacheClient.<init>(HttpCacheClient.java:242)

            at com.google.devtools.build.lib.remote.http.HttpCacheClient.create(HttpCacheClient.java:154)

            at com.google.devtools.build.lib.remote.RemoteCacheClientFactory.createHttp(RemoteCacheClientFactory.java:120)

            ... 10 more

    Caused by: java.security.KeyStoreException: problem accessing trust store

            at java.base/sun.security.ssl.TrustManagerFactoryImpl.engineInit(Unknown Source)

            at java.base/javax.net.ssl.TrustManagerFactory.init(Unknown Source)

            at io.netty.handler.ssl.ReferenceCountedOpenSslClientContext.newSessionContext(ReferenceCountedOpenSslClientContext.java:142)

            ... 16 more

    Caused by: java.security.KeyStoreException: Windows-ROOT not found

            at java.base/java.security.KeyStore.getInstance(Unknown Source)

            at java.base/sun.security.ssl.TrustStoreManager$TrustAnchorManager.loadKeyStore(Unknown Source)

            at java.base/sun.security.ssl.TrustStoreManager$TrustAnchorManager.getTrustedCerts(Unknown Source)

            at java.base/sun.security.ssl.TrustStoreManager.getTrustedCerts(Unknown Source)

            ... 19 more

I would like avoid using keytool to add any certificates to java trust store since the certifcates should be already available on Windows certificate store. 

Did anyone face similar issues and know how to fix it or have any suggestion what I could try? 

Thanks!