Release 0.16.0, bazel remote cache, SSL access failure

[matthew]

With the 0.16.0 release I'm starting to get SSL access errors when trying to reach bazel remote cache.

My remote cache server, running Bazel Remote Cache, uses a self-signed cert, successfully until now.

I'm able to establish a simple SSL connection using SSLPoke. 

  $ java SSLPoke my-cache-server 9090

  Successfully connected

But bazel builds fail. 

  Found 1 target...

  WARNING: Error reading from the remote cache:

  SSLEngine closed already

  WARNING: Error reading from the remote cache:

  ClosedChannelException

  WARNING: Error writing to the remote cache:

  SSLEngine closed already

  WARNING: Error writing to the remote cache:

  ClosedChannelException

  Target //blah/blah/blah/web_server failed to build

  Unhandled exception thrown during build; message: Unrecoverable error while evaluating node 'ActionLookupData{actionLookupKey=@boringssl//:crypto BuildConfigurationValue.Key[c215c61f119d666e61c4ec28c2abfb5c] false, actionIndex=317}' (requested by nodes 'external/xyz BuildConfigurationValue.Key[c215c61f119d666e61c4ec28c2abfb5c] false')

  INFO: Elapsed time: 5.754s, Critical Path: 4.43s

  INFO: 152 processes: 152 linux-sandbox.

  FAILED: Build did NOT complete successfully

  java.lang.RuntimeException: Unrecoverable error while evaluating node 'ActionLookupData{actionLookupKey=@xyz BuildConfigurationValue.Key[c215c61f119d666e61c4ec28c2abfb5c] false, actionIndex=317}' (requested by nodes 'external/xyz @xyz BuildConfigurationValue.Key[c215c61f119d666e61c4ec28c2abfb5c] false')

      at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:477)

      at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:355)

      at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

      at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

      at java.base/java.lang.Thread.run(Unknown Source)

  Caused by: io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: General OpenSslEngine problem

      at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:459)

      at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)

      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)

      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)

      at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)

      at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359)

      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)

      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)

      at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935)

      at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:138)

      at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)

      at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)

      at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)

      at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)

      at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)

      at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)

      ... 1 more

  Caused by: javax.net.ssl.SSLHandshakeException: General OpenSslEngine problem

      at io.netty.handler.ssl.ReferenceCountedOpenSslContext$AbstractCertificateVerifier.verify(ReferenceCountedOpenSslContext.java:648)

      at io.netty.internal.tcnative.SSL.readFromSSL(Native Method)

      at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.readPlaintextData(ReferenceCountedOpenSslEngine.java:489)

      at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1039)

      at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1146)

      at io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:211)

      at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1247)

      at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1158)

      at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1193)

      at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)

      at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)

      ... 16 more

  Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

      at java.base/sun.security.validator.PKIXValidator.doBuild(Unknown Source)

      at java.base/sun.security.validator.PKIXValidator.engineValidate(Unknown Source)

      at java.base/sun.security.validator.Validator.validate(Unknown Source)

      at java.base/sun.security.ssl.X509TrustManagerImpl.validate(Unknown Source)

      at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(Unknown Source)

      at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(Unknown Source)

      at io.netty.handler.ssl.ReferenceCountedOpenSslClientContext$ExtendedTrustManagerVerifyCallback.verify(ReferenceCountedOpenSslClientContext.java:221)

      at io.netty.handler.ssl.ReferenceCountedOpenSslContext$AbstractCertificateVerifier.verify(ReferenceCountedOpenSslContext.java:644)

      ... 26 more

  Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

      at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(Unknown Source)

      at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(Unknown Source)

      at java.base/java.security.cert.CertPathBuilder.build(Unknown Source)

      ... 34 more

Someone suggest a fix? Thanks.

[Jakob Buchgraber]

Hi Matthew,

I am somewhat surprised that self signed certificates ever worked for you without

you being able to specify the root cert in the Bazel client?

Can you provide a reproducer please and I ll look into a fix? Thanks!

Best,

Jakob

[matthew]

​

Thanks for 

​taking a look

. In case I failed to make it clear, the self-signed cert is installed in my client's local CA. It works with a minimal Java app and with 3 popular browsers.

Repro:

• Create a self-signed cert, following e.g. the first step described here. 
  

BRCROOT=/bazel-remote-cache

openssl req -x509 -nodes -days 180 \
          -newkey rsa:2048 \
          -keyout $BRCROOT/etc/selfsigned.key \
          -out $BRCROOT/etc/selfsigned.crt

• Start server, 
  

docker run --name bazel-remote-cache \
     -v $BRCROOT/data:/data \
     -v $BRCROOT/etc/selfsigned.crt:/etc/bazel-remote/server_cert \
     -v $BRCROOT/etc/selfsigned.key:/etc/bazel-remote/server_key \
     --restart=always \
     -p 9090:8080 buchgr/bazel-remote-cache \
     --tls_enabled=true \
     --tls_cert_file=/etc/bazel-remote/server_cert \
     --tls_key_file=/etc/bazel-remote/server_key

• Install cert on client machine
  

cp selfsigned.crt /usr/local/share/ca-certificates/
sudo update-ca-certificates

• bazel build on client machine
  
  

[Philipp Wollermann]

Hi Matthew,

I guess the issue is that your self-signed cert is added to your system JDK trust store, but Bazel uses the truststore of its embedded JDK by default.

Could you try the following to see if it fixes the issue for you?

bazel \
  --host_jvm_args=-Djavax.net.ssl.trustStore=/etc/ssl/certs/cacerts \
  --host_jvm_args=-Djavax.net.ssl.trustStorePassword=changeit \
  build //...

(The password is actually called "changeit" by default on Debian...)

If it works, you could add this to your bazelrc as startup options like this:

echo "startup --host_jvm_args=-Djavax.net.ssl.trustStore=/etc/ssl/certs/cacerts" >> ~/.bazelrc

echo "startup --host_jvm_args=-Djavax.net.ssl.trustStorePassword=changeit" >> ~/.bazelrc

Hope this helps!

Philipp

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/4dcd6df7-833d-46c2-aba0-8b3ae807107d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Philipp Wollermann
Software Engineer
phi...@google.com

Google Germany GmbH
Erika-Mann-Straße 33
80636 München

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg

Diese E-Mail ist vertraulich. Falls sie diese fälschlicherweise erhalten haben sollten, leiten Sie diese bitte nicht an jemand anderes weiter, löschen Sie alle Kopien und Anhänge davon und lassen Sie mich bitte wissen, dass die E-Mail an die falsche Person gesendet wurde.

This e-mail is confidential. If you received this communication by mistake, please don't forward it to anyone else, please erase all copies and attachments, and please let me know that it has gone to the wrong person.

[Matthew Weaver]

That fixed it. 

With one minor revision:

startup --host_jvm_args=-Djavax.net.ssl.trustStore=/etc/ssl/certs/java/cacerts \

        --host_jvm_args=-Djavax.net.ssl.trustStorePassword=changeit

Thanks Phillipp!

-M

To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discuss+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/4dcd6df7-833d-46c2-aba0-8b3ae807107d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Matthew Weaver]

The options can be placed in your .bazelrc file as:

startup --host_jvm_args=-Djavax.net.ssl.trustStore=/etc/ssl/certs/java/cacerts \

        --host_jvm_args=-Djavax.net.ssl.trustStorePassword=changeit

See this for info on the .bazelrc file: https://docs.bazel.build/versions/master/user-manual.html#where-are-the-bazelrc-files

Or you can put these options directly on the Bazel command line like:

bazel --host_jvm_args=-Djavax.net.ssl.trustStore=/etc/ssl/certs/java/cacerts \

  --host_jvm_args=-Djavax.net.ssl.trustStorePassword=changeit \
  build 

The BUILD file is not the place for these global bazel options. 

But I'm not sure I understand the question. Are you trying to use bazel remote cache? For that you will need a few other options too; details can be found here: https://docs.bazel.build/versions/master/remote-caching.html#run-bazel-using-the-remote-cache. 

On Fri, Aug 24, 2018, 3:48 AM Sandhyarani Gadamsetti <sandh...@gmail.com> wrote:

Hi Matthew,

Where to provide these options for Bazel, in the Build file.

Because we are having a showstopper from last 3 weeks and stuck up at one point.

we are trying to connect DashDB from Scala code using Bazel Build, in which connection is getting failed with a reset message.

But we are able able run the same code in SBT and also able to connect from DB Studio using Truststore certificates.

So kindly let me know , if we have our own truststore how to overwrite them in Bazel.

Thanks
Sandhya

-M

To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/4dcd6df7-833d-46c2-aba0-8b3ae807107d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Philipp Wollermann
Software Engineer
phi...@google.com

Google Germany GmbH
Erika-Mann-Straße 33
80636 München

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg

Diese E-Mail ist vertraulich. Falls sie diese fälschlicherweise erhalten haben sollten, leiten Sie diese bitte nicht an jemand anderes weiter, löschen Sie alle Kopien und Anhänge davon und lassen Sie mich bitte wissen, dass die E-Mail an die falsche Person gesendet wurde.

This e-mail is confidential. If you received this communication by mistake, please don't forward it to anyone else, please erase all copies and attachments, and please let me know that it has gone to the wrong person.

--
You received this message because you are subscribed to a topic in the Google Groups "bazel-discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/bazel-discuss/13uPDObyfQg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/5a60ac40-3afb-4c49-9b0c-45fb066a820e%40googlegroups.com.