Release 0.16.0, bazel remote cache, SSL access failure

1543 views
Skip to first unread message

matthew

unread,
Aug 2, 2018, 9:08:34 PM8/2/18
to bazel-discuss
With the 0.16.0 release I'm starting to get SSL access errors when trying to reach bazel remote cache.
My remote cache server, running Bazel Remote Cache, uses a self-signed cert, successfully until now.

I'm able to establish a simple SSL connection using SSLPoke
  $ java SSLPoke my-cache-server 9090
  Successfully connected

But bazel builds fail. 

  Found 1 target...
  WARNING: Error reading from the remote cache:
  SSLEngine closed already
  WARNING: Error reading from the remote cache:
  ClosedChannelException
  WARNING: Error writing to the remote cache:
  SSLEngine closed already
  WARNING: Error writing to the remote cache:
  ClosedChannelException
  Target //blah/blah/blah/web_server failed to build
  Unhandled exception thrown during build; message: Unrecoverable error while evaluating node 'ActionLookupData{actionLookupKey=@boringssl//:crypto BuildConfigurationValue.Key[c215c61f119d666e61c4ec28c2abfb5c] false, actionIndex=317}' (requested by nodes 'external/xyz BuildConfigurationValue.Key[c215c61f119d666e61c4ec28c2abfb5c] false')
  INFO: Elapsed time: 5.754s, Critical Path: 4.43s
  INFO: 152 processes: 152 linux-sandbox.
  FAILED: Build did NOT complete successfully
  java.lang.RuntimeException: Unrecoverable error while evaluating node 'ActionLookupData{actionLookupKey=@xyz BuildConfigurationValue.Key[c215c61f119d666e61c4ec28c2abfb5c] false, actionIndex=317}' (requested by nodes 'external/xyz @xyz BuildConfigurationValue.Key[c215c61f119d666e61c4ec28c2abfb5c] false')
      at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:477)
      at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:355)
      at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
      at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
      at java.base/java.lang.Thread.run(Unknown Source)
  Caused by: io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: General OpenSslEngine problem
      at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:459)
      at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
      at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
      at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359)
      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
      at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935)
      at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:138)
      at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
      at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
      at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
      at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
      at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
      at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
      ... 1 more
  Caused by: javax.net.ssl.SSLHandshakeException: General OpenSslEngine problem
      at io.netty.handler.ssl.ReferenceCountedOpenSslContext$AbstractCertificateVerifier.verify(ReferenceCountedOpenSslContext.java:648)
      at io.netty.internal.tcnative.SSL.readFromSSL(Native Method)
      at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.readPlaintextData(ReferenceCountedOpenSslEngine.java:489)
      at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1039)
      at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1146)
      at io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:211)
      at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1247)
      at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1158)
      at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1193)
      at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
      at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)
      ... 16 more
  Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
      at java.base/sun.security.validator.PKIXValidator.doBuild(Unknown Source)
      at java.base/sun.security.validator.PKIXValidator.engineValidate(Unknown Source)
      at java.base/sun.security.validator.Validator.validate(Unknown Source)
      at java.base/sun.security.ssl.X509TrustManagerImpl.validate(Unknown Source)
      at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(Unknown Source)
      at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(Unknown Source)
      at io.netty.handler.ssl.ReferenceCountedOpenSslClientContext$ExtendedTrustManagerVerifyCallback.verify(ReferenceCountedOpenSslClientContext.java:221)
      at io.netty.handler.ssl.ReferenceCountedOpenSslContext$AbstractCertificateVerifier.verify(ReferenceCountedOpenSslContext.java:644)
      ... 26 more
  Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
      at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(Unknown Source)
      at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(Unknown Source)
      at java.base/java.security.cert.CertPathBuilder.build(Unknown Source)
      ... 34 more

Someone suggest a fix? Thanks.

Jakob Buchgraber

unread,
Aug 6, 2018, 7:44:19 AM8/6/18
to bazel-discuss
Hi Matthew,

I am somewhat surprised that self signed certificates ever worked for you without
you being able to specify the root cert in the Bazel client?

Can you provide a reproducer please and I ll look into a fix? Thanks!

Best,
Jakob

matthew

unread,
Aug 6, 2018, 2:09:14 PM8/6/18
to bazel-discuss
Thanks for 
​taking a look
. In case I failed to make it clear, the self-signed cert is installed in my client's local CA. It works with a minimal Java app and with 3 popular browsers.

Repro:
  • Create a self-signed cert, following e.g. the first step described here
BRCROOT=/bazel-remote-cache

openssl req -x509 -nodes -days 180 \
          -newkey rsa:2048 \
          -keyout $BRCROOT/
etc/selfsigned.key \
         
-out $BRCROOT/etc/selfsigned.crt
  • Start server, 
docker run --name bazel-remote-cache \
     
-v $BRCROOT/data:/data \
     -v $BRCROOT/
etc/selfsigned.crt:/etc/bazel-remote/server_cert \
     
-v $BRCROOT/etc/selfsigned.key:/etc/bazel-remote/server_key \
     
--restart=always \
     
-p 9090:8080 buchgr/bazel-remote-cache \
     
--tls_enabled=true \
     
--tls_cert_file=/etc/bazel-remote/server_cert \
     
--tls_key_file=/etc/bazel-remote/server_key

  • Install cert on client machine
cp selfsigned.crt /usr/local/share/ca-certificates/
sudo update
-ca-certificates

  • bazel build on client machine

Philipp Wollermann

unread,
Aug 9, 2018, 9:17:47 AM8/9/18
to mat...@xnor.ai, bazel-discuss
Hi Matthew,

I guess the issue is that your self-signed cert is added to your system JDK trust store, but Bazel uses the truststore of its embedded JDK by default.

Could you try the following to see if it fixes the issue for you?

bazel \
  --host_jvm_args=-Djavax.net.ssl.trustStore=/etc/ssl/certs/cacerts \
  --host_jvm_args=-Djavax.net.ssl.trustStorePassword=changeit \
  build //...

(The password is actually called "changeit" by default on Debian...)

If it works, you could add this to your bazelrc as startup options like this:

echo "startup --host_jvm_args=-Djavax.net.ssl.trustStore=/etc/ssl/certs/cacerts" >> ~/.bazelrc
echo "startup --host_jvm_args=-Djavax.net.ssl.trustStorePassword=changeit" >> ~/.bazelrc

Hope this helps!

Philipp


--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/4dcd6df7-833d-46c2-aba0-8b3ae807107d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


--
Philipp Wollermann
Software Engineer
phi...@google.com

Google Germany GmbH
Erika-Mann-Straße 33
80636 München

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg

Diese E-Mail ist vertraulich. Falls sie diese fälschlicherweise erhalten haben sollten, leiten Sie diese bitte nicht an jemand anderes weiter, löschen Sie alle Kopien und Anhänge davon und lassen Sie mich bitte wissen, dass die E-Mail an die falsche Person gesendet wurde.

This e-mail is confidential. If you received this communication by mistake, please don't forward it to anyone else, please erase all copies and attachments, and please let me know that it has gone to the wrong person.

Matthew Weaver

unread,
Aug 9, 2018, 2:18:23 PM8/9/18
to Philipp Wollermann, bazel-discuss
That fixed it. 

With one minor revision:
startup --host_jvm_args=-Djavax.net.ssl.trustStore=/etc/ssl/certs/java/cacerts \
        --host_jvm_args=-Djavax.net.ssl.trustStorePassword=changeit

Thanks Phillipp!


-M

To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discuss+unsubscribe@googlegroups.com.
Message has been deleted

Matthew Weaver

unread,
Aug 24, 2018, 12:22:40 PM8/24/18
to Sandhyarani Gadamsetti, bazel-discuss
The options can be placed in your .bazelrc file as:

startup --host_jvm_args=-Djavax.net.ssl.trustStore=/etc/ssl/certs/java/cacerts \
        --host_jvm_args=-Djavax.net.ssl.trustStorePassword=changeit


Or you can put these options directly on the Bazel command line like:

bazel --host_jvm_args=-Djavax.net.ssl.trustStore=/etc/ssl/certs/java/cacerts \
  --host_jvm_args=-Djavax.net.ssl.trustStorePassword=changeit \
  build 

The BUILD file is not the place for these global bazel options. 

But I'm not sure I understand the question. Are you trying to use bazel remote cache? For that you will need a few other options too; details can be found here: https://docs.bazel.build/versions/master/remote-caching.html#run-bazel-using-the-remote-cache



On Fri, Aug 24, 2018, 3:48 AM Sandhyarani Gadamsetti <sandh...@gmail.com> wrote:
Hi Matthew,

Where to provide these options for Bazel, in the Build file.

Because we are having a showstopper from last 3 weeks and stuck up at one point.

we are trying to connect DashDB from Scala code using Bazel Build, in which connection is getting failed with a reset message.

But we are able able run the same code in SBT and also able to connect from DB Studio using Truststore certificates.

So kindly let me know , if we have our own truststore how to overwrite them in Bazel.

Thanks
Sandhya

-M

To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.


--
Philipp Wollermann
Software Engineer
phi...@google.com

Google Germany GmbH
Erika-Mann-Straße 33
80636 München

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg

Diese E-Mail ist vertraulich. Falls sie diese fälschlicherweise erhalten haben sollten, leiten Sie diese bitte nicht an jemand anderes weiter, löschen Sie alle Kopien und Anhänge davon und lassen Sie mich bitte wissen, dass die E-Mail an die falsche Person gesendet wurde.

This e-mail is confidential. If you received this communication by mistake, please don't forward it to anyone else, please erase all copies and attachments, and please let me know that it has gone to the wrong person.

--
You received this message because you are subscribed to a topic in the Google Groups "bazel-discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/bazel-discuss/13uPDObyfQg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/5a60ac40-3afb-4c49-9b0c-45fb066a820e%40googlegroups.com.
Reply all
Reply to author
Forward
0 new messages