Can't verify TLS certificate (Android)

[Paride]

Hello,

I'm trying to use a "turns:" relay on an Android app using WebRTC (M57).

If I use a regular "turn:" URI it works fine, and so it does if I use "turns:" but tell the library not to verify the TLS certificate, like this:

new PeerConnection.IceServer("turns:turn.example.com:8443?transport=tcp", "user", "pass", PeerConnection.TlsCertPolicy.TLS_CERT_POLICY_INSECURE_NO_CHECK);

When I switch to TLS_CERT_POLICY_SECURE (the default), without touching anything else, the connection to the TURN server fails.

Now, the TLS certificate I use is a valid letsencrypt certificate that is recognized by the device.

The TURN server (coturn-4.5.0.6) also works as an https server, and I can connect to:

https://turn.example.com:8443/

using a WebView browser on the same device and the certificate is recognized.

If I use Trickle ICE [1] I can connect to the "turns:" server and I get no warning/error on the certificate.

The problem seems to exist only on Android, and it's happening on both an emulated and physical device running stock Android 7.1.1

Is there anything obvious I should check?

Is there a way to get a more verbose logging?

Thank you,

Paride

[1] https://webrtc.github.io/samples/src/content/peerconnection/trickle-ice/

[Sorin Cocorada]

Hi Paride,
Same problem, have you found a solution?
I tried to find where against which CA certificate file is the actual turn server certificate checked  but didn't get it.

Sorin

[Paride]

Hi Sorin,

My guess at the moment is that the certificate is not checked using the system's CA certs, but those defined in webrtc/base/sslroots.h. These are the first lines of the file:

// This file is the root certificates in C form that are needed to connect to

// Google.

// It was generated with the following command line:

// > python tools/sslroots/generate_sslroots.py

//    https://pki.google.com/roots.pem

so it's probably good only for the certificate authorities used by Google, and there's a script to generate a custom version of the file.

I still have to try if it works, but seems reasonable to me.

What I'd like to understand is if there's a way to specify the CA roots from an API, but I don't think so, unfortunately.

Paride

[Taylor Brandstetter]

Hi Paride,

Your guess was correct. When webrtc is built with chromium, it defers to chromium for certificate validation, but native applications will end up using sslroots.h.

Currently there's no API to change the set of certificates, but I'll talk to my team about this. For now, it's probably safe to use TLS_CERT_POLICY_INSECURE_NO_CHECK; since media is secured end-to-end using DTLS-SRTP, verifying the certificate of the TURN server isn't necessary to ensure the media is secure.

--

---
You received this message because you are subscribed to the Google Groups "discuss-webrtc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss-webrtc+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/fa31ddc6-a7d6-46e9-9b7a-28e34562b2a2%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Paride]

Thank you Taylor. Do you think that bundling a more complete list of CA roots would be a viable option? I'm thinking for example of those by Mozilla:

https://hg.mozilla.org/releases/mozilla-release/raw-file/default/security/nss/lib/ckfw/builtins/certdata.txt

https://curl.haxx.se/ca/cacert.pem (the same in PEM format)

I understand this is not the right solution in the long term, but in the meantime it could work in many scenarios.

Paride

To unsubscribe from this group and stop receiving emails from it, send an email to discuss-webrt...@googlegroups.com.

[Andrea Bernabei]

Hi Taylor,

I've just hit the same issue.

After doing some research I found quite a few posts of people reporting SSL failures when trying to have videocalls using Android clients of Jitsi Meet and Riot IM (Android client for Matrix server) when connecting to self-hosted servers configured with TURNs and Let's Encrypt certificates.

I spent some time rebuilding webrtc, react-native-webrtc, matrix-sdk, riot-im, jitsi-meet sdk and realized that the failure in Android Jitsi Meet and Android Riot IM has the same root cause: it is indeed due to the DST Root CA X3 cert missing from WebRTC's trust store defined in ssl_roots.h

I am aware the media is encrypted even if you use plain TURN, but it would be great if we could get TURNs working.

Jitsi Meet, Matrix Synapse and Let's Encrypt have such a key role in digital freedom nowadays, it is a shame that their official implementations are not able to exploit the full potential of self-hosted protected setups because of a trusted CA missing in WebRTC's hardcoded list.

I am happy to spend some time contributing to fixing this for the community.

Would you be happy to consider my PR/MR to add DST Root CA X3 to ssl_root.h ?

Kind Regards,

Andrea Bernabei (faenil)

To unsubscribe from this group and stop receiving emails from it, send an email to discuss...@googlegroups.com.

[Andrea Bernabei]

or anyone else from the development team who can share the team's view on this?

[Taylor Brandstetter]

The root certificates were updated in this CL: https://webrtc-review.googlesource.com/c/src/+/173821

I don't specifically see "DST" anywhere, but can you test whether it works now?

[Andrea Bernabei]

Thanks for the update!
Considering it has been updated using Google's list, which does not include DST, I really doubt it will work.

In any case, I will try it and confirm :)

Assuming it won't work, would you be happy to accept my patch adding LetsEncrypt's root CAs?

[Andrea Bernabei]

Hi Taylor,

I have just checked and, as we expected, the update did not help.

I/openssl_adapter.cc: (line 808): Error with certificate at depth: 1
    (line 810):   issuer  = /O=Digital Signature Trust Co./CN=DST Root CA X3
    (line 812):   subject = /C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
    (line 813):   err     = 20:unable to get local issuer certificate
I/openssl_adapter.cc: (line 786): SSL3 alert write:fatal:unknown CA
    (line 793): SSL_connect:error in TLS 1.3 client read_server_certificate_verify

I tried adding the DST root CA to ssl_roots.h and the certificate error disappeared (I still have a TLS read_server_hello error, which I will investigate tomorrow).

Can I proceed with creating a change request?

[Harald Alvestrand]

Thanks for testing so thoroughly!

This seems to be confirming that the issue is that Google isn't including the DST root in its list of trust certificates - not something we can solve inside the WebRTC project.

You should have the same issue when connecting to the server from Chrome. (Do you?)

The official announcement of the roots.pem file seems to be this 2017 blog: https://security.googleblog.com/2017/01/the-foundation-of-more-secure-web.html

I haven't yet found the policy statement on what certificates are included in this file.

To unsubscribe from this group and stop receiving emails from it, send an email to discuss-webrt...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/de1e64d0-6374-4831-bbbf-1aa62f7dbe41%40googlegroups.com.

[Andrea Bernabei]

No problem, happy to help :)

I will double-check the TURNs connection with Chrome on Android later today.

Does Chrome on Android use this webrtc implementation to gather ICE candidates?

I can confirm the web loader in Chrome trusts DST Root CA X3 because it can verify the web page of my self-hosted Jitsi Meet instance, but I will doublecheck for any TURNs errors.

The official Matrix Android client (RiotIM) also successfully connects to a Let's Encrypt protected server, it is only the TURNs connection that fails, because it relies on the small trust store defined in ssl_roots.h

Please note, the Android platform also trusts DST Root CA X3

https://android.googlesource.com/platform/system/ca-certificates/+/refs/heads/master/files/12d55845.0

From the blog post you linked to:

Google maintains a sample PEM file at (https://pki.goog/roots.pem) which is periodically updated to include the Google Trust Services owned and operated roots as well as other roots that may be necessary now, or in the future to communicate with and use Google Products and Services.

At the pki.goog FAQ page, you can find the .pem file under the question:

What are the recommended minimum requirements for a Transport Layer Security (TLS) client to communicate with Google?

I believe the "sample PEM" file, as they define in the blog post, is just a list of the root CAs you need to trust to guarantee a successful connection to Google services.

I bet Google does not use Let's Encrypt for its certificates (there is no need), so Let's Encrypt's root CAs do not have to be part of that list.

That said,

would you mind elaborating on why this is "not something we can solve inside the WebRTC project"?

Does the WebRTC project explicitly forbid including CAs which are not part of Google's "sample" list?

Would you consider adding the DST Root CA X3 as a special case, given its impact on the privacy protection of private users?

Alternatively, what about aligning the list of trusted CAs to the one used by Android as defined in

https://android.googlesource.com/platform/system/ca-certificates/+/refs/heads/master/files/

or to the one defined by Mozilla NSS?

The main target here is providing private users with an easy way to protect their privacy by ensuring communication with Let's Encrypt certificates works OOB.

I'm happy to contribute towards that end :)

On Thursday, 30 April 2020 09:47:17 UTC+1, Harald Alvestrand wrote:

Thanks for testing so thoroughly!

This seems to be confirming that the issue is that Google isn't including the DST root in its list of trust certificates - not something we can solve inside the WebRTC project.

You should have the same issue when connecting to the server from Chrome. (Do you?)

The official announcement of the roots.pem file seems to be this 2017 blog: https://security.googleblog.com/2017/01/the-foundation-of-more-secure-web.html

I haven't yet found the policy statement on what certificates are included in this file.

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/de1e64d0-6374-4831-bbbf-1aa62f7dbe41%40googlegroups.com.

[Harald Alvestrand]

If this can be solved by switching which source of truth we're using for root CAs (and that passes security review), that would be great (and could be solved within the WebRTC project).

If we need to have pki.goog/roots.pem modified, that's another team that needs convincing.

To unsubscribe from this group and stop receiving emails from it, send an email to discuss-webrt...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/28e7344b-f83e-4a13-b7ef-ca595858db19%40googlegroups.com.

[Andrea Bernabei]

Indeed switching to a more comprehensive (yet trusted, such as Mozilla NSS) list of root CAs would be great.

However, that is a way bigger change and it is more likely to stir a discussion in the security team.

What about my suggestion of just manually adding DST Root CA X3 to ssl_roots.h in the WebRTC project, since it is known to be a trusted root CA?

That would effectively solve the issue with minimal change, hence much easier to review and more likely to be allowed by security team.

On Thursday, 30 April 2020 13:14:59 UTC+1, Harald Alvestrand wrote:

If this can be solved by switching which source of truth we're using for root CAs (and that passes security review), that would be great (and could be solved within the WebRTC project).

If we need to have pki.goog/roots.pem modified, that's another team that needs convincing.

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/28e7344b-f83e-4a13-b7ef-ca595858db19%40googlegroups.com.

[Andrea Bernabei]

Hi Harald,

I tried with Chrome and Firefox Nightly, on Android.

Both could successfully connect to the turn server via turns, which means they are probably using a different trust store than ssl_roots.h.

On Thursday, 30 April 2020 09:47:17 UTC+1, Harald Alvestrand wrote:

Thanks for testing so thoroughly!

This seems to be confirming that the issue is that Google isn't including the DST root in its list of trust certificates - not something we can solve inside the WebRTC project.

You should have the same issue when connecting to the server from Chrome. (Do you?)

The official announcement of the roots.pem file seems to be this 2017 blog: https://security.googleblog.com/2017/01/the-foundation-of-more-secure-web.html

I haven't yet found the policy statement on what certificates are included in this file.

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/de1e64d0-6374-4831-bbbf-1aa62f7dbe41%40googlegroups.com.

[Harald Alvestrand]

Andrea,

we got advice from the Chrome security folks: They recommend *strongly* that applications do *not* embed compiled-in certificate lists, the way ssl_roots.h seems to encourage.

Instead, the applications need to have their own policy for who they trust, and how they update the list.

So the question then becomes: Who is in a position to make this call for Android Jitsi Meet and Android Riot IM?

I think we should try to support people who have made a decision on this - and we should warn people that the list inside webrtc is a sample list, not one they should trust blindly.

How should we do this?

Harald

To unsubscribe from this group and stop receiving emails from it, send an email to discuss-webrt...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/83c4fa02-559a-4489-92d3-9d25a7d86aaf%40googlegroups.com.

[Philipp Hancke]

when in doubt better call saul. Err... saghul. Who happens to be kinda responsible for jitsi native apps.

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/CAOqqYVGNevO%3D-rsdJFA_YbZ0WRSExh7rWkSjQ7T12ZrhycR70A%40mail.gmail.com.

[Andrea Bernabei]

Hi Harald,

thanks for looking into it.

I spent the last couple of evenings looking at Chromium source code (my first go at it) and could eventually track down the full code path it follows to set a custom certificate verifier.

From IpcPacketSocket to P2PSocketClientImpl, P2PSocketManager, P2PSocketDispatcherHost::BindReceiver, NetworkContext::CreateP2PSocketManager and eventually NetworkContext::MakeURLRequestContext and finally

CertVerifier and CertVerifyProcAndroid and the other CertVerifyProc implementations.

For future reference or anyone else who might looking into this:

https://github.com/chromium/chromium/blob/ac20efa5f9866774c7e6bebe372f4463e99be8b7/net/cert/cert_verify_proc.cc#L497

That is why Chromium does not fail the verification...because it does not use ssl_roots.h, it uses different implementations depending on the platform it is running on...which was expected, after all.

Indeed, I agree that the ideal solution would be for all applications to maintain their own policy, however, as I am sure you agree, that comes with a maintenance cost which I am not sure they can afford.

At that point, it becomes a matter of using a possibly incomplete trust store vs. not using TLS at all.

While the former might. in the worst case, provide a false sense of security...I am not sure the latter is actually a better choice overall.

Let's see what saghul thinks.

Thanks Philipp for notifying him :)

On Thursday, 7 May 2020 21:49:34 UTC+1, Harald Alvestrand wrote:

Andrea,

we got advice from the Chrome security folks: They recommend *strongly* that applications do *not* embed compiled-in certificate lists, the way ssl_roots.h seems to encourage.

Instead, the applications need to have their own policy for who they trust, and how they update the list.

So the question then becomes: Who is in a position to make this call for Android Jitsi Meet and Android Riot IM?

I think we should try to support people who have made a decision on this - and we should warn people that the list inside webrtc is a sample list, not one they should trust blindly.

How should we do this?

Harald

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/83c4fa02-559a-4489-92d3-9d25a7d86aaf%40googlegroups.com.

[Andrea Bernabei]

Any update? :)

[Andrea Bernabei]

Hi there, any update? :) I'm happy to help to push this forward

[Sudheerkumar M]

Hi,

We were facing a similar issue while using the TURNs protocol. We were able to fix this using the full chain certificate deployment on the system where the turn server is running.

We used Digicert and coturn as the turn server. Let me know if I can be of any help.

Thanks 

Sudheer.

[Andrea Bernabei]

Thanks for your joining the discussion, unfortunately we are already using the fullchain.

This is a problem with LetsEncrypt's root CA not being part of the trust store used by webrtc clients such as Android's Jitsi app and Android's official Matrix client which both use WebRTC's default hardcoded list of root CAs (and that does not include LetsEncrypt's root CA)

[Sudheerkumar M]

Oh ya,

I was going through the other threads. Okay 👍

On Mon, 22 Jun 2020 at 14:25, Andrea Bernabei <and.be...@gmail.com> wrote:

Thanks for your joining the discussion, unfortunately we are already using the fullchain.

This is a problem with LetsEncrypt's root CA not being part of the trust store used by webrtc clients such as Android's Jitsi app and Android's official Matrix client which both use WebRTC's default hardcoded list of root CAs (and that does not include LetsEncrypt's root CA)

--

---
You received this message because you are subscribed to the Google Groups "discuss-webrtc" group.

To unsubscribe from this group and stop receiving emails from it, send an email to discuss-webrt...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/2cf6c328-b403-40f1-83ab-5d68ecc274e8o%40googlegroups.com.

[Taylor Brandstetter]

My current plan is to borrow the chromium code for using the OS trust store instead of relying on the built-in roots, at least for Android and iOS. Additionally we might add an API that would let an application inject their own list of roots.

This isn't my top priority at the moment, so if you want to help, here's a bug we can coordinate on: https://bugs.chromium.org/p/webrtc/issues/detail?id=11710

On Mon, Jun 22, 2020 at 1:41 AM Andrea Bernabei <and.be...@gmail.com> wrote:

Hi there, any update? :) I'm happy to help to push this forward

--

---
You received this message because you are subscribed to the Google Groups "discuss-webrtc" group.

To unsubscribe from this group and stop receiving emails from it, send an email to discuss-webrt...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/6df6696c-6325-4c60-93a4-11d5741fcd06o%40googlegroups.com.

[Andrea Bernabei]

Hi Taylor,

thanks for pushing this forward! I will track the bug you have created and see how I can help :)

On Tuesday, 23 June 2020 00:24:12 UTC+1, Taylor Brandstetter wrote:

My current plan is to borrow the chromium code for using the OS trust store instead of relying on the built-in roots, at least for Android and iOS. Additionally we might add an API that would let an application inject their own list of roots.

This isn't my top priority at the moment, so if you want to help, here's a bug we can coordinate on: https://bugs.chromium.org/p/webrtc/issues/detail?id=11710

On Mon, Jun 22, 2020 at 1:41 AM Andrea Bernabei <and.b...@gmail.com> wrote:

Hi there, any update? :) I'm happy to help to push this forward

--

---
You received this message because you are subscribed to the Google Groups "discuss-webrtc" group.

To unsubscribe from this group and stop receiving emails from it, send an email to discuss...@googlegroups.com.

[Prayas Lashkari]

Hi Guys,
I built a react native webrtc app, I am trying figure out how to keep my connection alive when the app is background, so far no luck
Do you guys have anything to say on this, any direction?
I read somewhere to create a foreground service, I created a foreground service but somehow it isnt working.
So, adding join function as an headless task, should I explicitly add audio related functions?


Somehow Jitsi Meet did the same thing.

[Harald Alvestrand]

Please start a new thread for a new subject; that makes it easier to keep different topics apart.

To unsubscribe from this group and stop receiving emails from it, send an email to discuss-webrt...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/8fa17693-d1a5-4911-84a9-dee8d8f3025bn%40googlegroups.com.