[R-SIG-Mac] Libre SSL bug on MacOS Monterey => error in download.file()

[Petr Bouchal]

Dear all,

In brief: on Monterey, R cannot reach certain web domains due to a bug in Libre SSL - and perhaps not relying on system curl/openssl in R would be a systematic solution to this and símilar issues.

Specifically: on MacOS Monterey 12.1 using R 4.1.2, download.file() and other functions that rely on system-provided curl/openssl/Libre SSL (including in the curl package) have been failing on specific domains.

So running

download.file(“https://www.czso.cz/”, tempfile())

returns:

status was ‘SSL connect error’

the underlying error being

error:06FFF089:digital envelope routines:CRYPTO_internal:bad key length.

This is caused by the Libre SSL bundled in MacOS Monterey and also affects several other domains, most notably https://libzip.org.

It is clearly an OS bug but infortunately also a situation where it affects R users because of how R relates to system libraries and is very difficult to work around.

It has manifested on CRAN (causing a package archival) and Github outside of R, so is not caused by a specific machine. It can be replicated on both M1 and Intel and also occurs when using curl in the system command line.

The czso.cz domain is the Czech Statistical Office, which makes it quite important for a number of users, also of a package I maintain (czso) which relies on accessing this domain. I have reported this to the server admin but since the problem is in the OS, I do not expect them to be able to help. I am not an expert in web security so cannot tell if there is anything in the certificates which could be causing this. In browsers, no such issue occurs and the server is configured correctly as per ssllabs.com testing. I have also reported to Apple but it is unclear whether they will fix this given the rare nature of the issue.

It is difficult to work around even on individual machines as replacing the system curl/openssl requires steps beyond what a most users are comfortable with (or should be doing to begin with). Using HTTP instead of HTTPS does not work, nor does using curl —insecure and equivalents.

This brings back the question of whether R on MacOS should include its own openssl instead of relying on the system-provided library. This has been discussed on the r-devel list: https://stat.ethz.ch/pipermail/r-devel/2020-June/079657.html.

Apple also recommends against relying on shared openssl, if I understand this correctly: https://developer.apple.com/forums/thread/89051. Given Apple’s approach to openssl/Libre SSL in MacOS (the bundled Libre SSL version is 3 years old), such hard-to-handle issues are likely to reappear over time. (I don’t have in-depth knowledge of how R is compiled, so apologies for any inaccuracies; hopefully it is clear what I mean.)

I’d be grateful for any thoughts on how this might be handled in the specific case and perhaps generally.

Kind regards
Petr Bouchal

_______________________________________________
R-SIG-Mac mailing list
R-SI...@r-project.org
https://stat.ethz.ch/mailman/listinfo/r-sig-mac

[Jeroen Ooms]

On Mon, Jan 10, 2022 at 11:22 AM Petr Bouchal <pbou...@gmail.com> wrote:
>
> Dear all,
>
> In brief: on Monterey, R cannot reach certain web domains due to a bug in Libre SSL - and perhaps not relying on system curl/openssl in R would be a systematic solution to this and símilar issues.
>
> Specifically: on MacOS Monterey 12.1 using R 4.1.2, download.file() and other functions that rely on system-provided curl/openssl/Libre SSL (including in the curl package) have been failing on specific domains.
>
> So running
>
> download.file(“https://www.czso.cz/”, tempfile())
>
> returns:
>
> status was ‘SSL connect error’
>
> the underlying error being
>
> error:06FFF089:digital envelope routines:CRYPTO_internal:bad key length.

I have to investigate this further (it looks like a buggy TLS server
actually), but as a workaround you can set an environment variable
CURL_SSL_BACKEND=SecureTransport when starting R, see for details:
https://curl.se/libcurl/c/libcurl-env.html

The version of libcurl that is included with the past few versions of
MacOS is actually built with support for 2 TLS back-ends: LibreSSL and
native apple TLS (aka SecureTransport). You can override the default
using the environment variable above, but you have to set it before
libcurl gets initiated, hence before making any http connections in
the R session, e.g. in your .Renviron.

You can see which version is active by looking at
curl::curl_version()$ssl_version, the version in parenthesis is Try
running:

CURL_SSL_BACKEND=openssl R -e "curl::curl_version()$ssl_version"
CURL_SSL_BACKEND=SecureTransport R -e "curl::curl_version()$ssl_version"

The same version of libcurl is also used by base-R in download.file().
I've also explained this a bit (mostly for windows) in this vignette:
https://cran.r-project.org/web/packages/curl/vignettes/windows.html

[Simon Urbanek]

Petře,

thanks, for the detailed analysis. It is rather curious that the issue appears only on _newer_ systems - we are more used to issues due to older CA chains and similar. It looks like an Apple bug on specific systems, so hopefully it will be fixed eventually. In general I was trying to avoid having to supply our own SSL library since that opens a whole can of worms - on one hand due the dependency issues (which libraries get compiled against what) and on the other hand we become responsible for security updates.

Thanks to Jeroen for the work-around (CURL_SSL_BACKEND=SecureTransport), using the native API is certainly preferred, there have been several issues with both OpenSSL and LibreSSL before. It seems that Apple has been flip-flopping with libcurl a lot - on El Capitan it was shipped with SecureTransport, on High-Sierra with LibreSSL, on Catalina and higher with both, but Libre the default.

I am somewhat less apprehensive to use static libcurl for R than SSL libraries as the fallout is a bit smaller. As a trial I have added static curl[2] which is close to the Apple build minus MultiSSL to big-sur nightly builds of R[3] and as expected that solves the problem. It may not be entirely unproblematic for package space, because packages often forget to prepend --static when using static builds of libraries, and so do other dependencies that may use curl, but I'll see what comes out of it.

Cheers,
Šimon

[1] - https://github.com/R-macos/recipes
[2] - https://github.com/R-macos/recipes/blob/add-ons/recipes/curl
[3] - https://mac.r-project.org/

[Petr Bouchal]

Many thanks Jeroen - that is very helpful, I was not aware of the capability in curl to switch SSL backends. (The backend also shows up in the curl package onLoad message, which is helpful).

Kind regards
Petr

[Petr Bouchal]

Šimone,

Thanks for this. I understand your concern about being responsible for the SSL side and can just about imagine the technical complexity of getting it done. Many thanks also for the trial build, I very much appreciate you doing this - once I can get it tested on an M1 machine, I will report back but it’s encouraging to hear that it is feasible and solves the issue.

As for Apple’s approach to SSL, my understanding from the bits of information I have been able to find is that they provide very little guarantee/stability/documentation around this, as also attested by the flip flopping you mentioned.

Thanks again.

Kind regards
Petr

[Jeroen Ooms]

On Tue, Jan 11, 2022 at 10:12 PM Simon Urbanek
<simon....@r-project.org> wrote:
>
> Petře,
>
> thanks, for the detailed analysis. It is rather curious that the issue appears only on _newer_ systems - we are more used to issues due to older CA chains and similar. It looks like an Apple bug on specific systems, so hopefully it will be fixed eventually. In general I was trying to avoid having to supply our own SSL library since that opens a whole can of worms - on one hand due the dependency issues (which libraries get compiled against what) and on the other hand we become responsible for security updates.
>
> Thanks to Jeroen for the work-around (CURL_SSL_BACKEND=SecureTransport), using the native API is certainly preferred, there have been several issues with both OpenSSL and LibreSSL before. It seems that Apple has been flip-flopping with libcurl a lot - on El Capitan it was shipped with SecureTransport, on High-Sierra with LibreSSL, on Catalina and higher with both, but Libre the default.
>
> I am somewhat less apprehensive to use static libcurl for R than SSL libraries as the fallout is a bit smaller. As a trial I have added static curl[2] which is close to the Apple build minus MultiSSL to big-sur nightly builds of R[3] and as expected that solves the problem. It may not be entirely unproblematic for package space, because packages often forget to prepend --static when using static builds of libraries, and so do other dependencies that may use curl, but I'll see what comes out of it.

I would much recommend to stick with the apple version of libcurl;
perhaps override the default ssl-backend if you like. There is some
example code to do this in the curl package that you could adapt for
base r: https://github.com/jeroen/curl/blob/master/src/ssl.c

The benefit of dynamically linking to apple's libcurl is that we
automatically get a version of libcurl+deps+certs that is tuned and
maintained for that version of macos, including future ones. If you
ship a version of base-R with a static libcurl now, that version of R
may not work anymore a few years from now or on a future version of
macos, when things have moved on (for example, when servers start to
require TLS1.3).

[Simon Urbanek]

Yes, but if you are using an old version of R on a new system, you have a lot of other worries - you can't expect new technologies to work with old software. CURL itself has fewer evolution issues than SSL libraries. As I said, I am a big proponent of re-using system libraries as much as possible, but, for example, High Sierra doesn't ship with ST back-end support, so using a static version that does is better there as Apple doesn't not maintain the curl CAs but it does the system ones so it's arguably better. The current issue is quite curious since breaking on the latest system is quite unusual, just preferring ST works only because it is the latest system that breaks and it has the ST option.

As Brian pointed out static curl has its own issues since its pkg-config flags are broken - that's why I have not activated the add-on recipes by default, I have seen those issues before.

For R itself there are thee options:

a) add CURL_SSL_BACKEND=${CURL_SSL_BACKEND-'SecureTransport'} to $R_HOME/etc/Renviron of the distribution

b) add something like your https://github.com/r-devel/r-svn/pull/75/commits/79b22b461e527e8a46de84c145e8e5fb59e75d14 to R

c) build against static libcurl


The big advantage of the first one is that it applies to all processes, so even command line curl will then work and so will all packages.

The drawback of the second one is that it only applies the R itself. The third one could be done both for R and packages, but causes headaches resp. requires slight patching of libcurl.pc. The advantage is that it can bring more recent curl to all older systems.

I don't have a strong opinion. I am not thrilled with option b) because that is a hack just to react to something which is never a good idea from maintenance point of view (we would require all curl-based code to use it). So I think a) and c) are more palatable with a) having the benefit of handling non-R cases. A slight benefit of c) is that some dependencies require more recent curl version than provided by older systems, so that would cover it at the cost of maintaining the curl binaries. Finally, the real benefit of c) is that if Apple screws things up even more we don't care - we may not be at that point yet, though.

Cheers,
Simon

[Jeroen Ooms]

I don't think apple screwed up per se; they probably tested several
configurations and picked this one to be the safest default. TLS is a
complex protocol with many versions and implementations; if some weird
server uses some non-standard cipher or unusual response, it may just
depend on the TLS library if it can handle that. I'm sure you'll be
able to find counter examples where libre/openssl works and
SecureTransport does not. For example, a case that we often encounter
on Windows are corporate networks which require connecting via
authenticated proxy servers or using a TLS client cert, which only
works on certain back-ends, see the table in:
https://cran.r-project.org/web/packages/curl/vignettes/windows.html

So I much favor of option A. This introduces the least complexity, and
keeps the ability for users to undo our change and switch back to
CURL_SSL_BACKEND=openssl in their .Renviron. Also it is a big benefit
in practice that curl in R behaves the same as command line curl on
that same machine, in order to narrow down if a connection problem is
a bug in our R code, or if it also exists outside of R.

[Kasper Daniel Hansen]

I am not an expert, but it seems to me that switching the backend is a
runtime setting. Couldn't we detect which version of OS X we're running and
then select the backend conditionally on that test?

Best,
Kasper

--
Best,
Kasper

[[alternative HTML version deleted]]

[Petr Bouchal]

Just an update: this particular issue (MacOS Libre SSL on Monterey vs. a particular cert/domain) is resolved in MacOS 12.3 beta (21E5206e) on M1 using the standard R 4.1.2 Arm64 binary.

Secure connections to the previously problematic server are now opened correctly in {curl}, curl in shell as well as download.file() without needing to switch SSL backends.

This is probably because the MacOS 12.3 beta comes with Libre SSL 3.3.5. Presumably this change resolves other similar issues with SSL on MacOS as well.

Kind regards
Petr