Static build crashes in Fedora

[jeroen]

Not sure if Fedora is a supported platform, but for example this Dockerfile:

FROM fedora:42

RUN dnf install -y R-core

RUN R -e 'install.packages("pak", repos = sprintf("https://r-lib.github.io/p/pak/stable/%s/%s/%s", .Platform$pkgType, R.Version()$os, R.Version()$arch))'

RUN R -e 'pak::pak("tidyverse")'

Gives me

R session closed the process connection, killed
NULL
NULL
NULL
29
1749305338

[gaborcsardi]

The static (lib)curl is not compatible with the Fedora 42 libc:

> library("curl", lib.loc="/usr/lib64/R/library/pak/library")
> curl::curl_fetch_memory("https://httpbin.org")

 *** caught segfault ***
address 0xfffffffffffffff9, cause 'memory not mapped'

Traceback:
 1: curl::curl_fetch_memory("https://httpbin.org")

Possible actions:
1: abort (with core dump, if enabled)
2: normal R exit
3: exit R without saving workspace
4: exit R saving workspace
Selection: 3
[root@62d02f08b347 ~]#

I need to start building a separate glibc build for pak.

[gaborcsardi]

A workaround is to install the R-devel system package, and probably libcurl-devel as well, and compile pak from source.

[jeroen]

Hmm interesting. Did Fedora switch to another libc or is this a bug in glibc?

Ah I forgot you build with static musl-c. So there is a change in glibc that makes in incompatible with MUSL?

[gaborcsardi]

I think in general you can't expect to be able to call glibc from musl. We were lucky that it was working until now. But e.g. I could not update openssl or libcurl either, because of this. So it was always the plan to create a glibc specific version. But now it is urgent...

[jeroen]

What I find odd is that there is no problem on e.g. debian:testing which has a more recent glibc than fedora:40.

[gaborcsardi]

Yeah, IDK what triggers it, maybe that Fedora packages are compiled for a more modern CPU, but maybe something completely different. Anyway, I'll get to this ASAP.

[jeroen]

Perhaps related to the 64-bit time_t transition https://wiki.debian.org/ReleaseGoals/64bit-time

[gaborcsardi]

My impression was that the memory layout can be slightly different between musl and glibc, so you cannot necessarily free() a piece of memory that was allocated with the other libc. But it can be anything, really, they are just not ABI compatible, I think.

[jeroen]

OK. FWIW the last version of Fedora that seems compatible is Fedora:37 which has gcc12 and glibc 2.36. The crash appears starting fedora:38 which has gcc-13 and glibc-2.37. Which does co-incide with the time64 ABI changes. But it probably doesn't matter indeed what is the cause.

[gaborcsardi]

That is very weird because we use it on some of the R-hub images that have more recent Fedora. One of them actually has Fedora 42. Hmmm. Maybe it is only an issue with the stock Fedora R then?

[gaborcsardi]

Yeah, if I install our R build with rig, then it works.

[jeroen]

Does your R build not use the system libcurl from Fedora? Perhaps there is a conflict with the libcurl version used by pak and the one used by base R.

[gaborcsardi]

It does use everything from Fedora.

[jeroen]

I think the problem is that your static build of the curl package is linked to a too old version of curl+openssl, and causes symbol collisions with the local curl+openssl loaded by base-R.

ldd /usr/lib64/R/library/pak/library/curl/libs/curl.so
#         statically linked

The libs are statically linked from pak-libs which currently seem to have openssl 1.1.1m. If we compare output:

docker run --rm -it ghcr.io/r-devel/fedora

This is the local curl/openssl on Fedora:

/opt/R/4.5.1/bin/R -e "libcurlVersion()"
# [1] "8.11.1"
# [1] "OpenSSL/3.2.4"
/usr/bin/R -e "libcurlVersion()"
# [1] "8.11.1"
# [1] "OpenSSL/3.2.4"

However the R package compiled by pak shows different curl and OpenSSL versions, somewhat surprisingly seems to pick up a different TLS backend on different R installations:

/usr/bin/R -e "library('curl', lib.loc='/usr/lib64/R/library/pak/library')"
# Using libcurl 7.68.0 with OpenSSL/3.2.0d ???

/opt/R/4.5.1/bin/R -e "library('curl', lib.loc='/usr/lib64/R/library/pak/library')"
# Using libcurl 7.68.0 with OpenSSL/1.1.1   ???

OpenSSL 1.1 and 3.0 are not ABI compatible that explains a crash.