uniffi-rs errors reading from large release staticlib ELF binaries #122

Johennes · 2024-10-10T19:41:07Z

With UBRN from 7afb49b and the following ubrn.yaml

---
crate:
  repo: https://github.com/matrix-org/matrix-rust-sdk
  rev: 19b9a73ecc3e31d502dbf0c5850bfdfaddf02afe # matrix-sdk-ffi/20241008
  manifestPath: bindings/matrix-sdk-ffi/Cargo.toml
android:
  cargoExtras: []
  directory: android
  jniLibs: src/main/jniLibs
  targets:
    - arm64-v8a
    - armeabi-v7a
ios:
  cargoExtras: []
  directory: ios
  targets:
    - aarch64-apple-ios
    - aarch64-apple-ios-sim
  xcodebuildExtras: []
  frameworkName: build/RnMatrixRustSdk
noOverwrite: []
turboModule:
  cpp: cpp
  ts: src
bindings:
  cpp: cpp/generated
  ts: src/generated

yarn ubrn build android --release --config ubrn.yaml --and-generate fails for me:

...
-- Copying into jniLibs directory
rm -Rf /Users/jm/Code/react-native-matrix-sdk/android/src/main/jniLibs
cp /Users/jm/Code/react-native-matrix-sdk/rust_modules/matrix-rust-sdk/target/aarch64-linux-android/release/libmatrix_sdk_ffi.a /Users/jm/Code/react-native-matrix-sdk/android/src/main/jniLibs/arm64-v8a/libmatrix_sdk_ffi.a
cp /Users/jm/Code/react-native-matrix-sdk/rust_modules/matrix-rust-sdk/target/armv7-linux-androideabi/release/libmatrix_sdk_ffi.a /Users/jm/Code/react-native-matrix-sdk/android/src/main/jniLibs/armeabi-v7a/libmatrix_sdk_ffi.a
Error: Failed to extract data from archive member `matrix_sdk_ffi.matrix_sdk_ffi.d37ea2d03792f472-cgu.12.rcgu.o`

@zzorba have you tried UBRN on the Rust SDK in release mode yet by any chance?

The text was updated successfully, but these errors were encountered:

zzorba · 2024-10-11T00:18:20Z

We primarily build release, to avoid the extra disk space. But I haven't run the latest version and we have a slightly modified build script still.

I will test tomorrow to confirm against main.

Johennes · 2024-10-11T06:17:45Z

Doing the same for iOS works, interestingly, as yarn ubrn build ios --release --config ubrn.yaml --and-generate doesn't fail for me.

As a workaround one can just build for both platforms and then run yarn generate all, pointing it to the iOS library:

yarn ubrn generate all --config ubrn.yaml build/RnMatrixRustSdk.xcframework/ios-arm64/libmatrix_sdk_ffi.a

jhugman · 2024-10-11T14:12:16Z

From your description, I thought it may be something to do with stripping of the Android binaries. This is done here.

However, I just quickly tried to reproduce, and could not. :( (on main)

I have not yet tried with 7afb49b .

Johennes · 2024-10-11T14:17:57Z

Hm, interesting. I'm not sure if it matters but I'm on rustc 1.81.0 (eeb90cda1 2024-09-04).

Johennes · 2024-10-11T18:27:53Z

So I've tried main and manually disabled stripping but am still left with the same error.

$ ../uniffi-bindgen-react-native/bin/cli build android --release --config ubrn.yaml --and-generate
Running cd "/Users/jm/Code/react-native-matrix-sdk/rust_modules/matrix-rust-sdk/bindings/matrix-sdk-ffi" && "cargo" "ndk" "--manifest-path" "/Users/jm/Code/react-native-matrix-sdk/rust_modules/matrix-rust-sdk/bindings/matrix-sdk-ffi/Cargo.toml" "--target" "arm64-v8a" "--platform" "21" "--no-strip" "--" "build" "--release"
    Building arm64-v8a (aarch64-linux-android)
    ...
    Finished `release` profile [optimized] target(s) in 10m 11s
-- Copying into jniLibs directory
rm -Rf /Users/jm/Code/react-native-matrix-sdk/android/src/main/jniLibs
cp /Users/jm/Code/react-native-matrix-sdk/rust_modules/matrix-rust-sdk/target/aarch64-linux-android/release/libmatrix_sdk_ffi.a /Users/jm/Code/react-native-matrix-sdk/android/src/main/jniLibs/arm64-v8a/libmatrix_sdk_ffi.a
Error: Failed to extract data from archive member `matrix_sdk_ffi.matrix_sdk_ffi.d37ea2d03792f472-cgu.12.rcgu.o`

Caused by:
    Unexpected metadata code: 255

Judging by the error message, it stems from https://github.com/mozilla/uniffi-rs/blob/438778213f6014dc44204acd3e6b92c29b439405/uniffi_bindgen/src/macro_metadata/extract.rs#L149. So might not actually be a UBRN issue.

Johennes · 2024-10-11T18:31:50Z

Caused by:
    Unexpected metadata code: 255

I don't recall seeing this yesterday but I might have just missed it. It appears to originate from https://github.com/mozilla/uniffi-rs/blob/438778213f6014dc44204acd3e6b92c29b439405/uniffi_meta/src/reader.rs#L62.

jhugman · 2024-10-17T16:21:17Z

@Johennes Are you still seeing this?

Johennes · 2024-10-18T06:51:39Z

I tried rebuilding and regenerating matrix-rust-sdk@matrix-sdk-ffi/20241008 with UBRN from current main and, sadly, its the same error.

jhugman · 2024-10-23T19:07:03Z

Stack trace from reproduction:

Error: Failed to extract data from archive member `matrix_sdk_ffi.matrix_sdk_ffi.bf8c76dbb4572677-cgu.07.rcgu.o`

Caused by:
    Unexpected metadata code: 88

Stack backtrace:
   0: std::backtrace_rs::backtrace::libunwind::trace
             at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/../../backtrace/src/backtrace/libunwind.rs:116:5
   1: std::backtrace_rs::backtrace::trace_unsynchronized
             at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2: std::backtrace::Backtrace::create
             at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/backtrace.rs:331:13
   3: anyhow::error::<impl anyhow::Error>::msg
             at /Users/jhugman/.cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.86/src/backtrace.rs:27:14
   4: anyhow::__private::format_err
             at /Users/jhugman/.cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.86/src/lib.rs:692:13
   5: uniffi_meta::reader::MetadataReader::read_metadata
             at /Users/jhugman/.cargo/registry/src/index.crates.io-6f17d22bba15001f/uniffi_meta-0.28.0/src/reader.rs:62:18
   6: uniffi_meta::reader::read_metadata
             at /Users/jhugman/.cargo/registry/src/index.crates.io-6f17d22bba15001f/uniffi_meta-0.28.0/src/reader.rs:10:5
   7: uniffi_meta::Metadata::read
             at /Users/jhugman/.cargo/registry/src/index.crates.io-6f17d22bba15001f/uniffi_meta-0.28.0/src/lib.rs:485:9
   8: uniffi_bindgen::macro_metadata::extract::ExtractedItems::extract_item
             at /Users/jhugman/.cargo/registry/src/index.crates.io-6f17d22bba15001f/uniffi_bindgen-0.28.0/src/macro_metadata/extract.rs:178:25
   9: uniffi_bindgen::macro_metadata::extract::extract_from_elf
             at /Users/jhugman/.cargo/registry/src/index.crates.io-6f17d22bba15001f/uniffi_bindgen-0.28.0/src/macro_metadata/extract.rs:53:13
  10: uniffi_bindgen::macro_metadata::extract::extract_from_bytes
             at /Users/jhugman/.cargo/registry/src/index.crates.io-6f17d22bba15001f/uniffi_bindgen-0.28.0/src/macro_metadata/extract.rs:29:29
  11: uniffi_bindgen::macro_metadata::extract::extract_from_archive
             at /Users/jhugman/.cargo/registry/src/index.crates.io-6f17d22bba15001f/uniffi_bindgen-0.28.0/src/macro_metadata/extract.rs:138:18
  12: uniffi_bindgen::macro_metadata::extract::extract_from_bytes
             at /Users/jhugman/.cargo/registry/src/index.crates.io-6f17d22bba15001f/uniffi_bindgen-0.28.0/src/macro_metadata/extract.rs:32:37
  13: uniffi_bindgen::macro_metadata::extract::extract_from_library
             at /Users/jhugman/.cargo/registry/src/index.crates.io-6f17d22bba15001f/uniffi_bindgen-0.28.0/src/macro_metadata/extract.rs:24:5
  14: uniffi_bindgen::library_mode::find_components
             at /Users/jhugman/.cargo/registry/src/index.crates.io-6f17d22bba15001f/uniffi_bindgen-0.28.0/src/library_mode.rs:108:17
  15: uniffi_bindgen::library_mode::generate_bindings
             at /Users/jhugman/.cargo/registry/src/index.crates.io-6f17d22bba15001f/uniffi_bindgen-0.28.0/src/library_mode.rs:49:26
  16: ubrn_bindgen::bindings::BindingsArgs::run
             at /Users/jhugman/workspaces/react-native-uniffi/work/uniffi-bindgen-react-native/crates/ubrn_bindgen/src/bindings/mod.rs:127:13
  17: uniffi_bindgen_react_native::generate::GenerateAllCommand::run
             at /Users/jhugman/workspaces/react-native-uniffi/work/uniffi-bindgen-react-native/crates/ubrn_cli/src/generate.rs:122:13
  18: uniffi_bindgen_react_native::generate::GenerateCmd::run
             at /Users/jhugman/workspaces/react-native-uniffi/work/uniffi-bindgen-react-native/crates/ubrn_cli/src/generate.rs:52:17
  19: uniffi_bindgen_react_native::generate::GenerateArgs::run
             at /Users/jhugman/workspaces/react-native-uniffi/work/uniffi-bindgen-react-native/crates/ubrn_cli/src/generate.rs:22:9
  20: uniffi_bindgen_react_native::cli::CliCmd::run
             at /Users/jhugman/workspaces/react-native-uniffi/work/uniffi-bindgen-react-native/crates/ubrn_cli/src/cli.rs:40:34
  21: uniffi_bindgen_react_native::main
             at /Users/jhugman/workspaces/react-native-uniffi/work/uniffi-bindgen-react-native/crates/ubrn_cli/src/main.rs:24:5
  22: core::ops::function::FnOnce::call_once
             at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/ops/function.rs:250:5
  23: std::sys::backtrace::__rust_begin_short_backtrace
             at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/sys/backtrace.rs:152:18
  24: std::rt::lang_start::{{closure}}
             at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/rt.rs:162:18
  25: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/ops/function.rs:284:13
  26: std::panicking::try::do_call
             at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panicking.rs:557:40
  27: std::panicking::try
             at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panicking.rs:521:19
  28: std::panic::catch_unwind
             at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panic.rs:350:14
  29: std::rt::lang_start_internal::{{closure}}
             at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/rt.rs:141:48
  30: std::panicking::try::do_call
             at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panicking.rs:557:40
  31: std::panicking::try
             at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panicking.rs:521:19
  32: std::panic::catch_unwind
             at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panic.rs:350:14
  33: std::rt::lang_start_internal
             at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/rt.rs:141:20
  34: std::rt::lang_start
             at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/rt.rs:161:17
  35: _main

jhugman · 2024-10-23T19:07:21Z

Trying to characterize this issue with the given YAML.

Once checked out, I'm building the matrix library, from within the rust_modules/matrix-rust-sdk directory.

Then feeding the resulting lib into the bundled uniffi-bindgen within that library. i.e. no uniffi-bindgen-react-native code.

This works:

$ cargo run --manifest-path uniffi-bindgen/Cargo.toml  -- generate --language kotlin --library target/aarch64-linux-android/release/libmatrix_sdk_ffi.so  --out-dir kt

We can also get swift to generate from this same library:

$ cargo run --manifest-path uniffi-bindgen/Cargo.toml  -- generate --language switt --library target/aarch64-linux-android/release/libmatrix_sdk_ffi.so  --out-dir swift

Swift is normally generated from a .a file, and python and kotlin usually need a .so file.

We seem to be having problems with the reading of staticlib release builds; we'd expect this to work, even though it's a build for a different architecture. However:

$ cargo run --manifest-path uniffi-bindgen/Cargo.toml  -- generate --language swift --library target/aarch64-linux-android/release/libmatrix_sdk_ffi.a  --out-dir swift
    Finished `dev` profile [unoptimized] target(s) in 0.11s
     Running `target/debug/uniffi-bindgen generate --language swift --library target/aarch64-linux-android/release/libmatrix_sdk_ffi.a --out-dir swift`
Failed to extract data from archive member `matrix_sdk_ffi.matrix_sdk_ffi.bf8c76dbb4572677-cgu.07.rcgu.o`

The debug build seems to work:

$ cargo run --manifest-path uniffi-bindgen/Cargo.toml  -- generate --language swift --library target/aarch64-linux-android/debug/libmatrix_sdk_
ffi.a  --out-dir swift
    Finished `dev` profile [unoptimized] target(s) in 0.11s
     Running `target/debug/uniffi-bindgen generate --language swift --library target/aarch64-linux-android/debug/libmatrix_sdk_ffi.a --out-dir swift`

So, what is different about the release build coming out of cargo ndk, that uniffi-rs cannot read?

I'd also like to reduce the size of the library that is being built: @Johennes has reported

Caused by:
    Unexpected metadata code: 255

and I am now seeing:

Caused by:
    Unexpected metadata code: 88

jhugman · 2024-10-24T11:14:05Z

I think we're firmly in the territory of "not holding it wrong, but not holding it as expected".

Cutting out uniffi-bindgen-react-native altogether, I'm building the matrix sdk ffi binding with:

cd /private/tmp/turbomodule-tests/122-repro/rust_modules/matrix-rust-sdk/bindings/matrix-sdk-ffi && \
  cargo ndk \
  --manifest-path /private/tmp/turbomodule-tests/122-repro/rust_modules/matrix-rust-sdk/bindings/matrix-sdk-ffi/Cargo.toml \
  --target arm64-v8a \
  --platform 21 --no-strip -- \
  build --release

I have lightly instrumented uniffi-rs to output the extracted metadata and types. Here is the output from the aarch64-linux-android/debug build.

And here is the output from aarch64-linux-android/release:

Metadata symbol UNIFFI_META_MATRIX_SDK_FFI_INTERFACE_ENCRYPTION
thread 'main' panicked at uniffi_meta/src/metadata.rs:84:18:
XXXX UNKNOWN ITEM 88 XXXX
stack backtrace:
   0: rust_begin_unwind
             at /rustc/7cf61ebde7b22796c69757901dd346d0fe70bd97/library/std/src/panicking.rs:647:5
   1: core::panicking::panic_fmt
             at /rustc/7cf61ebde7b22796c69757901dd346d0fe70bd97/library/core/src/panicking.rs:72:14
   2: uniffi_meta::metadata::codes::metadata_as_str
             at ./uniffi_meta/src/metadata.rs:84:18
   3: uniffi_meta::reader::MetadataReader::read_metadata
             at ./uniffi_meta/src/reader.rs:39:33
   4: uniffi_meta::reader::read_metadata
             at ./uniffi_meta/src/reader.rs:10:5
   5: uniffi_meta::Metadata::read
             at ./uniffi_meta/src/lib.rs:505:9
   6: uniffi_bindgen::macro_metadata::extract::ExtractedItems::extract_item
             at ./uniffi_bindgen/src/macro_metadata/extract.rs:184:25
   7: uniffi_bindgen::macro_metadata::extract::extract_from_elf
             at ./uniffi_bindgen/src/macro_metadata/extract.rs:53:13
   8: uniffi_bindgen::macro_metadata::extract::extract_from_bytes
             at ./uniffi_bindgen/src/macro_metadata/extract.rs:29:29
   9: uniffi_bindgen::macro_metadata::extract::extract_from_archive
             at ./uniffi_bindgen/src/macro_metadata/extract.rs:143:18
  10: uniffi_bindgen::macro_metadata::extract::extract_from_bytes
             at ./uniffi_bindgen/src/macro_metadata/extract.rs:32:37
  11: uniffi_bindgen::macro_metadata::extract::extract_from_library
             at ./uniffi_bindgen/src/macro_metadata/extract.rs:24:5
  12: uniffi_bindgen::library_mode::find_components
             at ./uniffi_bindgen/src/library_mode.rs:104:17
  13: uniffi_bindgen::library_mode::generate_bindings
             at ./uniffi_bindgen/src/library_mode.rs:46:26
  14: uniffi::cli::uniffi_bindgen::gen_library_mode
             at ./uniffi/src/cli/uniffi_bindgen.rs:207:38
  15: uniffi::cli::uniffi_bindgen::run_main
             at ./uniffi/src/cli/uniffi_bindgen.rs:297:17
  16: uniffi::cli::uniffi_bindgen_main
             at ./uniffi/src/cli/mod.rs:9:21
  17: uniffi_bindgen::main
             at ./cli/src/main.rs:2:5
  18: core::ops::function::FnOnce::call_once
             at /rustc/7cf61ebde7b22796c69757901dd346d0fe70bd97/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

i.e. it seems to be failing at the first metadata symbol UNIFFI_META_MATRIX_SDK_FFI_INTERFACE_ENCRYPTION with an unknown metadata item with code 88.

I don't ~~really~~ understand how the metadata is being written or how it differs between debug and release.

jhugman · 2024-10-25T15:42:44Z

This will be fixed by mozilla/uniffi-rs#2286.

Johennes changed the title ~~Generating from release build of matrix-rust-sdk fails~~ Generating from release build of matrix-rust-sdk fails for Android Oct 16, 2024

jhugman mentioned this issue Oct 21, 2024

yarn ubrn:ios intermittent failures #136

Closed

jhugman mentioned this issue Oct 24, 2024

Error: Failed to extract data from archive member mozilla/uniffi-rs#2285

Closed

jhugman changed the title ~~Generating from release build of matrix-rust-sdk fails for Android~~ uniffi-rs errors reading from large release staticlib ELF binaries Oct 28, 2024

jhugman added the waiting on uniffi-rs Waiting on a release from uniffi label Nov 11, 2024

jhugman linked a pull request Nov 15, 2024 that will close this issue

Bump uniffi version to 0.28.3 #164

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

uniffi-rs errors reading from large release staticlib ELF binaries #122

uniffi-rs errors reading from large release staticlib ELF binaries #122

Johennes commented Oct 10, 2024

zzorba commented Oct 11, 2024

Johennes commented Oct 11, 2024

jhugman commented Oct 11, 2024

Johennes commented Oct 11, 2024

Johennes commented Oct 11, 2024

Johennes commented Oct 11, 2024

jhugman commented Oct 17, 2024

Johennes commented Oct 18, 2024

jhugman commented Oct 23, 2024

jhugman commented Oct 23, 2024

jhugman commented Oct 24, 2024 •

edited

Loading

jhugman commented Oct 25, 2024

uniffi-rs errors reading from large release staticlib ELF binaries #122

uniffi-rs errors reading from large release staticlib ELF binaries #122

Comments

Johennes commented Oct 10, 2024

zzorba commented Oct 11, 2024

Johennes commented Oct 11, 2024

jhugman commented Oct 11, 2024

Johennes commented Oct 11, 2024

Johennes commented Oct 11, 2024

Johennes commented Oct 11, 2024

jhugman commented Oct 17, 2024

Johennes commented Oct 18, 2024

jhugman commented Oct 23, 2024

jhugman commented Oct 23, 2024

jhugman commented Oct 24, 2024 • edited Loading

jhugman commented Oct 25, 2024

jhugman commented Oct 24, 2024 •

edited

Loading