bootc v1.9.0 breaks custom build with rhsm enabled

[rriemann]

Dears,

For https://eu-os.eu, I build bootc with rhsm enabled in Fedora copr. I rely on the Fedora spec to build the rpm with two minimal changes applied via sed:

#!/usr/bin/env bash

set -euxo pipefail

git clone https://src.fedoraproject.org/rpms/bootc.git -b f42 --depth=1
cd bootc
sed -i '8s#%if 0%{?rhel}#%if 0%{?fedora} || 0%{?rhel}#' bootc.spec
sed -i '25s#autorelease#autorelease -e eu_os -b 100#' bootc.spec
spectool -g bootc.spec
rpmbuild -bs bootc.spec \
  --define "_topdir ." \
  --define "_sourcedir ." \
  --define "_srcrpmdir ."

mkdir -p "$COPR_RESULTDIR"
bsdtar xf *.src.rpm -C "$COPR_RESULTDIR"

Unfortunately, this stopped working for version 1.9.0:

Source: https://copr.fedorainfracloud.org/coprs/rriemann/eu-os/builds/

Looking into Copr logs

The copr build logs are here:

https://download.copr.fedorainfracloud.org/results/rriemann/eu-os/fedora-42-x86_64/09703373-bootc/builder-live.log.gz

The spec file is here: https://copr-dist-git.fedorainfracloud.org/packages/rriemann/eu-os/bootc.git/tree/?h=f42&id=7474a1c7e8f34e0c34b7effdee82124dc45de02c

Errors in the build log:

failures:

---- install::tests::test_remove_all_noxdev stdout ----

thread 'install::tests::test_remove_all_noxdev' panicked at crates/lib/src/install.rs:2256:46:
called `Result::unwrap()` on an `Err` value: Function not implemented (os error 38)

---- boundimage::tests::test_parse_spec_dir stdout ----

thread 'boundimage::tests::test_parse_spec_dir' panicked at crates/lib/src/boundimage.rs:290:49:
called `Result::unwrap()` on an `Err` value: Querying bound images

Caused by:
    Function not implemented (os error 38)
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

---- lints::tests::test_lint_inner stdout ----

thread 'lints::tests::test_lint_inner' panicked at crates/lib/src/lints.rs:865:67:
called `Result::unwrap()` on an `Err` value: Function not implemented (os error 38)

---- lints::tests::test_lint_main stdout ----

thread 'lints::tests::test_lint_main' panicked at crates/lib/src/lints.rs:850:75:
called `Result::unwrap()` on an `Err` value: Linting

Caused by:
    Function not implemented (os error 38)

---- lints::tests::test_non_utf8 stdout ----

thread 'lints::tests::test_non_utf8' panicked at crates/lib/src/lints.rs:1029:14:
called `Result::unwrap()` on an `Err` value: Function not implemented (os error 38)


failures:
    boundimage::tests::test_parse_spec_dir
    install::tests::test_remove_all_noxdev
    lints::tests::test_lint_inner
    lints::tests::test_lint_main
    lints::tests::test_non_utf8

test result: FAILED. 93 passed; 5 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.12s

error: test failed, to rerun pass `-p bootc-lib --lib`
     Running `/builddir/build/BUILD/bootc-1.9.0-build/bootc-1.9.0/target/rpm/deps/bootc_mount-b101c593bbd2209a`

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running `/builddir/build/BUILD/bootc-1.9.0-build/bootc-1.9.0/target/rpm/deps/bootc_sysusers-f6b75e3abe28594a`

running 9 tests
test nameservice::group::tests::test_parse_lines ... ok
test nameservice::passwd::tests::test_parse_lines ... ok
test nameservice::passwd::tests::test_write_entry ... ok
test nameservice::group::tests::test_write_entry ... ok
test nameservice::shadow::tests::test_parse_lines ... ok
test nameservice::shadow::tests::test_write_entry ... ok
test tests::test_sysusers_parse ... ok
test tests::test_missing ... ok
test tests::test_sysgroups_parse ... ok

test result: ok. 9 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running `/builddir/build/BUILD/bootc-1.9.0-build/bootc-1.9.0/target/rpm/deps/bootc_tmpfiles-f47d3e822f7efe89`

running 5 tests
test tests::test_canonicalize_escape_path ... ok
test tests::test_tmpfiles_entry_get_path ... ok
test tests::test_log_regfile ... FAILED
test tests::test_tmpfiles_d_translation ... FAILED
test tests::test_translate_to_tmpfiles_d ... ok

failures:

---- tests::test_log_regfile stdout ----

thread 'tests::test_log_regfile' panicked at crates/tmpfiles/src/lib.rs:633:49:
called `Result::unwrap()` on an `Err` value: Io(Os { code: 38, kind: Unsupported, message: "Function not implemented" })
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

---- tests::test_tmpfiles_d_translation stdout ----

thread 'tests::test_tmpfiles_d_translation' panicked at crates/tmpfiles/src/lib.rs:583:49:
called `Result::unwrap()` on an `Err` value: Io(Os { code: 38, kind: Unsupported, message: "Function not implemented" })


failures:
    tests::test_log_regfile
    tests::test_tmpfiles_d_translation

test result: FAILED. 3 passed; 2 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

error: test failed, to rerun pass `-p bootc-tmpfiles --lib`
     Running `/builddir/build/BUILD/bootc-1.9.0-build/bootc-1.9.0/target/rpm/deps/etc_merge-dbc993dd61dbfd02`

[…]

test tokio_util::tests::test_cancellable ... ok
test utils::tests::test_spinner ... ok

failures:

---- container::tests::test_merge_authopts stdout ----

thread 'container::tests::test_merge_authopts' panicked at crates/ostree-ext/src/container/mod.rs:643:87:
called `Result::unwrap()` on an `Err` value: Function not implemented (os error 38)

---- globals::tests::test_config_paths stdout ----

thread 'globals::tests::test_config_paths' panicked at crates/ostree-ext/src/globals.rs:121:46:
called `Result::unwrap()` on an `Err` value: Function not implemented (os error 38)


failures:
    container::tests::test_merge_authopts
    globals::tests::test_config_paths

test result: FAILED. 48 passed; 2 failed; 0 ignored; 0 measured; 0 filtered out; finished in 5.01s

error: test failed, to rerun pass `-p ostree-ext --lib`
     Running `/builddir/build/BUILD/bootc-1.9.0-build/bootc-1.9.0/target/rpm/deps/it-574e5916701a56f6`

Looking into local build (Fedora Kinoite 42)

I tried then in a fedora42 toolbox to reproduce the issue locally.

toolbox create --image registry.fedoraproject.org/fedora-toolbox:42  f42-rpm-build 
toolbox enter f42-rpm-build   
sudo dnf install copr-rpmbuild
sudo  /usr/bin/copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9703373-fedora-42-x86_64 --chroot fedora-42-x86_64

The build also fails.

From the log:

test tests::test_translate_to_tmpfiles_d ... ok
test tests::test_log_regfile ... ok
test tests::test_tmpfiles_d_translation ... ok
test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s
     Running `/builddir/build/BUILD/bootc-1.9.0-build/bootc-1.9.0/target/rpm/deps/etc_merge-dbc993dd61dbfd02`
running 3 tests
test tests::file_to_dir ... ok
test tests::test_etc_diff ... ok
test tests::test_merge ... FAILED
failures:
---- tests::test_merge stdout ----
Error: Merging
Caused by:
    0: Merging added files
    1: Copying xattrs
    2: setxattr "security.selinux" for "/proc/self/fd/11/a/b/new_file.txt"
    3: Operation not supported (os error 95)
failures:
    tests::test_merge
test result: FAILED. 2 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.02s
error: test failed, to rerun pass `-p etc-merge --lib`
     Running `/builddir/build/BUILD/bootc-1.9.0-build/bootc-1.9.0/target/rpm/deps/ostree_ext-c3247e4ef925a29c`
running 50 tests
test chunking::test::test_advanced_packing ... ok
test chunking::test::test_exclusive_chunks ... ok
test chunking::test::test_exclusive_chunks_isolation ...

build.log

[cgwalters]

called Result::unwrap() on an Err value: Io(Os { code: 38, kind: Unsupported, message: "Function not implemented" })

I'm pretty sure this is a recent regression actually in COPR, I did #1693 to narrow down my theory a bit more that this relates to seccomp.

2: setxattr "security.selinux" for "/proc/self/fd/11/a/b/new_file.txt"

3: Operation not supported (os error 95)

I thought we'd fixed this, but maybe not. We probably need to move that one to an integration test.

Hmmm...I am a bit confused how 1.9.0 and the other COPR builds were working. In https://kojipkgs.fedoraproject.org//work/tasks/3616/137923616/build.log it did work.

This smells like environmental skew between the Koji+mock and the COPR+mock.

[henrywang]

• bootc main COPR build does not have this issue https://copr.fedorainfracloud.org/coprs/rhcontainerbot/bootc/build/9699865/

• And I checked PRs in two weeks, we didn't merge any RPM build failed PRs.

So issue should be related with building environment.

[cgwalters]

rpm-software-management/mock#1613 (comment)