Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Could not find gn executable at: /ungoogled-chromium-debian/buildtools/linux64/gn OS: Ubuntu 22.04 #334

Open
raphael10-collab opened this issue Sep 30, 2023 · 24 comments

Comments

@raphael10-collab
Copy link

raphael10-collab commented Sep 30, 2023

generate-ninja is installed :

raphy@raohy:~$ sudo apt-get install generate-ninja
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
generate-ninja is already the newest version (0.0~git20220118.0725d78-1).
0 upgraded, 0 newly installed, 0 to remove and 4 not upgraded.
raphy@raohy:~$ sudo apt-get install gn
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Note, selecting 'generate-ninja' instead of 'gn'
generate-ninja is already the newest version (0.0~git20220118.0725d78-1).
0 upgraded, 0 newly installed, 0 to remove and 4 not upgraded.
raphy@raohy:~$ 

But following these building steps: https://github.com/ungoogled-software/ungoogled-chromium-debian :
UngoogleChromiumUbuntu22.04DesktopBuildingStepsFollowed.txt

I get this error:

gn.py: Could not find gn executable at: /home/raphy/ungoogled-chromium-debian/buildtools/linux64/gn

Other info:

O.S.: Ubuntu 22.04 Desktop
raphy@raohy:~/ungoogled-chromium-debian$ python3 --version
Python 3.10.12

How to make it work?

@raphael10-collab raphael10-collab changed the title Could not find gn executable even if generate-ninja already installed . OS: Ubuntu 22.04 Could not find gn executable at: /ungoogled-chromium-debian/buildtools/linux64/gn OS: Ubuntu 22.04 Sep 30, 2023
@Ark74
Copy link

Ark74 commented Oct 4, 2023

Interesting, I'm using sbuild to build on a chroot, it looks similar but not quite.
This is the error I get.

...
WARNING: Skipping non-existant path: /<<PKGBUILDDIR>>/third_party/opus/src/doc/release.txt
WARNING: Skipping non-existant path: /<<PKGBUILDDIR>>/third_party/re2/src/CMakeLists.txt
WARNING: Skipping non-existant path: /<<PKGBUILDDIR>>/third_party/re2/src/re2/testing/re2_test.cc
WARNING: Skipping non-existant path: /<<PKGBUILDDIR>>/third_party/re2/src/util/pcre.h
WARNING: Skipping non-existant path: /<<PKGBUILDDIR>>/third_party/snappy/linux/snappy-stubs-public.h
WARNING: Skipping non-existant path: /<<PKGBUILDDIR>>/third_party/snappy/mac/snappy-stubs-public.h
WARNING: Skipping non-existant path: /<<PKGBUILDDIR>>/third_party/snappy/src/snappy-test.h
WARNING: Skipping non-existant path: /<<PKGBUILDDIR>>/third_party/snappy/win32/snappy-stubs-public.h
WARNING: Skipping non-existant path: /<<PKGBUILDDIR>>/third_party/zlib/deflate.c
INFO: Path has no substitutions: tools/md_browser/base.css
cp /<<PKGBUILDDIR>>/debian/shims/jsoncpp.gn build/linux/unbundle
build/linux/unbundle/replace_gn_files.py --system-libraries flac fontconfig freetype jsoncpp libdrm libevent libjpeg libpng libwebp libxml libxslt opus re2 snappy zlib 
make[1]: Leaving directory '/<<PKGBUILDDIR>>'
   debian/rules override_dh_auto_build
make[1]: Entering directory '/<<PKGBUILDDIR>>'
gn gen out/Release --args="build_with_tflite_lib=false chrome_pgo_phase=0 clang_use_chrome_plugins=false disable_fieldtrial_testing_config=true enable_hangout_services_extension=false enable_mdns=false enable_mse_mpeg2ts_stream_parser=true enable_nacl=false enable_reading_list=false enable_remoting=false enable_reporting=false enable_service_discovery=false enable_widevine=true exclude_unwind_tables=true google_api_key=\"\" google_default_client_id=\"\" google_default_client_secret=\"\" safe_browsing_mode=0 treat_warnings_as_errors=false use_official_google_api_keys=false use_unofficial_version_number=false  custom_toolchain=\"//build/toolchain/linux/unbundle:default\" host_toolchain=\"//build/toolchain/linux/unbundle:default\" use_custom_libcxx=false is_official_build=true is_debug=false symbol_level=1 blink_enable_generated_code_formatting=false is_cfi=false use_thin_lto=false use_sysroot=false use_gnome_keyring=false ffmpeg_branding=\"Chrome\" proprietary_codecs=true use_pulseaudio=true link_pulseaudio=true use_vaapi=true use_ozone=true use_goma=false enable_vr=false enable_iterator_debugging=false optimize_webui=true use_gio=true use_lld=true is_clang=true use_kerberos=false use_cups=true v8_enable_backtrace=true  perfetto_use_system_zlib=true use_system_lcms2=true use_system_libopenjpeg2=true use_system_libpng=true use_system_zlib=true use_system_libjpeg=true " --fail-on-unused-args
ERROR at //build/config/compiler/BUILD.gn:1418:22: Script returned non-zero exit code.
    clang_revision = exec_script("//tools/clang/scripts/update.py",
                     ^----------
Current dir: /<<PKGBUILDDIR>>/out/Release/
Command: python3 /<<PKGBUILDDIR>>/tools/clang/scripts/update.py --print-revision --verify-version=17
Returned 1 and printed out:

The expected clang version is llvmorg-17-init-8029-g27f27d15-3 but the actual version is 
Did you run "gclient sync"?

See //build/config/BUILDCONFIG.gn:333:3: which caused the file to be included.
  "//build/config/compiler:afdo",
  ^-----------------------------
make[1]: *** [debian/rules:142: override_dh_auto_build] Error 1
make[1]: Leaving directory '/<<PKGBUILDDIR>>'
make: *** [debian/rules:109: binary] Error 2
dpkg-buildpackage: error: fakeroot debian/rules binary subprocess returned exit status 2
--------------------------------------------------------------------------------
Build finished at 2023-10-04T00:10:45Z

Finished
--------

Seems to be an issue at https://commondatastorage.googleapis.com/chromium-browser-clang where the specified clang pre-compiled binaries is no longer available.

  • llvmorg-17-init-8029-g27f27d15-3

Checking...

@Ark74
Copy link

Ark74 commented Oct 4, 2023

Indeed,

CLANG_REVISION = 'llvmorg-17-init-8029-g27f27d15'
CLANG_SUB_REVISION = 3

to

CLANG_REVISION = 'llvmorg-17-init-857-g4f2fde78'
CLANG_SUB_REVISION = 1

at tools/clang/scripts/update.py

That should get you pass that line.
Regards.

@Ark74
Copy link

Ark74 commented Oct 4, 2023

Oh well, later on seems that we need a specific version otherwise it will fail.

[14976/54516] CXX obj/chrome/common/channel_info/channel_info.o
FAILED: obj/chrome/common/channel_info/channel_info.o 
clang++-14 -MMD -MF obj/chrome/common/channel_info/channel_info.o.d -DUSE_UDEV -DUSE_AURA=1 -DUSE_GLIB=1 -DUSE_OZONE=1 -DOFFICIAL_BUILD -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D_FORTIFY_SOURCE=2 -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -DNO_UNWIND_TABLES -D_GNU_SOURCE -DCR_CLANG_REVISION=\"llvmorg-17-init-857-g4f2fde78-1\" -DNDEBUG -DNVALGRIND -DDYNAMIC_ANNOTATIONS_ENABLED=0 -DGLIB_VERSION_MAX_ALLOWED=GLIB_VERSION_2_56 -DGLIB_VERSION_MIN_REQUIRED=GLIB_VERSION_2_56 -DBASE_USE_PERFETTO_CLIENT_LIBRARY=1 -I../.. -Igen -I../../third_party/perfetto/include -Igen/third_party/perfetto/build_config -Igen/third_party/perfetto -Igen/shim_headers/zlib_shim -Igen/shim_headers/jsoncpp_shim -Igen/shim_headers/libevent_shim -I../../third_party/abseil-cpp -I../../third_party/boringssl/src/include -I../../third_party/protobuf/src -Igen/protoc_out -Wall -Wextra -Wimplicit-fallthrough -Wextra-semi -Wunreachable-code-aggressive -Wthread-safety -Wno-missing-field-initializers -Wno-unused-parameter -Wno-psabi -Wloop-analysis -Wno-unneeded-internal-declaration -Wenum-compare-conditional -Wno-ignored-pragma-optimize -Wno-deprecated-builtins -Wno-bitfield-constant-conversion -Wno-deprecated-this-capture -Wshadow -fno-delete-null-pointer-checks -fno-ident -fno-strict-aliasing --param=ssp-buffer-size=4 -fstack-protector -fno-unwind-tables -fno-asynchronous-unwind-tables -fPIC -pthread -fcolor-diagnostics -fmerge-all-constants -fcrash-diagnostics-dir=../../tools/clang/crashreports -mllvm -instcombine-lower-dbg-declare=0 -ffp-contract=off -m64 -msse3 -ffile-compilation-dir=. -no-canonical-prefixes -ftrivial-auto-var-init=pattern -O2 -fdata-sections -ffunction-sections -fno-unique-section-names -fno-omit-frame-pointer -gdwarf-4 -g1 -gdwarf-aranges -fvisibility=hidden -Wheader-hygiene -Wstring-conversion -Wtautological-overlap-compare -I/usr/include/glib-2.0 -I/usr/lib/x86_64-linux-gnu/glib-2.0/include -DPROTOBUF_ALLOW_DEPRECATED=1 -std=c++20 -Wno-trigraphs -gsimple-template-names -fno-exceptions -fno-rtti -fvisibility-inlines-hidden -Wdate-time -D_FORTIFY_SOURCE=2 -O2 -ffile-prefix-map=/<<PKGBUILDDIR>>=. -fstack-protector-strong -Wformat -Werror=format-security -Wno-unknown-warning-option -Wno-deprecated-declarations -c ../../chrome/common/channel_info.cc -o obj/chrome/common/channel_info/channel_info.o
In file included from ../../chrome/common/channel_info.cc:7:
../../components/version_info/version_info.h:30:23: error: constexpr function's return type 'std::string' (aka 'basic_string<char>') is not a literal type
constexpr std::string GetProductName() {

So yeah, we need to track the compatible version and try again, and so on as many times an issue is found.

@raphael10-collab
Copy link
Author

Hi @Ark74 !
What should I do then in order to make it work?

@Ark74
Copy link

Ark74 commented Oct 4, 2023

So far, no idea.
I also would like to build for a jammy base, but 'til a developer give us some light on the matter, I'm just like everyone else that is trying and failing. 😆

@Ark74
Copy link

Ark74 commented Oct 11, 2023

@raphael10-collab based on the open PR #328 there is already a PPA with the latest version.

I've been testing it and it works and builds correctly, thanks to @jhonny-oliveira

Regards.

@raphael10-collab
Copy link
Author

@Ark74 Thank you .

Actually I need the source code, and the correct working way to build it

@jhonny-oliveira
Copy link

@Ark74 Thank you .

Actually I need the source code, and the correct working way to build it

Check this out: #328 (comment)

You might no longer need the PRs (they were already merged) or you might needs new ones (with new fixes).

For a matter of alignment, I usually download the source from here: https://packages.debian.org/sid/chromium.

Hope this helps! Cheers!

@raphael10-collab
Copy link
Author

raphael10-collab commented Oct 12, 2023

Hi @jhonny-oliveira !

Thanks for the links.

I downloaded .deb file from here :https://packages.debian.org/sid/amd64/chromium/download

But during the source extraction I get this error:

raphy@raohy:~/ungoogled-chromium-debian$ dpkg-source -x /home/raphy/Downloads/chromium_118.0.5993.70-1_amd64.deb
dpkg-source: error: syntax error in /home/raphy/Downloads/chromium_118.0.5993.70-1_amd64.deb at line 2: line with unknown format (not field-colon-value)

@Ark74
Copy link

Ark74 commented Oct 12, 2023

@jhonny-oliveira I was able to build using your PPA source in amd64, for ppc64el I added this patch and some small changes in our helper, it is still building.

armhf and arm64 were failing, I'll test back again once ppc64el finish, if the whole set succeeds it's possible we could integrate it on the trisquel repo.

Cheers!

@Ark74
Copy link

Ark74 commented Oct 12, 2023

@raphael10-collab dpkg-source -x works with tarballs and .dsc files, .deb are binaries, they don't have source in it, unless it's stated the binary contains the source, but that's not the case for this package.

To get the source code go at the bottom of the page in:

Download the 3 files, 2 tar.xz and 1 dsc on the same directory, then do,

dpkg-source -x chromium_118.0.5993.70-1.dsc

Then you'll have the sid chromium source (not ungoogled-chromium), keep in mind that is very likely Debian Sid dependencies won't be compatible with Jammy's ones, as Sid is the "unstable" branch.

Regards.

@raphael10-collab
Copy link
Author

raphael10-collab commented Oct 12, 2023

@Ark74 Thank you Luis

Till the sid chromium source seems fine:

raphy@raohy:~/ungoogled-chromium-debian/convert$ wget deb.debian.org/debian/pool/main/c/chromium/chromium_118.0.5993.70-1.dsc
--2023-10-12 21:10:35--  http://deb.debian.org/debian/pool/main/c/chromium/chromium_118.0.5993.70-1.dsc
Resolving deb.debian.org (deb.debian.org)... 2a04:4e42:39::644, 151.101.242.132
Connecting to deb.debian.org (deb.debian.org)|2a04:4e42:39::644|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3659 (3,6K) [text/prs.lines.tag]
Saving to: ‘chromium_118.0.5993.70-1.dsc’

chromium_118.0.5993.70-1.dsc       100%[==============================================================>]   3,57K  --.-KB/s    in 0s      

2023-10-12 21:10:35 (386 MB/s) - ‘chromium_118.0.5993.70-1.dsc’ saved [3659/3659]

raphy@raohy:~/ungoogled-chromium-debian/convert$ wget deb.debian.org/debian/pool/main/c/chromium/chromium_118.0.5993.70.orig.tar.xz
--2023-10-12 21:10:47--  http://deb.debian.org/debian/pool/main/c/chromium/chromium_118.0.5993.70.orig.tar.xz
Resolving deb.debian.org (deb.debian.org)... 2a04:4e42:7d::644, 151.101.242.132
Connecting to deb.debian.org (deb.debian.org)|2a04:4e42:7d::644|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 771394968 (736M) [application/x-xz]
Saving to: ‘chromium_118.0.5993.70.orig.tar.xz’

chromium_118.0.5993.70.orig.tar.xz 100%[==============================================================>] 735,66M   110MB/s    in 7,0s    

2023-10-12 21:10:54 (106 MB/s) - ‘chromium_118.0.5993.70.orig.tar.xz’ saved [771394968/771394968]

raphy@raohy:~/ungoogled-chromium-debian/convert$ wget deb.debian.org/debian/pool/main/c/chromium/chromium_118.0.5993.70-1.debian.tar.xz
--2023-10-12 21:11:01--  http://deb.debian.org/debian/pool/main/c/chromium/chromium_118.0.5993.70-1.debian.tar.xz
Resolving deb.debian.org (deb.debian.org)... 2a04:4e42:39::644, 151.101.242.132
Connecting to deb.debian.org (deb.debian.org)|2a04:4e42:39::644|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 389388 (380K) [application/x-xz]
Saving to: ‘chromium_118.0.5993.70-1.debian.tar.xz’

chromium_118.0.5993.70-1.debian.ta 100%[==============================================================>] 380,26K  --.-KB/s    in 0,02s   

2023-10-12 21:11:01 (19,9 MB/s) - ‘chromium_118.0.5993.70-1.debian.tar.xz’ saved [389388/389388]

raphy@raohy:~/ungoogled-chromium-debian/convert$ ls -lah
total 737M
drwxrwxr-x 2 raphy raphy 4,0K ott 12 21:11 .
drwxrwxr-x 6 raphy raphy 4,0K ott 12 17:08 ..
-rw-rw-r-- 1 raphy raphy 381K ott 11 05:59 chromium_118.0.5993.70-1.debian.tar.xz
-rw-rw-r-- 1 raphy raphy 3,6K ott 11 05:59 chromium_118.0.5993.70-1.dsc
-rw-rw-r-- 1 raphy raphy 736M ott 11 05:59 chromium_118.0.5993.70.orig.tar.xz
-rwxrwxr-x 1 raphy raphy 2,3K ott 12 17:08 compare.sh
-rwxrwxr-x 1 raphy raphy 1,6K ott 12 17:08 editcontrol.pl
-rw-rw-r-- 1 raphy raphy  18K ott 12 21:07 Makefile
-rw-rw-r-- 1 raphy raphy 3,7K ott 12 17:08 README.md
-rw-rw-r-- 1 raphy raphy 1,5K ott 12 17:08 replace-name.pl
raphy@raohy:~/ungoogled-chromium-debian/convert$ dpkg-source -x chromium_118.0.5993.70-1.dsc
gpgv: Signature made mer 11 ott 2023, 05:48:02 CEST
gpgv:                using RSA key 6E58CA48BEAEF11C0345205EEC3C80BC57319B7E
gpgv: Can't check signature: No public key
dpkg-source: warning: cannot verify signature ./chromium_118.0.5993.70-1.dsc
dpkg-source: info: extracting chromium in chromium-118.0.5993.70
dpkg-source: info: unpacking chromium_118.0.5993.70.orig.tar.xz
dpkg-source: info: unpacking chromium_118.0.5993.70-1.debian.tar.xz
dpkg-source: info: using patch list from debian/patches/series
dpkg-source: info: applying debianization/manpage.patch
dpkg-source: info: applying debianization/sandbox.patch
dpkg-source: info: applying debianization/master-preferences.patch
dpkg-source: info: applying debianization/clang-version.patch
dpkg-source: info: applying fixes/ps-print.patch
dpkg-source: info: applying fixes/widevine-revision.patch
dpkg-source: info: applying fixes/widevine-locations.patch
dpkg-source: info: applying fixes/brandversion-construct.patch
dpkg-source: info: applying fixes/SkColor4f-init.patch
dpkg-source: info: applying fixes/gcc13-with-clang14.patch
dpkg-source: info: applying fixes/gcc13-headers.patch
dpkg-source: info: applying fixes/rust-clanglib.patch
dpkg-source: info: applying fixes/material-utils.patch
dpkg-source: info: applying fixes/perfetto.patch
dpkg-source: info: applying fixes/v8-compressed-ptrs.patch
dpkg-source: info: applying upstream/mojo.patch
dpkg-source: info: applying upstream/sensor-reading.patch
dpkg-source: info: applying upstream/lweight.patch
dpkg-source: info: applying upstream/freetype.patch
dpkg-source: info: applying upstream/sizet.patch
dpkg-source: info: applying disable/tests.patch
dpkg-source: info: applying disable/unrar.patch
dpkg-source: info: applying disable/signin.patch
dpkg-source: info: applying disable/android.patch
dpkg-source: info: applying disable/catapult.patch
dpkg-source: info: applying disable/font-tests.patch
dpkg-source: info: applying disable/swiftshader.patch
dpkg-source: info: applying disable/swiftshader-2.patch
dpkg-source: info: applying disable/google-api-warning.patch
dpkg-source: info: applying disable/third-party-cookies.patch
dpkg-source: info: applying disable/driver-chrome-path.patch
dpkg-source: info: applying disable/widevine-cdm-cu.patch
dpkg-source: info: applying disable/angle-perftests.patch
dpkg-source: info: applying disable/clang-version-check.patch
dpkg-source: info: applying disable/privacy-sandbox.patch
dpkg-source: info: applying system/icu-shim.patch
dpkg-source: info: applying system/jpeg.patch
dpkg-source: info: applying system/zlib.patch
dpkg-source: info: applying system/event.patch
dpkg-source: info: applying system/openjpeg.patch
dpkg-source: info: applying system/convertutf.patch
dpkg-source: info: applying system/clang-format.patch
dpkg-source: info: applying system/opus.patch
dpkg-source: info: applying system/eu-strip.patch
dpkg-source: info: applying system/rapidjson.patch
dpkg-source: info: applying system/rollup.patch
dpkg-source: info: applying bookworm/clang-attribs.patch
dpkg-source: info: applying bookworm/typename.patch
dpkg-source: warning: diff 'chromium-118.0.5993.70/debian/patches/bookworm/typename.patch' patches file chromium-118.0.5993.70/components/optimization_guide/core/tflite_model_executor.h more than once
dpkg-source: warning: diff 'chromium-118.0.5993.70/debian/patches/bookworm/typename.patch' patches file chromium-118.0.5993.70/ui/base/interaction/interactive_test_internal.h more than once
dpkg-source: info: applying bookworm/struct-ctor.patch
dpkg-source: info: applying bookworm/structured-binding-scope-bug.patch
dpkg-source: info: applying bookworm/stringpiece3.patch
dpkg-source: info: applying bookworm/initialize-const-ctor.patch
dpkg-source: info: applying i386/support-i386.patch
dpkg-source: info: applying i386/angle-lockfree.patch
dpkg-source: info: applying ppc64le/sandbox/0001-linux-seccomp-bpf-ppc64-glibc-workaround-in-SIGSYS-h.patch
dpkg-source: info: applying ppc64le/sandbox/0001-sandbox-Enable-seccomp_bpf-for-ppc64.patch
dpkg-source: info: applying ppc64le/sandbox/0001-services-service_manager-sandbox-linux-Fix-TCGETS-de.patch
dpkg-source: info: applying ppc64le/sandbox/0001-sandbox-linux-bpf_dsl-Update-syscall-ranges-for-ppc6.patch
dpkg-source: info: applying ppc64le/sandbox/0001-sandbox-linux-Implement-partial-support-for-ppc64-sy.patch
dpkg-source: info: applying ppc64le/sandbox/0001-sandbox-linux-Update-IsSyscallAllowed-in-broker_proc.patch
dpkg-source: info: applying ppc64le/sandbox/0001-sandbox-linux-Update-syscall-helpers-lists-for-ppc64.patch
dpkg-source: info: applying ppc64le/sandbox/0002-sandbox-linux-bpf_dsl-Modify-seccomp_macros-to-add-s.patch
dpkg-source: info: applying ppc64le/sandbox/0003-sandbox-linux-system_headers-Update-linux-seccomp-he.patch
dpkg-source: info: applying ppc64le/sandbox/0004-sandbox-linux-system_headers-Update-linux-signal-hea.patch
dpkg-source: info: applying ppc64le/sandbox/0005-sandbox-linux-seccomp-bpf-Add-ppc64-syscall-stub.patch
dpkg-source: info: applying ppc64le/sandbox/0005-sandbox-linux-update-unit-test-for-ppc64.patch
dpkg-source: info: applying ppc64le/sandbox/0006-sandbox-linux-disable-timedwait-time64-ppc64.patch
dpkg-source: info: applying ppc64le/sandbox/0007-sandbox-linux-add-ppc64-stat.patch
dpkg-source: info: applying ppc64le/sandbox/Sandbox-linux-services-credentials.cc-PPC.patch
dpkg-source: info: applying ppc64le/sandbox/0008-sandbox-fix-ppc64le-glibc234.patch
dpkg-source: info: applying ppc64le/third_party/0001-third_party-angle-Include-missing-header-cstddef-in-.patch
dpkg-source: info: applying ppc64le/third_party/0001-Add-PPC64-support-for-boringssl.patch
dpkg-source: info: applying ppc64le/third_party/0001-third_party-libvpx-Properly-generate-gni-on-ppc64.patch
dpkg-source: info: applying ppc64le/third_party/0001-third_party-lss-Don-t-look-for-mmap2-on-ppc64.patch
dpkg-source: info: applying ppc64le/third_party/0001-third_party-pffft-Include-altivec.h-on-ppc64-with-SI.patch
dpkg-source: info: applying ppc64le/third_party/0002-third_party-lss-kernel-structs.patch
dpkg-source: info: applying ppc64le/webrtc/Rtc_base-system-arch.h-PPC.patch
dpkg-source: info: applying ppc64le/crashpad/0002-Include-cstddef-to-fix-build.patch
dpkg-source: info: applying ppc64le/third_party/0004-third_party-crashpad-port-curl-transport-ppc64.patch
dpkg-source: info: applying ppc64le/workarounds/HACK-third_party-libvpx-use-generic-gnu.patch
dpkg-source: info: applying ppc64le/workarounds/HACK-debian-clang-disable-skia-musttail.patch
dpkg-source: info: applying ppc64le/libaom/0001-Add-ppc64-target-to-libaom.patch
dpkg-source: info: applying ppc64le/libaom/0001-Add-pregenerated-config-for-libaom-on-ppc64.patch
dpkg-source: info: applying ppc64le/third_party/0002-third_party-libvpx-Remove-bad-ppc64-config.patch
dpkg-source: info: applying ppc64le/third_party/0002-third-party-boringssl-add-generated-files.patch
dpkg-source: info: applying ppc64le/third_party/0003-third_party-libvpx-Add-ppc64-generated-config.patch
dpkg-source: info: applying ppc64le/third_party/0003-third_party-ffmpeg-Add-ppc64-generated-config.patch
dpkg-source: info: applying ppc64le/third_party/0004-third_party-libvpx-work-around-ambiguous-vsx.patch
dpkg-source: info: applying ppc64le/third_party/skia-vsx-instructions.patch
dpkg-source: info: applying ppc64le/ffmpeg/0001-Add-support-for-ppc64.patch
dpkg-source: info: applying ppc64le/breakpad/0001-Implement-support-for-ppc64-on-Linux.patch
dpkg-source: info: applying ppc64le/crashpad/0001-Implement-support-for-PPC64-on-Linux.patch
dpkg-source: info: applying ppc64le/database/0001-Properly-detect-little-endian-PPC64-systems.patch
dpkg-source: info: applying ppc64le/third_party/0001-Force-baseline-POWER8-AltiVec-VSX-CPU-features-when-.patch
dpkg-source: info: applying ppc64le/fixes/fix-breakpad-compile.patch
dpkg-source: info: applying ppc64le/fixes/fix-partition-alloc-compile.patch
dpkg-source: info: applying ppc64le/v8/0002-Add-ppc64-trap-instructions.patch
dpkg-source: info: applying ppc64le/third_party/0001-Fix-highway-ppc-hwcap.patch
dpkg-source: info: applying ppc64le/third_party/0001-Add-PPC64-support-for-libdav1d.patch
dpkg-source: info: applying ppc64le/third_party/0001-Fix-libdav1d-compilation-on-clang-ppc.patch
dpkg-source: info: applying ppc64le/sandbox/fix-ppc64-linux-syscalls-headers.patch
dpkg-source: info: applying ppc64le/third_party/0003-thirdparty-fix-dav1d-gn.patch
dpkg-source: info: applying ppc64le/third_party/use-sysconf-page-size-on-ppc64.patch
dpkg-source: info: applying ppc64le/third_party/dawn-fix-typos.patch
dpkg-source: info: applying ppc64le/third_party/dawn-fix-ppc64le-detection.patch
dpkg-source: info: applying ppc64le/fixes/fix-unknown-warning-option-messages.diff
dpkg-source: info: applying ppc64le/third_party/0002-Highway-disable-128-bit-vsx.patch
dpkg-source: info: applying ungoogled/core/ungoogled-chromium/disable-web-environment-integrity.patch


raphy@raohy:~/ungoogled-chromium-debian/convert$ ls -lah
total 737M
drwxrwxr-x  3 raphy raphy 4,0K ott 12 21:12 .
drwxrwxr-x  6 raphy raphy 4,0K ott 12 17:08 ..
drwxrwxr-x 55 raphy raphy 4,0K ott 12 21:12 chromium-118.0.5993.70
-rw-rw-r--  1 raphy raphy 381K ott 11 05:59 chromium_118.0.5993.70-1.debian.tar.xz
-rw-rw-r--  1 raphy raphy 3,6K ott 11 05:59 chromium_118.0.5993.70-1.dsc
-rw-rw-r--  1 raphy raphy 736M ott 11 05:59 chromium_118.0.5993.70.orig.tar.xz
-rwxrwxr-x  1 raphy raphy 2,3K ott 12 17:08 compare.sh
-rwxrwxr-x  1 raphy raphy 1,6K ott 12 17:08 editcontrol.pl
-rw-rw-r--  1 raphy raphy  18K ott 12 21:07 Makefile
-rw-rw-r--  1 raphy raphy 3,7K ott 12 17:08 README.md
-rw-rw-r--  1 raphy raphy 1,5K ott 12 17:08 replace-name.pl

But the make phase asks for rule to series.add : #328 (comment) :

raphy@raohy:~/ungoogled-chromium-debian/convert$ make
make: *** No rule to make target '/home/raphy/src/ungoogled-chromium/patches/series', needed by 'series.add'.  Stop.

@iskunk
Copy link
Contributor

iskunk commented Oct 18, 2023

Hi @Ark74, could you elaborate on why ppc64el needs ThinLTO disabled? I can integrate such a change into the XtraDeb source if needed.

(Please note that @jhonny-oliveira and I are still working on the automation to get ungoogled-chromium converted and updated automatically. Updates, let alone security updates, will be delayed until we have that in place. We will also be releasing the script we use to perform the Ubuntu conversion from the Debian source.)

@raphael10-collab, note that the files under ungoogled-chromium-debian/convert/ have their own documentation and procedure for use. The conversion framework cannot be used out of the box; there is some setup you will need to perform first.

@Ark74
Copy link

Ark74 commented Oct 18, 2023

@iskunk I did several testing, and I always hit the same error, something along the lines:

....
test -f   ungoogled-domain-substitution.orig.tar \
|| debian/ungoogled-domain-substitution.sh
Creating backup archive ...
Applying ungoogled-chromium domain substitution to 11885 files ...
gn gen out/Release --args="clang_use_chrome_plugins=false rust_sysroot_absolute=\"/usr\" rustc_version=\"rustc 1.66.1 (90743e729 2023-01-10) (built from a source tarball)\" enable_rust=false clang_base_path=\"/usr/lib/llvm-\" clang_version=\"""\" host_toolchain=\"//build/toolchain/linux/unbundle:default\" custom_toolchain=\"//build/toolchain/linux/unbundle:default\" is_debug=false use_goma=false use_sysroot=false use_libjpeg_turbo=true use_custom_libcxx=false use_unofficial_version_number=false enable_vr=false enable_nacl=false enable_swiftshader=false dawn_use_swiftshader=false build_dawn_tests=false enable_reading_list=false enable_iterator_debugging=false enable_hangout_services_extension=false angle_has_histograms=false build_angle_perftests=false treat_warnings_as_errors=false use_qt=false is_cfi=false use_thin_lto=true chrome_pgo_phase=0  use_gio=true is_official_build=true symbol_level=0 use_pulseaudio=true link_pulseaudio=true rtc_use_pipewire=true icu_use_data_file=true enable_widevine=true v8_enable_backtrace=true use_system_zlib=true use_system_lcms2=true use_system_libjpeg=true use_system_libpng=true use_system_freetype=true use_system_libopenjpeg2=true  proprietary_codecs=true ffmpeg_branding=\"Chrome\" disable_fieldtrial_testing_config=true  build_with_tflite_lib=false chrome_pgo_phase=0 clang_use_chrome_plugins=false disable_fieldtrial_testing_config=true enable_hangout_services_extension=false enable_mdns=false enable_nacl=false enable_reading_list=false enable_remoting=false enable_reporting=false enable_service_discovery=false enable_widevine=true exclude_unwind_tables=true google_api_key=\"\" google_default_client_id=\"\" google_default_client_secret=\"\" safe_browsing_mode=0 treat_warnings_as_errors=false use_official_google_api_keys=false use_unofficial_version_number=false "
ERROR at //build/config/compiler/BUILD.gn:696:5: Assertion failed.
    assert(use_lld, "LTO is only supported with lld")
    ^-----
LTO is only supported with lld
See //build/config/BUILDCONFIG.gn:333:3: which caused the file to be included.
  "//build/config/compiler:afdo",
  ^-----------------------------
make[1]: *** [debian/rules:164: override_dh_auto_build-arch] Error 1
make[1]: Leaving directory '/<<PKGBUILDDIR>>'
make: *** [debian/rules:127: binary] Error 2
dpkg-buildpackage: error: debian/rules binary subprocess returned exit status 2
--------------------------------------------------------------------------------
....

After not finding much information that Debian developers have decided to disable lto on chromium Debian#1015367 and comparing build environments for chromium and ungoogled-chromium I found that thin-lto is disabled in chromium, just by disabling thin-lto ppc64el started to build fine without any other change.

arm64(/hf) had been failing, since I've updated to not use thin-lto they seem will finish building, so I'm looking to disable to all but amd64.

ATTOW,

  • arm64 - [51275/57733] CXX obj/chrome/browser/extensions/extensions/scoped_active_install.o
  • armhf - [50587/57749] CXX obj/third_party/blink/renderer/core/core/focus_changed_observer.o

Which are the most progress or any of those archs.

You may be able to confirm my findings on the PPA build nodes.
About the PPA, and current stage, it's ok, I'm still testing this out, I'll wait it gets kind of stable before any move.

Thank you guys for all the work you are putting here.
Regards.

@raphael10-collab
Copy link
Author

I've restarted from scratch:

raphy@raohy:~/ungoogled-chromium$ wget deb.debian.org/debian/pool/main/c/chromium/chromium_118.0.5993.70.orig.tar.xz
--2023-10-18 09:31:25--  http://deb.debian.org/debian/pool/main/c/chromium/chromium_118.0.5993.70.orig.tar.xz
Resolving deb.debian.org (deb.debian.org)... 2a04:4e42:39::644, 151.101.242.132
Connecting to deb.debian.org (deb.debian.org)|2a04:4e42:39::644|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 771394968 (736M) [application/x-xz]
Saving to: ‘chromium_118.0.5993.70.orig.tar.xz’

chromium_118.0.5993.70.orig.tar.xz                                100%[===========================================================================================================================================================>] 735,66M   110MB/s    in 6,7s    

2023-10-18 09:31:31 (110 MB/s) - ‘chromium_118.0.5993.70.orig.tar.xz’ saved [771394968/771394968]

raphy@raohy:~/ungoogled-chromium$ wget deb.debian.org/debian/pool/main/c/chromium/chromium_118.0.5993.70-1.debian.tar.xz
--2023-10-18 09:31:36--  http://deb.debian.org/debian/pool/main/c/chromium/chromium_118.0.5993.70-1.debian.tar.xz
Resolving deb.debian.org (deb.debian.org)... 2a04:4e42:7d::644, 151.101.242.132
Connecting to deb.debian.org (deb.debian.org)|2a04:4e42:7d::644|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 389388 (380K) [application/x-xz]
Saving to: ‘chromium_118.0.5993.70-1.debian.tar.xz’

chromium_118.0.5993.70-1.debian.tar.xz                            100%[===========================================================================================================================================================>] 380,26K  --.-KB/s    in 0,07s   

2023-10-18 09:31:36 (5,32 MB/s) - ‘chromium_118.0.5993.70-1.debian.tar.xz’ saved [389388/389388]

raphy@raohy:~/ungoogled-chromium$ ls -lah
total 737M
drwxrwxr-x  2 raphy raphy 4,0K ott 18 09:31 .
drwxr-x--- 65 raphy raphy 4,0K ott 18 09:30 ..
-rw-rw-r--  1 raphy raphy 381K ott 11 05:59 chromium_118.0.5993.70-1.debian.tar.xz
-rw-rw-r--  1 raphy raphy 3,6K ott 11 05:59 chromium_118.0.5993.70-1.dsc
-rw-rw-r--  1 raphy raphy 736M ott 11 05:59 chromium_118.0.5993.70.orig.tar.xz


raphy@raohy:~/ungoogled-chromium$ dpkg-source -x chromium_118.0.5993.70-1.dsc 
gpgv: Signature made mer 11 ott 2023, 05:48:02 CEST
gpgv:                using RSA key 6E58CA48BEAEF11C0345205EEC3C80BC57319B7E
gpgv: Can't check signature: No public key
dpkg-source: warning: cannot verify signature ./chromium_118.0.5993.70-1.dsc
dpkg-source: info: extracting chromium in chromium-118.0.5993.70
dpkg-source: info: unpacking chromium_118.0.5993.70.orig.tar.xz
dpkg-source: info: unpacking chromium_118.0.5993.70-1.debian.tar.xz
dpkg-source: info: using patch list from debian/patches/series
dpkg-source: info: applying debianization/manpage.patch
dpkg-source: info: applying debianization/sandbox.patch
dpkg-source: info: applying debianization/master-preferences.patch
dpkg-source: info: applying debianization/clang-version.patch
dpkg-source: info: applying fixes/ps-print.patch
dpkg-source: info: applying fixes/widevine-revision.patch
dpkg-source: info: applying fixes/widevine-locations.patch
dpkg-source: info: applying fixes/brandversion-construct.patch
dpkg-source: info: applying fixes/SkColor4f-init.patch
dpkg-source: info: applying fixes/gcc13-with-clang14.patch
dpkg-source: info: applying fixes/gcc13-headers.patch
dpkg-source: info: applying fixes/rust-clanglib.patch
dpkg-source: info: applying fixes/material-utils.patch
dpkg-source: info: applying fixes/perfetto.patch
dpkg-source: info: applying fixes/v8-compressed-ptrs.patch
dpkg-source: info: applying upstream/mojo.patch
dpkg-source: info: applying upstream/sensor-reading.patch
dpkg-source: info: applying upstream/lweight.patch
dpkg-source: info: applying upstream/freetype.patch
dpkg-source: info: applying upstream/sizet.patch
dpkg-source: info: applying disable/tests.patch
dpkg-source: info: applying disable/unrar.patch
dpkg-source: info: applying disable/signin.patch
dpkg-source: info: applying disable/android.patch
dpkg-source: info: applying disable/catapult.patch
dpkg-source: info: applying disable/font-tests.patch
dpkg-source: info: applying disable/swiftshader.patch
dpkg-source: info: applying disable/swiftshader-2.patch
dpkg-source: info: applying disable/google-api-warning.patch
dpkg-source: info: applying disable/third-party-cookies.patch
dpkg-source: info: applying disable/driver-chrome-path.patch
dpkg-source: info: applying disable/widevine-cdm-cu.patch
dpkg-source: info: applying disable/angle-perftests.patch
dpkg-source: info: applying disable/clang-version-check.patch
dpkg-source: info: applying disable/privacy-sandbox.patch
dpkg-source: info: applying system/icu-shim.patch
dpkg-source: info: applying system/jpeg.patch
dpkg-source: info: applying system/zlib.patch
dpkg-source: info: applying system/event.patch
dpkg-source: info: applying system/openjpeg.patch
dpkg-source: info: applying system/convertutf.patch
dpkg-source: info: applying system/clang-format.patch
dpkg-source: info: applying system/opus.patch
dpkg-source: info: applying system/eu-strip.patch
dpkg-source: info: applying system/rapidjson.patch
dpkg-source: info: applying system/rollup.patch
dpkg-source: info: applying bookworm/clang-attribs.patch
dpkg-source: info: applying bookworm/typename.patch
dpkg-source: warning: diff 'chromium-118.0.5993.70/debian/patches/bookworm/typename.patch' patches file chromium-118.0.5993.70/components/optimization_guide/core/tflite_model_executor.h more than once
dpkg-source: warning: diff 'chromium-118.0.5993.70/debian/patches/bookworm/typename.patch' patches file chromium-118.0.5993.70/ui/base/interaction/interactive_test_internal.h more than once
dpkg-source: info: applying bookworm/struct-ctor.patch
dpkg-source: info: applying bookworm/structured-binding-scope-bug.patch
dpkg-source: info: applying bookworm/stringpiece3.patch
dpkg-source: info: applying bookworm/initialize-const-ctor.patch
dpkg-source: info: applying i386/support-i386.patch
dpkg-source: info: applying i386/angle-lockfree.patch
dpkg-source: info: applying ppc64le/sandbox/0001-linux-seccomp-bpf-ppc64-glibc-workaround-in-SIGSYS-h.patch
dpkg-source: info: applying ppc64le/sandbox/0001-sandbox-Enable-seccomp_bpf-for-ppc64.patch
dpkg-source: info: applying ppc64le/sandbox/0001-services-service_manager-sandbox-linux-Fix-TCGETS-de.patch
dpkg-source: info: applying ppc64le/sandbox/0001-sandbox-linux-bpf_dsl-Update-syscall-ranges-for-ppc6.patch
dpkg-source: info: applying ppc64le/sandbox/0001-sandbox-linux-Implement-partial-support-for-ppc64-sy.patch
dpkg-source: info: applying ppc64le/sandbox/0001-sandbox-linux-Update-IsSyscallAllowed-in-broker_proc.patch
dpkg-source: info: applying ppc64le/sandbox/0001-sandbox-linux-Update-syscall-helpers-lists-for-ppc64.patch
dpkg-source: info: applying ppc64le/sandbox/0002-sandbox-linux-bpf_dsl-Modify-seccomp_macros-to-add-s.patch
dpkg-source: info: applying ppc64le/sandbox/0003-sandbox-linux-system_headers-Update-linux-seccomp-he.patch
dpkg-source: info: applying ppc64le/sandbox/0004-sandbox-linux-system_headers-Update-linux-signal-hea.patch
dpkg-source: info: applying ppc64le/sandbox/0005-sandbox-linux-seccomp-bpf-Add-ppc64-syscall-stub.patch
dpkg-source: info: applying ppc64le/sandbox/0005-sandbox-linux-update-unit-test-for-ppc64.patch
dpkg-source: info: applying ppc64le/sandbox/0006-sandbox-linux-disable-timedwait-time64-ppc64.patch
dpkg-source: info: applying ppc64le/sandbox/0007-sandbox-linux-add-ppc64-stat.patch
dpkg-source: info: applying ppc64le/sandbox/Sandbox-linux-services-credentials.cc-PPC.patch
dpkg-source: info: applying ppc64le/sandbox/0008-sandbox-fix-ppc64le-glibc234.patch
dpkg-source: info: applying ppc64le/third_party/0001-third_party-angle-Include-missing-header-cstddef-in-.patch
dpkg-source: info: applying ppc64le/third_party/0001-Add-PPC64-support-for-boringssl.patch
dpkg-source: info: applying ppc64le/third_party/0001-third_party-libvpx-Properly-generate-gni-on-ppc64.patch
dpkg-source: info: applying ppc64le/third_party/0001-third_party-lss-Don-t-look-for-mmap2-on-ppc64.patch
dpkg-source: info: applying ppc64le/third_party/0001-third_party-pffft-Include-altivec.h-on-ppc64-with-SI.patch
dpkg-source: info: applying ppc64le/third_party/0002-third_party-lss-kernel-structs.patch
dpkg-source: info: applying ppc64le/webrtc/Rtc_base-system-arch.h-PPC.patch
dpkg-source: info: applying ppc64le/crashpad/0002-Include-cstddef-to-fix-build.patch
dpkg-source: info: applying ppc64le/third_party/0004-third_party-crashpad-port-curl-transport-ppc64.patch
dpkg-source: info: applying ppc64le/workarounds/HACK-third_party-libvpx-use-generic-gnu.patch
dpkg-source: info: applying ppc64le/workarounds/HACK-debian-clang-disable-skia-musttail.patch
dpkg-source: info: applying ppc64le/libaom/0001-Add-ppc64-target-to-libaom.patch
dpkg-source: info: applying ppc64le/libaom/0001-Add-pregenerated-config-for-libaom-on-ppc64.patch
dpkg-source: info: applying ppc64le/third_party/0002-third_party-libvpx-Remove-bad-ppc64-config.patch
dpkg-source: info: applying ppc64le/third_party/0002-third-party-boringssl-add-generated-files.patch
dpkg-source: info: applying ppc64le/third_party/0003-third_party-libvpx-Add-ppc64-generated-config.patch
dpkg-source: info: applying ppc64le/third_party/0003-third_party-ffmpeg-Add-ppc64-generated-config.patch
dpkg-source: info: applying ppc64le/third_party/0004-third_party-libvpx-work-around-ambiguous-vsx.patch
dpkg-source: info: applying ppc64le/third_party/skia-vsx-instructions.patch
dpkg-source: info: applying ppc64le/ffmpeg/0001-Add-support-for-ppc64.patch
dpkg-source: info: applying ppc64le/breakpad/0001-Implement-support-for-ppc64-on-Linux.patch
dpkg-source: info: applying ppc64le/crashpad/0001-Implement-support-for-PPC64-on-Linux.patch
dpkg-source: info: applying ppc64le/database/0001-Properly-detect-little-endian-PPC64-systems.patch
dpkg-source: info: applying ppc64le/third_party/0001-Force-baseline-POWER8-AltiVec-VSX-CPU-features-when-.patch
dpkg-source: info: applying ppc64le/fixes/fix-breakpad-compile.patch
dpkg-source: info: applying ppc64le/fixes/fix-partition-alloc-compile.patch
dpkg-source: info: applying ppc64le/v8/0002-Add-ppc64-trap-instructions.patch
dpkg-source: info: applying ppc64le/third_party/0001-Fix-highway-ppc-hwcap.patch
dpkg-source: info: applying ppc64le/third_party/0001-Add-PPC64-support-for-libdav1d.patch
dpkg-source: info: applying ppc64le/third_party/0001-Fix-libdav1d-compilation-on-clang-ppc.patch
dpkg-source: info: applying ppc64le/sandbox/fix-ppc64-linux-syscalls-headers.patch
dpkg-source: info: applying ppc64le/third_party/0003-thirdparty-fix-dav1d-gn.patch
dpkg-source: info: applying ppc64le/third_party/use-sysconf-page-size-on-ppc64.patch
dpkg-source: info: applying ppc64le/third_party/dawn-fix-typos.patch
dpkg-source: info: applying ppc64le/third_party/dawn-fix-ppc64le-detection.patch
dpkg-source: info: applying ppc64le/fixes/fix-unknown-warning-option-messages.diff
dpkg-source: info: applying ppc64le/third_party/0002-Highway-disable-128-bit-vsx.patch
dpkg-source: info: applying ungoogled/core/ungoogled-chromium/disable-web-environment-integrity.patch

It seems different now: I do not see the convert folder anymore

@iskunk
Copy link
Contributor

iskunk commented Oct 18, 2023

@Ark74: Thank you for bringing this to my attention. I was able to reproduce that error using QEMU.

It took some digging, but I was able to track down the issue to this bit in build/config/BUILDCONFIG.gn, around line 141:

  # Set to true when compiling with the Clang compiler.
  is_clang = current_os != "linux" ||
             (current_cpu != "s390x" && current_cpu != "s390" &&
              current_cpu != "ppc64" && current_cpu != "ppc" &&
              current_cpu != "mips" && current_cpu != "mips64" &&
              current_cpu != "riscv64")

The GN logic appears to assume that if you're building for any of those architectures, then you're not using Clang---which is clearly incorrect in the case of ppc64el. Please try removing/disabling the current_cpu != "ppc64" bit above, and give the build another go with use_thin_lto=true.

What errors were you seeing on arm64 / armhf?

Also, some clarifications on LTO:

  • The Debian issue you linked concerns "normal" (or "full") LTO, which is currently disabled by default in Debian, and enabled by default in Ubuntu. Before that issue was filed, the Chromium build configuration said nothing about LTO. When I tried to build that source on Ubuntu, however, the build failed due to the final link taking too long, causing the builder to time out. The issue was about having the Chromium build config disable LTO explicitly, for the benefit of building on Ubuntu. (There was also the matter that the LTO flags used by Debian/Ubuntu broadly are intended for GCC, not Clang.)
  • ThinLTO is different from "normal" LTO in that it is lighter weight. You get some of the benefit of those optimizations at a more modest cost of increased CPU/memory usage at link time. It is, in effect, "LTO lite."
  • So far, I've observed that Chromium builds with ThinLTO enabled fail on armhf due to insufficient RAM. It is possible that ThinLTO will need to be disabled on that arch in order to allow the build to complete. I believe armhf is uniquely limited by a lack of hardware options with lots of RAM.
  • Disabling ThinLTO should be a last resort. It's a one-time cost that pays off in better run-time performance for every user, so it's highly desirable. I have an open issue with Debian for them to switch it on in their builds. For now, I enable it in the XtraDeb source, since the Ubuntu builders can easily handle it (modulo armhf).

@raphael10-collab: If you're not already comfortable with handling and building Debian source packages, then there is not much I can do to get you through it. Using the conversion framework I've put together requires experience in that area.

Could you talk about why you wish to build this yourself, instead of using pre-built packages? If you have concerns about the integrity of the build, then there may be an alternative way of addressing that.

@Ark74
Copy link

Ark74 commented Oct 19, 2023

@iskunk you are right, removing current_cpu != "ppc64" does pass the initial error on ppc64el, there are some hours ahead to confirm build, hopefully it will pass.

arm64/hf didn't show some clear error output, just failing randomly, I'm using a couple of RPi4 8GB RAM, but even if you get access to some AWS ARM metal servers, they also failed on my first tests.

I'll run some new tests, and comment on the output, regards.

@Ark74
Copy link

Ark74 commented Oct 19, 2023

@iskunk I can confirm now that ppc64el passes with that change.

  • ppc64el - patch - log
  • aws arm builds ubuntu enviroment:
  • rpi4 arm builds trisquel environment (jammy based):

I'm not sure if the v4l2 issues comes from trisquel or plain ubuntu, last time I succeed using this patch and disabling thin-lto.

@iskunk
Copy link
Contributor

iskunk commented Oct 19, 2023

Yes, the v4l2 issue is due to Ubuntu jammy's kernel not being new enough. (Lunar and later are good.) That patch you linked is the right idea, but you should use these settings:

ifeq (arm64,$(DEB_HOST_ARCH))
defines+=host_cpu=\"arm64\" use_vaapi=true
endif
ifeq (armhf,$(DEB_HOST_ARCH))
defines+=host_cpu=\"arm\" use_vaapi=false arm_use_neon=true
endif

This is what Debian uses in their bullseye build. See this bug for background.

That html_to_wrapper.py error in the armhf build, however, that I have not seen before. I've successfully performed a full armhf build on jammy via QEMU, so I am not able to reproduce it. Could you try to see what's going on there? All I can say to help is that the third_party/node/linux/node-linux-x64/bin/node binary comes from this line in the rules file.

One thing I forgot to mention earlier: If you can, try to wrap your builds with env time --verbose. That will give you some useful statistics at the end, most notably the "Maximum resident set size." Knowing how much RAM a build requires, especially with ThinLTO vs. without, is critical on armhf. (My QEMU build yielded a max RSS of 2.4 GB without ThinLTO, so I think your RPis should be able to handle it with.)

@Ark74
Copy link

Ark74 commented Oct 20, 2023

Yes, the v4l2 issue is due to Ubuntu jammy's kernel not being new enough

I guess linux-hwe-6.2 could work, but that would require a non-standard dependency, so yeah, I guess it's better to disable it for now.

I've updated the logs for arm on a trisquel env, seems like the aws tripped somehow.
I'll rebuild the sources with the suggested proposal and give it another round.

Update: Here it is the AWS build log, success for arm64, I guess this proves the point so this could be proven to actually work for jammy, I'll continue testing on the rpi4, once I have room.

	Command being timed: "sbuild --no-run-lintian -v -j 64 -A --dist aramo --arch arm64 --build-dep-resolver=aptitude ungoogled-chromium_117.0.5938.149-1~xtradeb1+11.0trisquel1.dsc --extra-repository=deb http://ports.ubuntu.com/ubuntu-ports/ jammy main"
	User time (seconds): 203364.61
	System time (seconds): 11795.42
	Percent of CPU this job got: 5648%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 1:03:29
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 9264868
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 2480
	Minor (reclaiming a frame) page faults: 2492955282
	Voluntary context switches: 5779205
	Involuntary context switches: 4767860
	Swaps: 0
	File system inputs: 272
	File system outputs: 418176
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0

@Ark74
Copy link

Ark74 commented Oct 25, 2023

@iskunk news on the armhf build on the rpi4.
Well, it failed 😞, but just for the hardware limitation.

I guess the important part is:

[57730/57749] CXX obj/chrome/browser/browser/download_target_determiner.o
[57731/57749] CXX obj/chrome/browser/browser/chrome_download_manager_delegate.o
[57732/57749] AR obj/chrome/browser/libbrowser.a
[57733/57749] LINK ./v8_context_snapshot_generator
FAILED: v8_context_snapshot_generator 
"python3" "../../build/toolchain/gcc_link_wrapper.py" --output="./v8_context_snapshot_generator" -- clang++-15 -fuse-ld=lld-15 -Wl,--fatal-warnings -Wl,--build-id=sha1 -fPIC -Wl,-z,noexecstack -Wl,-z,relro -Wl,-z,now -Wl,--icf=all -Wl,--color-diagnostics -Wl,-mllvm,-instcombine-lower-dbg-declare=0 -flto=thin -Wl,--thinlto-jobs=all -Wl,--thinlto-cache-dir=thinlto-cache -Wl,--thinlto-cache-policy=cache_size=10\%:cache_size_bytes=40g:cache_size_files=100000 -Wl,-mllvm,-import-instr-limit=30 -fwhole-program-vtables -march=armv7-a -Wl,--undefined-version -Wl,--no-call-graph-profile-sort --target=arm-linux-gnueabihf -no-canonical-prefixes -Wl,-O2 -Wl,--gc-sections -rdynamic -Wl,-z,defs -Wl,--as-needed -Wl,--lto-O0 -pie -Wl,--disable-new-dtags -Wl,--icf=none -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,now -Wl,--stats -o "./v8_context_snapshot_generator" -Wl,--start-group @"./v8_context_snapshot_generator.rsp"  -Wl,--end-group  -latomic -ldl -lpthread -lrt -lgmodule-2.0 -lgobject-2.0 -lgthread-2.0 -lglib-2.0 -ljsoncpp -lnss3 -lnssutil3 -lsmime3 -lplds4 -lplc4 -lnspr4 -latk-1.0 -latk-bridge-2.0 -lcups -lz -ldouble-conversion -levent -lresolv -lgio-2.0 -lexpat -lpng16 -lwebpdemux -lwebpmux -lwebp -lfreetype -ljpeg -lfontconfig -ldrm -lopus -lopenh264 -lm -lX11 -lXcomposite -lXdamage -lXext -lXfixes -lXrender -lXrandr -lXtst -lgbm -lEGL -lxcb -lxkbcommon -lXi -lpci -l:libffi_pic.a -lXNVCtrl -ldbus-1 -lpangocairo-1.0 -lpango-1.0 -lharfbuzz -lcairo -lasound -lpulse -lFLAC -lsnappy -latspi -lxml2 -lminizip -lxslt -lwoff2dec -lbrotlidec 
ld.lld-15: error: failed to open ./v8_context_snapshot_generator: Cannot allocate memory
clang++-15: error: linker command failed with exit code 1 (use -v to see invocation)
ninja: build stopped: subcommand failed.
make[1]: *** [debian/rules:161: override_dh_auto_build-arch] Error 1
make[1]: Leaving directory '/<<PKGBUILDDIR>>'
make: *** [debian/rules:123: binary] Error 2
dpkg-buildpackage: error: debian/rules binary subprocess returned exit status 2

About env time --verbose, here the output:

Command exited with non-zero status 2
        Command being timed: "sbuild --no-run-lintian -v -j 4 -A --dist aramo --arch armhf --build-dep-resolver=aptitude ungoogled-chromium_117.0.5938.149-1~xtradeb1+11.0trisquel1.dsc --extra-repository=deb http://ports.ubuntu.com/ubuntu-ports/ jammy main"
        User time (seconds): 860058.18
        System time (seconds): 31953.59
        Percent of CPU this job got: 395%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 62:43:40
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 3598344
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 16621
        Minor (reclaiming a frame) page faults: 1662971621
        Voluntary context switches: 2424058
        Involuntary context switches: 56859450
        Swaps: 0
        File system inputs: 5405688
        File system outputs: 31998432
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 2

This rpi has a 8GB swap file, I guess I could easily double that, but I'm not sure if that would work / be enough, any suggestion?
I guess that this still proves it works, just need more resources than tested on this SBC.

Regards.

@iskunk
Copy link
Contributor

iskunk commented Oct 26, 2023

@Ark74: I didn't expect the armhf build max RSS to top out at under 4 GB RAM. Can armhf programs not address more than that, like (32-bit) x86 with PAE?

I'm currently running an armhf build in QEMU with use_thin_lto=true. That will not be limited by available RAM, but if the limitation is architectural, then I should run into the same failure as you did.

In which case, the only way to build a ThinLTO-optimized Chromium for armhf would be to cross-compile. I don't believe the source package is set up to allow cross-compilation...

@raphael10-collab: You might want to try ungoogled-chromium in the XtraDeb applications PPA. The source has been converted, and there are binary builds there too if you want. The downside is that the version is a bit behind---@jhonny-oliveira and I are still working on the update infrastructure.

@iskunk
Copy link
Contributor

iskunk commented Nov 3, 2023

An update: I've confirmed that the final link, on armhf with ThinLTO enabled, is impossible.

The build failed similarly at the v8_context_snapshot_generator link. I was able to get past that by running the link command directly, but then the final link of chrome fails even then:

build@xtradeb-jammy-armhf-ubuntu-con:~/chromium/chromium-118.0.5993.70/out/Release$ time -p "python3" "../../build/toolchain/gcc_link_wrapper.py" --output="./chrome" -- clang++-15 -Wl,--version-script=../../build/linux/chrome.map -fuse-ld=lld-15 -Wl,--fatal-warnings -Wl,--build-id=sha1 -fPIC -Wl,-z,noexecstack -Wl,-z,relro -Wl,-z,now -Wl,--icf=all -Wl,--color-diagnostics -Wl,-mllvm,-instcombine-lower-dbg-declare=0 -flto=thin -Wl,--thinlto-jobs=all -Wl,--thinlto-cache-dir=thinlto-cache -Wl,--thinlto-cache-policy=cache_size=10\%:cache_size_bytes=40g:cache_size_files=100000 -Wl,-mllvm,-import-instr-limit=30 -fwhole-program-vtables -march=armv7-a -Wl,--undefined-version -Wl,--no-call-graph-profile-sort --target=arm-linux-gnueabihf -no-canonical-prefixes -Wl,-O2 -Wl,--gc-sections -rdynamic -Wl,-z,defs -Wl,--as-needed -pie -Wl,--disable-new-dtags -Wl,--lto-O2 -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,now -Wl,--stats -o "./chrome" -Wl,--start-group @"./chrome.rsp"  -Wl,--end-group  -latomic -ldl -lpthread -lrt -lgmodule-2.0 -lgobject-2.0 -lgthread-2.0 -lglib-2.0 -ljsoncpp -lnss3 -lnssutil3 -lsmime3 -lplds4 -lplc4 -lnspr4 -latk-1.0 -latk-bridge-2.0 -lcups -lgio-2.0 -lfontconfig -ldbus-1 -lz -ldouble-conversion -levent -latspi -lresolv -lexpat -lpng16 -lwebpdemux -lwebpmux -lwebp -lfreetype -ljpeg -lopus -lopenh264 -lm -lX11 -lXcomposite -lXdamage -lXext -lXfixes -lXrender -lXrandr -lXtst -lgbm -lEGL -ldrm -lxcb -lxkbcommon -lXi -lpci -l:libffi_pic.a -lXNVCtrl -lpangocairo-1.0 -lpango-1.0 -lharfbuzz -lcairo -lasound -lpulse -lFLAC -lsnappy -lxml2 -lminizip -lxslt -lwoff2dec -lbrotlidec -lxshmfence -lopenjp2 -llcms2 -lpng
LLVM ERROR: out of memory
Allocation failed
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
Stack dump:
0.	Program arguments: /usr/bin/ld.lld-15 @/tmp/response-e65857.txt
clang++-15: error: unable to execute command: Aborted (core dumped)
clang++-15: error: linker command failed due to signal (use -v to see invocation)
real 295.54
user 279.98
sys 13.45

My hope that there was some way a 32-bit program could access more than 4 GB memory was for naught :-(

So that confirms, then, that building Chromium with ThinLTO is not possible on an armhf build host, and probably any 32-bit host as well (so also x86). Which is a bummer, because those platforms need all the speed-ups they can get.

For the record, here are the stats on the bulk of my (armhf via QEMU) build. @Ark74's little RPi beat the snot out of my 8-CPU VM:

Command exited with non-zero status 2
	Command being timed: "ionice -c 3 nice -99 dpkg-buildpackage -b"
	User time (seconds): 3631058.76
	System time (seconds): 30246.36
	Percent of CPU this job got: 2%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 127:29:08
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 3594472
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 37749
	Minor (reclaiming a frame) page faults: 2104661801
	Voluntary context switches: 7174060
	Involuntary context switches: 89119022
	Swaps: 0
	File system inputs: 216
	File system outputs: 13338312
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 2

@Ark74
Copy link

Ark74 commented Nov 8, 2023

@iskunk I wonder if linux-image-generic-lpae could make any difference on the memory usage.

I'm running a test, we'll find out later in the week.

Update: No change, so yeah seems like armhf should have thin-lto disabled.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants