Generated .dSYM doesn't contain debug info when using swift.full_lto & swift.emit.bc features

[sanju-naik]

Problem

I am trying to enable swift.full_lto & swift.emit.bc to experiment around the app size reduction, but looks like when these flags are enabled, the generated .dSYM file doesn't contain debug info. So, when crashes are symbolicated using this dSYM, they don't display file names and line numbers in the crash log.

I have verified that the generated .dSYM is missing __debug_info & __debug_line using dwarfdump --show-section-sizes //path-to.dSYM

Here is the dwarfdump output from my local

dwarfdump --show-section-sizes /private/var/tmp/_bazel_sanju/128aa60197c68471680d99b52b191635/execroot/_main/bazel-out/ios_sim_arm64-opt-ios-sim_arm64-min14.0-applebin_ios-ST-216e50c47d69/bin/BazelDemoHelloWorld.app.dSYM
----------------------------------------------------
file: /private/var/tmp/_bazel_sanju/128aa60197c68471680d99b52b191635/execroot/_main/bazel-out/ios_sim_arm64-opt-ios-sim_arm64-min14.0-applebin_ios-ST-216e50c47d69/bin/BazelDemoHelloWorld.app.dSYM/Contents/Resources/DWARF/BazelDemoHelloWorld
----------------------------------------------------
SECTION           SIZE (b)
----------------  --------
__debug_abbrev           1 (0.00%)
__debug_str              1 (0.00%)
__apple_namespac        36 (0.17%)
__apple_names           36 (0.17%)
__apple_types           48 (0.23%)
__apple_objc            36 (0.17%)

 Total Size: 158  (0.77%)
 Total File Size: 20638
----------------------------------------------------

Expected behaviour

The generated .dSYM should contain valid debug info so that crashes can be symbolicated correctly with file names and line numbers.

This works as expected when it's done on a vanilla Xcode project by passing -Xfrontend -lto=llvm-full -Xfrontend --emit-bc in OTHER_SWIFT_FLAGS .

Here is the dwarfdump output of .dSYM generated from Xcode

dwarfdump --show-section-sizes /Users/sanju/Downloads/DerivedData/BazelDemoHelloWorld-fdzzlllmmnlyfofgcchicknpofpz/Build/Products/Release-iphonesimulator/BazelDemoHelloWorld.app.dSYM/
----------------------------------------------------
file: /Users/sanju/Downloads/DerivedData/BazelDemoHelloWorld-fdzzlllmmnlyfofgcchicknpofpz/Build/Products/Release-iphonesimulator/BazelDemoHelloWorld.app.dSYM/Contents/Resources/DWARF/BazelDemoHelloWorld(arm64)
----------------------------------------------------
SECTION           SIZE (b)
----------------  --------
__swift_ast          51224 (4.02%)
__debug_line         96407 (7.56%)
__debug_info        425671 (33.37%)
__debug_aranges        672 (0.05%)
__debug_ranges        2128 (0.17%)
__debug_loc            413 (0.03%)
__debug_abbrev        3271 (0.26%)
__debug_str         475977 (37.31%)
__apple_namespac        36 (0.00%)
__apple_names         5872 (0.46%)
__apple_types       181176 (14.20%)
__apple_objc            36 (0.00%)

 Total Size: 1242883  (97.43%)
 Total File Size: 1275651
----------------------------------------------------

How to reproduce

Attcched BazelDemoHelloWorld.zip
contains both Bazel & Xcode reproducible examples.

For Bazel running this command produces .dSYM and can be verified that it doesn't contain debug info

bazel build --config=Release --ios_multi_cpus=sim_arm64 --swiftcopt=-v --linkopt=-v BazelDemoHelloWorld . Since the build is done with --linkopt=-v , we can see bazel linker step has these warnings

warning: (arm64) /tmp/lto.o unable to open object file: No such file or directory
warning: no debug symbols in executable (-arch arm64) 

Xcode project can be built by selecting BazelDemoHelloWorld scheme and the generated .dSYM file in DerivedData contains valid debug info.

[sanju-naik]

I tried comparing the Xcode and Bazel's compilation and linking steps to see what the difference is, it looks like Xcode produces a file called BazelDemoHelloWorld_lto.o and that's passed to clang during linking using -object_path_lto option. This file contains the DWARF debug information. Then seems like the linker adds debug information from this file to the final binary.

I was able to confirm this by removing -object_path_lto from the linker invocation extracted from Xcode, which results in the same error. The final binary misses the debug info. So confirmed it's this file that is adding debug info.

Also, when I run dsymutil on the generated binary from the above step, it throws the exact same warning we see during bazel's link step

warning: (arm64) /tmp/lto.o unable to open object file: No such file or directory

This further confirms the hypothesis.

Now with Bazel, how can I generate this App_lto.o file during compilation and pass it to clang? If someone can point me to source code where this is all being done, that would be helpful too.

[sanju-naik]

I think I got it to working 🎉 App_lto.o is not generated during compilation, rather generated during linking after performing LTO optimisations. -object_path_lto tells the linker where to write this file, then it seems like the Linker uses this while linking the final binary.

By passing - linkopts = ["-Wl,-object_path_lto,bazel-out/**/BazelDemoHelloWorld_lto.o"] I was able to generate valid dSYM which contains valid debug info.

But I think a better fix would be to fix this at source, i.e pass -object_path_lto option to the linker if the swift_library is compiled with -emit-bc / LTO flags.

[sanju-naik]

@keith I think this is a bug in the full LTO implementation, rules_swift supports full LTO builds by passing swift.full_lto & swift.emit.bc flags, but those flags aren't sufficient to get a working build & dSYM so its a broken feature, hence can this be reopened?

[luispadron]

@sanju-naik could you contribute a fix since it looks like you've figured out the root cause here. Let us know how we can help

[sanju-naik]

@luispadron sure. just need to understand the source code structure, will plan for this and let you folks know if I need any help.