Incomplete coredumps leading to confusing GDB output

[RoundofThree]

The GDB output when analysing v8 coredumps is confusing due to the program headers pointed by the AT_PHDR auxv value contained in the coredump not matching the program headers in the executable. It seems the coredump does not contain a PT_LOAD that corresponds to the mapping containing the in-memory program headers (pointed by the AT_PHDR auxv entry). I observe that the in-memory program headers are invalid.

gef>  info auxv 
3    AT_PHDR              Program headers for program    0x100040
4    AT_PHENT             Size of program header entry   56
5    AT_PHNUM             Number of program headers      12
6    AT_PAGESZ            System page size               4096
8    AT_FLAGS             Flags                          0x0
9    AT_ENTRY             Entry point of program         0x32afec1
7    AT_BASE              Base address of interpreter    0x47ce0000
24   AT_EHDRFLAGS         ELF header e_flags             0x10000
15   AT_EXECPATH          Executable path                0xffffbff7ff80 "/home/zyj20/v8/out/cheri-uncompressed-debug-fuzz/d8"
18   AT_OSRELDATE         OSRELDATE                      1500026
16   AT_CANARY            Canary for SSP                 0xffffbff7ff40
17   AT_CANARYLEN         Length of the SSP canary       64
19   AT_NCPUS             Number of CPUs                 4
20   AT_PAGESIZES         Pagesizes                      0xffffbff7ff20
21   AT_PAGESIZESLEN      Number of pagesizes            32
22   AT_TIMEKEEP          Pointer to timehands           0xfffffffff020
23   AT_STACKPROT         Initial stack protection       0x3
25   AT_HWCAP             Machine-dependent CPU capability hints 0x10119ffb
26   AT_HWCAP2            Extension of AT_HWCAP          0x0
27   AT_BSDFLAGS          ELF BSD flags                  0xa0000001
28   AT_ARGC              Argument count                 1
29   AT_ARGV              Argument vector                0xffffbff7f380
30   AT_ENVC              Environment count              31
31   AT_ENVV              Environment vector             0xffffbff7f3a0
32   AT_PS_STRINGS        Pointer to ps_strings          0xffffbff7ffc0
35   AT_USRSTACKBASE      Top of user stack              0xfffffff80000
36   AT_USRSTACKLIM       Grow limit of user stack       0x40000000
37   ???                                                 0xffffbff7fa60
0    AT_NULL              End of vector                  0x0
gef>  p *(Elf64_Phdr *)0x100040
$1 = {
  p_type = 0x4d0,
  p_flags = 0x0,
  p_offset = 0x100022001fff22,
  p_vaddr = 0x37fa10d,
  p_paddr = 0x48,
  p_filesz = 0x100022004e6ab1,
  p_memsz = 0x42857e5,
  p_align = 0x78
}

In svr4_exec_displacement in GDB, there is a verification that the program headers pointed by the AT_PHDR auxv value and the on-disk program headers match:
https://github.com/CTSRD-CHERI/gdb/blob/6b6284a9a2bbbe50548b7d1271e6984c26b12c24/gdb/solib-svr4.c#L2753
In this case, the verification fails and the svr4_exec_displacement function returns 0. Therefore, the displacement is not correct leading to confusing debug output.

As a temporary measure, I zero out the AT_PHDR value to skip the verification.

This is the original GDB output, without zeroing AT_PHDR.

Reading symbols from /home/zyj20/v8/out/cheri-uncompressed-debug-fuzz/d8...
[New LWP 219834]
Cannot access memory at address 0xdc5d40004040c04c
Cannot access memory at address 0xdc5d40004040c03c
Cannot access memory at address 0xdc5d40004040c03c
Core was generated by `./d8'.
Program terminated with signal SIGPROT, CHERI protection violation.
Capability tag fault.
#0  0x00000000033e82fc in v8::internal::ValidatePropertyCallbackInfo<v8::Boolean> (info=...) at ../../src/api/api.cc:11544
11544     CHECK(info.Data()->IsValue());
gef>  bt
#0  0x00000000033e82fc in v8::internal::ValidatePropertyCallbackInfo<v8::Boolean> (info=...) at ../../src/api/api.cc:11544
#1  0x00000000033e8154 in v8::internal::ValidatePropertyCallbackInfo<v8::Boolean> (info=...) at ../../src/api/api.cc:11539
#2  0x00000000032f0e08 in Cr_z_inflate_fast_chunk_ (strm=0x32f0e09 <Cr_z_inflate_fast_chunk_+524> [rxRE,0x100000-0x8b60000] (sentry), 
    start=0xdc5d4000) at ../../third_party/zlib/contrib/optimizations/inffast_chunk.c:142
#3  0x0000000047cff298 in ?? ()
Backtrace stopped: frame did not save the PC
gef>  info aux
3    AT_PHDR              Program headers for program    0x100040

This is the GDB output of the patched coredump, with zeroed AT_PHDR, and correct backtraces.

Reading symbols from /home/zyj20/v8/out/cheri-uncompressed-debug-fuzz/d8...
[New LWP 219834]
Core was generated by `./d8'.
Program terminated with signal SIGPROT, CHERI protection violation.
Capability tag fault.
#0  0x00000000033e82fc in __sanitizer_cov_trace_pc_guard (
    guard=0x7d0fe78 <.L__sancov_gen_.7231cbb79ecf2923ad1e8777c20ac37a.8> [rwRW,0x7d0fe78-0x7d0fe7c]) at ../../src/d8/cov.cc:89
89        shmem->edges[index / 8] |= 1 << (index % 8);
gef>  bt
#0  0x00000000033e82fc in __sanitizer_cov_trace_pc_guard (
    guard=0x7d0fe78 <.L__sancov_gen_.7231cbb79ecf2923ad1e8777c20ac37a.8> [rwRW,0x7d0fe78-0x7d0fe7c]) at ../../src/d8/cov.cc:89
#1  0x00000000033e8434 in v8::base::Malloc (size=0xdc5d40004d604840) at ../../src/base/platform/memory.h:32
#2  0x00000000033e8154 in __sanitizer_cov_trace_pc_guard_init (start=0x7d03010 <__start___sancov_guards> [rwRW,0x7d03000-0x7f9a800], 
    stop=0x7f9a624 [rwRW,0x7f9a624-0x7f9a624]) at ../../src/d8/cov.cc:45
#3  0x00000000032f0e08 in sancov.module_ctor_trace_pc_guard ()
#4  0x0000000047cff298 in objlist_call_init (list=<optimized out>, lockstate=<optimized out>) at /usr/src/libexec/rtld-elf/rtld.c:3686
#5  0x0000000047cfdb78 in _rtld (aux=<optimized out>, exit_proc=0xfffffff7ffe0 [rwRW,0xfffffff7ffe0-0xfffffff7fff0], 
    objp=<optimized out>) at /usr/src/libexec/rtld-elf/rtld.c:1133
#6  0x0000000047cfa498 in rtld_start () at /usr/src/libexec/rtld-elf/aarch64/rtld_start.S:77
gef>  info aux
3    AT_PHDR              Program headers for program    0x0

I can't attach the example d8 coredump and the corresponding d8 binary in this issue.

[jrtc27]

So is AT_PHDR wrong or is it pointing to memory that's wrong? You say there's no PT_LOAD, but yet you can read from that address, which wouldn't be the case if there was no segment.

[RoundofThree]

I think AT_PHDR is not wrong, but it is pointing to memory that is wrong. In the case of d8 coredumps, you can read from that address (which is why gdb mistakenly continues the comparison verification of in-memory and on-disk program headers), because it coincides with the file-backed .dynsym as you can see in info files of the original coredump (0x0000000000100040 is in [0x00000000000080c8 - 0x0000000000329e60]). I am not very sure why, so correct me if I am wrong.

Once I patch the coredump, and so gdb relocates correctly, x/10gx 0x0000000000100040 will give an error:

gef>  x/10gx 0x0000000000100040
0x100040:       Cannot access memory at address 0x100040

I compared this with the output in a Linux box, and the difference is that in Linux there is a PT_LOAD in the coredump that corresponds to the in-memory program headers.

This is info files for the original coredump:

gef>  info files
Symbols from "/home/zyj20/v8/out/cheri-uncompressed-debug-fuzz/d8".
Local core dump file:
        `/home/zyj20/v8/out/cheri-uncompressed-debug-fuzz/d8.orig.core', file type elf64-littleaarch64.
        0x0000000007697000 - 0x0000000007ce0000 is load1
        0x0000000007ce0000 - 0x0000000007f9a000 is load2
        0x0000000007f9a000 - 0x0000000008b51000 is load3
        0x0000000047d29000 - 0x0000000047d2d000 is load4
        0x0000000047d3c000 - 0x0000000047d40000 is load5
        0x0000000047d40000 - 0x0000000047d49000 is load6
        0x0000000047d8c000 - 0x0000000047d8e000 is load7
        0x0000000047d9d000 - 0x0000000047d9f000 is load8
        0x0000000047d9f000 - 0x0000000047db2000 is load9
        0x0000000047db2000 - 0x0000000047dbb000 is load10
        0x0000000047de0000 - 0x0000000047de2000 is load11
        0x0000000047df1000 - 0x0000000047df2000 is load12
        0x0000000047e15000 - 0x0000000047e16000 is load13
        0x0000000047e25000 - 0x0000000047e26000 is load14
        0x0000000047f46000 - 0x0000000047f63000 is load15
        0x0000000047f72000 - 0x0000000047f75000 is load16
        0x0000000047f75000 - 0x0000000047f7c000 is load17
        0x0000000047fb4000 - 0x0000000047fb8000 is load18
        0x0000000047fc7000 - 0x0000000047fc9000 is load19
        0x000000004802d000 - 0x000000004802f000 is load20
        0x000000004803e000 - 0x0000000048040000 is load21
        0x0000000048080000 - 0x0000000048083000 is load22
        0x0000000048092000 - 0x0000000048096000 is load23
        0x000000004827c000 - 0x0000000048298000 is load24
        0x00000000482a7000 - 0x00000000482b3000 is load25
        0x00000000482b3000 - 0x00000000486e3000 is load26
        0x000000004871b000 - 0x000000004871c000 is load27
        0x000000004872b000 - 0x000000004872c000 is load28
        0x000000004872c000 - 0x000000004872d000 is load29
        0x0000000048764000 - 0x0000000048766000 is load30
        0x0000000048775000 - 0x0000000048776000 is load31
        0x0000000048776000 - 0x0000000048778000 is load32
        0x000000004877f000 - 0x0000000048f80000 is load33
        0x0000000048f87000 - 0x0000000048f9d000 is load34
        0x0000000048f9f000 - 0x0000000048fc4000 is load35
        0x0000000048fc7000 - 0x0000000048fcc000 is load36
        0x0000000048fd3000 - 0x0000000048ff5000 is load37
        0x0000000048ffb000 - 0x000000004900b000 is load38
        0x000000004900b000 - 0x0000000049014000 is load39
        0x0000000049200000 - 0x0000000049400000 is load40
        0x0000000049400000 - 0x0000000049600000 is load41
        0x0000000049600000 - 0x0000000049c00000 is load42
        0x0000fbfdbffff000 - 0x0000fbfdc0000000 is load43
        0x0000ffffbfeff000 - 0x0000ffffbff80000 is load44
        0x0000fffffff60000 - 0x0000fffffff80000 is load45
        0x0000fffffffff000 - 0x0001000000000000 is load46
        While running this, GDB does not access memory from...
Local exec file:
        `/home/zyj20/v8/out/cheri-uncompressed-debug-fuzz/d8', file type elf64-littleaarch64.
        Entry point: 0x31afec0
        0x0000000000008000 - 0x0000000000008015 is .interp
        0x0000000000008018 - 0x000000000000807c is .note.tag
        0x000000000000807c - 0x0000000000008094 is .note.gnu.build-id
        0x0000000000008094 - 0x00000000000080c4 is .note.cheri
        0x00000000000080c8 - 0x0000000000329e60 is .dynsym
// [...]

This is info files for the patched coredump:

gef>  info files
Symbols from "/home/zyj20/v8/out/cheri-uncompressed-debug-fuzz/d8".
Local core dump file:
        `/home/zyj20/v8/out/cheri-uncompressed-debug-fuzz/d8.47612.core', file type elf64-littleaarch64.
        0x0000000007697000 - 0x0000000007ce0000 is load1
        0x0000000007ce0000 - 0x0000000007f9a000 is load2
        0x0000000007f9a000 - 0x0000000008b51000 is load3
        0x0000000047d29000 - 0x0000000047d2d000 is load4
        0x0000000047d3c000 - 0x0000000047d40000 is load5
        0x0000000047d40000 - 0x0000000047d49000 is load6
        0x0000000047d8c000 - 0x0000000047d8e000 is load7
        0x0000000047d9d000 - 0x0000000047d9f000 is load8
        0x0000000047d9f000 - 0x0000000047db2000 is load9
        0x0000000047db2000 - 0x0000000047dbb000 is load10
        0x0000000047de0000 - 0x0000000047de2000 is load11
        0x0000000047df1000 - 0x0000000047df2000 is load12
        0x0000000047e15000 - 0x0000000047e16000 is load13
        0x0000000047e25000 - 0x0000000047e26000 is load14
        0x0000000047f46000 - 0x0000000047f63000 is load15
        0x0000000047f72000 - 0x0000000047f75000 is load16
        0x0000000047f75000 - 0x0000000047f7c000 is load17
        0x0000000047fb4000 - 0x0000000047fb8000 is load18
        0x0000000047fc7000 - 0x0000000047fc9000 is load19
        0x000000004802d000 - 0x000000004802f000 is load20
        0x000000004803e000 - 0x0000000048040000 is load21
        0x0000000048080000 - 0x0000000048083000 is load22
        0x0000000048092000 - 0x0000000048096000 is load23
        0x000000004827c000 - 0x0000000048298000 is load24
        0x00000000482a7000 - 0x00000000482b3000 is load25
        0x00000000482b3000 - 0x00000000486e3000 is load26
        0x000000004871b000 - 0x000000004871c000 is load27
        0x000000004872b000 - 0x000000004872c000 is load28
        0x000000004872c000 - 0x000000004872d000 is load29
        0x0000000048764000 - 0x0000000048766000 is load30
        0x0000000048775000 - 0x0000000048776000 is load31
        0x0000000048776000 - 0x0000000048778000 is load32
        0x000000004877f000 - 0x0000000048f80000 is load33
        0x0000000048f87000 - 0x0000000048f9d000 is load34
        0x0000000048f9f000 - 0x0000000048fc4000 is load35
        0x0000000048fc7000 - 0x0000000048fcc000 is load36
        0x0000000048fd3000 - 0x0000000048ff5000 is load37
        0x0000000048ffb000 - 0x000000004900b000 is load38
        0x000000004900b000 - 0x0000000049014000 is load39
        0x0000000049200000 - 0x0000000049400000 is load40
        0x0000000049400000 - 0x0000000049600000 is load41
        0x0000000049600000 - 0x0000000049c00000 is load42
        0x0000fbfdbffff000 - 0x0000fbfdc0000000 is load43
        0x0000ffffbfeff000 - 0x0000ffffbff80000 is load44
        0x0000fffffff60000 - 0x0000fffffff80000 is load45
        0x0000fffffffff000 - 0x0001000000000000 is load46
        While running this, GDB does not access memory from...
Local exec file:
        `/home/zyj20/v8/out/cheri-uncompressed-debug-fuzz/d8', file type elf64-littleaarch64.
        Entry point: 0x32afec0
        0x0000000000108000 - 0x0000000000108015 is .interp
        0x0000000000108018 - 0x000000000010807c is .note.tag
        0x000000000010807c - 0x0000000000108094 is .note.gnu.build-id
        0x0000000000108094 - 0x00000000001080c4 is .note.cheri
        0x00000000001080c8 - 0x0000000000429e60 is .dynsym
// [...]

[jrtc27]

FreeBSD doesn't dump read-only mappings because they can just be re-read from the original file. The program headers are normally part of a read-only PT_LOAD, so I wouldn't expect them to be in the core dump. What looks to be the real problem is that the first output there doesn't have the addresses offset by 0x100000 (which is arm64's ET_DYN_LOAD_ADDR); it's using the offset in the file, not the actual run-time address. For whatever reason GDB has failed to figure out the load base address, it would appear. I suspect this is a GDB bug, not a CheriBSD one.

[jrtc27]

What does (llvm-)readelf -Wl path/to/d8 say?

[RoundofThree]

$ readelf -Wl /home/zyj20/v8/out/cheri-uncompressed-debug-fuzz/d8 

Elf file type is DYN (Shared object file)
Entry point 0x31afec1
There are 12 program headers, starting at offset 64

Program Headers:
  Type           Offset   VirtAddr           PhysAddr           FileSiz  MemSiz   Flg Align
  PHDR           0x000040 0x0000000000000040 0x0000000000000040 0x0002a0 0x0002a0 R   0x8
  INTERP         0x008000 0x0000000000008000 0x0000000000008000 0x000015 0x000015 R   0x1
      [Requesting program interpreter: /libexec/ld-elf.so.1]
  LOAD           0x000000 0x0000000000000000 0x0000000000000000 0x319fe84 0x319fe84 R   0x10000
  LOAD           0x319fec0 0x00000000031afec0 0x00000000031afec0 0x43d7400 0x43d7400 R E 0x10000
  LOAD           0x75772c0 0x00000000075972c0 0x00000000075972c0 0x648d40 0x648d40 RW  0x10000
  LOAD           0x7bc0000 0x0000000007be0000 0x0000000007be0000 0x2ba624 0xe70740 RW  0x10000
  TLS            0x75772c0 0x00000000075972c0 0x00000000075972c0 0x000010 0x000590 R   0x10
  DYNAMIC        0x7ba6ea0 0x0000000007bc6ea0 0x0000000007bc6ea0 0x000210 0x000210 RW  0x8
  GNU_RELRO      0x75772c0 0x00000000075972c0 0x00000000075972c0 0x648d40 0x648d40 R   0x1
  GNU_EH_FRAME   0x1fa3530 0x0000000001fa3530 0x0000000001fa3530 0x1feff4 0x1feff4 R   0x4
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000 0x000000 RW  0
  NOTE           0x008018 0x0000000000008018 0x0000000000008018 0x0000ac 0x0000ac R   0x4

 Section to Segment mapping:
  Segment Sections...
   00     
   01     .interp 
   02     .interp .note.tag .note.gnu.build-id .note.cheri .dynsym .gnu.version .gnu.version_r .gnu.hash .hash .dynstr .rela.dyn .rela.plt .rodata .eh_frame_hdr .eh_frame 
   03     .text .plt 
   04     .tdata .tbss .fini_array .data.rel.ro .init_array .dynamic .got .got.plt 
   05     .data __sancov_guards .bss 
   06     .tdata .tbss 
   07     .dynamic 
   08     .tdata .tbss .fini_array .data.rel.ro .init_array .dynamic .got .got.plt 
   09     .eh_frame_hdr 
   10     
   11     .note.tag .note.gnu.build-id .note.cheri

[RoundofThree]

In svr4_exec_displacement in GDB, there is a check that in-file and in-memory program headers exist and match. In the original coredump, the addresses are not offset by 0x100000 because this check unfortunately fails. So I assume GDB assumes that it can read in-memory program headers?

[jrtc27]

If phdrs_target isn't valid, i.e. the phdrs aren't in a PT_LOAD in the core dump, it doesn't run that. At least in theory. Then it falls back on assuming that the displacement is correct. It's only meant to not use the displacement when it has proven that it's definitely incorrect.

[RoundofThree]

I inserted a debug print at the start of the verification and ran gdb ./d8 ./d8.orig.core:

  /* Verify that the auxilliary vector describes the same file as exec_bfd, by
     comparing their program headers.  If the program headers in the auxilliary
     vector do not match the program headers in the executable, then we are
     looking at a different file than the one used by the kernel - for
     instance, "gdb program" connected to "gdbserver :PORT ld.so program".  */

  if (bfd_get_flavour (current_program_space->exec_bfd ())
      == bfd_target_elf_flavour)
    {
      /* Be optimistic and return 0 only if GDB was able to verify the headers
	 really do not match.  */
      int arch_size;

      gdb::optional<gdb::byte_vector> phdrs_target
	= read_program_header (-1, &arch_size, NULL);
      gdb::optional<gdb::byte_vector> phdrs_binary
	= read_program_headers_from_bfd (current_program_space->exec_bfd ());
      if (phdrs_target && phdrs_binary)
	{
          gdb_printf (_("Doing the verification\n")); // <---
	  enum bfd_endian byte_order = gdbarch_byte_order (target_gdbarch ());
// [...]

And the output confirms that the verification codepath is taken for d8 but not for many other smaller binaries.

Reading symbols from /home/zyj20/d8...
[New LWP 219834]
Doing the verification
Cannot access memory at address 0xdc5d40004040c04c
Cannot access memory at address 0xdc5d40004040c03c
Cannot access memory at address 0xdc5d40004040c03c
Core was generated by `./d8'.
Program terminated with signal SIGPROT, CHERI protection violation.
Capability tag fault.
#0  0x00000000033e82fc in v8::internal::ValidatePropertyCallbackInfo<v8::Boolean> (info=...) at ../../src/api/api.cc:11544

warning: 11544  ../../src/api/api.cc: No such file or directory
(gdb) 

I think gdb is also reading from file-backed data, which leads to an incorrectly set phdrs_target.

[rwatson]

Tatting @bsdjhb, also @brooksdavis as an FYI.

[bsdjhb]

Hmm, I thought we used the value of AT_PHDR to figure out the initial displacement?

[bsdjhb]

So I think the root issue here is that when GDB is reading the program headers from the core dump, it should only read from the core dump and not permit falling through to the next layer in the target stack that reads from the executable. I'll need to think about how to fix that.