Enabling libasan causes failures during tests

[tmiw]

@barjac reported an AddressSanitizer error during freedv-gui build:

[100%] Built target freedv_datac0c1_rx
make[1]: Leaving directory '/home/baz/build/BLD/BLD_MgaX_FREEDV/freedv-git/SOURCES/freedv-1.9.9-202403221300-3e864/codec2/build_linux'
/usr/bin/cmake -E cmake_progress_start /home/baz/BLD/BLD_MgaX_FREEDV/freedv-git/SOURCES/freedv-1.9.9-202403221300-3e864/codec2/build_linux/CMakeFiles 0
+ cd src
+ export LD_LIBRARY_PATH=/home/baz/BLD/BLD_MgaX_FREEDV/freedv-git/SOURCES/freedv-1.9.9-202403221300-3e864/LPCNet/build_linux/src
+ LD_LIBRARY_PATH=/home/baz/BLD/BLD_MgaX_FREEDV/freedv-git/SOURCES/freedv-1.9.9-202403221300-3e864/LPCNet/build_linux/src
+ ./freedv_tx 2020 /home/baz/BLD/BLD_MgaX_FREEDV/freedv-git/SOURCES/freedv-1.9.9-202403221300-3e864/LPCNet/wav/wia.wav -
+ ./freedv_rx 2020 - /dev/null
=================================================================
==36243==ERROR: AddressSanitizer: stack-use-after-scope on address 0x7ffefa65e990 at pc 0x000000402ffb bp 0x7ffefa6593d0 sp 0x7ffefa6593c8
READ of size 8 at 0x7ffefa65e990 thread T0
    #0 0x402ffa in my_get_next_tx_char /home/baz/BLD/BLD_MgaX_FREEDV/freedv-git/SOURCES/freedv-1.9.9-202403221300-3e864/codec2/src/freedv_tx.c:46
    #1 0x7f66ff6b3c02 in freedv_comptx_2020 /home/baz/BLD/BLD_MgaX_FREEDV/freedv-git/SOURCES/freedv-1.9.9-202403221300-3e864/codec2/src/freedv_2020.c:174
    #2 0x7f66ff6a575c in freedv_comptx /home/baz/BLD/BLD_MgaX_FREEDV/freedv-git/SOURCES/freedv-1.9.9-202403221300-3e864/codec2/src/freedv_api.c:389
    #3 0x7f66ff6a5ea5 in freedv_tx /home/baz/BLD/BLD_MgaX_FREEDV/freedv-git/SOURCES/freedv-1.9.9-202403221300-3e864/codec2/src/freedv_api.c:329
    #4 0x40284d in main /home/baz/BLD/BLD_MgaX_FREEDV/freedv-git/SOURCES/freedv-1.9.9-202403221300-3e864/codec2/src/freedv_tx.c:205
    #5 0x7f66ff47c736 in __libc_start_call_main (/lib64/libc.so.6+0x23736)
    #6 0x7f66ff47c7f4 in __libc_start_main@GLIBC_2.2.5 (/lib64/libc.so.6+0x237f4)
    #7 0x402ea0 in _start (/home/baz/build/BLD/BLD_MgaX_FREEDV/freedv-git/SOURCES/freedv-1.9.9-202403221300-3e864/codec2/build_linux/src/freedv_tx+0x402ea0)

Address 0x7ffefa65e990 is located in stack of thread T0 at offset 240 in frame
    #0 0x40231f in main /home/baz/BLD/BLD_MgaX_FREEDV/freedv-git/SOURCES/freedv-1.9.9-202403221300-3e864/codec2/src/freedv_tx.c:59

  This frame has 3 object(s):
    [32, 36) 'opt_idx' (line 97)
    [48, 128) 'f2020' (line 66)
    [160, 256) 'my_cb_state' (line 187) <== Memory access at offset 240 is inside this variable
HINT: this may be a false positive if your program uses some custom stack unwind mechanism, swapcontext or vfork
      (longjmp and C++ exceptions *are* supported)
SUMMARY: AddressSanitizer: stack-use-after-scope /home/baz/BLD/BLD_MgaX_FREEDV/freedv-git/SOURCES/freedv-1.9.9-202403221300-3e864/codec2/src/freedv_tx.c:46 in my_get_next_tx_char
Shadow bytes around the buggy address:
  0x10005f4c3ce0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10005f4c3cf0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10005f4c3d00: cb cb cb cb 00 00 00 00 00 00 00 00 00 00 00 00
  0x10005f4c3d10: 00 00 00 00 f1 f1 f1 f1 04 f2 00 00 00 00 00 00
  0x10005f4c3d20: 00 00 00 00 f2 f2 f2 f2 f8 f8 f8 f8 f8 f8 f8 f8
=>0x10005f4c3d30: f8 f8[f8]f8 f3 f3 f3 f3 00 00 00 00 00 00 00 00
  0x10005f4c3d40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10005f4c3d50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10005f4c3d60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10005f4c3d70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10005f4c3d80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==36243==ABORTING
frames decoded: 0  output speech samples: 0
+ cd /home/baz/BLD/BLD_MgaX_FREEDV/freedv-git/SOURCES/freedv-1.9.9-202403221300-3e864

Additionally, it appears a large number of ctests fail with AddressSanitizer enabled. Building with the following on an aarch64 Ubuntu VM:

$ CFLAGS=-fsanitize=address LDFLAGS=-fsanitize=address cmake -DLPCNET_BUILD_DIR=../../LPCNet/build_linux -DUNITTEST=1 ..
$ make -j6
$ ASAN_OPTIONS=detect_odr_violation=0 make test
....
61% tests passed, 60 tests failed out of 155

Total Test time (real) = 252.27 sec

The following tests FAILED:
          1 - test_clang_format (Failed)
         13 - test_ch_papr (Failed)
         19 - test_COHPSK_modem_octave_port (Failed)
         20 - test_COHPSK_modem_AWGN_BER (Failed)
         21 - test_COHPSK_modem_freq_offset (Failed)
         23 - test_OFDM_modem_octave_port (Failed)
         24 - test_OFDM_modem_octave_port_Nc_31 (Failed)
         34 - test_OFDM_modem_700D (Failed)
         35 - test_OFDM_modem_700D_ldpc (Failed)
         36 - test_OFDM_modem_2020_ldpc (Failed)
         37 - test_OFDM_modem_AWGN_BER (Failed)
         38 - test_OFDM_modem_fading_BER (Failed)
         39 - test_OFDM_modem_phase_est_bw (Failed)
         41 - test_OFDM_modem_time_sync_2020 (Failed)
         42 - test_OFDM_modem_700E_AWGN (Failed)
         43 - test_OFDM_modem_2020B_AWGN (Failed)
         47 - test_OFDM_modem_datac1 (Failed)
         51 - test_OFDM_modem_datac13_octave (Failed)
         52 - test_OFDM_modem_datac4_ldpc_burst (Failed)
         53 - test_OFDM_modem_datac13_ldpc_burst (Failed)
         66 - test_freedv_api_1600 (Failed)
         68 - test_freedv_api_700D_backwards_compatability (Failed)
         69 - test_freedv_api_700D_speech (Failed)
         71 - test_freedv_api_700D_AWGN_BER (Failed)
         72 - test_freedv_api_700D_AWGN_BER_USECOMPLEX (Failed)
         76 - test_freedv_api_2020_to_ofdm_demod (Failed)
         77 - test_freedv_api_2020_from_ofdm_mod (Failed)
         78 - test_freedv_api_2020_awgn (Failed)
         79 - test_freedv_api_2020B_mpp (Failed)
         80 - test_freedv_api_2400A (Failed)
         81 - test_freedv_api_2400B (Failed)
         82 - test_freedv_api_800XA (Failed)
         83 - test_freedv_api_rawdata_800XA (Failed)
         84 - test_freedv_api_rawdata_2400A (Failed)
         85 - test_freedv_api_rawdata_2400B (Failed)
         86 - test_peak_levels (Failed)
         87 - test_peak_levels_lpcnet (Failed)
         98 - test_freedv_reliable_text_ideal_2020 (Failed)
        100 - test_freedv_reliable_text_fade_2020 (Failed)
        101 - test_memory_leak_FreeDV_1600_tx (Failed)
        102 - test_memory_leak_FreeDV_1600_rx (Failed)
        103 - test_memory_leak_FreeDV_700D_tx (Failed)
        104 - test_memory_leak_FreeDV_700D_rx (Failed)
        105 - test_memory_leak_FreeDV_700C_tx (Failed)
        106 - test_memory_leak_FreeDV_700C_rx (Failed)
        107 - test_memory_leak_FreeDV_FSK_LDPC_tx (Failed)
        108 - test_memory_leak_FreeDV_DATAC0_tx (Failed)
        109 - test_memory_leak_FreeDV_DATAC1_tx (Failed)
        110 - test_memory_leak_FreeDV_DATAC3_tx (Failed)
        111 - test_memory_leak_FreeDV_DATAC4_tx (Failed)
        112 - test_memory_leak_FreeDV_DATAC13_tx (Failed)
        113 - test_memory_leak_FreeDV_700E_tx (Failed)
        114 - test_memory_leak_FreeDV_2020_tx (Failed)
        115 - test_memory_leak_FreeDV_2020_rx (Failed)
        116 - test_memory_leak_FreeDV_2020B_tx (Failed)
        117 - test_memory_leak_FreeDV_2020B_rx (Failed)
        125 - test_vq_mbest (Failed)
        145 - test_freedv_data_raw_ofdm_datac4_burst_file (Failed)
        146 - test_freedv_data_raw_ofdm_datac13_burst_file (Failed)
        153 - test_demo_700d_python (Failed)
Errors while running CTest
Output from these tests are in: /home/parallels/freedv-gui/codec2/build_linux/Testing/Temporary/LastTest.log
Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.
make: *** [Makefile:91: test] Error 8

(More tests were failing without disabling the ODR detection. I disabled above since it seemed like a false alarm.)

Example of one of the ctest failures:

test 69
    Start 69: test_freedv_api_700D_speech

69: Test command: /usr/bin/sh "-c" "cd /home/parallels/freedv-gui/codec2/build_linux/src;
                            ./freedv_tx 700D ../../raw/ve9qrp_10s.raw - |
                            ./ch - - --No -20 |
                            ./freedv_rx 700D - /dev/null --squelch -2 -vv"
69: Test timeout computed to be: 1500
69: ch: Fs: 8000 NodB: -20.00 foff: 0.00 Hz fading: 0 nhfdelay: 0 clip: 32767.00 ssbfilt: 1 complexout: 0
69: =================================================================
69: ==49821==ERROR: AddressSanitizer: stack-use-after-scope on address 0xffffce666d80 at pc 0xaaaab4232d98 bp 0xffffce662bb0 sp 0xffffce662bc0
69: READ of size 8 at 0xffffce666d80 thread T0
69:     #0 0xaaaab4232d94 in my_get_next_tx_char /home/parallels/freedv-gui/codec2/src/freedv_tx.c:46
69:     #1 0xffff841c6a40 in freedv_comptx_ofdm /home/parallels/freedv-gui/codec2/src/freedv_700.c:266
69:     #2 0xffff841bd2a4 in freedv_comptx /home/parallels/freedv-gui/codec2/src/freedv_api.c:375
69:     #3 0xffff841bd984 in freedv_tx /home/parallels/freedv-gui/codec2/src/freedv_api.c:329
69:     #4 0xaaaab42328f0 in main /home/parallels/freedv-gui/codec2/src/freedv_tx.c:205
69:     #5 0xffff83f273f8 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58
69:     #6 0xffff83f274c8 in __libc_start_main_impl ../csu/libc-start.c:392
69:     #7 0xaaaab4232c2c in _start (/home/parallels/freedv-gui/codec2/build_linux/src/freedv_tx+0x2c2c)
69: 
69: Address 0xffffce666d80 is located in stack of thread T0 at offset 128 in frame
69:     #0 0xaaaab42321ac in main /home/parallels/freedv-gui/codec2/src/freedv_tx.c:59
69: 
69:   This frame has 3 object(s):
69:     [32, 36) 'opt_idx' (line 97)
69:     [48, 144) 'my_cb_state' (line 187) <== Memory access at offset 128 is inside this variable
69:     [176, 256) 'f2020' (line 66)
69: HINT: this may be a false positive if your program uses some custom stack unwind mechanism, swapcontext or vfork
69:       (longjmp and C++ exceptions *are* supported)
69: SUMMARY: AddressSanitizer: stack-use-after-scope /home/parallels/freedv-gui/codec2/src/freedv_tx.c:46 in my_get_next_tx_char
69: Shadow bytes around the buggy address:
69:   0x200ff9cccd60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
69:   0x200ff9cccd70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
69:   0x200ff9cccd80: cb cb cb cb 00 00 00 00 00 00 00 00 00 00 00 00
69:   0x200ff9cccd90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
69:   0x200ff9cccda0: f1 f1 f1 f1 04 f2 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8
69: =>0x200ff9cccdb0:[f8]f8 f2 f2 f2 f2 00 00 00 00 00 00 00 00 00 00
69:   0x200ff9cccdc0: f3 f3 f3 f3 00 00 00 00 00 00 00 00 00 00 00 00
69:   0x200ff9cccdd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
69:   0x200ff9cccde0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
69:   0x200ff9cccdf0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
69:   0x200ff9ccce00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
69: Shadow byte legend (one shadow byte represents 8 application bytes):
69:   Addressable:           00
69:   Partially addressable: 01 02 03 04 05 06 07 
69:   Heap left redzone:       fa
69:   Freed heap region:       fd
69:   Stack left redzone:      f1
69:   Stack mid redzone:       f2
69:   Stack right redzone:     f3
69:   Stack after return:      f5
69:   Stack use after scope:   f8
69:   Global redzone:          f9
69:   Global init order:       f6
69:   Poisoned by user:        f7
69:   Container overflow:      fc
69:   Array cookie:            ac
69:   Intra object redzone:    bb
69:   ASan internal:           fe
69:   Left alloca redzone:     ca
69:   Right alloca redzone:    cb
69:   Shadow gap:              cc
69: ==49821==ABORTING
69: ch: SNR3k(dB):      nan  C/No....:      nan
69: ch: peak.....:     0.00  RMS.....:      nan   CPAPR.....:   nan 
69: ch: Nsamples.:        0  clipped.:      nan%  OutClipped:   nan%
69: frames decoded: 0  output speech samples: 0
69: 
69: =================================================================
69: ==49822==ERROR: LeakSanitizer: detected memory leaks
69: 
69: Direct leak of 9 byte(s) in 1 object(s) allocated from:
69:     #0 0xffffb5da8e30 in __interceptor_strdup ../../../../src/libsanitizer/asan/asan_interceptors.cpp:454
69:     #1 0xaaaac0d02888 in main /home/parallels/freedv-gui/codec2/src/ch.c:152
69:     #2 0xffffb59473f8 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58
69:     #3 0xffffb59474c8 in __libc_start_main_impl ../csu/libc-start.c:392
69:     #4 0xaaaac0d0442c in _start (/home/parallels/freedv-gui/codec2/build_linux/src/ch+0x442c)
69: 
69: SUMMARY: AddressSanitizer: 9 byte(s) leaked in 1 allocation(s).
1/1 Test #69: test_freedv_api_700D_speech ......***Failed  Required regular expression not found. Regex=[frames decoded: 62  output speech samples: 7
]  1.11 sec

A quick look seems to indicate that it's due to the following definition for my_cb_state (in both freedv_rx and freedv_tx):

  } else {
    /* set up callback for txt msg chars */
    struct my_callback_state my_cb_state;
    sprintf(my_cb_state.tx_str, "cq cq cq hello world\r");
    my_cb_state.ptx_str = my_cb_state.tx_str;
    my_cb_state.calls = 0;
    freedv_set_callback_txt(freedv, NULL, &my_get_next_tx_char, &my_cb_state);
  }

This definition should be moved to the top of main() to eliminate this error.

[drowe67]

Thanks @tmiw. If you think you have a quick fix and don't mind working on it pls feel free to go ahead and raise a PR.

[tmiw]

I created #45 with what I was able to get to tonight. There's still something like 14 ctest failures with libasan on, but I suspect at least a couple are due to configuration issues on my end.

[drowe67]

@tmiw - did any of the issues detected by AddressSanitizer cause any actual, real world issues, when running say freedv-gui? The changes in #45 are minor tweaks mainly in test code for soon to be deprecated modes. All this code was running fine before this tool was applied.

We already have a suite of valgrind tests that I guess traps most memory issues.

[tmiw]

@tmiw - did any of the issues detected by AddressSanitizer cause any actual, real world issues, when running say freedv-gui? The changes in #45 are minor tweaks mainly in test code for soon to be deprecated modes. All this code was running fine before this tool was applied.

We already have a suite of valgrind tests that I guess traps most memory issues.

This originally kicked off because @barjac couldn't build with AddressSanitizer while attempting to debug a freedv-gui issue. Fixing that of course ended up kicking off additional issues (for example, the behavior apparently varying wildly depending on what locale freedv-gui is set to). Also, I'm fairly sure he was able to duplicate the crashes in freedv-gui without AddressSanitizer enabled at one point, too.

For reference, the most recent crashes seemed to be happening while trying to decode 800XA:

==261635==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x602000185637 at pc 0x7f503ae47971 bp 0x7f501a342f50 sp 0x7f501a342700
WRITE of size 8 at 0x602000185637 thread T23
    #0 0x7f503ae47970 in __interceptor_memset (/lib64/libasan.so.8+0x47970)
    #1 0x7f503acbc176 in fvhff_extract_frame_voice /home/baz/BLD/BLD_MgaX_FREEDV/freedv-git/SOURCES/freedv-1.9.9-202403221300-3e864/codec2/src/freedv_vhf_framing.c:556
    #2 0x7f503acbef19 in fvhff_extract_frame /home/baz/BLD/BLD_MgaX_FREEDV/freedv-git/SOURCES/freedv-1.9.9-202403221300-3e864/codec2/src/freedv_vhf_framing.c:738
    #3 0x7f503acbef19 in fvhff_deframe_bits /home/baz/BLD/BLD_MgaX_FREEDV/freedv-git/SOURCES/freedv-1.9.9-202403221300-3e864/codec2/src/freedv_vhf_framing.c:855
    #4 0x7f503acbaf18 in freedv_comprx_fsk /home/baz/BLD/BLD_MgaX_FREEDV/freedv-git/SOURCES/freedv-1.9.9-202403221300-3e864/codec2/src/freedv_fsk.c:671
    #5 0x7f503aca9d8a in freedv_comprx /home/baz/BLD/BLD_MgaX_FREEDV/freedv-git/SOURCES/freedv-1.9.9-202403221300-3e864/codec2/src/freedv_api.c:806
    #6 0x4a45f8 in FreeDVReceiveStep::execute(std::shared_ptr<short>, int, int*) /home/baz/BLD/BLD_MgaX_FREEDV/freedv-git/SOURCES/freedv-1.9.9-202403221300-3e864/src/pipeline/FreeDVReceiveStep.cpp:100

(and earlier, were happening when decoding 2020 too)

There are also security implications, too. While admittedly unlikely, I imagine someone could come up with an OTA signal (or even just a WAV file that they convinced someone to play back in freedv-gui) that triggered one of the buffer overflows fixed by this PR and caused them to gain control of the user's system or something.

Anyway, that's a long way of saying that these modes should be fixed as long as the modes are still being used by something (e.g. freedv-gui). :)

[drowe67]

@tmiw - I'm not convinced this is an issue we should be addressing at this time.

We made a decision at PLT that non-critical bugs - especially in deprecated modes - would not be addressed at this time. Fixing ctests in deprecated code (that pass just fine) is not where we should working right now.

As codec 2 lead I'm making the call to pause any further libcodec2 work in this area - if you'd like to discuss further please raise at PLT.