AddressSanitizer and LeakSanitizer

Here is an overview of how to detect memory errors in OpenVINO using AddressSanitizer and LeakSanitizer tools.

Running OpenVINO tests and tools with sanitizers requires changing build options and environment variables.

Supported platforms

• Linux
• Mac OS
• Windows (Visual Studio 2019 version >= 16.9)

Enabling AddressSanitizer and LeakSanitizer for OpenVINO

Use the ENABLE_SANITIZER flag while building OpenVINO to enable AddressSanitizer. LeakSanitizer is enabled automatically with AddressSanitizer.

cmake .. -DENABLE_SANITIZER=ON

Inside Intel, you can use pre-built "sanitizer" binaries from nightly artifacts.

Running memory validation

Binaries compiled with -DENABLE_SANITIZER=ON report the runtime memory errors found to stdout. AddressSanitizer error messages start from the "AddressSanitizer" word.

AddressSanitizer and LeakSanitizer use ASAN_OPTIONS and LSAN_OPTIONS environment variables for their configuration.

Run OpenVINO tests and tools with the environment variables set to:

LSAN_OPTIONS=suppressions=<OpenVINO Repository>/tests/lsan/suppressions.txt NEOReadDebugKeys=1 DisableDeepBind=1

If running through Python you need to preload a sanitizer runtime before the python interpreter and tweak its memory allocator with the following environment variables set to:

# note libasan and libstdc++ paths may be different on your system
PYTHONMALLOC=malloc LD_PRELOAD=/lib/x86_64-linux-gnu/libasan.so.5:/lib/x86_64-linux-gnu/libstdc++.so.6

The default AddressSanitizer behavior is to halt execution on the first error found. To continue execution to see other errors add ASAN_OPTIONS=halt_on_error=0 environment variable.

Interpreting LeakSanitizer results

LeakSanitizer reports memory leaks to the stdout on the application exit.

LeakSanitizer distinguishes between directly leaked blocks (not reachable from anywhere) and indirectly leaked blocks (reachable from other leaked blocks). When fixing memory leaks start from "Direct leaks" first. Direct leaks can be a cause of indirect ones. If you have only indirect leaks those are the result of cyclic references.

Identifying googletest test case

To debug a leak sanitizer issue reported from the googletest suite you may want to use the --gtest_filter flag to execute a specific test case. The googletest test case name usually can be guessed from the call stack. For example, having a stack frame

#31 0x559299e76ca3 in ie_core_import_network_from_file_importNetworkFromFile_Test::TestBody() openvino/src/bindings/c/tests/ie_c_api_test.cpp:388

run googletest executable with the --gtest_filter=ie_core_import_network_from_file.importNetworkFromFile to reproduce the issue.

Sometimes it might be impossible to identify the test case from the log to reproduce the issues. In that case, run the googletest executable under https://github.com/google/gtest-parallel once so you will get log per-test and can easily identify the test cases having issues:

./gtest-parallel path/to/binary

Known limitations

1. An old version of OpenVINO GPU plugin uses compute-runtime which is incompatible with AddressSanitizer and LeakSanitizer: https://github.com/intel/compute-runtime/issues/376

2. AddressSanitizer and LeakSanitizer may conflict with signal handlers such as libSegFault.so