Memory leak when calling `var_set` on a pointer that is dereferenced with array syntax

[alexmanning9904]

Summary

I have a trick sim with a variable var along with a pointer to var, varptr. I can use python bindings to set var by dereferencing varptr using trick.var_set("varptr[0]"). This functions correctly, but seems to cause a memory leak within the Trick MemoryManager.

Reproduction

Environment

• Trick (my commit hash is 0482202, I'll take a TODO to reproduce this off of the head of master)
• Valgrind. I installed via my package manager which gave v3.24.0

Sim Files

My sim folder is contained in minimal_leak_demo.zip. This includes:

• An S_define file with a static variable and statically-defined pointer to it.
• An input.py file that forces the memory leak
• An S_overrides.mk
• A demo.sh script to force the memory leak

Steps

1. Extract minimal_leak_demo.zip
2. cd into the extracted directory
3. Run demo.sh
4. Wait. This demo took ~2 hours to give me data on my machine. You can get faster results by reducing the number of trick.add_read calls in input.py, this just makes the leak a bit less obvious when browsing the massif outputs.

Outputs

The most important output is the results.txt file, which is produced by massif/ms_print. It contains a memory usage vs instructions executed graph alongside a few memory utilization snapshots.

Analysis

From performing the steps listed above, I got this results.txt. I copied out the memory allocation trace from snapshot 49 and snapshot 97. Snapshot 49 was the first snapshot taken when the leak was large enough for massif to print it, and snapshot 97 is the final and largest snapshot. The size difference between these two snapshots is ~170MB. The allocation within:

->11.03% (207,756,640B) 0xA1459C: Trick::MemoryManager::make_reference_attr(ALLOC_INFO*) (in /home/amanning/trick-memory-leak-hell/SIM_minimal_leak_demo/S_main_Linux_11.5_x86_64.exe)
| ->11.03% (207,756,640B) 0xA19C3C: Trick::MemoryManager::ref_var(REF2*, char*) (in /home/amanning/trick-memory-leak-hell/SIM_minimal_leak_demo/S_main_Linux_11.5_x86_64.exe)
|   ->11.03% (207,756,640B) 0x9FFEB0: REF_parse(RefParseContext*) (in /home/amanning/trick-memory-leak-hell/SIM_minimal_leak_demo/S_main_Linux_11.5_x86_64.exe)
|     ->11.03% (207,756,640B) 0xA18D23: Trick::MemoryManager::ref_attributes(char const*) (in /home/amanning/trick-memory-leak-hell/SIM_minimal_leak_demo/S_main_Linux_11.5_x86_64.exe)
|       ->11.03% (207,756,640B) 0xA092CD: ref_attributes (in /home/amanning/trick-memory-leak-hell/SIM_minimal_leak_demo/S_main_Linux_11.5_x86_64.exe)
|         ->11.03% (207,756,640B) 0x720977: int var_set_base<long long>(char const*, long long, char const*) (in /home/amanning/trick-memory-leak-hell/SIM_minimal_leak_demo/S_main_Linux_11.5_x86_64.exe)
|           ->11.03% (207,756,640B) 0x72069B: var_set(char const*, long long, char const*) (in /home/amanning/trick-memory-leak-hell/SIM_minimal_leak_demo/S_main_Linux_11.5_x86_64.exe)
|             ->11.03% (207,756,640B) 0x971C15: _wrap_var_set__SWIG_3 (in /home/amanning/trick-memory-leak-hell/SIM_minimal_leak_demo/S_main_Linux_11.5_x86_64.exe)
|               ->11.03% (207,756,640B) 0x972EC1: _wrap_var_set (in /home/amanning/trick-memory-leak-hell/SIM_minimal_leak_demo/S_main_Linux_11.5_x86_64.exe)

Accounts for ~130MB of this. Snapshot 97 also includes 2 additional traces that include var_set that look like they can explain the extra 40MB.

---

I've started to dig into the memory manager with the hope of fixing this, but I don't think that I'm anywhere close to a solution yet. I would appreciate any insight on corrective actions or workarounds. Thanks.

[hchen99]

The attached example has trick.add_read(i/20000, f'trick.var_set("dyn[0].varptr", {i})') in the input file. Assuming it was meant to be dyn.varptr[0]? Have you tried var_set(dyn.var,...)) instead?

[alexmanning9904]

Assuming it was meant to be dyn.varptr[0]

Yes, thank you for catching that - the target should be to dyn.varptr[0].

Have you tried var_set(dyn.var,...)) instead?

Yes, setting the variable directly does not seem to cause the leak.

[hchen99]

So you could just use var_set dyn.var, or simply assign the value with dyn.varptr[0]=xxx or dyn.var=xxx. Meanwhile, taking a closer look at var_set dyn.varptr[0] will be on our to-do list.

[hchen99]

@alexmanning9904 Please give the branch a try to see if the issue still occurs on your end with the attached example.

[alexmanning9904]

Hi @hchen99, sorry for all the delays:

The dyn.varptr[0]=xxx seems to leak less/not at all. I've only run it through my little demo script, but I'll give it a shot in our production environment and see what happens.

Unforrunately, from commit 4d00c3348c6e8364813511c56011caa26c8364cd off of the linked branch, I still see a large leak, even in the demo environment. See the attached massif file.

Thanks again for all of the help!

results_devbuild.txt

[hchen99]

Thanks for testing. Wanted to verify if I am getting following right:

• var_set(dyn.var, ...) -- ok
• var_set(dyn.varptr[0], ...) -- leak
  • I actually think if the issue still exists for this case, var_set(dyn.var,...) should not be ok either.
• dyn.var = ... -- ok
• dyn.varptr[0] = ... -- ok

[alexmanning9904]

Hi, sorry for the delay -

I reran all 4 cases with a really high count of add_read calls (300000) and I'm seeing a leak on all 4, to differing degrees -

• var_set(dyn.var, ...) - 454MB leak
• var_set(dyn.varptr[0], ...) - 464MB leak
• dyn.var - 402MB leak
• dyn.varptr[0] - 430MB leak

Let me run through again without using add_read; I wonder if I'm conflating problems here.

[alexmanning9904]

Okay, it seems like calls to add_read() also leak memory regardless of the variable sets. Using an input of:

for i in range(n_reads):
    trick.add_read(n_reads/sim_time, f'pass')

trick.stop(sim_time)

with n_reads set to 300000, I saw a leak of 398MB.

I'll re-run the various variable-setting cases without the use of add_read()

[alexmanning9904]

without calls to add_read:

• Using the original trick build:
  • var_set(dyn.var, ...) - leaks to a peak of 46MB
  • var_set(dyn.varptr[0], ...) - leaks to a peak of 69MB
  • dyn.var - does not leak; static 18MB
  • dyn.varptr[0] - does not leak; static 18MB
• Using commit 4d00c33:
  • I see exactly the same things as with our original build. See the attached massif output from var_set(dyn.varptr[0], ...).

results_set_ptr.txt