Getting "malloc: Double free" error when overwriting keys too often

[mallman]

I haven't quite nailed down the exact circumstances in which this bug occurs, but I think you can reliably reproduce it by modifying examples/basic.c to add multiple calls to hatrack_dict_put(envp_dict, env_key, env_val) in the same loop iteration when populating envp_dict, like so:

    while (envp[i]) {
        p       = envp[i];
        env_eq  = strchr(p, '=');
        env_key = strndup(p, env_eq - p);
        env_val = strdup(env_eq + 1);

        hatrack_dict_put(envp_dict, env_key, env_val);
        hatrack_dict_put(envp_dict, env_key, env_val);
        hatrack_dict_put(envp_dict, env_key, env_val);
        hatrack_dict_put(envp_dict, env_key, env_val);
        hatrack_dict_put(envp_dict, env_key, env_val);
        hatrack_dict_put(envp_dict, env_key, env_val);
        hatrack_dict_put(envp_dict, env_key, env_val);
        hatrack_dict_put(envp_dict, env_key, env_val);
        hatrack_dict_put(envp_dict, env_key, env_val);

        i++;
    }

Obviously, this depends on the number of environment variables. I have about 40. Just add more calls hatrack_dict_put(envp_dict, env_key, env_val) if you still don't trigger the bug.

I think this is related to the value of HATRACK_RETIRE_FREQ, because the bug occurs in the function call to mmm_empty() in mmm_retire() in mmm.c.

I'm running this on an M1 macOS 13.4 with Xcode 14.3.1 clang:

Apple clang version 14.0.3 (clang-1403.0.22.14.1)
Target: arm64-apple-darwin22.5.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

I configured hatrack from scripts/config-debug.

Thank you!

[viega]

Hey I'm really sorry to have taken so long to notice this. I don't know why I didn't get the notification; I just saw it right now.

My recollection of that API is that when you pass a value into it, it takes ownership of the memory cell passed in by default (unless you provide an ejection callback), and will free it when the item is ejected. If you're putting in the same value multiple times, you need to stick each copy of the value in a different memory cell.

Looking at the example, I would absolutely expect the above loop to lead to a crash even in a single threaded program, because when you insert the memory cell the second time, it marks the first for deletion once it can 'prove' that no other threads might be holding a reference to it.

Hope that's helpful. I'll leave this open for a while and will spend more time on it soon just to double check.

And I'm going to look at my settings right now to make sure I am getting any tickets right away, really sorry about that!

[mallman]

Hi John. No worries. I'm glad to hear from you. I thought the project might be abandoned, and it looks like great work.

Would it be an error to reuse the same key to set different values? I'm not sure the fact that the values are the same in my code sample is germane to my original issue.

I set hatrack aside when I hit this stumbling block. Returning to my test case, I'm no longer getting the "double free" error.

Can you try running the example?

I can see that one thing that's changed on my side since I reported this is I updated my compiler:

Apple clang version 15.0.0 (clang-1500.0.40.1)
Target: arm64-apple-darwin22.6.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

I just tried again with clang 14.0.3, and still don't get an error. Hmmmm....

[viega]

Absolutely.

I just ran the example with the extra calls to re-add the same environment variable. That does cause double frees, as I would expect:

viega@UpDog examples % ./basic 
Freeing: TMPDIR: /var/folders/6n/n6z_d_2952g_kdzs7bfz4q8w0000gn/T/
Freeing: `A: "
basic(99960,0x1f14e1e00) malloc: *** error for object 0x600000044160: pointer being freed was not allocated
basic(99960,0x1f14e1e00) malloc: *** set a breakpoint in malloc_error_break to debug
zsh: abort      ./basic

I don't have an environment variable named A, of course.

If you look at the ejection handler in that file, envp_free_handler, it deletes the memory allocated for both the key and the value at some point after there's an overwrite.

So if you insert the same memory cell multiple times, it'll get ejected multiple times, and it'll free multiple times.

The handler getting called indicates that the ejected cell isn't being referenced, but it doesn't know if you've reused it.

So if you are re-using values, then freeing when the handler gets called is the wrong behavior; you might use an atomic reference count in this situation (with atomic_fetch_add()), bumping the count when inserting, lowering when the handler is called, and only freeing if it gets to 0.

Or, just wait till the table is torn down.

If you don't register an ejection handler, it won't try to free the cells.

As for reusing keys, it's the exact same mechanism-- if you want to be able to reuse them you can. If you don't want to ever free anything during the lifetime of the table, don't worry about registering a handle. But if you do want to free things, then you need to manage how to deal with items that might be duplicates.

[mallman]

Somehow I didn't notice the registration of the ejection handler. This makes sense now.

Obviously, this is not the exact code that led me to file this bug in the first place. If I come across the code that I was having trouble with, I will re-examine it. It's possible I was misusing the API.

Thanks for your help.

[mallman]

BTW, I found out why I was unable to reproduce the issue yesterday. I had commented out the code in hatrack that frees the memory. After undoing that change, I was able to reliably get the double free error.

[viega]

Thanks, if you do have any issues, just let me know. I have made sure this repo gets flagged, so I shouldn't miss it in the future.

[mallman]

Hi John. I found the original code that led me to file this issue, and it's still crashing even though I don't think it should be. I'm going to re-open this ticket, though we may end up with a different one.

I will look at how I can provide a test case that more accurately reflects the problem I'm still having. In the meantime, though, I'll mention the exact place where my code is crashing is at line 625 in

hatrack/src/hash/dict.c

Lines 617 to 628 in 1c66f8b

	static void
	hatrack_dict_record_eject(hatrack_dict_item_t *record,
	hatrack_dict_t *dict)
	{
	hatrack_mem_hook_t callback;
	callback = dict->free_handler;
	(*callback)(dict, record);
	return;
	}

The problem is that callback is NULL. This is irrespective of whether I assign a free handler or not. The variable dict in this context is not the same as the dict I'm allocating from my program.

I would have to dig deeper to figure out what dict in that function actually is, but can you have a look at that code and verify that dict->free_handler should be non-null as that code assumes?

[viega]

Sounds good, I will take a look (tho maybe not till tonight or in the morning); I assume if you do register a callback it's fine? The callback doesn't need to free anything...

I thought I'd made the callback optional, but my memory could also be hazy, I'm getting older :)

[mallman]

Sounds good, I will take a look (tho maybe not till tonight or in the morning); I assume if you do register a callback it's fine? The callback doesn't need to free anything...

I am setting a callback, yes. As I mentioned though, in the function I referenced above, dict is not the same as the dict my code inits, and dict->free_handler is NULL for some reason... 🤔

I thought I'd made the callback optional, but my memory could also be hazy, I'm getting older :)

Join the club. (BTW, I appreciate the reference to Zork.)

Here's a complete code sample that crashes. It's based on basic.c, but pared down:

#include <hatrack.h>
#include <string.h>
#include <stdio.h>
#include <stdbool.h>

static void
envp_free_handler(hatrack_dict_t *unused, hatrack_dict_item_t *item)
{
    void *key = item->key;
    return;
}

int
main(int argc, char *argv[], char *envp[])
{
    hatrack_dict_t *envp_dict;
    uint64_t        i;
    char           *env_eq; // Pointer to the equals mark.
    char           *env_key;
    char           *env_val;
    char           *p;

    envp_dict = hatrack_dict_new(HATRACK_DICT_KEY_TYPE_CSTR);

    hatrack_dict_set_free_handler(envp_dict,
                                  (hatrack_mem_hook_t)envp_free_handler);

    i = 0;

    while (envp[i]) {
        p       = envp[i];
        env_eq  = strchr(p, '=');
        env_key = strndup(p, env_eq - p);
        env_val = strdup(env_eq + 1);

        hatrack_dict_put(envp_dict, env_key, env_val);
        hatrack_dict_remove(envp_dict, env_key);

        i++;
    }

    hatrack_dict_delete(envp_dict);

    return 0;
}

There are two things in particular I want to point out in this example. First, I'm setting a free handler, but it doesn't actually free anything. However, it does access item. Second, I'm removing the key after I put it.

From what I understand about the API, this should be safe. My code isn't freeing env_key anywhere, including in the free handler. So my expectation is that hatrack shouldn't free env_key either. Further, I'm assuming that it's safe to reference item in the free handler.

(Some detailed API docs would be very welcome, BTW. 😄)

I appreciate your assistance.

[viega]

This is useful, thank you! I'll get to it by morning. I definitely didn't quite finish things up before going back to work, and that includes documentation :( In addition to docs, I really should strip everything down to what the core library should be, as opposed to the comparative algorithms, and rm the stuff that never got finished. I'll try to get to that stuff too. I probably will be using this library in a new project again soon, so I'll keep this in mind.

…

On Mon, Oct 2, 2023 at 1:24 PM Michael Allman ***@***.***> wrote: Sounds good, I will take a look (tho maybe not till tonight or in the morning); I assume if you do register a callback it's fine? The callback doesn't need to free anything... I am setting a callback, yes. As I mentioned though, in the function I referenced above, dict is not the same as the dict my code inits, and dict->free_handler is NULL for some reason... 🤔 I thought I'd made the callback optional, but my memory could also be hazy, I'm getting older :) Join the club. (BTW, I appreciate the reference to Zork.) Here's a complete code sample that crashes. It's based on basic.c, but pared down: #include <hatrack.h>#include <string.h>#include <stdio.h>#include <stdbool.h> static voidenvp_free_handler(hatrack_dict_t *unused, hatrack_dict_item_t *item) { void *key = item->key; return; } intmain(int argc, char *argv[], char *envp[]) { hatrack_dict_t *envp_dict; uint64_t i; char *env_eq; // Pointer to the equals mark. char *env_key; char *env_val; char *p; envp_dict = hatrack_dict_new(HATRACK_DICT_KEY_TYPE_CSTR); hatrack_dict_set_free_handler(envp_dict, (hatrack_mem_hook_t)envp_free_handler); i = 0; while (envp[i]) { p = envp[i]; env_eq = strchr(p, '='); env_key = strndup(p, env_eq - p); env_val = strdup(env_eq + 1); hatrack_dict_put(envp_dict, env_key, env_val); hatrack_dict_remove(envp_dict, env_key); i++; } hatrack_dict_delete(envp_dict); return 0; } There are two things in particular I want to point out in this example. First, I'm setting a free handler, but it doesn't actually free anything. However, it does access item. Second, I'm removing the key after I put it. From what I understand about the API, this should be safe. My code isn't freeing env_key anywhere, including in the free handler. So my expectation is that hatrack shouldn't free env_key either. Further, I'm assuming that it's safe to reference item in the free handler. (Some detailed API docs would be very welcome, BTW. 😄) I appreciate your assistance. — Reply to this email directly, view it on GitHub <#8 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABELGQJDZ4QNFKJENF2KLOLX5L2DLAVCNFSM6AAAAAAZNVRCXGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONBTGQ2DIMJYGA> . You are receiving this because you commented.Message ID: ***@***.***>

[mallman]

Thanks. It's no rush, really. I'm just on a roll now with this hatrack stuff, but I need to put my attention elsewhere, too.

[viega]

Okay, I might take another couple days then, it's turning out to be a busy week :)

[mallman]

Okay, I might take another couple days then, it's turning out to be a busy week :)

Hi @viega. I thought I'd just follow up with you to see how you're feeling about this project.

From my experience, hatrack is wonderful software. Its lack of API documentation makes its adoption more challenging than it need be. Adopters like me will struggle with problems that may come down to misunderstandings about how to use the API. Can you devote some time to shore up the documentation? It would help me debug my own issue here and decide whether this is an issue of misusing the API or an actual bug in hatrack.

Cheers.

(BTW, I'm using hatrack from a Swift project. Bet you never expected a Swift programmer to use hatrack.)