NTDS.dit Plugin

[B0TAxy]

Description

This merge request introduces an initial implementation of an NTDS.dit Plugin.
The parser can parse and decrypt secret records (e.g., user hashes, supplemental credentials) from an Active Directory NTDS database.

At this stage, the parser is in a late development preview:

Core functionality for parsing and decrypting secret records is available.

Support for additional record types (e.g., groups, ACLs, domain data) is not yet implemented.

Output support for BloodHound-compatible JSON is planned.

Automated tests have not been added yet — these will be included in a follow-up update.

The goal of creating this MR now is to gather early feedback and design review before finalizing the remaining features.

Checklist

• Implementation (core)
• Tests
• BloodHound export
• Full record support

[joost-j]

Regarding this PR, I've managed to get it to work with my own PR (fox-it/dissect.database#8) by changing a few things:

In the __init__():

ntds_path_key = target.registry.value(
    key="HKLM\\SYSTEM\\CurrentControlSet\\Services\\NTDS\\Parameters", 
    value="DSA Database file"
)
self.ntds = NTDS(target.fs.open(ntds_path_key.value))

Changing all the record[NAME_TO_INTERNAL["XXX"]] references to record.XXX where XXX refers to the official ldapDisplayName, since these names are already automatically converted. Lastly, by changing the database data collection part:

def _collect_pek_and_user_records(self) -> tuple[bytes, Generator[User]]:
    return next(self.ntds.lookup(objectCategory="domainDNS")).pekList, self.ntds.users()

and some other minor adjustments. Even on a small dataset that already seemed to result in a performance boost, since the NTDS class uses indexes to look for the pekList and User records.

Note: not all accounts seem to have entries for dBCSPwd, ntPwdHistory, lmPwdHistory and even unicodePwd, so that probably needs some try/exception wrapping.

Please let me know what you think of this and if you need any help with refactoring & editing the code. You can also DM me on the Dissect discord

[B0TAxy]

Hi @joost-j ,
Thanks for the suggestion! I’ve converted the PR to use your approach, and it’s working well with the changes you mentioned. I’ve also wrapped the optional attributes (dBCSPwd, ntPwdHistory, lmPwdHistory, unicodePwd etc) as you noted. Really appreciate the guidance — the adjustments to record.XXX and the NTDS data collection logic make the code cleaner and improve performance.

[joost-j]

Cool! Good to hear - a quick check from my side also confirms that it works well. When running your ntds.user_accounts plugin I can see the NTDS User class attributes, as well as the hashes, really nice! Hoping to wrap up the NTDS parser this year, so that this PR can be finished as well. Just a small remark; I did need to change the from Cryptodome.Cipher on line 9 to from Crypto.Cipher.

[B0TAxy]

Cool! Good to hear - a quick check from my side also confirms that it works well. When running your ntds.user_accounts plugin I can see the NTDS User class attributes, as well as the hashes, really nice! Hoping to wrap up the NTDS parser this year, so that this PR can be finished as well. Just a small remark; I did need to change the from Cryptodome.Cipher on line 9 to from Crypto.Cipher.

Hi @joost-j , Thanks for the insight! I’ve fixed that import as well.

By the way, do you plan to test your database parser against any older domain controllers? The extraction seems to be a bit different — especially the hash extraction — and I currently don’t have good test data from older DC versions to compare with.

[Schamper]

By the way, do you plan to test your database parser against any older domain controllers? The extraction seems to be a bit different — especially the hash extraction — and I currently don’t have good test data from older DC versions to compare with.

That is the goal, but we will incrementally improve the parser.

[codspeed-hq]

CodSpeed Performance Report

Merging #1347 will not alter performance

_{Comparing B0TAxy:feautre/ntds_dit_parsing (d3d658b) with main (1337e8a)}

Summary

✅ 9 untouched

[B0TAxy]

By the way, do you plan to test your database parser against any older domain controllers? The extraction seems to be a bit different — especially the hash extraction — and I currently don’t have good test data from older DC versions to compare with.

That is the goal, but we will incrementally improve the parser.

Makes sense. Once you feel the parser is ready for those older versions, could you let me know? I’d be happy to run a few tests on my end and give you some feedback on how it handles the extraction.

[joost-j]

PR fox-it/dissect.database#8 is now ready to be reviewed, and probably will be reviewed somewhere in the upcoming weeks. The PR now also includes most of the decryption routines for PEK lists etc. Once that PR is merged, you can edit your plugin once more to focus mainly on the plugin functionality and record publishing parts, leaving out most of the internals because you can now mostly wrap around those functions from dissect.database. I'll ping you once the PR has been merged!

[Schamper]

I’d be happy to run a few tests on my end and give you some feedback on how it handles the extraction.

If you have any test sets of old domains that are suitable for open-source sharing, that would be greatly appreciated!

[B0TAxy]

PR fox-it/dissect.database#8 is now ready to be reviewed, and probably will be reviewed somewhere in the upcoming weeks. The PR now also includes most of the decryption routines for PEK lists etc. Once that PR is merged, you can edit your plugin once more to focus mainly on the plugin functionality and record publishing parts, leaving out most of the internals because you can now mostly wrap around those functions from dissect.database. I'll ping you once the PR has been merged!

Thanks for the heads-up! I’ve already converted the plugin to use the new functions from your PR, and the code is definitely much cleaner now.

However, I’ve run into a couple of issues with the new routines. First, the hash decryption doesn't seem to be working—it’s returning the same buffer instead of the decrypted result. Second, the tool is now crashing when I try to decrypt the hash for the krbtgt account. Upon early investigation, the buffer I'm getting from unicodePwd for that account is significantly smaller than it used to be.

I'll keep digging into it, but let me know if there are any specific nuances in the new dissect.database functions I should be aware of regarding these cases!

[B0TAxy]

I’d be happy to run a few tests on my end and give you some feedback on how it handles the extraction.

If you have any test sets of old domains that are suitable for open-source sharing, that would be greatly appreciated!

Unfortunately, I don't currently have any test data from old domains that would be suitable for open-source sharing.

[Schamper]

@B0TAxy you need to “unlock” the PEK first now (ntds.pek.unlock(syskey)). Second, it only does the RC4/AES decryption. I meant for the DES decryption to be in dissect.target and re-used from the SAM plugin (it’s currently embedded inside decrypt_single_hash, but it should be separated to a des_decrypt).

If after that you still have some issues, please let me know and if possible provide some test data (PEK, syskey and encrypted blobs).

[B0TAxy]

@B0TAxy you need to “unlock” the PEK first now (ntds.pek.unlock(syskey)). Second, it only does the RC4/AES decryption. I meant for the DES decryption to be in dissect.target and re-used from the SAM plugin (it’s currently embedded inside decrypt_single_hash, but it should be separated to a des_decrypt).

If after that you still have some issues, please let me know and if possible provide some test data (PEK, syskey and encrypted blobs).

Yes, I'm aware of the issue—I had some trouble with Git and it didn't push the new code as expected. That’s fixed now and the latest version is pushed. However, I’m still running into the problem with the krbtgt decryption.

[B0TAxy]

@B0TAxy you need to “unlock” the PEK first now (ntds.pek.unlock(syskey)). Second, it only does the RC4/AES decryption. I meant for the DES decryption to be in dissect.target and re-used from the SAM plugin (it’s currently embedded inside decrypt_single_hash, but it should be separated to a des_decrypt).
If after that you still have some issues, please let me know and if possible provide some test data (PEK, syskey and encrypted blobs).

Yes, I'm aware of the issue—I had some trouble with Git and it didn't push the new code as expected. That’s fixed now and the latest version is pushed. However, I’m still running into the problem with the krbtgt decryption.

For test data, you can use the updated plugin along with the SYSTEM hive and the NTDS.dit file from dissect.database

[B0TAxy]

@B0TAxy you need to “unlock” the PEK first now (ntds.pek.unlock(syskey)). Second, it only does the RC4/AES decryption. I meant for the DES decryption to be in dissect.target and re-used from the SAM plugin (it’s currently embedded inside decrypt_single_hash, but it should be separated to a des_decrypt).

If after that you still have some issues, please let me know and if possible provide some test data (PEK, syskey and encrypted blobs).

Regarding the decryption, I noticed the output buffer remains unchanged, so I've left the function as-is for now. I'll merge it once I can re-test the plugin.

[Schamper]

Can you add the NTDS.dit file you're using as a test file under tests/_data/plugins/os/windows/ad/ntds.dit? GitHub does not allow LFS uploading to forks for maintainers (us).

That way, we can add a unit test. Maybe use the GOAD one as that one has credential history?

[B0TAxy]

Can you add the NTDS.dit file you're using as a test file under tests/_data/plugins/os/windows/ad/ntds.dit? GitHub does not allow LFS uploading to forks for maintainers (us).

That way, we can add a unit test. Maybe use the GOAD one as that one has credential history?

That’s a great idea. I’m adding both the GOAD data and my previous test samples to the repo so we can validate the logic against multiple sources and ensure coverage for credential history.

I'm also finishing up the shared DES function now to ensure the SAM and NTDS plugins remain consistent. I'll update the PR as soon as the tests are ready.

[B0TAxy]

Hello, apologies for the delay. I encountered several challenges with the tests, but I have made progress nonetheless. Here is a summary of the status and open items:

1. Fixture Mapping: I initially had trouble getting the fixture to work directly with fs_win (persisting "file not found" errors). I have currently worked around this by mapping through the target, but we may want to review this approach.

2. Test Data Needed:

• I need the known/ground truth data for the goad ntds.dit file to verify that my parser is extracting the correct information.
• I also need the known/ground truth data for Computer Accounts for both the large and goad datasets.

3. Bugs Requiring Review:

• Hash History Issue: There appears to be a bug in the hash history extraction. Occasionally, a hash is longer than 16 bytes, causing the DES removal step to fail. I have added a TODO and temporarily set it to return b"" to allow the rest of the code to run.
• member_of Exceptions: The member_of extraction is throwing multiple exceptions. I have added a temporary catch-all block and a TODO there as well. I would appreciate clarification on whether this behavior is intended or if it is a bug.

Could you please look into both the hash history length issue and the member_of exceptions? I would really appreciate your help debugging both of these specific issues. If you do not have time, let me know, and I can investigate further next week.

[Schamper]

I can take a look at the fixtures and hash length issue. What exactly do you mean with ground truth data? What would you need?

As for the exception, probably this is because I currently chose to throw an exception when trying to list the groups/members of objects that have been replicated from another DC (phantom objects), as that NTDS won't have that information available. My reasoning was that returning an empty list is not really accurate, since they probably do have groups/members, we just can't know about it. I wanted to experiment with how that would work out in dissect.target, if the exception is annoying (vs a boolean check on is_phantom), we can change that behavior.

[B0TAxy]

I can take a look at the fixtures and hash length issue. What exactly do you mean with ground truth data? What would you need?

As for the exception, probably this is because I currently chose to throw an exception when trying to list the groups/members of objects that have been replicated from another DC (phantom objects), as that NTDS won't have that information available. My reasoning was that returning an empty list is not really accurate, since they probably do have groups/members, we just can't know about it. I wanted to experiment with how that would work out in dissect.target, if the exception is annoying (vs a boolean check on is_phantom), we can change that behavior.

Thanks for looking into the fixtures and hash length!
Regarding ground truth data: I meant that I need the expected cleartext passwords (or target hashes) to verify my parsing logic. For example, @joost-j provided these credentials:

• henk.devries: Winter2025!
• beau.terham: Zomer2027!

Knowing these allows me to confirm that I am extracting the correct hash.
As for the exception: The logic regarding phantom objects makes sense and isn't annoying. However, the error I encountered is actually different. I checked, and it seems to be an IndexError (among others) originating in the backlinks function.

[Schamper]

Except for the changed passwords (which I unfortunately don't remember what I changed them too :)) all data is the same as here: https://github.com/Orange-Cyberdefense/GOAD/blob/main/ad/GOAD/data/config.json

[Schamper]

However, the error I encountered is actually different. I checked, and it seems to be an IndexError (among others) originating in the backlinks function.

I looked into this, and I believe this is a side-effect of the large NTDS.dit probably being "dirty" (can you confirm @joost-j?). Probably this could be fixed if we implement transaction files: fox-it/dissect.database#18

[Schamper]

I looked into this, and I believe this is a side-effect of the large NTDS.dit probably being "dirty" (can you confirm @joost-j?). Probably this could be fixed if we implement transaction files: fox-it/dissect.database#18

We confirmed this is the case and updated the test data in dissect.database. It's now merged too! 🥳 If you could update the file in this PR that'd be appreciated.