commandhistory and LDAP users

[liontruck]

On Linux, commandhistory does not seem to return shell history for LDAP users, which may be because those users are not present in /etc/passwd. I wanted to ask whether this is the intended behavior, and if there has been any consideration of falling back to enumerating /home/* (or similar paths) to locate shell history files.

Thank you!

[Schamper]

We added basic support for that here: #1207

However that only walks a single directory deep in /home. Are your LDAP users home directory one level deeper? E.g. /home/mydomain.local/some.user?

[liontruck]

All the users have home directories directly under /home (e.g. /home/someuser1, /home/someuser2), so they aren't nested deeper. Even so, commandhistory only seems to grab shell history for users present in /etc/passwd. LDAP users with valid home directories under /home aren't picked up.

[lhaagsma]

Had a brief look at this. tldr: seems like a bug/undesired behavior, potentially for any plugin that makes use of the all_with_home function.

Could not easily find a test that checks on Linux for /home/<user>/ where <user> is not in /etc/passwd. But did not look too hard.

In the code there might be an issue when I created a local testscenario.

At misc_user_paths entries in /home/ are yielded as (entry, None), None being the user details (later referred to as user_criterion:

dissect.target/dissect/target/plugins/os/unix/_os.py

Line 170 in 3c0d8ef

yield from ((entry, None) for entry in home_path.iterdir() if entry.is_dir())

But in all_with_home:

dissect.target/dissect/target/plugins/general/users.py

Line 102 in 3c0d8ef

if any(seen_path.samefile(misc_home_dir) for seen_path in seen) or not user_criterion:

any results with user_criterion=None is not yielded.

However the user_criterion is always None as returned by the misc_user_path implementation.

Plugin's such as commandhistory all use all_with_home:

dissect.target/dissect/target/plugins/os/unix/history.py

Line 63 in 3c0d8ef

for user_details in self.target.user_details.all_with_home():

I locally tested the following hacky workaround by changingplugins/general/users.py UserPlugin within all_with_home. It does not continue when user_criterion is None, and it will yield a UserDetails record with an user="" value:

        for misc_home_dir, user_criterion in self.target.misc_user_paths():
            # We skip the entry if no user can be found.
            # We cannot use set membership check here because of https://github.com/fox-it/dissect.target/issues/1203

            #if any(seen_path.samefile(misc_home_dir) for seen_path in seen) or not user_criterion:
            if any(seen_path.samefile(misc_home_dir) for seen_path in seen):
                continue

            # check if user_criterion is set.
            if user_criterion:
                if (user_details := self.find(**{user_criterion[0]: user_criterion[1]})) is None:
                    continue
                yield UserDetails(user=user_details.user, home_path=misc_home_dir)
                seen.add(misc_home_dir)
            else:
                # When no user_criterion is there just yield with empty user.
                yield UserDetails(user="", home_path=misc_home_dir)
                seen.add(misc_home_dir)

This yield records with empty user names:
<unix/history hostname='user-VMware20-1' domain=None ts=None order=2 command='ls -al' shell='bash' source='/home/dissect/.bash_history' username=None user_id=None user_group=None user_home=None>

[Schamper]

@twiggler what was your intended way for #1207 to be used in these cases?

[twiggler]

@Schamper Reconstructing from the PR description and the comment # We skip the entry if no user can be found.,

the intended way for plugins to iterate user_directories without user details is to use all_home_dirs.

I think it was designed that way because plugins expect the user part of UserDetails to be there when using all_with_home. (see

dissect.target/dissect/target/plugins/general/users.py

Lines 25 to 27 in 04720d9

	class UserDetails(NamedTuple):
	user: UserRecord
	home_path: TargetPath | None

, it is non nullable).@lhaagsma fix invalidates that invariant and potentially would result in regressions.

Moving forward, it appears that the logic of the command history plugin does not depend on user information, but only forwards the user information. For these cases, perhaps we can add a function which behaves like all_with_home except that the user details are optional. Plugins which just forward user information if available could opt to use that function.

We could also break things and rethink how users and home directories are related. Arguably, iterating over the intersection of directories and users is forensically unsound. We could do something like:

# Conceptual Fix
for home_path in self.target.home_paths # or profile in self.target.profiles
    # Try to bind a user, but proceed even if it fails
    user = self.target.user_details.find_owner(home_path) 
    
    yield ShellHistory(
        path=home_path,
        user=user, # Allow this to be None
        ...
    )

[Schamper]

We could also break things and rethink how users and home directories are related.

I'm generally in favor of this, since I think the current way we do users and home paths is a little annoying and has too much repeating boilerplate for every time you want to do something with users/home paths. The current implementation of user_details is very tacked on and never really properly revised. A lot of the user iterating logic stems from very early Dissect days where we just wanted quick and dirty solutions (and didn't have fancy things like SAM parsing yet).

I think in general the most important information is the tuple (user details | None, home path | None). Almost all plugins that do anything with users and/or home paths expect this tuple in some way. We should focus any new solution around this.

For specifically Windows, I think we should:

• Take SAM as leading source of truth for local users. We have a proper username and a SID. Resolve home path using SID matching in ProfileList (Add sam.groups plugin and reworking of t.users (feedback required) #1478 is a start for this)
• ProfileList is the second layer, where we can have "inferred" SID (from the registry key) and username (stem of the path)
• Iterating C:\Users is the third layer, where we have no SID, but we infer the username from the stem of the path

For Unix, it's similar:

• Take /etc/passwd as leading source of truth for local users. We have a proper username, uid and home path
• Iterating /home is the second layer, where we have no uid, but we infer the username from the stem of the path

So importantly, we can sometimes have no user details, but sometimes have partial (inferred) user details.

Brainfart: we could make t.users a namespace, where t.users() just yields the users, and further methods are t.users.all_with_*.

[twiggler]

With regard to the interface:

Brainfart: we could make t.users a namespace, where t.users() just yields the users, and further methods are t.users.all_with_*.

I think this is the right direction. It significantly improves DX by allowing us to use specific return types for specific methods, rather than forcing a generic Optional check on every plugin. If a method guarantees a user, the return type should reflect that.

However, we should distinguish the "driving entity" of the iteration:

• User-driven: t.users.all_with_home() implies we are iterating Users and filtering for those with homes. Here, user is non-null.
• Path-driven: The commandhistory case is actually iterating Home Directories and looking for owner metadata. Here, home_path is the non-null driving entity, while user is effectively an optional annotation.

If we treat everything as t.users..., we might be forcing a user-centric view on what is sometimes a filesystem-centric task.

I think in general the most important information is the tuple (user details | None, home path | None). Almost all plugins that do anything with users and/or home paths expect this tuple in some way.

I agree that the tuple is the practical unit of work, but I see a risk of confusion in the current data structure.
The user field in UserDetails already contains a home field

dissect.target/dissect/target/plugins/general/users.py

Lines 25 to 27 in 6b7fbf8

	class UserDetails(NamedTuple):
	user: UserRecord
	home_path: TargetPath | None

dissect.target/dissect/target/helpers/record.py

Lines 134 to 143 in 6b7fbf8

	COMMON_UNIX_FIELDS = [
	("string", "name"),
	("string", "passwd"),
	("varint", "uid"),
	("varint", "gid"),
	("string", "gecos"),
	("path", "home"),
	("string", "shell"),
	("string", "source"),
	]

If we pass a tuple like (user_details, home_path), we effectively have two "homes":

1. user_details.home: The configured home (string).
2. home_path: The resolved home (Path object).

If these differ, it creates ambiguity on which one the plugin should trust.

With regard to the implementation:

The layered approach seems sensible. I will take a more in-depth look at the details of #1478 later.

[Schamper]

If we treat everything as t.users..., we might be forcing a user-centric view on what is sometimes a filesystem-centric task.

This is partially semantics though. From a tool user perspective, it may not be a filesystem-centric task (as is the case in this very ticket, for example). While from a parsing perspective there's no association with a known user, the tool user expects these directories to be parsed as "user directories" and, in some cases, to have partial information inferred (such as the username, based on the stem of the home directory).

My primary question/concern would be whether this distinction is big enough to warrant its own separate namespace, or if we should make the distinction clear in docstring.

[twiggler]

Interesting.
I think semantics are crucial here because they define the contract for the plugin author. What I meant with filesystem-centric task is that the task is driven by history files on disks, not user accounts.

Having two functions allows the developer to choose the "driving entity":

def all_users_with_home_paths : Iterator[(User, HomePath | None)]

def all_home_paths_with_user : Iterator[(User | None, HomePath)]

# or def all_home_paths_with_user : Iterator[(HomePath, User | None)] # best to make it named

seems fine to me, even if they reside within the users namespace. The important part is that the return types explicitly tell the developer which part of the tuple is safe to access without checking for None.

Thus in this case, all_home_paths_with_user is the function we want.

[twiggler]

On the other hand, perhaps User is never null because (as you proposed above) the user name is at worst inferred from the home_path.

I am a bit on the fence if that is a good idea; it might give suggest false confidence.
At the minimum, it might be a good idea to add a source field to User indicating the origin of the data.

[lhaagsma]

At least the commandhistory does have a source field to indicate the source file of the .bash_history commands:
source='/home/dissect/.bash_history'

I have not strong opinion on the namespace matter. But grouping things together might make this easier/clearer.

Regarding the fact that users could be None perhaps a user_origin field is valuable, and set that to sam, profile_path or etc/passwd and and the likes to provide guidance.

For me I rather have more output (additional bash_history lines) and deal with the who is the related used account later than miss out on entries like in the current implementation because we are uncertain what user account is associated.

[twiggler]

At least the commandhistory does have a source field to indicate the source file of the .bash_history commands:
source='/home/dissect/.bash_history'

Good observation. In this specific case, a User object with a name inferred from the home directory doesn't add much value beyond what the source field already tells us.

That being said,

a user_origin field is valuable, and set that to sam, profile_path or etc/passwd and and the likes to provide guidance.

I agree. This effectively allows the User object to act as a Null Object pattern, as Schamper suggested.

I reviewed the usages of all_with_home, and most plugins only use the home_path or simply "forward" the user record without inspecting it. For the few that do computation, they already have to perform null checks (e.g., checking if sid exists). So, in practice, adopting an "Inferred User" model doesn't significantly degrade the developer experience—the null checks just shift from "is user None?" to "is user.field None?".

For me I rather have more output (additional bash_history lines) and deal with the who is the related used account later than miss out on entries like in the current implementation because we are uncertain what user account is associated.

I think we all agree on that, the current implementation is arguably a straw man.

While it feels theoretically unsound to shoehorn multiple types of users (Configured Users vs. Inferred/Zombie Users) into a single type, the addition of a mandatory source / origin field effectively turns the User object into a Tagged Union.

As long as we enforce that source field, I am comfortable moving forward with the "Inferred User" approach. It solves the immediate data loss issue while keeping the API signature simple. Perhaps having Bags of Maybes comes with the domain of forensics. And Python.