Initial commit for plugin internals refactor

[Schamper]

Initial commit for the plugin internals refactor. This removes a lot of slow and hard to understand code with (hopefully) faster and easier to understand code. Also fixes some consistency issues I came across while working on this.

There are some things I want to add, but I may do those in a separate PR:

• Re-add support for Nested namespaces do not work #758 (it was lost in between version 2 and 3)
• Add support for "plugin directories", similar to _os.py, but then _plugin.py or something similar (Add support for plugin directories #788)
• Add helper functions for getting runtime information from functions, e.g. the class object, the record descriptor objects, etc. This will finally allow us to query plugins by e.g. ones that generate records that have a path field, for example. (Add retrieval of runtime information to plugin descriptors #1007)

This should also cover #759.

Closes #889.

[codecov]

Codecov Report

Attention: Patch coverage is 88.29201% with 85 lines in your changes missing coverage. Please review.

Project coverage is 76.83%. Comparing base (f2f50a5) to head (552933d).
Report is 14 commits behind head on main.

Files with missing lines	Patch %	Lines
dissect/target/plugin.py	91.64%	38 Missing ⚠️
dissect/target/target.py	70.31%	19 Missing ⚠️
dissect/target/tools/query.py	77.35%	12 Missing ⚠️
dissect/target/plugins/general/plugins.py	88.00%	6 Missing ⚠️
dissect/target/helpers/docs.py	88.09%	5 Missing ⚠️
dissect/target/tools/utils.py	81.25%	3 Missing ⚠️
dissect/target/plugins/os/windows/lnk.py	0.00%	1 Missing ⚠️
dissect/target/plugins/os/windows/registry.py	50.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #763      +/-   ##
==========================================
- Coverage   77.72%   76.83%   -0.90%     
==========================================
  Files         326      327       +1     
  Lines       28575    28645      +70     
==========================================
- Hits        22210    22009     -201     
- Misses       6365     6636     +271     

Flag	Coverage Δ
unittests	76.83% <88.29%> (-0.90%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[pyrco]

I noticed plugins/general/example.py and plugins/os/unix/packagemanager.py still have a __findable__ property, which is now no longer needed.

Another thing I noticed is that the -l no longer lists the plugins from _os.py.

And a third thing I noticed that if I run a namespace plugin, e.g. webserver.access, which is listed when doing target-query MyTarget -l, before it printed an error like:

Error while parsing sysvol/windows/system32/inetsrv/config/applicationHost.config: ...

but now it errors with:

target-query: error: argument -f/--function contains invalid plugin(s): webserver.access

[Schamper]

I noticed plugins/general/example.py and plugins/os/unix/packagemanager.py still have a __findable__ property, which is now no longer needed.

Fixed package manager but we still don't want the example plugin to findable.

Another thing I noticed is that the -l no longer lists the plugins from _os.py.

Should be fixed again.

And a third thing I noticed that if I run a namespace plugin, e.g. webserver.access, which is listed when doing target-query MyTarget -l, before it printed an error like:

Error while parsing sysvol/windows/system32/inetsrv/config/applicationHost.config: ...

but now it errors with:

target-query: error: argument -f/--function contains invalid plugin(s): webserver.access

I had to largely rewrite the NamespacePlugin to properly behave without any of its old side effects, but this is now also fixed.

[twiggler]

Nice increment.

Left some comments, most are suggestions / ideas, not demands.

QA: did not find regressions but will test again after changes.

Processed

[JSCU-CNI]

Would it be possible to add registered arguments to generate_functions_json?

diff --git a/dissect/target/plugins/general/plugins.py b/dissect/target/plugins/general/plugins.py
index 334830b..6dd5ed9 100644
--- a/dissect/target/plugins/general/plugins.py
+++ b/dissect/target/plugins/general/plugins.py
@@ -59,14 +59,25 @@ def generate_functions_json(functions: list[plugin.FunctionDescriptor] | None =
 
     for desc in functions or _get_default_functions():
         plugincls = plugin.load(desc)
-        docstring = getattr(plugincls, desc.method_name).__doc__
+        func = getattr(plugincls, desc.method_name)
 
         loaded.append(
             {
                 "name": desc.name,
                 "output": desc.output,
-                "description": docstring.split("\n\n", 1)[0].strip() if docstring else None,
+                "description": func.__doc__.split("\n\n", 1)[0].strip() if func.__doc__ else None,
                 "path": desc.path,
+                "arguments": [
+                    {
+                        "name": name[0],
+                        "type": arg.get("type").__name__ if arg.get("type") is not None else None,
+                        "help": arg.get("help"),
+                        "default": arg.get("default"),
+                    }
+                    for name, arg in func.__args__
+                ]
+                if hasattr(func, "__args__")
+                else None,
             }
         )

[JSCU-CNI]

Additionally could FunctionDescriptor.record be populated with __record__ as discussed in #759?

[Schamper]

Additionally could FunctionDescriptor.record be populated with __record__ as discussed in #759?

This is not possible since we can't serialise the record descriptor, so it would just be a string. This functionality will be part of an incremental improvement of this PR as discussed in the main PR body:

• Add helper functions for getting runtime information from functions, e.g. the class object, the record descriptor objects, etc. This will finally allow us to query plugins by e.g. ones that generate records that have a path field, for example.

The goal of this PR is to agree and establish a better base plugin system, and then incrementally improve that.

[twiggler]

Code looks good.

Some tests are failing though.

After these have been fixed, I will do another QA pass.