Singularity rootkit

[solardiz]

Singularity is a new Open Source rootkit by @MatheuZSecurity (thank you, Matheu!), which has recently added an LKRG bypass. This "issue" is to formulate our stance on the topic, including how it fits our threat model (or does not), and what we're doing about it.

1. LKRG bypasses from kernel vulnerability exploits are more relevant for us and for LKRG users than bypasses from rootkits, since in that case the attacker doesn't yet have legitimate-looking root privileges at the initial stage. This is something to recognize and defend against with our current threat model. But in the case of rootkits, there isn't yet:
2. Currently, nothing prevents legitimate root from simply unloading LKRG with rmmod or effectively disabling it via sysctl before loading the rootkit. Therefore, an LKM rootkit bypassing LKRG, and us avoiding or breaking such bypass, will only make practical sense if and when we add protection against unloading LKRG and modifying its configuration LKRG Lockdown #439 @kcdev809. And even then, the question will be about protecting the entire system boot chain, including userland; otherwise, the rootkit could be loaded by modifying files used during the next boot-up. That is, Secure Boot, signatures on everything before LKRG is loaded and on LKRG itself (implemented in RLC-H from CIQ), and kernel lockdown. In such system configuration, an LKM rootkit that lacks a valid signature could not be loaded without exploiting a vulnerability even in absence of LKRG. So what LKRG could add on top is mitigation of kernel vulnerability exploits - the same thing that's already our focus - and then post-detection of the rootkit's dirty hacks if it does get loaded anyway.
3. LKRG monitors the integrity and behavior of the kernel, which often detects rootkits - for example, 8 out of 9 in one study. However, we intentionally allow kernel modules loaded by legitimate root user (not with root access just obtained through a kernel vulnerability) and using only official APIs. We even have additional code that intentionally allows kernel function hooking using ftrace (which Singularity uses). We try not to break useful functionality. Therefore, writing a rootkit that we don't detect can be done simply by making it as correct as possible, without hacks (perhaps giving up on hiding). I think this is why 1 out of 9 wasn't detected in that old study - it was just a keylogger, not a full-fledged rootkit, which was more likely implemented without hacks.
4. Nevertheless, Singularity (or some of its behavior) was apparently initially detected by LKRG. Hence the additional bypass code, added in a more recent commit after the project's initial announcement. In a sense, it's a success for LKRG that a rootkit written without regard for LKRG was detected, even if this did not really matter yet.
5. As far as we can judge from the bypass code, it won't work if the sysctl lkrg.hide=1 is enabled. Incidentally, this same setting also broke the bypass in @a13xp0p0v's exploit. We're once again considering making this option, or part of its effect, enabled by default.
6. We also remembered that notrace should be added to more functions in LKRG Add 'notrace' and 'static' attribute to the most of our internal functions #21 or enabled globally via build options Remove trace-related CFLAGS from the build #454 @kerneltoast. This should break ftrace for intercepting LKRG functions by the rootkit, which is what Singularity does. Incidentally, I see that Singularity itself uses notrace on its functions - I wouldn't be surprised if that's by our example; we just had not deployed ours widely enough.
7. We're also revisiting whether we really want to support the use of ftrace, especially without a way to disable such support. Perhaps we only need to remove or disable part of our code, and Singularity will break (as well as other uses of ftrace).

For now, we already proceeded to address item 6 above, via PR #454. So we probably already broke Singularity's bypass as of today - need to test.

[solardiz]

OK, testing this on Debian 13, I am able to load Singularity on top of LKRG, and Singularity is not immediately detected, and its hiding of itself from lsmod (shouldn't we detect that?) and hiding of directories works(*). However, LKRG remains functional, so when I try to actively use the rootkit with kill -59 $$, I get:

[7499030.643115] LKRG: ALERT: DETECT: Task: uid corruption (expected 1000 vs. actual 0) for pid 811033, name bash
[7499030.643977] LKRG: ALERT: DETECT: Task: euid corruption (expected 1000 vs. actual 0) for pid 811033, name bash
[7499030.644830] LKRG: ALERT: DETECT: Task: suid corruption (expected 1000 vs. actual 0) for pid 811033, name bash
[7499030.645667] LKRG: ALERT: DETECT: Task: fsuid corruption (expected 1000 vs. actual 0) for pid 811033, name bash
[7499030.646585] LKRG: ALERT: DETECT: Task: gid corruption (expected 1000 vs. actual 0) for pid 811033, name bash
[7499030.646999] LKRG: ALERT: DETECT: Task: egid corruption (expected 1000 vs. actual 0) for pid 811033, name bash
[7499030.646999] LKRG: ALERT: DETECT: Task: sgid corruption (expected 1000 vs. actual 0) for pid 811033, name bash
[7499030.646999] LKRG: ALERT: DETECT: Task: fsgid corruption (expected 1000 vs. actual 0) for pid 811033, name bash
[7499030.646999] LKRG: ALERT: BLOCK: Task: Killing pid 811033, name bash
[7499030.650683] LKRG: ALERT: BLOCK: Task: Killing pid 811033, name bash

When I try to access the reverse shell with python3 scripts/trigger.py 0 (I ran it from localhost, but I think it'd be same from another host if such traffic is permitted - not in my test here), I get:

[7499167.863162] LKRG: ALERT: BLOCK: UMH: Executing program name /bin/bash
[7499168.867229] LKRG: ALERT: BLOCK: UMH: Executing program name /bin/bash
[7499169.867123] LKRG: ALERT: BLOCK: UMH: Executing program name /bin/bash

corresponding to the 3 ICMP packets sent.

(*) but provides for an easy way to check for Singularity presence by shell commands if you know what to check for - e.g., create a directory called singularity or jira and it gets hidden and the parent directory's st_nlink appears non-increased (the first time I see this implemented in a rootkit, breaking my favorite generic method to detect hidden directories - kudos!)

[solardiz]

7. We're also revisiting whether we really want to support the use of ftrace, especially without a way to disable such support. Perhaps we only need to remove or disable part of our code, and Singularity will break (as well as other uses of ftrace).

I found that @Adam-pi3 and I had exchanged some comments on this in #400 (comment) but nothing came out of it back then.

I also wondered (again, after years of discussing this privately before) why we hook the sysctl handler for kernel.ftrace_enabled and disallow changing this setting (if I am reading the code right). I recall/guess this is because we think we don't correctly support ftrace getting enabled (or/and disabled?) after LKRG is loaded. But is this really so? I'm not entirely sure.

Anyway, I found that (successfully) setting sysctl kernel.ftrace_enabled=0 before loading Singularity breaks it, even without LKRG. Singularity gets loaded anyway, but fails to respond e.g. to its magic signal 59. If LKRG is also loaded, then while it is loaded this sysctl can't be simply re-enabled (of course, a modified Singularity could do it from kernel mode, but apparently currently does not?)

[solardiz]

I've just tried this hack:

+++ b/src/modules/database/arch/p_arch_metadata.c
@@ -103,11 +103,13 @@ int p_register_arch_metadata(void) {
    /*
     * Same for FTRACE
     */
+#if 0
    if (p_install_ftrace_modify_all_code_hook(0)) {
       p_print_log(P_LOG_FATAL, "Can't hook 'ftrace_modify_all_code'");
       return P_LKRG_GENERAL_ERROR;
    }
 #endif
+#endif
 
 #if defined(CONFIG_FUNCTION_TRACER)
    if (p_install_ftrace_enable_sysctl_hook(0)) {

which mostly excludes our handling/allowing of kernel modifications made via ftrace.

With the above, when I first load LKRG and then Singularity, everything works/fails the same as without the above patch, as far as I could tell from quick testing - e.g., kill -59 $$ results in LKRG detecting the attack and killing the shell, like before. This is unexpected, right @Adam-pi3? If we felt we had to handle ftrace specially, perhaps it isn't expected to work without tripping an integrity violation without us updating the hashes on those changes?

When I first load Singularity, then LKRG, the system freezes a few seconds later (maybe on the next periodic check), with nothing seen via LKRG remote logging.

I thought it could actually be the other way around - our handling needed to support ftrace hooks installed when we're already monitoring, not before we've started. So I'm puzzled.

[kerneltoast]

Currently, nothing prevents legitimate root from simply unloading LKRG with rmmod or effectively disabling it via sysctl before loading the rootkit.

I've been thinking about this: wouldn't this run afoul of a rootkit's goal of hiding its activities, provided that you have the remote logging set up in LKRG? If your remote logging dies, that would be an indicator that LKRG is gone, no? And if not, perhaps the remote logging should have some kind of keepalive mechanism to indicate LKRG still lives?

[solardiz]

@kerneltoast Yes, I also had such thoughts. LKRG logs a message when it's unloaded or reconfigured, which is instantly sent to the remote system. So it is already somewhat useful against rootkits like Singularity, at least in setups with remote logging and where the logs are actually looked at (or at least would be post-mortem).

There's also a keepalive mechanism, enabled via sysctl lkrg.heartbeat=1, in which case a message is logged (including to the remote) at regular intervals (perhaps a rootkit could fake that, but it's tricky, and anyway the message on unload is a giveaway).

I also thought of adding remote-only heartbeat messages to allow enabling them in cases where having so frequent local kernel messages is undesirable, #309

[Adam-pi3]

OK, testing this on Debian 13, I am able to load Singularity on top of LKRG, and Singularity is not immediately detected, and its hiding of itself from lsmod (shouldn't we detect that?) and hiding of directories works(*). However, LKRG remains functional, so when I try to actively use the rootkit with kill -59 $$, I get (...)

I actually tested it on Ubuntu and have the same results.

I also wondered (again, after years of discussing this privately before) why we hook the sysctl handler for kernel.ftrace_enabled and disallow changing this setting (if I am reading the code right). I recall/guess this is because we think we don't correctly support ftrace getting enabled (or/and disabled?) after LKRG is loaded. But is this really so? I'm not entirely sure.

Most (but not all) of LKRG hooks are being optimized to FTRACE (if CONFIG_OPTPROBES is enabled). If we don't protect that knob, you can disable LKRG by changing it. Of course as a side effect there will be LKRG verification failure at some point. That being said it's a self protection (kind of). I saw that the same trick Singularity is doing (I'm wondering if they "borrow" our idea ;-)).

This is unexpected, right @Adam-pi3? If we felt we had to handle ftrace specially, perhaps it isn't expected to work without tripping an integrity violation without us updating the hashes on those changes?

It depends. If during the same time some other modification happens (which is usually the case during module loading) then we will rebuild the entire hash. That's one of the limitation which we have and I actually discussed that with you @solardiz in the past. We discussed that Instead of recalculating entire .text section hash just update the specific page hash. However, this would mean we would need to change how we do hashing. I think it's and interesting research area but it may be costly (per page hash + modification tracking).

If your remote logging dies, that would be an indicator that LKRG is gone, no? And if not, perhaps the remote logging should have some kind of keepalive mechanism to indicate LKRG still lives?

Yes that's an indicator. However, the more stronger indicator is what @solardiz mentioned: sysctl lkrg.heartbeat=1. The entire idea behind heartbeat was exactly to cover that use-case - you don't see heartbeat? -> something is wrong (but it can be noisy).

and its hiding of itself from lsmod (shouldn't we detect that?) and hiding of directories works(*).

I wrote you some ideas on private message to you @solardiz regarding that. I think we may want to improve our current logic. Let me paste it here:

• Singularity hide itself using LKRG's concept / code. However, we should be able to easily detect it anyway. As of today we don't do it because singularity changes its module state to "MODULE_STATE_UNFORMED". Today we only track "LIVE" modules (it's a conscious decision, if needed we can talk why). However, as we can see it has limitation, we can have a simple logic that if we see the module with the state non-LIVE for more than 2-3 (we can tweak the number) verification routines it is obviously wrong because those other states (like "GOING" or "UNFORMED") supposed to have a very short and temporary lifetime.

I also believe we may start thinking how to "boost" the hiding option which also will make some of the offensive tools work more annoying:

• make hide enabled by default (as we track it on Consider enabling lkrg.hide=1 by default #456)
• TZ-LKRG
• generate random LKRG module name (instead of always being lkrg)
  -> during compilation we may generate a random string which we can be a module's name. Later we make a symlink to it an lkrg.ko name
• randomize symbols name (so you can't just simply try to search for lkrg-sepcific function names)

@solardiz @kerneltoast what's your take on all of those points (including detection of non-live module state) ?

[kerneltoast]

@Adam-pi3

Singularity hide itself using LKRG's concept / code. However, we should be able to easily detect it anyway. As of today we don't do it because singularity changes its module state to "MODULE_STATE_UNFORMED". Today we only track "LIVE" modules (it's a conscious decision, if needed we can talk why). However, as we can see it has limitation, we can have a simple logic that if we see the module with the state non-LIVE for more than 2-3 (we can tweak the number) verification routines it is obviously wrong because those other states (like "GOING" or "UNFORMED") supposed to have a very short and temporary lifetime.

Why not just probe do_init_module()? You can compute the hash for the module's text section at that point (no need to hold module_mutex), and then if do_init_module() returns 0 you know for certain that the module loaded successfully. That way, the module can't play tricks on its state variable.

TZ-LKRG

What does TZ mean here?

generate random LKRG module name (instead of always being lkrg)
-> during compilation we may generate a random string which we can be a module's name. Later we make a symlink to it an lkrg.ko name

A kernel module with a gibberish name would be very conspicuous. This can be trivially defeated by creating a whitelist of kernel module names.

randomize symbols name (so you can't just simply try to search for lkrg-sepcific function names)

Or just strip out the symbol table from the binary.

[Adam-pi3]

Why not just probe do_init_module()? You can compute the hash for the module's text section at that point (no need to hold module_mutex), and then if do_init_module() returns 0 you know for certain that the module loaded successfully. That way, the module can't play tricks on its state variable.

We used to have those hook but over the time we changed their functionalities and eventually dropped it. However, we can restore it. That being said, the problem is a bit different, when module is in not-LIVE state it may still be "forming" (which means relocation may need to be applied, JUMP-LABELs/FTRACEs not yet applied etc) and we can't reliably confirm/snapshot the .text. Yes, we will then update the modules .text since we do monitor those modifications anyway so maybe it's not a big deal (we would need to test it). Module can be also going away, which will have similar functionalities as "forming" one (and we would need to additionally remove it from our DB). What I mean, we can try but there are some corner cases which we need to take into account including modification of the verification routine (the biggest value is that we do monitor the linked list order itself, so if module hides, we know that order changed but module never went away). Verification routine must take into account that module list may not have yet the newly formed module (but we want to know that) but .text section should be guarded regardless of the state etc. What I'm proposing should be effective and simpler (but difficult the judge without research).

What does TZ mean here?

That's one of my old idea of enhancing LKRG via true-RO memory from more priv level (although i wrote it a few years ago so some tweaks may be needed):
https://site.pi3.com.pl/priv/TZ-LKRG.txt

A kernel module with a gibberish name would be very conspicuous. This can be trivially defeated by creating a whitelist of kernel module names.

Yes, however, that's not uncommon for some security tools. When GMER become effective and popular, they faced also similar problem of simple string-matching and one simple solution was when you download it, GMER generates a random name too (virus/malware didn't delete the file due to failure of string matching):
https://en.wikipedia.org/wiki/GMER

What I mean, if someone is using LKRG I assume the user is advanced enough to understand that (if we communicate it clearly). We can create a 'loader' which generate a random name (and rewrite the strings) before loading the module (instead of compile time) - we can use random GUID?

Or just strip out the symbol table from the binary.

Yes, that would work too. We just need to be sure to remove anything what can uniquely identify LKRG (including non-function strings)

[kerneltoast]

Why not just probe do_init_module()? You can compute the hash for the module's text section at that point (no need to hold module_mutex), and then if do_init_module() returns 0 you know for certain that the module loaded successfully. That way, the module can't play tricks on its state variable.

We used to have those hook but over the time we changed their functionalities and eventually dropped it. However, we can restore it. That being said, the problem is a bit different, when module is in not-LIVE state it may still be "forming" (which means relocation may need to be applied, JUMP-LABELs/FTRACEs not yet applied etc) and we can't reliably confirm/snapshot the .text.

The module's .text is finalized by the time do_init_module() is called, so you can snapshot it at that point.

A kernel module with a gibberish name would be very conspicuous. This can be trivially defeated by creating a whitelist of kernel module names.

Yes, however, that's not uncommon for some security tools. When GMER become effective and popular, they faced also similar problem of simple string-matching and one simple solution was when you download it, GMER generates a random name too (virus/malware didn't delete the file due to failure of string matching): https://en.wikipedia.org/wiki/GMER

The problem I'm pointing out is that there are not enough arbitrary kernel modules used on real systems. Because the list of possible kernel modules in use on a typical real system is finite, relatively small, and doesn't change much, that means someone can just make a whitelist of all legitimate kernel module names and assume that something not in their whitelist has a high probability of being LKRG with a randomized name.

In other words: although this technique works well in userspace, I don't think it works well for kernel modules.

[solardiz]

Oh, turns out there's already a response by @MatheuZSecurity:

## [Released] - 2026-02-02

### Changed

**LKRG Bypass Module - Complete Rewrite**
- Removed all LKRG internal function hooks (p_cmp_creds, p_cmp_tasks, p_check_integrity, etc.)
- New approach: hook kernel functions that LKRG uses instead of LKRG's own functions
- Direct manipulation of LKRG's global control structure (p_lkrg_global_ctrl)
- Memory offset-based control disable/restore (UMH validation, enforcement, PINT validation/enforcement)
- Hooks now target: vprintk_emit, signal functions (do_send_sig_info, send_sig_info, __send_signal_locked, force_sig), usermodehelper functions (call_usermodehelper_exec_async, call_usermodehelper_exec)
- Log filtering system intercepts and suppresses LKRG kernel messages
- SIGKILL interception prevents LKRG from killing hidden processes
- UMH protection bypass during usermode helper execution
- Automatic LKRG detection via symbol presence check
- Module notifier waits for LKRG load then locates control structure

**Technical Changes:**
- Removed 12 LKRG-internal hooks
- Added 7 kernel function hooks
- Added LKRG control structure offsets (UMH_VALIDATE: 0x30, UMH_ENFORCE: 0x34, PINT_VALIDATE: 0x08, PINT_ENFORCE: 0x0c)
- Added log buffer (512 bytes) with spinlock protection
- Added saved state variables for control restoration
- PID extraction from log messages for filtering
- Enhanced lineage checking (up to 64 levels)

### Impact

This version shifts from hooking LKRG's detection functions to disabling LKRG's protections at the source:
- More reliable bypass via direct control structure manipulation
- Prevents LKRG from detecting integrity violations instead of hiding from checks
- Suppresses all LKRG kernel log output for hidden processes
- Blocks LKRG from terminating hidden processes via signal interception
- Cleaner UMH bypass with automatic enable/disable during execution
- Better compatibility across LKRG versions (fewer internal function dependencies)

[kerneltoast]

Oh, turns out there's already a response by @MatheuZSecurity:

## [Released] - 2026-02-02

### Changed

**LKRG Bypass Module - Complete Rewrite**
- Removed all LKRG internal function hooks (p_cmp_creds, p_cmp_tasks, p_check_integrity, etc.)
- New approach: hook kernel functions that LKRG uses instead of LKRG's own functions
- Direct manipulation of LKRG's global control structure (p_lkrg_global_ctrl)
- Memory offset-based control disable/restore (UMH validation, enforcement, PINT validation/enforcement)
- Hooks now target: vprintk_emit, signal functions (do_send_sig_info, send_sig_info, __send_signal_locked, force_sig), usermodehelper functions (call_usermodehelper_exec_async, call_usermodehelper_exec)
- Log filtering system intercepts and suppresses LKRG kernel messages
- SIGKILL interception prevents LKRG from killing hidden processes
- UMH protection bypass during usermode helper execution
- Automatic LKRG detection via symbol presence check
- Module notifier waits for LKRG load then locates control structure

**Technical Changes:**
- Removed 12 LKRG-internal hooks
- Added 7 kernel function hooks
- Added LKRG control structure offsets (UMH_VALIDATE: 0x30, UMH_ENFORCE: 0x34, PINT_VALIDATE: 0x08, PINT_ENFORCE: 0x0c)
- Added log buffer (512 bytes) with spinlock protection
- Added saved state variables for control restoration
- PID extraction from log messages for filtering
- Enhanced lineage checking (up to 64 levels)

### Impact

This version shifts from hooking LKRG's detection functions to disabling LKRG's protections at the source:
- More reliable bypass via direct control structure manipulation
- Prevents LKRG from detecting integrity violations instead of hiding from checks
- Suppresses all LKRG kernel log output for hidden processes
- Blocks LKRG from terminating hidden processes via signal interception
- Cleaner UMH bypass with automatic enable/disable during execution
- Better compatibility across LKRG versions (fewer internal function dependencies)

Cool, Singularity has been forced to move onto non-portable ABI hacks, like hardcoded struct offsets. 🙂

[Adam-pi3]

The module's .text is finalized by the time do_init_module() is called, so you can snapshot it at that point.

Yes, we can do it (and verify return code first if it successful).

Oh, turns out there's already a response by [MatheuZSecurity]
...
Cool, Singularity has been forced to move onto non-portable ABI hacks, like hardcoded struct offsets. 🙂

We can randomize structure layout. Randomizing/stripping symbols is also something which will be useful. Changing how modules are tracked is also necessary,
Another idea is to verify if all LKRG's hooks are still present during verification routine.

In other words: although this technique works well in userspace, I don't think it works well for kernel modules.

Hm.. I see. However, reliable allow-list would not be as reliable on real-case system I assume (custom EDRs, custom drivers) but it may grow over-the-time. Yeah