Deadman's switch activated...killing process

[darrinh]

HI all,
Had trouble this morning with my workstation, it keep freezing when performing the same operation. I started fapolicyd in debug mode and performed those operations again, the output was:

Starting to listen for events

Mount change detected

Error (Permission denied) adding fanotify mark for /run/user/1000/doc

Mount change detected

Added /tmp/snap.rootfs_ALVYdD mount point

Mount change detected

Deleted /tmp/snap.rootfs_ALVYdD mount point

Mount change detected

Added /run/snapd/ns/firefox.mnt mount point

Mount change detected

Added /mnt mount point

Deadman's switch activated...killing process
Killed

The operations being performed was starting thunderbird, starting firefox, decrypting and mounting a usb drive, then using rsync to copy files from that drive to my local home dir, on every occasion it caused a freeze. Any ideas on what is going on? There doesn't seem to be anything related in the syslog. And why does this not activate when running in daemon mode instead of locking up the workstation?

Ubuntu 22.04
fapolicyd 1.1.3-105

[darrinh]

The version I'm using seems kind of old, will try a newer version.

[darrinh]

It may have been related to errors from the usb drive(?), on each occasion there was a long pause during which fapolicyd crashed out (deadmans switch), then copying resumed:

kernel: [  425.650463] sd 10:0:0:0: [sdd] tag#0 Sense Key : Hardware Error [current] 
kernel: [  425.650467] sd 10:0:0:0: [sdd] tag#0 Add. Sense: No additional sense information
kernel: [  425.650471] sd 10:0:0:0: [sdd] tag#0 CDB: Read(10) 28 00 13 f9 46 50 00 00 40 00
kernel: [  425.650474] I/O error, dev sdd, sector 335103568 op 0x0:(READ) flags 0x80700 phys_seg 8 prio class 0

but can't be 100% certain it was related as the debug output from fapolicyd isn't timestamped.

[darrinh]

I notice when fapolicyd has been running all day, the 'object slots in use' gets to 99% , does it mean the setting need more adjustment ?

[stevegrubb]

This is a timeout where the main decision thread is kind of hung up. So, it likely is related to the hardware failure. As to whether or not to increase the object cache, it depends on if you are getting lots of evictions. 99% is good utilization. But if you have lots of evictions, then bump it up. There is a performance section of the README.md page that describes how to tune it.

[Certezalito]

I got the same error Deadman's switch activated...killing process when attempting to run a debug via:
fapolicyd --permissive --debug-deny
If I stop Trend Micro DSM Agent Service, ds_agent then I can run a debug.

[stevegrubb]

We can increase the timeout. But the decision on the file is already taking 3 seconds. It should take microseconds.